Case Study: Using a Scenario to Select Business Intelligence Software

Summary: Testing products against a scenario is critical to making a sound selection. But the scenario has to reflect your own requirements. While this post shows results from one test, rankings could be very different for someone else.

I’m forever telling people that the only reliable way to select software is to devise scenarios and test the candidate products against them. I recently went through that process for a client and thought I’d share the results.

1. Define Requirements. In this particular case, the requirements were quite clear: the client had a number of workers who needed a data visualization tool to improve their presentations. These were smart but not particularly technical people and they only did a couple of presentations each month. This mean the tool had to be extremely easy to use, because the workers wouldn’t find time for extensive training and, being just occasional users, would quickly forget most of they had learned. They also wanted to do some light ad hoc analysis within the tool, but just on small, summary data sets since the serious analytics are done by other users earlier in the process. And, oh, by the way, if the same tool could provide live, updatable dashboards for clients to access directly, that would be nice too. (In a classic case of scope creep, the client later added mapping capabilities to the list, merging this with a project that had been running separately.)

During our initial discussions, I also mentioned that Crystal Xcelsius (now SAP Crystal Dashboard Design) has the very neat ability to embed live charts within Powerpoint documents. This became a requirement too. (Unfortunately, I couldn’t find a way to embed one of those images directly within this post, but you can click here to see a sample embedded in a pdf. Click on the radio buttons to see the different variables. How fun is that?)

2. Identify Options. Based on my own knowledge and a little background research, I built a list of candidate systems. Again, the main criteria were visualization, ease of use and – it nearly goes without saying – low cost. A few were eliminated immediately due to complexity or other reasons. This left:

• Xcelsius


• Advizor Solutions


• TIBCO Spotfire


• QlikView


• Lyzasoft


• Tableau Software

3. Define the Scenario. I defined a typical analysis for the client: a bar chart comparing index values for four variables across seven customer segments. The simplest bar chart showed all segment values for one variable. Another shows all variables for all segments, sorted first by variable and then by segment, with the segments ranked according to response rate (one of the five variables). This would show how the different variables related to response rate. It looked like this:


The tasks to execute the scenario were:

• connect to a simple Excel spreadsheet (seven segments x four variables.)*


• create a bar chart showing data for all segments for a single variable.


• create a bar chart showing data for all segments for all variables, clustered by variable and sorted by the value of one variable (response index).


• provide users with an option to select or highlight individual variables and segments.

Because my requirements assumed users would have little or no training, I specified that the scenarios be performed without taking time to learn each system. This made the testing easy, but I should stress it's an unusual situation: in most cases, systems are run by people who use them often enough to become experts. For situations like that, you should have an experienced user (often a vendor sales rep or engineer) execute the scenario for you. In fact, one of the most common errors we see is people judging a system by how easily they can run it without training – something that favors systems which are easy to use for simple tasks, but lack the functional depth clients will eventually need to do their real jobs.

4. Results. I was able to download free or trial versions of each system. I installed these and then timed how long it took to complete the scenario, or at least to get as far as I could before reaching the frustration level where a typical end-user would stop.

I did my best to approach each system as if I’d never seen it before, although in fact, I’ve done at least some testing on every product except SpotFire, and have worked extensively with Xcelsius and QlikView. As a bit of a double-check, I dragooned one of my kids into testing one system when he was innocently visiting home over Thanksgiving: his time was actually quicker than mine. I took that as a proof I'd tested fairly.

Notes from the tests are below.

• Xcelsius (SAP Crystal Dashboard Design): 3 hours to set up bar chart with one variable and allowing selection of individual variables. Did not attempt to create chart showing multiple variables. (Note: most of the time was spent figuring out how Xcelsius did the variable selection, which is highly unintuitive. I finally had to cheat and use the help functions, and even then it took at least another half hour. Remember that Xcelsius is a system I’d used extensively in the past, so I already had some idea of what I was looking for. On the other hand, I reproduced that chart in just a few minutes when I was creating the pdf for this post. Xcelsius would work very well for a frequent user, but it’s not for people who use it only occasionally.)
  

• Advizor: 3/4 hour to set up bar chart. Able to show multiple variables on same chart but not to group or sort by variable. Not obvious how to make changes (must click on a pull down menu to expose row of icons).

• Spotfire: 1/2 hour to set up bar chart. Needed to read Help to put multiple lines or bars on same chart. Could not find way to sort or group by variable.
  

• QlikView: 1/4 hour to set up bar chart (using default wizard). Able to add multiple variables and sort segments by response index, but could not cluster by variable or expose menu to add/remove variables. Not obvious how to make changes (must right-click to open properties box – I wouldn’t have known this without my prior QlikView experience).

• Lyzasoft: 1/4 hour to set up bar chart with multiple variables. Able to select individual variables, cluster by variable and sort by response index, but couldn’t easily assign different colors to different variables (required for legibility). Annoying lag each time chart is redrawn.

• Tableau: 1/4 hour to set up bar chart with multiple variables. Able to select individual variables, cluster by variable, and sort by variable. Only system to complete the full scenario.

Let me stress again that these results apply only to this particular scenario. Specifically, the ability to cluster the bars by segments within variables turned out to be critical in this test but doesn't come up very often in the real world. Other requirements, such as advanced collaboration, sophisticated dashboards or specialized types of graphics, would have yielded very different ranks.

5. Final Assessment. Although the scenario nicely addressed ease of use, there were other considerations that played into the final decision. These required a bit more research and some trade-offs, particularly regarding the Xcelsius-style ability to embed interactive charts within a Powerpoint slide. No one else on the list could do this without either loading additional software (often a problem when end-user PCs are locked down by corporate IT) or accessing an external server (a problem for mobile users and with license costs).

The following table shows my results:


6. Next Steps. The result of this project wasn’t a final selection, but a recommendation of a couple of products to explore in depth. There were still plenty of details to research and confirm. However, starting with a scenario greatly sped up the work, narrowed the field, and ensured that the final choice would meet operational requirements. That was well worth the effort. I strongly suggest you do the same.


____________________________________

* The actual data looked like this. Here's a link if you want to download it: