Thursday, July 31, 2008

How to Report on Ease of Use?

Yesterday’s post on classifying demand generation systems prompted some strong reactions. The basic issue is how to treat ease of use when describing vendors.

It’s hard to even define the issue without prejudicing the discussion. Are we talking about vendor rankings, vendor comparisons, or vendor analyses?

- Ranking implies a single score for each product. The approach is popular but it leads people to avoid evaluating systems against their own requirements. So I reject it.

- Vendor comparisons give each several scores to each vendor, for multiple categories. I have no problem with this, although it still leaves the question of what the categories should be.

- Vendor analyses attempt to describe what it's like to use a product. This is ultimately what buyers need to know, but it doesn’t lead directly to deciding which product is best for a given company.

Ultimately, then, a vendor comparison is what’s needed. Scoring vendors on several categories will highlight their strengths and weaknesses. Buyers then match these scores against their own requirements, focusing on the areas that are important to them. The mathematically inclined can assign formal weights to the different categories and generate a combined score if they wish. In fact, I do this regularly as a consultant. But the combined scores themselves are actually much less important than the understanding gained of trade-offs between products. Do we prefer a product that is better at function A than function B, or vice versa? Do we accept less functionality in return for lower cost or higher ease of use? Decisions are really made on that basis. The final ranking is just a byproduct.

The question, then, is whether ease of use should be one of the categories in this analysis. In theory I have no problem with including it. Ease of use does, however, pose some practical problems.

- It’s hard to measure. Ease of use is somewhat subjective. Things that are obvious to one person may not be obvious to someone else. Even a concrete measure like the time to set up a program or the number of keystrokes to accomplish a given task often depends on how familiar users are with a given system. This is not to say that usability differences don’t exist or are unmeasurable. But it does mean they are difficult to present accurately.

- ease depends on the situation. The interface that makes it easy to set up a simple project may make it difficult or impossible to handle a more complicated one. Conversely, features that support complex tasks often get in the way when you just want to do something simple. If one system does simple things easily and another does complicated things easily, which gets the better score?

I think this second item suggests that ease of use should be judged in conjunction with individual functions, rather than in general. In fact, it’s already part of a good functional assessment: the real question is usually not whether a system can do something, but how it does it. If the “how” is awkward, this lowers the score. This is precisely why I gather so much detail about the systems I evaluate, because I need to understand that “how”.

This leads me pretty much back to where I started, which is opposed to breaking out ease of use as a separate element in a system comparison. But I do recognize that people care deeply about it, so perhaps it would make sense to assess each function separately in terms of power and ease of use. Or, maybe some functions should be split into things like “simple email campaigns” and “complex email campaigns”. Ease of use would then be built into the score for each of them.

I’m still open to suggestion on this matter. Let me know what you think.


Jon Miller said...

I think your line "Do we accept less functionality in return for lower cost or higher ease of use? Decisions are really made on that basis" is the key thing to focus on. I believe those ('functionality' and 'ease') are the correct two dimensions to evaluate vendors.

First, keep in mind is that although technical folks like you or me may be comfortable with complex tools, it’s certainly not true for most marketers, especially the “generalists” who you typically find in demand generation roles. Most marketers do not have the time or desire to have to go to training or “get the hang of” complex tools – they want to be able to do things themselves, now.

Second, let’s not get too caught up in only usability. There are other important dimensions related to “ease” as opposed to “functional richness”. These include important factors like:
• IT-involvement required to get up and running / maintain the system (related to, but not exclusively, to on-premise vs. on-demand)
• Deployment time & costs (e.g. integration to CRM, website)
• Start-up / training time & costs
• Total time to first campaign, value
• Ongoing costs: dedicated resources, support, training
• Ease of buying: upfront payment required or monthly payments allowed; size of initial payment; overall cost

If you talk to customers, I believe you will hear that these issues matter as much or more to them than having extra features – and that this is exactly the right trend for marketing automation, which has been inhibited by bloated tools that most marketers couldn’t use.

As Ian Michaels of Aberdeen said, “In many cases, companies must balance the need for robust features with the complexity of the marketing automation tool.” Another consultant, Howard Sewell of CDI, said “There are products on the marketplace today that require a team of systems consultants to implement. The power of an effective marketing automation system is its flexibility – i.e. the ease with which you can customize and adapt your communications based on different audiences, product sets, demographics, changes in business or market conditions, or user activity.”

David Raab said...

Thanks Jon. (For readers who don't know him, Jon is VP of Marketing at Marketo.) The items you've listed are very similar to the components of Total Cost of Ownership (TCO). Certainly there's a case to be made that a reasonable matrix would compare cost vs. functionality. If I were looking for a way to rank vendors, that would be a sound approach.

But if you go back to my original post, that wasn't the goal. I was specifically looking for a needs-based way to segment customers, so I could then classify systems according to how well they matched the needs of each group. Now, you could quite validly argue that usability, or ease, or TCO is a customer need, and that it's useful for segmentation because it applies to different customers in different degrees. But I think that particular set of needs is a function of customer resources (those with fewer resources are more sensitive to cost, ease, etc.), and that resources are a function of marketing program size and complexity. Since I already had a dimension for complexity, I would just be measuring the same thing twice. (Conversely, if you argue that all marketers care about usability, etc., then it's not a meaningful segmentation dimension.)

Still, we're ultimately classifying systems, not marketers. It's certainly true that there are significant differences in usability/ease/TCO to do the same functions with different products. The question then becomes whether this particular dimension is the one I want to highlight given that I can only have two.

I'm still wrestling with that. One thing to bear in mind is that I'm trying to define a set of products for people to consider, not to recommend a single product they should select. The consideration set includes all products that meet their functional requirements. I want them to look at all those products before making a choice. I fully expect that major differences in usability, etc. will show up clearly when they do.

Somewhat ironically, it is actually harder for buyers to uncover the concrete functional differences between products during the purchase process than it is to see the differences in usability, etc. So once more I'm led back to the idea that I can be most useful to them by focusing on functionality.

To put it more succinctly: I expect that buyers will first determine which products meet their needs, and then choose the one that is most usable/easy/economical. My little matrix is intended to help them with the first step in the process, not the second.