Demand Generation Usability Scores - Part 1

(note: this is a slightly revised version of the original post, reflecting vendor feedback.)

For better or worse, I've taken my usability scoring project as far as I can. So, having run out of reasons to delay, let me present the results.

To start back at the beginning, the objective has been to find a practical, objective way to evaluate the usability of different demand generation systems. Lacking the resources for a detailed user survey or hands-on scenario testing, I chose to build a checklist of functions that I believe correlate with usability.

But I quickly ran into a problem. Because usability includes both ease-of-use and the suitability for a given task, a single usability score would be misleading because it must either favor construction of simple campaigns or of complex campaigns. I therefore chose to develop separate scores for each. The idea was to remind people that the question of which system is "best" has no simple answer precisely because different systems are good at different things. My hope is that people would then take the next logical step of assessing systems against their own requirements.

In working through the checklist items themselves, I quickly realized that some items apply to usability for both simple and complex campaigns. I therefore ended up with three groups of scoring elements: those for simple campaigns, those for complex campaigns, and those shared by both.

The final piece of background is my definition of simple vs. complex campaigns. In both cases, I see the basic flow as an outbound email, landing page, multi-step nurturing campaign, lead scoring, and transfer to a CRM system for sales followup. A simple campaign would do this for a single product, offer, customer segment and region, while a complex campaign could involve several of each. Obviously these aren't very specific scenarios, but I think the ability to efficiently deliver many different treatments to different customers is ultimately what separates simple from complex in a demand generation context.

On to the checklist items themselves. I tried to find items that could be judged objectively as present or not, without too much evaluation on my part of how well or poorly they had been implemented. This turned out to be reasonably easy and has the major benefit that I can assign a 1 or 0 in nearly all cases. The only exceptions were a couple of cases where it seemed most fair to give a vendor half-credit.

The harder part was deciding which items correlated with usability. Here I considered the functions needed to execute simple and complex campaigns, and focused on functions that made those campaigns easy to set up and run.

This means I excluded functions I consider important but not themselves directly related to ease of use. Or, more precisely, I excluded functions that are more or less equally easy to use in the different products. For example, every demand generation system provides an editor to create emails. But these are generally so similar that they don't really factor into differences in system usability.

Where complex campaigns are concerned, this approach also means I excluded functions having to do with the scope or sophistication of a system rather than bearing directly on ease of use. As I discussed in one of the earlier posts laying out this project, my final scoring system will include separate sophistication scores as well. Similarly, a final scoring system should include other items such as vendor viability, or at least a proxies such as years in business, numbers of clients, and funding. Once more, the logic is to provide enough different scores that people are led to consider which of those scores are important to their own business.

Since the actual scores for most items were 1 or 0, I simply added up the item scores to get composite scores for each vendor. I considered weighting different items, but it didn't appear that any reasonable set of weights would have much impact on the relative rankings of the different vendors. So it seemed best not to bother.

Okay then. Let's look at the items I've chosen and how I've scored the vendors listed in the Raab Guide to Demand Generation Systems. (Five of these were in the original Guide. Marketbright was added on March 3, and Neolane will be added in a week or two once we finalize their entry.)

Rather than overwhelm everyone with a single, huge blog post, I'm going to break this into three parts. This post will cover the shared items. The next will cover simple campaign items, and a third will cover complex campaign items. A final post will summarize and discuss the results.

One final caveat on the scores: they're based on my best information about the vendors, but it's possible something has changed or I missed something in my research. I expect the vendors will let me know if they have questions, and will certainly adjust the scores if appropriate.

Shared Items

These items apply to both simple and complex campaigns. They relate primarily to creation of marketing assets such as emails and landing pages, and to execution of lead scoring.

Select marketing assets from shared libraries. Users can draw on existing libraries of marketing assets when setting up a campaign, rather than creating them from scratch. These assets can be modified or used as is. Even though every system listed below can do this, it's included because some other products might not.

Select marketing assets from shared libraries
Eloqua	Manticore Technology	Market2Lead	Marketbright	Marketo	Neolane	Silverpop Engage B2B
1	1	1	1	1	1	1

Text search for assets. Users can enter a search string, such as a word or phrase, and get a list of all assets having that string in their name. This makes it easier to find specific assets without keeping track of their exact name or which campaigns used them previously. Again, although every system on this list has this capability, systems not listed might not.

Text search for assets
Eloqua	Manticore Technology	Market2Lead	Marketbright	Marketo	Neolane	Silverpop Engage B2B
1	1	1	1	1	1	1

Share marketing assets across campaigns. The same asset can be used in multiple campaigns without creating a new copy. This saves effort if an asset must be updated.

Share marketing assets across campaigns
Eloqua	Manticore Technology	Market2Lead	Marketbright	Marketo	Neolane	Silverpop Engage B2B
1	1	1	0	1	1	1

Live templates for asset frames. Assets are built in layered templates with common elements such as headers, footers and styles. The bodies of these assets may be different. The templates are "live" in the sense that a change to the template is applied to all assets using that template, even if the assets are already deployed to a campaign. Like shared assets, this saves effort if a common element must be updated. Silverpop gets a half point because it can share templates for Web forms, but not emails.

Live templates for asset frames
Eloqua	Manticore Technology	Market2Lead	Marketbright	Marketo	Neolane	Silverpop Engage B2B
1	1	1	0	1	1	.5

Trigger lead scoring outside a campaign step. The system will update lead scores without users building explicit steps into their campaigns to trigger this update. This simplifies campaign creation and ensures that scores are always current. Typically, scores are updated automatically after a data change. Sometimes, they are updated on a regular schedule instead.

Trigger lead scoring outside a campaign step
Eloqua	Manticore Technology	Market2Lead	Marketbright	Marketo	Neolane	Silverpop Engage B2B
1	1	1	1	1	1	1

Central definition of lead scoring rules. Scoring rules are defined in a central location rather than separately for each campaign. This saves effort and ensures consistency.

Central definition of lead scoring rules
Eloqua	Manticore Technology	Market2Lead	Marketbright	Marketo	Neolane	Silverpop Engage B2B
1	1	1	1	1	1	1

Total Score for Shared Items

Total Score for Shared Items
Eloqua	Manticore Technology	Market2Lead	Marketbright	Marketo	Neolane	Silverpop Engage B2B
6	6	6	4	6	6	5.5

The next post will look at scores for items specific to simple campaigns.

First Look at New Marketo Release

I’m going to diverge just slightly from my current obsession with usability to talk about a conversation I had today with Marketo President and CEO Phil Fernandez, who previewed the 3.0 release of his flagship product, scheduled for March 3.

The changes that Fernandez described seemed good but subtle. Major themes were greater access to detailed information, more precise targeting, and tighter integration with Salesforce.com. The company also reworked the user interface--of more than 200 total changes in this upgrade, about 75 were tied to usability--although it still takes the same basic approach of building campaigns as lists of steps, rather than branching flow charts. In Fernandez’ view, this reflects a fundamental philosophical difference from his competitors: Marketo sees marketing as reacting to prospect-initiated behaviors, not executing company-driven interaction paths. Although I actually think that quite a few demand generation vendors share the Marketo philosophy, it’s still helpful to hear the distinction made clearly.

What's ultimately more important than the uniqueness of Marketo's philosophy is how they have built it into their software. In Marketo, each campaign is a relatively small, self-contained sequence of steps that is triggered by a particular prospect need. Fernandez used the analogy of a cocktail party: you might have a few stories you expect to tell, but don’t know exactly when you’ll tell them or in what order. The simplicity of these individual campaigns is what lets Marketo use a simple interface to build them. There is certainly more to it than that—the company has a fanatical devotion to usability—but I’d argue that being structured around simple campaigns is the key to Marketo’s well-deserved reputation as a system that’s easy to use.

The challenge with all these simple campaigns is the same as the challenge of telling stories at a cocktail party: you have to be sure to tell the right story to the right person at the right time. Marketo’s approach is to trigger campaigns based on specified events or list criteria. But since this by itself won’t coordinate the separate campaigns, Marketo also lets users set up other campaigns (which sound suspiciously like flow charts, although they are not displayed that way) whose rules send prospects to one campaign or another. This is a perfectly straightforward approach, although I suspect that marketers may find it hard to keep track of the selection rules as they get increasingly complicated.

Here is where it’s worth considering the approaches of other vendors. Products including Silverpop Engage B2B (formerly Vtrenz), Market2Lead and Marketbright also let marketers set up small, sequential campaigns and embed them in selection framework. Each vendor takes a different approach to building this framework, and, like Marketo, they can all be difficult to grasp once you pass a certain threshold of complexity.

Still, I think its fair to say that the general approach of simple campaigns linked in a decision matrix is a more effective way to implement non-linear, prospect-driven interactions than conventional flow charts. My own experience over the years has been that flow charts quickly become too complicated for most marketers to deal with.

Hmm, I seem to have fallen back into a discussion of usability without planning to. You can just imagine how much fun I am at a cocktail party. Fernandez and I did in fact discuss other things, including Marketo’s phenomenal growth (he hopes to sign his 150th customer any day now), its success in selling to non-technology companies (healthcare, manufacturing and business services have been strong), and the growing importance of demand generation systems as buyers take control of the purchasing process.

That last point morphed into a discussion of the increased integration between marketing and sales. Today's prospects repeatedly bounce between the two during hyper-extended sales cycles that begin earlier and are far less linear. That resonated with me because I’ve noticed several other demand generation vendors offering features that cross into traditional sales department territories, such as prospect portals and reseller management.

In Marketo’s case, though, it’s less a matter of adding sales-type functions than tightening its integration with Salesforce.com. Changes to support this include faster response times, more extensive data sharing, and more precise control over synchronization rules. Still, the general point is the same: marketing and sales must cooperate more closely than ever. It's always interesting to see how a thoughtful company like Marketo looks at a trend like this and decides to react.

Simplifying Demand Generation Usability Assessment: No Obvious Answers

My feelings are hurt, people. No one has commented on last week’s post about usability measurement. I know it’s not the world’s most fascinating topic but I really wanted some feedback. And I do, after all, know how many people visit the site each day. Based on those numbers, there are a lot of you who have chosen not to help me.

Oh well, no grudges here -- ‘tis the season and all that. I’m guessing the reason for the lack of comment is that the proposed methodology was too complex for people to take the time to assess and critique. In fact, the length of the post itself may be an obstacle, but that in turn reflects the complexity of the approach it describes. Fair enough.

So the question is, how do you simplify the methodology and still provide something useful?

- One approach would be to reduce the scope of the assessment. Instead of defining scenarios for all types of demand generation processes, pick a single process and just analyze that. Note that I said “single” not “simple”, because you want to capture the ability of the systems to do complicated things as well. This is a very tempting path and I might try it because it’s easy to experiment with. But it still raises all the issues of how you determine which tasks are performed by which types of users and how you account for the cost of hand-offs between those users. This strikes me as a very important dimension to consider, but I also recognize that it introduces quite a bit of complexity and subjectivity into the process. I also recognize that measuring even a single process will require measuring system set-up, content creation and other preliminary tasks. Thus, you still need to do a great deal of work to get metrics on one task, and that task isn’t necessarily representative of the relative strengths of the different vendors. This seems like a lot of effort for a meager result.

- Another approach would be to ask users rather than trying to run the tests independently. That is, you would do a survey that lists the various scenarios and asks users to estimate the time they require and how this is distributed among different user types. That sounds appealing insofar as now someone else does the work, but I can’t imagine how you would get enough data to be meaningful, or how you would ensure different users’ responses were consistent.

- A variation of this approach would be to ask vastly simpler questions – say, estimate the time and skill level needed for a half-dozen or so typical processes including system setup, simple outbound email campaign, setting up a nurturing campaign, etc. You might get more answers and on the whole they’d probably be more reliable, but you’re still at the mercy of the respondents’ honesty. Since vendors would have a major incentive to game the system, this is a big concern. Nor is it clear what incentives users would have to participate, how you screen out people with axes to grind, or whether users would be constrained by non-disclosure agreements from participating. Still, this may be the most practical of all the approaches I’ve come up with, so perhaps it’s worth pursuing. Maybe we get a sample of ten clients from each vendor? Sure they’d be hand-picked, but we could still hope their answers would accurately reflect any substantial differences in workload by vendor.

- Or, we could ask the vendors to run their own tests and report the results. But who would believe them? Forget it.

- Maybe we just give up on any kind of public reporting, and provide buyers with the tools to conduct their own evaluations. I think this is a sound idea and actually mentioned it in last week’s post. Certainly the users themselves would benefit, although it’s not clear how many buyers really engage in a detailed comparative analysis before making their choice. (For what it’s worth, our existing usability worksheet is the third most popular download from the Raab Guide site. I guess that’s good news.) But if the results aren’t shared, there is no benefit to the larger community. We could offer the evaluation kit for free in return for sharing the results, but I doubt this is enough of an incentive. And you’d still have the issue of ensuring that reported results are legitimate.

So there you have it, folks. I’m between a rock and a hard place. No matter how much I talk about usability, the existing Raab Guide mostly lists features, and people will use it to compare systems on that basis. But I can’t find a way to add usability to the mix that’s both objective and practical. Not sure where to go next.

Marketbright Targets Sophisticated Demand Generation Users

I had a preliminary conversation last week with Mike Pilcher of Marketbright, one of the vendors I’ll probably end up adding to the Raab Guide to Demand Generation Systems. We didn’t look at the software itself, so I can’t comment on it in any detail. The slides did list a few unusual features, including “prospect portals” that help buyers and sellers to share information related to a project; a sales proposal builder; and features to work with sales partners. These seem pretty minor, although they do insert Marketbright more deeply into the sales process than most demand generation products. This is something of a theme for the company, although all demand generation vendors integrate closely with sales systems.

Marketbright sees its most important differentiator as a sophisticated architecture designed to coordinate marketing activities throughout a large organization. This doesn't strike me as a very effective selling point: buying a product because of its architecture is the software equivalent of reading Playboy for the articles. (Do I get credit for resisting the temptation to link to Playboy.com?) What really matters are the features facilitated by this architecture. According to the company Web site, these include “full document repository and asset management, multi-currency budget planning and management and a range of integrated collaboration features”. Now that's something to get excited about. Hubba hubba, eh?

At least this clarifies which end of the demand generation market will find Marketbright most attractive. Indeed, Pilcher told me that product sells best to people who have worked in large organizations and seen first-hand what it takes to support collaboration within marketing. These people may currently be working in small firms, so Marketbright has ended up with customers of all sizes. Pricing ranges from $20,000 to $200,000 per year based on the modules and number of users, so the system is financially competitive facross the spectrum.

Not having seen the product, I don’t know whether its sophisticated management features come at the price of end-user complexity. This is a common trade-off. One hint of Marketbright’s approach may be that Pilcher recommends his clients build separate campaigns for different customer segments, rather than “boiling the ocean” by creating a single campaign with branches to handle all contingencies. This suggests that Marketbright has at least tried to keep things as simple.

Pilcher and I had a lengthy subsequent email discussion about usability, “violently agreeing” that it’s an important though elusive measure. My final conclusion was similar to the positions I’ve taken before: usability has to be measured separately for different functions, levels of campaign sophistication, and user skill sets. Where I may have changed my mind is a grudging agreement that it’s legitimate to summarize the details into simple measures that could be plotted in a graph. The obvious ones are usability and functionality scores. I still fear this could mislead by obscuring important information: for example, deep functionality in a few areas could generate the same score as limited functionality across many areas. (Pilcher proposed the number of channels as a separate dimension, but then a system with weak functionality in many channels scores better than a system that is strong in just a few. I consider that equally misleading.) But if a two-dimensional summary offers an attractive entry point which in turn leads to deeper exploration, it’s better than scaring people away by showing them the details at the start.

Usability Is Just One Piece of the Puzzle

A funny thing happened as I was writing one of my usual rants on incorporating usability into the selection process. (The resulting paper is on the http://www.raabguide.com/ site, creatively titled "Building Usability into Your System Selection".)

After a few bon mots that probably no one else will find clever ("Usability is hard to measure; features are easy to count" "Small hard facts beat big blurry realities") I got to describing the steps in a usability-aware selection process:

1. define business needs
2. define processes to meet those needs
3. define tasks within each process
4. identify systems to consider, then, for each system:
5. determine which users will do each task
6. determine how much work each task will be
7. compare, rank and summarize the results

As a point of comparison, it's steps 3, 5 and 6 that differ from the conventional selection process. Step 3 in a conventional process would identify features needed rather than tasks, while steps 5 and 6 would be replaced with research into system features.

What I realized as I was writing this was that the real focus is not on usability, but on defining processes and tasks. Usability measures are something of a by-product. In fact, the most natural way to implement this approach would be to score each system for each task, with a single score that incoporates both functionality and as ease of use. Indeed, as I wrote not long ago, standard definitions of usability include both these elements, so this is not exactly an original thought.

Still, it does mean I have to restructure the terms of the debate (at least, the one inside my head). It's not usability vs. features, but process vs. features. That is, I'm essentially arguing that selection processes should invest their effort in understanding the company business processes that the new system must support, and in particular in which the tasks different users will perform.

The good news here is that you'll eventually need to define, or maybe redefine, those processes, tasks and user roles for a successful implementation. So you're not doing more work, but simply doing the implementation work sooner. This means a process-focused evaluation approach ultimately reduces the total work involved, as well as reducing implementation time and improving the prospects for success. By contrast, time spent researching system features is pretty much a waste once the selection process is complete.

Of course, this does raise the question of whether the feature information assembled in the Raab Guide to Demand Generation Systems is really helpful. You won't be surprised to find I think it is. This is not so much because of the feature checklist (truly my least favorite section) but because the Guide tries to show how the features are organized, which directly impacts system usability. Plus, of course, the absence of a necessary feature makes a system unusable for that particular purpose, and that is the biggest usability hit of all. What the Guide really does is save readers the work of assembling all the feature information for themselves, thereby freeing them to focus on defining their own business processes, tasks and users.

In conclusion, you should all go and buy the Guide immediately.

Free Usability Assessment Worksheet!

I won’t claim a direct cause-and-effect relationship, but is it really just a coincidence that the stock market finally had a good day exactly when my new Guide to Demand Generation Systems is about to be released? Think about it.

That said, the new Guide Web site is in the final testing and should be launched tomorrow. It might even be working by the time you read this: try http://www.raabguide.com/. The Guide itself has been circulating in draft among the vendors for about two weeks. The extra time was helpful since it allowed a final round of corrections triggered by the yes/no/maybe comparison matrix.

I still feel this sort of matrix oversimplifies matters, but it does seem to focus vendors’ attention in a way that less structured descriptions do not. In fact, I’m wondering whether I should drop the structured descriptions altogether, and just show the matrix categories with little explanatory notes. Readers would lose some nuance, but if nobody pays attention to the descriptions anyway, it might be a good choice for future editions. It would certainly save me a fair amount of work. Thoughts on the topic are welcome (yes, I know few of you have actually seen the Guide yet. I’m still considering how to distribute samples without losing sales.)

Part of my preparation for the release has been to once more ponder the question of usability, which is central to the appeal of several Guide vendors. A little external research quickly drove home the point that usability is always based on context: it can only be measured for particular users for particular functions in particular situations. This was already reflected in my thinking, but focusing on it did clarify matters. It actually implies two important things:

1. each usability analysis has to start with a definition of the specific functions, users and conditions that apply to the purchasing organization. This, in turn, means

2. there’s no way to create a generic usability ranking.

Okay, I’ll admit #2 is a conclusion I’m very happy to reach. Still, I do think it’s legitimate. More important, it opens a clear path towards a usability assessment methodology. The steps are:

- define the functions you need, the types of users who perform each function, and the conditions the users will work under. “Types of users” vary by familiarity with the system, how often they use it, their administrative rights, and their general skill sets (e.g. marketers vs. analysts vs. IT specialists). The effort required for a given task varies greatly for different user types, and so do the system features that are most helpful. To put it in highway terms: casual users need directions and guardrails; experienced users like short cuts.

“Conditions” are variables like the time available for a task, the number of tasks to complete, the cost of making an error, and external demands on the user’s time. A system that’s optimized for one set of conditions might be quite inefficient under another set. For example, a system designed to avoid errors through careful review and approvals of new programs might be very cumbersome for users who don’t need that much control.

- assess the effort that the actual users will spend on the functions. The point is that having a specific type of user and set of conditions in mind makes it much easier to assess a system’s suitability. Ideally, you would estimate the actual hours per year for each user group for each task (recognizing that some tasks may be divided among different user types). But even if you don't have that much detail, you should still be able to come up with a score that reflects which systems are more easier to use in a particular situation.

- if you want to get really detailed, break apart the effort associated with each task into three components: training, set-up (e.g. a new email template or campaign structure), and execution (e.g. customizing an email for a particular campaign). This is the most likely way for labor to be divided: more skilled users or administrators will set things up, while casual users or marketers will handle day-to-day execution. This division also matches important differences among the systems themselves: some require more set-up but make incremental execution very easy, while others need less set-up for each project but allow less reuse. It may be hard to actually uncover these differences in a brief vendor demonstration, but this approach at least raises the right question and gives a framework for capturing the answers.

- after the data is gathered, summarize it in a traditional score card fashion. If the effort measures are based on hours per year, no weighting is required; if you used some other type of scoring system, weights may be needed. You can use the same function list for traditional functionality assessments, which boil down to the percentage of requirements (essential and nice-to-have) each system can meet. Functional scores almost always need to be weighted by importance. Once you have functionality and usability scores available, comparing different systems is easy.

In practice, as I’ve said so many times before, the summary scores are less important than the function-by-function assessments going into them. This is really where you see the differences between systems and decide which trade-offs make the most sense.

For those of you who are interested, I’ve put together a Usability Assessment Worksheet that supports this methodology. This is available for free on the new Guide Web site: just register (if registration is working yet) and you’ll be able to download it. I’ll be adding other resources over time as well—hopefully the site will evolve into a useful repository of tools.