Demand Generation Usability Scores - Part 4

I spent a good part of yesterday talking with demand generation vendors about the usability scores I've been publishing all week. As a result, I've made a number of small adjustments which have been retroactively edited into the previous posts. There might still be some others: two of the vendors haven't spoken with me, which could mean they agree with my ratings but, more likely, means they just haven't been paying attention.

Perhaps I'm just starved for attention, but I do in fact appreciate the feedback. The main result of the conversations so far has been to clarify the items related to lead routing. The relative ranking of the different vendors hasn't been significantly affected.

Here, then, is the final usability matrix.

	Eloqua	Manticore Technology	Market2Lead	Marketbright	Marketo	Neolane	Silverpop Engage B2B
complex	11	8.5	12	9.5	9.5	12	11
simple	8	10	10	10	12	8.5	9

Now what? Although the focus of these last three posts has been on usability scoring, you may remember that the goal described in the blog post that started all this was to develop a summary comparison of demand generation vendors that marketers could match against their needs.

The idea is to quickly identify the most promising candidates so you have more time to explore them in depth. You can do that on your own through endless vendor meetings and random conversations with your peers, or can save time, effort and risk by purchasing the detailed, objective information conveniently assembled and beautifully packaged in the Raab Guide to Demand Management Systems for the low price of $595, satisfaction guaranteed or your money back. Choose whichever sounds best.

Anyway, the usability scores were the hardest measures to develop, which is why I did them first. What remains are scores for functionality, pricing and vendor background. Those are easy to create because the Raab Guide (buy now! all major credit cards accepted!) has already scored each vendor on more than 150 data points. All I need to do is add up the scores by category and summarize the results.

In fact, that's exactly what I did this morning, and I've been staring at the results for the past several hours.

But I can't show them to you.

Here's the problem. Usability requirements are roughly similar for all marketers. Yes, some features apply more to simple marketing programs and others to complex programs. But everyone needs their system to be intuitive, understandable, efficient, easy to learn, and the rest. So once you decide whether your focus is simple or complex marketing programs, the scores for those categories are a pretty good indicator of which systems you'll find most usable.

But functional requirements are different. Every company has a specific set of needs that are addressed by a specific set of features. In a study as broad-ranging as the Raab Guide, any one of those features will get lost when you aggregate scores into broad categories. So you can't just look at those categories to determine which systems fit your requirements.

For example, let's say that events like breakfast seminars are critical to your marketing program. Specialized features to manage those campaigns are available from several vendors, but they're not necessarily the ones with the highest scores for all campaign management features combined. If you decided to limit your search to the top vendors in that category, you'd miss at least one good candidate.

In other words, publishing aggregate functionality scores, or any type of simple vendor ranking, would actually do more harm than good. I've been saying this for years, but had briefly talked myself into publishing them because I know how popular they would be. I've now come back to my senses.

Still, I hate to disappoint those of you who were looking for more information. So here's what I will do.

First, I'll tell you that the aggregate functionality scores largely correlate with the scores for complex usability. That makes sense: vendors who have built powerful systems have tuned them to be usable in complex situations.

Second, I'll give you a matrix of that shows which systems provide features that are hard to find. At the end of the day, this is what you really need to know to decide which systems to explore in depth. Since we've already collected this information in the Guide (and so much more! order now and start reading in minutes!), it's not much work at all.

Of course, I do realize that publishing this matrix lead to many more phone calls from vendors who want to know why they weren't listed in a given category. That's okay, since I do want to be as accurate as possible. And it doesn't hurt that I'm about to leave town for a few days.

Eloqua	Manticore Technology	Market2Lead	Marketbright	Marketo	Neolane	Silverpop Engage B2B
Marketing Channels Beyond Email and System-generated Web Pages
0	0	0	1	0	0	0	dealer / channel management (register and co-promote with channel partners)
1	0	0	0.5	0	0	0	track RSS readership (publish RSS feeds and track who reads each item)
0.5	0	0	0	0	0.5	0	capture costs of Pay Per Click advertising programs directly from the vendor
0	0	0	0.5	0	0	0.5	online chat (display a button to request chat; monitor visitors and proactively offer a chat when appropriate)
1	0	0	0	0	0.5	0	fax (special relationships with broadcast fax vendors)
1	0	0.5	0	0	0	0.5	call center scripting (telephone agents can work from system screens)
1	0	0	0	0	1	0	mobile (special relationships with mobile message distribution vendors)
0	1	1	0	0	0	0	track conversions and stages via URLs (define which Web pages are conversions in a sales funnel and track leads as they visit them)
0	0	1	1	0	1	0	call vendor offers from external pages (display system-generated marketing offers in externally-hosted Web pages, such as the company Web site)
1	0	1	0	0	1	0.5	integrate w/direct mail printer (directly transmit personalized marketing materials)
1	0	1	1	0	0.5	0	events: quantity limits, wait lists, reminders (specialized campaigns to manage online and real-world events)
Tailoring Messages to Individuals
0	0	1	0	0	0.5	0	select next action based on highest value (automatically send different messages within a campaign stage based on lead characteristics and behavior history)
0	0	1	0.5	0	1	0	show highest-value offer on form (automatically display highest-value offers on a form based on lead characteristics and behaviors)
0.5	0.5	1	0.5	0	1	0	rule-based form customization (display different content blocks in an email or Web page based on user-defined rules)
Marketing Administration
0	0	0	0.5	0	1	0	project schedule (manage a schedule of tasks related to executing a campaign; includes workflow for copy approvals)
0	0	0	0	0	1	0	calculate costs from unit cost x volume (calculate campaign costs for line items based on cost per unit and selection or response quantity)
0	0	1	0	0	1	0	campaign cost detail (capture line items for costs and revenues within a campaign; track different types such as budget, plan, estimated, actual, etc.)
Content Management
0.5	0	0	0	0	1	0	expiration dates (assign expiration dates to marketing assets; warn users if an expired asset is still in use)
0.5	0	0.5	0.5	0	0.5	0	item-level security (assign access rights to specific marketing assets by individual user or user group)
0	0	0.5	1	0	1	1	check in/out, version control (manage asset creation to ensure only one person at a time is making changes; allow users to review and recall old versions)
Database Management
0	0	1	0	0	0	0	separate analytical (cube) database (copy operational tables into a separate database for reporting and analysis)
1	0	1	0	0	1	0.5	custom tables (allow users to add custom data tables to the standard data structure)

How to Use This Information

Armed with the usability scores and the preceding matrix, you should be able to narrow set of candidate vendors fairly quickly. Start by identifying your critical needs, and check if they're listed in the matrix. If so, the vendors that support your needs are the ones to start with. If not, decide whether your marketing program leans more towards the simple or complex, and choose the vendors with the best usability ratings in your group.