How to Compare Demand Generation Vendors: Choosing Summary Measures

I’ve more or less decided to offer a free summary of the demand generation vendor information in the Raab Guide. This might cost me a few sales, but it will enable many more people to benefit from the Guide’s contents and will make the Guide more valuable to the sponsoring vendors.

This raises the issue of exactly what should be in the summary listings. It seems that what people really want is an easy way to identify the best candidates for their particular situation. (Duh.) This suggests the summary should contain two components: a self-evaluation where people describe their situation, and a scoring mechanism to compare their needs with vendor strengths and weaknesses.

Categories for the self-evaluation seem pretty obvious: they would be the standard demand generation functions (outbound email, landing pages and forms, nurturing campaigns, lead scoring, and Salesforce.com integration), maybe a menu for less standard functions (e.g. events like Webinars, paid and organic search, online chat, direct mail, telemarketing, partner management, etc.), price range, and willingness to consider less established vendors. Once you’ve settled on these, you have the vendor ratings categories too, since they have to align with each other. That lets you easily find the vendors with the highest scores in your highest priority areas. Simple enough.

Except...some marketers only need simple versions of these functions while other marketers need sophisticated versions. But every system in the Guide can meet the simple needs, so there’s no point to setting up a separate scores for those: everyone would be close to a perfect 10. On the other hand, the vendors do differ in how easily they perform those basic tasks, and it’s important to capture that in the scoring.

The literal-minded solution is to score vendors on their ease-of-use for each of the functions. Sounds good, and as soon as someone offers me a couple hundred thousand dollars to sit with a stopwatch and time users working with different systems, I’ll get right on it. Until that happens, I need a simpler but still reasonably objective approach. Ideally, this would allow me to create the scores based on the information I’ve already assembled in the Guide.

My current thinking is to come up with a list of specific list of system attributes that make it easy to do basic functions, and then create a single “ease of use for basic tasks” score for each vendor. As a practical matter, this makes sense because I don’t have enough detail to create separate scores for each function. Plus, I’m pretty sure that each vendor’s scores would be similar across the different functions. I also think most buyers would need similar levels of sophistication across the functions as well. So I think one score will work, and be much simpler for readers to deal with.

The result would be a matrix that looks something like the one below. It would presumably be accompanied by a paragraph or two thumbnail description of each vendor:

	Ease of Basic Tasks	Sophistication of Tasks					Other Tasks (list)	Ease of Purchase (Cost)	Vendor Strength
		email campaigns	Web forms/ pages	nurture campaigns	lead scoring	Salesforce integration
Vendor A	5	8	4	8	4	9	4	7	5
Vendor B	8	4	8	4	7	4	7	5	8
Vendor C	4	8	4	9	4	7	5	8	6
...	...	...	...	...	...	...	...	...	...

So the first question is: would people find this useful? Remember that the detail behind the numbers would be available in the full Raab Guide but not as part of the free version.

The second question is: what attributes could support the “ease of basic tasks” score? Here is the set I’m considering at the moment. (Explanations are below; I’m listing them here to make this post a little easier to read):

- build a campaign as a list of steps.

- explicitly direct leads from one campaign to another.

- define campaign schedules and entry conditions at the campaign level, not for individual steps.

- select marketing contents from shared libraries.

- build decision rules from prebuilt functions for specific situations, such as ‘X web page visits in past Y days’.

- build split tests into content, not decision flows.

- explicitly define lead scoring as a campaign step, but have it point to a central lead scoring function.

- explicitly define lead transfer to sales as a campaign step, but have it point to a central lead transfer function.

These items are all fairly easy to judge: either a system works that way or it doesn’t. So in that sense they’re objective. But they’re subjective in that some people may not agree that they actually make a system easier to use for basic tasks. Again, I’d much prefer direct measures like the number of keystrokes or time required the complete a task. But I just don’t see a cost-effective way to gather that information.

Other measures I’d use in my scoring would be the amount of training provided during implementation, typical implementation time, and typical time for users to become proficient. These should be good indications of the difficulty of basic tasks. I do have some of this data, although it’s only what they vendors have told me. Answers will also be affected by differences in vendor deployment practices and in how sophisticated their users are. So I can’t give these too much weight.

Let me know what you think. Do these measures make sense? Are there others I can gather? Is there a better way to do this?

++++++++++

Explanations of the “ease of basic tasks” attributes:

- build a campaign as a list of steps. Each step can be an outbound message (email), an inbound message (landing page or form), or an action such as sending the lead for scoring or to another campaign. Campaigns can be saved as templates and reused, but the initial construction starts with a blank list. (This seems counterintuitive; surely it would be simpler to start with a prebuilt template? Yes for experienced marketers, but not for people who are just starting, who would have to explore the templates to figure out what they contain. For those people, it’s actually easier to start with something they’ve created for themselves and therefore fully understand.) Another aspect to this is that many marketers prefer a list of steps rather than a flow chart, even though they are logically the same. The reason is that simple campaigns tend to be very linear, so a flow chart adds more complexity than necessary. (Of all the items on this list, this is the one I’m probably least committed to. But the really simple systems I’ve seen do use lists more than flow charts, so I think there’s something to it.)

- explicitly direct leads from one campaign to another. Even basic marketing programs need to move leads from one treatment sequence to another, based on changes in lead status or behaviors. But building many different paths into a single campaign flow quickly becomes overwhelming. The easiest solution for basic marketing programs is to create separate, relatively simple campaigns and explicitly direct leads from one campaign to another at steps within the campaign designs. This approach works well only when there are relatively few campaigns in the program. More complex programs require other solutions, such as using rules to send different treatments to different people within the same campaign, having each campaign independently scan the database for leads that meet its acceptance criteria, or embedding small sequences (such as an email, landing page, thank you page and confirmation email) as a single step in a larger campaign. Systems built to handle sophisticated campaign requirements sometimes provide an alternative, simpler interface to handle simple campaign designs.

- define campaign schedules and entry conditions at the campaign level, not for individual steps. Basic marketing campaigns will have a single, campaign-level schedule and set of entry conditions. Even campaigns fed by other campaigns often impose additional entry criteria, for example to screen out leads who have already completed a particular campaign. Users will look for the schedule and entry conditions among the campaign attributes, not as attributes of the first step in the campaign or of a separate list attached to the campaign. More sophisticated marketing programs may also give each step its own schedule and/or entry conditions, instead of the campaign attributes or in addition to them.

- select marketing contents from shared libraries. The libraries contain complete documents with shared elements such as headers, footers, layouts and color schemes, plus a body of text and data entry fields. Marketers can either use a document as-is or make a copy and then edit it. In practice, nearly every system has this sort of library, so it’s not a differentiator. But it’s still worth listing just in case you come across a system that doesn’t. More sophisticated approaches, such as templates shared across multiple documents, only add efficiency for more advanced marketing departments.

- build decision rules from prebuilt functions for specific situations, such as ‘X web page visits in past Y days’. This saves users from having to figure out how to build those functions themselves, which can be quite challenging. In the example given, it would require knowing which field in the database is used to measure Web page visits, how to write a query that counts something entries in that field, and how to limit the selection to a relative time period. There are a great many ways to get these wrong.

- build split tests into content, not decision flows. The easiest way to test different versions of an email or landing page is to treat the versions as a single item in the campaign flow, and have the system automatically serve the alternatives during program execution. The common alternative approach is to directly split the campaign audience, either at the start of the campaign or at a stage in the flow directly before the split itself. This results in a more complex campaign flow and is often more difficult to analyze.

- explicitly define lead scoring as a campaign step, but have it point to a central lead scoring function. This is a compromise between independently defining the lead scoring rules every time they’re used, and automatically scoring leads (after each event or on a regular schedule) without creating a campaign step at all. Independent score definitions are a great deal of extra work and can lead to painful inconsistencies, so it’s pretty clear why you would want to avoid them. Doing the scoring automatically is actually simpler, but may confuse marketers who don’t see the scoring listed as a step in their campaign flow.

- explicitly define lead transfer to sales as a campaign step, but have it point to a central lead transfer function. The logic here is the same as applies to lead scoring. A case could be made that transfer to sales should in fact be part of the lead scoring, but if nothing else you’d still want marketers to decide on for each campaign whether sending a lead to sales should also remove the lead from the current campaign. Once you have campaign-level decisions of that sort, the process should be a step in the campaign flow.