Monday, February 16, 2009

How to Compare Demand Generation Vendors: Choosing Summary Measures

I’ve more or less decided to offer a free summary of the demand generation vendor information in the Raab Guide. This might cost me a few sales, but it will enable many more people to benefit from the Guide’s contents and will make the Guide more valuable to the sponsoring vendors.

This raises the issue of exactly what should be in the summary listings. It seems that what people really want is an easy way to identify the best candidates for their particular situation. (Duh.) This suggests the summary should contain two components: a self-evaluation where people describe their situation, and a scoring mechanism to compare their needs with vendor strengths and weaknesses.

Categories for the self-evaluation seem pretty obvious: they would be the standard demand generation functions (outbound email, landing pages and forms, nurturing campaigns, lead scoring, and Salesforce.com integration), maybe a menu for less standard functions (e.g. events like Webinars, paid and organic search, online chat, direct mail, telemarketing, partner management, etc.), price range, and willingness to consider less established vendors. Once you’ve settled on these, you have the vendor ratings categories too, since they have to align with each other. That lets you easily find the vendors with the highest scores in your highest priority areas. Simple enough.

Except...some marketers only need simple versions of these functions while other marketers need sophisticated versions. But every system in the Guide can meet the simple needs, so there’s no point to setting up a separate scores for those: everyone would be close to a perfect 10. On the other hand, the vendors do differ in how easily they perform those basic tasks, and it’s important to capture that in the scoring.

The literal-minded solution is to score vendors on their ease-of-use for each of the functions. Sounds good, and as soon as someone offers me a couple hundred thousand dollars to sit with a stopwatch and time users working with different systems, I’ll get right on it. Until that happens, I need a simpler but still reasonably objective approach. Ideally, this would allow me to create the scores based on the information I’ve already assembled in the Guide.

My current thinking is to come up with a list of specific list of system attributes that make it easy to do basic functions, and then create a single “ease of use for basic tasks” score for each vendor. As a practical matter, this makes sense because I don’t have enough detail to create separate scores for each function. Plus, I’m pretty sure that each vendor’s scores would be similar across the different functions. I also think most buyers would need similar levels of sophistication across the functions as well. So I think one score will work, and be much simpler for readers to deal with.

The result would be a matrix that looks something like the one below. It would presumably be accompanied by a paragraph or two thumbnail description of each vendor:


Ease of Basic
Tasks

Sophistication of Tasks

Other Tasks (list)

Ease of Purchase (Cost)

Vendor Strength

email campaigns

Web forms/
pages

nurture campaigns

lead scoring

Salesforce integration

Vendor A

5

8

4

8

4

9

4

7

5

Vendor B

8

4

8

4

7

4

7

5

8

Vendor C

4

8

4

9

4

7

5

8

6

...

...

...

...

...

...

...

...

...

...



So the first question is: would people find this useful? Remember that the detail behind the numbers would be available in the full Raab Guide but not as part of the free version.

The second question is: what attributes could support the “ease of basic tasks” score? Here is the set I’m considering at the moment. (Explanations are below; I’m listing them here to make this post a little easier to read):

- build a campaign as a list of steps.

- explicitly direct leads from one campaign to another.

- define campaign schedules and entry conditions at the campaign level, not for individual steps.

- select marketing contents from shared libraries.

- build decision rules from prebuilt functions for specific situations, such as ‘X web page visits in past Y days’.

- build split tests into content, not decision flows.

- explicitly define lead scoring as a campaign step, but have it point to a central lead scoring function.

- explicitly define lead transfer to sales as a campaign step, but have it point to a central lead transfer function.

These items are all fairly easy to judge: either a system works that way or it doesn’t. So in that sense they’re objective. But they’re subjective in that some people may not agree that they actually make a system easier to use for basic tasks. Again, I’d much prefer direct measures like the number of keystrokes or time required the complete a task. But I just don’t see a cost-effective way to gather that information.

Other measures I’d use in my scoring would be the amount of training provided during implementation, typical implementation time, and typical time for users to become proficient. These should be good indications of the difficulty of basic tasks. I do have some of this data, although it’s only what they vendors have told me. Answers will also be affected by differences in vendor deployment practices and in how sophisticated their users are. So I can’t give these too much weight.

Let me know what you think. Do these measures make sense? Are there others I can gather? Is there a better way to do this?

++++++++++

Explanations of the “ease of basic tasks” attributes:

- build a campaign as a list of steps. Each step can be an outbound message (email), an inbound message (landing page or form), or an action such as sending the lead for scoring or to another campaign. Campaigns can be saved as templates and reused, but the initial construction starts with a blank list. (This seems counterintuitive; surely it would be simpler to start with a prebuilt template? Yes for experienced marketers, but not for people who are just starting, who would have to explore the templates to figure out what they contain. For those people, it’s actually easier to start with something they’ve created for themselves and therefore fully understand.) Another aspect to this is that many marketers prefer a list of steps rather than a flow chart, even though they are logically the same. The reason is that simple campaigns tend to be very linear, so a flow chart adds more complexity than necessary. (Of all the items on this list, this is the one I’m probably least committed to. But the really simple systems I’ve seen do use lists more than flow charts, so I think there’s something to it.)

- explicitly direct leads from one campaign to another. Even basic marketing programs need to move leads from one treatment sequence to another, based on changes in lead status or behaviors. But building many different paths into a single campaign flow quickly becomes overwhelming. The easiest solution for basic marketing programs is to create separate, relatively simple campaigns and explicitly direct leads from one campaign to another at steps within the campaign designs. This approach works well only when there are relatively few campaigns in the program. More complex programs require other solutions, such as using rules to send different treatments to different people within the same campaign, having each campaign independently scan the database for leads that meet its acceptance criteria, or embedding small sequences (such as an email, landing page, thank you page and confirmation email) as a single step in a larger campaign. Systems built to handle sophisticated campaign requirements sometimes provide an alternative, simpler interface to handle simple campaign designs.

- define campaign schedules and entry conditions at the campaign level, not for individual steps. Basic marketing campaigns will have a single, campaign-level schedule and set of entry conditions. Even campaigns fed by other campaigns often impose additional entry criteria, for example to screen out leads who have already completed a particular campaign. Users will look for the schedule and entry conditions among the campaign attributes, not as attributes of the first step in the campaign or of a separate list attached to the campaign. More sophisticated marketing programs may also give each step its own schedule and/or entry conditions, instead of the campaign attributes or in addition to them.

- select marketing contents from shared libraries. The libraries contain complete documents with shared elements such as headers, footers, layouts and color schemes, plus a body of text and data entry fields. Marketers can either use a document as-is or make a copy and then edit it. In practice, nearly every system has this sort of library, so it’s not a differentiator. But it’s still worth listing just in case you come across a system that doesn’t. More sophisticated approaches, such as templates shared across multiple documents, only add efficiency for more advanced marketing departments.

- build decision rules from prebuilt functions for specific situations, such as ‘X web page visits in past Y days’. This saves users from having to figure out how to build those functions themselves, which can be quite challenging. In the example given, it would require knowing which field in the database is used to measure Web page visits, how to write a query that counts something entries in that field, and how to limit the selection to a relative time period. There are a great many ways to get these wrong.

- build split tests into content, not decision flows. The easiest way to test different versions of an email or landing page is to treat the versions as a single item in the campaign flow, and have the system automatically serve the alternatives during program execution. The common alternative approach is to directly split the campaign audience, either at the start of the campaign or at a stage in the flow directly before the split itself. This results in a more complex campaign flow and is often more difficult to analyze.

- explicitly define lead scoring as a campaign step, but have it point to a central lead scoring function. This is a compromise between independently defining the lead scoring rules every time they’re used, and automatically scoring leads (after each event or on a regular schedule) without creating a campaign step at all. Independent score definitions are a great deal of extra work and can lead to painful inconsistencies, so it’s pretty clear why you would want to avoid them. Doing the scoring automatically is actually simpler, but may confuse marketers who don’t see the scoring listed as a step in their campaign flow.

- explicitly define lead transfer to sales as a campaign step, but have it point to a central lead transfer function. The logic here is the same as applies to lead scoring. A case could be made that transfer to sales should in fact be part of the lead scoring, but if nothing else you’d still want marketers to decide on for each campaign whether sending a lead to sales should also remove the lead from the current campaign. Once you have campaign-level decisions of that sort, the process should be a step in the campaign flow.

5 comments:

Jon Miller said...

David -- I think a summary is a great idea, and as you know we consistently hear usability as the #1 factor driving vendor selection these days.

A few comments on your list:

- We've done quite a bit of customer research on this, and building split tests into content is an advanced feature, not a basic one.

- Defining the list for a campaign combining demographic and behavioral filters should be on the list of basic tasks

- Updating data values as a campaign step should be a basic task

One more thought: Ease of use is not necessarily the same as usability. Usability is how it feels, how it works, the little elements (like auto-complete for dropdowns) that make using the system faster, and more fun. Think of how an iPhone animates when you scroll through pictures; without that animation, it would be just as easy to use but would have less usability.

David Raab said...

Thanks Jon. Your points are very well taken. Any other specific things like auto-complete that would be on your 'usability' list?

Steven Woods said...

David,
I like your approach to analyzing vendors in the space. We’ve worked with a lot of teams doing vendor evaluation on usability, and they have usually used the following approach:

- Identify a business scenario/challenge and what it would take to achieve success
- Understand the functionality/approach needed to tackle that scenario
- Assess the usability of that functionality/approach

This achieves the same end goal as what you describe, but is more top-down than bottom-up. In guide form, it also allows guide users to quickly decide whether they care about the business scenario or not. This avoids the advanced/basic split, which can be a bit challenging as a rare/advance usage scenario for one organization can be a core need for another organization.

Take lead scoring, as one individual example, the following business challenges can be explored:

- Do you need to score leads differently based on a strategic/named account structure?
o If so, you’ll need a way to identify named accounts (and their various ways of being written), and applying a score
o Vendors can be assessed on the usability of setting up this aspect of the scoring process

- Do you need to score based on data that is often non-standardized, like title or industry (V.P., Vice Pres, Vice President, etc)
o If so you’ll need a way of handling or standardizing the data that forms the basis for scoring
o Vendors can be assessed on the usability of setting up those processes and rules

- Do you want to differentiate between a score on “who” the prospect is (explicit data) vs “how interested” they are (implicit data) in order to understand where they are in the buying process
o If so, you’ll need a way of handling both aspect of this score and building appropriate rules based on it
o Vendors can be assessed on how easily they are able to create and manage these rules

Similar business case assessments can be made with any other aspects that are common to today’s systems, which will give your audience a much deeper view into how they should be assessing their selection.

The clients we’ve worked with who have done a good job of assessing usability have followed this business-case-first methodology, and it might be one worth following in your guide as it allows an assessment of usability in your guide that is specific to each businesses unique needs.

Hope that helps, glad to chat further if you’re interested,
Steve

David Raab said...

Thanks Steve. I absolutely agree that use cases are the right way to assess usability. But as your example so well illustrates, even one small feature involves many options that must be assessed separately. Multiply that by the many features that people need, the variations in how they use those features, and the number of vendors to assess, and you quickly come up with an impossibly large and expensive project. Plus, sadly, most readers wouldn't take the time to look at the details even if you made them available, so it's not clear that anyone would benefit from all that work. Hence my quest for a small feature checklist that works as a proxy for the larger analysis. In a way, NOT providing the detail actually makes it more obvious to buyers that they have to do the detailed work for themselves.

A couple other notes while I'm commenting here:

- I'm thinking of creating a parallel list of features that measure "ease of use for sophisticated tasks", since that is also relevant. Then the users would simply have to decide whether they are "basic" or "sophisticated" and could pick the vendors to consider based on that. I don't mean to be cynical, but this really seems to be what people want.

- regarding use cases, one thing I do consider practical is to build a library of cases that people could use as the basis for their own evaluations. This goes back to the "cookbook" I mentioned in an earlier post. I haven't had time to flesh this out but it seems worth pursuing. Maybe people would "donate" recipes, although I suppose putting together a basic set wouldn't be that hard. Or maybe vendors could each donate one recipe and illustrate how to do it with their tool?

Unknown said...

I fully agree with Jon: Ease of use is not necessarily the same as usability.

Great job Raab!