Summary: Web usability guru Jakob Nielsen has proven that simple usability tests are highly effective. Marketing automation vendors should take heed. Come to think of it, so should marketers.
A vendor very proudly showed me their new Adobe Flex-based user interface the other day. Flex is a “rich Internet application” technology, meaning it gives you drag-and-drop, pop-up windows, and other features of a desktop graphical user interface. I found the new interface a bit confusing, so I asked whether the vendor had done any usability testing. He said they hadn’t done anything formal, because of the cost, but had shown it to many current users who were very enthusiastic about the change. Fair enough.
By coincidence, a random Twitter post this morning pointed me to the blog of Web usability guru Jakob Nielsen. The specific post had to do with a study of mobile phone usability, which you won’t be surprised to find is dismal. But what particularly struck me was the relative simplicity of Nielsen’s methods – the study involved just a few dozen users doing selected tasks and its primary metric was the straightforward one of success rate. (I don’t mean to suggest the study itself is simple – the full report runs 132 pages and costs $198.)
Poking a bit more around Nielsen’s blog and Web site, I saw that simple usability studies are a recurring theme. For example, he showed back in 2000 Why You Only Need to Test with Five Users and made a more recent case for Fast, Cheap, and Good testing methods. Indeed, yet another post on Guesses vs. Data as Basis for Design Recommendations demonstrates that testing even two users is better than guessing.
Now, it’s true that Nielsen is measuring Web site usability, which is considerably less complicated than usability for a software application such as a marketing automation system. But I still think his point and his methods are valid: even a little usability testing goes a long way to help designers make the right choices. It’s something I hope more vendors will keep in mind.
Of course, Nielsen's work is even more directly applicable to the landing pages and other Web site components that marketers construct for their own use. Most marketers don't test enough in general, so you can see usability testing as just one example of the larger problem. But usability testing methods are different enough from standard marketing tests to think about them separately. The fact that they can be done very simply and before a page or site is launched actually means there is less reason not to do them, and greater value when you do.
Showing posts with label software usability measurement. Show all posts
Showing posts with label software usability measurement. Show all posts
Thursday, July 23, 2009
Wednesday, April 08, 2009
Pedowitz Group Offers Free Support for New Eloqua Clients
The Pedowitz Group announced this morning that it was offering $15,000 in free consulting services and guaranteeing a five-day implementation to new clients who purchase Eloqua demand generation software . (Click here for the announcement.)
(If you’re not familiar with The Pedowitz Group, President and CEO Jeff Pedowitz ran the professional services group at Eloqua for several years before starting the company. The firm also works with Marketo, Silverpop Engage B2B and MarketingGenius depending on client needs. )
My initial reaction to the announcement was to wonder if it would be interpreted as evidence that Eloqua requires a lot of consulting to implement. Certainly that’s how I’d spin it if I were a competitor. But then again, maybe I wouldn't, because it focuses attention on how much support is really needed to deploy other systems.
On the surface, this is a strength of vendors who promise free implementation and deployment within a few days. But the reality is that most marketers need outside help to design their email campaigns, nurture programs, scoring rules, and CRM integration. This has less to do with learning the software than with knowing what works and how to apply it to their own business.
Yes, some products really are easier to use than others, especially for simple programs. That's one reason Pedowitz works with several vendors. (Download the Raab Guide report on Vendor Usability Scores for more on this.) And yes, some marketers will get their system running with no more than telephone support.
But it’s just plain silly to think that most marketers can quickly deploy sophisticated demand generation programs without some expert help. This is what’s highlighted by the Pedowitz Group offer – and it’s a discussion that vendors selling the dream of an instant deployment should probably avoid.
(If you’re not familiar with The Pedowitz Group, President and CEO Jeff Pedowitz ran the professional services group at Eloqua for several years before starting the company. The firm also works with Marketo, Silverpop Engage B2B and MarketingGenius depending on client needs. )
My initial reaction to the announcement was to wonder if it would be interpreted as evidence that Eloqua requires a lot of consulting to implement. Certainly that’s how I’d spin it if I were a competitor. But then again, maybe I wouldn't, because it focuses attention on how much support is really needed to deploy other systems.
On the surface, this is a strength of vendors who promise free implementation and deployment within a few days. But the reality is that most marketers need outside help to design their email campaigns, nurture programs, scoring rules, and CRM integration. This has less to do with learning the software than with knowing what works and how to apply it to their own business.
Yes, some products really are easier to use than others, especially for simple programs. That's one reason Pedowitz works with several vendors. (Download the Raab Guide report on Vendor Usability Scores for more on this.) And yes, some marketers will get their system running with no more than telephone support.
But it’s just plain silly to think that most marketers can quickly deploy sophisticated demand generation programs without some expert help. This is what’s highlighted by the Pedowitz Group offer – and it’s a discussion that vendors selling the dream of an instant deployment should probably avoid.
Sunday, April 05, 2009
True Influence Opens a Window into Future Demand Generation
People develop new products because they feel they can offer something existing products do not. In the early stages of an industry, the new products are often similar because several people have independently spotted the same opportunity and built something to tap it. As the industry matures, second-generation products are designed to improve on the original products, either by adding new capabilities or by delivering the same capabilities faster, easier or cheaper. This leads to more variety as vendors experiment with different approaches to a now-defined problem. In a third stage, variety diminishes as widely successful approaches become templates for standard configurations.
Demand generation systems are in that second stage. This means new products reflect the lessons each vendor has drawn from the industry’s history to date.
True Influence illustrates this nicely. CEO and co-founder Brian Giese had extensive experience in business sales and marketing and with existing demand generation systems when he began developing True Influence two years ago. So it’s possible to see True Influence, which was released at the end of last year, as his well-educated guess at what future demand generation systems will look like.
Giese’s conclusion was that marketers’ overwhelming need is simplicity. In fact, he said he has actually removed features from the system because customers weren’t using them. But he also decided that marketers want Webinar integration, digital asset management, APIs to capture data from external Web forms, and a dedicated IP address for email. These are not yet standard features on most demand generation products. But if Giese is right, they will be.
Like all demand generation systems, True Influence can import lists, send emails, create Web forms and surveys, score leads, set up multi-step campaigns, and integrate with CRM systems. Capabilities in these areas tend to be adequate but minimal. For example:
- emails and Web forms can be personalized with lead data, but don’t incorporate rule-selected content blocks.
- the list selection interface uses a form that lets users apply values to a list of all data elements. This is simple but doesn’t easily support complex conditions. Nor does True Influence support random splits for tests.
- Answers to Web surveys are limited to list box or radio button formats.
- The system allows an unlimited number of survey questions, but it will overwrite previous replies if a question is answered more than once.
- The lead record allows only four user-defined fields.
- Lead scores can be based on just a few attributes and activities: industry, job title, company size, location, lead source, email status, activity indicator, most recent activity date, and visits to specific Web pages. Giese said that other elements could be exposed but clients haven’t requested them.
On the brighter side:
- The system API lets users easily adopt externally-built and -hosted Web forms to post into the True Influence database. This saves clients the trouble of replacing existing Web forms when they deploy the system. Clients can also build and host their forms within True Influence if they prefer.
- The base price includes a separate IP address. This protects the client if any other True Influence customers run afoul of the anti-spam police. Most other vendors charge extra for a dedicated IP address if they make it available at all.
- The system includes a resource library for both internal assets (templates, emails, Web forms, etc.) and downloadable collateral such as white papers and brochures. This might replace a separate digital asset management system for clients who don't need approval workflows or fine-grained user rights management. The system does support some version control.
Campaign management reflects a particularly interesting set of design choices. Users define campaigns by building a flow chart with icons for steps and delays. Most simplicity-oriented systems avoid flow charts, so I was surprised to see them in True Influence. The system does simplify the diagrams a bit by embedding the decision rules within the lines that link the icons instead of creating separate decision icons. I was also surprised to find that each icon can have its own schedule – another feature typically reserved for advanced systems.
Giese assured me that his clients like the flow charts and successfully use them for "very complex" campaigns. But we didn't discuss the meaning of "very complex" and I suspect my definition is more demanding than his.
More in line with what I’d expect from this system, decision rules are limited to a few essential functions (opened email, clicked link, registered for Webinar, joined Webinar, submitted Web form, completed step, action completed, action failed). Campaign actions are also constrained: for example, the system can send emails but has no particular support for other media such as direct mail or call centers.
Campaigns can also add a lead to a list (which might in turn trigger another campaign), update the lead score, convert a prospect to a lead and send it to the CRM system, send an email alert to a salesperson or sales manager, and publish a landing page or Web form. Treating lead scoring, conversion to a CRM lead and publication of Web pages as campaign functions, rather than executing these outside of individual campaigns, is typical of simplicity-oriented demand generation systems. So is moving leads among campaigns by adding them to lists rather than assigning them directly.
Wait steps can either delay the campaign for a specified period (e.g., wait seven days for a reply), or be keyed to a specific date (e.g., send a reminder three days before a Webinar). This is more flexible than some other products. However, event-based triggers are limited to submission of a Web form. Otherwise, users can achieve near-real-time triggers by scheduling a campaign step to check for specified conditions at regular intervals defined in hours or minutes.
Webinar support includes prebuilt campaigns with registration and confirmation forms; emails for invitations, reminders and thank-you messagess; and salesperson alerts. These are all part of one campaign flow. More important, the vendor has prebuilt integration with Webinar vendor DimDim (a pretty interesting product in its own right). This lets True Influence capture actual attendance and automatically load it into the contact history. Less extensive integration is also available with GoToWebinar.
True Influence has existing integrations with Salesforce.com and SugarCRM. These provide bi-directional synchronization at five minute intervals. The system can exchange data with other CRM systems through batch updates as needed.
Campaign reporting in True Influence includes campaign activities (emails sent and received, page visits, etc.) and lets users drill down to the list of individuals in each campaign. Users can also view the profile, list memberships and activity details for an individual. The system provides its own Web analytics, based on page tags.
Pricing of True Influence is also designed for on simplicity. Fees are based on the number of “actionable” names in the client database, which basically means valid email addresses. This accommodates the large volume of bad data often imported from mailing lists. There are no separate charges for deployment, training, individual modules, extra seats or dedicated IP addresses. Prices are being revised at this writing but are generally intended to be competitive in the middle tier of the demand generation market.
True Influence was released in late 2008 and currently has about fifteen clients.
Thursday, March 26, 2009
Multi-Step Campaign Interfaces: A Quick Vendor Survey
If you read this blog regularly (and who doesn't?), you know that I see a lot of demand generation systems. Naturally, the vendors showing them to me have all thought very carefully about their designs and made the best choices they could, typically based on feedback from their customers. As a result, they tend to be quite certain that they have made the right decisions and are correspondingly unreceptive should I suggest otherwise.
The area where this comes up more often than any other is the design of multi-step campaign flows. There are two basic approaches to this: use a flow chart, or present a list of steps. My own opinion is quite firm: flow charts don't work. They look good in demonstrations and can lay out simple processes quite nicely. But they get impossibly convoluted once you try to do something complex.
I say this with the fervor of a reformed sinner, since for many years I looked at flow charts as the mark of an advanced marketing automation system. But in all that time I never saw a flow chart interface that actually did a good job handling complexity. So I've reluctantly concluded that flow charts are only suitable for serious technical experts.
Such specialists are fairly common at big consumer marketers such as banks and retailers. These organizations have marketing operations staff members whose entire job is to set up campaigns. They do perfectly well with flow chart interfaces, which indeed are standard on enterprise marketing automation products (e.g. Unica, Teradata Relationship Manager, SAS Marketing Automation, Siebel Marketing, Alterian, Aprimo, SmartFocus).
But the vast majority of demand generation systems are installed in much smaller marketing departments, where setting up campaigns is just a fraction of the user's job. Those people don't have the time or inclination to master the subtleties of a flow chart interface.
Of course, holding an opinion strongly doesn't make it correct, even when I'm the one doing the holding. So I thought I'd take a little scan of the demand generation systems I've looked at to see how the vendors themselves had voted. I'm pleased to see that the majority of vendors (11 of 16) , and particularly those tending towards serving smaller clients, have in fact chosen against the flow chart approach. (Gone against the flow, as it were.)
I don't expect this news to change the minds of vendors who made the opposite decision. In fact, if they're any kind of marketers at all, they'll argue it's a competitive differentiator. But at least I'll feel more justified the next I tell one of them I disagree with their choice.
Here, then, is a list of vendors (alphabetically) and their interfaces.
(1) embeds simple list- based campaign in a larger flow chart. I'm in favor of that approach.
The area where this comes up more often than any other is the design of multi-step campaign flows. There are two basic approaches to this: use a flow chart, or present a list of steps. My own opinion is quite firm: flow charts don't work. They look good in demonstrations and can lay out simple processes quite nicely. But they get impossibly convoluted once you try to do something complex.
I say this with the fervor of a reformed sinner, since for many years I looked at flow charts as the mark of an advanced marketing automation system. But in all that time I never saw a flow chart interface that actually did a good job handling complexity. So I've reluctantly concluded that flow charts are only suitable for serious technical experts.
Such specialists are fairly common at big consumer marketers such as banks and retailers. These organizations have marketing operations staff members whose entire job is to set up campaigns. They do perfectly well with flow chart interfaces, which indeed are standard on enterprise marketing automation products (e.g. Unica, Teradata Relationship Manager, SAS Marketing Automation, Siebel Marketing, Alterian, Aprimo, SmartFocus).
But the vast majority of demand generation systems are installed in much smaller marketing departments, where setting up campaigns is just a fraction of the user's job. Those people don't have the time or inclination to master the subtleties of a flow chart interface.
Of course, holding an opinion strongly doesn't make it correct, even when I'm the one doing the holding. So I thought I'd take a little scan of the demand generation systems I've looked at to see how the vendors themselves had voted. I'm pleased to see that the majority of vendors (11 of 16) , and particularly those tending towards serving smaller clients, have in fact chosen against the flow chart approach. (Gone against the flow, as it were.)
I don't expect this news to change the minds of vendors who made the opposite decision. In fact, if they're any kind of marketers at all, they'll argue it's a competitive differentiator. But at least I'll feel more justified the next I tell one of them I disagree with their choice.
Here, then, is a list of vendors (alphabetically) and their interfaces.
flow | list | |
| Act-On Software | x | |
| ActiveConversion |
| x |
| Eloqua | x | |
| Infusionsoft | x | |
| LeadLife | x | |
| LoopFuse | x | |
| Manticore Technology | x | |
| Market2Lead | (1) | x |
Marketbright | (1) | x |
| Marketo | x | |
| Neolane | x | |
| OfficeAutoPilot | x | |
| Pardot | x | |
| Silverpop Engage B2B | x | |
| Treehouse Interactive | x | |
| True Influence | x |
(1) embeds simple list- based campaign in a larger flow chart. I'm in favor of that approach.
Tuesday, March 10, 2009
Notes from SAS Analyst Conference
I spent yesterday at an analyst conference for SAS, being briefed on the company’s plans for 2009. The presentations are considered confidential, so I can’t go into details. But I think it’s okay to share a few points that struck me.
- credit and other financial risk analysis are a very strong business for the company right now. Of course this makes perfect sense, but it’s nice to see on two levels: first, that financial institutions are working on the issue and second, that there’s some good news somewhere.
- SAS sees itself as selling software applications for specific tasks (like credit risk analysis) rather than the underlying analytical tools. This isn’t news, but I hadn’t quite realized how fully the company had committed to this approach. One big difference with SAS is that they essentially create a separate data warehouse using SAS technology, instead of drawing on other systems for data integration and database management. The SAS “framework” (they avoid the term “platform”) might draw from a conventional warehouse, exist alongside it, or replace it altogether. But however you structure it, the SAS approach implies a relatively substantial investment to bring up the initial SAS application. The advantage is it's easier to add later applications – a major selling point when SAS tries to expand its footprint within an existing client.
This approach also puts SAS in conflict with several other parties: corporate IT departments who may view its foundation as redundant; other application software developers; and channel partners who may want to integrate solutions from several sources. SAS did mention several initiatives to cooperate with other players (except maybe other application software developers), but also seems quite willing to go it alone when necessary. The underlying attitude seemed to be that their solutions are good enough to win on their merits, and ultimately that’s what matters.
- improved user interfaces are a major initiative this year. SAS leaders mentioned several times that they had lost deals to less capable products that looked more attractive, and clearly this rankles. However, SAS seems to believe that better user interfaces are all about flash (and, in this case, specifically Adobe Flash). I saw and heard little that suggested a deeper understanding of usability as a way of helping people to do their jobs more efficiently and effectively. One red flag: the sample dashboard they proudly displayed included many gauges, which are almost universally rejected as a poor use of screen real estate. Ironically, one of SAS’s own presentations made this exact point about gauges—but then someone illustrated emailing a picture of a gauge as a way of alerting a colleague to a value. Ouch!
(In case it's not clear why you wouldn't email a gauge: if you just wanted to tell someone the number, you'd just send the number and save the person the need to look at the gauge, find the needle, and then relate it to the scale. If you wanted to show the number in context, you'd send a note along the lines of "revenue is $45,000, 20% ahead of plan". Again, much easier than trying to interpret values and color zones on the gauge, and conveys precisely the point you intended.)
But those are just my little quibbles. SAS remains a tremendously strong company with great technology, far-sighted management and dedicated employees. I have no doubt that they’ll continue to succeed and, more important, deliver great value to their clients.
- credit and other financial risk analysis are a very strong business for the company right now. Of course this makes perfect sense, but it’s nice to see on two levels: first, that financial institutions are working on the issue and second, that there’s some good news somewhere.
- SAS sees itself as selling software applications for specific tasks (like credit risk analysis) rather than the underlying analytical tools. This isn’t news, but I hadn’t quite realized how fully the company had committed to this approach. One big difference with SAS is that they essentially create a separate data warehouse using SAS technology, instead of drawing on other systems for data integration and database management. The SAS “framework” (they avoid the term “platform”) might draw from a conventional warehouse, exist alongside it, or replace it altogether. But however you structure it, the SAS approach implies a relatively substantial investment to bring up the initial SAS application. The advantage is it's easier to add later applications – a major selling point when SAS tries to expand its footprint within an existing client.
This approach also puts SAS in conflict with several other parties: corporate IT departments who may view its foundation as redundant; other application software developers; and channel partners who may want to integrate solutions from several sources. SAS did mention several initiatives to cooperate with other players (except maybe other application software developers), but also seems quite willing to go it alone when necessary. The underlying attitude seemed to be that their solutions are good enough to win on their merits, and ultimately that’s what matters.
- improved user interfaces are a major initiative this year. SAS leaders mentioned several times that they had lost deals to less capable products that looked more attractive, and clearly this rankles. However, SAS seems to believe that better user interfaces are all about flash (and, in this case, specifically Adobe Flash). I saw and heard little that suggested a deeper understanding of usability as a way of helping people to do their jobs more efficiently and effectively. One red flag: the sample dashboard they proudly displayed included many gauges, which are almost universally rejected as a poor use of screen real estate. Ironically, one of SAS’s own presentations made this exact point about gauges—but then someone illustrated emailing a picture of a gauge as a way of alerting a colleague to a value. Ouch!
(In case it's not clear why you wouldn't email a gauge: if you just wanted to tell someone the number, you'd just send the number and save the person the need to look at the gauge, find the needle, and then relate it to the scale. If you wanted to show the number in context, you'd send a note along the lines of "revenue is $45,000, 20% ahead of plan". Again, much easier than trying to interpret values and color zones on the gauge, and conveys precisely the point you intended.)
But those are just my little quibbles. SAS remains a tremendously strong company with great technology, far-sighted management and dedicated employees. I have no doubt that they’ll continue to succeed and, more important, deliver great value to their clients.
Thursday, March 05, 2009
Demand Generation Usability Scores - Part 4
I spent a good part of yesterday talking with demand generation vendors about the usability scores I've been publishing all week. As a result, I've made a number of small adjustments which have been retroactively edited into the previous posts. There might still be some others: two of the vendors haven't spoken with me, which could mean they agree with my ratings but, more likely, means they just haven't been paying attention.
Perhaps I'm just starved for attention, but I do in fact appreciate the feedback. The main result of the conversations so far has been to clarify the items related to lead routing. The relative ranking of the different vendors hasn't been significantly affected.
Here, then, is the final usability matrix.
Now what? Although the focus of these last three posts has been on usability scoring, you may remember that the goal described in the blog post that started all this was to develop a summary comparison of demand generation vendors that marketers could match against their needs.
The idea is to quickly identify the most promising candidates so you have more time to explore them in depth. You can do that on your own through endless vendor meetings and random conversations with your peers, or can save time, effort and risk by purchasing the detailed, objective information conveniently assembled and beautifully packaged in the Raab Guide to Demand Management Systems for the low price of $595, satisfaction guaranteed or your money back. Choose whichever sounds best.
Anyway, the usability scores were the hardest measures to develop, which is why I did them first. What remains are scores for functionality, pricing and vendor background. Those are easy to create because the Raab Guide (buy now! all major credit cards accepted!) has already scored each vendor on more than 150 data points. All I need to do is add up the scores by category and summarize the results.
In fact, that's exactly what I did this morning, and I've been staring at the results for the past several hours.
But I can't show them to you.
Here's the problem. Usability requirements are roughly similar for all marketers. Yes, some features apply more to simple marketing programs and others to complex programs. But everyone needs their system to be intuitive, understandable, efficient, easy to learn, and the rest. So once you decide whether your focus is simple or complex marketing programs, the scores for those categories are a pretty good indicator of which systems you'll find most usable.
But functional requirements are different. Every company has a specific set of needs that are addressed by a specific set of features. In a study as broad-ranging as the Raab Guide, any one of those features will get lost when you aggregate scores into broad categories. So you can't just look at those categories to determine which systems fit your requirements.
For example, let's say that events like breakfast seminars are critical to your marketing program. Specialized features to manage those campaigns are available from several vendors, but they're not necessarily the ones with the highest scores for all campaign management features combined. If you decided to limit your search to the top vendors in that category, you'd miss at least one good candidate.
In other words, publishing aggregate functionality scores, or any type of simple vendor ranking, would actually do more harm than good. I've been saying this for years, but had briefly talked myself into publishing them because I know how popular they would be. I've now come back to my senses.
Still, I hate to disappoint those of you who were looking for more information. So here's what I will do.
First, I'll tell you that the aggregate functionality scores largely correlate with the scores for complex usability. That makes sense: vendors who have built powerful systems have tuned them to be usable in complex situations.
Second, I'll give you a matrix of that shows which systems provide features that are hard to find. At the end of the day, this is what you really need to know to decide which systems to explore in depth. Since we've already collected this information in the Guide (and so much more! order now and start reading in minutes!), it's not much work at all.
Of course, I do realize that publishing this matrix lead to many more phone calls from vendors who want to know why they weren't listed in a given category. That's okay, since I do want to be as accurate as possible. And it doesn't hurt that I'm about to leave town for a few days.
How to Use This Information
Armed with the usability scores and the preceding matrix, you should be able to narrow set of candidate vendors fairly quickly. Start by identifying your critical needs, and check if they're listed in the matrix. If so, the vendors that support your needs are the ones to start with. If not, decide whether your marketing program leans more towards the simple or complex, and choose the vendors with the best usability ratings in your group.
Perhaps I'm just starved for attention, but I do in fact appreciate the feedback. The main result of the conversations so far has been to clarify the items related to lead routing. The relative ranking of the different vendors hasn't been significantly affected.
Here, then, is the final usability matrix.
| | Eloqua | Manticore Technology | Market2Lead | Marketbright | Marketo | Neolane | Silverpop Engage B2B |
| complex | 11 | 8.5 | 12 | 9.5 | 9.5 | 12 | 11 |
| simple | 8 | 10 | 10 | 10 | 12 | 8.5 | 9 |
Now what? Although the focus of these last three posts has been on usability scoring, you may remember that the goal described in the blog post that started all this was to develop a summary comparison of demand generation vendors that marketers could match against their needs.
The idea is to quickly identify the most promising candidates so you have more time to explore them in depth. You can do that on your own through endless vendor meetings and random conversations with your peers, or can save time, effort and risk by purchasing the detailed, objective information conveniently assembled and beautifully packaged in the Raab Guide to Demand Management Systems for the low price of $595, satisfaction guaranteed or your money back. Choose whichever sounds best.
Anyway, the usability scores were the hardest measures to develop, which is why I did them first. What remains are scores for functionality, pricing and vendor background. Those are easy to create because the Raab Guide (buy now! all major credit cards accepted!) has already scored each vendor on more than 150 data points. All I need to do is add up the scores by category and summarize the results.
In fact, that's exactly what I did this morning, and I've been staring at the results for the past several hours.
But I can't show them to you.
Here's the problem. Usability requirements are roughly similar for all marketers. Yes, some features apply more to simple marketing programs and others to complex programs. But everyone needs their system to be intuitive, understandable, efficient, easy to learn, and the rest. So once you decide whether your focus is simple or complex marketing programs, the scores for those categories are a pretty good indicator of which systems you'll find most usable.
But functional requirements are different. Every company has a specific set of needs that are addressed by a specific set of features. In a study as broad-ranging as the Raab Guide, any one of those features will get lost when you aggregate scores into broad categories. So you can't just look at those categories to determine which systems fit your requirements.
For example, let's say that events like breakfast seminars are critical to your marketing program. Specialized features to manage those campaigns are available from several vendors, but they're not necessarily the ones with the highest scores for all campaign management features combined. If you decided to limit your search to the top vendors in that category, you'd miss at least one good candidate.
In other words, publishing aggregate functionality scores, or any type of simple vendor ranking, would actually do more harm than good. I've been saying this for years, but had briefly talked myself into publishing them because I know how popular they would be. I've now come back to my senses.
Still, I hate to disappoint those of you who were looking for more information. So here's what I will do.
First, I'll tell you that the aggregate functionality scores largely correlate with the scores for complex usability. That makes sense: vendors who have built powerful systems have tuned them to be usable in complex situations.
Second, I'll give you a matrix of that shows which systems provide features that are hard to find. At the end of the day, this is what you really need to know to decide which systems to explore in depth. Since we've already collected this information in the Guide (and so much more! order now and start reading in minutes!), it's not much work at all.
Of course, I do realize that publishing this matrix lead to many more phone calls from vendors who want to know why they weren't listed in a given category. That's okay, since I do want to be as accurate as possible. And it doesn't hurt that I'm about to leave town for a few days.
Eloqua | Manticore Technology | Market2Lead | Marketbright | Marketo | Neolane | Silverpop Engage B2B | |||||
| Marketing Channels Beyond Email and System-generated Web Pages | |||||||||||
0 | 0 | 0 | 1 | 0 | 0 | 0 | dealer / channel management (register and co-promote with channel partners) | ||||
1 | 0 | 0 | 0.5 | 0 | 0 | 0 | track RSS readership (publish RSS feeds and track who reads each item) | ||||
0.5 | 0 | 0 | 0 | 0 | 0.5 | 0 | capture costs of Pay Per Click advertising programs directly from the vendor | ||||
0 | 0 | 0 | 0.5 | 0 | 0 | 0.5 | online chat (display a button to request chat; monitor visitors and proactively offer a chat when appropriate) | ||||
1 | 0 | 0 | 0 | 0 | 0.5 | 0 | fax (special relationships with broadcast fax vendors) | ||||
1 | 0 | 0.5 | 0 | 0 | 0 | 0.5 | call center scripting (telephone agents can work from system screens) | ||||
1 | 0 | 0 | 0 | 0 | 1 | 0 | mobile (special relationships with mobile message distribution vendors) | ||||
0 | 1 | 1 | 0 | 0 | 0 | 0 | track conversions and stages via URLs (define which Web pages are conversions in a sales funnel and track leads as they visit them) | ||||
0 | 0 | 1 | 1 | 0 | 1 | 0 | call vendor offers from external pages (display system-generated marketing offers in externally-hosted Web pages, such as the company Web site) | ||||
1 | 0 | 1 | 0 | 0 | 1 | 0.5 | integrate w/direct mail printer (directly transmit personalized marketing materials) | ||||
1 | 0 | 1 | 1 | 0 | 0.5 | 0 | events: quantity limits, wait lists, reminders (specialized campaigns to manage online and real-world events) | ||||
| Tailoring Messages to Individuals | |||||||||||
0 | 0 | 1 | 0 | 0 | 0.5 | 0 | select next action based on highest value (automatically send different messages within a campaign stage based on lead characteristics and behavior history) | ||||
0 | 0 | 1 | 0.5 | 0 | 1 | 0 | show highest-value offer on form (automatically display highest-value offers on a form based on lead characteristics and behaviors) | ||||
0.5 | 0.5 | 1 | 0.5 | 0 | 1 | 0 | rule-based form customization (display different content blocks in an email or Web page based on user-defined rules) | ||||
| Marketing Administration | |||||||||||
0 | 0 | 0 | 0.5 | 0 | 1 | 0 | project schedule (manage a schedule of tasks related to executing a campaign; includes workflow for copy approvals) | ||||
0 | 0 | 0 | 0 | 0 | 1 | 0 | calculate costs from unit cost x volume (calculate campaign costs for line items based on cost per unit and selection or response quantity) | ||||
0 | 0 | 1 | 0 | 0 | 1 | 0 | campaign cost detail (capture line items for costs and revenues within a campaign; track different types such as budget, plan, estimated, actual, etc.) | ||||
| Content Management | |||||||||||
0.5 | 0 | 0 | 0 | 0 | 1 | 0 | expiration dates (assign expiration dates to marketing assets; warn users if an expired asset is still in use) | ||||
0.5 | 0 | 0.5 | 0.5 | 0 | 0.5 | 0 | item-level security (assign access rights to specific marketing assets by individual user or user group) | ||||
0 | 0 | 0.5 | 1 | 0 | 1 | 1 | check in/out, version control (manage asset creation to ensure only one person at a time is making changes; allow users to review and recall old versions) | ||||
| Database Management | |||||||||||
0 | 0 | 1 | 0 | 0 | 0 | 0 | separate analytical (cube) database (copy operational tables into a separate database for reporting and analysis) | ||||
1 | 0 | 1 | 0 | 0 | 1 | 0.5 | custom tables (allow users to add custom data tables to the standard data structure) | ||||
How to Use This Information
Armed with the usability scores and the preceding matrix, you should be able to narrow set of candidate vendors fairly quickly. Start by identifying your critical needs, and check if they're listed in the matrix. If so, the vendors that support your needs are the ones to start with. If not, decide whether your marketing program leans more towards the simple or complex, and choose the vendors with the best usability ratings in your group.
Wednesday, March 04, 2009
Demand Generation Usability Scores - Part 3
Usability Items for Complex Marketing Programs
(note: this is a slightly revised version of the original post, reflecting vendor feedback.)
This post will complete the demand generation vendor usability scores by looking at items that contribute to usability for complex marketing programs.
Explicitly direct leads from one campaign to another. Users can directly send leads from one stage in a campaign to a different campaign. The underlying logic was explained in last Friday's post: marketers running complex programs need precise control over flows among many different campaigns. Marketo gets a half point because it can direct leads to lists, which in turn feed campaigns, but not to campaigns directly.
Linear campaigns embedded in larger campaign flows. Marketers can define a large campaign flow that links the small, linear campaigns. Each small campaign would complete a single marketing project that may have multiple stages, such as prospecting campaign or Webinar. This approach lets users specify the flows among linear campaigns without opening up the rules within those campaigns. This saves effort and makes the flows easier to understand. Encapsulating the linear campaigns as single objects within the larger flow also makes the flow easier to work with.
Integrated a/b testing. The system has features to support split tests of content versions or other treatments. These may be set up by associating multiple versions with a piece of content or by setting up splits within the campaign flow. Inclusion of this item is somewhat aspirational on my part: too few marketers actually do this sort of testing. But they should, and their systems should make it easy.
Rule-based content selection. Marketers can embed logic within an email, Web page or survey that shows different contents depending on the data associated an individual. This is more than simple personalization, which displays a name or other data read from the individual's record. It simplifies complex marketing programs by allowing a single asset to support different customer segments, products, offers, and regions. In the case of Market2Lead, this entry also encompasses an ability for rules to select entirely different assets at the same point in a campaign flow. Manticore Technology and Marketbright receive half credit because they users must write selection rules in a scripting language, which many marketers would find difficult. Silverpop gets a half point because it supports rule-based content for email but not forms.
Multiple scores per lead. The system can calculate and store multiple scores for a single lead record. Different scores might relate to different products or business units.
Central rules control when leads are transferred to sales. Leads are transferred to sales according to a set of rules that operates outside of individual marketing campaigns. The central rules typically execute on a regular schedule or are triggered by events such as a data change or score update. Central rules are necessary in complex marketing programs where there are too many campaigns to define the transfer conditions separately for each one.
Group-based security. Access to system functions, campaigns, and other marketing assets are assigned to individuals, either directly or based on group membership. This degree of control is necessary for large operations where different people may be responsible for different products, regions, customer segments or other slices of the entire marketing program. This reduces the amount of information presented to each user, making their jobs easier as well as preventing unauthorized activities.
Total for Complex Items
Combined Score for Complex Marketing Programs
The complete usability score for complex marketing programs is the sum of the scores for the complex items and for the shared items listed in part 1 of this series. These show Neolane, Market2Lead, Eloqua and Silverpop as the leaders.
Other Items
My original list of complex items included several candidates that I later discarded because they had more to do with program sophistication than usability. These are shown below. Since the highest scores again belong to Market2Lead, Neolane and Eloqua, the only impact of including these items would be a slight decline for Silverpop.
Tomorrow I'll look at some other ratings needed to present a complete picture of the vendors.
(note: this is a slightly revised version of the original post, reflecting vendor feedback.)
This post will complete the demand generation vendor usability scores by looking at items that contribute to usability for complex marketing programs.
Explicitly direct leads from one campaign to another. Users can directly send leads from one stage in a campaign to a different campaign. The underlying logic was explained in last Friday's post: marketers running complex programs need precise control over flows among many different campaigns. Marketo gets a half point because it can direct leads to lists, which in turn feed campaigns, but not to campaigns directly.
Explicitly direct leads from one campaign to another | ||||||
Eloqua | Manticore Technology | Market2Lead | Marketbright | Marketo | Neolane | Silverpop Engage B2B |
1 | 1 | 1 | 1 | 0.5 | 1 | 1 |
Linear campaigns embedded in larger campaign flows. Marketers can define a large campaign flow that links the small, linear campaigns. Each small campaign would complete a single marketing project that may have multiple stages, such as prospecting campaign or Webinar. This approach lets users specify the flows among linear campaigns without opening up the rules within those campaigns. This saves effort and makes the flows easier to understand. Encapsulating the linear campaigns as single objects within the larger flow also makes the flow easier to work with.
Linear campaigns embedded in larger campaign flows | ||||||
Eloqua | Manticore Technology | Market2Lead | Marketbright | Marketo | Neolane | Silverpop Engage B2B |
0 | 0 | 1 | 1 | 0 | 0 | 1 |
Integrated a/b testing. The system has features to support split tests of content versions or other treatments. These may be set up by associating multiple versions with a piece of content or by setting up splits within the campaign flow. Inclusion of this item is somewhat aspirational on my part: too few marketers actually do this sort of testing. But they should, and their systems should make it easy.
Integrated a/b testing | ||||||
Eloqua | Manticore Technology | Market2Lead | Marketbright | Marketo | Neolane | Silverpop Engage B2B |
1 | 0 | 1 | 0 | 1 | 1 | 1 |
Rule-based content selection. Marketers can embed logic within an email, Web page or survey that shows different contents depending on the data associated an individual. This is more than simple personalization, which displays a name or other data read from the individual's record. It simplifies complex marketing programs by allowing a single asset to support different customer segments, products, offers, and regions. In the case of Market2Lead, this entry also encompasses an ability for rules to select entirely different assets at the same point in a campaign flow. Manticore Technology and Marketbright receive half credit because they users must write selection rules in a scripting language, which many marketers would find difficult. Silverpop gets a half point because it supports rule-based content for email but not forms.
Rule-based content selection | ||||||
Eloqua | Manticore Technology | Market2Lead | Marketbright | Marketo | Neolane | Silverpop Engage B2B |
1 | 0.5 | 1 | 0.5 | 0 | 1 | 0.5 |
Multiple scores per lead. The system can calculate and store multiple scores for a single lead record. Different scores might relate to different products or business units.
Multiple scores per lead | ||||||
Eloqua | Manticore Technology | Market2Lead | Marketbright | Marketo | Neolane | Silverpop Engage B2B |
1 | 1 | 0 | 1 | 1 | 1 | 0 |
Central rules control when leads are transferred to sales. Leads are transferred to sales according to a set of rules that operates outside of individual marketing campaigns. The central rules typically execute on a regular schedule or are triggered by events such as a data change or score update. Central rules are necessary in complex marketing programs where there are too many campaigns to define the transfer conditions separately for each one.
Central rules control when leads are transferred to sales | ||||||
Eloqua | Manticore Technology | Market2Lead | Marketbright | Marketo | Neolane | Silverpop Engage B2B |
0 | 0 | 1 | 1 | 1 | 1 | 1 |
Group-based security. Access to system functions, campaigns, and other marketing assets are assigned to individuals, either directly or based on group membership. This degree of control is necessary for large operations where different people may be responsible for different products, regions, customer segments or other slices of the entire marketing program. This reduces the amount of information presented to each user, making their jobs easier as well as preventing unauthorized activities.
Group-based security | ||||||
Eloqua | Manticore Technology | Market2Lead | Marketbright | Marketo | Neolane | Silverpop Engage B2B |
1 | 0 | 1 | 1 | 0 | 1 | 1 |
Total for Complex Items
Total for Complex Items | ||||||
Eloqua | Manticore Technology | Market2Lead | Marketbright | Marketo | Neolane | Silverpop Engage B2B |
5 | 2.5 | 6 | 5.5 | 3.5 | 6 | 5.5 |
Combined Score for Complex Marketing Programs
The complete usability score for complex marketing programs is the sum of the scores for the complex items and for the shared items listed in part 1 of this series. These show Neolane, Market2Lead, Eloqua and Silverpop as the leaders.
Combined Score for Complex Marketing Programs | |||||||
Eloqua | Manticore Technology | Market2Lead | Marketbright | Marketo | Neolane | Silverpop Engage B2B | |
Shared | 6 | 6 | 6 | 4 | 6 | 6 | 5.5 |
Complex | 5 | 2.5 | 6 | 5.5 | 3.5 | 6 | 5.5 |
Combined | 11 | 8.5 | 12 | 9.5 | 9.5 | 12 | 11 |
Other Items
My original list of complex items included several candidates that I later discarded because they had more to do with program sophistication than usability. These are shown below. Since the highest scores again belong to Market2Lead, Neolane and Eloqua, the only impact of including these items would be a slight decline for Silverpop.
Other Items | |||||||
Eloqua | Manticore Technology | Market2Lead | Marketbright | Marketo | Neolane | Silverpop Engage B2B | |
| company-level lead scores | 1 | 0 | 1 | 1 | 0.5 | 1 | 0 |
| progressive profiling in surveys | 1 | 1 | 1 | 1 | 1 | 1 | 0 |
| report on asset usage by campaign | 1 | 0 | 1 | 0 | 1 | 0 | 1 |
| campaign costs | 1 | 1 | 1 | 1 | 0 | 1 | 1 |
| total | 4 | 2 | 4 | 3 | 2.5 | 3 | 2 |
Tomorrow I'll look at some other ratings needed to present a complete picture of the vendors.
Tuesday, March 03, 2009
Demand Generation Usability Scores - Part 2
Usability Items for Simple Marketing Programs
(note: this is a slightly revised version of the original post, reflecting vendor feedback.)
Yesterday's post described the background of this usability scoring project and gave scores for several items that apply to both simple and complex marketing programs. This post will continue with the scores for items that apply to usability for simple campaigns.
Build a campaign as a list of stages. Users can build a simple campaign by defining a linear sequence of stages. As discussed in last Friday's post , I see this is one of two truly key features for making simple campaigns easy to build. The primary alternative, laying out a campaign on a Visio-style flow chart, is harder for most marketers to grasp.
In case you're wondering, the practical difference between a list and a flow chart is that stages in a flow chart are linked by separate decision icons, while stages in a list are connected directly. Many list interfaces allow decision rules within each stage, so there's not necessarily a functional difference. But embedding the rules gives a much cleaner, simpler view of the campaign flow. Of course, this only works for simple flows, because any complex branching is hidden and would soon become unmanageable.
Build linear campaigns outside of a larger campaign structure. The linear campaigns described above can be created independently, rather than as part of a larger structure involving multiple linear campaigns. This is the other key to simple campaign usability because it allows the linear campaigns to be added and removed directly. Otherwise, they must be fit into a larger structure which adds complexity that is unnecessary for a simple marketing program. The flow of leads among these independent campaigns may be managed either through explicit routing (i.e., one campaign directly sends leads to another campaign) or implicitly (i.e., each campaign has its own entry conditions, and leads flow to whichever campaigns they are qualified for.)
Define campaign schedules only at the campaign level, not for individual steps. Execution schedules are specified for the campaign as a whole, and not separately for individual stages. This reduces setup labor, eliminates an item from the user interface, and makes it easier to understand how the campaign will function. The campaign schedules are often derived from the update schedules of the lists that feed the campaign. Neolane is scored with a half point because it allows stage-level schedules but can be configured to hide the capability.
Decision rules are built with functions for specific data types, such as ‘X Web site visits in past Y days'. This contrasts with generic rule building interfaces, which require users to know which field holds a particular type of data and may require complex specifications for calculations. Every system scored here meets this requirement in one way or another, but some other demand generation products do not.
Campaign stages send leads directly to sales. A stage within the flow for each campaign specifies when the system will transfer qualified leads to sales. This is easier to understand than having a separate function that scans for transfer opportunities independently of campaigns. Silverpop gets a half point because it has rules within each campaign that scan for opportunities.
Training time during implementation. This reflects the amount of training the vendors say they provide for end-users during system implementation. It is included here on the assumption that less training time indicates an easy-to-use system. Vendors providing less than one day of training are scored with a 1; those providing more than one day are scored with a 0. Where vendors offer different versions of their system, the scores are based on training for the simplest version with complete functionality. In practice, of course, vendors offer different training packages which are tailored to the client's situation.
Total for Simple Items
Combined Score for Simple Programs
The total usability score for simple marketing programs is the sum of simple items listed in this post plus the shared items listed yesterday. (Figures for Eloqua were changed from the original values based on information provided by the company.) It's no surprise that Marketo has the highest score. The three-way tie for second place among Manticore Technology, Market2Lead and Marketbright is more intriguing. The rating for Market2Lead in particular reflects its just-released new interface, which goes a long way to simplifying what had previously been a very complicated system.
Tomorrow's post will look at scores for complex campaigns.
(note: this is a slightly revised version of the original post, reflecting vendor feedback.)
Yesterday's post described the background of this usability scoring project and gave scores for several items that apply to both simple and complex marketing programs. This post will continue with the scores for items that apply to usability for simple campaigns.
Build a campaign as a list of stages. Users can build a simple campaign by defining a linear sequence of stages. As discussed in last Friday's post , I see this is one of two truly key features for making simple campaigns easy to build. The primary alternative, laying out a campaign on a Visio-style flow chart, is harder for most marketers to grasp.
In case you're wondering, the practical difference between a list and a flow chart is that stages in a flow chart are linked by separate decision icons, while stages in a list are connected directly. Many list interfaces allow decision rules within each stage, so there's not necessarily a functional difference. But embedding the rules gives a much cleaner, simpler view of the campaign flow. Of course, this only works for simple flows, because any complex branching is hidden and would soon become unmanageable.
Build a campaign as a list of stages | ||||||
Eloqua | Manticore Technology | Market2Lead | Marketbright | Marketo | Neolane | Silverpop Engage B2B |
0 | 0 | 1 | 1 | 1 | 0 | 1 |
Build linear campaigns outside of a larger campaign structure. The linear campaigns described above can be created independently, rather than as part of a larger structure involving multiple linear campaigns. This is the other key to simple campaign usability because it allows the linear campaigns to be added and removed directly. Otherwise, they must be fit into a larger structure which adds complexity that is unnecessary for a simple marketing program. The flow of leads among these independent campaigns may be managed either through explicit routing (i.e., one campaign directly sends leads to another campaign) or implicitly (i.e., each campaign has its own entry conditions, and leads flow to whichever campaigns they are qualified for.)
Build linear campaigns outside of a larger campaign structure | ||||||
Eloqua | Manticore Technology | Market2Lead | Marketbright | Marketo | Neolane | Silverpop Engage B2B |
0 | 0 | 1 | 1 | 1 | 0 | 0 |
Define campaign schedules only at the campaign level, not for individual steps. Execution schedules are specified for the campaign as a whole, and not separately for individual stages. This reduces setup labor, eliminates an item from the user interface, and makes it easier to understand how the campaign will function. The campaign schedules are often derived from the update schedules of the lists that feed the campaign. Neolane is scored with a half point because it allows stage-level schedules but can be configured to hide the capability.
Define campaign schedules only at the campaign level, not for individual steps. | ||||||
Eloqua | Manticore Technology | Market2Lead | Marketbright | Marketo | Neolane | Silverpop Engage B2B |
0 | 1 | 1 | 1 | 1 | 0.5 | 0 |
Decision rules are built with functions for specific data types, such as ‘X Web site visits in past Y days'. This contrasts with generic rule building interfaces, which require users to know which field holds a particular type of data and may require complex specifications for calculations. Every system scored here meets this requirement in one way or another, but some other demand generation products do not.
Decision rules are built with functions for specific data types | ||||||
Eloqua | Manticore Technology | Market2Lead | Marketbright | Marketo | Neolane | Silverpop Engage B2B |
1 | 1 | 1 | 1 | 1 | 1 | 1 |
Campaign stages send leads directly to sales. A stage within the flow for each campaign specifies when the system will transfer qualified leads to sales. This is easier to understand than having a separate function that scans for transfer opportunities independently of campaigns. Silverpop gets a half point because it has rules within each campaign that scan for opportunities.
Campaign stages send leads directly to sales | ||||||
Eloqua | Manticore Technology | Market2Lead | Marketbright | Marketo | Neolane | Silverpop Engage B2B |
1 | 1 | 0 | 1 | 1 | 1 | 0.5 |
Training time during implementation. This reflects the amount of training the vendors say they provide for end-users during system implementation. It is included here on the assumption that less training time indicates an easy-to-use system. Vendors providing less than one day of training are scored with a 1; those providing more than one day are scored with a 0. Where vendors offer different versions of their system, the scores are based on training for the simplest version with complete functionality. In practice, of course, vendors offer different training packages which are tailored to the client's situation.
Training time during implementation | ||||||
Eloqua | Manticore Technology | Market2Lead | Marketbright | Marketo | Neolane | Silverpop Engage B2B |
0 | 1 | 0 | 1 | 1 | 0 | 1 |
Total for Simple Items
Total for Simple Items | ||||||
Eloqua | Manticore Technology | Market2Lead | Marketbright | Marketo | Neolane | Silverpop Engage B2B |
2 | 4 | 4 | 6 | 6 | 2.5 | 3.5 |
Combined Score for Simple Programs
The total usability score for simple marketing programs is the sum of simple items listed in this post plus the shared items listed yesterday. (Figures for Eloqua were changed from the original values based on information provided by the company.) It's no surprise that Marketo has the highest score. The three-way tie for second place among Manticore Technology, Market2Lead and Marketbright is more intriguing. The rating for Market2Lead in particular reflects its just-released new interface, which goes a long way to simplifying what had previously been a very complicated system.
Combined Score for Simple Programs | |||||||
Eloqua | Manticore Technology | Market2Lead | Marketbright | Marketo | Neolane | Silverpop Engage B2B | |
| Shared | 6 | 6 | 6 | 4 | 6 | 6 | 5.5 |
| Simple | 2 | 4 | 4 | 6 | 6 | 2.5 | 3.5 |
Combined | 8 | 10 | 10 | 10 | 12 | 8.5 | 9 |
Tomorrow's post will look at scores for complex campaigns.
Monday, March 02, 2009
Demand Generation Usability Scores - Part 1
(note: this is a slightly revised version of the original post, reflecting vendor feedback.)
For better or worse, I've taken my usability scoring project as far as I can. So, having run out of reasons to delay, let me present the results.
To start back at the beginning, the objective has been to find a practical, objective way to evaluate the usability of different demand generation systems. Lacking the resources for a detailed user survey or hands-on scenario testing, I chose to build a checklist of functions that I believe correlate with usability.
But I quickly ran into a problem. Because usability includes both ease-of-use and the suitability for a given task, a single usability score would be misleading because it must either favor construction of simple campaigns or of complex campaigns. I therefore chose to develop separate scores for each. The idea was to remind people that the question of which system is "best" has no simple answer precisely because different systems are good at different things. My hope is that people would then take the next logical step of assessing systems against their own requirements.
In working through the checklist items themselves, I quickly realized that some items apply to usability for both simple and complex campaigns. I therefore ended up with three groups of scoring elements: those for simple campaigns, those for complex campaigns, and those shared by both.
The final piece of background is my definition of simple vs. complex campaigns. In both cases, I see the basic flow as an outbound email, landing page, multi-step nurturing campaign, lead scoring, and transfer to a CRM system for sales followup. A simple campaign would do this for a single product, offer, customer segment and region, while a complex campaign could involve several of each. Obviously these aren't very specific scenarios, but I think the ability to efficiently deliver many different treatments to different customers is ultimately what separates simple from complex in a demand generation context.
On to the checklist items themselves. I tried to find items that could be judged objectively as present or not, without too much evaluation on my part of how well or poorly they had been implemented. This turned out to be reasonably easy and has the major benefit that I can assign a 1 or 0 in nearly all cases. The only exceptions were a couple of cases where it seemed most fair to give a vendor half-credit.
The harder part was deciding which items correlated with usability. Here I considered the functions needed to execute simple and complex campaigns, and focused on functions that made those campaigns easy to set up and run.
This means I excluded functions I consider important but not themselves directly related to ease of use. Or, more precisely, I excluded functions that are more or less equally easy to use in the different products. For example, every demand generation system provides an editor to create emails. But these are generally so similar that they don't really factor into differences in system usability.
Where complex campaigns are concerned, this approach also means I excluded functions having to do with the scope or sophistication of a system rather than bearing directly on ease of use. As I discussed in one of the earlier posts laying out this project, my final scoring system will include separate sophistication scores as well. Similarly, a final scoring system should include other items such as vendor viability, or at least a proxies such as years in business, numbers of clients, and funding. Once more, the logic is to provide enough different scores that people are led to consider which of those scores are important to their own business.
Since the actual scores for most items were 1 or 0, I simply added up the item scores to get composite scores for each vendor. I considered weighting different items, but it didn't appear that any reasonable set of weights would have much impact on the relative rankings of the different vendors. So it seemed best not to bother.
Okay then. Let's look at the items I've chosen and how I've scored the vendors listed in the Raab Guide to Demand Generation Systems. (Five of these were in the original Guide. Marketbright was added on March 3, and Neolane will be added in a week or two once we finalize their entry.)
Rather than overwhelm everyone with a single, huge blog post, I'm going to break this into three parts. This post will cover the shared items. The next will cover simple campaign items, and a third will cover complex campaign items. A final post will summarize and discuss the results.
One final caveat on the scores: they're based on my best information about the vendors, but it's possible something has changed or I missed something in my research. I expect the vendors will let me know if they have questions, and will certainly adjust the scores if appropriate.
Shared Items
These items apply to both simple and complex campaigns. They relate primarily to creation of marketing assets such as emails and landing pages, and to execution of lead scoring.
Select marketing assets from shared libraries. Users can draw on existing libraries of marketing assets when setting up a campaign, rather than creating them from scratch. These assets can be modified or used as is. Even though every system listed below can do this, it's included because some other products might not.
Text search for assets. Users can enter a search string, such as a word or phrase, and get a list of all assets having that string in their name. This makes it easier to find specific assets without keeping track of their exact name or which campaigns used them previously. Again, although every system on this list has this capability, systems not listed might not.
Share marketing assets across campaigns. The same asset can be used in multiple campaigns without creating a new copy. This saves effort if an asset must be updated.
Live templates for asset frames. Assets are built in layered templates with common elements such as headers, footers and styles. The bodies of these assets may be different. The templates are "live" in the sense that a change to the template is applied to all assets using that template, even if the assets are already deployed to a campaign. Like shared assets, this saves effort if a common element must be updated. Silverpop gets a half point because it can share templates for Web forms, but not emails.
Trigger lead scoring outside a campaign step. The system will update lead scores without users building explicit steps into their campaigns to trigger this update. This simplifies campaign creation and ensures that scores are always current. Typically, scores are updated automatically after a data change. Sometimes, they are updated on a regular schedule instead.
Central definition of lead scoring rules. Scoring rules are defined in a central location rather than separately for each campaign. This saves effort and ensures consistency.
Total Score for Shared Items
The next post will look at scores for items specific to simple campaigns.
For better or worse, I've taken my usability scoring project as far as I can. So, having run out of reasons to delay, let me present the results.
To start back at the beginning, the objective has been to find a practical, objective way to evaluate the usability of different demand generation systems. Lacking the resources for a detailed user survey or hands-on scenario testing, I chose to build a checklist of functions that I believe correlate with usability.
But I quickly ran into a problem. Because usability includes both ease-of-use and the suitability for a given task, a single usability score would be misleading because it must either favor construction of simple campaigns or of complex campaigns. I therefore chose to develop separate scores for each. The idea was to remind people that the question of which system is "best" has no simple answer precisely because different systems are good at different things. My hope is that people would then take the next logical step of assessing systems against their own requirements.
In working through the checklist items themselves, I quickly realized that some items apply to usability for both simple and complex campaigns. I therefore ended up with three groups of scoring elements: those for simple campaigns, those for complex campaigns, and those shared by both.
The final piece of background is my definition of simple vs. complex campaigns. In both cases, I see the basic flow as an outbound email, landing page, multi-step nurturing campaign, lead scoring, and transfer to a CRM system for sales followup. A simple campaign would do this for a single product, offer, customer segment and region, while a complex campaign could involve several of each. Obviously these aren't very specific scenarios, but I think the ability to efficiently deliver many different treatments to different customers is ultimately what separates simple from complex in a demand generation context.
On to the checklist items themselves. I tried to find items that could be judged objectively as present or not, without too much evaluation on my part of how well or poorly they had been implemented. This turned out to be reasonably easy and has the major benefit that I can assign a 1 or 0 in nearly all cases. The only exceptions were a couple of cases where it seemed most fair to give a vendor half-credit.
The harder part was deciding which items correlated with usability. Here I considered the functions needed to execute simple and complex campaigns, and focused on functions that made those campaigns easy to set up and run.
This means I excluded functions I consider important but not themselves directly related to ease of use. Or, more precisely, I excluded functions that are more or less equally easy to use in the different products. For example, every demand generation system provides an editor to create emails. But these are generally so similar that they don't really factor into differences in system usability.
Where complex campaigns are concerned, this approach also means I excluded functions having to do with the scope or sophistication of a system rather than bearing directly on ease of use. As I discussed in one of the earlier posts laying out this project, my final scoring system will include separate sophistication scores as well. Similarly, a final scoring system should include other items such as vendor viability, or at least a proxies such as years in business, numbers of clients, and funding. Once more, the logic is to provide enough different scores that people are led to consider which of those scores are important to their own business.
Since the actual scores for most items were 1 or 0, I simply added up the item scores to get composite scores for each vendor. I considered weighting different items, but it didn't appear that any reasonable set of weights would have much impact on the relative rankings of the different vendors. So it seemed best not to bother.
Okay then. Let's look at the items I've chosen and how I've scored the vendors listed in the Raab Guide to Demand Generation Systems. (Five of these were in the original Guide. Marketbright was added on March 3, and Neolane will be added in a week or two once we finalize their entry.)
Rather than overwhelm everyone with a single, huge blog post, I'm going to break this into three parts. This post will cover the shared items. The next will cover simple campaign items, and a third will cover complex campaign items. A final post will summarize and discuss the results.
One final caveat on the scores: they're based on my best information about the vendors, but it's possible something has changed or I missed something in my research. I expect the vendors will let me know if they have questions, and will certainly adjust the scores if appropriate.
Shared Items
These items apply to both simple and complex campaigns. They relate primarily to creation of marketing assets such as emails and landing pages, and to execution of lead scoring.
Select marketing assets from shared libraries. Users can draw on existing libraries of marketing assets when setting up a campaign, rather than creating them from scratch. These assets can be modified or used as is. Even though every system listed below can do this, it's included because some other products might not.
Select marketing assets from shared libraries | ||||||
Eloqua | Manticore Technology | Market2Lead | Marketbright | Marketo | Neolane | Silverpop Engage B2B |
1 | 1 | 1 | 1 | 1 | 1 | 1 |
Text search for assets. Users can enter a search string, such as a word or phrase, and get a list of all assets having that string in their name. This makes it easier to find specific assets without keeping track of their exact name or which campaigns used them previously. Again, although every system on this list has this capability, systems not listed might not.
Text search for assets | ||||||
Eloqua | Manticore Technology | Market2Lead | Marketbright | Marketo | Neolane | Silverpop Engage B2B |
1 | 1 | 1 | 1 | 1 | 1 | 1 |
Share marketing assets across campaigns. The same asset can be used in multiple campaigns without creating a new copy. This saves effort if an asset must be updated.
Share marketing assets across campaigns | ||||||
Eloqua | Manticore Technology | Market2Lead | Marketbright | Marketo | Neolane | Silverpop Engage B2B |
1 | 1 | 1 | 0 | 1 | 1 | 1 |
Live templates for asset frames. Assets are built in layered templates with common elements such as headers, footers and styles. The bodies of these assets may be different. The templates are "live" in the sense that a change to the template is applied to all assets using that template, even if the assets are already deployed to a campaign. Like shared assets, this saves effort if a common element must be updated. Silverpop gets a half point because it can share templates for Web forms, but not emails.
Live templates for asset frames | ||||||
Eloqua | Manticore Technology | Market2Lead | Marketbright | Marketo | Neolane | Silverpop Engage B2B |
1 | 1 | 1 | 0 | 1 | 1 | .5 |
Trigger lead scoring outside a campaign step. The system will update lead scores without users building explicit steps into their campaigns to trigger this update. This simplifies campaign creation and ensures that scores are always current. Typically, scores are updated automatically after a data change. Sometimes, they are updated on a regular schedule instead.
Trigger lead scoring outside a campaign step | ||||||
Eloqua | Manticore Technology | Market2Lead | Marketbright | Marketo | Neolane | Silverpop Engage B2B |
1 | 1 | 1 | 1 | 1 | 1 | 1 |
Central definition of lead scoring rules. Scoring rules are defined in a central location rather than separately for each campaign. This saves effort and ensures consistency.
Central definition of lead scoring rules | ||||||
Eloqua | Manticore Technology | Market2Lead | Marketbright | Marketo | Neolane | Silverpop Engage B2B |
1 | 1 | 1 | 1 | 1 | 1 | 1 |
Total Score for Shared Items
Total Score for Shared Items | ||||||
Eloqua | Manticore Technology | Market2Lead | Marketbright | Marketo | Neolane | Silverpop Engage B2B |
6 | 6 | 6 | 4 | 6 | 6 | 5.5 |
The next post will look at scores for items specific to simple campaigns.
Subscribe to:
Posts (Atom)
