I spoke earlier this week with SQLStream, which offers software to execute queries against data streams such as stock market prices, Web logs and credit card transactions. These queries can include on-the-fly calculations such as moving averages, as well as scans for patterns like a sequence of failed log-in attempts. Typical applications include security monitoring, fraud detection, and general business activity monitoring. Marketers can use the queries to identify new leads and select cross-sell and upsell offers. Although the connection is a little less obvious, the system can also be used as an alternative to conventional batch data preparation methods for tasks like customer data integration.
SQLStream’s particular claim to fame is that its queries are almost identical to garden-variety SQL. Other vendors in this space apparently use more proprietary approaches. I say “apparently” because I haven’t researched the competition in any depth. A quick bit of poking around was enough to scare me off: there are many vendors in the space and it is a highly technical topic. It turns out that stream processing is one type of “complex event processing,” a field which has attracted some very smart but contentious experts. To see what I mean, check out Event Processing Thinking (Opher Etzion) and Cyberstrategics Complex Event Processing Blog (Tim Bass). This is clearly not a group to mess with.
That said, SQLStream’s more or less direct competitors seem to include: Coral8, Truviso, Progress Apama, Oracle BAM, TIBCO BusinessEvents, KX Systems, StreamBase and Aleri . For a basic introduction to data stream processing, see this presentation from Truvisio.
Back to SQLStream. As I said, it lets users write what are essentially standard SQL queries that are directed against a data stream rather than a static table. The data stream can be any JDBC-accessible data source, which includes most types of databases and file structures. The system can also accept streams of XML data over HTTP, which includes RSS feeds, Twitter posts and other Web sources. Its queries can also incorporate conventional (non-streaming) relational database tables, which is very useful when you need to compare streamed inputs against more or less static reference information. For example, you might want to check current activity against a customer’s six-month average bank balance or transaction rate.
The advantages of using SQL queries are that there are lots of SQL programmers out there and that SQL is relatively easy to write and understand. The disadvantage (in my opinion; not surprisingly, SQLStream didn’t mention this) is that SQL is really bad at certain kinds of queries, such as queries comparing subsets within the query universe and queries based on record sequence. Lack of sequencing may sound like a pretty big drawback for a stream processing system, but SQLStream compensates by letting queries specify a time “window” of records to analyze. This makes queries such as “more than three transactions in the past minute” quite simple. (The notion of “windows” is common among stream processing systems.) To handle subsets within queries, SQLStream mimics a common SQL technique of converting one complex query into a sequence of simple queries. In SQLStream terms, this means the output of one query can be a stream that is read by another query. These streams can be cascaded indefinitely in what SQLStream calls a “data flow architecture”. Queries can also call external services, such as address verification, and incorporate the results. Query results can be posted as records to a regular database table.
SQLStream does its actual processing by holding all the necessary data in memory. It automatically examines all active queries to determine how long data must be retained: thus, if three different queries need a data element for one, two and three minutes, the system will keep that data in memory for three minutes. SQLStream can run on 64-bit servers, allowing effectively unlimited memory, at least in theory. In practice, it is bound by the physical memory available: if the stream feeds more data than the server can hold, some data will be lost. The vendor is working on strategies to solve this problem, probably by retaining the overflow data and processing it later. For now, the company simply recommends that users make sure they have plenty of extra memory available.
In addition to memory, system throughput depends on processing power. SQLStream currently runs on multi-core, single-server systems and is moving towards multi-node parallel processing. Existing systems process tens of thousands of records per second. By itself, this isn't a terribly meaningful figure, since capacity also depends on record size, query complexity, and data retention windows. In any case, the vendor is aiming to support one million records per second.
SQLStream was founded in 2002 and owns some basic stream processing patents. The product itself was launched only in 2008 and currently has about a dozen customers. Since the company is still seeking to establish itself, pricing is, in their words, “very aggressive”.
If you’re still reading this, you probably have a pretty specific reason for being interested in SQLStream or stream processing in general. But just in case you’re wondering “Why the heck is he writing about this in a marketing blog?” there are actually several reasons. The most obvious is that “real time analytics” and “real time interaction management” are increasingly prominent topics among marketers. Real time analytics provides insights into customer behaviors at either a group level (e.g., trends in keyword response) or for an individual (e.g., estimated lifetime value). Real time interaction management goes beyond insight to recommend individual treatments as the interaction takes place (e.g., which offer to make during a phone call). Both require the type of quick reaction to new data that stream processing can provide.
There is also increasing interest in behavior detection, sometimes called event driven marketing. This monitors customer behaviors for opportunities to initiate an interaction. The concept is not widely adopted, even thought it has proven successful again and again. (For example, Mark Holtom of Eventricity recently shared some very solid research that found event-based contacts were twice as productive as any other type. Unfortunately the details are confidential, but if you contact Mark via Eventriicty perhaps he can elaborate.) I don’t think lack of stream processing technology is the real obstacle to event-based marketing, but perhaps greater awareness of stream processing would stir up interest in behavior detection in general.
Finally, stream processing is important because so much attention has recently been focused on analytical databases that use special storage techniques such as columnar or in-memory structures. These require processing to put the data into the proper format. Some offer incremental updates, but in general the updates run as batch processes and the systems are not tuned for real-time or near-real-time reactions. So it’s worth considering stream processing systems as a complement to that lets companies employ these other technologies without giving up quick response to new data.
I suppose there's one more reason: I think this stuff is really neat. Am I allowed to say that?
Thursday, January 29, 2009
Tuesday, January 20, 2009
Salespeople: One Question Matters Most
Back in December, the Sales Lead Management Association and LEADTRACK published a survey on lead management practices that I haven’t previously had time to write about. (The survey is still available on the SLMA Web site.) It contained 10 questions, which is about as many as I can easily grasp.
The two clearest answers came from questions about the information salespeople want and why they don’t follow up on inquiries. By far the most desired piece of information about a lead was purchasing time frame: this was cited by 41% of respondents, compared with budget (17%), application (15%), lead score (15%) and authority (12%). I guess it’s a safe bet that salespeople jump quickly on leads who are about to purchase and pretty much ignore the others, so this finding strongly reinforces the need for nurturing campaigns that allow marketers to keep in contact with leads who are not yet ready to buy.
Note that none of listed categories included behavioral information such as email clickthroughs or Web page visits, which demand generation vendors make so much of. I doubt they would have ranked highly had they been included. Although behavioral data provides some insights into a lead’s state of mind, it's useful to be reminded that wholly pragmatic facts about time frame are a salesperson's paramount concern.
The other clear message from the survey was that the main reason leads are not followed up is “not enough info”. This was cited by 55% of respondents, compared with 14% for “inquired before, never bought”, 12% for “no system to organize leads”, 10% for “no phone number”, 7% for "geo undesirable" and 2% because of "no quota on product". This is an unsurprising result, since (a) good information is often missing and (b) salespeople don’t like to waste time on unqualified leads. Based on the previous question, we can probably assume that the critical piece of necessary information is time frame. So this answer reinforces the importance of gathering that information and passing it on.
One set of answers that surprised me a bit were that 77% or 80% of salespeople were working with an automated lead management system, either “CRM/lead management” or “Software as a Service”. I’ve given two figures because the question was purposely asked two different ways to check for consistency. The categories don’t make much sense to me because they overlap: products like Salesforce.com are both CRM systems and SaaS. Still, this doesn't affect the main finding that nearly everyone has some type of automated system to “update lead status” and “manage your inquires” (the two different questions that were asked). This is higher market penetration than I expected, although I do recognize that those questions deal more with lead management (a traditional sales automation function) than lead generation (the province of demand generation systems). Still, to the extent that CRM systems can offer demand generation functions, there may be a more limited market for demand generation than the vendors expect.
One final interesting set of figures had to do with marketing measurement. The survey found that 23% of companies measure ROI for all lead generation tactics, 30% measure it for some tactics, and 47% don’t measure it at all. The authors of the survey report seem to find these numbers distressingly low, particularly in comparison with the 80% of companies that have a system in place and, at least in theory, are capturing the data needed for measurement. I suppose I come at this from a different perspective, having seen so many surveys over the years showing that most companies don’t do much measurement. To me, 23% measuring everything seems unbelievably high. (For example, Jim Lenskold's 2008 Marketing ROI and Measurements Study found 26% of respondents measured ROI on some or all campaigns; the combination of "some" and "all" in the SLMA study is 53%.) Either way, of course, there is plenty of room for improvement, and that's what really counts.
The two clearest answers came from questions about the information salespeople want and why they don’t follow up on inquiries. By far the most desired piece of information about a lead was purchasing time frame: this was cited by 41% of respondents, compared with budget (17%), application (15%), lead score (15%) and authority (12%). I guess it’s a safe bet that salespeople jump quickly on leads who are about to purchase and pretty much ignore the others, so this finding strongly reinforces the need for nurturing campaigns that allow marketers to keep in contact with leads who are not yet ready to buy.
Note that none of listed categories included behavioral information such as email clickthroughs or Web page visits, which demand generation vendors make so much of. I doubt they would have ranked highly had they been included. Although behavioral data provides some insights into a lead’s state of mind, it's useful to be reminded that wholly pragmatic facts about time frame are a salesperson's paramount concern.
The other clear message from the survey was that the main reason leads are not followed up is “not enough info”. This was cited by 55% of respondents, compared with 14% for “inquired before, never bought”, 12% for “no system to organize leads”, 10% for “no phone number”, 7% for "geo undesirable" and 2% because of "no quota on product". This is an unsurprising result, since (a) good information is often missing and (b) salespeople don’t like to waste time on unqualified leads. Based on the previous question, we can probably assume that the critical piece of necessary information is time frame. So this answer reinforces the importance of gathering that information and passing it on.
One set of answers that surprised me a bit were that 77% or 80% of salespeople were working with an automated lead management system, either “CRM/lead management” or “Software as a Service”. I’ve given two figures because the question was purposely asked two different ways to check for consistency. The categories don’t make much sense to me because they overlap: products like Salesforce.com are both CRM systems and SaaS. Still, this doesn't affect the main finding that nearly everyone has some type of automated system to “update lead status” and “manage your inquires” (the two different questions that were asked). This is higher market penetration than I expected, although I do recognize that those questions deal more with lead management (a traditional sales automation function) than lead generation (the province of demand generation systems). Still, to the extent that CRM systems can offer demand generation functions, there may be a more limited market for demand generation than the vendors expect.
One final interesting set of figures had to do with marketing measurement. The survey found that 23% of companies measure ROI for all lead generation tactics, 30% measure it for some tactics, and 47% don’t measure it at all. The authors of the survey report seem to find these numbers distressingly low, particularly in comparison with the 80% of companies that have a system in place and, at least in theory, are capturing the data needed for measurement. I suppose I come at this from a different perspective, having seen so many surveys over the years showing that most companies don’t do much measurement. To me, 23% measuring everything seems unbelievably high. (For example, Jim Lenskold's 2008 Marketing ROI and Measurements Study found 26% of respondents measured ROI on some or all campaigns; the combination of "some" and "all" in the SLMA study is 53%.) Either way, of course, there is plenty of room for improvement, and that's what really counts.
Monday, January 19, 2009
New Best Practices White Paper
As promised, I've written a white paper with the 37 Marketing Automation Best Practices listed in last week's post. This is in the Resources section of the Raab Guide to Demand Generation Systems site; you have to register and then log in. Registration is free.
Saturday, January 17, 2009
Best Practices for Marketing Automation and Demand Generation Campaigns
I enjoyed my little presentation on BrightTalk last Wednesday, which you can still view by clicking here. (If that doesn’t work, go to the BrightTalk site and key my name into the site search function. This will also bring up a roundtable discussion from Tuesday, which I think was interesting as well.) The BrightTalk platform itself worked nicely and was about as simple as possible. They offer a limited version for free (one 30 minute Webinar per month), which is worth considering if you’d like to dip your toe into this sort of thing. The next level up is $949 per month, which is rather pricey compared with $99 per month for Go To Webinar, a platform we’ve used here which does roughly the same thing. I'm not saying they're identical: BrightTalk lets you upload your slides rather than sharing the screen of your PC, which makes it more reliable, and seems to offer some promotional services too. So you’d want to look more closely at both paid services before making a choice.
But I digress. The heart of my presentation on Wednesday ended up as a list of 37 “best practices” for marketing automation / demand generation programs. I’ll probably embed them in a white paper for the Raab Guide Web site in the near future, but for now I thought I’d share them here. (If you want the full slide deck, complete with moderately witty speaker notes, drop me at email at draab@raabassociates.com.)
A bit of context: the presentation listed a sequence of steps for marketing campaign creation, deployment and analysis. The best practices are organized around those steps.
Step 1: Gather Data. The marketer assembles information about the target audience. Best practices here involve the types of data, and, in particular, expanding beyond traditional sources.
• leads, promotions, responses, orders: these are the traditional data sources used in most marketing systems. Best practices would link actual orders back to the individual leads, and do accurate customer data integration.
• external demographics, preferences, contact names: the best practice here is to supplement internal data with external sources such as D&B, Hoovers, LexisNexis, ZoomInfo, etc. More information allows more accurate targeting and better lead scoring.
• social networks: these can be another source of contact names, and sometimes of introductions via mutual friends. A close look at what individuals have said and done in these networks could provide deep insight into a particular person’s needs, attitudes and interests, but this is more an activity for salespeople than marketers.
• summarized activity detail: marketing systems gather an overwhelming mass of detail about prospect activity, down to every click on every Web page. Best practice is to make this more usable by flagging summaries such as “three visits in the past seven days” and making them available for segmentation and event-triggered marketing.
• self-adjusting surveys: once a lead has answered a survey question, the system should automatically replace that question with new one. This builds a richer customer profile and avoids annoying the lead by asking the same question twice. For a bonus best practice, the system should choose the next question based on user-defined rules that select the most useful question for each individual.
• order detail, payments, customer service: the best practice is to gather information beyond the basic order history from operational systems. This also allows more precise targeting and may uncover opportunities that are otherwise invisible, such as follow-up to service problems.
• near real-time updates: fast access to information about lead behaviors allows quick response, particularly in the form of event-triggered messages. This can be critical to engaging a prospect when her interest is at its peak, and before she turns to a competitor.
• household and company levels: consumers should be grouped by households and business leads by their company, division or project. This grouping permits selections and scoring based on activity of the entire group, which may display patterns that are not visible when looking at just a single individual.
Step 2: Design Campaign. The marketer now designs the flow of the campaign itself. Traditional marketing programs use a small number of simple campaigns, each designed from scratch and often used just once. Even traditional campaigns often include multiple steps, so this itself isn’t listed separately as a new best practice.
• many specialized campaigns: the best practice marketer deploys many campaigns, each tailored to a specific customer segment or business need. These are more effective because they are more tightly targeted.
• cross sell, up sell and retention campaigns: demand generation focuses primarily on campaigns to acquire new leads. The best practice is to supplement these with campaigns that help sell more to existing customers and to retain those customers. Marketing automation has generally included these types of campaigns, at least in theory, but many firms could productively expand their efforts in these areas.
• share and reuse components (structure, rules, lists): when marketers are running many specialized campaigns, they have greater opportunity to share common components, and greater benefit from doing so. Sharing makes it possible to build more complex, sophisticated components and to ensure consistency both in how each customer is treated and in how company policies are implemented.
• new channels (search, Web ads, mobile, social): these new channels are often more efficient than traditional channels, and many have other benefits such as being easier to measure. Best practice marketers test new channels aggressively to find out what works and how they can best be used. Even if the new channels are not immediately cost-effective, marketers can limit their investment but still build some experience that will be useful later.
• multiple channels in same campaign: true multichannel campaigns contact customers through different media. A mix of media allows you to reach customers who are responsive in different channels, thereby boosting the aggregate response. Channels may also be chosen based on stated customer preferences and the nature of a particular contact. Marketing automation systems make it easy to switch between media within a single campaign.
Step 3: Develop Content. This step creates the actual marketing materials needed by each step in the campaign design. These are emails, call scripts, landing pages, brochures, and so on.
• rule-selected content blocks and best offers: content is tailored to individuals not simply by inserting data elements (“Dear [First_Name]” but by executing rules that select different messages based on the situation. For example, a rule might send different messages based on the customer’s account balance.
• map drip-marketing message to buyer stage: best practice nurturing campaigns deliver messages that move the lead through a sequence of stages, typically starting with general information and becoming more product oriented. This is more effective than sending the same message to everyone or always sending product information.
• standard templates: messages are built using standard templates that share a desired look-and-feel and contain common elements such as headers and footers. This provides consistency, saves work, and ensures that policies are followed.
• share and reuse components (items, content blocks, images): like shared campaign components, shared marketing contents minimize the work needed to create many different, tailored campaigns. Sharing also makes it easy to deploy changes, such as a new price or new logo, without individually modifying every item in every campaign.
• unified content management system across channels: even though most marketing materials are channel-specific, many components such as images and text blocks can in fact be shared across different channels. Managing these through a single system further saves work, supports sharing, and ensures consistency.
Step 4: Execute Campaign. The campaign is deployed to actual customers. Best practice campaigns often run continuously, rather than being executed once and then replaced with something new. This lets marketers refine them over time, testing different treatments for different conditions and keeping the winners.
• separate treatments by segment: messages and campaign flows are tailored to the needs of each segment. This could be done by creating one campaign with variations for different segments or by creating separate campaigns for each segment. Which works best depends largely on your particular marketing automation system. Either way, shared components should keep the redundant work to a minimum.
• statistical modeling for segmentation: predictive model scores can often define segments more accurately than manual segmentations. Perhaps more important, they can be less labor-intensive to create, allowing marketers to build more segments and rebuild them more often. This matters because best practice marketing involves so many specialized campaigns and is constantly adjusting to new conditions.
• change campaign flow based on responses, events, activities: best practice campaigns change lead treatments in response to their behaviors. Thus, instead of a fixed sequence of treatments, they send leads down different branches and, in some cases, move them from one campaign to another. Changes may be triggered by activities within the campaign, such as response to a message or data provided within a form, or by information recorded elsewhere and reported to the marketing automation system.
• advanced scoring (complex rules, activity patterns, event depreciation, point caps): simple lead scoring formulas are often inaccurate predictors of future behavior. Best practice scoring may involve complex calculations based on relationships among several data elements, summarized activity detail, reduced value assigned to less recent events, and caps on the number of points assigned for any single type of activity. A related challenge for system designers is making complex formulas reasonably easy to set up and understand.
• company-level scores and activity tracking: the best practice campaign can use aggregated company or household data to calculate scores, guide individual treatments, and issue alerts. This allows more appropriate treatment than looking at each individual in isolation.
• multiple scores per lead: for companies with several products, the best practice to calculate a separate lead score for each. The scores may also have different thresholds for sending the lead to sales.
• define score formula jointly with sales: the salesperson is the ultimate judge of whether a lead is qualified. But many marketing departments still set up lead scoring formulas without sales input. Best practice is to work together on defining the criteria and then to periodically review the results to see if the formula can be improved.
• let sales return leads for more nurturing: traditional lead management is a one-way street, with leads sent from marketing to sales and then never heard from again. Best practice marketers allow salespeople to return leads to marketing for further nurturing. This improves the chances of a lead ultimately making a purchase, even if it doesn’t happen right away.
Step 5: Analyze Results. Learning from past campaigns may be the most important best practice of all. Having many targeted campaigns allows for continuous incremental improvement, achieved by quickly evaluating the return on each project and adjusting future programs based on the results.
• advanced response attribution: traditional methods often credit a lead to whichever campaign contacted them first, or whichever generated the first response. Best practice marketers look more deeply at the factors which may have influenced a lead’s behavior, often applying sophisticated analytics to estimate the incremental impact of different campaigns.
• standard metrics, within and across channels: resources can only be allocated to their optimal use if return on investment can be compared across campaigns. This requires standard metrics, which must be calculated consistently and clearly understood throughout the organization.
• formal test designs (a/b, multivariate): traditional marketers often do little testing, and the tests they do are often poorly designed. Best practice marketing involves continuous, formal testing designed to answer specific questions and lead to actionable results.
• capture immediate and long-term results: initial response rate or cost per lead fails to take into account the value of the leads generated, which can differ hugely from campaign to campaign. Best practice requires measuring the long-term value and building it into standard campaign metrics.
• evaluate on customer profitability, not revenue: customers with the same revenue can vary greatly in the actual profit they bring to the company, depending on the profit margins of their purchases and other costs such as customer support. Best practice metrics include accurate profitability measures, preferably drawn from an activity-based costing system.
• continually assess and reallocate spending: best practice marketers have a formal process to shift resources to the most productive marketing investments. These will change as campaigns are refined, business conditions evolve, and new opportunities emerge. A formal assessment process is essential because organizations otherwise tend to resist change.
Infrastructure. Individual campaigns are made possible by an underlying infrastructure that has best practices of its own.
• consolidated systems (multi-channel content management, campaign management and analytics): today’s marketing systems can usually handle multiple channels, so decommissioning older channel-based systems may save money as well as making multi-channel campaigns easier to execute. Consolidated multi-channel analytics, which may occur outside of the marketing automation system, are particularly important for gaining a complete view of each customer.
• advanced system training: marketing departments often provide workers with the minimum training needed to gain competency in their tools. Best practice departments recognize that additional training can make users more productive, particularly as the tools themselves add new capabilities that users would otherwise not be able to exploit.
• advanced analytics training: analytics play a central role in the continuous improvement process. Solid analytics training ensures that users can set up proper tests and interpret the results. Because data and tools are often already available, lack of training is frequently the main obstacle that prevents marketers from using analytics effectively.
• formal processes: best practice marketers develop, document and enforce formal, consistent business processes. This both ensures that work is done efficiently and makes it possible to execute changes when opportunities arise.
• cross-department cooperation: working with sales, service, finance and other departments is essential to sharing systems, data and metrics. A cross-department perspective ensures that each department considers the impact of its decisions on the rest of the company and on the customers themselves.
Summary
The best practice vision is many marketing campaigns, each precisely targeted, efficiently executed, and carefully designed to yield the greatest possible value. The campaigns are supported by detailed analysis to understand results and identify potential improvements. This information is quickly fed into new campaigns, ensuring that the company continually evolves its approaches and makes the best possible use of marketing resources. Continuous optimization is the ultimate best practice that all other practices should support.
But I digress. The heart of my presentation on Wednesday ended up as a list of 37 “best practices” for marketing automation / demand generation programs. I’ll probably embed them in a white paper for the Raab Guide Web site in the near future, but for now I thought I’d share them here. (If you want the full slide deck, complete with moderately witty speaker notes, drop me at email at draab@raabassociates.com.)
A bit of context: the presentation listed a sequence of steps for marketing campaign creation, deployment and analysis. The best practices are organized around those steps.
Step 1: Gather Data. The marketer assembles information about the target audience. Best practices here involve the types of data, and, in particular, expanding beyond traditional sources.
• leads, promotions, responses, orders: these are the traditional data sources used in most marketing systems. Best practices would link actual orders back to the individual leads, and do accurate customer data integration.
• external demographics, preferences, contact names: the best practice here is to supplement internal data with external sources such as D&B, Hoovers, LexisNexis, ZoomInfo, etc. More information allows more accurate targeting and better lead scoring.
• social networks: these can be another source of contact names, and sometimes of introductions via mutual friends. A close look at what individuals have said and done in these networks could provide deep insight into a particular person’s needs, attitudes and interests, but this is more an activity for salespeople than marketers.
• summarized activity detail: marketing systems gather an overwhelming mass of detail about prospect activity, down to every click on every Web page. Best practice is to make this more usable by flagging summaries such as “three visits in the past seven days” and making them available for segmentation and event-triggered marketing.
• self-adjusting surveys: once a lead has answered a survey question, the system should automatically replace that question with new one. This builds a richer customer profile and avoids annoying the lead by asking the same question twice. For a bonus best practice, the system should choose the next question based on user-defined rules that select the most useful question for each individual.
• order detail, payments, customer service: the best practice is to gather information beyond the basic order history from operational systems. This also allows more precise targeting and may uncover opportunities that are otherwise invisible, such as follow-up to service problems.
• near real-time updates: fast access to information about lead behaviors allows quick response, particularly in the form of event-triggered messages. This can be critical to engaging a prospect when her interest is at its peak, and before she turns to a competitor.
• household and company levels: consumers should be grouped by households and business leads by their company, division or project. This grouping permits selections and scoring based on activity of the entire group, which may display patterns that are not visible when looking at just a single individual.
Step 2: Design Campaign. The marketer now designs the flow of the campaign itself. Traditional marketing programs use a small number of simple campaigns, each designed from scratch and often used just once. Even traditional campaigns often include multiple steps, so this itself isn’t listed separately as a new best practice.
• many specialized campaigns: the best practice marketer deploys many campaigns, each tailored to a specific customer segment or business need. These are more effective because they are more tightly targeted.
• cross sell, up sell and retention campaigns: demand generation focuses primarily on campaigns to acquire new leads. The best practice is to supplement these with campaigns that help sell more to existing customers and to retain those customers. Marketing automation has generally included these types of campaigns, at least in theory, but many firms could productively expand their efforts in these areas.
• share and reuse components (structure, rules, lists): when marketers are running many specialized campaigns, they have greater opportunity to share common components, and greater benefit from doing so. Sharing makes it possible to build more complex, sophisticated components and to ensure consistency both in how each customer is treated and in how company policies are implemented.
• new channels (search, Web ads, mobile, social): these new channels are often more efficient than traditional channels, and many have other benefits such as being easier to measure. Best practice marketers test new channels aggressively to find out what works and how they can best be used. Even if the new channels are not immediately cost-effective, marketers can limit their investment but still build some experience that will be useful later.
• multiple channels in same campaign: true multichannel campaigns contact customers through different media. A mix of media allows you to reach customers who are responsive in different channels, thereby boosting the aggregate response. Channels may also be chosen based on stated customer preferences and the nature of a particular contact. Marketing automation systems make it easy to switch between media within a single campaign.
Step 3: Develop Content. This step creates the actual marketing materials needed by each step in the campaign design. These are emails, call scripts, landing pages, brochures, and so on.
• rule-selected content blocks and best offers: content is tailored to individuals not simply by inserting data elements (“Dear [First_Name]” but by executing rules that select different messages based on the situation. For example, a rule might send different messages based on the customer’s account balance.
• map drip-marketing message to buyer stage: best practice nurturing campaigns deliver messages that move the lead through a sequence of stages, typically starting with general information and becoming more product oriented. This is more effective than sending the same message to everyone or always sending product information.
• standard templates: messages are built using standard templates that share a desired look-and-feel and contain common elements such as headers and footers. This provides consistency, saves work, and ensures that policies are followed.
• share and reuse components (items, content blocks, images): like shared campaign components, shared marketing contents minimize the work needed to create many different, tailored campaigns. Sharing also makes it easy to deploy changes, such as a new price or new logo, without individually modifying every item in every campaign.
• unified content management system across channels: even though most marketing materials are channel-specific, many components such as images and text blocks can in fact be shared across different channels. Managing these through a single system further saves work, supports sharing, and ensures consistency.
Step 4: Execute Campaign. The campaign is deployed to actual customers. Best practice campaigns often run continuously, rather than being executed once and then replaced with something new. This lets marketers refine them over time, testing different treatments for different conditions and keeping the winners.
• separate treatments by segment: messages and campaign flows are tailored to the needs of each segment. This could be done by creating one campaign with variations for different segments or by creating separate campaigns for each segment. Which works best depends largely on your particular marketing automation system. Either way, shared components should keep the redundant work to a minimum.
• statistical modeling for segmentation: predictive model scores can often define segments more accurately than manual segmentations. Perhaps more important, they can be less labor-intensive to create, allowing marketers to build more segments and rebuild them more often. This matters because best practice marketing involves so many specialized campaigns and is constantly adjusting to new conditions.
• change campaign flow based on responses, events, activities: best practice campaigns change lead treatments in response to their behaviors. Thus, instead of a fixed sequence of treatments, they send leads down different branches and, in some cases, move them from one campaign to another. Changes may be triggered by activities within the campaign, such as response to a message or data provided within a form, or by information recorded elsewhere and reported to the marketing automation system.
• advanced scoring (complex rules, activity patterns, event depreciation, point caps): simple lead scoring formulas are often inaccurate predictors of future behavior. Best practice scoring may involve complex calculations based on relationships among several data elements, summarized activity detail, reduced value assigned to less recent events, and caps on the number of points assigned for any single type of activity. A related challenge for system designers is making complex formulas reasonably easy to set up and understand.
• company-level scores and activity tracking: the best practice campaign can use aggregated company or household data to calculate scores, guide individual treatments, and issue alerts. This allows more appropriate treatment than looking at each individual in isolation.
• multiple scores per lead: for companies with several products, the best practice to calculate a separate lead score for each. The scores may also have different thresholds for sending the lead to sales.
• define score formula jointly with sales: the salesperson is the ultimate judge of whether a lead is qualified. But many marketing departments still set up lead scoring formulas without sales input. Best practice is to work together on defining the criteria and then to periodically review the results to see if the formula can be improved.
• let sales return leads for more nurturing: traditional lead management is a one-way street, with leads sent from marketing to sales and then never heard from again. Best practice marketers allow salespeople to return leads to marketing for further nurturing. This improves the chances of a lead ultimately making a purchase, even if it doesn’t happen right away.
Step 5: Analyze Results. Learning from past campaigns may be the most important best practice of all. Having many targeted campaigns allows for continuous incremental improvement, achieved by quickly evaluating the return on each project and adjusting future programs based on the results.
• advanced response attribution: traditional methods often credit a lead to whichever campaign contacted them first, or whichever generated the first response. Best practice marketers look more deeply at the factors which may have influenced a lead’s behavior, often applying sophisticated analytics to estimate the incremental impact of different campaigns.
• standard metrics, within and across channels: resources can only be allocated to their optimal use if return on investment can be compared across campaigns. This requires standard metrics, which must be calculated consistently and clearly understood throughout the organization.
• formal test designs (a/b, multivariate): traditional marketers often do little testing, and the tests they do are often poorly designed. Best practice marketing involves continuous, formal testing designed to answer specific questions and lead to actionable results.
• capture immediate and long-term results: initial response rate or cost per lead fails to take into account the value of the leads generated, which can differ hugely from campaign to campaign. Best practice requires measuring the long-term value and building it into standard campaign metrics.
• evaluate on customer profitability, not revenue: customers with the same revenue can vary greatly in the actual profit they bring to the company, depending on the profit margins of their purchases and other costs such as customer support. Best practice metrics include accurate profitability measures, preferably drawn from an activity-based costing system.
• continually assess and reallocate spending: best practice marketers have a formal process to shift resources to the most productive marketing investments. These will change as campaigns are refined, business conditions evolve, and new opportunities emerge. A formal assessment process is essential because organizations otherwise tend to resist change.
Infrastructure. Individual campaigns are made possible by an underlying infrastructure that has best practices of its own.
• consolidated systems (multi-channel content management, campaign management and analytics): today’s marketing systems can usually handle multiple channels, so decommissioning older channel-based systems may save money as well as making multi-channel campaigns easier to execute. Consolidated multi-channel analytics, which may occur outside of the marketing automation system, are particularly important for gaining a complete view of each customer.
• advanced system training: marketing departments often provide workers with the minimum training needed to gain competency in their tools. Best practice departments recognize that additional training can make users more productive, particularly as the tools themselves add new capabilities that users would otherwise not be able to exploit.
• advanced analytics training: analytics play a central role in the continuous improvement process. Solid analytics training ensures that users can set up proper tests and interpret the results. Because data and tools are often already available, lack of training is frequently the main obstacle that prevents marketers from using analytics effectively.
• formal processes: best practice marketers develop, document and enforce formal, consistent business processes. This both ensures that work is done efficiently and makes it possible to execute changes when opportunities arise.
• cross-department cooperation: working with sales, service, finance and other departments is essential to sharing systems, data and metrics. A cross-department perspective ensures that each department considers the impact of its decisions on the rest of the company and on the customers themselves.
Summary
The best practice vision is many marketing campaigns, each precisely targeted, efficiently executed, and carefully designed to yield the greatest possible value. The campaigns are supported by detailed analysis to understand results and identify potential improvements. This information is quickly fed into new campaigns, ensuring that the company continually evolves its approaches and makes the best possible use of marketing resources. Continuous optimization is the ultimate best practice that all other practices should support.
Thursday, January 08, 2009
Company-Level Data in Demand Generation Systems
I had an interesting email conversation last month with a Raab Guide buyer about the nuances of company-level data management in demand generation systems. He started from the perfectly reasonable premise that the demand generation system should give an overview of activity for all leads associated with a given company. This is a topic we do cover in the Guide, but not in the detail he needed. The conversation has sharpened my own thinking on the subject and prompted a couple of conversations with vendors. I think it’s worth discussing here in some detail.
There are really two separate issues at play. The first is how the demand generation system treats company-level data. This, to start at the very beginning, is data about the company associated with an individual. Typically this is the company they work for, although there might be another relationship such as consultant or services vendor. From a database design standpoint, having one company record that is linked to multiple individuals avoids redundant data and, therefore, potential inconsistencies between company data for different people. (The technical term for this is “normalization”, meaning that the database is designed so a given piece of information is stored only once.) Some of the demand generation systems do indeed have a separate company table: it is one of the marks of a sophisticated design.
But the wrinkle here is that most CRM systems in general, and Salesforce.com in particular, also have separate company and individual levels. In fact, Salesforce.com actually has two types of individuals: “leads” which are unattached individuals, and “contacts”, which are individuals associated with an “account” (typically a company, although it might a smaller entity such as a division or department). Most demand generation systems make little distinction between CRM “leads” and “contacts”, converting them both to the same record type when data is loaded or synchronized.
The common assumption among demand generation vendors is that the CRM system is the primary source of information for which individuals are associated with which companies and for the company information itself. This makes sense, given that the salespeople who manage the CRM data are much closer to the companies and individuals than the marketers who run the demand generation system. Demand generation systems therefore generally import the company / individual relationships as part of their synchronization process. Systems with a separate company table store the imported company data in it; those without a separate company table copy the (same) company data onto each individual record. So far so good.
However, this raises the question of whether the demand generation system should be permitted to override company / individual relationships defined in the CRM system or to edit the company (or, for that matter, individual) data itself. I have to get back to the vendors and ask the question, but I believe that most vendors do NOT let the demand generation system assign individuals to companies or change the imported relationships. (In at least some cases, users can choose whether or not to allow such changes.) Interestingly enough, these limits apply even to systems that infer a visitor’s company from their IP address and show it in reports. Whether demand generation can change company-level data and have those changes flow back into the CRM system is less clear: again, it may be matter of configuration in some products. The vendors who don’t provide this capability will argue, probably correctly, that few marketers really want to do this and or in fact should do it.
So what information DOES originate in the demand generation system? Basically, it is new individuals, which correspond to “lead” records in Salesforce.com, and attributes for existing individuals, which may relate to either CRM “leads” or “contacts”. The demand generation system may also capture company information, but this is stored on the individual record and kept distinct from the company information in the CRM system. When a new individual is sent from demand generation to the CRM system, it is set up in CRM as a “lead” (that is, unattached to a company). The CRM user can later convert it to a contact within an account. But the demand generation system cannot generally set up accounts and contacts itself. (Again, let me stress that there may be some exceptions—I’ll let you know when I find out.)
Bottom line: leads, lead data and contact data may originate in either demand generation or CRM, and the synchronization is truly bi-directional in that changes made in either system will be copied to the other. But accounts and account / contact relationships are only maintained in the CRM system: most demand generation systems simply copy that data and don’t allow changes. Thus, it is essentially a unidirectional synchronization.
This leads us to the second issue, which is company-level reporting. It’s touted by most demand generation vendors, but a closer look reveals that some actually rely on the account-level reporting in Salesforce.com to deliver it. This isn’t necessarily a problem, since the Salesforce.com report can incorporate activities captured by the demand generation system. Per the earlier discussion, these activities will be linked to individuals (leads or contacts in Salesforce.com terms). Of course, if the individuals have not been linked to a company in the CRM system, they cannot be included in company-level reports.
One problem with relying on CRM for company-level reporting is that the demand generation system may have some nice reporting capabilities that you’d want to use. So there is some advantage to capturing the company / individual links within the demand generation system, even if the links themselves are created in CRM. (Not to get too technical here, but those “links” could simply be a company ID on the individual record, even if there is no separate company-level table in the data model. So lacking a company table does not preclude company-level reporting.)
A second issue is that some marketers want to use company-level information in their lead scoring calculations. This is another question we ask explicitly in the Guide, and only some companies say they can do it. Whether they make it simple is another question—some products who claim the facility in fact require considerable effort to make it happen. Again, the vendors who don’t offer this would presumably say that it isn’t very important to their clients.
I hope this clarifies both the mechanics of handling company-level data and some of underlying business issues. It’s one of those aspects of demand generation systems where vendor differences are real but subtle, and where the true importance of those differences is clear only to experienced users. All I can do in the Guide and this blog is try to describe things as clearly and accurately as possible, and thereby help you to make sound judgments about your own business needs.
There are really two separate issues at play. The first is how the demand generation system treats company-level data. This, to start at the very beginning, is data about the company associated with an individual. Typically this is the company they work for, although there might be another relationship such as consultant or services vendor. From a database design standpoint, having one company record that is linked to multiple individuals avoids redundant data and, therefore, potential inconsistencies between company data for different people. (The technical term for this is “normalization”, meaning that the database is designed so a given piece of information is stored only once.) Some of the demand generation systems do indeed have a separate company table: it is one of the marks of a sophisticated design.
But the wrinkle here is that most CRM systems in general, and Salesforce.com in particular, also have separate company and individual levels. In fact, Salesforce.com actually has two types of individuals: “leads” which are unattached individuals, and “contacts”, which are individuals associated with an “account” (typically a company, although it might a smaller entity such as a division or department). Most demand generation systems make little distinction between CRM “leads” and “contacts”, converting them both to the same record type when data is loaded or synchronized.
The common assumption among demand generation vendors is that the CRM system is the primary source of information for which individuals are associated with which companies and for the company information itself. This makes sense, given that the salespeople who manage the CRM data are much closer to the companies and individuals than the marketers who run the demand generation system. Demand generation systems therefore generally import the company / individual relationships as part of their synchronization process. Systems with a separate company table store the imported company data in it; those without a separate company table copy the (same) company data onto each individual record. So far so good.
However, this raises the question of whether the demand generation system should be permitted to override company / individual relationships defined in the CRM system or to edit the company (or, for that matter, individual) data itself. I have to get back to the vendors and ask the question, but I believe that most vendors do NOT let the demand generation system assign individuals to companies or change the imported relationships. (In at least some cases, users can choose whether or not to allow such changes.) Interestingly enough, these limits apply even to systems that infer a visitor’s company from their IP address and show it in reports. Whether demand generation can change company-level data and have those changes flow back into the CRM system is less clear: again, it may be matter of configuration in some products. The vendors who don’t provide this capability will argue, probably correctly, that few marketers really want to do this and or in fact should do it.
So what information DOES originate in the demand generation system? Basically, it is new individuals, which correspond to “lead” records in Salesforce.com, and attributes for existing individuals, which may relate to either CRM “leads” or “contacts”. The demand generation system may also capture company information, but this is stored on the individual record and kept distinct from the company information in the CRM system. When a new individual is sent from demand generation to the CRM system, it is set up in CRM as a “lead” (that is, unattached to a company). The CRM user can later convert it to a contact within an account. But the demand generation system cannot generally set up accounts and contacts itself. (Again, let me stress that there may be some exceptions—I’ll let you know when I find out.)
Bottom line: leads, lead data and contact data may originate in either demand generation or CRM, and the synchronization is truly bi-directional in that changes made in either system will be copied to the other. But accounts and account / contact relationships are only maintained in the CRM system: most demand generation systems simply copy that data and don’t allow changes. Thus, it is essentially a unidirectional synchronization.
This leads us to the second issue, which is company-level reporting. It’s touted by most demand generation vendors, but a closer look reveals that some actually rely on the account-level reporting in Salesforce.com to deliver it. This isn’t necessarily a problem, since the Salesforce.com report can incorporate activities captured by the demand generation system. Per the earlier discussion, these activities will be linked to individuals (leads or contacts in Salesforce.com terms). Of course, if the individuals have not been linked to a company in the CRM system, they cannot be included in company-level reports.
One problem with relying on CRM for company-level reporting is that the demand generation system may have some nice reporting capabilities that you’d want to use. So there is some advantage to capturing the company / individual links within the demand generation system, even if the links themselves are created in CRM. (Not to get too technical here, but those “links” could simply be a company ID on the individual record, even if there is no separate company-level table in the data model. So lacking a company table does not preclude company-level reporting.)
A second issue is that some marketers want to use company-level information in their lead scoring calculations. This is another question we ask explicitly in the Guide, and only some companies say they can do it. Whether they make it simple is another question—some products who claim the facility in fact require considerable effort to make it happen. Again, the vendors who don’t offer this would presumably say that it isn’t very important to their clients.
I hope this clarifies both the mechanics of handling company-level data and some of underlying business issues. It’s one of those aspects of demand generation systems where vendor differences are real but subtle, and where the true importance of those differences is clear only to experienced users. All I can do in the Guide and this blog is try to describe things as clearly and accurately as possible, and thereby help you to make sound judgments about your own business needs.
Tuesday, January 06, 2009
Raab Marketing Automation Webinar on January 14
Just a brief note to let you know that I'll be giving a Webinar on "How to Get the Most Value from Your Marketing Automation System" on Wednesday, January 14 at 4 p.m. Eastern / 1 p.m. Pacific. You can register at http://www.brighttalk.com/webcasts/2017/attend This is part of a two-day Marketing Automation Summit organized by BrightTALK, a vendor of webcasting technology that sponsors channels of specialized content, as well as running regular online conferences. As near as I can tell, BrightTALK makes its money selling the technology and related services: according to their Web site, you can have your own "channel" with unlimited webcasts for $949 per month. Whether their conferences are simply a way to promote this or have another revenue stream attached, I don't know.
In any case, my presentation on the 14th will look at short-term ways of getting value from existing systems, on the theory that money for new systems will be scarce in the near future. This is a bit of a switch from my usual focus on system selection. But system selection always boils down to how the system will be used, so it isn't much of a stretch.
Tune in if you have a chance, and if you're, say, recovering from a skiing accident over the holidays, you can listen to the full lineup of sessions starting on January 13. I'll also be participating in a round table discussion at 7 p.m. Eastern / 4 p.m. Pacific on the 13th.
In any case, my presentation on the 14th will look at short-term ways of getting value from existing systems, on the theory that money for new systems will be scarce in the near future. This is a bit of a switch from my usual focus on system selection. But system selection always boils down to how the system will be used, so it isn't much of a stretch.
Tune in if you have a chance, and if you're, say, recovering from a skiing accident over the holidays, you can listen to the full lineup of sessions starting on January 13. I'll also be participating in a round table discussion at 7 p.m. Eastern / 4 p.m. Pacific on the 13th.
Wednesday, December 24, 2008
ADVIZOR's In-Memory Database Supports Powerful Visualization
Back when I was writing a great deal about QlikView, I proposed that its fundamental value came from empowering business analysts to do work for themselves that would otherwise require IT support. (See, for example, this post, which has the virtue of pretty graphics.) This same notion of considering which users do which work has permiated my ideas of usability measurement for demand generation systems and usability in general. But to get back specifically to business intelligence systems, I think there is a particularly large gap between the capabilities available to business analysts and those available to IT. That is, even though the business intelligence systems like Cognos and Business Objects give analysts many ways to slice and present prepared data, they do not let analysts add new data or restructure existing data to meet new needs. This still requires the IT staff to design new data cubes and loading processes.
This gap is partly filled by analytical technologies such as columnar systems and database appliances, which can give good performance without schemas tailored for each task. But those systems are purchased and managed by the IT department, so they still leave analysts largely reliant on IT’s tender mercies.
A much larger portion of the gap is filled by products like QlikView, which the analysts can largely control for themselves. These can be divided into two subcategories: database engines like QlikView and illuminate, and visualization tools like Tableau and TIBCO Spotfire. The first group lets analysts do complex data manipulation and queries without extensive data modeling, while the latter group lets them do complex data exploration and presentation without formal programming. This distinction is not absolute: the database tools offer some presentation functions, and the visualization tools support some data manipulation. Both capabilities must be available for the analysts the work independently.
This brings us to ADVIZOR from ADVIZOR Solutions. ADVIZOR features an in-memory database and some data manipulation, but its primary strength is visualization. This includes at least fifteen chart types, including some with delightfully cool names like Multiscape, Parabox and Data Constellations. Analysts can easily configure these by selecting a few menu options. The charts are also somewhat interactive, allowing users to select records by clicking on a shape or drawing a box around data points. Some settings can be changed within the chart, such as selecting a measure to report on. Others, such as specifying the dimensions, require modifying the chart setup. The distinction won’t matter much to business analysts, who will have the tools build and modify the charts. But final consumers of the analyses typically run a viewer that does not permit changes to the underlying graph configuration.
On the other hand, that in-memory database can link several charts within a dashboard so selections made on one chart are immediately reflected in all others. This is arguably the greatest strength of the system, since it lets users slice data across many dimensions without writing complex queries. Colors are also consistent from one chart to the next, so that, for example, the colors assigned to different customer groups in a bar chart determine the color of the dot assigned to each customer in a scatter plot. Selecting a single group by clicking on its bar would turn the dots of all the other customers to gray. Keeping the excluded records visible in this fashion may yield more insight than simply removing them, although the system could also do that. These adjustments appear almost instantly even where millions of records are involved.
Dashboards can easily be shared with end-users through either a zero-footprint Web client or a downloadable object. Both use the Microsoft .NET platform, so Mac and Linux users need not apply. Images of ADVIZOR dashboards can easily be exported to Office documents, and can actually be manipulated from within Powerpoint if they are connected to the underlying dashboard. It’s also easy to export results such as lists of selected records.
Circling back to that database: it employs technology developed at Bell Labs during the 1990’s to support interactive visualization. The data model itself is a fairly standard one of tables linked by keys. Users can import data from text files, relational databases, Excel, Access, text files, Business Objects or Salesforce.com. They can map the table relationships and add some transformations and calculated fields during or after the import process. Although the mapping and transformations are executed interactively, the system records the sequence so the user can later edit it or repeat it automatically.
The import is fairly quick: the vendor said that an extract of three to four gigabytes across thirty tables runs in about twenty minutes, of which about five minutes is the build itself. The stored data is highly compressed but expands substantially when loaded into RAM: in the previous example, the three to four GB are saved as a 70 MB project file, but need 1.4 GB of RAM. The current version of ADVIZOR runs on 32 bit systems which limits it to 2-4 GB of RAM, although a 64 bit version is on track for release in January 2009. This will allow much larger implementations.
Pricing of ADVIZOR starts at $499 for a desktop version limited to Excel, Access or Salesforce.com source data and without table linking. (A 30-day trial version of this costs $49.) The full version starts at around $10,000, with additional charges for different types of user seats and professional services. Few clients pay less than $20,000 and a typical purchase is $50,000 to $60,000. Most buyers are business analysts or managers with limited technical skills, so the company usually helps set up their initial data loads and applications. ADVIZOR was introduced in 2004 and has several thousand end users, with a particular concentration in fund-raising for higher education. The bulk of ADVIZOR sales come through vendors who have embedded it within their own products.
This gap is partly filled by analytical technologies such as columnar systems and database appliances, which can give good performance without schemas tailored for each task. But those systems are purchased and managed by the IT department, so they still leave analysts largely reliant on IT’s tender mercies.
A much larger portion of the gap is filled by products like QlikView, which the analysts can largely control for themselves. These can be divided into two subcategories: database engines like QlikView and illuminate, and visualization tools like Tableau and TIBCO Spotfire. The first group lets analysts do complex data manipulation and queries without extensive data modeling, while the latter group lets them do complex data exploration and presentation without formal programming. This distinction is not absolute: the database tools offer some presentation functions, and the visualization tools support some data manipulation. Both capabilities must be available for the analysts the work independently.
This brings us to ADVIZOR from ADVIZOR Solutions. ADVIZOR features an in-memory database and some data manipulation, but its primary strength is visualization. This includes at least fifteen chart types, including some with delightfully cool names like Multiscape, Parabox and Data Constellations. Analysts can easily configure these by selecting a few menu options. The charts are also somewhat interactive, allowing users to select records by clicking on a shape or drawing a box around data points. Some settings can be changed within the chart, such as selecting a measure to report on. Others, such as specifying the dimensions, require modifying the chart setup. The distinction won’t matter much to business analysts, who will have the tools build and modify the charts. But final consumers of the analyses typically run a viewer that does not permit changes to the underlying graph configuration.
On the other hand, that in-memory database can link several charts within a dashboard so selections made on one chart are immediately reflected in all others. This is arguably the greatest strength of the system, since it lets users slice data across many dimensions without writing complex queries. Colors are also consistent from one chart to the next, so that, for example, the colors assigned to different customer groups in a bar chart determine the color of the dot assigned to each customer in a scatter plot. Selecting a single group by clicking on its bar would turn the dots of all the other customers to gray. Keeping the excluded records visible in this fashion may yield more insight than simply removing them, although the system could also do that. These adjustments appear almost instantly even where millions of records are involved.
Dashboards can easily be shared with end-users through either a zero-footprint Web client or a downloadable object. Both use the Microsoft .NET platform, so Mac and Linux users need not apply. Images of ADVIZOR dashboards can easily be exported to Office documents, and can actually be manipulated from within Powerpoint if they are connected to the underlying dashboard. It’s also easy to export results such as lists of selected records.
Circling back to that database: it employs technology developed at Bell Labs during the 1990’s to support interactive visualization. The data model itself is a fairly standard one of tables linked by keys. Users can import data from text files, relational databases, Excel, Access, text files, Business Objects or Salesforce.com. They can map the table relationships and add some transformations and calculated fields during or after the import process. Although the mapping and transformations are executed interactively, the system records the sequence so the user can later edit it or repeat it automatically.
The import is fairly quick: the vendor said that an extract of three to four gigabytes across thirty tables runs in about twenty minutes, of which about five minutes is the build itself. The stored data is highly compressed but expands substantially when loaded into RAM: in the previous example, the three to four GB are saved as a 70 MB project file, but need 1.4 GB of RAM. The current version of ADVIZOR runs on 32 bit systems which limits it to 2-4 GB of RAM, although a 64 bit version is on track for release in January 2009. This will allow much larger implementations.
Pricing of ADVIZOR starts at $499 for a desktop version limited to Excel, Access or Salesforce.com source data and without table linking. (A 30-day trial version of this costs $49.) The full version starts at around $10,000, with additional charges for different types of user seats and professional services. Few clients pay less than $20,000 and a typical purchase is $50,000 to $60,000. Most buyers are business analysts or managers with limited technical skills, so the company usually helps set up their initial data loads and applications. ADVIZOR was introduced in 2004 and has several thousand end users, with a particular concentration in fund-raising for higher education. The bulk of ADVIZOR sales come through vendors who have embedded it within their own products.
Thursday, December 18, 2008
Simplifying Demand Generation Usability Assessment: No Obvious Answers
My feelings are hurt, people. No one has commented on last week’s post about usability measurement. I know it’s not the world’s most fascinating topic but I really wanted some feedback. And I do, after all, know how many people visit the site each day. Based on those numbers, there are a lot of you who have chosen not to help me.
Oh well, no grudges here -- ‘tis the season and all that. I’m guessing the reason for the lack of comment is that the proposed methodology was too complex for people to take the time to assess and critique. In fact, the length of the post itself may be an obstacle, but that in turn reflects the complexity of the approach it describes. Fair enough.
So the question is, how do you simplify the methodology and still provide something useful?
- One approach would be to reduce the scope of the assessment. Instead of defining scenarios for all types of demand generation processes, pick a single process and just analyze that. Note that I said “single” not “simple”, because you want to capture the ability of the systems to do complicated things as well. This is a very tempting path and I might try it because it’s easy to experiment with. But it still raises all the issues of how you determine which tasks are performed by which types of users and how you account for the cost of hand-offs between those users. This strikes me as a very important dimension to consider, but I also recognize that it introduces quite a bit of complexity and subjectivity into the process. I also recognize that measuring even a single process will require measuring system set-up, content creation and other preliminary tasks. Thus, you still need to do a great deal of work to get metrics on one task, and that task isn’t necessarily representative of the relative strengths of the different vendors. This seems like a lot of effort for a meager result.
- Another approach would be to ask users rather than trying to run the tests independently. That is, you would do a survey that lists the various scenarios and asks users to estimate the time they require and how this is distributed among different user types. That sounds appealing insofar as now someone else does the work, but I can’t imagine how you would get enough data to be meaningful, or how you would ensure different users’ responses were consistent.
- A variation of this approach would be to ask vastly simpler questions – say, estimate the time and skill level needed for a half-dozen or so typical processes including system setup, simple outbound email campaign, setting up a nurturing campaign, etc. You might get more answers and on the whole they’d probably be more reliable, but you’re still at the mercy of the respondents’ honesty. Since vendors would have a major incentive to game the system, this is a big concern. Nor is it clear what incentives users would have to participate, how you screen out people with axes to grind, or whether users would be constrained by non-disclosure agreements from participating. Still, this may be the most practical of all the approaches I’ve come up with, so perhaps it’s worth pursuing. Maybe we get a sample of ten clients from each vendor? Sure they’d be hand-picked, but we could still hope their answers would accurately reflect any substantial differences in workload by vendor.
- Or, we could ask the vendors to run their own tests and report the results. But who would believe them? Forget it.
- Maybe we just give up on any kind of public reporting, and provide buyers with the tools to conduct their own evaluations. I think this is a sound idea and actually mentioned it in last week’s post. Certainly the users themselves would benefit, although it’s not clear how many buyers really engage in a detailed comparative analysis before making their choice. (For what it’s worth, our existing usability worksheet is the third most popular download from the Raab Guide site. I guess that’s good news.) But if the results aren’t shared, there is no benefit to the larger community. We could offer the evaluation kit for free in return for sharing the results, but I doubt this is enough of an incentive. And you’d still have the issue of ensuring that reported results are legitimate.
So there you have it, folks. I’m between a rock and a hard place. No matter how much I talk about usability, the existing Raab Guide mostly lists features, and people will use it to compare systems on that basis. But I can’t find a way to add usability to the mix that’s both objective and practical. Not sure where to go next.
Oh well, no grudges here -- ‘tis the season and all that. I’m guessing the reason for the lack of comment is that the proposed methodology was too complex for people to take the time to assess and critique. In fact, the length of the post itself may be an obstacle, but that in turn reflects the complexity of the approach it describes. Fair enough.
So the question is, how do you simplify the methodology and still provide something useful?
- One approach would be to reduce the scope of the assessment. Instead of defining scenarios for all types of demand generation processes, pick a single process and just analyze that. Note that I said “single” not “simple”, because you want to capture the ability of the systems to do complicated things as well. This is a very tempting path and I might try it because it’s easy to experiment with. But it still raises all the issues of how you determine which tasks are performed by which types of users and how you account for the cost of hand-offs between those users. This strikes me as a very important dimension to consider, but I also recognize that it introduces quite a bit of complexity and subjectivity into the process. I also recognize that measuring even a single process will require measuring system set-up, content creation and other preliminary tasks. Thus, you still need to do a great deal of work to get metrics on one task, and that task isn’t necessarily representative of the relative strengths of the different vendors. This seems like a lot of effort for a meager result.
- Another approach would be to ask users rather than trying to run the tests independently. That is, you would do a survey that lists the various scenarios and asks users to estimate the time they require and how this is distributed among different user types. That sounds appealing insofar as now someone else does the work, but I can’t imagine how you would get enough data to be meaningful, or how you would ensure different users’ responses were consistent.
- A variation of this approach would be to ask vastly simpler questions – say, estimate the time and skill level needed for a half-dozen or so typical processes including system setup, simple outbound email campaign, setting up a nurturing campaign, etc. You might get more answers and on the whole they’d probably be more reliable, but you’re still at the mercy of the respondents’ honesty. Since vendors would have a major incentive to game the system, this is a big concern. Nor is it clear what incentives users would have to participate, how you screen out people with axes to grind, or whether users would be constrained by non-disclosure agreements from participating. Still, this may be the most practical of all the approaches I’ve come up with, so perhaps it’s worth pursuing. Maybe we get a sample of ten clients from each vendor? Sure they’d be hand-picked, but we could still hope their answers would accurately reflect any substantial differences in workload by vendor.
- Or, we could ask the vendors to run their own tests and report the results. But who would believe them? Forget it.
- Maybe we just give up on any kind of public reporting, and provide buyers with the tools to conduct their own evaluations. I think this is a sound idea and actually mentioned it in last week’s post. Certainly the users themselves would benefit, although it’s not clear how many buyers really engage in a detailed comparative analysis before making their choice. (For what it’s worth, our existing usability worksheet is the third most popular download from the Raab Guide site. I guess that’s good news.) But if the results aren’t shared, there is no benefit to the larger community. We could offer the evaluation kit for free in return for sharing the results, but I doubt this is enough of an incentive. And you’d still have the issue of ensuring that reported results are legitimate.
So there you have it, folks. I’m between a rock and a hard place. No matter how much I talk about usability, the existing Raab Guide mostly lists features, and people will use it to compare systems on that basis. But I can’t find a way to add usability to the mix that’s both objective and practical. Not sure where to go next.
Saturday, December 13, 2008
A Modest Proposal for Demand Generation Usability Measurement
As Tuesday’s post suggested, my thoughts on usability measurement have now crystallized. To provide a meaningful and consistent comparison of usability across demand generation vendors, you could:
1. Define a set of business scenarios that must be supported by the system. Each scenario would describe a type of marketing campaign and the system tasks required to run it. These tasks would cover system set-up, materials creation, campaign design, execution and evaluation. Some tasks would be common to several scenarios, while others would be unique. For example, pretty much any kind of campaign would involve creating an email, but only some campaigns require adding custom fields to the system database.
The result would be a grid with tasks listed down the side, scenarios across the top, and checkmarks showing which tasks are used in which scenarios. A small sample is below. Note that you can build a single task list for any combination of scenarios by simply combining the checkmarks in their columns. Thus, in the sample table, scenarios 1 and 2 require tasks 1, 2 and 3. In many cases, there will be two entries for each task, one for setting it up and another for repeating it.
| scenario 1 | scenario 2 | scenario 3 | ... | |
| task 1 | x | x | x | |
| task 2 | x | x | ||
| task 3 | x | |||
| task 4 | x | |||
| ... |
2. Develop a specific package for each scenario, with both a task list and standard materials such as a list of users to set up, data elements to capture, email contents to produce, campaign logic, etc. You also need a standard Salesforce.com installation and perhaps company Web site to integrate during testing. Assembling these packages would be quite a bit of work, but only has to be done once. The project could start with relatively simple packages and expand them over time.
3. Have an expert user (typically a vendor employee) run through the required tasks while the tester tracks their time and results. As I noted in the earlier post, this means a single expert user is simulating different users at a real client. (These are marketing managers, operations specialists, system administrators, database managers, etc.) This makes sense if we assume that the client users will all be experts in their own areas. But it also means that the tester must assess which type of user would perform each task. The test packages would include score sheets to make capturing this information as easy as possible.
4. Check the test results by executing the scenario campaigns and identifying any errors. You need this step to ensure the work was actually completed correctly—otherwise, the experts could simply zoom through their tasks without worrying about accuracy. Part of the process would be for the tester to “respond” to the promotions to ensure that the system reacts correctly. This is another labor-intensive process. Results will be summarized in an error report that is part of the final evaluation.
5. Have users select their scenarios they wish to evaluate. Then generate reports for the tasks in those scenarios, showing the (a) tasks completed (b) time required (c) workload on different users and (d) error rates. Comparing the results for different systems will give a good sense of strengths and weaknesses.
* * *
Of course, the process won’t end with the detailed reports. People will want to combine the results in a single score that can be used to rank the vendors. **sigh**. Building such a score requires adjusting for factors including:
- differences in system features (some systems lack some features and, thus, can’t execute all the specified tasks)
- differences in the importance of different tasks
- differences in the value of difference users’ time
- the impact of handing off tasks among users (this adds time and errors that won’t be captured in a single-user test)
- differences in error rates and in the importance of different errors
- differences in the mix of tasks and their importance at different companies
I’m sure there are other factors as well. A simple approach might just be to assign scores for each separate dimension, say on a 1-10 scale, and then add them for a combined score. You could evolve a more elaborate approach over time, but the resulting figures will never have any specific meaning. Still, they should provide a reasonably valid ranking of the competitors.
The report could also supplement or replace the single figure with a graph that plots the results on two or more dimensions. For example, a classic scatter plot could position the each system based on breadth (number of tasks completed) vs. productivity (error-free tasks completed per hour). This would more clearly illustrate the trade offs between the products.
The good news in all this is that the closer you get to a specific company’s requirements, the more you can replace any generic assumptions with that company’s own data. This means that any aggregate score becomes much more meaningful for that company.
Let me clarify and expand that last point, because it’s very important. The tests just need to be done once because the results (tasks completed, work time, user workload, error rate) don’t change based on the individual client. So you could store those results in a database and then apply client-specific parameters such as scenarios chosen, task mix, and user costs to get a client-specific ranking without actually conducting any client-specific tests.
Of course, no one would purchase a system based solely on someone else’s tests. But having the data available could greatly speed the evaluation process and improve buyers’ understanding of the real differences between systems. In addition, the test scenarios themselves should be help buyers to decide what they want to see demonstrated.
(Before I lose track of it, let me point out that this approach doesn’t address the ease-of-learning component of usability. That requires a larger base of testers, or at least an assessment of what looks difficult to learn. It’s possible that assessing ease-of-learning really involves the same judgment as assessing which type of user will perform each task. Both, after all, are based on how hard the task looks. In any case, this issue needs more thought.)
What Do You Think?
Well, this all sounds just great to me, but I’m not an objective observer. I’m really curious to learn what you think (on the assumption that “you”, the readers of this blog, include many demand generation vendors and users).
Users: Would you use something like this during the selection process? Would you be able to prioritize your requirements and estimate the numbers of different tasks (campaigns, emails, landing pages, etc.) per year? What would you pay for a customized report based on your inputs? Would you prefer to do this sort of testing for yourself? If so, would you pay for the task lists and scenario packages to help you run your own tests? Would you want consulting help with conducting those tests? In general, do you actually conduct a detailed vendor comparison before making a choice?
Vendors: Does this approach seem fair? Is it very different from the standard scenarios you’ve already worked up for training and sales demonstrations? Would the standard task lists and scenarios make it easier to gather prospects' business requirements? Would the test results accelerate your sales cycles and deployment times? Would testing provide a useful benchmark for your development efforts? Would you participate in the testing, knowing the results were going to be published in a database? Would you pay to have the tests done? Would you help to fund initial development?
Everybody: Does this make sense? What flaws or risks do you see? What’s the best way to build the scenarios, task lists and packages? In particular, could be they be built cooperatively (“crowd sourced”) with a Wiki or something similar?
Please comment and ask others to comment as well.
Tuesday, December 09, 2008
Measuring Usability: A Task-Based Approach
I think we all know that the simplest practical measure of intelligence is how often someone agrees with you. On that scale, University of Ottawa Professor Timothy Lethbridge must be some kind of genius, because his course notes on Software Usability express my opinions on the topic even better and in more detail than I’ve yet to do for myself. Specifically, he lists the following basic process for measuring usability:
- understand your users, and recognize that they fall into different classes
- understand the tasks that users will perform with the system
- pick a representative set of tasks
- pick a representative set of users
- define the questions you want to answer about usability
- pick the metrics that answer those questions
- have the users perform the tasks and measure their performance
This is very much the approach that I’ve been writing about, in pretty much the same words. Happily, Lethbridge provides additional refinement of the concepts. Just paging through his notes, some of his suggestions include:
- classifying users in several dimensions, including the job type, experience with the tasks, general computer experience, personality type, and general abilities (e.g. language skills, physical disabilities, etc.). I’d be more specific and add skills such as analytical or technical knowledge.
- defining tasks based on use cases (I tend to call these business processes, but it’s pretty much the same); understanding how often each task is performed, how much time it takes, and how important it is; and testing different tasks for different types of users. “THIS STEP CAN BE A LOT OF WORK” the notes warn us, and, indeed, building the proper task list is probably the hardest step in the whole process.
- a list of metrics:
- proficiency, defined as the time to complete the chosen tasks. That strikes me as an odd label, since I usually think of proficiency as an attribute of a user not a system. The obvious alternative is efficiency, but as we’ll see in a moment, he uses that for something else. Maybe “productivity” would be better; I think this comes close to the standard definition of labor productivity as output per hour.
- learnability, defined as time to reach a specified level of proficiency.
- efficiency, defined as proficiency of an expert. There’s no corresponding term for “proficiency of a novice”, which I think there should be. So maybe what you really need is “expert efficiency” and “novice efficiency”, or “expert and novice “productivity”, and discard “proficiency” altogether.
- memorability, defined as proficiency after a period of non-use. If you discard proficiency, this could be “efficiency (or productivity) after a period of non-use”, which makes just as much sense.
- error handling, defined as number or time spent on deviations from the ideal way to perform a task. I’m not so sure about this one. After all, time spent on deviations is part of total time spent, which is already captured in proficiency or efficiency or whatever you call it. I’d rather see a measure of error rate, which would be defined as number or percentage of tasks performed correctly (by users with a certain level of training). Now that I think about it, none of Lethbridge’s measures incorporate any notion of output quality—a rather curious and important omission.
- satisfaction, defined subjectively by users on a scale of 1 to 5.
- plot a “learning curve” on the two dimensions of proficiency and training / practice time; the shape of the curve provides useful insights into novice productivity (what can new users do without any training); learnability (a steep early curve means people learn the system quickly) and eventual efficiency (the level of proficiency where the curve flattens out).
- even expert users may not make best use of the system if stop learning before they master all its features. So they system should lead them to explore new features by offering tips or making contextual suggestions.
At this point, we’re about half way through the notes. The second half provides specific suggestions on:
- measuring learnability (e.g. by looking at features that make systems easy to learn);
- causes of efficiency problems (e.g. slow response time, lack of an easy step-by-step route to perform a task);
- choosing experts and what to do when experts are unavailable (basically, plot of learning curve of new users);
- measuring memorability (which may involve different retention periods for different types of tasks; and should also distinguish between frequently and infrequently used tasks, with special attention to handling emergencies)
- classifying errors (based on whether they were caused by user accidents or confusion [Lethbridge says that accidents are not the system’s fault while confusion is; this is not a distinction I find convincing]; also based on whether the user discovers them immediately or after some delay, the system points them out, or they are never made known to the user)
- measuring satisfaction (surveys should be based on real and varied work rather than just a few small tasks, should be limited to 10-15 questions, should use a “Likert Scale” of strongly agree to strongly disagree, and should vary the sequence and wording of questions)
- measuring different classes of users (consider their experience with computers, the application domain and the system being tested; best way to measure proficiency differences is to compare the bottom 25% of users with the 3rd best 25%, since this will eliminate outliers)
This is all good stuff. Of course, my own interest is applying it to measuring usability for demand generation systems. My main take-aways for that are:
1. defining user types and tasks to measure are really important. But I knew that already.
2. choosing the actual metrics takes more thought than I’ve previously given it. Time to complete the chosen tasks (I think I’ll settle on calling it productivity) is clearly the most important. But learnability (which I think comes down to time to reach a specified level of expertise) and error rate matter too.
For marketing automation systems in particular, I think it’s reasonable to assume that all users will be trained in the tasks they perform. (This isn’t the case for other systems, e.g. ATM machines and most consumer Web sites, which are used by wholly untrained users.) The key to this assumption is that different tasks will be the responsibility of different users; otherwise, I’d be assuming that all users are trained in everything. So it does require determining which users will do which tasks in different systems.
On the other hand, assuming that all tasks are performed by experts in those tasks does mean that someone who is expert in all tasks (e.g., a vendor sales engineer) can actually provide a good measure of system productivity. I know this is a very convenient conclusion for me to reach, but I swear I didn’t start out aiming for it. Still, I do think it’s sound and it may provide a huge shortcut in developing usability comparisons for the Raab Guide. What is does do is require a separate focus on learnability so we don’t lose sight of that one. I’m not sure what to do about error rate, but do know it has to be measured for experts, not novices. Perhaps when we set up the test tasks, we can involve specific contents that can later be checked for errors. Interesting project, this is.
3. the role of surveys is limited. This is another convenient conclusion, since statistically meaningful surveys would require finding a large number of demand generation system users and gathering detailed information about their levels of expertise. It would still be interesting to do some preliminary surveys of marketers to help understand the tasks they find important and, to the degree possible, to understand the system features they like or dislike. But the classic usability surveys that ask users how they feel about their systems are probably not necessary or even very helpful in this situation.
This matters because much of the literature I’ve seen treats surveys as the primary tool in the usability measurement. This is why I am relieved to find an alternative.
As an aside: many usability surveys such as SUMI (Software Usability Measurement Inventory) are proprietary. My research did turn up what looks like a good public version
Measuring Usability with the USE Questionnaire by Arnold M. Lund from the
Society for Technical Communication (STC) Usability SIG Newsletter of October 2001. The acronym USE stands for the three main categories: Usefulness, Satisfaction and Ease of Use/Ease of Learning. The article provides a good explanation of the logic behind the survey, and is well worth reading if you’re interested in the topic. The questions, which would be asked on a 7-point Likert Scale, are:
Usefulness
- It helps me be more effective.
- It helps me be more productive.
- It is useful.
- It gives me more control over the activities in my life.
- It makes the things I want to accomplish easier to get done.
- It saves me time when I use it.
- It meets my needs.
- It does everything I would expect it to do.
Ease of Use
- It is easy to use.
- It is simple to use.
- It is user friendly.
- It requires the fewest steps possible to accomplish what I want to do with it.
- It is flexible.
- Using it is effortless.
- I can use it without written instructions.
- I don't notice any inconsistencies as I use it.
- Both occasional and regular users would like it.
- I can recover from mistakes quickly and easily.
- I can use it successfully every time.
Ease of Learning
- I learned to use it quickly.
- I easily remember how to use it.
- It is easy to learn to use it.
- I quickly became skillful with it.
Satisfaction
- I am satisfied with it.
- I would recommend it to a friend.
- It is fun to use.
- It works the way I want it to work.
- It is wonderful.
- I feel I need to have it.
- It is pleasant to use.
Apart from the difficulties of recruiting and analyzing a large enough number of respondents, this type of survey only gives a general view of the product in question. In the case of demand generation, this wouldn’t allow us to understand the specific strengths and weaknesses of different products, which is a key objective of any comparative research. Any results from this sort of survey would be interesting in their own right, but couldn’t themselves provide a substitute for the more detailed task-based research.
Two Interesting Blogs on Demand Generation
I noticed a visitor to the Raab Guide to Demand Generation Systems site from a new source the other day, which turned out to be a mention on blog by Jason Stewart of Demandbase. The context was a discussion of system selection and the Raab Guide was paired with a reference to a very excellent series of posts on system selection on Maria Pergolino’s Inbound Marketer blog. Actually, only two of promised three parts are posted, but the first two are well worth a look. In particular, part 2 has a long list of evaluation criteria. I haven’t compared it with my own but it’s surely a good starting point for anyone interested in that sort of thing. Pergolino’s blog also has an entry with a long list of “Marketing Automation People on Twitter”, which has already wasted several hours of my precious time. Thanks a bunch, Maria.
Wednesday, December 03, 2008
Pardot Offers Refined Demand Generation at a Small Business Price
My little tour of demand generation vendors landed at Pardot just before Thanksgiving. As you’ll recall from my post on Web activity statistics, Pardot is one of the higher-ranked vendors not already in the Raab Guide to Demand Generation Systems. So I was quite curious to see what they had to offer.
What I found was intriguing. While last week’s post found that Marketbright aims at more sophisticated clients, Pardot explicitly targets small and midsize businesses (or SMBs as we fondly acronymize them [yes, that’s a word, at least according to http://www.urbandictionary.com/]). Actually I don’t know why I find the contrast between Pardot and Marketbright intriguing, except for the implication that marketers can be divided into two simple categories, SMB and Enterprise, and no further distinctions are necessary. The analyst in me rejects this as an obvious and appalling over-simplification, but there’s a sneaky, almost guilty pleasure in contemplating whether it might be correct.
What’s odd about the SMB vs. Enterprise dichotomy is that both sets of systems are quite similar. Pardot and other SMB systems don’t just offer a few simple features. In fact, Pardot in particular provides advanced capabilities including progressive profiling (automatically changing the questions on forms as customer answer them) and dynamic content (rule-driven selection of content blocks within emails and Web pages). The only common feature that’s missing in Pardot is rule-based branching within multi-step programs. Even this is far from a fatal flaw, since (a) users can simulate it with rules that move customers from one program to another and (b) intra-program branching will be added by the end of this month.
What really distinguishes the Enterprise vendors is the ability to limit different users to different tasks. This involves rights management and content management features that seem arcane but are nevertheless critical when marketing responsibilities are divided by function, channel, region and product organizations. Although enterprise marketing programs are more complex than SMB programs, most SMB systems can actually handle complex programs quite well. Conversely, although SMB vendors stress their products’ ease of use, simple things are not necessarily harder to do in the Enterprise products. I’m still trying to work out a systematic approach to measuring usability, but my current feeling is that there are large variations among products with both the SMB and Enterprise groups.
Back to Pardot. It certainly considers ease of use to be one of its advantages, and I saw nothing to contradict this. Functionally, it has all the capabilities you’d expect of a demand generation product: users create personalized emails and Web pages with a drag-and-drop interface; track responders with cookies; look up visitors' companies based on their IP address; run multi-step drip marketing campaigns; score leads based on activities and attributes; and integrate tightly with Salesforce.com and other CRM systems. These are nicely implemented with refinements including:
- integrated site search, including the ability to use visitor queries as part of their behavior profiles (something I haven’t seen in other demand generation products)
- different scoring rules for customers in different segments (other systems could achieve this but not so directly)
- auto-response messages tied to completion of an email form (again, this often requires more work in other systems)
- ability to post data from externally-hosted forms via API calls (not just batch file imports) and to forward posted data to external systems
- email address validation that goes beyond the format checking available in most products to include rejecting addresses from free domains such as gmail or yahoo, and validating that the address is active on the specified host
- ability to capture campaign costs by importing data from Google AdWords and other sources
- plug-ins that let the system track emails sent by Outlook, Thunderbird and Apple email clients
This is an impressive list that suggests a thoughtfully designed system. But I didn’t check Pardot against my full list of possible features, so don’t get the impression that it does everything. For example, its approach to revenue reporting is no better than average: the system imports revenue from the sales automation opportunity records, and then assigns it to the first campaign of the associated lead. This is a common approach, but quite simplistic. More sophisticated methods give more control over which campaigns are credited and can divide revenue among multiple campaigns. Nor does Pardot have the refined user rights management and content management features associated with enterprise systems. It also limits database customization to adding user-defined fields to the prospect table. (This is another area where Enterprise vendors tend to be more flexible than SMB systems, albeit with considerable variation within each group.)
The point here is that Pardot, like all systems, has its own strengths and weaknesses. This is why the simple SMB vs. Enterprise dichotomy isn’t enough. People who need specific features won’t necessarily find them in all products of one group or the other. You really do have to look closely at the individual products before making a choice. QED.
One other factor clearly distinguishes SMB from Enterprise systems, and that’s pricing. Pardot’s lowest-price system, $500 per month, may be too constrained for most companies (no CRM integration, maximum of five landing pages, etc.),. But its $750 per month offering should be practical for many SMBs and a $1,250 per month option allows still higher volumes. (Pricing details are published on their Web site – which is itself typical of SMB products.) This pricing is low even among SMB demand generation systems. By comparison, limited versions cost $1,500 per month for Marketo and $1,000 for Manticore Technology, and both charge $2,400 per month for their cheapest complete offering. (Note: other SMB-oriented vendors including ActiveConversion and OfficeAutoPilot also have entry pricing in the $500 per month range, although neither publishes the details.)
The Pardot product began as an internal project for the marketing group at Hannon Hill, a content management system developer. Pardot was spun off about two years ago and launched its product at the end of 2007. It recently signed its 100th client.
Tuesday, November 25, 2008
Marketbright Targets Sophisticated Demand Generation Users
I had a preliminary conversation last week with Mike Pilcher of Marketbright, one of the vendors I’ll probably end up adding to the Raab Guide to Demand Generation Systems. We didn’t look at the software itself, so I can’t comment on it in any detail. The slides did list a few unusual features, including “prospect portals” that help buyers and sellers to share information related to a project; a sales proposal builder; and features to work with sales partners. These seem pretty minor, although they do insert Marketbright more deeply into the sales process than most demand generation products. This is something of a theme for the company, although all demand generation vendors integrate closely with sales systems.
Marketbright sees its most important differentiator as a sophisticated architecture designed to coordinate marketing activities throughout a large organization. This doesn't strike me as a very effective selling point: buying a product because of its architecture is the software equivalent of reading Playboy for the articles. (Do I get credit for resisting the temptation to link to Playboy.com?) What really matters are the features facilitated by this architecture. According to the company Web site, these include “full document repository and asset management, multi-currency budget planning and management and a range of integrated collaboration features”. Now that's something to get excited about. Hubba hubba, eh?
At least this clarifies which end of the demand generation market will find Marketbright most attractive. Indeed, Pilcher told me that product sells best to people who have worked in large organizations and seen first-hand what it takes to support collaboration within marketing. These people may currently be working in small firms, so Marketbright has ended up with customers of all sizes. Pricing ranges from $20,000 to $200,000 per year based on the modules and number of users, so the system is financially competitive facross the spectrum.
Not having seen the product, I don’t know whether its sophisticated management features come at the price of end-user complexity. This is a common trade-off. One hint of Marketbright’s approach may be that Pilcher recommends his clients build separate campaigns for different customer segments, rather than “boiling the ocean” by creating a single campaign with branches to handle all contingencies. This suggests that Marketbright has at least tried to keep things as simple.
Pilcher and I had a lengthy subsequent email discussion about usability, “violently agreeing” that it’s an important though elusive measure. My final conclusion was similar to the positions I’ve taken before: usability has to be measured separately for different functions, levels of campaign sophistication, and user skill sets. Where I may have changed my mind is a grudging agreement that it’s legitimate to summarize the details into simple measures that could be plotted in a graph. The obvious ones are usability and functionality scores. I still fear this could mislead by obscuring important information: for example, deep functionality in a few areas could generate the same score as limited functionality across many areas. (Pilcher proposed the number of channels as a separate dimension, but then a system with weak functionality in many channels scores better than a system that is strong in just a few. I consider that equally misleading.) But if a two-dimensional summary offers an attractive entry point which in turn leads to deeper exploration, it’s better than scaring people away by showing them the details at the start.
Marketbright sees its most important differentiator as a sophisticated architecture designed to coordinate marketing activities throughout a large organization. This doesn't strike me as a very effective selling point: buying a product because of its architecture is the software equivalent of reading Playboy for the articles. (Do I get credit for resisting the temptation to link to Playboy.com?) What really matters are the features facilitated by this architecture. According to the company Web site, these include “full document repository and asset management, multi-currency budget planning and management and a range of integrated collaboration features”. Now that's something to get excited about. Hubba hubba, eh?
At least this clarifies which end of the demand generation market will find Marketbright most attractive. Indeed, Pilcher told me that product sells best to people who have worked in large organizations and seen first-hand what it takes to support collaboration within marketing. These people may currently be working in small firms, so Marketbright has ended up with customers of all sizes. Pricing ranges from $20,000 to $200,000 per year based on the modules and number of users, so the system is financially competitive facross the spectrum.
Not having seen the product, I don’t know whether its sophisticated management features come at the price of end-user complexity. This is a common trade-off. One hint of Marketbright’s approach may be that Pilcher recommends his clients build separate campaigns for different customer segments, rather than “boiling the ocean” by creating a single campaign with branches to handle all contingencies. This suggests that Marketbright has at least tried to keep things as simple.
Pilcher and I had a lengthy subsequent email discussion about usability, “violently agreeing” that it’s an important though elusive measure. My final conclusion was similar to the positions I’ve taken before: usability has to be measured separately for different functions, levels of campaign sophistication, and user skill sets. Where I may have changed my mind is a grudging agreement that it’s legitimate to summarize the details into simple measures that could be plotted in a graph. The obvious ones are usability and functionality scores. I still fear this could mislead by obscuring important information: for example, deep functionality in a few areas could generate the same score as limited functionality across many areas. (Pilcher proposed the number of channels as a separate dimension, but then a system with weak functionality in many channels scores better than a system that is strong in just a few. I consider that equally misleading.) But if a two-dimensional summary offers an attractive entry point which in turn leads to deeper exploration, it’s better than scaring people away by showing them the details at the start.
Wednesday, November 19, 2008
Ranking the Demand Generation Vendors by Popularity (Yes, Life Really Is Just Like High School)
As you might imagine, I’ve been trying to decide how to expand the set of products covered in the Raab Guide to Demand Generation Systems. My original plan had been to add several marketing automation vendors with significant presence in this market. The tentative list is Unica, Aprimo, Alterian, and Neolane.
But I’ve also been approached by some of the other demand generation specialists. My original set of products was based on a general knowledge of which companies are most established, plus some consultation with vendors to learn who they felt were their main competitors. So far the original list of Eloqua, Vtrenz, Marketo, Manticore Technology and Market2Lead has proven a good set of choices. Yet there are so many more vendors I could add. How to choose?
The general rule is pretty obvious: pick the vendors that people are most interested in. We do, after all, want people to buy this thing. Of course, you want some wiggle room to add intriguing new products that they may not know about. Still, you mostly want to the report to include the vendors they are already asking about.
But although the general rule is obvious, which vendors are most popular is not. Fortunately, we have the Internet to help. It offers quite a few ways to measure interest in a vendor: Web searches, blog mentions, Google hits, and site traffic among them. All are publicly available with almost no effort. After a close analysis of the alternatives, I have decided the Alexa.com traffic statistics are the best indicator of vendor market presence. (You can read about the analysis in fascinating detail on my marketing measurement blog, MPM Toolkit.)
The table below shows the Alexa rankings and share statistics for the current Guide entries, the four marketing automation vendors already mentioned, and a dozen or so contenders.
The figures themselves need a little explaining. The Alexa rank is a “combined measure of page views and number of users”, with the most popular site ranked number 1, next-most-popular ranked number 2, etc. (In case you're wondering, the top three are Yahoo!, Google and YouTube.) Alexa share represents “percent of global Internet users who visit this site”. The rank and share figures correlate closely, but share is probably for comparing sites, since the ratio directly reflects relative traffic. That is, a share figure twice as large as another share figure indicates twice as many visitors, while a rank that is one half as large as another rank doesn’t necessarily mean twice as much traffic.
The figures for the existing vendors, in the first block of the table, give pretty much the ranking you’d expect. One wrinkle is that Vtrenz is owned by Silverpop, so Silverpop.com presumably siphons off a great deal of traffic from Vtrenz.com. On the other hand, Silverpop is a major email service provider in its own right, so a large share of the Silverpop.com traffic probably has nothing to do with Vtrenz. In any event, I’ve listed both sites in the table. Vtrenz is clearly a major vendor, so nothing is at stake here except bragging rights.
What’s more interesting is the figures for the Marketing Automation group. Unica is quite popular, while the other vendors are much less visited. This doesn’t particularly surprise me, although seeing Alterian, Aprimo and Neolane rank well below Manticore Technology and Market2Lead is odd. Perhaps these vendors are more obscure than I had realized. Still, they are much larger firms and do much more marketing than Manticore or Market2Lead. Interestingly, the other measure I found somewhat credible, IceRocket’s count of blog mentions, ranks Alterian, Aprimo and Neolane considerably higher than Manticore and Market2Lead. (See the MPM Toolkit post for details.) So the marketing automation vendors are probably a little more important to potential Guide buyers than the Alexa numbers suggest.
But my real concern was the Other Demand Generation group. Here, the Alexa figures do provide some very helpful insights. Basically they suggest that Marketbright, Pardot, Marqui and ActiveConversion, are all pretty much comparable in market presence to Manticore and Market2Lead. I spoke with Marketbright and Pardot this week and connected with ActiveConversion some time ago. Based on those conversations, this seems about right. (Marqui is a special case because they fell on financial hard times and the assets were recently purchased.) Rankings fall off sharply for the other vendors on the list, providing a reasonable cut-off point for the next round of Guide entries.
Of course, nothing is set in stone. Perhaps one of the smaller vendors can convince me that they have something special enough to justify including them. Plus there is still the question of whether I should invest the effort to expand the Guide at all, and what sequence I do the additions. But, whatever the final result, it’s nice to have an objective way to measure vendor market presence.
But I’ve also been approached by some of the other demand generation specialists. My original set of products was based on a general knowledge of which companies are most established, plus some consultation with vendors to learn who they felt were their main competitors. So far the original list of Eloqua, Vtrenz, Marketo, Manticore Technology and Market2Lead has proven a good set of choices. Yet there are so many more vendors I could add. How to choose?
The general rule is pretty obvious: pick the vendors that people are most interested in. We do, after all, want people to buy this thing. Of course, you want some wiggle room to add intriguing new products that they may not know about. Still, you mostly want to the report to include the vendors they are already asking about.
But although the general rule is obvious, which vendors are most popular is not. Fortunately, we have the Internet to help. It offers quite a few ways to measure interest in a vendor: Web searches, blog mentions, Google hits, and site traffic among them. All are publicly available with almost no effort. After a close analysis of the alternatives, I have decided the Alexa.com traffic statistics are the best indicator of vendor market presence. (You can read about the analysis in fascinating detail on my marketing measurement blog, MPM Toolkit.)
The table below shows the Alexa rankings and share statistics for the current Guide entries, the four marketing automation vendors already mentioned, and a dozen or so contenders.
Alexa | Alexa | |
rank | share | |
Already in Guide: | ||
| Eloqua | 20,234 | 0.007070 |
| Silverpop | 29,080 | 0.003050 |
| Marketo | 68,088 | 0.001700 |
| Manticore Technology | 213,546 | 0.000610 |
| Market2Lead | 235,244 | 0.000480 |
| Vtrenz | 295,636 | 0.000360 |
Marketing Automation: | ||
| Unica / Affinium* | 126,215 | 0.000850 |
| Alterian | 345,543 | 0.000250 |
| Aprimo | 416,446 | 0.000220 |
| Neolane | 566,977 | 0.000169 |
| Other Demand Generation: | ||
| Marketbright | 167,306 | 0.000540 |
| Pardot | 211,309 | 0.000360 |
| Marqui Software | 211,767 | 0.000440 |
| ActiveConversion | 257,058 | 0.000340 |
| Bulldog Solutions | 338,337 | 0.000320 |
| OfficeAutoPilot | 509,868 | 0.000200 |
| Lead Genesys | 557,199 | 0.000145 |
| LoopFuse | 734,098 | 0.000109 |
| eTrigue | 1,510,207 | 0.000043 |
| PredictiveResponse | 2,313,880 | 0.000033 |
| FirstWave Technologies | 2,872,765 | 0.000017 |
| NurtureMyLeads | 4,157,304 | 0.000014 |
| Customer Portfolios | 5,097,525 | 0.000009 |
| Conversen* | 6,062,462 | 0.000007 |
| FirstReef | 11,688,817 | 0.000001 |
The figures themselves need a little explaining. The Alexa rank is a “combined measure of page views and number of users”, with the most popular site ranked number 1, next-most-popular ranked number 2, etc. (In case you're wondering, the top three are Yahoo!, Google and YouTube.) Alexa share represents “percent of global Internet users who visit this site”. The rank and share figures correlate closely, but share is probably for comparing sites, since the ratio directly reflects relative traffic. That is, a share figure twice as large as another share figure indicates twice as many visitors, while a rank that is one half as large as another rank doesn’t necessarily mean twice as much traffic.
The figures for the existing vendors, in the first block of the table, give pretty much the ranking you’d expect. One wrinkle is that Vtrenz is owned by Silverpop, so Silverpop.com presumably siphons off a great deal of traffic from Vtrenz.com. On the other hand, Silverpop is a major email service provider in its own right, so a large share of the Silverpop.com traffic probably has nothing to do with Vtrenz. In any event, I’ve listed both sites in the table. Vtrenz is clearly a major vendor, so nothing is at stake here except bragging rights.
What’s more interesting is the figures for the Marketing Automation group. Unica is quite popular, while the other vendors are much less visited. This doesn’t particularly surprise me, although seeing Alterian, Aprimo and Neolane rank well below Manticore Technology and Market2Lead is odd. Perhaps these vendors are more obscure than I had realized. Still, they are much larger firms and do much more marketing than Manticore or Market2Lead. Interestingly, the other measure I found somewhat credible, IceRocket’s count of blog mentions, ranks Alterian, Aprimo and Neolane considerably higher than Manticore and Market2Lead. (See the MPM Toolkit post for details.) So the marketing automation vendors are probably a little more important to potential Guide buyers than the Alexa numbers suggest.
But my real concern was the Other Demand Generation group. Here, the Alexa figures do provide some very helpful insights. Basically they suggest that Marketbright, Pardot, Marqui and ActiveConversion, are all pretty much comparable in market presence to Manticore and Market2Lead. I spoke with Marketbright and Pardot this week and connected with ActiveConversion some time ago. Based on those conversations, this seems about right. (Marqui is a special case because they fell on financial hard times and the assets were recently purchased.) Rankings fall off sharply for the other vendors on the list, providing a reasonable cut-off point for the next round of Guide entries.
Of course, nothing is set in stone. Perhaps one of the smaller vendors can convince me that they have something special enough to justify including them. Plus there is still the question of whether I should invest the effort to expand the Guide at all, and what sequence I do the additions. But, whatever the final result, it’s nice to have an objective way to measure vendor market presence.
Thursday, November 13, 2008
Usability Is Just One Piece of the Puzzle
A funny thing happened as I was writing one of my usual rants on incorporating usability into the selection process. (The resulting paper is on the http://www.raabguide.com/ site, creatively titled "Building Usability into Your System Selection".)
After a few bon mots that probably no one else will find clever ("Usability is hard to measure; features are easy to count" "Small hard facts beat big blurry realities") I got to describing the steps in a usability-aware selection process:
After a few bon mots that probably no one else will find clever ("Usability is hard to measure; features are easy to count" "Small hard facts beat big blurry realities") I got to describing the steps in a usability-aware selection process:
- define business needs
- define processes to meet those needs
- define tasks within each process
- identify systems to consider, then, for each system:
- determine which users will do each task
- determine how much work each task will be
- compare, rank and summarize the results
As a point of comparison, it's steps 3, 5 and 6 that differ from the conventional selection process. Step 3 in a conventional process would identify features needed rather than tasks, while steps 5 and 6 would be replaced with research into system features.
What I realized as I was writing this was that the real focus is not on usability, but on defining processes and tasks. Usability measures are something of a by-product. In fact, the most natural way to implement this approach would be to score each system for each task, with a single score that incoporates both functionality and as ease of use. Indeed, as I wrote not long ago, standard definitions of usability include both these elements, so this is not exactly an original thought.
Still, it does mean I have to restructure the terms of the debate (at least, the one inside my head). It's not usability vs. features, but process vs. features. That is, I'm essentially arguing that selection processes should invest their effort in understanding the company business processes that the new system must support, and in particular in which the tasks different users will perform.
The good news here is that you'll eventually need to define, or maybe redefine, those processes, tasks and user roles for a successful implementation. So you're not doing more work, but simply doing the implementation work sooner. This means a process-focused evaluation approach ultimately reduces the total work involved, as well as reducing implementation time and improving the prospects for success. By contrast, time spent researching system features is pretty much a waste once the selection process is complete.
Of course, this does raise the question of whether the feature information assembled in the Raab Guide to Demand Generation Systems is really helpful. You won't be surprised to find I think it is. This is not so much because of the feature checklist (truly my least favorite section) but because the Guide tries to show how the features are organized, which directly impacts system usability. Plus, of course, the absence of a necessary feature makes a system unusable for that particular purpose, and that is the biggest usability hit of all. What the Guide really does is save readers the work of assembling all the feature information for themselves, thereby freeing them to focus on defining their own business processes, tasks and users.
In conclusion, you should all go and buy the Guide immediately.
Thursday, November 06, 2008
LucidEra and Birst Blaze New Trails for On-Demand BI
I spent a few minutes last week on the Web sites of about eight or nine on-demand business intelligence vendors, and within a few days received emails from two of them ostensibly asking about much earlier visits where I must have registered with my email address. Given my current obsession with demand generation systems, I’m pretty sure this was no coincidence: they had deposited a cookie linked to my email address during the earlier visit, and used this address to react when I returned. I suppose I should admire this as good marketing, although the disingenuousness of the messages was a bit disturbing. I suppose they felt that was better than the creepy feeling I might get if they said they knew I had visited. [Postscript: I later spoke with one of the salespeople, who sadly assured me it was a total coincidence. She only wished her firm could react so effectively.]
(I was about to coin the word “disingenuity” to mean something that is ingeniously disingenuous [i.e., cleverly deceptive], but see that the dictionary already lists it as a synonym for disingenuousness. Pity. )
Whatever. The reason I was looking at the on-demand BI sites was I’d spoken recently with two vendors in the field and wanted to get some context. One of the two was LucidEra , which was giving me an update since my post about them in July.
They’re doing quite well, thanks, and most excited about a new services offering they call a “pipeline healthcheck”. This is a standardized analysis of a company’s sales pipeline to find actionable insights. LucidEra says it has been tremendously successful in demonstrating the value of their system and thus closing sales. Apparently, many marketers never learned how to analyze the information buried within their sales automation systems, simply because it wasn’t available back when they were being trained. So doing it for them, and helping them learn to do it for themselves, adds great value.
This reinforced one of my few really profound insights into the software business, which is that the marketing software vendors who succeed have been the ones who provide extensive services to help their clients gain value from their systems. (Well, I think it's profound.) Interestingly, when I told LucidEra I have recently been applying this insight to demand generation vendors, they said they had recently switched to a new demand generation vendor and—this is the interesting part—found the new system was so much simpler to use that very little vendor support was necessary. That’s an interesting tidbit, although it doesn’t necessary confirm my service-is-essential thesis. Perhaps it needs a corollary of some sort when the applications are obvious or the users are already trained. Facts can be so pesky.
The other vendor on my mind was Birst. I actually spoke to them back in early September, but their product announcement was under embargo until September 30 and in any case I’ve been focused since then on the demand generation guide (have I mentioned http://www.raabguide.com/ yet today?) I’m glad to get back to Birst, though, because I was quite intrigued by what they showed me. Basically they claim to have fully automated the entire business intelligence implementation process: loading the data, designing the warehouse, identifying interesting information, and creating dashboards to display the results.
I’ll admit to being skeptical of how well they can do this, but the company’s managers have some excellent credentials and Birst itself is a project of a Success Metrics, which has been providing Web-based opportunity discovery to insurance and pharmaceutical sales forces since 2006. They offered me an online workspace to play with the tool, but I haven’t had time to take them up on it. (I think their Web site makes that same offer to anyone.)
I did spend a few minutes playing with a prebuilt demo on the Web site: it’s a reasonable user interface for ad hoc analysis and building dashboard reports. There was a lag of up to five seconds between each click when I was working with the data, which would quickly get annoying if I were trying to do real work. Part of the lag may be caused by the underlying technology, which generates relational OLAP cubes on the fly in response to user queries. But it also appears the system uses a traditional Web interface, which redraws the screen after each click, rather than AJAX and similar technologies which provide a smoother, faster user experience.
I don’t want to dwell on the Birst user interface, partly because I haven’t tested it thoroughly and partly because you can judge it for yourself, but mostly because their more important claim is the automated implementation. As I said last March, I think the labor involved with building the system is the biggest obstacle to on-demand BI, so Birst’s claim to have solved this is the real news.
It would take some serious testing to assess how good a job Birst’s automated systems can really do. Still, the system can be useful even if it’s not perfect and it will presumably improve over time. So if you’re thinking about on-demand business intelligence, either for a specific purpose or just to better understand what’s possible, Birst is certainly worth a look.
Incidentally, my quick scan of other on-demand business intelligence vendors (Autometrics, BlinkLogic, Good Data, oco, OnDemandIQ, and PivotLink) showed that only oco made a similar claim about having automated the implementation process.
On the other hand, Good Data, PivotLink LucidEra and possibly oco are using in-memory or columnar databases (PivotLink’s is in-memory and columnar: they win). In theory these should give quicker response than Birst’s on-the-fly OLAP cubes, although actual performance depends on the implementation details. (Speaking of experience, Birst’s database technology has been running at Success Metrics for several years, and has scaled to the terabyte range. I don’t know what scales the other vendors have reached.) It also seems to me that in-memory and columnar databases should be particularly compatible with automated implementation because their simpler structures and greater efficiency make them more forgiving than conventional databases if the automated design is less than optimal. But no one in this particular group of vendors seems to have put the two together.
I don’t know when I’ll have time to give all these other vendors the attention they deserve. But based on what I’ve heard from LucidEra and Birst, and seen on the other vendors’ Web sites, I’m more optimistic about the potential of on-demand business intelligence than I was back in March.
(I was about to coin the word “disingenuity” to mean something that is ingeniously disingenuous [i.e., cleverly deceptive], but see that the dictionary already lists it as a synonym for disingenuousness. Pity. )
Whatever. The reason I was looking at the on-demand BI sites was I’d spoken recently with two vendors in the field and wanted to get some context. One of the two was LucidEra , which was giving me an update since my post about them in July.
They’re doing quite well, thanks, and most excited about a new services offering they call a “pipeline healthcheck”. This is a standardized analysis of a company’s sales pipeline to find actionable insights. LucidEra says it has been tremendously successful in demonstrating the value of their system and thus closing sales. Apparently, many marketers never learned how to analyze the information buried within their sales automation systems, simply because it wasn’t available back when they were being trained. So doing it for them, and helping them learn to do it for themselves, adds great value.
This reinforced one of my few really profound insights into the software business, which is that the marketing software vendors who succeed have been the ones who provide extensive services to help their clients gain value from their systems. (Well, I think it's profound.) Interestingly, when I told LucidEra I have recently been applying this insight to demand generation vendors, they said they had recently switched to a new demand generation vendor and—this is the interesting part—found the new system was so much simpler to use that very little vendor support was necessary. That’s an interesting tidbit, although it doesn’t necessary confirm my service-is-essential thesis. Perhaps it needs a corollary of some sort when the applications are obvious or the users are already trained. Facts can be so pesky.
The other vendor on my mind was Birst. I actually spoke to them back in early September, but their product announcement was under embargo until September 30 and in any case I’ve been focused since then on the demand generation guide (have I mentioned http://www.raabguide.com/ yet today?) I’m glad to get back to Birst, though, because I was quite intrigued by what they showed me. Basically they claim to have fully automated the entire business intelligence implementation process: loading the data, designing the warehouse, identifying interesting information, and creating dashboards to display the results.
I’ll admit to being skeptical of how well they can do this, but the company’s managers have some excellent credentials and Birst itself is a project of a Success Metrics, which has been providing Web-based opportunity discovery to insurance and pharmaceutical sales forces since 2006. They offered me an online workspace to play with the tool, but I haven’t had time to take them up on it. (I think their Web site makes that same offer to anyone.)
I did spend a few minutes playing with a prebuilt demo on the Web site: it’s a reasonable user interface for ad hoc analysis and building dashboard reports. There was a lag of up to five seconds between each click when I was working with the data, which would quickly get annoying if I were trying to do real work. Part of the lag may be caused by the underlying technology, which generates relational OLAP cubes on the fly in response to user queries. But it also appears the system uses a traditional Web interface, which redraws the screen after each click, rather than AJAX and similar technologies which provide a smoother, faster user experience.
I don’t want to dwell on the Birst user interface, partly because I haven’t tested it thoroughly and partly because you can judge it for yourself, but mostly because their more important claim is the automated implementation. As I said last March, I think the labor involved with building the system is the biggest obstacle to on-demand BI, so Birst’s claim to have solved this is the real news.
It would take some serious testing to assess how good a job Birst’s automated systems can really do. Still, the system can be useful even if it’s not perfect and it will presumably improve over time. So if you’re thinking about on-demand business intelligence, either for a specific purpose or just to better understand what’s possible, Birst is certainly worth a look.
Incidentally, my quick scan of other on-demand business intelligence vendors (Autometrics, BlinkLogic, Good Data, oco, OnDemandIQ, and PivotLink) showed that only oco made a similar claim about having automated the implementation process.
On the other hand, Good Data, PivotLink LucidEra and possibly oco are using in-memory or columnar databases (PivotLink’s is in-memory and columnar: they win). In theory these should give quicker response than Birst’s on-the-fly OLAP cubes, although actual performance depends on the implementation details. (Speaking of experience, Birst’s database technology has been running at Success Metrics for several years, and has scaled to the terabyte range. I don’t know what scales the other vendors have reached.) It also seems to me that in-memory and columnar databases should be particularly compatible with automated implementation because their simpler structures and greater efficiency make them more forgiving than conventional databases if the automated design is less than optimal. But no one in this particular group of vendors seems to have put the two together.
I don’t know when I’ll have time to give all these other vendors the attention they deserve. But based on what I’ve heard from LucidEra and Birst, and seen on the other vendors’ Web sites, I’m more optimistic about the potential of on-demand business intelligence than I was back in March.
Subscribe to:
Posts (Atom)
