Tuesday, February 27, 2007

Are Multi-Variate Testing Systems Under-Priced?

I’ve had a series of conversations over the past few days regarding the distinction I made last November between “behavioral targeting” systems and “multivariate testing” systems. Both types of products tailor Web contents to individual visitors. Both work similarly: place code snippets in slots on the Web page where the personalized content will appear; when the page is loaded, send visitor information to a hosted server; run server-side rules to select the content; then return the selection to the Web page for display. The difference is in how they select the contents.

The testing systems I’ve looked at closely (Optimost, Offermatica, Memetrics) rely on users to define customer segments and assign the contents shown to each segment. They usually assign multiple content items to test their performance. But the systems can also send targeted contents in non-test situations. It’s just a matter of specifying the default contents to serve each segment.

By contrast, the behavioral systems (Certona, Touch Clarity [recently purchased by Omniture], [x+1]) automatically build their own segments. Specifically, they create groups of visitors which are likely to respond to different contents. Thus both the segment definitions and segment-to-content match-ups evolve over time as the system gains more experience and, perhaps, as user behavior evolves.

What was interesting, and frustrating, about my recent conversations was that my counterpart kept insisting that the testing systems do not allow segment-based targeting. He even showed me a recent analyst report that said this. Having personally researched Memetrics and Offermatica and taken a close look at the Optimost Web site, I know this is wrong. Of course, I’ve been around long enough to know you can’t trust anything but your own eyes where software is concerned (and sometimes not even those!) So the misinformation—which I’m certain was unintentional—was no surprise.

More intriguing was the realization that sophisticated testing systems could probably charge more if they positioned themselves as targeting tools. Apparently the prices for behavioral targeting products are higher—perhaps because at least some of them base their fees on incremental profits earned for their clients (I know Certona does this). Maybe the testing systems are missing other functions needed for targeting. But if they really could raise their fees by repositioning themselves, it looks like a missed opportunity.


E Hansen said...


while I can't speak for all testing vendors, our own customers use SiteSpect both for multivariate testing *and* targeting. I think the difference is in how much you expect to predefine your segments (much as a direct marketer does) vs. letting the computer divine those segments for you. Interestingly, the two approaches are not mutually exclusive - we have customers who use both SiteSpect for targeted testing as well as 3rd party solutions for automated behavioral targeting.

I think your observations motivate a broader discussion about micro vs. macro segmentation, and how to strike the right balance. In the extreme of micro-segmentation, if you're classifying every user as their own segment, can you really achieve statistically significant lifts in site performance? Conversely, if you segment your users into too few predefined buckets (e.g. new vs. returning), are you missing opportunity for finer-grained targeting and optimization?

While the answer will vary based on the makeup of each site's user base, my sense is that web marketers should infuse as much human knowledge into the segmentation process as possible. Refer to your historical data (web analytics) for insight, and use persona design to make sure you actually have the right content available for each stage of the purchase cycle. Then work with your vendors (whether multivariate testing like SiteSpect and/or others you've mentioned) to target users with the right content at the right time, making sure you have sufficient samples to yield statistical significance on an ongoing basis.

Eric Hansen
SiteSpect, Inc.

p.s. thanks for the heads-up on pricing ... we're raising our rates at the end of the week ;)

David Raab said...

Thanks for the comment Eric. Sorry to have left SiteSpect out of my list--no negative implications intended, it's just that I haven't looked at the product closely enough to comment with confidence on what it does and doesn't do. And I totally agree that there is a place for both marketer-defined and system-discovered segmentations.