It’s been fun to explore multi-variate testing in some depth over these last few entries, but I fear you Dear Readers will get bored if I continue to focus on a single topic. Also, my stack of white papers to review is getting taller and I do so enjoy critiquing them. (At the top of the pile is Optimost’s “15 Critical Questions to Ask a Multivariable Testing Provider” available here. This covers many of the items I listed on Monday although of course it puts an Optimost-centric spin on them. Yes I read it before compiling my own list; let’s just call that “research”.)
Before I leave the topic altogether, let me share some final tidbits. One is that I’m told it’s not possible to force a particular combination of elements into a Taguchi test plan because the combinations are determined by the Taguchi design process itself. I’m not 100% sure this is correct: I suspect that if you specified a particular combination as a starting point, you could design a valid plan around it. But the deeper point, which certainly does make sense, is that substituting a stronger for weaker combination within an active test plan would almost surely invalidate the results. The more sensible method is to keep tests short, and either complete the current test unchanged or replace it with a new one if an important result becomes clear early on. Remember that this applies to multi-variate tests, where results from all test combinations are aggregated to read the final results. In a simpler A/B test, you have more flexibility.
The second point, which is more of a Note To Self, is that multi-variate testing still won't let me use historical data to measure the impact of different experiences on future customer behavior. I want to do this for value formulas in the Customer Experience Matrix. The issue is simple: multi-variate tests require valid test data. This means the test system must determine which contents are presented, and everything else about the test customers must be pretty much the same. Historical data won’t meet these conditions: either everyone saw the same content, or the content was selected by some existing business rule that introduces its own bias. The same problem really applies to any analytical technique, even things like regression that don’t require data generated from structured tests. When a formal test is possible, multi-variate testing can definitely help to measure experience impacts. But, as one of the vendors pointed out to me yesterday, it’s difficult for companies to run tests that last long enough to measure long-term outcomes like lifetime value.
Hi again, Mark Ogne from Kefta - standard disclosure:-)
ReplyDeleteIn your final paragraph, you touch upon a point that's near and dear to me... visitors to a site are not a static, they are a dynamic... from both the dimensions of quantity as well as the makeup of their needs and expectations. The latter being my point in this note.
In your final paragraph you describe the struggle to use MVT as a predictive tool, identifying LTV as a hope. One way that I look at this dilemma is to understand what assertions one has to prove true for their site before a typical MVT can make sense:
1) All page visitors have the same needs and expectations, they are a homogeneous group. Because of this homogeneity, aggregation of results and projections of needs (reflected in best performing creatives, offers, etc.) can be accomplished. If this assertion is not correct, a MVT will simply tell you which page assets are best for the larger of the differing groups of site visitors… at the expense of the needs of smaller groups.
2) The composition of differing needs does not change over time. This builds upon the first point. If you believe the needs to be only tolerably different and you decide to continue with a MVT, this assumption must hold true in the long term. If in the future, you acquire new sources of traffic or any facet of the competition or your offering changes, the results of the test may be hurting rather than helping. Here’s an example… a consumer electronics ecommerce site typically gets their traffic from audiophile sites via banners and links. Arguably, the needs of the traffic are high-end or quality oriented in nature and one could argue relatively homogeneous. A MVT is run and the “best” page is chosen. Later, the marketing team finds a large source of traffic from comparison shopping sites… arguably price sensitive in nature. Would the same page be acceptable to this new traffic? What if the new traffic is now of larger quantity than the original quality oriented traffic… what would happen to overall performance?
Is it possible that a site has very homogeneous traffic, the needs and expectations of all site visitors are the same? Sure, I suppose so. Is it possible that all of the people who have used MVT or are discussing to use MVT as a primary tool have this homogeneity? Certainly no. MVT only makes sense if it’s used on a targeted group of site visitors.
Hi Mark,
ReplyDeleteSo near as I can tell, most if not all of the MVT tools I've been looking at do support separate tests per segment. From this, I'd infer that the vendors largely agree with you that segmentation is important.
Perhaps a more interesting question is segment- vs. individual level customization. Although the deployment mechanisms of at least some MVT systems do allow business rules to help select contents, I don't think that MVT itself can be used to identify the best treatment for each individual. But I haven't researched that particular question so I could be wrong. Based on my current understanding, I see individual customization as more in the domain of behavioral targeting systems.
Hi David,
ReplyDeleteI just found your blog and saw your excellent posts about MVT. Our company wants to sign-up with a MVT vendor, but we're not sure which one. Our company has a big emphasis on phone sales, so we'd like a MVT solution that can be integrated with our salespeople(Example: If we have dozens of different offers on our website (dozens of different recipes), and someone calls in to our salesperson, how can the salesperson know what offer that customer is seeing?)
Do you have any knowledge about this?
Thanks!
Hi Walt,
ReplyDeleteI think you might have preferred a private reply, but I've never figured out how to get at commenters' email addresses. (If you or anybody else knows, I'd be thrilled if you tell me.) As to your question: most if not all of the MVT systems establish profiles for visitors, so they do track which offers have been shown to each individual. I don't know off hand which systems expose that information but imagine at least some of them do. Sorry I can't be more specific, but the good news is you're asking a reasonable question that should be answerable in the affirmative. Hope this helps. (If you want more private comments, please email me at draab@clientxclient.com.)