Is Taguchi Good for Multivariate Testing?

I’ve spent a lot of time recently talking to vendors of Web site testing systems. One topic that keeps coming up is whether Taguchi testing—which tests selected combinations of variables and infers the results for untested combinations—is a useful technique for this application. Some vendors use it heavily; some make it available but don’t recommend it; others reject it altogether.

Vendors in the non-Taguchi camp tell me they’ve done tests comparing Taguchi and “full factorial” tests (which test all possible combinations), and gotten different results. Since the main claim of Taguchi is that it finds the optimum combination, this is powerful practical evidence against it. On the theoretical level, the criticism is that Taguchi assumes that there are no interactions among test variables, meaning results for each variable are not affected by the values of other variables, when such interactions are in fact common. Moreover, how would you know whether interactions existed if you didn’t test for them? (Taguchi tests are generally too small to find interactions.)

Taguchi proponents might argue that careful test design can avoid interactions. But the more common justification seems to be that Taguchi makes it possible to test many more alternatives than conventional A/B tests (which change just one item at a time) or full-factorial designs (which need a lot of traffic to get adequate volume for each combination.)

So, the real question is not whether Taguchi ignores interactions (it does), but whether Taguchi leads to better results more quickly. This is possible even if those results not optimal, because Taguchi lets users test a wider variety of options with a given amount of traffic. I’m guessing Taguchi does help, at least for sites without huge visitor volumes.

Incidentally, I tried to do a quick classification of which vendors favor Taguchi. But it’s not so simple, because even vendors who prefer other methods still offer Taguchi as an option. And some alternative methods can be seen more as refinements of Taguchi than total rejections of it. So I think I’ll avoid naming names just now, and let the vendors speak for themselves. (Vendors to check: Offermatica, Optimost, Memetrics, SiteSpect, Vertster.)