Operational Systems Should Be Designed with Testing in Mind

A direct marketing client pointed out to me recently that it can be very difficult to set up tests in operational systems.

There is no small irony in this. Direct marketers have always prided themselves on their ability to test what they do, in pointed contrast to the barely measurable results of conventional media. But his point was well taken. Although it’s fairly easy to set up tests for direct marketing promotions, testing customer treatments delivered through operational systems such as order processing is much more difficult. Those systems are designed with the assumption that all customers are treated the same. Forcing them to treat selected customers differently can require significant contortions.

This naturally led me to wonder what an operational system would look like if it had been designed from the start with testing in mind. A bit more reflection led me to the multivariate testing systems I have been looking at recently—Optimost, Offermatica, Memetrics, SiteSpect, Vertster. These take control of all customer treatments (within a limited domain), and therefore make delivering a test message no harder or easier than delivering a default message. If we treated them as a template for generic customer treatment systems, which functions would we copy?

I see several:

- segmentation capabilities, which can select customers for particular tests (or for customized treatment in general). You might generalize this further to include business rules and statistical models that determine exactly which treatments are applied to which customers: when you think about it, segmentation is just a special type of business rule.

- customer profiles, which make available all the information needed for segmentation/rules and hold tags that identify customers already tagged for a particular test

- content management features, which make existing content available to apply in tests. Some systems provide content creation as well, although I think this is a separate specialty that should usually remain external.

- test design functions, that help users create correctly-structured tests and then link them to the segmentation rules, data profiles and content needed to execute them. These design functions also include parameters such as date ranges, preview features for checking that they are set up correctly, workflow for approvals, and similar administrative features.

- reporting and analysis, so users can easily read test results, understand their implications, and use the knowledge effectively.

I don’t mean to suggest that existing multivariate testing systems should replace operational systems. The testing systems are mostly limited to Web sites and control only a few portions of a few pages within those sites. They sit on top of general purpose Web platforms which handle many other site capabilities. Using the testing systems to control all pages on a site without an underlying platform would be difficult if not impossible, and in any case isn’t what the testing systems are designed for.

Rather, my point is that developers of operational systems should use the testing products as models for how to finely control each customer experience, whether for testing or simply to tailor treatments to individual customer needs. Starting their design from the perspective of creating a powerful testing environment should enable them to understand more clearly what is needed to build a comprehensive solution, rather than trying to bolt on particular capabilities without grasping the underlying connections.