How Do You Measure the Influence of Marketing Messages?

My review of Coremetrics Lifestyle raised the issue of measuring the impact of marketing materials on customer behavior. Of course, this is just one piece of the marketing attribution puzzle. But it’s worth a separate discussion because it’s such a common question – and, unlike so many measurement problems, this one actually has an answer.

Let’s start with the original impetus. This was an “influence” report that showed the percentage of people reaching a marketing stage who had received specific marketing treatments (or had other attributes such as source, product history, demographic, etc.). The idea was that treatments received by a higher percentage of customers were more influential. In other words, if 100% of new buyers saw a white paper offer and just 50% saw a Webinar invitation, then the white paper has more influence than the Webinar.

Plausible, yes. But wrong.

Let’s think through the example. What if the white paper is offered to everyone? Yes, 100% of new buyers saw it, but so did 100% of non-buyers. We know exactly nothing about whether it made its recipients more or less likely to purchase.

Now, let’s say just 10% of prospects see the Webinar invitation, compared with 50% of buyers. Can we say it has a positive influence? Still no: maybe the Webinar attracts hot prospects who would have purchased anyway. It’s even possible that the Webinar offer annoys people and actually reduces purchase rates. You can’t tell from these figures.

In other words, it’s not enough to know what was seen by customers who became buyers (or, more generally, by people who took any particular action). You also need to know what was seen by non-buyers and, ideally, to compare results for groups that are similar except for that particular treatment.

So, what measures do make sense for assessing influence?

- the simplest measure compares the result rate of treated customers with results for non-treated customers. You might find that 20% of people who receive a white paper became buyers, compared with 10% of people who don’t receive the white paper. These two figures can be combined in a single ratio: 20% of treated / 10% of non-treated = 2.0. The higher the ratio, the more it seems that receiving the white paper increased the likelihood that someone would purchase. But it’s no more than a suggestion: maybe the white paper was sent to people who were stronger prospects to begin with.

- a more advanced measure adjusts for the audience by attempting to limit the non-treated group (e.g., non-buyers) to customers similar to the target group. This could be done by building a statistical model that uses all other attributes to predict behavior. Or, you could apply lead scores or funnel stage definitions. Whatever the technique, the result is to divide the audience into groups that are expected to behave similarly. The calculation would then compare results of treated vs. non-treated customers in each same group. So, a report might find that 40% of “stage 3 leads” (whatever they are) made a purchase after attending a Webinar, while just 15% of “stage 3 leads” made a purchase if they didn't attend a Webinar. Again, the treated and non-treated figures could be combined in a ratio (40% / 15% = 2.7)

- of course, the only true measure is a structured test. This ensures that the only difference between the treated and non-treated groups is the treatment itself. Without such tests, there's a good chance that the customers selected for treatment would have performed differently in any event.

A proper reporting system would present the ratios along with actual result rates, trends over time, the number of customers receiving each treatment, and comparisons with ratios for other treatments. These figures help marketers focus their energies on the most valuable opportunities. Still, the starting point is always a comparison of treated vs. non-treated performance: without that, the numbers could mean anything.