Analytics & Testing, , Marketing Books

Marcom Valuation: An Alternative to A/B Testing

So we always wanna know how marcom (marketing communications) is performing, both as a vehicle and for an individual campaign. In evaluating marcom it is common to employ simple A/B testing. This is a technique wherein random sampling populates two cells for campaign treatment.

One cell gets the test and the other cell will not. Then response rate or net revenue is compared between the two cells. If the test cell outperforms the control cell (within testing parameters of lift, confidence, etc.) the campaign is deemed significant and positive.

Why Do Something Else?

However, this procedure lacks insight generation. It optimizes nothing, is performed in a vacuum, gives no implications for strategy and there are no controls for other stimuli.

Secondly, all too often, the test is polluted in that at least one of the cells has accidently received other offers, brand messages, communications, etc. How many times has the test results been deemed inconclusive, even non-sensical? So they test again and again. They learn nothing, except that testing does not work.

That’s why I recommend using ordinary regression to control for all other stimuli. Regression modeling also gives insights into marcom valuation which can generate an ROI. This is not done in a vacuum, but provides options as a portfolio to optimize budget.

An Example

Let’s say we were testing two emails, test vs. control and the results came back non-sensical. Then we found out that our brand department accidentally sent a direct mail piece to (mostly) the control group. This piece was not planned (by us) nor accounted for in randomly choosing the test cells. That is, the business-as-usual group got the usual direct mail but the test group–which was held out–did not. This is very typical in a corporation, wherein one group does not work nor communicate with another business unit.

So instead of testing wherein each row is a customer, we roll up the data by time period, say weekly. We add up, by week, the number of test emails, control emails and direct mails sent out. We also include binary variables to account for season, in this case quarterly. TABLE 1 shows a partial list of the aggregates with the email test starting in week 10. Now we do a model:

net\_rev = f(em\_test,em\_cntrl,dir\_mail,q_1,q_2,q_3,etc)

The ordinary regression model as formulated above produces TABLE 2 output. Include any other independent variables of interest. Of particular notice should be that (net) price is excluded as an independent variable. This is because net revenue is the dependent variable and is calculated as (net) price * quantity.

TABLE 1

weekem_testem_cntrldir_mailq_1q_2q_3net_rev
90055100$1,950
102235125100$2,545
112344155100$2,100
12302175100$2,675
13352380100$2,000
144137125010$2,900
152254200010$3,500
1600115010$4,500
170025010$2,875
180035010$6,500

To include price as an independent variable means having price on both sides of the equation, which is inappropriate. (My book, Marketing Analytics: A Practical Guide to Real Marketing Science, provides extensive examples and analysis of this analytic problem.) The adjusted R2 for this model is 64%. (I dropped q4 to avoid the dummy trap.) emc = control email and emt = test email. All of the variables are significant at the 95% level.

TABLE 2

q_3q_2q_1dmemcemtconst
coeff-949-1,402-2,2941244775,039
st err474.1487.2828.12.522.430.8
t-ratio-2-2.88-2.774.851.972.49

In terms of the email test, the test email outperformed the control email by 77 vs 44 and was much more significant. Thus, accounting for other things, the test email worked. These insights come even when the data is polluted. An A/B test would not have produced this.

TABLE 3 takes the coefficients to calculate marcomm valuation, a contribution of each vehicle in terms of net revenue. That is, to calculate the value of direct mail, the coefficient of 12 is multiplied by the mean number of direct mails sent of 109 to get $1,305. Customers spend an average amount of $4,057. Thus $1,305 / $4,057 = 26.8%. That means direct mail contributed nearly 27% of the total net revenue. In terms of ROI, 109 direct mails generate $1,305. If a catalog costs $45 then ROI = ($1,305 – $55) / $55 = 2300%!

Because price was not independent variable, it is usually concluded that price’s impact is buried in the constant. In this case the constant of 5039 includes price, any other missing variables and a random error, or about 83% of net revenue.

TABLE 3

q_3q_2q_1dmemcemtconst
Coeff-949-1,402-2,2941244775,039
mean0.370.370.11109.236.114.941
$4,875-$352-$521-$262$1,305$269$379$4,057
value-7.20%-10.70%-5.40%26.80%5.50%7.80%83.20%

Conclusion

Ordinary regression offered an alternative to provide insights in the face of dirty data, as is often the case in a corporate testing scheme. Regression also provides a contribution to net revenue as well as a business case for ROI. Ordinary regression is an alternative technique in terms of marcomm valuation.

2 Comments

  1. 1

    Nice alternative to a practical issue, Mike.
    In the way you have done, I guess there is no overlap of target communicators in the immediate prior weeks. Otherwise would you have an auto-regressive and / or time-lagged component?

  2. 2

Leave a Reply