Marcom Valuation: An Alternative to A/B Testing

So we always wanna know how marcom (marketing communications) is performing, both as a vehicle and for an individual campaign. In evaluating marcom it is common to employ simple A/B testing. This is a technique wherein random sampling populates two cells for campaign treatment.

One cell gets the test and the other cell will not. Then response rate or net revenue is compared between the two cells. If the test cell outperforms the control cell (within testing parameters of lift, confidence, etc.) the campaign is deemed significant and positive.

Why Do Something Else?

However, this procedure lacks insight generation. It optimizes nothing, is performed in a vacuum, gives no implications for strategy and there are no controls for other stimuli.

Secondly, all too often, the test is polluted in that at least one of the cells has accidently received other offers, brand messages, communications, etc. How many times has the test results been deemed inconclusive, even non-sensical? So they test again and again. They learn nothing, except that testing does not work.

That’s why I recommend using ordinary regression to control for all other stimuli. Regression modeling also gives insights into marcom valuation which can generate an ROI. This is not done in a vacuum, but provides options as a portfolio to optimize budget.

An Example

Let’s say we were testing two emails, test vs. control and the results came back non-sensical. Then we found out that our brand department accidentally sent a direct mail piece to (mostly) the control group. This piece was not planned (by us) nor accounted for in randomly choosing the test cells. That is, the business-as-usual group got the usual direct mail but the test group–which was held out–did not. This is very typical in a corporation, wherein one group does not work nor communicate with another business unit.

So instead of testing wherein each row is a customer, we roll up the data by time period, say weekly. We add up, by week, the number of test emails, control emails and direct mails sent out. We also include binary variables to account for season, in this case quarterly. TABLE 1 shows a partial list of the aggregates with the email test starting in week 10. Now we do a model:

The ordinary regression model as formulated above produces TABLE 2 output. Include any other independent variables of interest. Of particular notice should be that (net) price is excluded as an independent variable. This is because net revenue is the dependent variable and is calculated as (net) price * quantity.

TABLE 1

week	em_test	em_cntrl	dir_mail				net_rev
9	0	0	55	1	0	0	$1,950
10	22	35	125	1	0	0	$2,545
11	23	44	155	1	0	0	$2,100
12	30	21	75	1	0	0	$2,675
13	35	23	80	1	0	0	$2,000
14	41	37	125	0	1	0	$2,900
15	22	54	200	0	1	0	$3,500
16	0	0	115	0	1	0	$4,500
17	0	0	25	0	1	0	$2,875
18	0	0	35	0	1	0	$6,500

To include price as an independent variable means having price on both sides of the equation, which is inappropriate. (My book, Marketing Analytics: A Practical Guide to Real Marketing Science, provides extensive examples and analysis of this analytic problem.) The adjusted R2 for this model is 64%. (I dropped q4 to avoid the dummy trap.) emc = control email and emt = test email. All of the variables are significant at the 95% level.

TABLE 2

				dm	emc	emt	const
coeff	-949	-1,402	-2,294	12	44	77	5,039
st err	474.1	487.2	828.1	2.5	22.4	30.8
t-ratio	-2	-2.88	-2.77	4.85	1.97	2.49

In terms of the email test, the test email outperformed the control email by 77 vs 44 and was much more significant. Thus, accounting for other things, the test email worked. These insights come even when the data is polluted. An A/B test would not have produced this.

TABLE 3 takes the coefficients to calculate marcomm valuation, a contribution of each vehicle in terms of net revenue. That is, to calculate the value of direct mail, the coefficient of 12 is multiplied by the mean number of direct mails sent of 109 to get $1,305. Customers spend an average amount of $4,057. Thus $1,305 / $4,057 = 26.8%. That means direct mail contributed nearly 27% of the total net revenue. In terms of ROI, 109 direct mails generate $1,305. If a catalog costs $45 then ROI = ($1,305 – $55) / $55 = 2300%!

Because price was not independent variable, it is usually concluded that price’s impact is buried in the constant. In this case the constant of 5039 includes price, any other missing variables and a random error, or about 83% of net revenue.

TABLE 3

				dm	emc	emt	const
Coeff	-949	-1,402	-2,294	12	44	77	5,039
mean	0.37	0.37	0.11	109.23	6.11	4.94	1
$4,875	-$352	-$521	-$262	$1,305	$269	$379	$4,057
value	-7.20%	-10.70%	-5.40%	26.80%	5.50%	7.80%	83.20%

Conclusion

Ordinary regression offered an alternative to provide insights in the face of dirty data, as is often the case in a corporate testing scheme. Regression also provides a contribution to net revenue as well as a business case for ROI. Ordinary regression is an alternative technique in terms of marcomm valuation.