Analytics & Testing Email Marketing & Automation Marketing Books

MarCom Valuation: An Alternative to A/B Testing

Michael GrigsbyJul 22, 2020

3 minutes read

MarCom Valuation: An Alternative to A/B Testing

So, we always want to know how marketing communications (MarCom) are performing, both as a vehicle and for individual campaigns. In evaluating MarCom, it is common to employ simple A/B testing. This is a technique wherein random sampling populates two cells for campaign treatment.

One cell receives the test, and the other cell does not. The response rate or net revenue is compared between the two cells. If the test cell outperforms the control cell (within testing parameters of lift, confidence, etc.), the campaign is deemed significant and positive.

Why Do Something Else?

However, this procedure lacks insight generation. It optimizes nothing, is performed in a vacuum, gives no implications for strategy, and there are no controls for other stimuli.

Secondly, the test is often polluted, as at least one of the cells has accidentally received other offers, brand messages, or communications. How many times have the test results been deemed inconclusive, even nonsensical? So they test again and again. They learn nothing, except that testing is ineffective.

That’s why I recommend using ordinary regression to control for all other stimuli. 1 This is not done in a vacuum, but rather provides options as part of a portfolio to optimize the budget.

A Regression Modeling Example

Let’s say we were testing two emails, a test versus a control, and the results came back nonsensical. Then we discovered that our brand department had accidentally sent a direct mail piece to (mostly) the control group. This piece was not planned (by us) nor accounted for in randomly choosing the test cells. That is, the business-as-usual group received the usual direct mail, but the test group, which was held out, did not. This is very typical in a corporation, wherein one group does not work or communicate with another business unit.

Instead of testing where each row represents a customer, we aggregate the data by time period, such as weekly. We add up, by week, the number of test emails, control emails, and direct mails sent out. We also include binary variables to account for season, in this case, quarterly. Table 1 presents a partial list of the aggregates, with the email test commencing in week 10. Now we do a model:

Loading formula...

The ordinary regression model, as formulated above, produces the output shown in Table 2. Include any other independent variables of interest. Of particular notice should be that the (net) price is excluded as an independent variable. This is because net revenue is the dependent variable and is calculated as (net) price * quantity.

Table 1

week	em_test	em_cntrl	dir_mail	Loading formula...	Loading formula...	net_rev
9	0	0	55	1	0	$1,950
10	22	35	125	1	0	$2,545
11	23	44	155	1	0	$2,100
12	30	21	75	1	0	$2,675
13	35	23	80	1	0	$2,000
14	41	37	125	0	1	$2,900
15	22	54	200	0	1	$3,500
16	0	0	115	0	1	$4,500
17	0	0	25	0	1	$2,875
18	0	0	35	0	1	$6,500

To include price as an independent variable means having price on both sides of the equation, which is inappropriate. (My book, Marketing Analytics: A Practical Guide to Real Marketing Science, provides extensive examples and analysis of this analytic problem.) The adjusted R2 value for this model is 0.64. (I dropped q4 to avoid the dummy trap.) emc = control email and emt = test email. All variables are significant at the 95% confidence level.

Table 2

	Loading formula...	Loading formula...	Loading formula...	dm	emc	emt	const
coeff	-949	-1,402	-2,294	12	44	77	5,039
st err	474.1	487.2	828.1	2.5	22.4	30.8
t-ratio	-2	-2.88	-2.77	4.85	1.97	2.49

In terms of the email test, the test email outperformed the control email by 77 vs 44, and the difference was much more significant. Thus, accounting for other things, the test email worked. These insights come even when the data is polluted. An A/B test would not have yielded this result.

Table 3 presents the coefficients used to calculate MarCom valuation, which represents the contribution of each vehicle in terms of net revenue. That is, to estimate the value of direct mail, the coefficient of 12 is multiplied by the mean number of direct mails sent, which is 109, to get $1,305. Customers spend an average amount of $4,057. Thus $1,305 / $4,057 = 26.8%. That means direct mail contributed nearly 27% of the total net revenue. In terms of ROI, 109 direct mails generate $1,305. If a catalog costs $45 then ROI = ($1,305 – $55) / $55 = 2300%!

Because price was not an independent variable, it is usually concluded that price’s impact is buried in the constant. In this case, the constant of 5039 includes the price, any other missing variables, and a random error, accounting for approximately 83% of the net revenue.

Table 3

	Loading formula...	Loading formula...	Loading formula...	dm	emc	emt	const
Coeff	-949	-1,402	-2,294	12	44	77	5,039
mean	0.37	0.37	0.11	109.23	6.11	4.94	1
$4,875	-$352	-$521	-$262	$1,305	$269	$379	$4,057
value	-7.20%	-10.70%	-5.40%	26.80%	5.50%	7.80%	83.20%

Conclusion

Ordinary regression offered an alternative to provide insights in the face of dirty data, as is often the case in a corporate testing scheme. Regression also contributes to net revenue as well as a business case for ROI. Ordinary regression is an alternative technique in terms of MarCom valuation.