Most direct marketers find statistical formulas difficult to understand. Many just ignore them. Or they take the easy road and heed consultants who say things such as, “You're fine as long as you have 50 responders.”
This widespread lack of sophistication costs direct marketers dearly in the form of unrealized revenue. This article will help illustrate the magnitude of the problem via a two-part statistical formula. This formula, first presented in “How Big Should My Test Be?” (DM News, Oct. 1, 2001), assists in determining the quantity of names to include in test panels. You will see how inputting this formula into a spreadsheet and investing an hour or two in experimentation might generate millions of dollars for your direct marketing business.
A quick review of the October article. This is the formula for determining how many names to include in a test panel:
Part 1: (Expected Response Rate X (1 – Expected Response Rate) X Z2) ÷
Precision2
Part 2: Answer to Part 1 ÷ (1 + (Answer to Part 1 ÷ Rollout Universe Quantity))
Understanding “Z” would require a statistics lesson. All we need to know is that it corresponds to the level of confidence that we have in the accuracy of our test panel response rate.
The following are six combinations of Z and confidence levels: a Z of 1.96 corresponds to a confidence level of 95 percent; a Z of 1.645 to 90 percent; 1.282 to 80 percent; 1.04 to 70 percent; 0.84 to 60 percent; and 0.67 to 50 percent.
Precision describes the degree of plus/minus uncertainty around a test panel response rate. We can never know for sure from a test panel response rate what the true rollout rate will be. For example, with a test panel response rate of 0.8 percent and a universe size of 100,000, a test panel size of 5,273 will result in being 50 percent confident that the rollout response rate will be between 0.72 percent and 0.88 percent. In other words, one out of every two times the rollout rate will be within 10 percent of the test panel rate.
You can check this by substituting “0.67” for “Z” in the formula, and “0.08 percent” for “Precision.” You will get an answer of 5,273.
We have just established that half of the time the true rollout response rate will be between 0.72 percent and 0.88 percent. By definition, one out of every two times it will be outside of this range. Therefore, by extension, one out of every four times the rollout rate will be less than 0.72 percent.
Generating millions of dollars. For most direct marketers, the effort involved in understanding this formula is less pleasant than tasks such as sourcing new merchandise and working on promotional layouts. However, it is just as important. We will build upon the example from the previous section to illustrate why this formula might generate millions of dollars for your business.
We established that one out of every four times the true rollout response rate will be less than 0.72 percent, one out of two times it will be between 0.72 percent and 0.88 percent, and one out of four times it will be more than 0.88 percent. Of course, there is no way to know what the true rollout rate is without going through the effort of contacting everyone. However, assume that we magically know in advance that the true rate is identical to the test panel rate of 0.8 percent. Though rare in direct marketing testing, it does happen on occasion that the two are the same. Also assume that 0.72 percent — or, 10 percent less than 0.8 percent — is the minimum test panel response rate required for rollout.
With this information, we know that one out of every four times a test panel size of 5,273 will result in failing to roll out the list select, because the test panel response rate will be less than the required 0.72 percent. This is a missed opportunity because the true rollout response rate of 0.8 percent is comfortably more than the minimum. The financial ramifications of this missed opportunity are profound because rollouts generally are repeated many times.
So assume that we contact proven rental lists three times a year. By failing to roll out the 100,000 list select, we will have failed to cost-effectively generate 300,000 promotions a year, or 1.5 million over a five-year period. Using our response rate assumption, that's 12,000 missed customers!
Now, assume that each new customer will, on average, order one additional time, and that the size of each order is $90. That translates into 24,000 missed orders and $2.16 million of missed revenue over the five years. And that is from just a single missed rollout.
There is no right or wrong answer for test panel quantities. No one size will be optimal for every direct marketer. The appropriate quantity will depend on factors such as the amount of money available for testing and the level of risk the direct marketer is willing to assume that the rollout response rate will be significantly different from the test rate. Nevertheless, most direct marketers are appalled when they are made to understand the profound financial ramifications of small test panel quantities.
Fortunately, it is possible to minimize the risk of failing to identify the $2.16 million in the example. What has to be done is to enlarge the test panel. Let us explore the effects of various panel quantities on the accuracy of our test reads. Throughout, we will assume that promotional costs, including list rental, are $1 per thousand.
If we increase our test panel quantity from 5,273 to 8,046, we will spend an extra $2,763. With a response rate of 0.8 percent, we will raise our responder quantity from 42 to 64. As a result, we will fail to identify the $2.16 million opportunity just one out of every five times rather than one out of every four.
If we move from 5,273 to 11,826, we will invest an extra $6,543 in order to fail to identify the $2.16 million just 15 percent of the time. That is going from 42 to 95 responders. Likewise, with 16,930, we will spend an incremental $11,647 to fail just one out of 10 times. Finally, with 25,124, we will invest an extra $19,841 to fail just one out of 20 times.
Final thoughts. You will have to decide which quantity and corresponding cost is the best balance for you. As the test quantities increase, the chances of failing to identify the $2.16 million decrease. At the same time, the costs of testing increase. There is a specific point that reflects your personal equilibrium.
Be mindful throughout that the incremental costs presented earlier for increased certainty are based on gross rather than net calculations. Orders will be generated from the additional promotional quantities, which will defray a portion of the costs. And, when you are more confident in your results, you can proceed to full rollout more quickly.
Most direct marketers have no idea of the extent of the missed opportunities that result from small test panels. This is understandable because missed opportunities are, by definition, a hidden phenomenon. Nevertheless, direct marketers unwilling to understand the basics of statistical sampling theory pay a steep tax for their lack of knowledge.
Unfortunately, the list brokerage industry generally does little to educate its clients on these issues. This is because most list professionals are no more comfortable with sampling theory than their clients. However, that is the subject for another article.
