Common Traps in Market Testing
So to the class of 2000 and our newest direct marketers, allow me to offer this compilation of lessons about market testing, learned - some more easily than others - by this direct marketer of 17 years, with the support of my colleagues Kevin Lyons and Lisa Hamilton in our company's analytics division.
Here's our advice:
Control groups. Though it might sound simple, it's critical to ensure that control group characteristics match those of the group with which they're being compared. They must mirror one another. This presents a particular challenge when conducting extended tests. With long-term tests, you have two choices: Either leave the groups as they are during the course of the test or change all of them at the same time in the same way and in the same proportion. For example, if you want to test the long-term effect of a retail loyalty program, you should not add new members to your mail pool unless you also add them to your "no-mail" control.
Rollouts. If you mail your entire universe upfront, thinking it's your best shot, you might be wasting money. One solution is to start with a smaller but statistically valid test mailing. For instance, if you eventually want to do a new product mailing to more than 1 million names, limit your liability by starting with 25,000. Analyze the response. If it meets your objectives, then roll out telescopically by mailing 125,000, then 1,250,000 and so on.
Testing size. Make sure your test cells are large enough to produce statistically valid results (it's often tempting to skimp on the size of test cells, especially no-mail test cells, in an attempt to maximize sales). A common rule of thumb suggests that your smaller cells should contain 5,000 to 10,000 names. However, the only proper way to determine accurately the sample and cell size is through the careful use of statistical tests to determine significance and confidence (if you don't have one of the statistical software packages, use probability tables).
Testing variables. Another common mistake is to test too many variables at once. Either test one thing or test everything. For example, you may test one variable, such as a change in the approach used in copy. Or test everything by completely redesigning the package. And note that major advances generally come from testing everything, while incremental gains may result from single component testing.
External factors. Not accounting for unavoidable or uncontrollable factors can prove deadly. Changes in the economy or competitive market, seasonality, the political environment and geographical issues such as major weather events (i.e., hurricanes) are examples. Conducting tests in multiple geographies can minimize the effects of these conditions. And remember, only tests conducted at the same time are valid. Rollouts will behave differently from upfront tests.
Managing rollout expectations. When rolling out after a test, it's important to understand and manage expectations. Consider what differences between the test and the rollout may exist (to meet your mail quantity, you might be forced to loosen your selection criteria, for instance). Another common mistake is to get caught up in the excitement of the test, then pay less attention to critical details in the rollout. All in all, it's best to discount the test percent of response when projecting the rollout response; a 15 percent deduction is safe, though this thumbnail may vary for your product category - monitor over time to establish your own.
Tracking your responses. It might sound rudimentary, but be sure to code accurately your mail/marketing pieces and record responses. Without a code, you may not be able to determine accurately to which offer a customer is accurately responding or from which list. And when recording responses, be sure that accurate and complete data are entered into your marketing database.
Test only what's important. This usually means list, creative and offer. But test everything you can think of that fits within these categories: products or services; media selection and mix; copy; format; timing; and positioning.
Finally, know when to test and when not to test. Be careful about getting involved in testing. If your mail quantity is so minute that nothing will be statistically accurate, your conclusions may be faulty. In this case, you might be better off stressing the art of marketing rather than the science of marketing and letting each subsequent mailing to such small universes be the testing ground for garnering better response.
Most importantly, the No. 1 rule to make the testing work is ensuring that you understand overall business goals before developing a test program. Answer questions such as, "What are we trying to learn about our customers and how do they use our products/services?" and "What questions am I trying to answer?" before putting any testing plans into place.
And one final piece of advice: Be sure to partner with an agency that has been around the block and is experienced in market testing. In the long run you'll be saved from the pain and anguish of learning lessons the hard way.