8 Secrets to Better Insert Media Tests
The challenge is that testing in insert media is often more of an art than a science, and is much tougher than testing in direct mail. Unlike direct mail, where you can control the universe of names being tested, mail all pieces on the same date and perform exact A-B splits, insert media is much less scientific and much less reliable.
Yet this should not discourage an insert mailer from doing regular testing for its product. Here are a few rules to ensure that you obtain the most accurate and reliable test results:
Look for big lifts. Unlike direct mail, where a 15 to 25 percent lift on a test is very strong, insert media requires bigger lifts before deeming a new creative or new offer the control. To safely roll out with a winning test, look for lifts of 50 percent or more. Increases of less than 50 percent are nice, but must be discounted a bit because of the unscientific nature of insert testing.
Back-test to confirm results. Once you've declared a test package the control, back-test the old control in the rollout to ensure that your winner was as good as your original test indicated. Often you'll find that an old control that lost to a test will beat the new control in a back-test, especially if the lift on the original test was not that significant.
Consider using two control panels in your tests and averaging the results. You'll get a stronger indication of how good a test panel is. If it wins two control panels handily, you know you have a winner.
Commingle pieces when you can. Assuming the size and weight of your test pieces are the same as the control, ask your printer to commingle the pieces at the production stage. This will give you a better geographic split when you perform tests.
If your insert piece is not the same size and weight as your control, try to arrange with your broker to have the test pieces interspersed with the control when the program owner is collating packages. It won't be perfect, but it might help counter the geographic bias that results from inserting pieces in their entirety before switching to a new insert.
Back into the universes you need to obtain for a reliable result. Generally, you'll want about 50-100 paid buyers to get a reliable read on your test. You'll need to assume certain response rates and conversions (depending on whether you are going out with a lead generation offer, soft offer or hard offer), which will then tell you how many total pieces you need to insert for an accurate analysis of your test. The goal is to get numbers that are statistically significant and that will let you feel comfortable on a rollout.
Choose programs that have a good track record on inserting your piece. There's nothing worse than picking a program for your test that gets delayed, or that ships half of its expected volume. Go with the programs that have shown the most reliability for you in the past, and that you feel confident will insert your piece in the required time frame. Your broker can help here, too.
Choose different categories of programs for your test. You might find that certain creative approaches or offers work in one category, but not in another. By testing in different categories, you'll learn about these potential differences. You might end up with different controls for different categories. Also, if a test beats the control in all categories, you'll have that much more confidence in your test results.
Get actual quantities of pieces inserted and plug in these numbers after the fact to perform a proper analysis. Because of printing overs and other factors, mailers sometimes insert slightly more or less than your estimated quantity. Get the final numbers of pieces inserted and change the universe count in your analysis reports. A 10 percent rise in the universe often makes a big difference in whether a test is a winner, so ensure to account for these numbers. Alternatively, you can ask the program owner to stop inserting at a fixed quantity.
Testing in insert media is a challenge, but by following these few basic rules, you can make the task easier and find the testing breakthroughs that we all want to achieve. n