What marketers are doing wrong

Share this content:
Darrell Lester
Darrell Lester

If you're a direct marketer, then you know about panel testing. However, there is something that you are likely doing wrong, and it could be hurting your bottom line.

The recommendations from the statisti­cians pertaining to what size test panels to mail would indicate, for example, that an outgoing test panel size of 50,000 is required for a test result variability of plus or minus 5% (at a 95% confidence level).

This is absolutely correct from a statisti­cal theory point of view. But does it hold true when you roll out? Even though you do everything right, with randomly select­ed names, and so on? The answer is no.

It turns out that these decisions based on statistical theory are not accurate. Follow­ing is an example that explains how I know this to be true.

During my time at Publishers Clearing House, testing was our lifeblood. In fact, every week for 30 years I looked at and scrutinized test results. I did this as a young marketing analyst when I first started in 1973 and kept doing this when I was an SVP and officer in the company before I retired a few years ago.

And we tested everything. We performed about 125 tests every year, from copy changes to format changes to product of­fers to pricing to contest techniques. You name it, we tested it.

We were using the traditional formulas for test panel sizes. However, we realized that the risk of a wrong test result could be very harmful.

So what did we do? We tested it. We tested the validity of test panel size and variability. We did this by running several control panels under the typical random se­lection (including several other techniques to maintain randomness) and mailed them. We ran this several times with varying out­going mail volumes.

And after all this testing, what did we find out? That the bottom line about panel testing is that you need to increase outgo­ing test panel sizes by about 50% (versus what statistical theory indicates) to be able to have a high confidence level in your findings.


Sign up to our newsletters

Company of the Week

We recently were named B2B Magazine's Direct Marketing Agency of the Year, and with good reason: We make real, measureable, positive change happen for our clients. A full-service agency founded in 1974, Bader Rutter expertly helps you get the right message to the right audience at the right time through the right channels. As we engage our clients' audiences along their journey, direct marketing (email, direct mail, phone, SMS) and behavioral marketing (SEM, retargeting, contextual) channels deliver information relevant to the needs of each stage. We are experts at implementing and leveraging marketing technologies such as CRM and marketing automation in order to synchronize sales and marketing communications. Our team of architects and activators plan, execute, measure and adjust in real time to ensure the strategy is working as needed and change things if it's not.

Find out more here »

Career Center

Check out hundreds of exciting professional opportunities available on DMN's Career Center.  
Explore careers in digital marketing, sales, eCommerce, marketing communications, IT, data strategies, and much more. And don't forget to update your resume so employers can contact you privately about job opportunities.

>>Click Here

Relive the 2017 Marketing Hall of Femme

Click the image above