The only hard part about testing is that there is no “right” answer because the target is always moving. The things that delight customers go in and out of fashion and all you are doing by testing is trying to catch and ride that wave. û John Wall (www.themshow.com)
One of my first jobs in industry, years ago, was as a production control clerk at the Thermoid Corporation in Trenton, NJ. To control production they had a library of 5 x 8 cards, one for each of the thousands of different automotive rubber products that they manufactured. At the top of each card, a senior analyst had written the minimum on hand stock level, and standard order quantity. On the body of the card, junior analysts posted daily additions and subtractions to the quantity on hand. My job was to look at each card every day to place orders for manufacture of any part whose stock level fell below the minimum.
This activity was, of course, before the days of computers. Today this simple process is done electronically, eliminating not only my old job, but the jobs of the analysts who posted the data, and the senior analyst who controlled the system. I am sure that there was some sort of a fight before management could replace them. At the time all of us working there assumed that what we were doing was state of the art – which it probably was. If it ain’t broke, don’t fix it. The changes when they were made, of course, made Thermoid and every other manufacturing company tremendously more efficient, bringing prices down and profits up.
The same attitude toward change exists today in the direct marketing industry. I recently saw a quote by a junior analyst in a company doing customer e-mail marketing. “Testing: A process designed to: 1) make marketing more complex, 2) slow down production and 3) prove to that pigheaded boss that yes, we really do know what already works.” I even heard a senior marketing VP at another major corporation with 25 million annual customers say about a year ago in regard to their daily direct mail program, “We don’t need testing. We know what works.”
More people than you would think have the attitudes of the junior analyst and the marketing VP. Yet, without testing, it is almost impossible to improve. What we are doing may work, but would something else work better? That is the question that cannot be answered without constant testing. If we do not test, our competition may be testing, and learning something that we have not discovered. Our perfectly good direct marketing program may be beaten by our competitor’s even better one.
Testing can involve considerable internal corporate contention. If your regular promotion produces a response rate of 2 percent and your test produces only 1 percent, you will be accused of wasting the company’s money by doing the test. If your test produces 3 percent, you will be accused of wasting money by mailing your regular 2 percent promotion. If you do no testing at all, no one will complain. This situation makes it difficult to get tests approved.
Control group problems
To test in the e-mail and direct mail industries you have to have control groups. Control groups are folks who get what you have always sent, while test groups get something new and different. You learn by comparing the response and conversion rates of the two groups. Each group has to be large enough to give statistically valid results.
Setting aside a control group can be even more difficult for you than getting approval to do a test. To be valid, a control group has to be an exact statistical replica of your existing customer base. That means that it must include some of your very best customers, as well as the others. It also means that other marketers at your company should not stimulate your control group during your testing period. Readers of Strategic Database Marketing may remember the case study of what happened when a major retail chain created a control group for their gold customer program. The control group, like the test group, had to be composed of some of the chain’s best customers. The benefits given to the gold members were so significant that when some members of the control group discovered they were not getting these benefits, they called customer service to complain. Customer service was told that when that happened, they were to say, “There must have been a computer error. You are enrolled in the gold program effective immediately.” The marketers at this retail chain called those folks who were switched, the “Out of Control Group.”
Measuring the half-life effect
Many marketers today are still unaware of the usefulness of half-life analysis. In any promotion without a firm closing date, the responses and sales may come in over a period of weeks or even months. If you do such promotions on a regular basis you usually cannot afford to wait for the results of any one promotion before you get the next one out the door. Half-life is a phenomenon that is almost always true in direct marketing: if the results come in over a period of time (such as 90 days) there is one day on which half of your results have arrived. That day is usually the same for all of your promotions. Sears Canada, for example, mails 13 books a year. Their half-life day is day 20 after the first purchase from any book. By keeping track of your sales by day, and multiplying the sales achieved by the half-life day by two, you can learn after 20 days what will have happened after 90 days. It speeds up the analysis of your testing program. Study your promotions and learn what your half-life day is.
Measuring the effect on customer segments
An important part of your testing should be to determine the effect of your communications on your best customers. To learn this, you should segment your customer base by their spending habits. If you are like most enterprises, you have about 20 percent of your customers that give you 80 percent of your revenue. This gold group may be willing to pay full price because they like your products, your service and your sales staff. The remaining sales come from the other 80 percent of your customers who may be motivated mainly by bargains and sales. If your e-mail promotions, for example, are chiefly announcing discounts, you may be doing a good job with your least valuable customers, but turning off your best ones. How can you learn that important information?
Paul Wang used to discuss the case study of a hotel that hired a new manager for their restaurant. To build up business, he invited rock bands to entertain diners in the evening, turning the hotel restaurant into a very loud and very successful night club. Unnoticed in the commotion every night was the patronage of the hotel’s business customers who represented 80 percent of the hotel’s revenue. With all the noise every night, these business customers gradually began to locate other quieter hotels in the area. The restaurant prospered, while the hotel lost its most valuable customers.
So if your – campaign, for example, is bringing in business, is it the business that you want? What is happening to your best customers and to your bottom line? This question must be included in your analysis of your test results.
Long-Term effects of promotions
A promotion campaign typically is measured by return on investment. You spend so much on the mailing, and you get so much net revenue. What is seldom measured is the long-term effect on the customers who respond and who do not respond. A mailing, for example, may bring customers in to your Web site or retail store. They may or may not buy. You measure your ROI by those who buy. But what is the long-term effect? Some customers drove to the mall, parked their car, and walked into the store as a result of your promotion. Perhaps they bought something, or did not buy. But now they know where your store is, and they have seen a lot of things that may have attracted their interest. They may return tomorrow or next week. There is usually a long-term effect of any promotion beyond the ROI of any particular campaign. Sophisticated testing will measure this long-term effect. It may be double the ROI of the campaign, or even higher if this customer becomes a regular.
A testing checklist
So, if you have gotten religion, and realize that to be successful at customer (and prospect) communications you must test, what are the rules you should follow?
1) Make every single promotion a test. By not testing, you are throwing away a golden opportunity to improve, an opportunity that may give your competitors an edge.
2) Study and act on the results of your tests. Too many marketers are so busy getting out the next promotion that they fail to determine what they learned from the last one. Not acting on test results is an even worse mistake than not testing at all.
3) Don’t test too many things at once. Design your tests to give a definitive answer to a simple question: Which gets better results, A or B? If you have the ability, you can simultaneously test A vs. B vs. C vs. D. But make sure that all of them are identical except for the single thing that you are varying (offer, subject line, first paragraph, etc.)
4) In e-mail testing, you absolutely need to test frequency of mailing. Mailing too often can turn off your best customers û a disaster. Not mailing enough can let sales slip through your fingers.
5) Segment your customers and use analysis to determine the effect of your promotions on your best customers.
6) Make your test groups large enough to give statistically valid results. Be sure that each group is an exact statistical replica of your total database, so, if the test is successful, you can roll it out to your entire customer or prospect base.
7) Use half-life analysis to speed up the measurement of your results.
8) Find a way to measure the long-term effect of your promotions.
