A Fortune 1000 retailer recently worked with Quris to create a comprehensive e-mail program for its customers. The company had not used e-mail extensively before. The goal was to show a general increase in store activity and net revenue through e-mail to opt-in e-mail customers. The company also wanted to prove that it could influence customer behavior through e-mail.
With this experiment, the company created one of the most comprehensive, valuable exercises in the direct marketing industry.
The company set up a test population of just under 220,000 customers who previously had given the company their e-mail address and permission to use it. Several barriers had to be overcome to be successful:
Two-step process. The company does not sell products by mail or through its Web site. Customers must come to the stores to buy. So, to learn whether the e-mails were working, the company had to measure store activity by the folks who got the e-mails. A lot of store transaction data had to be analyzed.
Setting up a control group. Some people asked the company to send them e-mails about new product releases. To prove the value of the e-mails, the company had to withhold e-mails from 16,000 people who had asked for them.
Data formats and location. As is common with large corporations, data was in different formats and different places.
Channel overlap. Many channels touched the customers. Those who got e-mails also saw print ads, direct mail, Web sites and TV news about the products. How could the effect of e-mail be determined while controlling for the influence of other channels?
Electronic coupons. These were new to the company’s customers. Technology and procedures for distribution and redemption had to be developed, and customer acceptance/familiarity developed.
In its planning the company learned that:
o E-mail programs have different goals from direct mail. Though both aim to increase store activity, direct mail is offer-based while e-mail has an information, entertainment and long-term relationship-building purpose.
o E-mail recipients had to receive consistent communications. To track e-mail success it was necessary to measure the store activity of the targeted customers before and after the e-mails.
Direct mail has unknown recipients until the campaign list criteria are specified. E-mail requires opt-in to receive the communication, but the recipient does not have to be a customer. The company found that the recency of the opt-in had a significant effect on the customer’s behavior. Those whose opt-in was older than a year had an overall response (at least one e-mail open) during the program of 20 percent. E-mail opt-ins newer than one year had a response exceeding 85 percent.
What customers received was a newsletter about the products. Quris also set up an interactive Web page where customers could subscribe to the newsletter. Members also could sign up for the newsletter at the stores, through partner agreements and through sweepstakes.
Setting up the test and control groups. The sample of just under 220,000 was divided into several groups for testing purposes. A control group of nearly 16,000 opt-in e-mail customers was assigned at random before the program began. This group got no e-mail messages.
Customers with opt-ins older than a year (nearly 90,000) got a reconnect message to which they could opt out. Then the total test universe of more than 170,000 was divided into four basic groups of about 43,000 each. Cells 1 and 2 got a constant message every two weeks. Cells 3 and 4 got messages based on their prior month store visits. Cells 2 and 4 received e-coupons, while cells 1 and 3 did not. The coupons tested three offers:
o Get one product, get one free anytime.
o Get one product, get one free in a five-day period.
o Get one product, get one free Monday through Thursday.
A particular difficulty involved how to measure customer activity. Was it total expenditures, number of transactions, frequency of transactions, etc. The company emerged from this test with valid quantitative measures to show the lift from e-mail on an ongoing basis.
The company determines ROI based on incremental total net revenue. Quris augmented this traditional analyses by isolating the source of the lift: i.e., did e-mail drive new customers to the store? Or the same customers more frequently? Or the same customers to spend more? Also, Quris looked at purchase propensities (e.g., comparing different types of transactions).
Results. The lift for the entire test population versus the control sample was 28 percent. In other words, sending e-mails twice a month increased sales by 28 percent over sending nothing.
The total lift for customers who received electronic coupons was higher than for those with no coupons. The coupons worked. Previously active customers had a higher lift in ROI than less-active customers. The e-mails were successful in reactivating inactive members.
Quris and the company used all the techniques available to do a thorough job of exploring the relationship between e-mails and customer behavior. Too few companies have rigorously used control groups to prove that what they were doing was working. This analysis validated the e-mail channel to the company’s senior management. As of 2003, the company sends eight newsletters and more than 1.5 million e-mails a month.
