Catalog Testing the Right Way

Share this content:
Becky Jewitt, president/CEO of Norm Thompson, Hillsboro, OR, made a presentation this month to the DMA/Catalog & E-commerce Club of Northern California Road Show on the specialty retailer's efforts to become a sustainable company.

She mentioned that they were testing how the use of recycled paper for their catalogs affected sales.

That made me think that many people believe they understand how to conduct and read a test.

However, my experience is that when it involves concepts or the recipients' perceptions of the company, they do not know how to test.

Testing in cataloging is unlike that in general direct marketing. In traditional direct marketing, a company will create several mailing pieces.

Then it will determine a winner among them by mailing all at the same time and measuring response. The winner becomes the control in subsequent mailings against which all other concepts are measured. In cataloging, since the product mix and creative presentation changes from catalog to catalog, you never develop a true control mailing piece.

This makes it impossible to conduct a clear test.

For instance, if a cataloger is testing a new cover concept, the normal cataloging method is to mail two different concepts at the same time to determine which one performs better. However, once a winner is determined, the cataloger never uses that cover again. This is because most catalogers show product on their front covers and they will not use a cover depicting last year's or last season's product.

The change in product normally necessitates a not-so-subtle change in presentation so that frequently the only common thread is the company logo on the cover.

Thus, unlike traditional direct marketing, all mailings are, in essence, a test. This even carries over to list testing and rolling out the winners.

Here the catalog offerings change as well as the composition of the rented list. The offer/presentation that was presented to the test cell is different than that used for the rollout. Also, the list itself is dynamic, and the composition of the test cell is different than that of the rollout. However, catalog companies have learned to compensate for this over time by not expecting the same level of response on the rollout, which is proof of the dynamics of changing the offer and presentation.

One of the many things not discernible before testing is the intent of the prospective buyer to buy.

All catalogers and remote shopping companies try their best to put their offers in front of those who are most likely to purchase based on their prospects' past transactional behavior. However, everyone knows that in most cases the response will be only 3 percent or less.

Does this mean that the list mailed is bad? No.

The problem is that a person's propensity to purchase is difficult to gauge, especially in a push marketing technology such as cataloging.

Therefore, creating and maintaining a clear image/brand is critical to success in push marketing.

If a product has wide distribution, such as a soft drink, the image is built and maintained through advertising and people just seeing it on shelves. If it is a retail store, its physical facility is a major, if not the major, method of building awareness, one of the reasons behind the saying "location, location, location."

For catalogers that constantly change their cover image, maintenance is harder to achieve. Only those that can afford saturation mailings like Lands' End or Victoria's Secret are likely to burn their images into consumers' memory banks. And that is why elements such as paper are so critical to image building, as people may not remember seeing a catalog before, but the quality and weight of the paper immediately makes a subliminal statement about a company and its products.

However, typically when a catalog company tests for anything, it only measures the results over a mailing or two. This is because of the dynamic nature of the catalog, always changing product offerings and presentation and the desire for instant answers.

While catalogers have learned to work with this uncertainty for list testing, it is dangerous when trying to measure the effect of less obvious changes such as paper type or quality.

If you tried to measure the effect of the proposed paper tests by Norm Thompson over just a mailing or two, there is a high probability that the results could be misleading. If a company mailed to 1,000 individuals who had never purchased before a catalog every month for 12 months with a 3 percent response rate, there still would be almost 70 percent who do not purchase. Does that mean they are not interested in the company's offerings or presentation? Not necessarily.

However, even with a 70 percent unknown or nonresponse return on the 12 mailings, this is far superior to the knowledge gained in two or three mailings where the unknown/nonresponse factor exceeds 90 percent. At least by measuring a subliminal impact factor such as paper change, and assuming an announcement that it is recycled paper over a lengthy period, such as 12 to 18 months, a company will reduce the possibility of a misread of the data.

As mentioned above, to do lengthy tests is not in the nature of catalogers.

However, on subjects such as paper or delivery charges, to not measure it over time greatly increases the risk of drawing the wrong conclusions. And that will have a major effect in the future.

• Bill Dean is the founding partner of W.A. Dean & Associates, San Francisco, a catalog consultancy that provides companies with professional recommendations to aid them in their decisions about their remote shopping options. Reach him at or 415/547-2930.

Next Article in Multichannel Marketing

Sign up to our newsletters

Company of the Week

We recently were named B2B Magazine's Direct Marketing Agency of the Year, and with good reason: We make real, measureable, positive change happen for our clients. A full-service agency founded in 1974, Bader Rutter expertly helps you get the right message to the right audience at the right time through the right channels. As we engage our clients' audiences along their journey, direct marketing (email, direct mail, phone, SMS) and behavioral marketing (SEM, retargeting, contextual) channels deliver information relevant to the needs of each stage. We are experts at implementing and leveraging marketing technologies such as CRM and marketing automation in order to synchronize sales and marketing communications. Our team of architects and activators plan, execute, measure and adjust in real time to ensure the strategy is working as needed and change things if it's not.

Find out more here »

Career Center

Check out hundreds of exciting professional opportunities available on DMN's Career Center.  
Explore careers in digital marketing, sales, eCommerce, marketing communications, IT, data strategies, and much more. And don't forget to update your resume so employers can contact you privately about job opportunities.

>>Click Here

Relive the 2017 Marketing Hall of Femme

Click the image above