Managing test campaigns can be a critical step for any organization that wants to continuously improve response rates, ROI and the effectiveness of its communications to customers. Gleaning the relevant information from these campaigns can be challenging, but it is vital in a competitive market. In recent years, marketing organizations have made strides to re-invent how these campaigns are executed, but many companies still have progress to make.
Today’s Landscape
Test and control — The most common approach to creating test campaigns is to use simple control groups. The purpose of the control group is to compare marketing effectiveness of a group that received a communication against a group that did not receive any communication. This comparison allows the marketer to determine who may have bought a product regardless of being targeted. Test and control groups are created by a process of cell-splitting — once a segment has been determined, it is divided in a random fashion to get representative customers in each of the cells.
How big should a control group be? There are common approaches to determining the size of a control group. Ideally, a marketer would employ some statistical analysis to ensure that the size of the control group is large enough to guarantee that the post-campaign analysis is valid. Very often, organizations follow standard procedures that stipulate a certain number or percentage be used for the hold-out group. For example, test campaigns or control groups must be at least 50,000. This approach is less than ideal and reduces the efficiency of the test.
Champion/challenger – Another common testing technique is when a direct marketer uses an existing campaign as a base to compare against a different treatment. An example of a champion/challenger campaign might be changing the envelope or the color of the copy. The audience of the campaign might be split using similar methods described above and considerable thought may go into determining why a certain creative might prove more effective for the segment that is being tested.
Tomorrow’s Promise
Test and learn – An emerging technique in direct marketing, test and learn, is a far more complex approach than the traditional methods listed above, yet it holds tremendous promise. It overcomes the shortcomings of traditional methods that can only test one factor at a time. To test the full range of factors, the marketer may need to test hundreds of cells. Such sweeping tests are often impractical, not to mention expensive and time-consuming. This older method also assumes that each factor that is being tested is independent of the campaign’s other attributes.
Test and learn solves this problem by using advanced analytics to select the appropriate cells to test. Done correctly, the marketer can make the same, powerful conclusions about test cells that were not tested. For example, if the algorithm reduced the number of test cells from 150 down to 20, conclusions about the 130 test cells that were not part of the test can be statistically derived from the results of the 20 cells. This approach manages the complexity of a campaign without reducing the value of such an approach.
This technique, also referred to as experimental design, has great potential for organizations that want to employ a scientific approach to marketing. It allows an organization to respond quickly to test campaigns and then change messaging and content based on what is successful. The competitive marketplace demands that offers and content remain relevant and appropriate. There is no better method than testing to make sure this is true.
Sequential analysis – Marketers are becoming more interested in the sequence of communications and finding the right balance across channels. Forward-looking marketers are beginning to ask these questions: “How many different offers should I send the same customer before I give up? Does the order matter? What effect does changing the channel have on response rates?”
To analyze the order of communications and their effect on productivity, new data mining techniques, such as sequential analysis, have emerged. Information gleaned from these campaigns can be directly applied to similar segments in the customer base. Analytical methods are critical here because it can be difficult to ascertain the impact of an individual offer. At the surface, it may seem that one offer was highly effective, but by looking at its context, marketers can discern some otherwise hidden patterns. The concept of significance is quite important in this analysis.
Number of unique offers – Another common question that is being tested is, “How many of the same offer should be extended to the same customer?” We know that customers frequently fail to respond the first time an offer is sent. How often should an offer be extended before marketers stop? Should they try other channels? The difficult answer, as always, is “It depends.” Finding the right balance requires a variety of testing approaches to determine what types of strategies are working on what types of segments or individuals. An experimental design may offer insight into some of these questions and unlock value in future communications.
Managing Complexity
Test campaigns can be an intensive analytical process. Yet this testing need not be intimidating since there are software solutions that can help. Successful test campaigns require a combination of data management, analytics and marketing automation. These processes, once they are implemented, can reap substantial long-term benefits as you continuously learn what is most effective and appropriate for your customers.
