It’s a safe bet that few readers of this column care deeply about the technical differences among multivariate testing methodologies. Taguchi, optimal design and discrete choice models each have strengths and weaknesses, but all are ways to quickly and efficiently identify optimal combinations of marketing treatments.
The underlying technology can’t be ignored totally, because it can affect practical issues such as scalability and flexibility to handle unexpected needs. But overall, users evaluate testing systems like any other software: by looking at what it would be like to use them, with little concern for what goes on under the hood. What matters is the result: better-performing promotions.
Memetrics xOs (Memetrics, 415/513-5120) is a multivariate testing system based on discrete choice models. This approach measures consumer preferences by asking them to choose among versions of a complete offer, each having a different combination of attribute values. Think of product design: Attributes might be size, price, color, etc. Consumers would choose among sample products with different values for each. Discrete choice models have proven more effective at determining the impact of each attribute and value than asking about single attributes or values in isolation.
This is all heady stuff, and there are Nobel Prizes involved. But for projects such as Web page optimization, the practical result resembles other multivariate testing methods: Each page is divided into one or more zones (attributes) such as message, image and offer, and these are assigned multiple test values.
If all possible value combinations have been tested, something called a “full factorial” design, the system will identify the best-performing combination as optimal and identify any relationships (“interactions”) among values. If only some combinations have been tested, the system ignores any interactions, identifies the best-performing value for each attribute and proposes the combination of best-performing values as optimal. Testing only some combinations is a typical multivariate approach that yields faster results from smaller, simpler tests. It requires proper test design, which, like other multivariate testing systems, Memetrics does automatically.
Either way, the system also builds a “choice model” that can estimate results for any combination of values. This can be helpful if the user is interested in multiple outcomes – say, gross revenue, profit margin and number of orders – and wants to balance them against each other rather than maximizing just one. The Enterprise version of xOs can track several outcomes and model them separately or combine them into a single measure and model that. Enterprise users also can assign an offer cost and selection value to each outcome and combine these into a target measure. Outcomes can be based on information captured during an interaction or imported from external sources such as an order processing system. These features are unavailable in the simpler Express version of xOS, which tracks only one outcome.
Setting up a test in Memetrics begins with defining the attributes, values, outcomes and proportion of traffic to be tested. Attribute values can be defined with a name, Internet address (URL) or by uploading the actual content to the Memetrics server. A sample size calculator helps users determine the number of attributes and values to test based on traffic volume, time available, conversion rates, expected response variations and target confidence level.
Once the elements are specified, xOs generates a block of Javascript that identifies the test and its attributes. The user then embeds the Javascript in the Web page to be tested. The Javascript calls the Memetrics server each time the page is displayed, letting Memetrics assign each visitor to a test, control or default group and present the appropriate content. Memetrics uses persistent cookies to identify site visitors so it can ensure consistent treatment when they return.
The Memetrics Javascript also can capture information from the user’s URL, such as the search query that led the user to the page. This is stored on the Memetrics server and used to analyze results by visitor segment. Javascript for the same test can be embedded in several pages, allowing consistent treatments and tracking of results such as registration or purchases.
The Express version of xOs is limited to the Javascript approach. Enterprise also can use techniques such as .NET, PHP and JSP to communicate with Web servers or other interaction systems such as call centers. This lets Enterprise manage tests across multiple channels. Enterprise, but not Express, also can use real-time filters to limit tests to predefined customer segments. These filters can access data provided by the interaction system or read from other sources such as a customer database.
After a test is complete, both systems let users define segments based on whatever visitor data are available. Users can test alternative segmentation approaches to find the best results. xOs can build one model against the entire test universe or build separate models for each segment. Reports for each model show the effect of each value and its statistical significance.
Users can accept the system’s choice of optimal values or select their own, and then deploy this combination as a default. Either way, the system will show the expected results for the specified combination. There is no automated adjustment of default values as customer behavior changes over time. Memetrics argues that humans should examine each test result and make conscious decisions about what to do next. A typical Web page test runs two to four weeks and evaluates five or six attributes, each with multiple values.
The default values will be shown to all visitors outside of a test sample and also are displayed if the Memetrics server is unavailable. The default contents also are viewed by Web search engines.
Memetrics was founded in 1999 and has more than 30 clients. Its original product was Enterprise, which is priced at $150,000 yearly plus consulting. It can run in-house or be hosted by Memetrics. Express is a hosted service that was introduced in 2006. Price begins at $40,000 per year and is based on volume.
Memetrics xOs testing system finds the best X’s and O’s
