It’s a cliché, but testing truly is the heart of direct marketing. Yet
your grandfather’s A/B split is as obsolete as his Philco radio. Direct
marketers today use a more sophisticated technique known as multivariate
testing to evaluate many factors simultaneously and identify the best
possible combination, even if it’s one that was never actually tested.
In multivariate testing, the idea is that an experiment can measure the
effects of several elements, each with multiple versions, by testing a
small fraction of all possible combinations. Results for each element
are read separately: All results for headline A are compared with all
results for headline B, even though some people saw different copy,
prices, photos and so on. Results from different combinations of
elements are estimated by adding up the effects of their components.
This is a simplified explanation: There are nuances that only a
statistician could love.
Offermatica offers multivariate and A/B testing for Web marketers.
The system not only helps marketers design the tests, but
executes them by taking over specified areas on a Web page and
controlling their contents for each site visitor. The execution
functions are critical because the mechanics of test delivery are much
more demanding than the test design itself.
Each multivariate test in Offermatica starts with a test name, start and
end dates and percentage of visitors to include. Users can optionally
specify targeting rules to narrow the audience and definitions of
segments within the audience to report on. The user then builds a list
of elements and versions, which the system automatically converts into a
test grid. Finally, the user links the elements to Web page locations,
which Offermatica calls “mboxes.”
The mboxes are physically added to Web pages by inserting a few lines of
display content that Offermatica returns. Each mbox has its own
also can transmit the page ID, visitor ID and URL parameters such as
search terms. These can be used to update visitor profiles and capture
test outcomes such as clicking on an ad or placing an order. Users also
can track outcomes by importing external data such as order logs.
Visitors are identified by first-party cookies, which contain an ID
linked to a detailed profile stored at Offermatica.
Offermatica automatically builds its list of available mboxes by adding
each one when it first calls the server – that is, the first time the
page with the mbox is viewed in any browser. This usually happens
Adding mboxes is pretty much the only involvement that Web site
technicians have with Offermatica. Test content can be uploaded to
Offermatica using the system interface, stored on a client server or
reside with a third party. When the content is elsewhere, Offermatica
stores an identifier that tells the other system what to deliver. Users
can view each piece of content within the Offermatica interface and
preview full Web pages as contents are assigned to mboxes.
Reports show both the winning version for each element and the winning
combination that actually was tested. Results are updated in real time.
Users specify the success metric, with options including conversion
rate, lift, average order value, revenue per visit and total sales. The
system also shows the influence and confidence level of each result.
Users can filter results by segment, weekday vs. weekend and time
period, and can exclude very large orders that might skew results. A
“push winner” button makes the winning combination the default for all
visitors in one step. If the best combination was not tested, users must
set it up manually.
The system does not identify interactions or correlations among test
elements. Users seeking interactions can download element-by-element
statistics or have Offermatica staff explore the data for them.
Offermatica generally suggests that tests be designed so that
interactions are not a major concern, arguing that quick, simple tests
are more productive than larger, more complicated ones.
But in practice, Offermatica places few constraints on its users. The
system does not limit the number of variables or elements per test. The
same mbox can be used by multiple tests and appear in multiple
locations. Any content can be assigned to any test or mbox. Visitors can
be kept within the same test over multiple visits or not. Users can set
priorities across tests and apply target segments within tests to
control how such conflicts are resolved. This flexibility makes the
system very powerful, though it also opens opportunities for error.
Offermatica account managers and consultants help users interpret their
Because Offermatica controls the mbox content seen by all visitors, it
can do more than testing. One approach is to set rules that deliver
different content based on the visitor’s source, site behavior or
profile. This could, for example, treat existing customers differently
from prospects or make offers related to previous purchases. Another
approach uses “self-optimizing” tests that automatically increase the
proportion of visitors shown the best-performing combination as the test
progresses. Such tests review results every two hours so they can adjust
to changes in user behavior over time. A new offering, “AdBox,” manages
online ads served outside the client’s own site.
Offermatica is sold as a hosted service. Contracts run for one year or
more and range from $5,000 to $25,000 monthly based on the volume of
test visits and staff hours. The original version of the product was
released in 2004, and the company says it has more than 100 active