Marketers have always known that transactions are the most powerful data for predicting customer behavior. But acting on this knowledge is hard: Transaction data often are inaccessible; when accessible, the volumes can be overwhelming; when resources exist to manage the volume, it is difficult to make sense of the details.
As a result, marketers often have settled for summary measures such as recency-frequency-monetary value cells, or still simpler segmentation techniques. But today's systems make it easier to share transaction data, and they can handle huge volumes at acceptable cost. As these preliminary hurdles disappear, marketers face the final challenge: finding something useful to do with all the information they now have available.
Some applications are simple enough. Sending a welcome kit to new customers or a thank-you note after each order requires little information outside the current transaction itself. But marketers want to look past individual transactions to patterns that indicate threats or opportunities. Yet even something as obvious as looking for a decline in transaction volume — often a precursor to customer attrition — turns out to pose technical difficulties. The brute force approach, of continually reading the detail data to create summary measures, requires more computer power than even today's technology makes economically acceptable. Moreover, writing and running such queries takes computer skills that most marketers lack. And linking such analysis to appropriate marketing responses takes another unwieldy round of systems integration.
These problems have not gone unnoticed by software entrepreneurs. Several products have appeared in recent years to address these issues. They are often described as “state-based” systems, meaning that they assign customers to various “states,” such as satisfied or dissatisfied, and flag customers when they change from one state to another.
Of course, nearly any segmentation scheme could be described in terms of states, and nearly any marketing system could react when customers migrate from one segment to another. But the state-based systems share several features that distinguish them from their more generic cousins:
· Tracking complex behaviors. The truly interesting customer behaviors turn out to be complicated things like a change in transaction frequency or deviating from a pattern of weekly deposits or a sequence of service problems followed by complaints. Describing such behaviors with conventional database structures and query languages can be difficult or impossible. State-based systems provide special tools to streamline the process.
· Efficient data storage. Constantly searching for patterns in raw transaction data is not practical when volumes get large. Nor does it provide quick response when evaluating the significance of a new transaction. State-based systems use specialized storage methods that summarize past transactions without losing the detail needed to fit new transactions into the patterns. They nearly always pack this data into a single customer record that is quickly accessible for real-time processing.
· External connections. Today's state-based systems are designed to accept transactions from multiple sources and feed responses back to multiple channels. This makes them highly adaptable and allows them to complement rather than compete with existing touch points, marketing databases and campaign managers. Still, there is nothing inherent in state-based technology that requires this flexibility. Over time, it is conceivable that state-based methods will be embedded in operational systems like call centers or Web servers. They may even turn up as functions in statistical scoring systems, particularly the real-time versions connected directly to transaction data.
Elity InSight (Elity Systems, 732/564-1850, www.elity.com) fits squarely into the category of state-based systems, though the vendor does not use the term. The system reads streams of transaction data, picks out transactions that identify user-defined behavior patterns and executes specified actions when a pattern occurs. All this is organized in a framework of nested rules: A top-level rule might describe a business policy such as “place a phone call when a high-value customer significantly reduces her transaction frequency,” while subsidiary rules would define the terms “place a phone call,” “high-value customer” and “significantly reduces transaction frequency.” These rules themselves may draw on still other rules to define their own components. Lower-level rules can be shared, easing rule development and encouraging consistency.
Rules are constructed by filling out pre-defined forms with variables such as data elements, logical operators, calculations and time intervals. Standard forms let users define patterns, trends, simple relationships, mathematical formulas and inactivity. Action rules such as “place a phone call” would define a text message, data elements, format and destination system.
Data elements are mapped into the system by technical staff. Elements include transaction attributes such as date, amount and type, and can include transactions in different formats from different systems. The transactions can be loaded into the system in a batch process or fed individually in real time. The system uses XML as its primary data language but can also load relational database tables and flat files directly. A rule can also query an external database, although this slows performance.
InSight evaluates each transaction to determine whether it applies to any rule that affects the customer. If so, the customer's record is updated with the relevant information only. This is a subset of all the information in the transaction: For example, if a rule is counting transactions in a given time frame, the system will record the transaction date but not its amount. The result is a single, relatively compact customer record that is stored as a single row on a conventional relational database. For efficiency, the contents are placed in a binary format and are dropped when no longer needed to track an active rule. But InSight also copies information into a historical database that maintains a more complete record. This can be accessed directly by the system's own reporting tools or by third-party tools via an application program interface or API. Elity is also developing a viewer to read the customer records directly.
InSight was first introduced in 1999, which makes it relatively mature for a system of this type. This shows most clearly in administrative functions, which include rule- and role-based security, reports on rule utilization, automated checking for invalid or circular rule references, an option to limit the total number of outputs a rule generates, start and end dates on rules and a separate environment to test new rules against sample data. However, the system does not include structures to capture revenue, cost or responses for rules. It also does not allow random splitting within a rule, has limited prioritization capabilities when multiple rules apply and lacks integrated statistical modeling. Nor can one rule manage a sequence of actions and responses, although a “follow-up” function lets users do this by linking separate rules. Several of these features are planned for future development.
Another measure of the system's maturity is its proven scalability: The largest client processes more than 1 million transactions for 10 million customers in about one hour each night. The system has about six total installations, mostly in financial services and telecommunications. Pricing is based on the number of processors in the server and is about $140,000 per processor. InSight runs on Windows NT/2000 servers and uses a Web browser on user workstations.
