Topological data analysis or TDA if you prefer, is a term you might need to get up to speed on in the near future. For simplicity’s sake we can leave the higher math out of this discussion, but suffice it to say that some bright minds at Stanford and Darpa (the government agency that invented the Internet when Al Gore worked there) spent a lot of free time on the idea.
The best way to get a handle on TDA is to think about Big Data and how inadequate our current tools are for chopping it up and finding interesting nuggets. Current tools are built to help answer specific questions we ask of our data: what’s the relationship between X and Y; how, when, and where do things happen; and stuff like that. Such a process has to touch nearly each piece of data to do its work and that’s time and resource intense. You always seem to have more data than resource, and so a critical need to find a short cut.
Big Data is, well, too big and the corollary is that there’s too much of it coming at us too fast. But also, current technology requires us to ask good questions—no question, no answer. It’s as simple as that. Unfortunately, as data multiplies it becomes more complex because the more of it you have, the more connections and linkages are possible. So, very quickly the amount of data (and its linkages) overwhelm the analysis paradigm we all grew to know and, if not love, to at least respect.
Really Big Data is generated in areas like oil and gas exploration and new drug development. It also grows like weeds in government and financial verticals—basically anywhere you have lots of individuals with tenuous relationships generating lots of data from interactions.
TDA’s approach to the problem is to evaluate the shape of the data monster, which is where the topological part comes in. With a topological approach you don’t have to come up with all the right questions to ask of your data in order to find what’s really interesting in it. TDA makes that first cut almost like a good metal detector would tell you where to dig on the beach. So with TDA you can be reasonably sure that you aren’t leaving something undiscovered in your data that could be valuable.
I spoke with one company in the TDA space recently, Ayasdi, which is just out of a Stanford lab and using some Darpa funding. They tell me their beta process used some 20-year-old data from drug development and you know what? They found more information that had been left undiscovered all these years.
You know where this is going, right? Marketing. Back when we had focus groups and surveys, the data was manageable, we made a few assumptions, asked some questions and generated products, messages, and programs that we broadcast to the world. The good old days. But today we have an avalanche of data coming from social media and crowded markets where we fight for any advantage we can find.
Enter TDA. With it, the pile of social data you may be having trouble storing today will become a fountain of information unique to your company. There’s another name for that too, it’s called intellectual property or IP, and it will make your social data as valuable as the patents, designs, and other proprietary ideas inside your walls.
I figure TDA has some maturing to do and it might be a couple of years before it gets to marketing. A related approach called machine learning seeks to model your best customers and then apply those findings to the universe of potential customers to tell you which ones are the best prospects.
Either way, if you’re at the point where your data is overwhelming you or you just want to know if there’s something you’re missing, help is out there. This is one of the many reasons why Gartner and other firms are saying that the CMO will be spending more on technology than the CIO in a few years. You might want to begin investigating this on your own because a couple of years will go by like nothing.