There is no dearth of data today. Every Internet channel of worth – from Google to Twitter to Facebook – is collecting sh*t loads of it. Then there are all these analytics vendors who are doing their own crawling. But what good is all that data if there are no insights from it?
Zach Hofer-Shall from Forrester considers these questions on his blog. He rightly suggests that before diving into the ocean of data consider what you might be able to do with all that data. Will it turn up insights that are valuable? Are those who intend to use this data even prepared to be bombarded with the never-ending flow of real time data?
But there’s a bigger question at hand: what will these vendors do with the firehose? Back to my rhetorical question before: what good is a plethora of data if you can’t find what you’re looking for? With more data comes increased importance of text analysis. Being able to pull in 65 million tweets a day is great in theory, but requires vast amounts of processing and analysis to actually turn it into something of value. When someone tweets about what they had for lunch or I tweet about a soccer game I’m watching, these irrelevant items build into the countless others that make up that firehose.
Having worked in the area of analytics I can vouch for what he says. Whatever channel you choose, there is an overwhelming amount of data that is available. But all that is loaded with lots of noise. Extracting meaningful analysis out from it is the big challenge. Furthermore if you are analytics vendor there is the added challenge of presenting all the insights and enabling your customer to use those insights in a way that is trivial.