Big Data: Going Beyond the Landing

Analytics in enterprise is typically highly manual and bespoke.  Select data sets are curated and prepared by analysts and then analyzed or modeled on an as needs basis to produce insights or models.  When models need to be put into ‘production’ then the data preparation steps are carefully replicated by a database programmer.  A lead time of months between model generation and production is a typical industry standard. 

In the world of Big Data data preparation becomes an order of magnitude more complex.  Whilst previously some SAS scripts or SQL queries would suffice, within the Big Data realm the Data Scientist has to deal with a combination of Structured, Semi-Structured and Unstructured sources in a variety of file formats persisted onto a data lake.  Not only are the input sources more varied and complex but the Data Scientist needs to manufacture more attributes or features from these to feed increasingly data hungry algorithms that attempt to draw from every more subtle signals in the data. 

To be successful in this paradigm building pipelines that enable an enterprise to amortize to cost of data preparation across the entire Data Science team becomes critical.  Ensuring that the time to create a model is reduced to a timeframe of relevance, and that it doesn’t take months to take a model from the lab to the factory. 

The next major challenge to applying Big Data to drive personalization is having the ability to deploy data driven strategies to drive front line decisions.  Digital platforms are a relatively easy target.  There the main challenge is identity.  Most traditional enterprises have a wonderful understanding of offline identity of their customers, but joining this to online identity is very challenging.  There is no doubt that we will see this improve as Internet properties continue to get us to engage with their platforms in exchange for content, however, today this is usually available to enterprise in exchange for ad-revenue.  This leaves enterprise with a big challenge in activating individual customer insights on their own domains. 

If Big Data is to continue to add value to enterprise then two key problems that need to be solved is customer identity management and attribute preparation.