We’ve lit a candle under the Data Lake — it’s going to take a while until we boil the (Data) Ocean

Data is flowing into the data lake, from social media to clickpaths, from log files to demographics to phone recordings. The dawn of 5G will bring more and more IoT and collaboration data streaming through the data taps.

Cloud processing and storage provide the platform; the demands of businesses to grow and increase efficiency provides the requirement and funding; data science provides the recipe for success at scale — sounds like data utopia … we should be flying but there’s something that is not quite right…

The last decade has seen a seismic shift of focus on regulatory compliance, data protection, cyber security, governance and process. We’ve build controls, rules, systems, programmes and the inbred discipline required to police our businesses and that’s just not compatible with the agility and demands of exploiting huge amounts of data.

It will be about boiling the ocean, understanding the widest needs of the organisation and making sure that integrated systems across the enterprise can satisfy those needs … but … we need to get there over time with agility and experimentation; small step by small step.

Every data lake programme needs to take a leaf out of Geoffrey Moore’s classic —” Crossing the Chasm”… find a niche that will provide quick wins, focus and deliver this without allowing energy to be lost on the bigger picture. Then let the data scientists loose on it to see what other benefits or lessons can be taken from this step. And then do it again and again.

As the volume of the data lake increases over time, improvements in data science, platform capability and governance models will enable businesses to take an evolutionary approach to technology, analytics, governance and outcomes.

And it is this agile and evolving tactic which will warm up your data ocean; enable change; establish a 360-degree view of operations and customers; and modernise operating models and governance … and do so while reducing your storage cost and identifying previously undiscovered opportunities “the pots of gold” within your business data.