Managing your Data as an Asset: The View Ahead

I’ve spent a good part of the last two decades working with large companies on their data. During that time there have been some massive underlying technological and behavioral shifts in how large companies manage and leverage their data as an asset.

In the early 2000’s there were few options for innovative database systems. Most companies were stuck with whatever Oracle, Microsoft, IBM or Teradata gave them. Fast forward ten years and the situation was the opposite – there were hundreds of options of new database systems – from unstructured database systems like MongoDB and Couchbase, to streaming systems like Kafka, and the emergence of Postgres as the replacement of MySQL as the default opensource database of choice. Move into the 2020’s and the advent of cloud native data platforms such as Snowflake and Databricks – as well as the options on GCP (BigTable/BigQuery), AWS (Redshift/RDS) and Azure(Synapse) – have created an opportunity for a new level of productivity within a modern enterprise that wants to use it’s data as a strategic asset.

On the organizational side, the rise of the Chief Data Officer as well as the popularity of Data Science has led to the obvious (in my humble opinion) recognition that there is a massive need to improve the sophistication and discipline of Data Engineering within the enterprise. Most data scientists spend the majority of their time organizing and cleaning data vs. using the data – early surveys reported 80% of time spent on data prep. And it’s generally known that the majority of workloads on DataBricks are not fancy data models – but rather are traditional ETL and data movement workloads. Data Science is basically bottlenecked on great Data Engineering.

I’ve been experiencing these changes over the past two decades as a customer and vendor. In response, I began working on an aspirational experience for mastering data as an asset in the enterprise. At Tamr we have expressed many of these design patterns into our product and are seeing tremendous success on the ground as we help our customers clean up their data one business entity at a time.

We are also looking out 2-3 years and designing the aspirational experience for modern data organizations. I have a prototype for this aspirational experience and am anxious to get feedback from the best and most innovative Chief Data Officers in the world. If you are interested in checking out this prototype, I’d be happy to personally walk you through it and get your feedback. Just drop me an email at andy.palmer@tamr.com and I’d be happy to find a time that works for you.

‍