DataOps: A Unique Moment in Time for Next Generation Data Engineering

Judging from the engagement on my latest post from across the spectrum of business leaders, data engineers, data scientists and thought leaders, DataOps is clearly picking up steam. There were too many comments for me to respond to them all on LinkedIn, so I decided to post this follow up.

My overall takeaway? The sense of optimism that people are feeling about next-generation data engineering in the enterprise. It finally feels like large organizations have embraced their “data debt” and are figuring out how to monetize their data. It helps when they realize that great analytics depend on great data.

We’re at a unique point in time when fundamental changes in data management in the enterprise, the low barrier to cloud migration and the volume and value of enterprise data combine into a massive opportunity to create next generation data engineering pathways.

Figure 1: A unique moment in time for Data as an Asset

One of the foundational ideas behind DataOps, and the reason Mike and I started Tamr, is that enterprise technology and the people responsible for data have finally reached a place where companies that are not data-native can start achieving the types of analytical outcomes that have allowed Google, Facebook and Amazon to sprint so far ahead of the competition. These companies achieved their primacy because they could devote vast amounts of technical resources to solving the problems of software and data engineering at scale when it was a seemingly impossible task. DataOps technologies and processes have the potential to bring Google/Amazon-level data engineering to rest of us in the Global 2000.

Below, I’ve taken a few threads from comments I received and addressed them. I love the discussion — so let’s keep it up!

Starting with outcomes

A common point was expertly summed up: “putting [this] into practice will be a long journey, and will need to bring cultural change.” This is right on the nose. While developing a data-driven engineering infrastructure will take organizations a long time, getting started won’t. Building a foundation for the repeatable delivery of production-ready data starts with a single use case, and is often preceded by shockingly simple but shockingly hard-to-answer questions from business users: “How many customers do we have?” “How much do we pay each of our suppliers?” Answering these questions well is the best way to get started on DataOps. By attacking the question not as a one-and-done project, but as an analytic that should be readily available to many users at any point in time, the principles of DataOps necessarily come into focus. You can’t answer that question for many users over time without integrating data from many different sources, without involving users and without automation. DataOps will be successful when driven by use cases with clear business value — whether it’s generating more revenue through cross-sell and upsell, preparing for GDPR or finding savings in the supply chain. The success of these analytics is addictive; once an organization is successful for one project; the next most important use case is immediately apparent.

Isn’t DataOps just a new marketing name for an old trend?

This was a common and valid comment. As someone who has been on both sides of the CIO/vendor table for over three decades, I’ve been in the midst of every new attempt at solving this problem. My partner Mike gives a great talk on the generations of data curation.

DataOps is a discipline that allows organizations to repeatedly create production-ready data from a wide variety of sources for a wide variety of use cases. There are two reasons why DataOps is materially different than the generations of data curation which came before it. The first differentiator is the is the ability to create bi-directionality between end users at the point of consumption into the data models themselves. The key to repeatability is the tight coupling of feedback from users at the point of consumption into the creation of the data models themselves. The quintessential example, in my mind, is Google Maps. Google keeps informational incredibly up-to-date simply by asking a few users to add a few data points. Integrating this feedback into its architecture is at the core of why Google Maps is so good and constantly getting better. When we’re setting up next-generation data engineering processes, we have to keep this bi-directionality in mind; it’s the only way to scale data curation. The technology is there to allow closely-coupled integration between data production and consumption, and coupling them allows repeatability and scale.

The second differentiator is automation. The use of technologies like machine learning on very tactical but predictable chores like transformations or schema-mapping allow DataOps-ground data engineering pipelines to do the vast majority of work — upwards of 95% — without human intervention; saving the tough work for humans. Combined with scale-out computing and the easy availability of cloud resources, we are at a place where the rest of us can get a handle on the crazy amounts of siloed enterprise data.

Data Quality: Cold War or Overt Conflict within your organization?

At the heart of the problem is an age-old battle — the desire of IT folks to not let systems and data get out of hand, and the desire of business users to have useful, efficient operations with the data they need to get the job done. Business process automation has been spewing out data for decades; until recently, we thought of this as as a type of exhaust. As business users realize how valuable that data is, the desire to treat data as an asset has highlighted just how messy enterprise data systems are. DataOps projects need to acknowledge that business users need some freedom to operate — but that data needs to be handled and treated like an asset. Engineering for these realities, rather than ignoring them, is a key component of DataOps. A great example comes from my friends at Novartis Institute for Biomedical Research, who realized that while they wanted to glean insights from the tens of thousands of experiments, forcing data entry standards on hundreds of scientists was a non-starter. Instead, they had to adjust their data engineering to accommodate how scientists actually produce data, and use the machine to prepare the data how the business wants to use it.

It some companies, this tension is overt; in others, it’s a cold war. In these cases, technological changes should be deployed to facilitate cultural changes. In every DataOps project I’ve been a part of, the people side of things is much harder than the technology side. Giving business users a taste of the possible analytical outcomes from having unified, trusted data givens them a foundation of trust to give up data hoarding and political games. It’s tough, but integrating users into the process as experts, and seeing the results of their feedback improve the data they have access to, is a powerful tool for detente.

Isn’t another system for copying data just adding to the problem?

Data isn’t code; as one commenter mentioned, the instant you copy data it becomes corrupted. This is why Master Data Management, for example, failed to reach its promise. Again, DataOps is about acknowledging reality. Data is going to multiply, duplicate and live where it’s not suppose to. A common system of reference — a unified data hub — tackles this problem by allowing the data models to be flexible enough to accommodate duplication and resolve uncertainties by incorporating experts for data curation. Only processes that acknowledge and account for how data is produced and used have any hope of success. The alternatives have all been tried: imposing standards, mastering data, system consolidations, etc. They are necessary but insufficient, because as soon as they become cumbersome to business users, those users ignore them, circumvent them or procure their own, new system.

DataOps done poorly will just result in another competing golden record. This happens when the tools and process takes primacy over results. When DataOps projects are focused on driving transformational and repeatable analytical outcomes, users will clamor to gain access for their own use cases. I’ve seen this virtuous cycle over and over again.