Current Trends in DataOps: Survey Results

As enterprises continue to accumulate data at increasingly large volumes and speeds, they are looking for ways to effectively and efficiently leverage that data to drive value for the organization. This has led to the emergence of DataOps–an automated, process-oriented methodology used by analytic and data teams to improve the quality and reduce the cycle time of data analytics. 

The value of the DataOps infrastructure is that it enables enterprises to close the gap between the potential value of their vastly siloed legacy data and the consistent realization of the data’s value by all the people in their organization. Building on the wisdom of the Agile process and borrowing tenets from the long-practiced discipline of DevOps, DataOps focuses the entire organization on timely delivery of working data.

Implementing an infrastructure like DataOps requires a lot of changes–across people, processes and, of course, technology. But for many organizations, the discipline is becoming necessary. In particular, you can tell that your organization needs DataOps if:

• Your data team is in full burnout mode, because they’re being inundated with too many minor request tickets.
• Business users don’t understand why it takes so long to get data, and even when they do get it, they often don’t trust it because the data contains too many errors.
• Data analysts write the same jobs and reports with minor variations.
• Data scientists wait for months for data and computing resources.
• Your organization has started self-service initiatives, but this strategy has spawned hundreds of data silos.

With so many enterprises facing the challenges that come with Big Data, we wondered: how many organizations have adopted DataOps, and how successful have they been? Answering this question was part of the purpose of a survey of 175 BI Directors and Managers conducted in April 2019 by the Eckerson Group. The following is an overview of some of the key results, but you can also download the full report here.

DataOps Adoption and Challenges

Despite the obvious and growing need for a discipline like DataOps across many enterprises, a majority of companies have yet to really embrace the practice, with 43% of survey respondents saying their organization currently has no DataOps initiative.

Does your organization have a DataOps initiative?

This is likely due to a number of key factors. To begin with, as companies that have already initiated DataOps processes at their organization can attest to, implementing an infrastructure like this is not easy. As one survey respondent put it:

“Make sure your team is prepared to go slow before it can go fast. Less regular work is going to get done. Tools need evaluating and learning. Processes need research and documenting. Things will speed up, but it takes time.”

As we have discussed previously, sometimes the people in an organization are a bigger bottleneck than the technology when it comes to implementing new infrastructures and processes. This can be attributed to a number of factors, but often resistance to change can be very strong at first.

Other challenges to implementing DataOps that were commonly noted by respondents are outlined in the chart below:

DataOps Principles and Benefits

At Tamr, we’ve talked a lot about the key principles of a DataOps ecosystem and the importance of each. This survey also took a look at what respondents view as the most important principles or characteristics of a successful DataOps implementation. The responses included:

• Agile development (58%)
• Continuous delivery (54%)
• Continuous integration (53%)
• Collaboration and reuse (50%)
• Code repositories (50%)
• Data pipeline orchestration (46%)
• Performance and application monitoring (46%)
• Continuous testing (41%)

Despite the challenges of implementing DataOps, survey respondents agree that the benefits to this type of infrastructure are numerous. The biggest benefit was that of ‘faster cycle times’ selected by 60% of respondents. Also related to this benefit were:

• Delivering new applications more quickly (50%)
• Ingest new data sources more quickly (48%)
• Faster change requests (47%)

Clearly, survey respondents acknowledge that DataOps, at its core, offers the ability to improve data-related bottlenecks that have plagued most enterprises for years. Whether or not enterprises are ready to fully embrace DataOps to implement the needed changes to tackle these challenges head on, however, is another discussion.

To read the full results from the Eckerson Group survey, download the report below.

‍