Transforming Data Management: Insights from Anthony Deighton, CEO of Tamr

Summary:

• AI is revolutionizing data management, addressing challenges improving data quality, especially at scale.
• AI-native data mastering creates golden records and improves data integrity which, together, improve business performance.
• AI technologies like machine learning and LLMs play a crucial role in data quality, MDM, and entity resolution.
• Tamr uses AI to automate data curation and provide trusted, high-quality data for better decision-making.

Stephen Hawking once said "AI is not just another tool; it’s the next stage of evolution for humanity’s capacity to solve problems." To me, these words capture the essence of the seismic shift happening in the world of data management. As organizations worldwide continue to grapple with the exponential growth of data, many are finally realizing that traditional methods of data management no longer suffice. They must shift their approach to one that embraces new, modern approaches to data management.

That's where artificial intelligence (AI) comes into play. AI is proving to be a transformative force in the way organizations worldwide approach data management. Through the automation of common data management tasks, AI is redefining how companies approach data and transforming the ways in which they manage, improve, and generate business value from it.

The #1 Challenge with Enterprise Data

Organizations have more data than ever before. And many are finally recognizing the importance of harnessing the power of their data to grow the business and gain competitive edge. But despite advances in BI and analytics, data visualization, and data access, one fundamental issue remains: bad data.

According to a study by Harvard Business Review, only 3% of data, on average, meets data quality standards, while Gartner reports that poor data quality costs organizations, on average, $12.9 million. In Wavestone's 2024 Data and AI Leadership Executive Survey, it's clear that bad data remains a challenge - 63% of respondents indicate they struggle with data quality while even fewer - only 37% - indicate they've made progress in improving the quality of their data.

Messy data is a pervasive problem. And for many, the issue stems from the inability to scale. Imagine for a moment that your task is to compare 10 million customer records to find the duplicates. Doing so would require you to make 50 trillion comparisons, a task that is extremely difficult and incredibly time-consuming with traditional data management techniques.

To say this N-squared problem is overwhelming is an understatement. The good news, however, is that AI can tackle this scale problem effectively. Using AI-native data mastering, organizations can finally address this seemingly impossible challenge by employing AI to identify entities within the data and unify them into golden records that represent the best version of the entities across disparate systems and sources.

The Evolution of AI and its Role in Data Management Transformation

Reflecting on insights from Tamr co-founder Mike Stonebraker, it’s clear that machine learning was always poised to transform data management. After all, AI is capable than doing more than just powering autonomous vehicles or delivering your Amazon order via a drone. Using AI, organizations can resolve key business entities into single, unified views called golden records.

Golden records represent the peak of data management: a single, authoritative, accurate version of a business entity’s data across multiple data sources and datasets. This hard-to-achieve level of data integrity and comprehensiveness is only possible because of AI. By employing AI to create golden records, companies can automate and streamline data management, leading to more accurate insights, improved decision-making, and stronger business performance.

In the past, “golden records” had a bad rap because traditional, rules-based master data management (MDM) projects have high failure rates. Not only do traditional MDM solutions demand that humans manage tedious, large-scale tasks, but they're also built for static data and therefore cannot scale. AI-native data mastering, on the other hand, uses AI to automate these tasks, handling 95-99.9% of data curation and leaving interesting or complicatededge cases to humans.

Connecting Data Quality, MDM, and AI

Data quality, MDM, and entity resolution all aim to identify the best version of data within an enterprise. Whether it’s customer data, supplier data, or product data, the objective is to curate and merge data from various, disparate internal and external sources to provide a trusted, unified view. And AI plays a pivotal role in this process.

AI encompasses a broad range of technologies. Machine learning, a subset of AI, and large language models (LLMs) work well together. LLMs excel at text generation and extraction, making them valuable for tasks like summarizing customer data or generating emails. However, they aren’t suited for entity resolution, which requires specialized machine learning techniques. Combining these tools under the AI umbrella enables comprehensive data management solutions.

AI and the Role of the Chief Data Officer (CDO)

A promising innovation in AI that could disrupt the CDO role in a positive way is retrieval augmented generation (RAG), a method used in natural language processing (NLP) that combines the strengths of retrieval-based and generative models. Using Tamr’s virtual chief data officer capability, driven by RAG, AI can answer routine questions, freeing up CDOs to focus on more strategic work.

However, despite its potential, the adoption of RAG remains in the early stages because of the cutting-edge nature of the technology and the current focus on experimenting with large language models.

Tamr: Using AI to Transform Data Management

Founded in 2013, Tamr is based on Turing award winner Mike Stonebraker’s research at MIT’s Computer Science and AI Lab. Tamr's mission is to empower companies to unify and master the expansive and constantly shifting landscape of highly-diverse data. At the forefront of using AI to improve the quality and trustworthiness of data, Tamr aims to deliver the perfect fusion of AI and human intellect, enabling businesses to uncover new insights from their data so they can work smarter and reach new levels of success.

Tamr's AI-first approach to data mastering and entity resolution solves the MDM problem. Unlike primitive, rules-based tools of the past, Tamr uses advanced AI to automatically identify and resolve inconsistencies across data sources. As a result, businesses can finally deliver the golden records every stakeholder needs to realize value from their data.

By automating tedious tasks and providing trusted data, Tamr enables organizations to make better decisions and focus on strategic goals. And as AI technology evolves, its role in enhancing data quality and management will only grow, promising a future where clean, reliable data is the norm rather than the exception.

To see for yourself how Tamr's AI-native data mastering can transform data management to help your organization deliver the high-quality data needed to drive better decisions, please book a demo.