Why Data Mesh? ft. Ben Stopford

Streaming Audio: Apache Kafka® & Real-Time Data

With experience in data infrastructure and distributed data technologies, author of the book “Designing Event-Driven Systems” Ben Stopford (Lead Technologist, Office of the CTO, Confluent) explains the data mesh paradigm, differences between traditional data warehouses and microservices, as well as how you can get started with data mesh.

Unlike standard data architecture, data mesh is about moving data away from a monolithic data warehouse into distributed data systems. Doing so will allow data to be available as a product—this is also one of the four principles of data mesh:

1. Data ownership by domain
2. Data as a product
3. Data available everywhere for self-service
4. Data governed wherever it is

These four principles are technology agnostic, which means that they don’t restrict you to a programming language, Apache Kafka®, or other databases. Data mesh is all about building point-to-point architecture that lets you evolve and accommodate real-time data needs with governance tools.

Fundamentally, data mesh is more than a technological shift. It’s a mindset shift that requires cultural adaptation of product thinking—treating data as a product instead of data as an asset or resource. Data mesh invests ownership of data by the people who create it with requirements that ensure quality and governance. Because data mesh consists of a map of interconnections, it’s important to have governance tools in place to identify data sources and provide data discovery capabilities.

There are many ways to implement data mesh, event streaming being one of them. You can ingest data sets from across organizations and sources into your own data system. Then you can use stream processing to trigger an application response to the data set. By representing each data product as a data stream, you can tag it with sub-elements and secondary dimensions to enable data searchability. If you are using a managed service like Confluent Cloud for data mesh, you can visualize how data flows inside the mesh through a stream lineage graph.

Ben also discusses the importance of keeping data architecture as simple as you can to avoid derivatives of data products.

EPISODE LINKS

• Data Mesh 101 course
• Data Mesh 101 with Live Walkthrough Exercise
• Introduction and Guide to Data Mesh
• The Definitive Guide to Building a Data Mesh with Event Streams
• What is Data Mesh, and How Does it Work? ft. Zhamak Dehghani
• Designing Event-Driven Systems
• Watch the video version of this podcast
• Join the Confluent Community
• Learn more with Kafka tutorials, resources, and guides at Confluent Developer
• Live demo: Intro to Event-Driven Microservices with Confluent
• Use PODCAST100 to get an additional $100 of free Confluent Cloud usage (details)

فصل ها

1. Intro (00:00:00)

2. What is a data mesh? (00:01:14)

3. Data domain ownership (00:05:12)

4. The 4 principles (00:09:12)

5. Data governance, availability (00:11:01)

6. Data discovery (00:15:55)

7. How does data mesh relates to event streaming? (00:19:22)

8. Why build a data mesh and how to get started? (00:20:54)

9. Can you build a data mesh with one Kafka cluster? (00:31:37)

10. Operational impacts (00:35:57)

11. Cultural shift (00:39:04)

12. It's a wrap! (00:42:46)