Looks like the publisher may have taken this series offline or changed its URL. Please contact support if you believe it should be working, the feed URL is invalid, or you have any other concerns about it.
با برنامه Player FM !
پادکست هایی که ارزش شنیدن دارند
حمایت شده
Rerelease of #65 What's a Data Contract Between Friends - Setting Expectations with Data Contracts - Interview w/ Abe Gong
بایگانی مجموعه ها ("فیدهای غیر فعال" status)
When? This feed was archived on February 08, 2025 14:08 (
Why? فیدهای غیر فعال status. سرورهای ما، برای یک دوره پایدار، قادر به بازیابی یک فید پادکست معتبر نبوده اند.
What now? You might be able to find a more up-to-date version using the search function. This series will no longer be checked for updates. If you believe this to be in error, please check if the publisher's feed link below is valid and contact support to request the feed be restored or if you have any other concerns about this.
Manage episode 376793528 series 3293786
Due to health-related issues, we are on a temporary hiatus for new episodes. Please enjoy this rerelease of episode #65 with Abe Gong all about how people are implementing data contracts in the wild. There are so many ways people can just do only defensive data contracts and I think that is such a missed opportunity. Maybe it's where you will have to start but there's a much better way and we talk a bit about why I think that is so distressing that people aren't talking to each other.
Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/
Please Rate and Review us on your podcast app of choice!
If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here
Episode list and links to all available episode transcripts here.
Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.
Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here
Abe's Twitter: @AbeGong / https://twitter.com/AbeGong
Abe's LinkedIn: https://www.linkedin.com/in/abe-gong-8a77034/
Great Expectations Community Page: https://greatexpectations.io/community
In this episode, Scott interviewed Abe Gong, the co-creator Great Expectations (an open source data quality / monitoring / observability tool) and co-founder/CEO of Superconductive.
One caveat before jumping in is that Abe is passionate about the topic and has created tooling to help address it. So try to view Abe's discussion of Great Expectations as an approach rather than a commercial for the project/product.
To start the conversation, Abe shared some of his background experience living the pain of unexpected upstream data changes causing data chaos / lots of work to recover from and adapt. Part of where we need to get to using something like data contracts is to remove the need to recover in addition to adapting and move towards controlled/expected adaptation. Abe believes that the best framing for data contracts is to think about them as a set of expectations.
To define expectations here, this would include not just schema but also the content of data, such as value ranges/types/distributions/relationships across tables/etc. So for instance, a column may be a one to five for rankings and then the application team changes it one to 10. The schema may not be broken - it is still passing whole numbers - but the new range is not within expectations so the contract is broken.
At current, Abe sees the best way to not break social expectations is via getting consumers and producers in a meeting to talk about the upcoming changes and prepare, such as with versioning. But, as tooling improves, Abe sees a world where we won't even need a lot of those meetings going forward - either because data pipelines can be "self-healing" and automatically adapt to changes upstream or because metadata and tools for context-sharing will reduce the need for meetings.
Abe sees two distinct use cases in general for data contracts or more specifically how people are using Great Expectations to implement data contracts. The first is purely defensively - put some validation on the data you are ingesting to prevent data that doesn't match from blowing up your own work; the second type is when the consuming team shares their expectations with the producers and there is a more formal agreement - or contract - with a shared set of expectations. The first often leads to the second, via an agreement conversation that happens after there was an upstream breaking change.
Abe also mentioned there is a third constituent on data contracts in the room: the data. Sometimes the consumers and producers may agree on what they expect, but if that’s different than what’s in the actual data, then it's hard or dangerous to move forward. The data has a veto.
There was an interesting discussion on the push versus pull of data contracts - should the producer team create an all-encompassing contract or should we have consumer-driven contracts? Would producer-driven contracts be too restrictive, preventing the serendipity insights data mesh aims to produce? Would consumer-driven contracts mean multiple contracts for each data product that the producer agrees to? Is that sustainable?
So, to sum it up, the idea of a set of explicit expectations around a data product that are the result of collaboration between producers and consumers sounds like where we should all head if possible. If the expectation set is only coming from the producer side, it might be overly restrictive and miss a lot of the nuance necessary to actually create consumer trust. And exclusively consumer-driven contracts don't sound sustainable or scalable.
Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/
If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/
If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here
All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf
422 قسمت
بایگانی مجموعه ها ("فیدهای غیر فعال" status)
When?
This feed was archived on February 08, 2025 14:08 (
Why? فیدهای غیر فعال status. سرورهای ما، برای یک دوره پایدار، قادر به بازیابی یک فید پادکست معتبر نبوده اند.
What now? You might be able to find a more up-to-date version using the search function. This series will no longer be checked for updates. If you believe this to be in error, please check if the publisher's feed link below is valid and contact support to request the feed be restored or if you have any other concerns about this.
Manage episode 376793528 series 3293786
Due to health-related issues, we are on a temporary hiatus for new episodes. Please enjoy this rerelease of episode #65 with Abe Gong all about how people are implementing data contracts in the wild. There are so many ways people can just do only defensive data contracts and I think that is such a missed opportunity. Maybe it's where you will have to start but there's a much better way and we talk a bit about why I think that is so distressing that people aren't talking to each other.
Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/
Please Rate and Review us on your podcast app of choice!
If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here
Episode list and links to all available episode transcripts here.
Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh.
Transcript for this episode (link) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here
Abe's Twitter: @AbeGong / https://twitter.com/AbeGong
Abe's LinkedIn: https://www.linkedin.com/in/abe-gong-8a77034/
Great Expectations Community Page: https://greatexpectations.io/community
In this episode, Scott interviewed Abe Gong, the co-creator Great Expectations (an open source data quality / monitoring / observability tool) and co-founder/CEO of Superconductive.
One caveat before jumping in is that Abe is passionate about the topic and has created tooling to help address it. So try to view Abe's discussion of Great Expectations as an approach rather than a commercial for the project/product.
To start the conversation, Abe shared some of his background experience living the pain of unexpected upstream data changes causing data chaos / lots of work to recover from and adapt. Part of where we need to get to using something like data contracts is to remove the need to recover in addition to adapting and move towards controlled/expected adaptation. Abe believes that the best framing for data contracts is to think about them as a set of expectations.
To define expectations here, this would include not just schema but also the content of data, such as value ranges/types/distributions/relationships across tables/etc. So for instance, a column may be a one to five for rankings and then the application team changes it one to 10. The schema may not be broken - it is still passing whole numbers - but the new range is not within expectations so the contract is broken.
At current, Abe sees the best way to not break social expectations is via getting consumers and producers in a meeting to talk about the upcoming changes and prepare, such as with versioning. But, as tooling improves, Abe sees a world where we won't even need a lot of those meetings going forward - either because data pipelines can be "self-healing" and automatically adapt to changes upstream or because metadata and tools for context-sharing will reduce the need for meetings.
Abe sees two distinct use cases in general for data contracts or more specifically how people are using Great Expectations to implement data contracts. The first is purely defensively - put some validation on the data you are ingesting to prevent data that doesn't match from blowing up your own work; the second type is when the consuming team shares their expectations with the producers and there is a more formal agreement - or contract - with a shared set of expectations. The first often leads to the second, via an agreement conversation that happens after there was an upstream breaking change.
Abe also mentioned there is a third constituent on data contracts in the room: the data. Sometimes the consumers and producers may agree on what they expect, but if that’s different than what’s in the actual data, then it's hard or dangerous to move forward. The data has a veto.
There was an interesting discussion on the push versus pull of data contracts - should the producer team create an all-encompassing contract or should we have consumer-driven contracts? Would producer-driven contracts be too restrictive, preventing the serendipity insights data mesh aims to produce? Would consumer-driven contracts mean multiple contracts for each data product that the producer agrees to? Is that sustainable?
So, to sum it up, the idea of a set of explicit expectations around a data product that are the result of collaboration between producers and consumers sounds like where we should all head if possible. If the expectation set is only coming from the producer side, it might be overly restrictive and miss a lot of the nuance necessary to actually create consumer trust. And exclusively consumer-driven contracts don't sound sustainable or scalable.
Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/
If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/
If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here
All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm, MondayHopes, SergeQuadrado, ItsWatR, Lexin_Music, and/or nevesf
422 قسمت
همه قسمت ها
×
1 Summer Hiatus Announcement - Back in August 4:28

1 #306 Building with People for People - Swisscom's Data Mesh Approach and Learnings - Interview w/ Mirela Navodaru 1:09:06

1 #305 Combining the Technical and Business Perspectives for Data Mesh - Interview w/ Alyona Galyeva 1:05:59

1 #304 Getting Your Data Mesh Journey Moving Forward - Interview w/ Chris Ford and Arne Lapõnin 1:01:50

1 #303 Delivering What Matters - Value - Through Strong Business Collaboration - Interview w/ Saba Ishaq 1:10:37

1 No Episode This Week 1:31

1 #302 Finding and Delivering on a Good Initial Data Mesh Use Case - Interview w/ Basten Carmio 1:11:47

1 #301 Learnings From 25+ Years in Data Quality - Interview w/ Olga Maydanchik 1:01:57

1 #300 Panel: How to Treat Your Data Platform as a Product - Led by Michael Toland w/ Sadie Martin, Marta Diaz, and Sean Gustafson 1:03:01

1 #299 Empowering Development with Actionable Data - Interview w/ Carol Assis and Eduardo Santos 1:13:01

1 #298 Effective Partnering With Business Execs - Learnings from Another Data Mesh Journey - Interview w/ Jessika Milhomem 1:07:39

1 #297 Panel: Understanding and Leveraging the Data Value Chain - Led by Marisa Fish w/ Tina Albrecht, Karolina Stosio, and Kinda El Maarry, PhD 58:10

1 #296 Patience in Product Thinking in Data - Building to Large-Scale Behavior Change - Interview w/ Darren Wood 1:02:58

1 #295 Data Shouldn't be a Four-Letter Word - Making Data a Forethought - Interview w/ Wendy Turner-Williams 1:16:25

1 #294 Panel: Product Discovery and Data Discoverability in a Data Mesh World - Led by Ecem Biyik w/ Frannie Helforoush, Marta Debska-Barcinska, and Ole Olesen-Bagneux 1:03:15

1 #293 Adapting Product Management to Data - Finding the Customer Pain and the Value - Interview w/ Amritha Arun Babu Mysore 1:05:31

1 #292 Aligning Your Data Transformation to the Business - Interview w/ Nailya Sabirzyanova 1:05:30

1 #291 Panel: Data as a Product in Practice - Led by Jen Tedrow w/ Martina Ivaničová and Xavier Gumara Rigol 1:01:48

1 #290 Applying Platform Engineering Best Practices to Your Mesh Data Platform - Interview w/ Tom De Wolf 1:05:45

1 #289 Building the Right Foundations for Generative AI - Interview w/ May Xu 51:26

1 Major Programming Announcement 4:25

1 #288 Panel: Master Data Management in a Data Mesh World - Led by Ole Olesen-Bagneux w/ Liz Henderson, Piethein Strengholt, and Samia Rahman 1:04:59

1 #287 Driving Data Value Through Creativity, Curiosity, Collaboration, and Communication - Interview w/ Tiankai Feng 56:16

1 #286 Mastering Master Data Management in a Modern World - Interview w/ Sue Geuens 54:07

1 #285 Getting Depth and Value From Generative AI - In Data Mesh and in General - Zhamak's Corner 33 19:15

1 #284 Breaking Down the Monolith - Incentivizing Good Choices - Interview w/ Frederik Nielsen 1:03:13

1 #283 Selling Data Mesh to Your C-Suite and Board - Mesh Musing 58 20:05

1 #282 Not Sweating the Small Stuff in Data Mesh - Interview w/ Mandeep Kaur 1:15:54

1 Weekly Episode Summaries and Programming Notes – Week of December 31, 2023 14:39

1 #281 Panel: Data Contracts and Data Mesh - Led by Jean-Georges Perrin w/ Amy Raygada and Andrew Jones 1:06:28

1 #280 Enabling Your Domains to Create Maintainable Data Products - Interview w/ Alexandra Diem, PhD 59:40

1 Weekly Episode Summaries and Programming Notes – Week of December 24, 2023 28:06

1 #279 Driving Towards a Cohesive Developer Experience - At the Expense of Snowflake and Databricks? - Zhamak's Corner 32 18:40

1 #278 Data Contracts for the Rest of Us - Approaching Contracts in Evolving Companies - Interview w/ Ryan Collingwood 1:19:26

1 Weekly Episode Summaries and Programming Notes – Week of December 17, 2023 14:41

1 #277 Mesh Momentum Versus Value - What to Choose When and Why - Mesh Musings 57 13:30

1 #276 Making Self-Service Actually Work Well Safely - Interview w/ Kate Carruthers 1:02:55

1 Weekly Episode Summaries and Programming Notes – Week of December 10, 2023 15:18

1 #275 Panel: Why Data Mesh Needs Digital and Org Transformation - Led by Benny Benford w/ Nailya Sabirzyanova, Iulia Varvara, and Stefan Zima 1:05:41

1 #274 Your Data Platform is a Product, Treat it Like One! - Interview w/ Sean Gustafson 1:01:32

1 Weekly Episode Summaries and Programming Notes – Week of December 3, 2023 23:41

1 #273 An API-First World in Data Integration - An Actual Modern Data Stack - Zhamak's Corner 31 22:52

1 #272 Understanding and Valuing Your Organization's Data - Interview w/ Lauren Cascio and Chris Ensey 55:46

1 Weekly Episode Summaries and Programming Notes – Week of November 26, 2023 14:30

1 #271 The Importance of Repeatability of Language to Scalability - Mesh Musings 56 11:22

1 #270 Sustainable Data Transformation to Drive Towards Data Mesh - RBI's Journey So Far - Interview w/ Stefan Zima 1:13:44

1 Weekly Episode Summaries and Programming Notes – Week of November 19, 2023 13:03

1 #269 Panel: Leading a Data Mesh Implementation (2nd Iteration) - Led by Vanessa Eriksson w/ Stefan Zima, Duncan Cooper, and Sid Shah 1:03:53

1 #268 Adapting to and Adopting Product Thinking - Transforming Your Org for Sustainable Data Mesh - Interview w/ Iulia Varvara 56:02

1 Weekly Episode Summaries and Programming Notes – Week of November 12, 2023 23:30

1 #267 The Developer Experience - How Do We Delight Data Developers? - Zhamak's Corner 30 19:49

1 #266 Leveraging Decades of Information Architecture Learnings to Do Data Well - Interview w/ Akins Lawal 1:03:39

1 Weekly Episode Summaries and Programming Notes – Week of November 5, 2023 11:45

1 #265 Are You (Even) Doing Data Mesh™? - Mesh Musings 55 17:03

1 #264 Will GenAI and Data Mesh Really Mix? - Interview w/ Madhav Srinath 51:09

1 Weekly Episode Summaries and Programming Notes – Week of October 29, 2023 14:53

1 #263 Panel: Applying Site Reliability Engineering Practices to Data - Led by Emily Gorcenski w/ Amy Tobey and Alex Hidalgo 59:22

1 #262 Setting the Groundwork to Become Data Driven - Interview w/ Corrin Shlomo Goldenberg 1:06:52

1 Weekly Episode Summaries and Programming Notes – Week of October 22, 2023 27:17

1 #261 Just What the Heck is a Data Product Container? - Zhamak's Corner 29 20:37

1 #260 Driving the Big Picture Forward - More on Northern Trust's Data Mesh Implementation - Interview w/ Jimmy Kozlow 1:13:56

1 Weekly Episode Summaries and Programming Notes – Week of October 15, 2023 21:06

1 #259 The Fellowship of the Data Mesh: The Journey to Meshtopia - Mesh Musings 54 14:50

1 #258 Data Mesh on Hard Mode: Learnings From Airtel's Early Data Mesh Journey - Interview w/ Sid Shah 1:16:15

1 Weekly Episode Summaries and Programming Notes – Week of October 8, 2023 21:34

1 #257 Panel: Doing Data Mesh Data Governance Well - Led by Andrew Sharp w/ Nicola Askham, Kinda El Maarry, PhD, and Jay Como 1:09:24

1 #256 How to Drive Towards a Data-Driven Culture with Data Mesh - Interview w/ Amy Edwards 1:21:28

1 Weekly Episode Summaries and Programming Notes – Week of October 1, 2023 33:10

1 #255 Zhamak's Corner 28 - Generative AI and Data Mesh: The Start of a Long Road 22:31

1 #254 Easing Into a Data Mesh Journey - Ocean Spray's Pre-Data Mesh Preparations - Interview w/ Paul Cavacas 1:06:46

1 Weekly Episode Summaries and Programming Notes – Week of September 24, 2023 17:23

1 #253 Data Mesh Implementation Success Metrics - Data Quality - Mesh Musings 53 40:58

1 #252 Designing and Building a Better Data Governance Approach - Interview w/ Lauren Maffeo 59:37

1 Weekly Episode Summaries and Programming Notes – Week of September 17, 2023 14:05

1 Rerelease of #130 Making the Data Quantum Leap - Starting from the Data Quantum at PayPal - Interview w/ Jean-Georges Perrin (JGP) 1:23:55

1 Rerelease of #65 What's a Data Contract Between Friends - Setting Expectations with Data Contracts - Interview w/ Abe Gong 1:01:37

1 Rerelease of #48 Overcoming Obstinate Organizational Obstacles in Data Mesh - Interview w/ Scott Hawkins 57:27

1 Weekly Episode Summaries and Programming Notes – Week of September 10, 2023 31:50

1 Rerelease of #150 3 Years in, Data Mesh at eDreams: Small Data Products, Consumer Burden, and Iterating to Success, Oh My! - Interview w/ Carlos Saona 1:24:33

1 Rerelease of #133 Nitty Gritty From the Deployment Committee: Crucial Learnings on Driving Buy-in and Data Product Discovery - Interview w/ Ammara Gafoor 1:19:06

1 Rerelease of #177 - Zhamak's Corner 14 - The Data Can't Protect Itself 20:22

1 Weekly Episode Summaries and Programming Notes – Week of September 3, 2023 37:36

1 Rerelease of #52 Data Mesh Data Governance: Getting Out of Your Own Way - Interview w/ Sarita Bakst 1:11:30

1 Rerelease of #44 A Pragmatic Approach to Getting Started with Data Mesh at Northern Trust - Interview w/ Khanh Chau 1:16:11

1 Rerelease of #25 The Most Key Data Mesh Takeaway: It's Not Just You - Mesh Musings 5 21:00

1 Weekly Episode Summaries and Programming Notes – Week of August 27, 2023 16:30

1 New Episode Hiatus Announcement 1:39

1 #251 Panel: Tying the Business Strategy to the Data Work and Vice Versa - Led by Burce Gültekin w/ Ghada Richani, Beth Bauer, and Michael Toland 1:02:10

1 #250 Staying Aligned on the Vision in Data Mesh - Lessons from Volkswagen's Journey - Interview w/ Christoph Spohr 1:11:07

1 Weekly Episode Summaries and Programming Notes – Week of August 20, 2023 29:31

1 #249 Zhamak's Corner 27 - Creating Truly Scalable Interconnectivity for Data 21:30

1 #248 Doing Data Quality Right by Building Trust - Interview w/ Ale Cabrera 1:10:11

1 Weekly Episode Summaries and Programming Notes – Week of August 13, 2023 13:11

1 #247 Data Mesh Implementation Success Metrics - The Data Platform - Mesh Musings 52 28:26

1 #246 Making Federated Data Governance Approachable and Effective - Interview w/ Kinda El Maarry, PhD 1:16:27

1 Weekly Episode Summaries and Programming Notes – Week of August 6, 2023 20:10

1 #245 Panel: Lessons From Doing Data Mesh Again: The Second Timers Club - Led by Samia Rahman w/ Khanh Chau and Sheetal Pratik 1:17:32

1 Weekly Data Mesh Roundtables - Join Us! 3:05

1 #244 Leading a Data Transformation the Empathetic (and Right) Way - Going Far Together - Interview w/ Benny Benford 1:04:36

1 Weekly Episode Summaries and Programming Notes – Week of July 30, 2023 33:44

1 #243 Zhamak's Corner 26 - The Fundamental Data Need: Autonomy with Interconnectivity 21:51

1 #242 Making Data Accessible Makes Your Data Work Successful - More on PayPal's Data Mesh Journey - Interview w/ Kim Thies 1:17:32

1 Weekly Episode Summaries and Programming Notes – Week of July 23, 2023 15:42

1 #241 Data Product Success Metrics - A Kinda Deep Dive - Mesh Musings 51 17:31

1 #240 Driving to Better Healthcare Patient Outcomes Through Data - Interview w/ Smriti Kirubanandan 59:59
به Player FM خوش آمدید!
Player FM در سراسر وب را برای یافتن پادکست های با کیفیت اسکن می کند تا همین الان لذت ببرید. این بهترین برنامه ی پادکست است که در اندروید، آیفون و وب کار می کند. ثبت نام کنید تا اشتراک های شما در بین دستگاه های مختلف همگام سازی شود.