Weekly Episode Summaries And Programming Notes – Week Of August 27, 2023 Data Mesh Radio podcast

D

Data Mesh Radio

1
Summer Hiatus Announcement - Back in August 4:28

1 year پیش4:28

4:28

Taking a needed break to focus on getting healthy. Be back in August!

D

Data Mesh Radio

1
#306 Building with People for People - Swisscom's Data Mesh Approach and Learnings - Interview w/ Mirela Navodaru 1:09:06

1 year پیش1:09:06

1:09:06

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Mirela's LinkedIn: https://www.linkedin.com/in/mirelanavodaru/ In this episode, Scott interviewed Mirela Navodaru, Enterprise and Solution Architect for Data, Analytics, and AI at Swisscom. Some key takeaways/thoughts from Mirela's point of view: Specifically at Swisscom, it's not about doing data mesh. They want to make data a key part of all their major decisions - operational and strategic - and data mesh means they can put the data production and consumption in far more people's hands. Data mesh is a way to achieve their data goals, not the goal. When you are trying to get people bought in to something like data mesh, you always have to consider what is in it for them. Yes, the overall organization benefiting is great but it’s not the best selling point 😅 try to develop your approach to truly benefit everyone. Data literacy is crucial to getting the most value from data mesh. Data mesh is not about throwing away the important knowledge your data people have but it's about unlocking the value of the knowledge your business people have to be shared with the rest of the organization effectively, reliably, and scalably. ?Controversial? You really have to talk to a lot of people early in your data mesh journey to discover the broader benefits to the organization. That way you can talk to people's specific challenges to get them bought in. When designing your journey, it is important to get input from a large number of people. When talking data as a product versus data products, the first is the core concept and the second is the deliverables. Scott note: this is a really simple but powerful delineation "No value, no party." If there isn't a value proposition, there shouldn't be any action. You need to stay focused on value because there are so many potential places to focus in a data mesh implementation. You have to balance value at the use case level to the domain versus more global value to the organization. At the end of the day, everything you do should add value to the organization but sometimes use cases are much more focused at the domain and that's perfectly expected and acceptable. Data mesh, to really change the organization in the right way, needs top level buy-in. You can't only be the data team trying to head down the data mesh path. Everything in data mesh is about iterating to better. You need the space and room to learn as you go along. You can - and must - deliver value before you've got everything figured out perfectly. Relatedly, you will learn how to better iterate towards value throughout your journey. It will be tough at the start as with any learning journey. Obviously, data mesh is a large cultural change. You need to have empathy and give people the chance to grow instead of trying to move too fast. Upskilling, especially around data literacy, is crucial. There are two very valuable aspects of data mesh: the value you deliver via use cases along the way and the value you get from learning to do data better across your organization. The first is from integrating data into far more of your decisions and the second means you can react more quickly to new opportunities and build scalable and reliable approaches to data management. Something like data mesh is a big change. But it shouldn't be a shock to people. You can do it gradually and incrementally while you deliver value. One of the best ways to lose people is to thrust disruptive change on them instead of working with them through the change to prevent large-scale negative disruptions. There are so many areas where data mesh helps organizations, whether it is getting away from silos, reducing redundancy, improving quality and reliability, etc. It's not just about doing data management itself better, which has been the focus of most data approaches historically. Again, data work is not the point. The point is to make your colleagues better at their job through being more informed. That comes down to the data but it's never the actual point, it's the vehicle to delivering value. Transparency and managing expectations - and communication in general - are crucial to doing data mesh well. You need to have that space to learn and iterate. Let people know what you are doing and especially why you are doing it. Data modeling in data mesh is of course a challenge. But it's important to have some level of common language between the domains or you will have data silos. It's a balance but it's crucial to give domains flexibility but also create easy paths for people to combine data across domains. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#305 Combining the Technical and Business Perspectives for Data Mesh - Interview w/ Alyona Galyeva 1:05:59

1 year پیش1:05:59

1:05:59

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Alyona's LinkedIn: https://www.linkedin.com/in/alyonagalyeva/ In this episode, Scott interviewed Alyona Galyeva, Principal Data Engineer at Thoughtworks. To be clear, she was only representing her own views on the episode. Some key takeaways/thoughts from Alyona's point of view: ?Controversial? People keep coming up with simple phrasing and a few sentences about where to focus in data mesh. But if you're headed in the right direction, data mesh will be hard, it's a big change. You might want things to be simple but simplistic answers aren't really going to lead to lasting, high-value change to the way your org does data. Be prepared to put in the effort to make mesh a success at your organization, not a few magic answers. !Controversial! Stop focusing so much on the data work as the point. It's a way to derive and deliver value but the data work isn't the value itself. Relatedly, ask what are the key decisions people need to make and what is currently preventing them from making those decisions. Those are likely to be your best use cases. When it comes to Zhamak's data mesh book, it needs to be used as a source of inspiration instead of trying to use it as a manual. Large concepts like data mesh cannot be copy/paste, they must be adapted to your organization. It's really important to understand your internal data flows. Many people inside organizations - especially the data people - think they know the way data flows across the organization, especially for key use cases. But when you dig in, they don't. Those are some key places to deeply investigate first to add value. On centralization versus decentralization, it's better to think of each decision as a slider rather than one or the other. You need to find your balances and also it's okay to take your time as you shift more towards decentralization for many aspects. Change management is best done incrementally. ?Controversial? A major misunderstanding of data mesh that some long-time data people have is that it is just sticking a better self-serve consumption layer on top and we can continue to do monolithic data work under the hood. Be prepared for lots of friction in convincing some data architects that this isn't just a reskin or another layer on top of the enterprise warehouse or data lake. For data mesh, it's crucial to understand necessary changes at the technical and the business level. You can't only work on one but you also don't have to 'solve' them at the start, make progress. It's like with the four principles, you need to thin slice change across the technical and business aspects rather than only focusing on one. You can sell data engineers on data mesh by making their work more meaningful and impactful. Instead of mostly firefighting - which is the case in many organizations - they can focus on shipping new features and adding incremental value. With data mesh, you want people focusing on more than just making data valuable - what is valuable will change so how do you make your data products evolvable and maintainable? You always want to be focused on addressing people's pain points in data mesh, driving towards value. That's how you can get data people bought in as well, not just business people. !Controversial! Doing aggregated data products across domains is usually the data mesh inflection point - basically answering can data mesh work in your organization. If you can't get that cross domain collaboration going well, you should consider another model like hub and spoke. Relatedly, aggregating data across multiple domains is where there is usually the most value for an organization. But it's very hard to find good champions there because you need more vision and more hard work to collaborate across domain boundaries. Identify the people with the vision early in your journey, even if it's often better to actually only start working with them once you have more momentum and data products. Too often, there is a rush to build _something_ instead of the right thing. Don't get fooled by the idea that data work always creates value. Even if the client or business partner asks you to build something of value, always circle back to the use cases. As much as we'd like to build universal data products, they just don't exist. Relatedly though, don’t get so focused only on trying to build the consumer-aligned data products for hyper-specific use cases that you miss the forest for the trees. Sometimes the use case is something like 'we need to understand what data we even have to be able to use it to address our current problems in XYZ business line.' To kind of sum it up: stop focusing on what you can build first. Focus on what you should build and then look at the realities. What matters to the business and why? Then focus on what's possible and what will deliver sustainable and maintainable value through data work. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#304 Getting Your Data Mesh Journey Moving Forward - Interview w/ Chris Ford and Arne Lapõnin 1:01:50

1 year پیش1:01:50

1:01:50

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Arne's LinkedIn: https://www.linkedin.com/in/arnelaponin/ Chris' LinkedIn: https://www.linkedin.com/in/ctford/ Foundations of Data Mesh O'Reilly Course: https://www.oreilly.com/videos/foundations-of-data/0636920971191/ Data Mesh Accelerate workshop article: https://martinfowler.com/articles/data-mesh-accelerate-workshop.html In this episode, Scott interviewed Arne Lapõnin, Data Engineer and Chris Ford, Technology Director, both at Thoughtworks. From here forward in this write-up, I am combining Chris and Arne's points of view rather than trying to specifically call out who said which part. Some key takeaways/thoughts from Arne and Chris' point of view: Before you start a data mesh journey, you need an idea of what you want to achieve, a bet you are making on what will drive value. It doesn't have to be all-encompassing but doing data mesh can't be the point, it's an approach for delivering on the point 😅 Relatedly, there should be a business aspiration for doing data mesh rather than simply a change to the way of doing data aspiration. What does doing data better mean for your organization? What does a "data mesh nirvana" look like for the organization? Work backwards from that to figure where to head with your journey. A common early data mesh anti-pattern is trying to skip both ownership and data as a product. There are existing data assets that leverage spaghetti code and some just rename them to data products and pretend that's moved the needle. "A data product is a data set + love." The real difference between a data product and a data set is that true ownership and care. ?Controversial?: Another common mesh anti-pattern is trying to get too specific with definitions or prescriptive advice. There isn't a copy/paste approach that will work and getting a specific definition of a data product doesn't really change much. Mindset is far more important than definitions. It can be very helpful to have some simple checklists around your data products. While there is no prescriptive way to build, checklists remove a lot of the uncertainty for teams asking 'am I doing this right?' It gives some simple reassurances that you aren't missing out on key pieces of what they're building. ?Controversial?: Most organizations probably don't need to do a ton of pre-work before starting on a data mesh implementation. They need some achievable goals, a roadmap for how they plan to achieve those goals, and a lot of willpower to push things forward and keep going when the going gets tough. You also need an enticing vision for people to buy into. THIN SLICE! Don't try to take everything on at once but also don't try to skip over any of the four pillars. There's a reason they haven't changed from Zhamak's initial blog post. Scott note: don't try to argue the governance pillar wasn't in the first blog post, it just wasn't called out separately… Three key questions to answer if you are considering data mesh: A) Do you have sufficient scale? B) Do you have a strategy that depends on deriving value from data? C) Are you prepared to take advantage of the autonomy Data Mesh will afford to your product teams? If you don't have satisfactory answers to those three questions, data mesh is probably not right/overkill for you. If people don't see the strong need to transform your business through data, it's likely to lead to troubles 6-9 months into your data mesh journey. If you aren't addressing key organizational pain points or delivering value, you will likely lose support for your data mesh implementation initiative. Doing data better has to be valued to get more budget to keep going. Another anti-pattern is focusing too much on use cases at the expense of the platform and the journey. Data mesh is designed to work at scale and that only works by finding repeatable processes. You can't treat each data product like a one-off. In order to get buy-in from the data engineers - or whoever are your data product developers - you need to invest in changing hearts and minds through the platform. If creating and managing data products is significantly harder than the old way of dealing with data, you will lose people quickly. Read about the data mesh accelerate workshop 😅 When you think about first steps with data mesh, A) build buy-in at the strategic level that you want to actually start leveraging your data for high-value purposes; B) find use cases to support those strategic initiatives; and C) make sure you are ready to actually thin slice and not try to only tackle on pillar - you have to be ready to take on a LOT of challenging work. !Controversial!: None of the four data mesh principles are all that useful on their own. Scott note: there's a figure Zhamak has that explains why all four are necessary in conjunction that's very helpful here. It's easy to want to skip bringing all your key stakeholders into alignment early in your data mesh journey. But you need matching expectations and shared understandings of what you are trying to accomplish and why. Scott note: this doesn't mean everyone has to be bought in that they are first, there's a balance to be found here. You need room to make mistakes and adjust your data mesh implementation because you will not get it all right at the start. Data mesh is as much about learning how to do data well as doing data well. It's crucial to not just ask if you are succeeding with your data mesh implementation but measure that. It can be hard to measure but consider what matters to your implementation's success and find things to measure if you're succeeding in each of those areas. Otherwise, how do you know where to focus and optimize? Subsidiarity: "everything should be decided as locally as possibly but no more so." Basically, there are many decisions that should be made in the domains but there are some that need to be made centrally. The challenge is figuring out which decisions should be made where 😅 There will be capability challenges in every organization when doing data mesh. That will impact initial decisions around how much to centralize or decentralize but as you upskill the teams, you may want to decentralize more. Find your equilibriums but equilibriums change. It's all about trade-offs! Many people are too focused on exactly if they are doing data mesh instead of are they delivering value in a scalable way through data. That happened in microservices too and it took them 10+ years to really get to best practices. Data mesh is only 5 years old and only ~3 with any number of organizations attempting it - focus on getting better instead of being worried if you're the perfect picture of data mesh yet. When talking about your data mesh success internally, you need to talk about the value from use cases AND the value of improving your data capabilities in general. You prove out you are delivering specific value along the way but also that you are getting more and more capable at doing the data work to make the organization better. Both are of valuable and you should promote the value of both aspects: use case value and capability value. When talking about data mesh, use the ADKAR method: create Awareness and Desire, give them the Knowledge about how you're doing it, upskill people so they have the Ability to do data mesh, and finally constantly Reinforce the value and that it's important. Without touting your mesh successes, you'll lose momentum. When looking for your first data mesh use case, look for something that has a customer impact - what can you do for them that you couldn't before. Personalization is a good example. Legal is potentially another place re reducing risk. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#303 Delivering What Matters - Value - Through Strong Business Collaboration - Interview w/ Saba Ishaq 1:10:37

1 year پیش1:10:37

1:10:37

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode (link) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Saba's LinkedIn: https://www.linkedin.com/in/sabaishaq/ Decide Data website: ttps://www.decidedata.com/ In this episode, Scott interviewed Saba Ishaq, CEO and Founder of her own data as a service consultancy, Decide Data, which also provides 3rd party DAaaS (Data Analytics as a Service) solutions. Some key takeaways/thoughts from Saba's point of view: "If you don't know what you want, you're going to end up with a lot of what you don't want." This is especially true in collaborating with business stakeholders when it comes to data 😅 Focus on delivering value through data instead of delivering data and assuming it has value. – “Not all data is created equal.” As a data leader, it's your role to help people figure out what they actually want by asking great questions and being a strong partner when it comes to the data/data work. Don't only focus on the data work itself but it's very easy to do data work for the sake of it instead of something that is valuable. To deliver data work that actually moves the needle, we need to start from what are the key business processes and then understand the pain points and opportunities. Then, good data work is about how do we support and improve those business processes. Relatedly, that's also the best way to drive exec alignment - talking about their business processes and how they can be improved first, data work second. They will feel seen and heard and are far more likely to lean in. At the end of the day addressing business and operational challenges is what data and analytics is all about. Deliver something valuable early in any data collaboration with a business stakeholder. You don't have to deliver an entire completed project but time to first insight is time to value and you build momentum and credibility with that stakeholder. At the beginning of a project - and delivering a data product is itself a project - you should work with stakeholders to not just define target outcomes but also define how are you going to collaborate and communicate. You can't just get requirements, go away and build. Working with data should be iterative and should have an element of continuous improvement to evolve what you deliver as you build value. Start any data work by asking someone about their business objectives, challenges, and target outcomes. You need your business stakeholders to have a clear vision of what they want to achieve, otherwise you are likely to be delivering only data work instead of business value that leverages your data work. By doing deep discovery work, you can find where are the key lynchpins and value drivers in a use case. There are points of criticality that are easy to lose in a sea of potential requirements that are really requests. Find those crucial value leverage points! Relatedly, you can use those value leverage points to keep your business execs engaged. They will - hopefully - see the importance and help you narrow in on what matters in their use case. Then it's no longer about the data work but the value to them. ?Controversial?: For data people, you have to balance career management and interesting project/technology work versus value delivery. That doesn't mean delivering value isn't interesting but it doesn't always mean getting to play with the latest and greatest. But if data people never get to have fun and play with cool tech, many will leave. It's a tough balance. Try to make the valuable work also interesting 😅. Relatedly, try understanding the data team’s learning areas of interest and see how you can build seeds to foster their skill growth while making data work valuable. Sometimes it turns out to be a win-win situation. Relatedly, be very transparent and communicate a lot to your data teams about what you are prioritizing and why. It's very easy to get lost in telling data people to do certain work rather than why they are doing that work. Keeping your data people in the loop of the why will keep them focused on what matters. For many organizations, the rate of change of their technology - application and data technologies - is growing at faster than the rate of their people change management/transformation processes. You need parallel streams to modernize both or your people will fall further behind, leading to chaos. ?Controversial?: Relatedly, your overall org and/or digital transformation strategy should be tied to your data strategy. Otherwise, they will likely be heading in different directions, creating more challenges. Scott note: Benny Benford talked a lot about this in episode #244, going far together. Data management is a very crucial element of digital transformation but it’s not the same thing as change management. The data team shouldn't be the ones leading the overall digital transformation of the organization. That's too much on a team that specializes in data rather than change management. If you are in that situation, it's a very tough spot to do well. It's very important to focus on communication to stakeholders when you think about data governance and digital transformation. For many execs, these are foreign topics so you have to work hard to engage them and keep them leaning forward on the necessary work. Data governance is beneficial for everyone, so if explained and defined well people will engage willingly after knowing what’s in it for them. As someone in the data team, you have to be well informed about digital transformation initiatives inside your organization. Otherwise, you will miss opportunities to align to those initiatives AND have all your data sources break when there is a migration you weren't told about 😅 It's easy to screw up the data steward/ownership conversation letting someone know they are responsible for the governance of their data. It's often a scary conversation for both parties. But it's necessary and you can show people why it makes sense and adds value to their work too! Relatedly, link people's pain points to current weaknesses in the data governance. Show them they are causing issues for themselves and give them an easier path to fix it without having to learn everything about data work. Data governance doesn't have to be some wholly - or holy 😅 - separate practice. It should just be part of normal work related to data. Make it less scary and more approachable for your business stakeholders. It's a team effort and it drives real, measurable benefits and value. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
No Episode This Week 1:31

1 year پیش1:31

1:31

Craziness of the overseas move (including a faulty office chair... long story) are to blame. Back to the normally scheduled one episode a week next week! Episode list and links to all available episode transcripts here .

D

Data Mesh Radio

1
#302 Finding and Delivering on a Good Initial Data Mesh Use Case - Interview w/ Basten Carmio 1:11:47

1 year پیش1:11:47

1:11:47

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Basten's LinkedIn: https://www.linkedin.com/in/basten-carmio-2585576/ In this episode, Scott interviewed Basten Carmio, Customer Delivery Architect of Data and Analytics at AWS Professional Services. To be clear, he was only representing his own views on the episode. Some key takeaways/thoughts from Basten's point of view: Your first use case - at the core - should A) deliver value in and of itself and B) improve your capabilities to deliver on incremental use cases. That's balancing value delivery, improving capabilities, and building momentum which are all key to a successful long-term mesh implementation. When thinking about data mesh - or really any tech initiative - it's crucial to understand your starting state, not just your target end state. You need to adjust any approach to your realities and make incremental progress. ?Controversial?: Relatedly, it's very important to define what success looks like. Doing data mesh cannot be the goal. You need to consider your maturity levels and where you want to focus and what will deliver value for your organization. That is different for each organization. Scott note: this shouldn't be controversial but many companies are not defining their mesh value bet… Even aligning everyone on your organization's definition of mesh success will probably be hard. But it's important to do. For a data mesh readiness assessment, consider where you can deliver incremental value and align it to your general business strategy. If you aren't ready to build incrementally, you aren't going to do well with data mesh. A common value theme for data mesh implementations is easier collaboration across the organization through data; that leads to faster reactions to changes and opportunities in your markets. Mesh done well means it's far faster and easier for lines of business to collaborate with each other - especially in a reliable and scalable way - and there are far better standard rules/policies/ways of working around that collaboration. But organizations have to see value in that or there may be mesh resistance. As many have said, you must approach data mesh in a thin slice. Trying to focus too much on any pillar at the expense of the others leads to challenges. Scott note: Zhamak literally has a figure on this she shares often. It's easy to get unbalanced if you ignore a principle and fixing that takes more effort than thin slicing. ?Controversial?: As you build out your mesh capabilities, especially your platform, think about what you need to actually deliver on the use case(s) at hand and deliver only that. Don't get ahead of yourself. It's fun to build capabilities but it's easy to build a monstrosity of a platform instead of proper abstractions to make building and managing data products simple. Doing data mesh well is all about managing trade-offs. There are trade-offs at the use case and implementation level. And it's okay to get those wrong, just look to fail fast rather than hold on to bad decisions. Don't be precious with your decisions and build in ways to make evolving and improving easier. MVP can be minimum viable product or minimum valuable product. You need to define what value actually means for each use case. It's easy to point to pain and start solutioning but not end up addressing the pain or creating value. This will help you prioritize and deliver the most value compared to effort early. Relatedly, having a clear perspective on value in your MVP means you are more easily able to change and adapt as you learn and deliver. If you thought value was going to come from X but early indications are Y is way more important or is much different than expectations, you can more easily pivot towards Y. Data products don't magically create value. They should be bets on value delivery. So you need to have ways to gracefully retire data products when the cost exceeds the incremental value. Often times a bet was a good bet even when it didn't pay off or it is no longer paying off. Continuous communication and driving buy-in, especially with C-level execs, is unfortunately crucial to your data mesh implementation success. If they don't value better data capabilities, few people will invest their time and effort to really reinvent the way your organization does data. Relatedly, you need to tie your data mesh implementation to the overall business strategy. Execs need to know why you are doing something like data mesh and how it will deliver value. It's not an overnight success but it also can't be a 3 year project before value. Speaking to that gets more people to lean in so you can build momentum. Finding and leveraging your data success champions is always hard but it's necessary to build that momentum. Think about building champions quite early. When you're trying to get an exec to buy in to something like data mesh, always put it in terms of what's in it for them. Don't try to get them bought in to some grand vision first, why would they invest their valuable time and resources here? Find their pain points and what something like data mesh will do to address those specifically. Relatedly, trying to get someone to just take on data ownership without the understanding of what the rest of the organization is going to do to make ownership much easier - mainly through the platform and federated governance - is probably going to be a … not fun conversation. When thinking about data mesh, it should all come back to value. If you can't specifically point to value that will exceed your effort for doing something like data mesh, it's probably not worth it for your organization. What is valuable to your organization and how will data mesh help you capture that value? Relatedly, as you are on your data mesh journey, you have to be honest in assessing if data mesh is really working for your organization. It's okay to stop if it's not. The data mesh principles sound really great and helpful in the abstract. But they just might not be aligned to value with the way your company does business. ?Controversial?: It's perfectly acceptable to have hybrid approaches in a data mesh journey. There is a perception that everything should be decentralized and that just isn't always the best approach, don't get caught up in dogma. Scott note: the "decentralize everything" is a misunderstanding of Zhamak. Also, as you are on a journey, your current state won't match an ideal end state. Focus on evolving while delivering value, not trying to be perfect today instead of _getting_ to good. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#301 Learnings From 25+ Years in Data Quality - Interview w/ Olga Maydanchik 1:01:57

1 year پیش1:01:57

1:01:57

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Olga's LinkedIn: https://www.linkedin.com/in/olga-maydanchik-23b3508/ Walter Shewhart - Father of Statistical Quality Control: https://en.wikipedia.org/wiki/Walter_A._Shewhart William Edwards Deming - Father of Quality Improvement/Control: https://en.wikipedia.org/wiki/W._Edwards_Deming Larry English - Information Quality Pioneer: https://www.cdomagazine.tech/opinion-analysis/article_da6de4b6-7127-11eb-970e-6bb1aee7a52f.html Tom Redman - 'The Data Doc': https://www.linkedin.com/in/tomredman/ In this episode, Scott interviewed Olga Maydanchik, an Information Management Practitioner, Educator, and Evangelist. Some key takeaways/thoughts from Olga's point of view: Learn your data quality history. There are people who have been fighting this good fight for 25+ years. Even for over a century if you look at statistical quality control. Don't needlessly reinvent some of it :) Data literacy is a very important aspect of data quality. If people don't understand the costs of bad quality, they are far less likely to care about quality. Data quality can be a tricky topic - if you let consumers know that the data quality isn't perfect, they can lose trust. But A) in general, that conversation is getting better/easier to have and B) we _have_ to be able to identify quality as a problem in order to fix it. Data quality is NOT a project - it's a continuous process. Even now, people are finding it hard to use the well-established data quality dimensions. It's a framework for considering/measuring/understanding data quality so it’s not very helpful to data stewards / data engineers in creating data quality rules. The majority of quality errors are not random, they come from faulty data mapping / bugs in pipelines. Having good quality rules will catch a large percentage of errors that can be fixed in bulk. When thinking about getting started around data quality, it doesn't have to be complex and with lots of tools. It can be people looking at the data for potential issues and talking to producers. Then you can build a business case for fixing the data to get funding. You have to roll up your sleeves and talk to people but you can get forward momentum. Data quality issues aren't inherently material to the business processes - they are only bad when they cause issues for the business. You have to find those actual business issues to get people to care and get funding for fixing it. Quality for the sake of quality is just extra cost. Do not create too many data quality rules that do not matter. Relatedly, being able to show someone a relatively basic quality indicator early is far better than asking for a lot of budget to figure out the quality levels. You can do that with something as simple as random sampling 100-200 records and an hour of 1-2 people's time. To understand which data quality challenges and use cases are the most important, data people simply have to learn more about the business. Good data quality is about fit for purpose and that means understanding the purposes :) To find your initial good data quality use cases, look to mission criticality. What dashboards or reports are actually important to the company and why? Then work backwards to see if quality is an issue for those dashboards and reports. That's how you find your early buy-in to work on a quality initiative that can scale. !Controversial!: Data contracts are not at all new, we just now have a good enough set of tools and technologies to be able to do them better at scale. ?Controversial?: Most are doing data contracts … not that well. For them, it's about the technology and not the process. There isn't a continuous approach. Scott note: Andrew Jones has said the same. It's about ensuring a process that results in quality data, not the tools. For data contracts, there MUST be a feedback loop or we aren't actually delivering to needs, especially as needs evolve. Look to the widely used customer supply model for insights into what we need to achieve and how when it comes to data contracts. Many companies are creating actual financial incentives tied to data quality in order to ensure people care about data quality. That's not right for every organization but it does send a clear message as to the importance of data quality. You have to consider your data supply chain - if your interface for data input is bad, your data is very likely to be bad. People will simply enter garbage to move forward. Doing data quality manually is not sustainable/scalable. But you don't need to start with expensive tools, you can get your arms around things initially pretty easily. It will help you identify your actual problems instead of spending time specifically on tools. ?Controversial?: Many vendors are selling their tools as the fix to data quality. But detecting data errors with the tools is only the start of the data quality improvements. Once errors are detected, root cause analysis for the errors needs to be performed and the processes / code need to be fixed. None of the data quality tools can do this. It is human’s job. Beware the snake oil. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#300 Panel: How to Treat Your Data Platform as a Product - Led by Michael Toland w/ Sadie Martin, Marta Diaz, and Sean Gustafson 1:03:01

1 year پیش1:03:01

1:03:01

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Michael's LinkedIn: https://www.linkedin.com/in/mjtoland/ Marta's LinkedIn: https://www.linkedin.com/in/diazmarta/ Sadie's LinkedIn: https://www.linkedin.com/in/sadie-martin-06404125/ Sean's LinkedIn: https://www.linkedin.com/in/seangustafson/ The Magic of Platforms by Gregor Hohpe: https://platformengineering.org/talks-library/the-magic-of-platforms Start with why -- how great leaders inspire action | Simon Sinek: https://www.youtube.com/watch?v=u4ZoJKF_VuA In this episode, guest host Michael Toland Senior Product Manager at Pathfinder Product Labs/Testdouble and host of the upcoming Data Product Management in Action Podcast facilitated a discussion with Sadie Martin, Product Manager at Fivetran (guest of episode #64), Sean Gustafson, Director of Engineering - Data Platform at Delivery Hero (guest of episode #274), and Marta Diaz, Product Manager Data Platform at Adevinta Spain. As per usual, all guests were only reflecting their own views. The topic for this panel was how to treat your data platform as a product. While many people in the data space are talking about data products, not nearly as many are treating the platform used for creating and managing those data products as a product itself. This is about moving beyond the IT services model for your data work. Platforms have life-cycles and need product management principles too! Also, in data mesh, it is crucial to understand that 'platform' can be plural, it doesn't have to be one monolithic platform, users don't care. Scott note: As per usual, I share my takeaways rather than trying to reflect the nuance of the panelists' views individually. Scott's Top Takeaways: You will hear "product mindset" a lot in this panel. It's important to embrace product management as a mindset and not an exact set of things to do and approaches to take. The whole point of a product mindset is to find what works and deliver reliably while focusing on value. Be ruthless. Ruthlessly prioritize and ruthlessly focus on user centricity. In data, we have a tendency to fall in love with the tools instead of the jobs to be done. But a good platform is about making it easy to get high-value work done and that's rarely about exposing tooling to users. Your platform isn't going to magically drive usage. There is change management required to get people to leverage your data platform more. Be prepared for that change management work and closely aligning with users, whether they are 'data people' or not. Platforms and products in general are about scalability. Through a platform, instead of reacting to tickets, you build services and capabilities to address the problems that caused the tickets - far more scalable than reacting to the individual tickets, addressing the disease not the symptom. By building your platform as a product, you focus on what actual capabilities are within scope so you can continue to manage and expand your data platform. As much as it would be amazing to build a data platform from scratch - think how amazing you could build it! - in many cases, you will have to build off of existing services and platforms. Don't be too dogmatic - what matters is continuing to get better not being perfect. Give yourself the space to improve the platform but products live in the real world and the real world of your organization has current/existing business needs and constraints. Products - especially in software - are as much or more about evolution as they are about their form/function at initial launch. The same should happen with your platform. And the same should happen with your vision. Don't get locked in, don't get tunnel vision. Generally speaking, most data platforms are still not serving the role a software platform - data or otherwise - should serve. They are still about the tech instead of the capabilities. You aren't alone if your platform doesn't meet the ideal vision. Your role is to make it better but it's still probably not going to be a sparkling beacon of perfection anytime soon 😅 Your data platform needs to be aligned to your company culture. So you have to meet people where they are and properly set down the easy path to where you want them to go. It's a long journey. Other Important Takeaways (many touch on similar points from different aspects): The product mindset is for the entire team, not just the platform leader and/or product manager. Your entire team will have to change their approach and mindset. Not overnight but it's hard to break old ticket-taker habits. Make sure to put your work in the context of the business. It's easy to get bogged down in the platform engineering aspects or the data tools but your data platform is there to serve business purposes. Focus on what delivers value for the business. Your platform doesn't have to address all your organization's data challenges at once, right at the start. Find where you are seeing the biggest challenges with delivering value - maybe listen back to episode #297 on the Data Value Chain - and look to focus there first. Build up to better. "Build with [your users], not for them." Don't treat your platform as a project. Yes, that's the subject of the panel but it's very important to always keep close at heart. User experience is such a crucial aspect of good software platforms. It's probably even more important when it comes to data platforms if you want to bring new users to producing and consuming data. But it's HARD and rarely discussed. What is the goal of your platform? That might sound like an obvious question but it's not really when you go to answer it. Maybe it's "make it easier to work with data" but easier for who? Really map out good outcomes of a well-made platform. Good products - at least good software products - aren't designed to be perfect at launch. Give yourself the space to not be perfect but set yourself up to understand where things can be improved and then iterate. Test, learn, iterate. What are your data platform KPIs? What will measure if you are being successful? Really consider what bets you are making and why. Usage isn't always good but it's a good first indicator as you are getting moving. To make your platform a product, you have to come back to the vision (or goal mentioned above). Otherwise, you have a collection of services. How is that vision tied to business value? If you need budget, do those holding the purse strings care about your tech decisions or business impact? A good platform helps people get what they need done with the agency to do it. It limits - and where possible eliminates - bottlenecks. A crucial aspect of product management is taking user needs and then translating that to build something to serve those needs. But when it comes to your data platform, the users and the builders (the data engineering team) usually don't speak the same language so you will probably have to spend more time translating between the users and the team building the platform than even in software. Be prepared for you data team to grumble about more meetings too 😅 Combine Simon Sinek's 'Start from the why' and user centricity. The why should be about what are you enabling your users to actually accomplish and what value that drives. Really focus on building something that drives value for the business instead of leveraging the coolest tech. Relatedly, it's likely going to be very difficult to measure the return on investment on your data platform. Be prepared for those conversations but it can be pretty squishy. At their heart, good products are about creating good user experiences and delivering/capturing value. Products need to deliver value to users and producers need to capture value as well. That is a complex topic when the value captured is internal, but it is still an appropriate mindset. If you aren't able to prove value, will you get further funding? Consider your company needs relative to data maturity. An incredibly cutting edge platform for an organization that has low data maturity is not the right fit. Even if you have a cutting edge vision for their work, your 4 year old won't be able to out sculpt Michealangelo. Be realistic and use the platform to drive towards better data capabilities but you have to meet your organization at least close to where they are. You might have difficulty getting people bought in on the vision of your data platform. If people view data as ticket-takers/a service instead of an integral part of the company's way of doing business, be prepared for the fun of lots of stakeholder alignment work. Where possible, don't only look to sell people on your vision. Try to change - or possibly nudge at most - people's behavior through your platform. It's a subtle art and hard to pull off but it will mean people do what you and they both need but without having to build a million PowerPoint decks 😅 It's important to separate your concept of a data platform and the approach of treating it as a product. Otherwise, your platform can easily fall into the trap of designing to user wants instead of what is a platform's function - to make it easier to do work in a scalable, reliable, and repeatable fashion. You need to consider the purpose of the platform instead of throw product management at a collection of tools. Leaders/execs want change. Leaders/exec do not want _to change_. Be prepared for that. Users aren't necessarily ready for the data platform to be a product. They are used to the ticket-taker model. Be prepared to shift their mindset too, not just that of the (data) engineers building the platform. It won't be a simple switch over either - very deep topic… Relatedly, you will probably need a longer run-way to transition your team to a product model for the platform. It is a mindset shift but it's also a big change in the ways of working. You will need to have some patience, you can't switch from a ticket model to only building your platform as a product overnight. A key potential area of value for the platform is making it fast to prototype and get data in consumers' hands. It can reduce a ton of back and forth and also help you quickly discover if further work on a potential data product is useful or if it should be scrapped. Research spikes are very important in data and enabling them can be very valuable - if not always valued 😅 Make sure to think about the actual scope of your data platform. Products have scope, don't try to be all things to all people and don't bite off more than you can chew. There's a reason SaaS offerings in the data space have post-sales and customer success engineers. Many of your users won't just simply be able to read the documentation and use your platform no matter how well you build and document it. Be prepared to ensure success through a bit of hand-holding. Ensuring usage that drives value is (usually) what makes software and data offerings successful. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#299 Empowering Development with Actionable Data - Interview w/ Carol Assis and Eduardo Santos 1:13:01

1 year پیش1:13:01

1:13:01

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Carol's LinkedIn: https://www.linkedin.com/in/carol-assis/ Eduardo's LinkedIn: https://www.linkedin.com/in/eduardosan/ Continuous Integration book: https://www.amazon.com/Continuous-Integration-Improving-Software-Reducing/dp/0321336380 Measure What Matters book: https://www.amazon.com/Measure-What-Matters-Google-Foundation/dp/0525536221 Inspired by Marty Cagan: https://www.amazon.com/INSPIRED-Create-Tech-Products-Customers/dp/1119387507 Empowered by Marty Cagan: https://www.amazon.com/EMPOWERED-Ordinary-Extraordinary-Products-Silicon/dp/111969129X In this episode, Scott interviewed Carol Assis, Data Analyst/Data Product Manager and Eduardo Santos, Professor and Consultant, both at Thoughtworks. To be clear, they were only representing their own views on the episode. From here forward in this write-up, I will be generally combining both Carol and Eduardo's views into one rather than trying to specifically call out who said which part. Some key takeaways/thoughts from Eduardo and Carol's point of view: At the end of the day, the team that produces the data will get the most use out of it 9/10 times. Getting teams used to developing with data in mind isn't just useful for the organization, it is for maximizing their own team's success. Continuous integration is a crucial concept in general for learning how to automate and focus on delivering more, which leads to focusing on value. Read the book :) ?Controversial?: Data mesh is an extension of the continuous integration book/concept because of the focus on delivering value quickly and building to scale reliably. There are many methodologies for understanding value delivery in software. We just have to adapt them better to data. Don't reinvent the wheel. Far more organizations need to think about the goals of their products and then how to measure success against those goals _at product inception_. Design data into your products from the start. Data people often make the data overly complicated for non-data people to grasp. What does the data tell us and what are some simple numbers? Then people can feel like they understand without going too deep into stochastic modeling or something. Relatedly, engage data consumers' curiosity - including the producers that will consume their own data. Try to meet them where they are to get them to engage with data more. Lower the perceived bar to leveraging data. Application development teams need convincing that working with data is 1) essential to understand their own success to further improve their products and 2) much easier than it has been historically. There is a LOT of scar tissue out there… A potential good hook to get people to build their applications with data in mind is being able to show metrics and measure success from day one. The business side can get a better idea and are more likely to engage; it gives a communication bridge between developers and the business people. Having data early in the application development cycle means you have more proof points for making your decisions - assuming you side with the data 😅 that makes it easier to justify decisions instead of people making guesses. Measuring what matters is a crucial concept for the entire team to adopt. It will help people understand what data they need and why. Pressing people on what to measure and then how to measure it crystalizes what bets they are making. How are they expecting users to interact with their applications? For many business people, you may need someone playing the data translator role, translating data to business and business to data. Most organizations' data literacy is still quite low and again, you need to lower the bar to business people leveraging data. ?Controversial?: You don't need to start with data products. Yes, they are great, but teaching your teams that data matters is groundwork to head in the direction of something more scalable. A spreadsheet is a fine place to start. Focus on delivering insights that deliver value, then work towards the productized aspects :) Ask people what action they will take once they have data. Get them in the mindset that data drives action and data that won’t drive action isn't where they should focus. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#298 Effective Partnering With Business Execs - Learnings from Another Data Mesh Journey - Interview w/ Jessika Milhomem 1:07:39

1 year پیش1:07:39

1:07:39

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Jessika's LinkedIn: https://www.linkedin.com/in/jmilhomem/ In this episode, Scott interviewed Jessika Milhomem, Analytics Engineering Manager and Global Fraud Data Squad Leader at Nubank. To be clear, she was only representing her own views on the episode. Some key takeaways/thoughts from Jessika's point of view: There are no silver bullets in data. Be prepared to make trade-offs. And make non data folks understand that too! Far too often, people are looking only at a target end-result of leveraging data. Many execs aren't leaning in to how to actually work with the data, set themselves up to succeed through data. Data isn't a magic wand, it takes effort to drive results. Relatedly, there is a disconnect between the impact of bad quality data and what business partners need to do to ensure data is high enough quality for them. Poor data quality results in 4 potential issues that cost the company: regulatory violations/fines, higher operational costs, loss of revenue, and negative reputational impact. There's a real lack of understanding by the business execs of how the data work ties directly into their strategy and day-to-day. It's not integrated. Good data work isn't simply an output, it needs to be integrated into your general business initiatives. More business execs really need to embrace data as a product and data product thinking. Instead of a focus on only the short-term impact of data - typically answering a single question - how can we integrate data into our work to drive short, mid, and long-term value? ?Controversial?: In data mesh, within larger domains like Marketing or Credit Cards in a bank, it is absolutely okay to have a centralized data team rather than trying to have smaller data product teams in each subdomain. Scott note: this is actually a common pattern and seems to work well. Relatedly, the pattern of centralized data teams in the domains leads to easier compliance with regulators because there is one team focused on reporting one view instead of trying to have multiple teams contribute to that view. When you really start to federate data ownership, business execs can now partner far easier with other business execs in other domains leveraging data. Instead of having the central data team trying to translate, there is a focus on what needs to get done and the data work flows from that instead of the data work being the focus. It's the engine that powers their collaboration but it's no longer 'the point'. Partnering with those who "are closer to the reality" of the business, it's easier and more likely to drive good outcomes. Meaning: not the senior execs. But the senior execs often have to be on board with the work and the target results. So work on communicating up but closely collaborating at lower levels. Data for regulators often has a LOT of potential reuse for your own organization. Lean into finding those areas where you can do the data work once and get value twice :) ?Controversial?: Really consider role titles in data mesh. Data product owner might be too nebulous and quickly accumulate too many responsibilities. Data product manager is easier to understand the scope of responsibilities and the specific areas of focus. Scott note: this comes up A LOT and is generally starting with data product owner and moving to data product manager. ?Controversial?: Data leaders need to understand product management. To really scale data work, we have to start treating all aspects as a product practice. CTOs down to software engineers need to understand product management, it's time for the data org to as well. Data leaders need to have significant communication skills while maintaining their understandings of data best practices. It's all a delicate balance but the data work doesn't speak for itself. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#297 Panel: Understanding and Leveraging the Data Value Chain - Led by Marisa Fish w/ Tina Albrecht, Karolina Stosio, and Kinda El Maarry, PhD 58:10

1 year پیش58:10

58:10

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Marisa's LinkedIn: https://www.linkedin.com/in/marisafish/ Karolina's LinkedIn: https://www.linkedin.com/in/karolinastosio/ Tina's LinkedIn: https://www.linkedin.com/in/christina-albrecht-69a6833a/ Kinda's LinkedIn: https://www.linkedin.com/in/kindamaarry/ In this episode, guest host Marisa Fish (guest of episode #115), Senior Technical Architect at Salesforce facilitated a discussion with Kinda El Maarry, PhD, Director of Data Governance and Business Intelligence at Prima (guest of episode #246), Tina Albrecht, Senior Director Transformation at Exxeta (guest of episode #228), and Karolina Stosio, Senior Project Manager of AI at Munich Re. As per usual, all guests were only reflecting their own views. The topic for this panel was understanding and leveraging the data value chain. This is a complicated but crucial topic as so many companies struggle to understand the collection + storage, processing, and then specifically usage of data to drive value. There is way too much focus on the processing as if upstream of processing isn't a crucial aspect and as if value just happens by creating high-quality data. A note from Marisa: Our panel is comprised of a group of data professionals who study business, architecture, artificial intelligence, and data because we want to know how (direct) data adds value to the development of goods and services within a business; and how (indirect) data enables that development. Most importantly, we want to help stakeholders better understand why data is critical to their organization's business administration strategy and is a keystone in their value chain. Also, we lost Karolina for a bit there towards the end due to a spotty internet connection. Scott note: As per usual, I share my takeaways rather than trying to reflect the nuance of the panelists' views individually. Scott's Top Takeaways: If you want to dig deeper into the data value chain, consider looking into the value streams concept. What flows through your business in terms of process to generate value? Where are there points of value leakage? The same concepts are crucial in your value chain. Organizations need to really educate their entire organization on the data value chain. Part of why there are so many issues in data from upstream changes by developers breaking downstream data is they simply don't know what parts of their data are used and why. Communication is a much bigger aspect of doing data than people think. Even talking about the specific data value chain can cause people to focus too much on the data work instead of the business value delivered via data. The data value chain is crucial to understand but it's also crucial to understand data work doesn't inherently create value, it's about how it's used in the business. Dig into the value created and focus on working backwards from that to what data work needs to be done. The data value chain is crucial for companies of all sizes across all industries. At its heart, the concept is about focusing on ensuring you aren't leaking value in your business value streams/pipelines. You need to focus on what drives value and how to improve the processes there. Data value chains often cross line of business/domain boundaries. After all, a lot of the value of data is about combining information across those boundaries. That can mean cross-team handoffs, which make understanding and ensuring the success of those data value chains even harder. Who owns what isn't inherently understood/agreed to, you need to get specific. It's important to not get overly focused on a single end-point of value when it comes to data work, especially when it comes to a data product. If we want re-use, we have to focus on the processes of creating reusable value. Maintaining that larger picture focus while still ensuring each data consumer can still get value from a data product is a very hard balance. Focusing heavily on your data value chain is going to be hard. It means hard work and a lot of internal collaboration - and thus negotiation - across domain boundaries. You all have to be in it together to really get the best results - and some organizations aren't ready for that. But the hard work pays off because you are ensuring value actually gets created. As with anything in data, you have to make bets. That doesn't mean every bit of data work will create significant value or even exceed the investment. But an approach like data value chain is crucial to understand 1) what bets are you making and why and 2) who owns what aspects of the data work. That can help you really focus on the what and why rather than focusing on outputs. Other Important Takeaways (many touch on similar points from different aspects): As with many things in data, ownership is crucial to understanding your value chain. The weakest points in a value chain are the handoffs between teams. Strong ownership, including of those handoffs, prevents value leakage (from the value streams concept). To understand your data value chain, you will have to go deeper than many are willing to in the (dreaded?) operational plane. You have to understand what data you have, how it's collected, what data you can collect, etc. Some of it is working backward from what data you need/want but a lot of it is working from what data you have or can get. Relatedly, the value you can create from data is heavily reliant on what matters to the business. To think about value, you have to understand your business processes and what generates actual value. You really need to consider your approach to data collection and storage. How do you want to consider data that may have value but hasn't yet proven to have value? You don't want to have costs go out of control and most data is never back-cleaned/filled if it wasn't collected and stored for use. But you can't know all your data use cases at the launch of a new application or product. It's a balancing act. There is a question of how mature do you need to be as an organization to actually really consider using data value chain as a framework instead of merely some principles to guide your work. It can be hard to get people to understand the value and what drives that value in data when they don't understand data work in general. Relatedly, really digging into the data value chain can shine a light on underperforming activities inside and outside the data function. So you need to be prepared for some hard realizations and questions. Are you ready for transparency? What aspect of data value chains fall on the business? It's a hard question. At the end of the day, data value chains are supportive of the business value chains/streams but it depends on who has ownership over data work: the lines of business or a centralized team. Your data value chains should have explicit ownership, at least of the different 'links' of the chain. In data mesh especially but true in any data work, it's important to not see the data product as the end of the data value chain. The data product is there to make it easy for producers to reliably and scalably deliver value through data. But there is only value if that data is consumed, the value happens when someone takes action. When launching new applications/products, you have to consider what data you might want to collect even if you don't need it right at the start. Especially if that is something like hardware where you can't augment many aspects of the devices once they've been deployed. Focusing on data value chains is a mindset shift for most organizations, much like data as a product thinking. You need to get people to stop handwaving about aspects of data work and focus specifically on value and understanding that all parts of the data creation and transformation process are crucial to driving rich and sustainable value from data. Even if you do a good job at understanding your data value chains, there will still need to be rework. But it can help you prioritize data rework - you aren't going to get your data preparation perfect, especially for multiple consumers, on the first try. You have to be realistic about your data value. Your company probably won't value data and analytics that are internal facing as much as they do external-facing interactions until you prove out the value of treating those internal users with as much care. Part of that is getting specific about how much value you are generating and how :) At some point in your value chain, you aren't dealing with raw data anymore. Think about who wants what and why. Most execs want aggregated information - again, that point of driving business value instead of data work. Make sure there is clear communication to drive outcomes instead of outputs. A data value chain isn't about getting everything perfect upfront. Everything is about incremental delivery and getting better. What is the cost/benefit of that improvement? Get something out that works and is supportable/stable and then improve. Iteration is your friend. When thinking about your data value chain, it's usually best to focus again on target business outcomes/objectives. After all, that is where the value is. You can get more business people interested in data work if you are constantly talking in their language about their key objectives. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#296 Patience in Product Thinking in Data - Building to Large-Scale Behavior Change - Interview w/ Darren Wood 1:02:58

1 year پیش1:02:58

1:02:58

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Darren's LinkedIn: https://www.linkedin.com/in/darrenjwoodagileheadofproduct/ Darren's Big Data LDN Presentation: https://youtu.be/vUjoJrl_MEs?si=WzB0sBStVIAyqDJs In this episode, Scott interviewed Darren Wood, Head of Data Product Strategy at UK media and broadcast company ITV. To be clear, he was only representing his own views on the episode. Scott note: I use "coalition of the willing" to refer to those willing to participate early in your data mesh implementation. I wasn't aware of the historical context here, especially when it came to being used in war, e.g. the Iraq war of the early 2000s. I apologize for using a phrase like this. Some key takeaways/thoughts from Darren's point of view: Overall, when thinking about moving to product thinking in data, it's as much about behavior change as action. You have to understand how humans react to change and support that. You can't expect change to happen overnight - patience, persistence, and empathy are all crucial aspects. Transformation takes time and teamwork. ?Controversial?: In data mesh, it's crucial to think about flexibility and adaptability of your approach. Things will change, your understanding of how you deliver value will change. Your key targets will change. Be prepared or you will miss the main point of product thinking in data. When choosing your initial domains and use cases in data mesh, think about big picture benefits. You aren't looking for exact value measurements for return on investment but you also want to target a tangible impact, e.g. if we do X, we think we can increase Y part of the business revenue Z%. Zhamak defines a data product quite well in her book on data mesh. But data as a product is a much broader definition of bringing product management best practices to data. That's harder to define but quite important to get right. When thinking about product discovery - what do data consumers actually need producers to create - there is often a big difference between consumers' initial suggested requirements and what they actually want. It's the role of data product management to bridge that gap and deliver what they need instead of what they requested. ?Controversial?: There is a big difference between data product management and regular product management: in regular product management, the ability to take requirements and go away and come back with something months later works. But data is about what's happening with the business now and needs to evolve as the understanding of requirements evolves. Relatedly, data products need to be even less rigid than regular products because they are a reaction to the real world as it changes; you must focus on building something that flexible. Otherwise, you aren't going to be reflecting what matters. When building data products, get to an MVP fast. Get something in people's hands and evolve it to what they need/want. Don't try to get it perfect. When it comes to doing data products well, there is far more collaboration than most people are used to around data work. When considering what to build, data producers need to ask consumers what question they are actually trying to answer rather than what dashboard do they want. Outcomes over outputs. It's about what you are trying to do. Scott note: And then come to an agreement on the specifics - are you delivering data, the insights, or the 'so what'? ?Controversial?: Data products can - and probably should? - absolutely look to address multiple business questions. But when you are thinking about your MVP, focus on what is the most critical question and the minimum requirements to address that question. You can improve the product but getting to first valuable insights is a great initial milestone to build off. ?Controversial?: Similarly, in regular product management, you can go away for six months and come back with something new and shiny but in data, it might take that long just to change two metrics on a dashboard because it took that long to clean up the data. And at the end no one cares, no one understands what the value was, and worst still people are annoyed because they don't trust the new metrics. Relatedly, maybe consider MLP instead of MVP. No, not 'My Little Pony' but Minimum Lovable Product. What is the minimum you can deliver that someone will love - what are the need to haves instead of the nice to haves? What is at least one feature that users will love and can you deliver that early to maintain engagement as you deliver more aspects of value? Actually sit with users and see how they leverage your products. That's a crucial aspect of product management, it shouldn't be any different in data. It can be a bit harder sometimes to get to specifics but that doesn't let you off the hook. It's necessary work to do. ?Controversial?: Bring as many of the folks on the data product producer team as you can into the discussions with the initial data product consumer. That will give a more complete picture to the team. Don't treat the rest of the team as ticket takers internally. And you can probably find and address challenges earlier - e.g. flagging a low quality data source - if everyone is more informed. As with anything in product management, prioritization is crucial. Focus on delivering value, not simply data products. Again, outcomes over outputs. Initial delivery of the data product to a consumer isn't a mark of 'done', it's a great place to focus on how the consumer actually uses and interacts with the data product. Get a sense of the friction points to find places to further improve it. No more throwing things - data projects - over the wall and treating them as done. Teams that understand product thinking in general are easier to teach product thinking around data. But for those teams that don't understand product thinking at all yet, you will need to spend more time with them. It can seem obvious but each team has different educational needs to bring them to product thinking for data. You can't try to win over everyone to something like treating data as a product yourself. You need to find your champions and advocates and then provide platforms internally for them to spread the messages of value and success. It's not only about delivering value but showing that off a bit too to get others excited. Use momentum levers. When choosing your first domains to work with and finding your first data products for data mesh, look for people that are enthusiasts. You want partners early on, not someone you have to constantly convince. Also look for data products that will be reusable to find additional users to provide additional value. It's as much about building momentum as it is about delivering value early. ?Controversial?: Behavior change is a crucial aspect of implementing data mesh. You have to understand behavior change takes time. And you have to know where you will be rigid and where you will be more flexible. If you say my way or the highway, most people are just going to head for the highway. You have to work out what is non-negotiable early. Relatedly, behavior change happens at different paces for different people. That will happen inside your organization too. Look to build up the critical mass, that momentum, over time so people feel like they are joining a successful movement. But you need to do your internal PR to make sure they know about it - data success isn't self-evident. During your transition to data mesh - and is it ever really done? -, you will have data sources that aren't part of the mesh. It will potentially be hard to integrate those with data from the mesh because your data in your data mesh has legal and governance at the core. Sources outside the mesh tend to have one-off policy approaches applied. Be prepared for some tension and consternation. In most cases, people will inherently trust existing data sources over new data sources. That will be a challenge to your data mesh adoption. Even if they objectively know the quality is better with the new source, it's human nature to trust what you already know more all else being equal. Involve consumers heavily in the conversations about what is changing and why. Pushing change on people is likely to cause pushback. No one wants change to happen to them instead of with them. When doing Domain Driven Design (DDD) in data mesh, do NOT try to start with all your domains at once. Look to find ones that are capable enough and that you can learn from to make it easier to partner with other domains in the future. Relatedly, your domain map will change. That's a part of DDD. Don't try to hold onto things rigidly. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#295 Data Shouldn't be a Four-Letter Word - Making Data a Forethought - Interview w/ Wendy Turner-Williams 1:16:25

1 year پیش1:16:25

1:16:25

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Wendy's LinkedIn: https://www.linkedin.com/in/wendy-turner-williams-8b66039/ Culstrata website: https://www.culstrata-ai.com/ TheAssociation.AI website: https://www.theassociation.ai/ In this episode, Scott interviewed Wendy Turner-Williams, Managing Partner at both TheAssociation.AI and Culstrata and the former CDO of Tableau. TheAssociation.AI is "a global nonprofit business organization …focused on bridging the disciplines of AI, data, ethics, privacy, robotics, and security." It is focusing on things like networking and knowledge sharing to drive towards better outcomes including ethical AI. Some key takeaways/thoughts from Wendy's point of view: Right now, we try to break up the aspects of data into discrete disciplines - and then work on each completely separately - far too much. Privacy, security, compliance, performance, etc. Instead, we need to focus on the holistic picture of what we're trying to do and why. Communication is key to effective data work and driving value from data. Hire product managers and focus on the why. Break through the historical perceptions of data as a service organization. Drive to what matters - outcomes over outputs - and focus on delivering value. "What's the point of being focused on the data if you don't understand the business that the data is supposed to be used for?" ?Controversial?: "There is no transformation without automation." If you want data to play a part in transforming the business, you need to focus on automation. Data related work can't be toil work or most won't even do it. "You will never be as successful as you can be as a data organization if you're not able to influence your IT partners, your product teams, your business teams." For far too many companies, data is just an afterthought. It's not the core around how they build out initiatives. When you bolt-on the data to any aspect of the business instead of integrate it from the start and build with data in mind, it's far less impactful. You're always playing catch-up. Make data a forethought. In many respects, data has become a 'four letter word' to lots of people - meaning it has a bad connotation. There are a lot of internal politics around data. Data can mean power and it can also give people perspective on your team's performance. Try to work towards removing the politics if possible but also good luck… 😅 There is so much data in many large organizations that execs can't make sense of it. They often don't understand what data they will need to support their decisions or how to get in place the data they do know they need. There's also often a disconnect between strategy and targets/feasibility when it comes to data. There may be a strategy of grow X product with a target of 'grow X product 15%' but there isn't a good reason why 15% is the target. It becomes a dartboard instead of data feeding into creating the goals. Execution and tactical decisions are powered by data far less often than they should be. There is far too little thought or process around strategy and tactics enabled by and about creating data. Many line of business or domain leaders are simply not great at data. They may be able to leverage insights but they don't get the information cycle, especially sourcing necessary data. Data teams need to partner with them effectively - that is definitely a two-way street. ?Controversial?: Relatedly, too many data people are focused on the data work itself instead of the impact of the work. There needs to be a better understanding of what teams are trying to accomplish with the data work. It's not about the pipeline, it's about the goal of the work and the impact. You need internal processes and clear delineation of ownership or you will have multiple teams measuring the same things and getting different answers. Far too often, people are myopic in focusing on their own job instead of how they fit into the bigger picture of the organization and delivering value to customers. That leads to teams not considering how they exchange information internally, only focusing on their own usage. ?Controversial?: Data teams need to spend more time creating their own data around the impact of data work and impact of issues like data downtime. Move past the service-only/cost-center perspective. Lack of data fluency, especially among execs, causes so many issues. If people don't understand data, they don't understand how much they can trust it and thus won't rely on it. Relatedly, there is a significant lack of understanding of upstream and downstream data and business processes and needs that could be fixed by better communication. What do you need from your upstream and how are your downstream users leveraging your data? Communicate! Automating data work enables business partners to identify "business choke-points" and address them. ?Controversial?: You can't have AI without really understanding your business processes and how data supports those, how people combine data and their degree of trust and understanding of the data. Wendy started out with her perspective that in some respects, data has become "a four letter word". There's so much data and everyone is trying to use it but everyone also feels inundated. Instead of being data-driven, we are data-flooded or data-dragged. And there is a major lack of tying the data work to the actual strategy and execution. Where do we need data to support our decisions? We need a strategy to get that data in place. Relatedly, Wendy sees the major breakdown between strategy and goals when it comes to data. There may be a strategy to grow a product but how much growth is feasible, a good target? Why is that growth feasible? What does the data say about growing that product, e.g. the market dynamics and your positioning in the market? So when goals are set, it is a 'finger in the air' type guess as to how much it could grow or worse, simply how much leaders want a product to grow. And then what data do we need in place to enable the team managing that product to actually be able to grow it that much? How do we enable them to make smart tactical decisions? Basically, it's a lot of things looping back on each other. We need data to set good strategic decisions. But we need a strategy to set up our ability to capture and analyze that data. We need data to make better tactical decisions. But most companies lack the ability to make good tactical decisions to get the necessary data in place. It's top-down driven but far too often the ones who understand what needs to be done for and with data are too far down in the organization and there's a communication gap. Thus, there is a significant lack of being data-driven. We have to admit the problem first. How to fix all of that is another fun process 😅 For Wendy, far too many organizations have data as an afterthought. And that leads to subpar understanding of what's actually happening with the business and lacking the information to fine-tune their strategic decisions. There isn't a strong strategic connection flowing from the business strategy to the data work. Execs aren't spending the time to really follow-through on exactly what the tactics should be to get the right data in place. Scott note: we literally have a panel on doing that, tying the data work to the business strategy and vice versa 😎 episode #251 In many parts of many organizations, e.g. Marketing, Wendy sees there being very competent people who just don't really understand how to do data well. They need a great partner. Should the marketing leader be focused on what data sources they need and why? Or should we be able to translate their needs into the work? But first, we need to actually be able to partner and they need to understand their needs. Data people can supercharge their efforts but the business partners need to lean in. Scott note: in data mesh, part of the role is enabling them to get better. We need people to up their fluency but doing data mesh or not, everyone starts somewhere and we need to help them level up. On the flip side, Wendy also sees how often data people are stuck focusing on the data work instead of the business aspects of what that data work is tied to. Without the business context, all you are doing is pushing 1s and 0s. What do business partners need and why?! There needs to be the ability and the courage to just hammer out the understanding differences or the problems will persist. Wendy also gave some specific examples of too many cooks in the kitchen relative to certain measurements. Instead of there being one official perspective or measurement for something like usage of a cloud product, in a previous role there were many measurements across engineering, finance, marketing, sales, etc. And every single one was different because they all used slightly different methodologies and even sources. So when they tried to look at success of the product, everything told a different story. And when they tried to have a simple bill the customers could understand, it was just not possible. While single source of truth is a complicated and overloaded term, one official source of truth for a question is something you should be able to rally around. A big problem in many organizations is people are only focused on their own job and lose sight of the bigger picture and especially how they play into that bigger picture of the organization's success according to Wendy. Even if your role isn't directly improving the customer experience, your work can have a positive impact on that if you drive towards that goal. Sometimes, politics around data also gets in the way of collaboration across teams and lines of business. Wendy talked about another persistent problem in data: the service model. If your data teams are only focused on supporting other teams, you can lose sight of your big picture impact as well as the impact of bad data. She believes data teams need to spend more time creating their own data around their impact and also quantifying the costs of data issues. What are the actual impacts to the organization? And do execs outside the data team understand data well enough to understand those impacts? If they don't understand data, can they even trust it enough to rely on it? Circling back to the bigger picture, Wendy believes that teams can drive significant process improvements if they just understand the impact of their work - especially through data - upstream and downstream. What do they actually need from others? Who is consuming their data and why? What impact will changes have? How are communications set up to prevent issues and create strong understanding and trust? And then of course, try to automate as much as possible to lower the burden on everyone involved in the data flowing around :) As part of that, please just hire good product managers 😅 Wendy said, "You will never be as successful as you can be as a data organization if you're not able to influence your IT partners, your product teams, your business teams." Data is a team sport, data is about making the organization better. You need others to play with you or it won't work. When thinking about actual business transformation around data, Wendy said, "There is no transformation without automation." Historically, doing data work has required a lot of effort. The business side just wants to leverage the data, help them automate as much as possible. Otherwise many - most? - business partners won't want to engage with the data and leverage data to improve their work - it's too much effort. Also, removing the friction from data work helps people identify the friction in general business processes. So automating the data work allows them to more easily identify and then address "business choke-points". For Wendy, too many aspects of data work are treated as wholly separate disciplines instead of treating it as all part of one whole. Security, privacy, compliance/regulatory, performance, etc. We have to shift it left but also stop trying to treat them as discrete challenges to overcome instead of interoperating aspects of a working, scalable solution. Think data by design 😎 That's why she created TheAssociation.AI, "a global nonprofit business organization …focused on bridging the disciplines of AI, data, ethics, privacy, robotics, and security." She said, "there is no security, there is no privacy, there is no ethics, there is no AI without data," but we also need organizations to actually implement their policies into their data and data work. There isn't going to be ethical AI without someone leading that charge and TheAssociation.AI is looking to push that effort forward. In wrapping up, Wendy circled back to the start. What is the point of doing data work. She said, "What's the point of being focused on the data if you don't understand the business that the data is supposed to be used for?" Being a data leader, especially the CDAO, is VERY tough because you often don't own much of the infrastructure if at all and have to do your work essentially via influence. But if you build the right relationships and understanding of the business, you can still have a major impact and drive significant value for your organization. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#294 Panel: Product Discovery and Data Discoverability in a Data Mesh World - Led by Ecem Biyik w/ Frannie Helforoush, Marta Debska-Barcinska, and Ole Olesen-Bagneux 1:03:15

۲ years پیش1:03:15

1:03:15

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#293 Adapting Product Management to Data - Finding the Customer Pain and the Value - Interview w/ Amritha Arun Babu Mysore 1:05:31

۲ years پیش1:05:31

1:05:31

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Amritha's LinkedIn: https://www.linkedin.com/in/amritha-arun-babu-a2273729/ In this episode, Scott interviewed Amritha Arun Babu Mysore, Manager of Technical Product Management in ML at Amazon. To be clear, she was only representing only own views on the episode. In this episode, we use the phrase 'data product management' to mean 'product management around data' rather than specific to product management for data products. It can apply to data products but also something like an ML model or pipeline which will be called 'data elements' in this write-up. Some key takeaways/thoughts from Amritha's point of view: "As a product manager, it's just part of the job that you have to work backwards from a customer pain point." If you aren't building to a customer pain, if you don't have a customer, is it even a product? Always focus on who you are building a product for, why, and what is the impact. Data product management is different from software product management in a few key ways. In software, you are focused "on solving a particular user problem." In data, you have the same goal but there are often more complications like not owning the source of your data and potentially more related problems to solve across multiple users. In data product management, start from the user journey and the user problem then work back to not only what a solution looks like but also what data you need. What are the sources and then do they exist yet? Product management is about delivering business value. Data product management is no different. Always come back to the business value from addressing the user problem. Even your data cleaning methodology can impact your data. Make sure consumers that care - usually data scientists - are aware of the decisions you've made. Bring them in as early as possible to help you make decisions that work for all. ?Controversial?: Try not to over customize your solutions but oftentimes you will still need to really consider the very specific needs of your consumers. Build for reuse but also build where your consumers are actually having their needs met. A mediocre solution for all is usually worse than a few specialized solutions. Prioritization is crucial in product management. That applies to features within the products but also the products themselves. There are many potential use cases that won't be met because there isn't enough value. That's the name of the game, return on investment; it's not about capturing all value possible. Communication and building relationships/trust are foundational in product management. It's an art as much as a science. If you can't have tough conversations and get alignment, it is FAR harder to build a product that meets customer's needs. Relatedly, establish regular communication with your customers. You shouldn't only be talking to them when things go wrong. Stay on top of what is driving value for them and look to augment your product proactively, not only reactively. Product management requires patience as much as diligence. Sometimes your data product/element violates its SLAs but it's an outlier, a one-off. Don't look to overreact and jump to changing things. But you obviously need to have serious conversations if elements aren't meeting expectations over a more extended time period. If you aren't sure what products you should create in a new area, talk to people and find the points of friction. What are the pain points and is there enough value in addressing them to justify doing the work? It's crucial to deeply converse with potential users of a data product/element to assess if it's really going to be worth the effort. There is always a chance you build something that isn't used/valuable but through deep investigation and ideation with potential customers, you can avoid that far more often. When you are building something, even before it hits 'GA', get validation. You can save yourself a ton of effort in rework as you find a better solution sooner. Product management is about collaborating to drive towards value. You are there to prioritize and coordinate. You don't have to know everything, but your job is to uncover as much understanding as possible to maximize your value creation and minimize wasted work. Always ask what value building something for your customer will drive. But also ask what happens if we don't build it. What is the cost of not acting? The only constant is change, especially in data. Leverage a "loosely dependent architecture" to be able to adapt to change. And be open and honest with customers that things will change. Emphasize you'll work with them to adapt to those changes. Amritha started the conversation on some key differences between software product management and product management around data - whether specific to 'data products' or not. One similarity is the focus on solving a particular user problem but in data, you might have to build something to address multiple users' problems. A much bigger difference is that in data, you often don't own the entire process as you might be reliant on others to source your data. In software, you are generally building the data sourcing because you own the interaction creating the data. How the data is stored and collected throughout the upstream process impacts what you can do. The user problem, the business value, and the user journey are some key guides to doing data product management well for Amritha. Keep coming back to those as you build out your solution. Focus on understanding what the user really needs and work backwards to the sources. And then of course focus on making sure you are actually addressing user needs when you deploy the solution. There are many reasons a data element may not be performing up to expectations so be prepared to deep dive; is there a problem with what you've built, what's feeding your data element - maybe sources have changed or there is a quality issue -, or is it just not performing to expectations because the hypothesis was wrong? Amritha dug a bit more into some challenges specific to product management in machine learning and AI. While data scientists want clean data, when possible they want to even be part of the process of selecting the cleaning methodologies - even that can impact the data enough to change outcomes. So really start from the process of bringing them in as a stakeholder as soon as you can and don't throw data over the wall at them. And if you already have something developed, share your methodologies and help them figure out if it's the right fit for them or if something new needs to be developed. Again, we want reuse but we also want solutions that address their problems. Always a hard set of needles to thread. "As a product manager, it's just part of the job that you have to work backwards from a customer pain point." Amritha questions if you are even building a product if you don't have a customer. What is the business value of the work? For a product manager of product without a customer, are you focused on your own thoughts and biases rather than the needs of consumers? "So the point here is that at any given point, you have to be cognizant of who are you building this for, why, and what that is the primary customer. And the secondary is: who else if I build this, what are the impacts it will have on my secondary customers, or other downstream or interacting applications?" Amritha talked about one crucial rule in product management: prioritize. There are many use cases you _could_ solve but are they actually worth the effort? Think about what will impact your organization the most. Don't try to solve every use case and don't try to make products that can serve every potential customer - focus on delivering value. Scott note: this can be a slippery slope in data mesh. You want to take on use cases you actually can tackle when you are learning. Don't only go for the biggest value but also tackle problems where the juice is worth the squeeze, where the outcome is worth the effort. In product management, Amritha believes it's absolutely crucial to understand the art and the science. The science is more about is this product specifically meeting the needs it was designed for. Basically, measuring the level of success and determining if that's good enough or especially is it _still_ good enough. But even that last bit can be a bit of art. The real art is all about communication and building relationships. If you build the world's objectively best product but no one trusts it or understands it enough to use it, it's not a valuable product. You must build strong relationships and have the tough conversations with stakeholders, earning their trust, to align on what needs to get built and why as well as when a product isn't meeting expectations. Establish regular lines of communication so it's not that the only time you talk to your customers, it's bad news or big changes. Continue to extract information from them to drive to business value. When it comes back to the science, that's when Amritha believes you should dig into the why something isn't meeting expectations from the technical perspective :) And have some patience around that. Sometimes it's a blip on the radar, not anything more. When figuring out what products/data elements you might want to build in a specific area, Amritha recommends digging into the potential workflows and user journeys. Start to really think about what you think could exist and why. But, instead of trying to ideate only yourself, go and talk to people and listen for their pain and points of friction. They may not even realize they have pain but you can find the challenges that people will want to address. Again, work backwards from the user journeys to discover what products you should build 😅 Amritha talked about how to make maximize the chance that what you're building will be used/valuable. A lot of it is simply digging in deep with potential customers in the ideation phase to make sure this will actually drive value. There are ways to do that but a lot of it is simply spending the time to really understand the likely impact of what you're building. As Alla Hale said in episode #122, "What would having this unlock for you?" Also, ask, "what if we don't do this, what is the impact of not doing this?" And make sure to get validation as you're building. It might be the value hypothesis was wrong or that you're building something that is the wrong or suboptimal way to address the challenge/opportunity. You can save yourself a lot of headaches and rework. It's all about that collaboration to drive to value. In wrapping up, Amritha talked about how changes, especially in data, are inevitable. Make sure to communicate with consumers so they have realistic expectations. Sometimes those are proactive changes but often, you don't have that much control over changes, especially coming from upstream in data. Look to build in a way that can adapt and leverage a "loosely dependent architecture". Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#292 Aligning Your Data Transformation to the Business - Interview w/ Nailya Sabirzyanova 1:05:30

۲ years پیش1:05:30

1:05:30

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Nailya's LinkedIn: https://www.linkedin.com/in/nailya-sabirzyanova-5b724310b/ In this episode, Scott interviewed Nailya Sabirzyanova, Digitalization Manager at DHL and a PhD Candidate around data architecture and data driven transformation. To be clear, she was only representing her own views on the episode. Some key takeaways/thoughts from Nailya's point of view: When it came to microservices and digital transformation, we aligned our application and business architectures. Now, we have to align our application, business, and data architectures if we want to really move towards being data-driven. To do data transformation well, you must align it to your application architecture transformation. Otherwise, you have two things transforming simultaneously but not in conjunction. It's crucial to involve business counterparts in your data architectural transformation. They know the business architecture best and the data architecture is there to best serve the business. That is a prerequisite to enable continuous business value-generation from the transformation. Re a transformation, ask two simple questions to your stakeholders: What should this transformation enable? How should we enable it? It will give them a chance to share their pain points and their ideas on how to address them. The business stakeholders know their business problems better than the data people 😅 Your approach to data mesh, at the start and throughout your journey, MUST be adapted to your organization's organizational model and ways of working. Everyone starts from completely different places. Data mesh won't work if you overly decentralize. You must find your balances between centralization and decentralization yourself. ?Controversial?: Historically, teams were charged for data work and resources but with something like data mesh, they can manage their data and data costs far more efficiently. Framework processes, tools, and skills help teams to identify which data is valuable for their own or other domains and requires investment, and which data or data processing operations are redundant, and thus, a source of savings. ?Controversial?: You should consider two phases of your early data mesh implementation: “foundation” - enabling teams to own their data by building corresponding teams, processes, and tools and “operationalization and scaling” - enabling/incentivizing them to share their data well with others. They have overlap but if you don't focus on enabling to own data for themselves, you may have trouble incentivizing them to even own their own data let alone share it. To drive incentivization and prioritization well to do something like data mesh - or really most large-scale transformations - you need top-down support from the highest levels in the organization. ?Controversial?: In highly regulated industries, you will have domains that already have very strong governance practices. Focus on enabling them to safely manage their data within a new framework, rather than trying to change their ways of working. Relatedly, focus on the frameworks and guidelines as well as the tooling to enable those domains that aren't nearly as advanced in their governance. If you are looking to federate/decentralize to all domains at the same time, consider how you leverage central committees. If you don't have someone helping guide people towards some degree of consistency, you can't find repeatable scaling patterns/best practices and are likely to create data silos. The level of importance your company places on your data transformation - mesh or otherwise - should determine how you combine it with your digital transformation. It might be under it as part of digital transformation, entirely separate but at a peer level, etc. But they should get aligned no matter what to have the best impact. Everyone must understand that data mesh is journey. You will learn and adjust along the way. Getting budget for your data transformation / data mesh journey is probably more political than many expect. The easiest way to get budget is high-level management attention. Scott note: but of course, it can be hard to get that buy-in first 😅 Nailya started the conversation on the need for application, business, and data architectures to all be aligned. She gave some history about how application and business architecture were brought into alignment when we started with microservices and digital transformation. But now we have to add data into the mix which makes things even more difficult. We need both the application and data architectures to be designed to specifically support the business goals. When it comes to actually transforming your data architecture, Nailya believes the transformation should be led from the business side where possible. At the very least, the business side should be involved. They understand the business needs best so they can help direct the transformation to serve those needs. Great data work that doesn't support the business needs is often just a well-designed money pit 😅 You obviously need the strong data expertise but a transformation led exclusively by the data team is far less likely to align to business goals and priorities. In a successful past digital and data transformation, Nailya used two simple questions to the key business stakeholders: What should this transformation enable? And how should we enable it? It gave the business stakeholders a chance to fully lay out their pain points as well as some ideas how to address them. That way, the team had a very broad perspective and could come back to each of the stakeholders with solutions that worked for them somewhat tailored to their needs and thoughts. You need repeatable patterns/approaches but you also need people to feel seen and heard in order to drive buy-in where you address their specific pains and ideas. When asked about adapting data mesh to an organization's specific challenges, Nailya pointed to how every culture is so different and you need to take into account how people internally exchange information and work together as you design how you want to go forward. Overly decentralizing - so not doing any federation - or decentralizing too quickly won't work well. You have to find your balance between centralization and decentralization throughout your journey. One interesting buy-in point Nailya mentioned was cost control over data work. Because teams have traditionally been charged for data resources and work by central data teams, they were not as involved in managing costs. Data mesh empowers business teams with tools to control cost-effectiveness of their data, and thus they can identify easier which data is valuable for their business and requires investment, and which data or data processing operations are redundant, and thus, a source of savings. They can see it as a chance to do things better and align better on what work is worth doing - the central data team might have done work that the domain doesn't see as valuable when really considering it more deeply. At first, it might be only for their internal-facing to the domain data work before we can get them bought in that they are now responsible for also sharing their data with other domains. Nailya talked about her experience with a large-scale data mesh implementation. They focused on first enabling teams to own their own data. So again, giving them the chance to gain transparency to their most valuable data as well as define, align, and prioritize their data initiatives. Then, they started to work to incentivize and better enable them to share their data with the rest of the organization identifying new use cases and data customers. This may delay the biggest benefits of data mesh - high-quality, reusable data across domain boundaries - but it does mean that teams aren't struggling to own their data at the same time as learning to share it with others; this also helps with the incentivization challenge as they can take advantage of their data first for themselves before being asked to focus on sharing it with other domains. Data governance in data mesh will - unsurprisingly - be hard for every organization in Nailya's view. If there are domains that already know how to handle their data well, work to enable them to better share their information but also don't try to push them towards central ways of working. If they can safely secure and share their data in a way the rest of the organization can consume it, don't get in their way. But you should also look to create frameworks and standards for those that aren't as mature to help guide them along. Scott note: this is an adjustment for those that are already somewhat decentralized with their data work. Again, adjust for your circumstances! Nailya also recommends central committees to ensure teams are meeting some degree of conceptual consistency and also technical/architectural consistency. That way, you can really find your scalability patterns and best practices. Scott note: if you decentralize/federate all at once, this might be the best bet. But many - most? - are going domain by domain so this function is already embedded in an enabling team. In her experience, Nailya believes that aligning your data and digital transformation is very important to be able to succeed at data mesh. Again, you need to align the application and data architectures with the business architecture. Really take stock - is your data transformation part of your overall digital transformation, are they at the same level and should be partnered, etc.? Nailya again circled back to engaging with your business partners/stakeholders to have them help design your transformation efforts. They will lean in from feeling seen and heard but also, they know the most critical pain points best and can help direct where you should focus first. The conversation finished up around getting and maintaining a budget around your data mesh implementation. For Nailya, this is typically far more political than many might expect. But if you have the proper top-down support and management attention/buy-in, you should at least be able to get going. You have to show value along the way but that should be part of the prerequisite to start: what value are you trying to capture and how will you measure it? Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#291 Panel: Data as a Product in Practice - Led by Jen Tedrow w/ Martina Ivaničová and Xavier Gumara Rigol 1:01:48

۲ years پیش1:01:48

1:01:48

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Jen's LinkedIn: https://www.linkedin.com/in/jentedrow/ Martina's LinkedIn: https://www.linkedin.com/in/martina-ivanicova/ Xavier's LinkedIn: https://www.linkedin.com/in/xgumara/ Xavier's blog post on data as a product versus data products: https://towardsdatascience.com/data-as-a-product-vs-data-products-what-are-the-differences-b43ddbb0f123 Results of Jen's survey 'The State of Data as a Product in the Real World' (NOT info-gated 😎👍): https://pathfinderproduct.com/wp-content/uploads/2023/12/2023-State-of-DaaP-Real-World-Study.pdf?mtm_campaign=daap-study&mtm_source=pp-blog&mtm_content=pdf-daap-study In this episode, guest host Jen Tedrow, Jen Tedrow, Director, Product Management at Pathfinder Product, a Test Double Operation (guest of episode #98) facilitated a discussion with Martina Ivaničová, Data Engineering Manager and Tech Ambassador at Kiwi.com (guest of episode #112), and Xavier Gumara Rigol, Data Engineering Manager at Oda (guest of episode #40). As per usual, all guests were only reflecting their own views. The topic for this panel was data as a product generally and especially how can we actually apply it to data in the real world. This is Scott's #1 most important aspect to get when it comes to doing data - especially data mesh - well. It's the holistic practice of applying product management approaches to data. It ends up shaping all the other data mesh principles and is a much broader topic than data mesh is in his view. But it can also be quite simple in concept when you really boil it down, it just takes patience and focus. Scott note: I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views individually. Scott's Top Takeaways: At its core, data as a product is more an organizational mindset approach than anything else. It is something you work towards. It's not an overnight change but getting the mindset right first - at least with a small core group - will help the organization figure out how to best move towards treating your data as a product. Data as a product and data product thinking must not fall into only thinking about data products - It's product thinking about data! In general product management, it isn't about creating a product, it's about creating an experience for the customer that generates value for them in some way. The best way to do that sustainably in data is a data product. But the value and experience are the point, that's your focus, the data product is merely a vehicle to deliver those. Think about good and bad product experiences. The incomprehensible manuals and unintuitive design of a bad product. The awesome tutorials and documentation around a good one. We need to create easy paths for people to not only discover our data products but discover the best ways to generate value from them. Data as a product includes considering your data sourcing strategy. If you think about a physical good, you don't just have the labor to put it together, you need to consider the materials that need to be combined to create the product. It's not just about what parts you have laying around, you manage the supply chain - hopefully reliably and scalably - to make your widgets. The same should be part of data product practices. It's not just what data you have but what data you will need. If we actually want our organizations to be data-driven (whatever that means to you 😅), people need to be able to rely on the data. If we want to embed leveraging data into the DNA, the day-to-day operations, of the company, we need to make our data creation and management processes reliable. The best way to do that is via data products because it makes it easy and reliable for consumers to leverage data. The data isn't the point, it's the mechanism to drive business value. A first step on the road to your organization treating data as a product is getting away from the data team as a service mindset. That they are ticket takers. Data shouldn't be a cost center or ticket-taking organization rather than a value generating one. A key aspect of good products is usability. We need to focus more on usability in data. That is somewhat wrapped into user experience but there are other aspects that are even more often overlooked than UX. A lot of that usability falls on the platform as well so there isn't a different user experience for each data product. Moving to a data as a product approach, instilling that data as a product mindset in your organization, will be hard. And it is quite a bit of cognitive load for people who haven't focused on data historically. Look to introduce the concept and changes needed to implement data as a product over time. This isn't a switch you flip. Other Important Takeaways (many touch on similar points from different aspects): Data as a product is a lot about the mindset and approach you bring to data. What have we learned from product management - in software and elsewhere - that we can bring to data? Most good general product experiences - at least on the consumer side - don't need a ton of hand-holding. Can we actually get there on data? Do we want it to be that easy since it's still easy to misinterpret the data? This balance will be different for every organization and probably every data product. Relatedly, fully self-service is a real question. You want to lower the amount of information requests to data owners but some of those questions can be value generating. So you want to build out the experience - especially documentation - to answer the basics but there's a question of how far you go relative to trying to document everything. There's a major question about how much change management is involved in learning to treat your data as a product. Is it just a mindset shift? Probably not. But then how do you actually change the organization to start focusing on data as a product? Jen said, "… the primary purpose of data as a product is to maximize data as utility." There isn't a single solution or approach to 'solving' product management. There won't for data as a product either 😅 prepare yourself and your teams for that. It's going to take sustained learning and evolution to get better and better. There is no 'done' but there is a ton of value to accrue along that learning journey. It's okay - if not ideal - to have multiple things in your organization called 'data products'. Not all have to meet the data mesh definition. But make it clear to people what you mean around data products - potentially call them mesh data products - or their thinking on data products will be, "okay, but just what the heck is a data product?" 😅 Software product management is to software products as data product management (the application of data as a product) is to data products. No one thinks software products and product management are the same. We shouldn't in data either. Data work and learning how to do data well both generally have a high cognitive load. Be prepared for that. Don't expect everyone to get it right away, whether that is the why of treating data as a product or the how do we actually do this :) Data as a product will be hard to instill even inside your data team. Again, this will take time and you have to let people know the why and the how. Similarly, you need to be prepared for sustained effort to communicate the benefits to those in the business. People aren't jumping up and down to own their own data and especially not to do it well 😅 Are people ready for the best product approaches in data? Doing product management well is about trying, fast failing, learning, then iterating. Are people ready for there to be more continual change in how data is served? Part of product management is stakeholder management and especially communication. In data, we need to move past requirement gathering but we also have to find better ways to communicate value and get - and then retain - buy-in. Product management, at the end of the day, is about capturing value through creating value for others. With data as a product, you need to understand what value is expected to be created from your work and where you might increase that value. But data doesn't have inherent value unless it is used so you need to stimulate data usage to create value. An important benefit of managing your data as a product is the improvements in data user experience. That means more time spent on creating value instead of wrangling data. Part of good product management is product marketing. That means discovering what data should exist but also finding champions, internally marketing successes, etc. You need people to see the value of the data work to get them to want to lean in. People aren't paying close enough attention to inherently know the impact of the data work, you have to tell them 😅 Products have owners. Treating your data as a product means your data has strong ownership. It's easy to say you are creating data products but really instilling that ownership is crucial to doing data at scale reliably. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#290 Applying Platform Engineering Best Practices to Your Mesh Data Platform - Interview w/ Tom De Wolf 1:05:45

۲ years پیش1:05:45

1:05:45

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Tom's LinkedIn: https://www.linkedin.com/in/tomdw/ Data Mesh Belgium: https://www.meetup.com/data-mesh-belgium/ Video by Tom: 'Platform Building for Data Mesh - Show me how it is done!': https://www.youtube.com/watch?v=wG2g67RHYyo ACA Group Data Mesh Landing Page: https://acagroup.be/en/services/data-mesh/ In this episode, Scott interviewed Tom De Wolf, Senior Architect and Innovation Lead at ACA Group and Host of the Data Mesh Belgium Meetup. Some key takeaways/thoughts from Tom's point of view: Platform engineering, at its core, is about delivering a great and reliable self-service experience to developers. That's just as true in data as in software. Focus on automation, lowering cognitive load, hiding complexity, etc. If provisioning decision specifics don't matter, why make developers deal with them? The key to a good platform is something your users _want_ to use not simply must use. That's your user experience measuring stick. When building a platform, you want to hide a lot of the things that don't matter. But when you start, especially with a platform in data mesh, there will be many things you aren't sure if they matter. That's okay, automate those decisions that don't matter as you find them but exposing them early is normal/fine. Relatedly, make that hiding easy to see through the curtain if the developer cares. Sometimes it matters to 5% of use cases but also often, engineers really want to understand the details just because they are engineers 😅 Make a platform where people can customize their experience where possible without going overboard. ?Controversial?: Few - if any - current tools in data are "aware" of the data product, they are still focused on their specific tasks instead of the target of creating an actual data product. Relatedly, the developers should be able to focus on creating and maintaining data products instead of focusing on leveraging specific tools. We need platforms that allow them to deliver value through creating and managing data products, not a focus on working with tools. ?Controversial?: Data mesh without technology is just theory. It can't only be about the people - if you focus on evangelizing without anything practical to show, it is too theoretical or abstract for people. You need a platform early to be able to show people what you mean. Scott note: you need a thin slice that has at least some aspect of all the 4 mesh principles early or your implementation becomes lopsided. Relatedly, get to something to show people in a demo as soon as possible with your mesh implementation so they can picture it and understand what you're trying to accomplish. In data mesh, you will still have data developer power users that really want to dig deep. But a key focus of your platform should be to make it easier for non-power users to still build and maintain great and valuable data products. Expand the potential number of people creating data products by lowering the bar. ?Controversial?: The platform team shouldn't be a blocker to new data products being developed. However, you should probably have certain cost guidelines/guardrails so someone doesn't develop a very expensive data product - it should only go to a central team for oversight when cost becomes an issue. That way, you prevent unnecessary friction and costs simultaneously. When there is an escalation because of a problem with a data product related to cost or governance, look to frame it as a collaborative deep-dive into an issue. Rather than a central 'you can't do this', it’s much more of a 'why is this made this way? Is this optimal or can we change it?' That collaborative discussion can keep people engaged and leaning in. You can get domains more bought in on something like data mesh/data products by showing them how this new approach won't directly tie data schema to their application schema. That way, they can still easily make changes to their application schemas and not break their data products. Good engineering is all about managing tradeoffs. Platform engineering is no different. You'll have to look at what should be specific versus generic. Orchestration is one area that should be very generic. In data mesh, you want to think about the holistic flow of data and data product work. That's why we need a platform. But the tools aren't really that well built to work together. Be ready for that frustration of having to build on top of the tools to get them to play nicely. ?Controversial?: Even if you aren't doing data mesh, your platform should focus on abstractions. What matters and why is a fundamental question. You want people solving challenges that add value. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#289 Building the Right Foundations for Generative AI - Interview w/ May Xu 51:26

۲ years پیش51:26

51:26

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . May's LinkedIn: https://www.linkedin.com/in/may-xu-sydney/ In this episode, Scott interviewed May Xu, Head of Technology, APAC Digital Engineering at Thoughtworks. To be clear, she was only representing her own views on the episode. We will use the terms GenAI and LLMs to mean Generative AI and Large-Language Models in this write-up rather than use the entire phrase each time :) Some key takeaways/thoughts from May's point of view: Garbage-in, garbage-out: if you don't have good quality data - across many dimensions - and "solid data architecture", you won't get good results from trying to leverage LLMs on your data. Or really on most of your data initiatives 😅 There are 3 approaches to LLMs: train your own, start from pre-trained and tune them, or use existing pre-trained models. Many organizations should focus on the second. Relatedly, per a survey, most organizations understand they aren't capable of training their own LLMs from scratch at this point. It will likely take any organization around three months at least to train their own LLM from scratch. Parallel training and throwing money at the problem can only take you so far. And you need a LOT of high-quality data to train an LLM from scratch. There's a trend towards more people exploring and leveraging models that aren't so 'large', that have fewer parameters. They can often perform specific tasks better than general large parameter models. Similarly, there is a trend towards organizations exploring more domain-specific models instead of general purpose models like ChatGPT. ?Controversial?: Machines have given humanity scalability through predictability and reliability. But GenAI inherently lacks predictability. You have to treat GenAI like working with a person and that means less inherent trust in their responses. Generative AI is definitely not the right approach to all problems. As always, you have to understand your tradeoffs. If you don’t feed your GenAI the right information, it will give you bad answers. It only knows what it has been told. Always start from the problem you are trying to solve rather than the approach you are trying to use. Then evaluate if GenAI is the right approach for that problem. Simple, fundamental stuff but it's crucial to remember: start with the problem before the proposed solution. Many people are leaping to use GenAI because their past approaches to certain problems haven't worked. Dig into those pains. GenAI may or may not be the right approach but either way it can be great for surfacing persistent challenges. Leverage people's enthusiasm for GenAI to have deeper conversations about general business challenges. It can really start to highlight friction points across organizational boundaries and who is responsible for what. Scott note: But as the data team, be careful not to try to fix the entire organization, that's not what you are responsible for 😅 Right now, despite all the hype, most organizations are still at most in small-scale PoCs around GenAI. There is less of an initial focus on return on investment versus what capabilities GenAI might unlock but there is also a focus on what risks GenAI may introduce. Despite the hype, many to most organizations are doing their diligence. May started with three general approaches organizations are taking to generative AI (GenAI): 1) building their own LLMs from scratch, 2) fine tune specific, pre-trained existing LLMs, or 3) leverage pre-trained LLMs as is. Many organizations may want to do the first but it is prohibitively expensive to train your own LLMs from scratch just for the compute and you also need (very expensive) people with very specific expertise to do so. Tuning pre-trained models will likely become the standard approach for many organizations. However, being able to leverage LLMs on internal data in general requires "existing good quality data and solid data architecture." When considering training a model from scratch, May also pointed to time as an issue. Typically, it takes at least three months to properly train an LLM from scratch. Parallel training is helpful but you need to fine-tune results and retrain so you can't just throw compute at it and make the process that much faster. So again, you need high quality data - and you need a LOT of it - plus a fair amount of time plus a ton of money. Once you are in production, it also takes a lot of money and effort to keep them running and tuned properly 😅 Luckily, according to some surveys Thoughtworks did, most organizations recognize training LLMs from scratch isn't the right call for them just yet. May is seeing a trend of people moving away from the 'bigger is better' mentality. More people are starting to explore more targeted and specialized models that have fewer parameters. And often, for specific tasks, they perform better than the first L in LLMs. So we may see a trend towards more and more targeted LLMs/models. Scott note: Madhav Srinath really leaned into this in his episode, #264. Humanity in general has benefited greatly from machines through predictability and reliability according to May. Essentially, if they are made well, you essentially know what you should/will get from machines. But GenAI is designed specifically to act like humans and humans are not predictable and often not that reliable. So people have to get used to interacting with machines that may give wrong answers and are designed - in a way - to do so 😅 We can't expect predictability and reliability from GenAI. Relatedly, when thinking about where is GenAI the right choice versus like traditional machine learning/AI, May believes you really have to dig into the tradeoffs. If you really understand the problem set and what you are trying to accomplish, traditional ML/AI is probably the better approach for you. You need to really understand where the strengths of GenAI will play and feed it the data/information it needs to succeed, otherwise you'll be asking an uniformed and unpredictable entity to solve your most pressing business problems. That's probably not going to go well… May talked about going back to the basics of problem solving when it comes to Generative AI: what problem are you trying to solve instead of what way are you trying to solve a problem and then finding your way back to the problem. It can sound obvious but really, many are in such a rush to leverage these tools, it's crucial to stop and consider. Start with the problem before the solution 😅 GenAI may also surface a number of internal business challenges that aren't spoken about or people have essentially given up on tackling previously according to May. We have a new tool in the toolbox so people want to see if it will be useful to tackle something they haven't been able to address well previously. Lean into GenAI as a conversational lubricant. GenAI may not be the right tool for every one of these challenges but it means there is more internal conversation and sharing :) From what May is seeing, many to most organizations are still in the early experimenting and PoC phase with Generative AI. They are trying to figure out what opportunities GenAI brings and also what risks. Despite the hype, people are taking their time but they aren't as focused on initial return on investment, more to validate if they can actually leverage GenAI to create value. Also, there is strong trend towards domain-specific LLMs rather than general purpose ones, e.g. financial sector or media specific models. May finished on the idea that data mesh and other data management paradigms are crucial to doing something like GenAI right. There is still a strong need for quality data that is accessible, interoperable, privacy-aware, secured, etc. to be able to leverage GenAI well. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Major Programming Announcement 4:25

۲ years پیش4:25

4:25

Announcing moving to one episode per week :)

D

Data Mesh Radio

1
#288 Panel: Master Data Management in a Data Mesh World - Led by Ole Olesen-Bagneux w/ Liz Henderson, Piethein Strengholt, and Samia Rahman 1:04:59

۲ years پیش1:04:59

1:04:59

IRM UK Conference, March 11-14: https://irmuk.co.uk/dgmdm-2024-2-2/ use code DM10 for a 10% off discount! Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Ole's LinkedIn: https://www.linkedin.com/in/ole-olesen-bagneux-2b73449a/ Piethein's LinkedIn: https://www.linkedin.com/in/pietheinstrengholt/ Samia's LinkedIn: https://www.linkedin.com/in/samia-rahman-b7b65216/ Liz's LinkedIn: https://www.linkedin.com/in/lizhendersondata/ Ole's book The Enterprise Data Catalog: https://www.oreilly.com/library/view/the-enterprise-data/9781492098706/ Piethein's book Data Management at Scale (2nd Edition): https://www.oreilly.com/library/view/data-management-at/9781098138851/ Liz's blog: https://lizhendersondata.wordpress.com/ In this episode, guest host Ole Olesen-Bagneux, Chief Evangelist at Zeenea (guest of episode #82) facilitated a discussion with Piethein Strengholt, CDO at Microsoft Netherlands (guest of episode #20), Liz Henderson AKA The Data Queen, a board advisor, non-executive director, and mentor in digital and data at Capgemini (guest of episode #106), and Samia Rahman, Director of Enterprise Data Strategy, Architecture, and Governance at SeaGen/Pfizer (guest of episode #67). As per usual, all guests were only reflecting their own views. The topic for this panel was modernizing master data management (MDM) and applying that to data mesh. It's a very challenging topic to cover because even people's general definition of MDM can be pretty different and there is a question between simply mastering data versus trying to globally compared to locally manage master data. It's a very tricky topic in data mesh. I sometimes use the term 'mastered data' because I think it is far more applicable in that situation than 'master data' - audio goes through a mastering process, so must the master data to actually reach a certain quality level. 'Master data' is more the core linking data. But even that is still just one person's definition. Scott note: As per usual, I share my takeaways rather than trying to reflect the nuance of the panelists' views individually. Also, there was a bit of a misstep around intros if that gets a bit lost in there. Scott's Top Takeaways: The historical impression of MDM - striving for that single golden record - needs to change. Trying to head down that path in a federated/decentralized approach is even more difficult that in a centralized world. And the benefits just keep proving out to not be worth costs. You need to consider your master data management strategy to do data mesh well. Do you want official sources of truth relative to specific questions? Who owns data quality for core linking data? Etc. You don't have to get it perfect at the start but if people don't trust the data on the mesh, your mesh implementation probably fails. Mastering data can improve the quality - especially the provability of that quality - and thus trust. And without strong linking data between data products, do you just have high quality data silos? Relatedly, there is a tipping point in an organization related to size and complexity where you need to consider data mesh. There is a tipping point in a mesh implementation where you need to really start to push MDM. It doesn't have to be on day one but you should plan from the start around MDM. MDM - at least done well - is not about getting to 'perfect' data, it's about understanding the needs of your organization and helping people get to the right quality level and providing core data when it's needed. Not all data needs mastering. Not all data is master data. There's a difference between 'single source of truth' and 'most trusted source of truth'. It's absolutely okay to designate 'sources of truth' for specific questions. Other data sources may provide different perspectives on the same topic - e.g. customer is different in sales, marketing, and finance - but there needs to be one right, repeatable answer for things like regulatory reporting or financial statements 😅 One reason MDM is such a risk to data mesh is that, to be effective, some part of master data essentially has to be centrally managed. You need ways for domains to adhere to central standards, guidelines, policies, etc. Otherwise you risk silos. BUT centrally managed often leads to inflexibility. It can be tough to thread this needle. Look to provide the value add from MDM but limit the overhead and rigidity. How that will apply to your organization will be quite specific but go talk to others in your space to very specifically understand their approaches. No one-size-fits-all, no copy/paste. This is going to be very hard but don't skip it. If you are going to do MDM in your data mesh implementation, much like with anything in data mesh: test, learn, and iterate. Don't do a huge upfront implementation or it will cost far too much and limit your agility and flexibility far too much. Plan some aspects of your MDM implementation out ahead of time but trying to do everything at the start is a massive anti-pattern. There is a massive push and pull in data mesh - one where you have to find the right balance specific to your organization - between master data and maintaining the domain-level meaning and understanding of data in domain-specific data products. This is where the enterprise data warehouse often goes wrong: focusing on fitting the data together (master data) at the expense of its actual meaning and uniqueness. It's a balance. Scott note: I often call this local versus global maximization, this time related to business context :) Other Important Takeaways (many touch on similar points from different aspects): Liz asked an interesting question: if we are creating true high-quality data products, have we by definition mastered the data inside? Is that good enough? The answer to the first is maybe? The answer to the second seems to be no, we need to look at how data in a data product fits in the broader scope of the organization's data needs or we're just creating high-quality data silos as data products. That is that master data versus mastered data. Do we only need to consider data that is broadly reusable across the organization for mastering? It can be quite complex - and political too 😅 - figuring out what data is deserving of being mastered and what data should be master data. Historical MDM approaches have been extremely costly often without the return on investment to justify them. Look to consider a point Khanh Chau made way back in episode #44: if everyone is doing similar cleaning work instead of that being in the data product itself, your total cost of ownership skyrockets. In MDM, when there isn't a clear owner for data because it's so broadly used for linking, that may fall back to the data team. NOT IDEAL but such is life/the real world. Be super clear internally as to your definitions around what MDM means and its application. It will look different in any organization. Much like the fun "what is a data product?" conversation. Dive into the difference between reference data - e.g. standardized country codes - and master data. Data that can be combined but isn't of a high enough quality still needs to be mastered. It's not only about creating interoperable data but data you would actually want to use upon combining it. As Piethein mentioned, MDM should not be managed at the domain levels. That creates a massive mess of contention. MDM is especially key in industries with lots of externally purchased data - e.g. financial services or life sciences - because many domains all leverage the same data. It saves time and money to have that managed once instead of 10s of times 😅 Should data consumers push insights and data improvements back to their data producers? How do business logic and transformations of the data play into MDM? Does it matter what we call it as long as we allow value-add work to easily flow and become scalable? Because of the data products approach with very clear ownership, data mesh may actually make MDM easier than the traditional centralized approach. There is - at least there is supposed to be - clear lineage, documentation, other metadata, ownership, etc. so we can all understand more easily what data we should use and for what purposes. Or at least ask the owners for more information to get a better understanding of that. MDM without strong metadata ownership and management is just like data mesh without them: a disaster waiting to happen. MDM and DDD: there is a complex interplay between how MDM and Domain Driven Design work together. Every domain has its own unique 'language', even a somewhat unique language for communicating externally, but you do need some more broad language of the organization. Getting that crisp so your data products fit a broader taxonomy or ontology is going to be a MASSIVE challenge. If you don't have strong documentation and metadata management, you are far less likely to see a valuable MDM implementation, data mesh or otherwise. People will discover data and use it whether they really understand it or not 😅 Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#287 Driving Data Value Through Creativity, Curiosity, Collaboration, and Communication - Interview w/ Tiankai Feng 56:16

۲ years پیش56:16

56:16

IRM UK Conference, March 11-14: https://irmuk.co.uk/dgmdm-2024-2-2/ use code DM10 for a 10% off discount! Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#286 Mastering Master Data Management in a Modern World - Interview w/ Sue Geuens 54:07

۲ years پیش54:07

54:07

IRM UK Conference, March 11-14: https://irmuk.co.uk/dgmdm-2024-2-2/ use code DM10 for a 10% off discount! Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Sue's LinkedIn: https://www.linkedin.com/in/suegeuens/ In this episode, Scott interviewed Sue Geuens, Director of Data Governance and Product Data at Elsevier. To be clear, she was only representing her own views on the episode. We use the phrase MDM to mean master data management throughout the episode. Some key takeaways/thoughts from Sue's point of view: At the end of the day, if you want to do data governance well, it's about the people. Go talk to them, find out their specific needs and desires and work to tailor your language - and presumably your application of policies when possible - to their situations. People want good data, help them get there! Relatedly, get good at telling stories about data work. Get people to lean in and get them involved. Personalize your communication! While policies and standards are crucial, they are about creating better data for the organization. Try to leverage them as a carrot instead of a stick. ?Controversial?: Don't talk about someone owning data. That's scary for most. Find ways to get them excited about owning the data without making it scary by using different phrasing. The key to doing data governance well is getting people to care. We need them to care about the data because others have to use it. And that means the people are the most important focus. Data governance is too focused on 'governance' and that means oversight. The word governance has a bad connotation for a reason - it makes many potential allies uncomfortable. So governance folks have to really work to make it less scary. Don't focus so much on the data aspects of data work when talking with stakeholders. It's about achieving outcomes through data, not data work itself. Focus on what gets your business partners excited and that's (unfortunately) usually not the data itself. It's easy to fall into thinking about what you want from others in governance. But where you can add far more value is starting with what others want from you and working back towards solutions that accomplish both your and their goals. ?Controversial?: Prioritization in data governance work is crucial. A good method is looking for who shouts loudest for help. They are ready to lean in and are more likely to be leveraged as your governance advocates once you help them. The two big reasons MDM initiatives have failed historically are 1) not having the governance, quality, and metadata embedded into the data and 2) striving for "perfect" data, that single golden record. Relatedly, in data we don't need perfection, the juice isn't worth the squeeze. We need good enough and we need to reflect the realities that our world is ever changing and there are multiple perspectives on the same data that can all be right/correct. It's easy to lose people if you start talking the 1s and 0s of data. Focus on finding stories that resonate with them. If they see the value of the work, you have a much better chance of getting them to actually do the work 😅 MDM in data mesh should be all about "ensuring that you get the right data for the right purpose at the right time for the right person." Sue started the conversation as other data governance experts have - the word governance strikes at least discomfort if not fear into the hearts of many of our colleagues. We need to expect that discomfort and be active in dispelling the myths around data governance as it really is about achieving better outcomes for all. But that means more carrots than sticks, which can be a tall task when it comes to things like regulatory compliance. Basically, it's not easy 😅 Another aspect Sue pointed to is that many - most? - data people really like to talk data. So, instead of talking to outcomes, they talk about the data work, and data work for the sake of data work has kind of been one of the big historical challenges of data - instead we need to focus on the value that comes from the data work. If your business partners are already uncomfortable simply by the phrase data governance, not leaning into their value from the data work and target outcomes is likely to lose them even further. Start the conversation with what they might want from you, not what you might want from them. Sue specifically said she starts partnering with people by focusing on those target outcomes and how might she be helpful to them. Especially, what are their expectations of her? By trying to walk in their shoes, she can come to better conclusions and find working solutions. It's about getting them to lean in. Scott note: and then she can trap them! In a virtuous bi-directional value trap of course… Relatedly, prioritization in data governance is key in Sue's view. What are the problems that really matter? While the "who shouts loudest" test may not point to the most valuable problems, it often points to the problems people value most and thus you can find willing partners. Trying to enforce others to care about their data is a hard road but if people are ready for your help, you can make a huge difference and they are willing to lean in. Those are also likely to be your biggest advocates once you help them, gaining your governance efforts more momentum by leveraging champions. There are many reasons why Sue believes people are skeptical of master data management (MDM). Historically, there were two big reasons MDM projects failed. The first is not really focusing on integrating MDM into the data so not having the governance, quality, and metadata embedded into the data and processes. The second is the drive towards perfection. Instead of focusing on what was good enough, there was this focus on the 'golden record'. That led to inflexibility, poor scaling, high costs, etc. Good data work isn't about being perfect, it's about being good enough. Sue circled back to her focus on working with people. Good governance isn't about perfect data, it's about getting people to care about the quality of the data. That means working to get them to understand what is good enough and why should they care. It's not all just empathy - there needs to be some oversight and making it part of their job - but with humans in the loop, your data quality will be much better if you get people to care about who else uses their data and why. When it comes to actually getting people to understand data governance work - whether MDM or anything else - Sue recommends personalizing your communication. While that may not scale perfectly, again, find your key stakeholders and partners. Stories about data work in a vacuum just don't resonate - Scott note: is there a physics/sound joke in there as there is no air for sound to resonate in a vacuum…? 😅 - Getting people to understand that the work has a purpose and it really is useful to specifically them is crucial. Don't talk to the 1s and 0s of data! When it comes to specifically data ownership, Sue has seen just how scary that ownership word can be. It's not an easy task but we need to find ways to instill people with the excitement around ownership without the fear. Again, easier said than done but it's about getting things to the right place not about doing something right now. It will take time but it's better to do it right. If you don’t take care with implementing data mesh well, Sue believes it will be a far bigger mess than if you didn't try data mesh at all. (Scott note: strong agree) You need to focus again back to what are you trying to accomplish and what data needs to be put in place to do that. MDM in data mesh should be about "ensuring that you get the right data for the right purpose at the right time for the right person." In wrapping up, Sue emphasized the need for personalizing your communication around getting people to do data work with care and prioritize it. You need to be able to speak to them in their language and get them excited about the impact of the work. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#285 Getting Depth and Value From Generative AI - In Data Mesh and in General - Zhamak's Corner 33 19:15

۲ years پیش19:15

19:15

Key points: Thus far, most of the generative AI stuff Zhamak has seen is not that much of a differentiator. They are doing far better chat bots but that hasn't really changed the game. When it comes to any ML work - and GenAI is just a subset of ML work - engineers need data products to make their data work easy. Reliable sources of data, ability to version, etc. Data mesh obviously plays well there. Relatedly, we need to continue to make things easier for people to leverage data products for GenAI. Engineers shouldn't have to spend all their time moving data around and using many systems. GenAI really could be game changing in data mesh but right now we don't have enough information to really do it well. We need far more metadata around things like data products. GenAI often gives extremely shallow answers that just aren't that helpful. If we can get better answers, amazing. But right now, it's not there. Sponsored by NextData , Zhamak's company that is helping ease data product creation. For more great content from Zhamak, check out her book on data mesh , a book she collaborated on , her LinkedIn , and her Twitter . Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Data Mesh Radio episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#284 Breaking Down the Monolith - Incentivizing Good Choices - Interview w/ Frederik Nielsen 1:03:13

۲ years پیش1:03:13

1:03:13

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Frederik's LinkedIn: https://www.linkedin.com/in/frederikgnielsen/ In this episode, Scott interviewed Frederik Nielsen, Engineering Manager at Pandora (the jewelry one, not the music one 😅). Some key takeaways/thoughts from Frederik's point of view: Your data technology and architecture choices incentivize certain behaviors. Consider what behaviors you want before you lock yourself in to anything. Advice to past data mesh self: "construct a data architecture and platform that can adapt to the business requirements and wishes [which] will change over time." Build a composable platform as it's "easier to adapt to changing business requirements." Focus on decentralization features and make it decoupled and composable. Trying to go too wide with your data mesh implementation at the start with all your domains makes it harder to really find your groove and build momentum. Cost transparency can be a big driver for data mesh adoption. Teams want to understand their costs and many organizations are driving cost cutting initiatives. Decomposing the monolithic approach to data means better understanding the cost of individual pieces of data work. Relatedly, when teams are responsible for their own costs, it's easier to spot when someone is making tradeoffs related to cost. It's a more tangible decision and can be a conscious decision to take on tech debt. When taking a concept like data mesh to the highest levels in the organization, attach it to tangible use cases. Make it something that is worth their while, the 'juice must be worth the squeeze'. Focus on the strategic business goals and priorities. It's okay to leverage management consultants. But your data ownership should very clearly be internal - external parties should not own any aspects if you want long-term success. Regarding consultants: "you would rather be driving them than them driving you." It's absolutely normal for some teams to be more data mature than others. If teams raise their hands saying they need help with their data work, your culture is mature enough where teams ask for help and they should get it where possible. ?Controversial?: It's potentially better to focus on your more data mature teams first when going with data mesh so you can move faster early. If possible, create golden paths or pre-configured approaches for less data mature teams to be able to still create data products. It can be hard to show domains why they should move to data mesh. Focusing on use cases is probably the best approach but finding use cases enticing enough to each domain can be a challenge 😅 Tying your data initiatives to the company strategic priorities is crucial to get buy-in. E.g. personalization and omni-channel experience - how do you tie your use cases back to what is most important to the business? At the heart of it, data mesh should be about driving business outcomes - especially the ones people really care about. Focus on that and you will have a far higher chance of success and getting/maintaining buy-in. Make sure you build your data products in a scalable way. That means understanding when you need to put information into separate data products instead of trying to combine it all into one - that is just a mini enterprise data warehouse / microlith. If your data team is remaining quite productive but the backlog is ever increasing as is the time between request and delivery, then your central data team might be a bottleneck. Consider addressing that with something like data mesh. ?Controversial?: Less mature domains can get a more "watered down" version of data mesh as they learn to actually manage and own their data. You don't need to start with the most complicated aspects and use cases first. Scott note: this can be a slippery slope When mapping out potential use cases, ask the amount of effort - if it's even possible - to execute in your existing (non-data mesh) architecture. If it's not possible, data mesh can mean far more data capability for the organization, which can be a great selling point. A decentralized architecture can mean cost savings by getting far more fine grained, e.g. shutting off test environments over the weekend or at night. You can find places to be more efficient far more easily. Frederik started with a bit about how their initial data mesh journey started - and it wasn't great 😅 It was led by management consultants and was focused on real-time data with a very tangible use case. However, two things came from it: 1) a better understanding of what data mesh should actually be used for and 2) buy-in around a very specific use case at the highest levels. So while there was a misinterpretation of data mesh and the use case wasn't the best fit, there was still excitement about the term - and somewhat the actual meaning broadly internally. Making it tangible got people to see the potential benefits. Cost transparency has been a major driver for data mesh internally according to Frederik. Because the costs in a large monolithic stack are very opaque, decomposing the architecture has led to a far better understanding of the cost of individual pieces of work. Because inflation concerns were a big factor for retail in 2023, there was a bigger focus on cost reductions. Being able to give teams the freedom to take different approaches but making them responsible for the costs has led to better cost efficiency - teams can choose more costly methods but those decisions are more exposed. Also, because you have much finer-grained control, there are far more levers to pull when it comes to cost savings, e.g. shutting off test and dev environments at night or scaling up and down dynamically. Frederik talked about a common pattern when moving to data mesh: some teams are more data mature than others. There will be plenty of teams that need help when it comes to data mesh, especially building good data products. They are considering creating a sort of golden path or easy button approach for those teams that aren't as mature as well to make things relatively pre-configured instead of having to make many complex decisions. When driving buy-in at the wider level for data mesh, Frederik talked about pitching data mesh as an entire organization transformation versus pitching use case by use case. He believes it's probably better to focus on the use cases but it can be hard to focus on the complete picture of everything you need when you are also focused on specific use cases. It's always a balance between what is needed only for the use case and what is good for the overall company approach to data. For Frederik, there are two big company strategic priorities: personalization and omni-channel experience (experience across in-store and online). So much of what they have been focusing on is finding use cases that tie into at least one of the priorities because then there will be executive support. Constantly tying the data work back to what people care about shows an understanding of the business instead of doing data work for the sake of data work. However, these are very big challenges across many domains and teams. So making sure to do things in a scalable way and finding the right balance between data products with still high interoperability is crucial. When discussing bottlenecks, Frederik talked about how the measure for the centralized data team becoming a bottleneck was when the time between a data request and the actually delivery was expanding. The backlog was ballooning even though the data team was quite productive. Many people will feel the pain of the increasing time to delivery, leverage that while still showing a productive team. If you are executing well but aren't succeeding, you need a new strategy. Frederik talked about the fact your data technology and architecture decisions will incentivize certain behaviors. A monolithic platform incentivizes monolithic ownership and handing off work, responsibilities, etc. When they introduced Kafka, it enabled them to push ownership upstream to data producers because the new technology allowed data producers to more easily own their data. It's of course difficult to incentivize your desired behaviors but always think about what you want to happen and try to make that the easy/happy path. When it comes to ownership of data, Frederik thinks maturity really matters. When you want to go down the path of data mesh, trying to get every domain to really be advanced with data is just not that realistic. Some teams just don't see data as their focus so if they won't leverage much data for analytical or ML/AI use cases, they are less likely to want to own their data. And less capable quite frankly. Circling back to tangible use cases, Frederik talked about one really key use case that saw a really big uptake that they couldn't really accomplish before going the data mesh route. Being able to tie something to actual impact, that really helped people get more interested. Whether that is a business capability or directly impacting a business metric. Similarly, when trying to find new use cases, the team did a lot of user journey mapping. The data for that user journey lives in many systems so you need lots of teams participating to make the data available but it can have a big impact on business. Many companies probably can't do something that complex in their existing architecture. You can use the inability to do amazing things in your existing architecture as a potential selling point. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#283 Selling Data Mesh to Your C-Suite and Board - Mesh Musing 58 20:05

۲ years پیش20:05

20:05

Quick Summary Points Talk to the business strategy importance - data is there to make things better for the business. What could being better informed mean for your execs? When people ask about the strategy, that is when you can mention data mesh. It isn't about doing data mesh but you also aren't inventing this whole-cloth. 100s to 1000s of organizations are already on the journey. But data mesh is not some magic phrase, it is merely a framing for doing data better at scale. Think of the first hidden data demon from my upcoming mini-book: this is about getting to data driven, not being data dragged. This is about better equipping the people you thought were good enough to hire for their expertise and making them even better. Think of the second hidden data demon: data isn't only about strategic decisions - this gets us into a place where we can make better day-to-day execution decisions too. We don't get to skip leg day. I originally typed 'leg data' and maybe that's what we call the foundations 😅 Please Rate and Review us on your podcast app of choice! Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#282 Not Sweating the Small Stuff in Data Mesh - Interview w/ Mandeep Kaur 1:15:54

۲ years پیش1:15:54

1:15:54

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Mandeep's LinkedIn: https://www.linkedin.com/in/kaurmandeep80/ In this episode, Scott interviewed Mandeep Kaur, Enterprise Information Architect at Nordea Asset Management. To be clear, she was only representing her own views on the episode. Nordea has been on their data mesh journey for a while and Mandeep has been trying to figure out best practices for the hundreds - thousands - of micro decisions in a journey. So how do we get comfortable with making so many calls? Some key takeaways/thoughts from Mandeep's point of view: "1) don't overthink it; 2) bring value out as soon as possible; [and] 3) evolution before completion." The micro decisions in data mesh do matter, give them some thought. But it's important to simply get some perspective from the people who should know best and move forward. That can be from people inside or outside your organization but think about the blast radius of getting something wrong before you fix it. Most times it's smaller than you'd expect. Your first question when considering data mesh: what value am I trying to get out of it? Think about what are the target value propositions and what does it do for the business if this is successful. If you don't have good answers, should you do data mesh? The answers to the 'what value' question of your own mesh journey above should drive your strategy, where you should focus early and what will measure your success. And every organization will have different answers. ?Controversial?: There's a LOT of overthinking in most data mesh implementations 😅 come back to your anchoring points around ownership/accountability, product thinking, value proposition, etc. What's important? You can try something and see if it works and change it if it doesn't, don't get caught in analysis paralysis. Relatedly, always focus on the value proposition. If you are delivering value, you can improve the other aspects as you move along and learn to do aspects of your journey better. There's a major challenge in abstract communication, especially about something like data mesh: those doing the abstracting have so much more time and specific research that there will always be logic leaps and things that only map to their own mental model. Get specific in your data mesh examples and anecdotes while providing abstractions. And be prepared to dive deeper in 1:1 conversations when it makes sense. The internal communication of something like data mesh shouldn't only fall on the data team. Find/create your ambassadors/champions. They can communicate concepts and aspects well in the language of the business and have mental models closer to their peers. ?Controversial?: As the data team, our role is best served guiding people to the right ways to answer their own questions rather than answering things for them. That's part of data mesh but it's a good practice outside it too. With data mesh, you will probably feel it's about 'getting it right'. It's as much about learning how to 'get it right' as it is about actually getting it right. That's product thinking for you. When making your decisions, it's all about trade-offs. Think about what you aren't willing to trade-off. That will guide you more and more to your crucial decisions. Nothing is perfect and nothing is 100% sure. When making your data mesh journey plan, you must take into account your competency gaps - either to fill them before your journey or how you will compensate for them and fill them as you go along. Scott note: ignore your competency gaps at your own - significant? - risk. Set milestones for your data mesh journey so you can measure your success to some degree. It also makes you feel better about the progress you've made. Celebrate the success even if there is far to go! We need to make business users understand data and technology are there to enable them to be better. Too many still see them as a threat. ?Controversial?: Something like data mesh might be seen as threatening to some data consumers by reducing their control and their value creation. Previously, many consumers were in charge of getting access to the data and doing the analysis but that is often pushed to the producer in data mesh. Scott note: this was a really interesting point that hasn't come up before A product is only a product if it's providing value. Ingrain this into people doing data work. Really drive to a common understanding of terms like a data product or data as a product. It's very easy for many people across the organization to interpret/understand them differently making communication extra challenging. Product thinking isn't only focusing on the end product but the supply chain and component parts. That's even more important when that is relating to data; the inputs all matter and the reliability/sustainability of the inputs matter. Transformation journeys are often quite disjointed at the start. You have to align people to start the transformation and people aren't moving together. There is a disjointed phase before the planning phase 😅 That's normal and to be expected. If you focus on the target instead of the journey to get to the target, you're as likely to make things worse. You can't simply jump to the end target state, you have to learn how to get there and improve along the way. This is especially true with data mesh. Mandeep started by discussing one of the key challenges in talking about data mesh: there are so many areas to cover that we often discuss things abstractly. Those abstractions are based on a significant amount of research, discussions, and related work that create a mental model. When we communicate those abstractions, it's hard to communicate the mental model as well. The listeners just aren't as deep into it so much of it goes over their (our) heads. So, we need to get far more specific with anecdotes and examples. We also can't forget the value of 1:1 conversations to drive to deeper understanding. That might not always be the most scalable but it is the best way to prevent misunderstandings. Basically, communication around data mesh is hard! Go talk to people. Scott note: Data Mesh Understanding exists for this reason… When looking at how to get specific internally with data mesh communication, Mandeep is always on the lookout for her ambassadors or champions. Within a domain, they have strong domain knowledge to connect what you are trying to achieve with data mesh to what the domain is specifically focused on. And they can obviously communicate well in the language of the domain. Connecting the changes data mesh brings to real world problems helps people understand the what and the why. There is a lot of risk of analysis paralysis in any data mesh implementation according to Mandeep. There are hundreds of 'micro decisions' but if you focus on the core aspects of what you're trying to do, that should guide you to the ones that matter the most. A bit of don't sweat the small stuff. Always come back to the value proposition because you can change things as you learn more. That's not to say be sloppy or careless, there are important aspects like using the right architecture, having strong ownership/accountability, product thinking, etc. But data mesh is as much about learning to get it right as getting it right. And always return to your trade-offs. What aren't you willing to trade-off and why? Once you answer that, more and more solutions become tenable and you can weigh the pros and cons. Mandeep started to dig into the crucial first question to a data mesh implementation: what is the value you hope to get out of it? And there are different answers for each organization. Those answers will start to inform where you should focus and when in your mesh journey. That will help you set your plan because "a target without a plan is just a dream". And when you form your data mesh plan, think about what you have to adapt to your organization and why. This is not a copy/paste approach! You almost certainly will have competency gaps so how do you plan to fill those gaps and make progress while doing that? Or do you have to fill those gaps before starting the journey because they are journey blockers? Really consider the journey, not only the target outcome. Relatedly, set some milestones for your journey to help you measure your progress and celebrate the progress you've made. They might not be the best success measures once you're further along in your journey but that's okay, you can adjust. That's product thinking. Mandeep wanted to stress three quick points: "1) don't overthink it. 2) bring value out as soon as possible. 3) evolution before completion." Many business users still see technology as a threat rather than an enabler in Mandeep's view. They aren't even thinking about data yet, they are still on just tech 😅So there is a lot of work in communication to get them to see data as a major innovation enabler, something to drive their part of the business to new heights. Another interesting aspect Mandeep talked about was that self-serve might actually be seen as threatening to data consumers. Previously, they controlled - to some degree - their own ability to get access to data but now, it's on the producing team and consumers only get what producers are willing to share. The consumers created the business value by doing the analysis and transformation and now that is pushed much more onto the data producers. Will consumers feel their power and importance is diminished? If the value of data work is attributed to the producers, will data fluent consumers still lean in to leveraging data as much as they did previously? Mandeep returned to product thinking and her view that a product is only a product if it's providing value. You start from the value you are trying to deliver and work backwards. Build your KPIs around actually delivering value instead of simply creating data products with the hope they create value. When thinking about data as a product, Mandeep encourages everyone to have conversations about it in their organization and what it will actually mean and look like in their specific organization. Because it's easy to assume everyone is on the same page when they really aren't. And that confusion will bite you in the end with more friction than clearing it up early. Mandeep believes that in a transformation journey, it almost always starts as somewhat disjointed - a disruptive phase before the planning phase. Part of going on a journey is preparing for that journey. Once people are aligned, that is when you can really start all heading forward. You need pioneers or leaders, those front runners to show people it's safe. But it will still take some time before that alignment. Don't get concerned when that happens even if it feels like everyone should align after the first presentation 😅 Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of December 31, 2023 14:39

۲ years پیش14:39

14:39

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#281 Panel: Data Contracts and Data Mesh - Led by Jean-Georges Perrin w/ Amy Raygada and Andrew Jones 1:06:28

۲ years پیش1:06:28

1:06:28

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . JGP's LinkedIn: https://www.linkedin.com/in/jgperrin/ Amy's LinkedIn: https://www.linkedin.com/in/amy-raygada/ Andrew's LinkedIn: https://www.linkedin.com/in/andrewrhysjones/ Andrew's website: https://andrew-jones.com/daily/ Andrew's book: https://data-contracts.com/ Data contract standard project Bitol: https://lfaidata.foundation/projects/bitol/ JGP's blog: https://jgp.ai/ In this episode, guest host Jean-Georges Perrin, Data Innovation Consultant at ProfitOptics (guest of episode #130 and panelist in episode #227), facilitated a discussion with Amy Raygada, Senior Data Product Manager at Swiss Marketplace Group (guest of episode #165), and Andrew Jones, Principal Engineer and Author of the book on Data Contracts (guest of episode #29). As per usual, all guests were only reflecting their own views. The topic for this panel was all about data contracts and how do we go about getting them in place. Much of it was about the general concept but some of it was specifically about how do we think about data contracts applying to data mesh. This was the first topic I really did a deep dive into in early 2022 and it has evolved but is definitely still evolving. Scott note: As per usual, I share my takeaways rather than trying to reflect the nuance of the panelists' views individually. Scott's Top Takeaways: Data contracts are about trust and understanding. Trust that there is an owner and there are rules, there is a minder that knows this data matters. Trust that things aren't going to break - at least as often as many things in data have historically and they will be told if it breaks. And understanding that what you're getting isn't perfect and there are rules but also limitations. It's no longer buyer beware, consumers can understand what they should get. To do data products well, you almost certainly need some concept of a data contract. Otherwise, you are essentially just putting out a data asset and calling it a product. Products come with guarantees of some sort. Data contracts are about ensuring better outputs with less effort for all parties. They are a quality assurance mechanism but also a scaling mechanism. It's a printing press for data in a sense - reusability where you don't have to carve things in wood each time, you assemble the tiles to have it say what you want but it's more about arranging tiles than defining everything - carving from scratch in this analogy. Standardized aspects of contracts help both producers and consumers communicate about the aspects of a data product. Like with anything related to data mesh - or really any good data practices - you can roll out data contracts over time. It's not a switch you flip and suddenly everything is covered. Start small and find value. Start with one or two teams / data products, figure how this can work in your organization, and then scale from there. While many may see data contracts as additional overhead for data producers, it's quite often a safety mechanism for them. They (hopefully) don't want to break things for downstream consumers but they often don't know exactly how their data is used. Now we have a way for them to understand the impacts of their changes and easy mechanisms to get in touch with the users of their data. Far fewer emergency response tickets to data breakages. Data contracts are very useful - potentially necessary? - when we think about interoperability between data products in a larger context. The contract isn't only about what is in the specific data product but how it relates to the rest of your data products, mesh or not. If you have interoperability standards or linking keys, those are important aspects to mention in a contract. To realize the vision of data mesh, we have to be technology agnostic. There will be tons of vendors releasing their own versions and visions. But at the end of the day, to actually be able to let teams have the freedom to develop their data products to best serve users, we need approaches over tools. Scott note: If you can't tell, I am skeptical of tooling in this space… Other Important Takeaways (many touch on similar points from different aspects): Define your contracts where it's most likely to be updated. That's probably in the code for the data product, not having to go to some separate tool. Circling back to understanding, data contracts set expectations. Literally, they contain the expectations of what you should get with the data product. Expectations setting and boundaries are crucial to good human communication :) As always with data work, data contracts don't come for free. They take time for producers to engage with. Reduce the friction of dealing with contracts for producers but also incentivize them to actually leverage data contracts. Otherwise, it's just a request not a requirement. There are two different main approaches to data contracts when it comes to breaking changes - to collaborate on changes before they happen or to alert people a breaking change has occurred. It's better to be the first but you might start with only the second capability and that's okay. ?Controversial?: Using data contracts only as a blame mechanism when data breaks is missing the point. They can be a GREAT collaboration tool for negotiating between producers and consumers. They are a great starting point for those negotiations and then an agreement tracking and enforcement mechanism. ?Controversial?: As I've noted many times before, contracts can be a double-edged sword. If you have consumers that never meet with producers and share information, that can lead to someone leveraging data they don't fully understand. Contracts can give people trust in the data products they discover without digging deep enough. It's a very nuanced and hidden issue. Like any products practice, you will probably start out pretty raw and unsophisticated when doing data contracts. It's about getting to good, not starting there. Find value, find scale, find repeatability. Iterate to good, here's your permission to suck when you start. Data contracts can behave as great automated communication tools. Instead of trying to find all your users to update them about an upcoming change, it's automatic. Without automation, Amy said data contracts are "just a bit more paperwork." Data contract standards are important but must be extensible. Don't expect a standard to solve all your problems or fit all your needs, especially as they are just emerging. There are many choices you have to make for your organization around your data contract setup. Who owns the data contract is especially important. It should probably lie with the owner but if you don't have clear ownership of a data product/asset, then it's more likely a fact sheet about your data product/asset, not a contract. That said, consumer-driven testing is great in software, will we have some aspect of it in data? Circling back to communication, to get producers to lean into contracts, look to have real conversations with producers about the challenges the organization is having with data and things breaking. Work with them to find a better solution. They are typically software engineers, they like solving problems. Give them the KPIs that let them focus on data and solving things through contracts. It can't simply be more work, it needs prioritization. Data consumers need to be accountable to watching for changes to the data products they use. You need a good mechanism to alert them but if they aren't paying attention and something breaks, that's on them. Everyone has accountabilities. Should you have your security and privacy encoded in the data contract itself? I think it's early days there. It might live in the contract as the place of record for the platform or not. It's an interesting concept. ?Controversial?: Should we try to create the contract automatically and have a human change and validate or the other way around? Probably human with a template works best right now. Automation would be great but there's probably too much room for error. We want super clean implementations of something like data contracts - one standard contract across the organization. But it's just not realistic at the end of the day, especially early in your data contract journey. Every organization is messy in its own way especially with multi-cloud and many platforms, this is no different. How data quality plays into data contracts is a bit more complicated than people think. There are quality standards but checking if a data product actually complies with its SLAs and the standards is another interesting question that people are approaching differently, whether that quality enforcement is in the contract or not. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#280 Enabling Your Domains to Create Maintainable Data Products - Interview w/ Alexandra Diem, PhD 59:40

۲ years پیش59:40

59:40

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Alexandra's LinkedIn: https://www.linkedin.com/in/dralexdiem/ In this episode, Scott interviewed Alexandra Diem, PhD, Head of Cloud Analytics and MLOps at Norwegian insurance company Gjensidige. Gjensidige's approach closely aligns with data mesh but they are starting with a focus on consumer-aligned data products as they have a well-functioning data warehouse and are not looking to replace what isn't broken. Some key takeaways/thoughts from Alexandra's point of view: Advice to past data mesh self: stop talking to people about data mesh, talk to the changes in the way of working. It can be very tiresome to try to explain data mesh instead of those changes. Data mesh isn't the point. There aren't really any reasons we can't apply many software engineering best practices to data, it's simply we haven't done it broadly in the data world. There is a push and pull between software best practices and data understanding. Consider which you see as more important and when. Do you bring data understanding to software engineers or software best practices to those with data understanding. When you leverage pair programming between enablement software engineers and data analysts that understand the domain, the software engineers learn more about data and the domain and the analysts learn good software engineering/product practices. It's a win-win. The people you enable to do work in a data mesh way should serve as ambassadors of your ways of working, especially within the domain. Both helping others learn and as champions. That provides organizational scale. You can't individually enable every person in a large company. "Too many cooks spoil the broth." Think about having that 'two pizza team' kind of approach so you have concentrated understanding by those involved in creating data products who then can again help others learn. This is good for those in the domain and also for an enablement team bringing learnings back to a platform team. Having a team with intimate knowledge of what data products/data product features have been built can speed time to market for other teams and improve reuse. Each time they sit with a team, that new team has far greater access to what's been built before, whether that is existing data sources, existing models or transformations, output ports, etc. ?Controversial?: With a central enablement team, your job is more to teach the domains how to do the work, get them to minimum viable data products. Otherwise, that central enablement just isn't scalable in a large organization. ?Controversial?: A perfectly filled data catalog still won't connect all the dots for consumers. Yes, good documentation is important but there still is a significant value in helping people connect the dots. Scott note: this shouldn't be controversial but is. It's also my 'data sherpa' pattern emerging yet again as highly valuable If you can, make sure you have a shielding and prioritization mechanism for any central team or you can head back down the overloaded central data team as a bottleneck pattern/challenge. As anyone in the organization, your ultimate role is value generation. Consider how the data teams do that. If it's an enabling team, it's helping teams to do data work quicker and better. Those teams don't care that you're doing that via data mesh. Relatedly, terms like data product and self-service platform resonate far more than data mesh. Lean in to what generates value, not the implementation details behind the scenes. Potentially read The Lean Startup to dig deeper into this philosophy. Data reuse is not actually that obvious of a concept to many. Probably because it's meant so much cleaning and manual work in the past from finding poorly owned data sources or processes. Train your domain teams to look for places to reuse what has come before them. !Controversial!: Potentially look to build out your data products from a source of already clean data. That may be an existing data warehouse or something centrally managed. Scott note: this is a data mesh end-state anti-pattern but is it an anti-pattern when in transition? If something isn't broken, do you need to 'fix' it? Relatedly, "[I] don't really see the point of having to destroy value before I should be able to generate new value. I can very happily just generate value on top of the value that I already have." Hypothesis testing and fast fail are great software engineering practices but in data 1) it can be hard to hypothesis test value and 2) it can be quite hard culturally to get people to learn to iterate and embrace fast fail. Alexandra started with a little about her background coming from academia into the commercial world and how that shaped her views of things. She was the first data scientist at a company so before she could really do data science, she essentially had to work as a software engineer; that meant learning many good software engineering practices. When she moved to data, she thought 'we should use these practices here too' and then she also came across Zhamak's posts on Martin Fowler's site and it all started to click. Specifically at Gjensidige, Alexandra was brought in to lead of team of software engineers acting as an enablement team plus a platform team. Their role was to focus on bringing these good software practices - e.g. DevOps, automation, testing, etc. - to the data/business analysts to help them build data products. It has evolved to be more sophisticated but the team is still about enabling people to build data products. Gjensidige already had embedded analyst teams in many of their domains so when Alexandra and team started to roll out the data mesh implementation, there were already a number of data-capable folks who understood the actual business aspects of the domains. That meant there wasn't the typical pushback on the domain actually owning their data products, it was more about enabling them to do so and build a maintainable and scalable product. This process of pair programming between her team of software engineers and the domain data experts means her team becomes more and more data fluent while the domain learns how to write good software code. They specifically leverage a model of two of her team and two of the analysts in a data product creation team to provide enough information exchange but not too much overhead. That intimate understanding of what has been created also helps her team to help find reuse in other domains - they more deeply understand what has been built and can direct teams towards it quickly. That speeds time to market as well for the new team. Lots of wins all around! When looking at the central enablement team's strategy, Alexandra strongly believes in a minimum viable data product approach. Her team only has a handful of people and they have 25 analyst teams to work with. The team has to focus on getting each analyst team to capable via the first data product - again with only two analysts on the team - and then letting those two analysts propagate the knowledge to the rest of their own teams. Otherwise, the central team would be too overloaded. So again, the focus is on teaching the analyst teams how to build good data products and then moving on. Otherwise the central team just isn't scalable or you have so many people in the central team that it becomes far harder to find patterns and share information. The domains have to deliver value themselves so teaching them to do so and then moving on is a sustainable strategy. When communicating with the rest of the organization, Alexandra rarely uses the term data mesh. She points to data product and self-service platform as things that resonate with people and help communicate what she's actually focused on doing: generating value. Most people don't care that the way you are generating value is data mesh. It's simply a mechanism. 'Lean' into that. Scott note: lean is a bad pun here because she mentioned how helpful The Lean Startup is to focusing on value generation. One very interesting note Alexandra talked about was training the domains in reusing data. Historically, it's been very difficult to reuse data because you didn't have the information about how it was created and didn't really have a reliable source. Getting the spreadsheet from a colleague each month isn't that reliable 😅 so, you will likely need to train your domains on reuse, especially finding sources to reuse and how to see if something fits their purpose. That can be the producing team too, teaching them how to share what they've built to other parties that might want their data. Alexandra noted that most of the data products they are building use an existing clean and well understood data source: the cloud data warehouse. They are leveraging a hub and spoke pattern from that warehouse for their products. People already know and trust the warehouse so it made sense to them to start there. Essentially, everything ends up as consumer-aligned data products in a sense. Relatedly, for Alexandra and team, they don't see a need to adhere to every aspect of data mesh, especially at the start of their journey. She said, "[I] don't really see the point of having to destroy value before I should be able to generate new value. I can very happily just generate value on top of the value that I already have." They had some things that were working well already and breaking it all down to fit the paradigm didn't make sense to them. However, she is aware of the additional challenges this can bring and made the conscious trade-off. Scott note: this is an obvious data mesh anti-pattern because the upstream isn't directly from source systems - the teams building the data products don't control the source or their source-aligned data products. But if you don’t have an existing bottleneck from your cloud data warehouse, why fix something that isn't broken? This may become a bigger challenge later - Zhamak has written why not owning source data creates challenges - but if they are willing to take the tradeoff and understand those tradeoffs, is it a bad approach? I don't think so _in their case_ because the data warehouse is functioning well/isn't a bottleneck. Alexandra talked about how to really embrace a culture around 'minimum viable x' in data. In data science, at least there is a good understanding of hypothesis testing but even then, it's often hard to embrace the necessary 'fast fail' model touted by things like The Lean Startup. Trying to understand how to hypothesis test value is also difficult and people have historically seen the challenges in iterating on anything data related. So there is a learning curve but also generally a necessary cultural change to embrace hypothesis testing and fast fail around data. On advice to her past 'data mesh self', Alexandra gave a reasonably common response, circling back to an earlier point: stop talking about data mesh, at least early in the process. Data mesh is a set of guiding principles, not the answer. Talk to people about changes to their ways of working and target outcomes. Why are we taking on change? People hear data mesh and expect it to be some technology or technological approach. You can use the name when people ask for what you're calling the approach but selling it as doing data mesh doesn't help your business partners get it. It becomes a much more tiresome approach to specifically focus on data mesh instead of the ways things change and what matters to them. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of December 24, 2023 28:06

۲ years پیش28:06

28:06

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#279 Driving Towards a Cohesive Developer Experience - At the Expense of Snowflake and Databricks? - Zhamak's Corner 32 18:40

۲ years پیش18:40

18:40

Key Points: We need API-first technologies in data. Not just offering APIs but being able to integrate seamlessly with each other via API. We have that in software but it's been a long-time coming in data. If we want an actual modern data stack, we need to have tooling providers make a real change. Simple made easy: we need to make things simple for data product development and consumption. It's not simplistic but it removes unnecessary complexities. Overall, there is such a trend in data where people aren't building things that remove toil - there is this assumption of increasing complexity of use cases but so much of the work is not that complicated. We need to make it so most people can do most of the work relatively easily without making it overly simplistic - easier said than done of course. Sponsored by NextData , Zhamak's company that is helping ease data product creation. For more great content from Zhamak, check out her book on data mesh , a book she collaborated on , her LinkedIn , and her Twitter . Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Data Mesh Radio episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#278 Data Contracts for the Rest of Us - Approaching Contracts in Evolving Companies - Interview w/ Ryan Collingwood 1:19:26

۲ years پیش1:19:26

1:19:26

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Ryan's LinkedIn: https://www.linkedin.com/in/ryancollingwood/ In this episode, Scott interviewed Ryan Collingwood, Head of Data and Analytics at OrotonGroup. To be clear, he was only representing his own views on the episode. Some key takeaways/thoughts from Ryan's point of view: Have empathy for yourselves and others in all things you do around data. You won't always get it right the first time. Build the relationships, build the trust to continually drive and iterate towards better. In tech, far too often we hear what people need and provide a poor solution to actually solving their needs. It's focusing on the tech instead of the people. Far too many technical solutions/approaches - e.g. data mesh, data contracts, etc. - are really presented for tech-heavy/forward companies e.g. startups. Most companies, large or small, are not capable to leverage the approaches as presented so they must be adapted for 'the rest of us' companies. Scott note: data mesh is like this Far too often, these tech approaches focus purely on the tech instead of the people. That's partially because every org has a different culture so you can't cover them all; but if you only follow the approach as presented instead of focus on the people/ways of working in your org, it's far less likely to go well. You've implemented a great technical solution that no wants to or can use. ?Controversial?: "What are the trade-offs that I can make, while still being true to the value and the benefits that I want to get out of this?" Scott note: SO important to consider when looking at any technical pattern/approach. What is true to the value of the approach? Data contracts really rely on 3 things: at least two parties, an agreement of some kind that is recorded, and access to data that conforms to that agreement. You can add value building beyond those 3 but you have to start somewhere and you can deliver value with something that only satisfies those 3. ?Controversial?: It's hard not to have a sense of imposter syndrome when you actually strip a concept down and implement something that doesn't look like the public examples. That's okay and to be expected. If you're delivering value reliably, you're probably doing something right 😅 The world changes all the time. Your systems will change. Your data sources will change. Your understanding of the world will change. Your processes will change. Create/use approaches that can handle change or you're just going to create more headaches down the road. With a centralized data team, the data team is often considered the data producer, at least to the consumer in a data contract. So with strong testing, the data team can be far more sure about meeting their contracts with consumers. Express data quality "in a way that engages people who are not you." Make it understandable. It doesn't have to be - and shouldn't be - rocket science. Data contracts should be the culmination of multiple conversations. Because then the producer isn't just posting data, they understand the needs of the consumer(s) and can best serve them. It can be incredibly helpful to just go talk to your business colleagues about data in their applications or that they use in the warehouse. They can explain the context of what is going on in the real world but you can show them how that is represented in data. Both people gain a lot of understanding. Similar to data mesh, most people in your organization won't care you are specifically doing data contracts. Talk to them about what changes and why you are doing it and how it will deliver value. Speak in the language of the business, meet them where they are. Also similar to data mesh, you don't need to convert the whole company upfront. Find an ally to test at a small scale and prove value. Use that to learn, get a champion, and show value to gain more converts. Some data isn't worth cleaning if it fails your contracts testing. Really consider what is of value and negotiate that with your consumers. They may want 6 years of data but really only 6 months or weeks is of considerable value. What is the risk, what is the fear of a piece of data being wrong? Really assess that. If it's relatively close, especially for now, will that be good enough? People need to consider that data is never 100% accurate so what is good enough and how comfortable are they with uncertainty. ?Controversial?: Relatedly, signal is what matters, not (usually) exact measurement. Get people used to finding the signal but also get them used to understanding how reliable that signal is and then acting on it to an appropriate degree. If something you are measuring, some data point, isn't going to cause action no matter the result, why measure it? Ryan started off with some framing of how he looks at tech approaches in general but especially how he started looking at data contracts. Most paradigms are presented as if every organization is very tech-y, like a tech startup. With data contracts, much of the content "…there was this assumption that you had multiple teams of people that had a fairly high degree of technical sophistication, … or maybe even data was their primary focus." So when a less tech-y company wants to leverage the paradigm, there is always some adjustments necessary 😅 and when it comes to those types of companies, it’s so much more about the people than the way most paradigms are presented. It makes some sense because every org's ways of working and culture are different but it still can feel very removed from reality for less tech-heavy companies. When focusing specifically on data contracts, Ryan's company is far more batch than streaming. So trying to even leverage the best advice (Scott note: I highly recommend Andrew Jones for that), he had to adjust some aspects to a world where things were a bit more messy and with teams that aren't as data mature. When approaching how to tweak data contracts to still work, he asked the rhetorical but crucial question: "What are the trade-offs that I can make, while still being true to the value and the benefits that I want to get out of this?" Ryan moved into what he sees as the minimum viable value aspects of data contracts. You need two parties, you need an agreement of some kind that is recorded, and you need access to data that conforms to the agreement*. As to the parts of the agreement, Ryan focused on two factors at the start: semantics and data quality. If people can't understand the data can they use it? If they don't understand the quality, can they really trust it enough to rely on it? So they worked to create a data dictionary and also provide people a better understanding of the different angles on data quality. * Scott note: this could somewhat disagree with the idea many have around data contracts of merely publishing data with SLAs because while there is a consuming party, they aren't really part of the agreement, they only choose to use the data based on the existing SLAs/contract around it. There's lots of nuance but I HIGHLY believe in the communication-heavy aspect Ryan and Andrew Jones both present. Often, when comparing with what was presented for a tech-heavy company to what is possible at a more regular organization can be disheartening according to Ryan. The idea that the end picture at your organization should look like the one presented is pervasive. So it's not only hard to adapt the approach but then you wonder if you even captured the value 😅 Can you even call it 'data contracts' or whatever you are working on?! Imposter syndrome is very common here. Scott note: you could definitely call what Ryan and team are doing data contracts :) Ryan also talked about how in data contracts, you must build for change. Change is the only constant after all. So creating systems that don't handle change well is a great way to manufacture more headaches down the road. Much like in software testing, you can more easily tell when something no longer works and needs to be changed. And when the data team is the actual data producer - if the data team are the ones transforming the data, that's often the case or at least is the only group of people consumers talk to with a centralized data team - they are much more sure that what they are doing is correct. Another key learning Ryan had along the journey was that when displaying data quality, make the metrics more easy to understand to the layperson. Historically, data quality has been measured with complex statistics. Most people can't easily read the charts from that to understand what's going on. Make the data quality metrics understandable so people can see progress but also get a sense of how well they can rely on data. It is a sad truth that you can deliver value but if you can't get others to see that value, it isn't valued. Showing that value gets people to lean in. Ryan dug a bit deeper into creating systems that act with empathy. If you approach data contracts as consumers only get what the producer shares, that doesn't end up serving the end needs that well. But if you are treating the contracts as the culmination of multiple conversations, the producer can start to really understand the impact of bad data. How much work do data consumers have to do to actually use the data? This is where empathy and product thinking come in. "…data, as we know, it is merely a side effect of activity, of stuff happening." Ryan believes we need to move past the 1s and 0s thinking in data and focus on what it reflects and how that impacts the people in the organization. Conversations can be hard but they give you the context necessary to maximize the impact of your deep systems work. Talking with people can help both parties bridge the gap between understanding what is happening in the real world versus the data 😅 Internally in Ryan's org, they wanted to review their general processes. Part of that was the uncomfortable truth that change, especially to processes, impacts the data. So that review created a great opportunity to start to implement data contracts. It wasn't about telling people they were doing data contracts, it was about getting people bought in to what value could be delivered if they did data quality and trust better. It just happened to be via data contracts. When actually starting out, Ryan looked for one ally that was willing to take on some of the complexity of dealing with data contracts and saw the potential benefits. Instead of trying to convert the whole organization, it was contained and let Ryan learn how to implement data contracts well in his specific organization. That initial success gave him the confidence to move further and the success story to entice additional partners/allies. Ryan discussed the push and pull of data quality and value. While it might be valuable to have a long history of data, is the cleanup worth it? Really have conversations and make hard choices that align to return on investment instead of merely do consumers want it. Similarly, people need to confront the idea of data being right or wrong. They need to consider what is the cost of some data being wrong, especially slightly off. If that's for a regulator, potentially high. But if it's your weekly marketing leads report and it's off by 0.2%, how big of a deal is that? And how much trust is lost if it's wrong? Can we get people to understand data is never 100% clean/right? Getting people to act on signals will likely be somewhat challenging but it's a better way to navigate than trying to wait for exact measurement in many - most? - cases. Ryan wrapped up back on dealing with yourself and others with empathy. You might not get it right at first but if there's trust, you can iterate towards better together. That goes for your data, your processes, and your relationships. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of December 17, 2023 14:41

۲ years پیش14:41

14:41

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#277 Mesh Momentum Versus Value - What to Choose When and Why - Mesh Musings 57 13:30

۲ years پیش13:30

13:30

Please Rate and Review us on your podcast app of choice! Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#276 Making Self-Service Actually Work Well Safely - Interview w/ Kate Carruthers 1:02:55

۲ years پیش1:02:55

1:02:55

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Kate's LinkedIn: https://www.linkedin.com/in/katecarruthers/ Kate's 'Data Revolution' Podcast: https://datarevolution.tech/ In this episode, Scott interviewed Kate Carruthers, Head Of Business Intelligence at the UNSW AI Institute and Chief Data & Insights Officer at UNSW (University of New South Wales). To be clear, she was only representing her own views on the episode. UNSW is not currently implementing data mesh but are preparing to be able to do so. This is a great lesson in building up the capabilities to move forward towards your goals but not rush. Some key takeaways/thoughts from Kate's point of view: Universities can teach us some really interesting perspectives on self-serve. Because universities are such complex organizations and so many departments are involved in deep investigations in very specific areas, they really are the only domain experts. So enabling them to even just own their own data can be very challenging, let alone helping them share with others safely. Relatedly, each academic researcher is essentially a micro-domain themselves with their own ways of working. That just adds to the need to enable freedom in ways of working but still "keep them safe." Scott note: safety was a key theme of the conversation "At the end of the day, data mesh is about controlling the bits that you need to control, and giving people the freedom to do what they need to do, safely." "Technology is kind of the least of your problems." When it comes to data, be prepared to start with some people not even recognizing there is a problem with the current ways of working or a need to improve. Connect their pain to data immaturity to win them over. The best way to win people over is show, don't tell. Show them the power of self-service instead of pitch them on it. Get a PoC going and get people to tangibly see - and hopefully soon touch - your self-service capabilities early. Always look to anchor your data work - especially things like platform work - to a business need. How will doing the work impact the business? Why is it important to do and to do now? When tying your data work to the overall business strategy for your organization, do NOT forget the people aspect. The relationships matter. Your work on the data team definitely isn't only about technical execution. ?Controversial?: Build a culture around data that is as focused on building human relationships as it is on building data pipelines and platforms. ?Controversial?: To share personal/sensitive information - e.g. PII - a producer should justify why it's appropriate and a data controller should review that. Keep humans in the loop. Giving data owners (UNSW calls them data controllers) a say in how their data is actually used can get them more excited to share their data. It isn't a silver bullet to data sharing incentivization but it adds value to them. Good conversations about access to sensitive data shouldn't be yes/no. They are about getting to what is acceptable and maximizing value within that framing. Get people to share what they are trying to accomplish and partner to best achieve it! Invest in business analysts. They are your front-line to figuring out how to proceed around data and generate value. You need people who can speak business and data simultaneously to drive to great outcomes. Find ways to prevent data puddles, especially places where people are copying data and not securing it well. "People overestimate the power of making change really fast, and underestimate the power of … sustained incremental change." Give people a mental map for change. It removes the fear of the change and lets them lean in. You are creating change with and through them instead of pushing change on them. ?Controversial?: ChatGPT and other GenAI can actually be a great benefit to education. We have to lean in to it as it's not as though students won't have access to these tools in their work life. So getting them to still learn but leveraging better tools is essential to their progress. Kate started out with a bit about the catalyst for her current data journey towards data mesh. About 10 years ago, she saw that universities and especially UNSW were going to "undergo a very big digital transformation and that data would underpin it as an organization. [So] we would need to be on top of our data if we were going to be able to ride that wave." She also gave some color on what running a data office at a university entails. At UNSW, it's split into three general areas of administration, learning + teaching, and pure research. There are some major challenges when it comes to providing data capabilities - especially self-service - to the academic research arm of a university according to Kate. They all have their own ways of working and want - demand? - freedom to work the way they want. Yet, the data team's job is also to "keep them safe." That safety has many facets as well. And the research capabilities of a university can mean some truly world-changing interdisciplinary collaboration. But that only happens if the teams can actually, you know, collaborate 😅 When it comes to the non-research area, Kate believes data mesh is an even better fit. "At the end of the day, data mesh is about controlling the bits that you need to control, and giving people the freedom to do what they need to do, safely." As many guests have noted, Kate believes when it comes to your organization's data journey, "technology is kind of the least of your problems." It’s about people and often even getting them to recognize the problem with their ways of working and how better data maturity will help alleviate their problems. It's not just the data itself but their understanding and relationship to data. Kate and team built a quick cloud warehouse PoC that showed people the ability to onboard new data sources in weeks instead of taking up to six months. Showing them instead of simply telling them really won people over. People could connect moving to a cloud data warehouse to business benefits. They also anchored it all to business needs. Yes, rebuilding their architecture to move to the cloud was going to be work but it meant speed to new data use cases and easier management. When Kate was working to tie her team's work to the overall business strategy, she remained focused on the human relationships and people aspects of doing business. She really recommends building relationships with "customers" of your data work because then they feel comfortable to come to you with more types of problems and challenges. And sometimes that kind of culture/approach isn't for everyone and that's okay. If people aren't willing to treat customers as people, they aren't right for her team. When asked about her frequent use of the word "safe", Kate talked about keeping people from misusing data or even misusing the trust people who provided that data - e.g. the students at UNSW - gave the organization. Anytime someone wants to share sensitive information like PII, there is a data controller that needs to review the justification. Keeping that human in the loop means there is a real understanding and consideration of 'is this okay?' On the flip side, the team has been proactive in sharing information that someone should have access to, e.g. a professor being able to know who is in their class and being able to contact them. Kate mentioned that when they implemented the data controller review, the data producers were much happier. Previously, they had no real say in how their data was used but now, they are listened to. It also strengthened relationships because consumers had to actually collaborate with producers to get access to their data. It's creating interesting conversations and people can get more creative around data to achieve their goals with more data safety. And her investment in hiring a bunch of business analysts has created some great value leverage points. Going back to keeping people and data safe, Kate talked about their struggles with data puddles - where people are copying data into lots of areas instead of accessing the data where it is. And they aren't securing that data well, which leads to more challenges and potential issues. But it's still a process to give people all the access they need and make that copying data less attractive. As like many areas, it's a work in progress 😅 Kate sees the attractiveness of moving fast but believes people need to focus more on sustained incremental change, that they overestimate the value of the former and underestimate the value of the latter. It's similar to transformation versus a change that will revert. Fast changes are far less likely to stick or even work. And people feel less of the suddenness and fight against it far less if at all when it is gradual incremental progress. Another point Kate emphasized was that people need a mental map for change. If they don't understand what is changing and why, they will inherently fight back, even if the change is good for them. It's simply human nature to not want change. So take away the fear of change to make it easier for people. Basically create change with and through people instead of pushing change on them whether they want it or not 😅 The conversation wrapped up around GenAI, especially because Kate is involved in the UNSW AI Institute. She is seeing the open source large-language models (LLMs) improving at a rapid pace, sometimes multiple times a day. And there is a lot of promise even if things are early days. At UNSW, they are figuring out good ways to leverage GenAI in education instead of trying to fight against it like some math teachers did against calculators. It's here to stay so they have to adapt and adopt. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of December 10, 2023 15:18

۲ years پیش15:18

15:18

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#275 Panel: Why Data Mesh Needs Digital and Org Transformation - Led by Benny Benford w/ Nailya Sabirzyanova, Iulia Varvara, and Stefan Zima 1:05:41

۲ years پیش1:05:41

1:05:41

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Benny's LinkedIn: https://www.linkedin.com/in/bennybenford/ Iulia's LinkedIn: https://www.linkedin.com/in/iuliavarvara/ Nailya's LinkedIn: https://www.linkedin.com/in/nailya-sabirzyanova-5b724310b/ Stefan's LinkedIn: https://www.linkedin.com/in/stefan-zima-650229b7/ In this episode, guest host Benny Benford, Founder and CEO at Datent - a data transformation focused consultancy/community - and guest of episode #244 facilitated a discussion with Iulia Varvara, Advisory Consultant in Digital and Organizational Transformation at Thoughtworks (guest of episode #268), Nailya Sabirzyanova, Digitalization Manager at DHL (guest of a soon-to-be-released episode), and Stefan Zima, Data Transformation Lead at Raiffeisen Bank International AG (guest of episode #270). As per usual, all guests were only reflecting their own views. The topic for this panel was transformation when it comes to data and data mesh in general but especially understanding how organizational transformation must play a large part in a data mesh implementation to be successful. And that transformation is not simply making changes, it is making _lasting_ changes. Organizational transformation is a crucial aspect of doing data mesh even if it's not spoken about all that often. Scott note: I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views individually. Scott's Top Takeaways: Transformation means changing something. We aren't starting from scratch. You have to consider the starting points, not only the target end points - and in data mesh, there isn't really an end. Every organization's transformation starting point, whether a data mesh transformation or otherwise, will be unique so adjust your transformation journey plan accordingly. There are so many reasons transformation initiatives, especially data transformations, can fail but a big one is not preparing for the long-term change necessary to make changes actually stick. It's easy to try to make changes but actually making them to last for the long run is something else entirely. There needs to be a sense of urgency to drive forward a large-scale top down-driven organizational transformation. If there isn't a real business reason and one where there is a need - or at least a strong desire - to be addressed in the near-term, you are far more likely to lose momentum/sponsorship. And you need lots of momentum and sponsorship for large-scale sustainable transformation. If you are trying to pitch something like data mesh, speak to real pain points. Just selling the potential benefits instead of solving real, painful existing challenges is not likely to win you as many converts. There's a reason painkillers are easier to sell than vitamins. Transformation and product thinking have a lot in common. Org transformation is treating the organization as something like a product to improve over time. That means prioritization. You can't take everything on at once. Work with your stakeholders to make progress on what matters most. Driving data transformation - data mesh or otherwise - will likely take a lot of education. There is a general sense people should be using data for additional use cases but really, many aren't thinking of the great ways they could use data. Help them find the link between their business priorities/pains and data. Your business partners don't need to know the particulars of your transformation initiative. Sell them a story, give them an enticing vision. Why is this worth doing and what is the payoff? Stop taking them on the sausage factory tour against their will, give them - or promise them - a wonderful sausage tasting party instead. Organizational transformation - data or otherwise - only happens when things change, when they transform. Sounds obvious but you really have to get your business partners to engage or your transformation won't be as successful and is likely to stall/fail. Trying to change the entire organization from just the data team is daunting at best. Other Important Takeaways (many touch on similar points from different aspects): Transformation might not be the best word since transformation implies an end state, an end to transforming. And while you should have some kind of target future state in mind, that's likely to be a target state along the journey rather than an end state. The only constant is change as Iulia said. If you try to break your data mesh transformation down into very separate component parts and transform them separately, it's going to make it more difficult. You need to transform across technology, mindset, understanding, ownership, etc. simultaneously - pushing in the same cohesive direction. The organization has to be ready and capable for a change. Relatedly, you can't start with some massive change or only look to deliver value starting in year 3. Find ways to make progress and deliver value - especially provable/marketable value - along the way. Incremental value delivery is the key to maintaining exec attention and sponsorship, which are crucial to maintaining momentum. Also relatedly, this doesn't mean your progress on different aspects will all be at the same pace. Maturity, buy-in, capacity, etc. will determine how far you can transform different aspects and when. Don't try to wait for everything to move together. A data mesh transformation driven from the bottom up and mostly by the data team is possible but will likely be harder - far harder? - than top-down. You need to constantly win more support but that can also have its advantages than something with fanfare but not a lot of specific direction. Prioritization is key to doing org transformation well; so is measuring progress. If you aren't addressing the real pain points - or at least the pain points your exec sponsors care about 😅 - you will likely lose that sponsorship. Show them you are making progress against those pain points. Get your business partners to tell you about their actual pain points. Not just about data but about areas where data may be able to improve their work. They will often literally tell you how to sell doing something like data to them by making them feel seen and heard, actually creating a plan to address their specific pain. Relatedly, work with stakeholders to define success metrics around your progress. If you can continually show them incremental value and that you are addressing their needs, you are far more likely to be successful with your transformation initiative. But getting to clear metrics around the data work will be _hard_. Scott note: I'm writing a book on this for a reason 😅 ?Controversial?: If you want to 'prove' value from data work, create a way for other teams to measure the value created. A data team claiming they created value versus Finance claiming the data team created value is a world of difference when it comes to credibility. When it comes to prioritization of data transformation, should the data team really be setting the priorities? For certain aspects like the platform, probably. But really, the business should tell you what are the highest priorities where you should focus your work. ?Controversial?: The head of the data org should be there to enable other parts of the business to derive value from data. It's about making everyone else better. Because of the central nature of many - most? - data teams in large organizations, too many people are used to them essentially being free - incremental data work doesn't typically cost the line of business or at least doesn't cost much. Transforming that mindset to get them to focus on extracting value from data work might be challenging. As constantly comes up in almost every data mesh conversation, incentivizing data producers is hard. You should try to create structural incentivization at the organizational level to push some of the value created back to the data producers. Data mesh isn't the point. It should never be the point. Zhamak has said this as well. We are looking for ways to achieve our goals and data mesh (hopefully) provides good framing to do that. At the end of the day, data work should be about impact. Focus on impact with your business partners and they will be far more likely to continue to engage. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#274 Your Data Platform is a Product, Treat it Like One! - Interview w/ Sean Gustafson 1:01:32

۲ years پیش1:01:32

1:01:32

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Sean's LinkedIn: https://www.linkedin.com/in/seangustafson/ In this episode, Scott interviewed Sean Gustafson, Director of the Data Platform at Delivery Hero. Delivery Hero has been on the data mesh journey for longer than most organizations, at least over 3 years. Some key takeaways/thoughts from Sean's point of view: It's extremely hard but still important to try to impact your culture through things like your data platform. Who are you trying to make information available to? How do you make it accessible? How do you make data ownership easier? A key role of the data platform is that golden/easy path. Showing people easy ways to accomplish what they need with data products. Embed best practices into the platform when possible. You need a product manager in your data platform team. It's easy-ish to build cool things in data but understanding and building to user needs is harder and a must. Treat your data platform as a product! Relatedly, there isn't anything all that special about product management around the data platform. You can take what we've learned from other disciplines - especially software - and tweak it a bit for data. But it's not some arcane art. Focus on KPIs around what you are building and why, especially for your data platform. It's very hard to measure developer productivity but that doesn't mean you just don't measure it. ?Controversial?: Be prepared to deal with a lot of qualitative data when measuring success around your data platform. Surveys work far better than most might think. Good product managers balance the short and long-term. You don't want to make drastic and breaking changes to your data platform often but that doesn't mean you can't take bigger bets and shake things up. Just balance iterative improvements and the bigger picture. Scott note: Zhamak talks about Thomas Kuhn and cumulative progress versus paradigm shifts In the same vein, make small bets where small bets will do but don't be afraid to make big bets when necessary. ?Controversial?: It will be hard to iteratively change a traditional centralized-focused data platform to do data mesh/decentralized ownership well. You want to at least consider a fresh start when looking at your mesh platform. Tools like dbt have given a much broader group of people the ability to model their data. There are inherent problems if they don't do it well but we still need to encourage more people to do data work so we can get them better and producing great work. Data products are a lot like APIs. There are many best practices we can take from APIs and apply them to data products. Explaining data mesh to software engineers can be tough. They probably get the concepts given most are just software engineering concepts reconceptualized for data. But the biggest challenge is they will probably see data as a second class citizen to the underlying back-end systems. Scott note: Unfortunate but extremely common In data incident management, e.g. data loss, you have to look at the prioritization but our general historical focus - how much money did we lose - just doesn't make sense in data. We have to take reliability engineering practices from software and tweak them to work with data but we can take incident management essentially as is. We just have to understand prioritization far better. Sean started with a bit about how he sees his role as leading the data platform team. It is very challenging but still important in his view to try to shape culture even through the data platform. There are so many places in data mesh where there is friction, how do you make things easier as everyone transitions to product thinking and decentralized ownership? Just because you have mandates from the top, people need new ways to accomplish new goals. Make your platform reflect the type of data culture you want. Instill in people the understanding that they can and should participate in your data culture/work. Easier said than done of course. Relatedly, Sean believes the data platform should show people the right way to do things, give them that easy path where possible. But still give them the freedom to do some aspects … not so right 😅 Treat your data platform as a product is something Sean strongly believes in. And to do that, you need someone acting as a product manager. It's not rocket science, we know how product management works and it's not very different when it comes to building a data platform. But you need someone specifically focusing on user needs. And part of that role is also to advocate new features and using the platform. Just because you built it, that doesn't mean people will use it. When asked about iterating to good, Sean talked about how in product management, good practice is about making constant and small improvements but also balancing the bigger picture/big bets. It's not always about the big new platform but sometimes, it's okay to shake things up - make small bets when small bets are good enough but make big bets when necessary. But you have to do that by balancing the short-term and long-term picture. Fail fast and iterative improvements are crucial to good product thinking in software and we need to apply that to data. But again, big changes are okay if you properly build to them instead of trying to flip a switch. He specifically mentioned that it will be hard to iterate to a platform that does decentralized ownership well from one that was highly centralized. Not impossible but at least consider building that out more from scratch. Sean talked about Generative AI and how it's starting to change lots of people's views internally about data. While previously, many software teams were at best reluctant/hesitant to model their data, there is a big interest from the software engineers to directly interact with the large language models (LLMs). Tools like dbt previously brought many new people to the data party, making it easy to model data - at least structurally - so hopefully GenAI will mean more people learning to model their data. There are inherent challenges but the more the merrier when it comes to people working to produce good data. We just have to make sure they learn how to do it well 😅many who are new to data modeling do it… not so well… When it comes to product management, you need to measure how well you're doing. For Sean, that of course extends to the data platform. While KPIs can be somewhat hard around your data platform, that doesn't mean you get to slack off and not measure things. At Delivery Hero, right now they are using surveys to measure a number of things around their data platform rather than trying to measure things automatically without context. It also creates a lot of conversations in the data platform team about what are you trying to do and why, which prevents a lot of waste. It's not perfect but it's getting better. Scott Note: this is why I am writing a book on success factors then one on success metrics in data mesh 😅 this is HARD Sean talked a bit about APIs and how much data products _should_ be treated like APIs. Not just versioning but tracking usage and having users register to use them. There's a lot to learn from how APIs evolved so we don't have to make the same mistakes in data. Scott note: Zhamak comments on this VERY frequently that API approaches are crucial to data mesh When talking to software engineering people, Sean has found using data terminology, especially data mesh terminology, doesn't really resonate with them. We probably need to come up with new terms - or potentially use the terms Zhamak took from software and just make them about data too instead of inventing new terms. But be prepared for it all to fall back to that most software people will see the back-end systems as more important than the data. If you get them over that hump, it's far easier to get them bought in on data mesh. You may be able to win them over by showing them how the data is used internally. Incident management in data is still pretty nascent in Sean's view. While on the software engineering side, there are very well established processes, often in data it has been more slapdash at best. No escalation, no prioritization, no formal process, no post mortem + shared learning, etc. The traditional measure around data issues - how much money did we lose - often isn't applicable to data. So we have to rethink what matters and why because our prioritization is often skewed. Sean wrapped back to the start about how important culture is. Not just getting your organization to be data driven but setting up more and more people for success in your organization through their work with data. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of December 3, 2023 23:41

۲ years پیش23:41

23:41

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#273 An API-First World in Data Integration - An Actual Modern Data Stack - Zhamak's Corner 31 22:52

۲ years پیش22:52

22:52

Key Points: The rush to categorize all of our tooling in data has caused many issues - we will see a big shake-up coming in the future much like happened in application development tooling. So much of data people's time is spent on things that don't add value themselves, it's work that should be automated. We need to fix that so the data work is about delivering value. We can learn a lot from virtualization but data virtualization is not where things should go in general. Containerization is merely an implementation detail. Much like software developers don't really care much about process containers, the same will happen in data product containers - it's all about the experience and containers significantly improve the experience. The pendulum swung towards decoupled data tech instead of monolithic offerings with 'The Modern Data Stack' but most of the technologies were not that easy to stitch together. Going forward, we want to keep the decoupled strategy but we need a better way to integrate - APIs is how it worked in software, why not in data? Sponsored by NextData , Zhamak's company that is helping ease data product creation. For more great content from Zhamak, check out her book on data mesh , a book she collaborated on , her LinkedIn , and her Twitter . Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Data Mesh Radio episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#272 Understanding and Valuing Your Organization's Data - Interview w/ Lauren Cascio and Chris Ensey 55:46

۲ years پیش55:46

55:46

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Contact email: Swimwith[at]gulpdata.com Lauren's LinkedIn: https://www.linkedin.com/in/laurencascio/ Chris' LinkedIn: https://www.linkedin.com/in/censey/ In this episode, Scott interviewed Lauren Cascio, Chief Fish Wrangler, and Chris Ensey, CTO at Gulp Data. From here forward in this write-up, L&C will refer to the combination of Lauren and Chris rather than trying to specifically call out who said which part. Some key takeaways/thoughts from L&C's point of view: ?Controversial?: Many organizations have an incorrect perspective that they mostly have a single type of data that's useful for each use case or need. Typically, their data is useful for many more internal use cases and also to organizations in far different industries. Often, there is a lack of a data sharing culture in many organizations. There isn't anyone that really understands how data flows throughout the organization or especially how it _could_ flow to serve many untapped use cases. There are many people emotionally attached to owning their own data but not in the product sense, they are focused on maintaining control rather than structuring it to be shared. So there are organizational challenges to data sharing in addition to technology. Many organizations have a tough time justifying updating their data infrastructure, leading to more and more challenges with progressing their data journey. It's often hard to point to a tangible ROI on updating the data platform for instance. Far too often, companies and LOBs know they want to analyze some information but they don't really know what they are analyzing it for. Instead of shaping data to make specific decisions, there is a focus on the visualization without a clear action in mind once the data tells them something. Drive towards what you care about and use data to answer those questions, the data doesn't speak for itself. Your upper management has limited patience and a limited attention span. Focus on what matters to them and be crisp on delivering an outcome with data, not outputs. A dashboard is just a pretty picture unless it drives action/creates insights. ?Controversial?: It's often easier to get funding to prepare your data for external sale than investing in internal use cases. The simple reason is a tangible ROI. Look to frame your internal investments in data in the same way. Find ways to open more communication about what data you have internally. You will be surprised by the number of new use cases emerging. There's so much untapped data internally that people would use if they only knew about it and could easily use it. Many organizations aren't really thinking of the value of their data and how they protect it. If the data is so valuable to your organization, what kind of investment are you making in security and compliance to protect it? !Controversial!: People's personal data getting shared, at least with some modicum of regulatory oversight, is for the greater good - e.g. more patient data to help fight disease or more financial information to help unbanked people get access to credit/capital. If an organization wants to understand their overall data landscape, the best way to start is simply by starting and also having an end purpose in mind. Essentially, get conversations going and know why you are trying to understand your data. Is it to unlock new use cases, save costs, sell your data, etc.? L&C started with discussing how many organizations view their internal data landscape/estate and how it's not a complete picture. There tends to be a perspective that an organization's data is only useful for their internal use cases and often that each set of data is only useful for one type of use case. And L&C just haven't seen that be true - internally, most orgs have data that could be useful to existing use cases . How that typically manifests is data silos where data that should be shared isn't because people aren't aware it exists. Or the other side is that data producers have no real idea of how their data is being used downstream by other parts of the company. Externally, most companies' data is often very useful to other organizations in entirely different sectors. When asked about why lines of business have such a hard time understanding what data other LOBs have, L&C talked a bit about the technical challenges but much more about the organizational. In many - most? - organizations, lines of business have treated their internal data as overly precious, making sure it was structured specifically so they could use it. Trying to get them to structure it so others can use it is an emotional hurdle because it can feel like giving up that control. Thus, it's been hard for other LOBs to even know about the data across the organization, let alone use it. Add to that the challenge of businesses treating the data team and especially their infrastructure as a cost center and that further impedes their data journey. When it's hard to make the tangible business case for updating your data infrastructure, it's easy to fall further behind. If much of this sounds familiar, it's frequent hurdles towards implementing data mesh. L&C talked about how typical it is where organizations understand they want to visualize their data but without a specific goal in mind. Just visualizing without an expectation of what it will be used for is not product thinking. What information do you need to make your external facing products better? What will cause you to act? Instead, it's about "what does the data tell us" which is not often aligned with taking actual action. That leads to wasted cycles and money; it also often leads to wasted buy-in from upper management - they really only have limited patience, spend it on what matters. Be crisp on what goals you are going after then develop the data and analysis to help you actually go after those goals. It's far easier to get exec buy-in on selling your data externally than investing further for internal use in L&C's experience. That's because there is a tangible outcome at the end of the road. Look to try to shape your asks for additional funding based on that principle: a tangible ROI makes decisioning easier. For L&C, there are many use cases that could be unlocked in most organizations if only people knew what data was available. Finding ways to discover and share more about what data you have internally is very helpful. Yes, a data catalog is great but finding better ways to make people aware of the available data will unlock new valuable use cases. An audit for what data to sell externally is one way to spark these conversations but there are many others :) L&C pointed to two differing types of companies regarding selling their data. The first is low margin businesses. Because they are so reliant on volume, they end up with a considerable amount of data that they could potentially monetize. The other type of company is early stage companies that have yet to reach product market fit, especially B2B. They often think their data will be very valuable but selling data becomes a distraction far too easily. Focus on your core business, not small external monetization streams. On the somewhat controversial topic of data monetization, how people's information is protected versus leveraged, L&C believe there is a greater good in general to your information being shared. While something like GDPR gives the perception of your data being protected, it's not really all that true - everyone's data is out there already 😅. Meanwhile, there is lots of potential good that can come out of more comprehensive data sharing, e.g. better information to fight diseases from more patient information or lower cost of items in retail stores from data generated in loyalty programs. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of November 26, 2023 14:30

۲ years پیش14:30

14:30

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#271 The Importance of Repeatability of Language to Scalability - Mesh Musings 56 11:22

۲ years پیش11:22

11:22

Important points: There are places where nuance adds value. Many times, explicit definitions around data aspects like quality or even SRE metrics like uptime and query performance are not one. Provide a simple way for producers to apply these scalable approaches - the platform should measure data quality metrics for example. Data producers are having a hard enough time in general learning how to leverage data better. Find places to make it about learning about the information encapsulated in the data product, not learning a new set of SLAs for each data product. Consumers will thank you too since it make their lives easier. With that, you should see more of an uptick in data usage. Please Rate and Review us on your podcast app of choice! Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#270 Sustainable Data Transformation to Drive Towards Data Mesh - RBI's Journey So Far - Interview w/ Stefan Zima 1:13:44

۲ years پیش1:13:44

1:13:44

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Stefan's LinkedIn: https://www.linkedin.com/in/stefan-zima-650229b7/ In this episode, Scott interviewed Stefan Zima, Data Transformation Lead at RBI (Raiffeisen Bank International AG). To be clear, he was only representing his own views on the episode. Some key takeaways/thoughts from Stefan's point of view: No one has data mesh all figured out. Go talk to each other. But also don't be ashamed that you are running into challenges. So is everyone else. Data mesh implementers also need to share more of the anti-patterns they are finding. Agile transformation really focuses a lot on communication and transparency. Both are very crucial to really any successful transformation initiative. Humans struggle with uncertainty and change so giving them a lot of information especially about the why prevents unnecessary pushback. Relatedly, there are many things we can take from Agile transformation practices to apply to data/data mesh transformation. It's not a copy/paste but there's still much that is very relevant with some tweaks. Many organizations are still focusing on technology-led transformation, whether data or digital in general. You must also change the mindset and organizational approaches if you want to be successful. In banking, the rise of fintechs (financial technology companies) has made it clear that being nimble and quickly acting on data is crucial. Being data driven is required to remain competitive. Data mesh can mean far less friction in getting to serving use cases. Instead of fighting against the data protection office, they are involved from the start. That time to market is especially crucial in banking now. If you can, look to make your data sharing policies and approaches generic enough to only create friction when there truly is something different that should be examined further. If you really want to be 'data-driven', if you really want to be a data company, you have to find and address the friction points in your data processes. Stop trying to simply get better at processes that stopped scaling and have high friction, find new ways. Don't try to sell everyone on flying to the moon when you're struggling to get the garbage taken out every night. Get grounded in people's day-to-day challenges and sell them on getting better and better on delivering on them so they actually can start to get buy-in to the bigger picture. Relatedly, talk to the vision and the goal but show the pathway to getting there. You can't just drop product thinking on the organization and expect overnight uptake. Communication is incredibly important to transformation. You need top-down support, not just behind the scenes but in day-to-day internal communications. People need to understand transformation is a priority. Leverage your communication department. To maintain mesh momentum, you really do have to prove out value and then get further buy-in from upper management - you don't just get to build without proving value. That proof of value will add more positive pressure on other domains to get on board. To get people bought in, focus on incentives. Words and asking are all well and good but really, incentivization is the key to getting participation and buy-in. In data mesh, it's easy to get too focused on either the small scale - only focusing on use cases - or the overall implementation. As a leader, you need to split your focus between the execution and the big picture. At some point, mesh needs to scale to make it worthwhile so that big picture matters even early. ?Controversial?: When doing an Agile transformation, the role of Agile coaches was quite helpful. Is there something similar that could work for data mesh? Take the data product concept and find a clear definition for your own organization. And it's okay to have something that extends the concept. For example, at RBI, a dashboard is a business product, not a data product. When it comes to risk mitigation, the Data Protection/Privacy Officer can just say no to every request and there is no risk taken, so job done, right? As data people, our job is to clearly explain the risks and how we are mitigating them to make it easy for the DPO to say yes. Talk to them early to design compliant solutions. ?Controversial?: Self-service is more than just providing access to the data and it's certainly more than just tooling/the platform. You have to teach people how to find, access, consume, and especially understand the data. And then teach them to also share what they've learned via sharing data in a safe and scalable way. While incentivization is crucial, it's also often hard. E.g. what is the incentive for someone to put really complete metadata around a data product into the catalog. Can't any users just come to the owners with questions? When you start your data mesh journey, make sure to start with FinOps. So much of data work is opaque, get your arms around costs and also cost savings. This will prevent challenges down the road 😅 Simply put, transformation takes time and energy/effort. If you try to do it as a big bang, all you do is blow things up. Stefan started with a bit about his background and current focus. From having worked in data and transformation, the current transformation needs - data and digital in general - for many organizations are accelerating. There is a need to move towards more modern platforms and a self-service orientation of course but the organizational change especially around mindset cannot be overlooked. As for why the banking industry is embracing data mesh and being data driven, Stefan mentioned that over the last five years or so, there have been many major industry changes. One aspect is the rise of fintechs that are built from day one with data in mind - to stay competitive, more traditional banks need to be able to move extremely quickly and the best - only? - way to do that intelligently is with data. Another aspect is simply the domains are getting more and more capable around leveraging information to gain competitive advantages so they need to feed that with quality data to win in the market. When the bank started to see more and more friction - e.g. more pushback from the data protection officers to comply with ever advancing and increasing regulation - Stefan and team realized they needed a new approach. Instead of trying to make improvements to the existing processes, there needed to be new ways to get things done instead of relying on existing bottleneck creating interdependencies. That's part of what led them to data mesh - taking points of friction and shifting them left and reducing handovers meant faster time to market with new information and use cases. Stefan talked about how many people in transformation and data push too far, too fast with their vision. When people are stuck in their day-to-day, getting them to imagine the company that could be in 5 years is at best inspiring but often not and it certainly doesn't help them today. Help them connect that vision of that 'data-driven future' to what you will do for them now. Don't expect people's mindsets to shift overnight. Relative to transformation, Stefan discussed how important internal communications is. There is of course the messaging to drive understanding but also the communication by the leaders to show support. If you lack visibility of your top-down support, it's much harder to get people to take your efforts - data mesh or otherwise - seriously. At RBI, Stefan noted that they are through their data mesh PoC phase. They were able to prove out significant value to upper management and get upper management to further buy-in. With that buy-in, there was more communication internally, getting more and more people aligned that data mesh was the way forward. But of course, data mesh is in some ways just a label and approach, it's not the point. Stefan talked about what can we take from Agile transformation and successful business transformation in general to use for data/data mesh transformation. A big focus in Agile is on communication and transparency. When people feel informed and heard, they are far more likely to buy-in. Humans struggle with uncertainty. When it comes to data mesh, there will be all kinds of new roles and responsibilities so you need strong communication to keep people informed and not feeling lost. Even if that is about trying something and seeing if it works. That honesty and transparency will have far more people leaning in than trying to issue proclamations. And of course, be ready for some politically driven issues because we are changing the way people operate. The ever present 'what is a data product?' conversation also happened at RBI according to Stefan. And it was important to define it in their own world - every organization has their own needs, understandings, and requirements/restrictions that will mean data products look slightly different. Specifically for RBI, a dashboard isn't a data product but is a business product, meaning there is still a clear ownership model. They also focused a lot on automation and risk assessment/mitigation. In a heavily regulated industry, risk is always a crucial factor. When it comes to working with the Data Protection Officers, it's important to make it safe and worthwhile for them to say yes when saying no is easy and prevents all risk. Stefan then went into a bit about where RBI is headed around self-service and why that's so crucial to the company's data mesh ambitions. For him, self-service is far more than just giving people access to data. Yes, you need a platform but you need upskilling and data literacy, an understanding of compliance, an understanding of your overall data ecosystem, and an understanding of the tooling and processes. You also need to build a platform that can be leveraged by non-experts. You need to make it easy for the general populace of your organization to actually consume, understand, and produce data. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of November 19, 2023 13:03

۲ years پیش13:03

13:03

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#269 Panel: Leading a Data Mesh Implementation (2nd Iteration) - Led by Vanessa Eriksson w/ Stefan Zima, Duncan Cooper, and Sid Shah 1:03:53

۲ years پیش1:03:53

1:03:53

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn . Transcript for this episode ( link ) provided by Starburst. You can download their Data Products for Dummies e-book (info-gated) here and their Data Mesh for Dummies e-book (info gated) here . Vanessa's LinkedIn: https://www.linkedin.com/in/vanessaeriksson/ Sid's LinkedIn: https://www.linkedin.com/in/siddharthin/ Stefan's LinkedIn: https://www.linkedin.com/in/stefan-zima-650229b7/ Duncan's LinkedIn: https://www.linkedin.com/in/duncan-cooper-1113722/ In this episode, guest host Vanessa Eriksson, the first CDO in Sweden and the head of data advisory company Vanessa Eriksson AB facilitated a discussion with Duncan Cooper, Chief Data Officer for Northern Trust Asset Servicing, Sid Shah, Head of Data Monetization and Platform at Airtel (guest of episode #258), and Stefan Zima, Data Transformation Lead at Raiffeisen Bank International AG (guest of episode #270). As per usual, all guests were only reflecting their own views. The topic for this panel was about the leader's role in a data mesh implementation and what these four panelists have learned in that role. This was the second iteration of a panel we will likely have about every six months or so - the first was episode #215 from April of 2023. Scott note: I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views individually. Scott's Top Takeaways: Regarding data mesh: get going but don't rush. Essentially, get started now but don't be in a hurry to try to get to some picture perfect end state. You need to take your time to make sure you are transforming instead of making changes that will unravel. Be brave and move forward into some uncertainty! Relatedly, you will absolutely get many things "wrong" but wrong in a data mesh world can simply mean not right yet. We have ways to adapt/adjust and evolve as we learn and grow. Data mesh provides you the ability to iterate towards better constantly. You _really_ should look to take inspiration and learnings from other data mesh implementations but also understand that your journey will look _considerably_ different, especially your starting point and initial focus areas. If you aren't prepared to adjust for your own situation, you aren't ready to do data mesh. Sid especially has focused on learning from other implementers and leveraging them to create credibility internally. Stop treating data problems as technical problems. At least start to treat them as business problems first but really, they are sociotechnical problems - social/business challenges married to technical challenges wrapped in trying to share understanding. It's people/process/technology and people most importantly. Relatedly, as Stefan said, data mesh is much more about mindset shifts than it is about technology shifts. You need to show people new ways of working and guide them to value. If you don't, you very likely won't change how people work with data and won't deliver the promised value of data mesh. And then you will lose your sponsorship. While data mesh can be a powerful paradigm, one of the best things about it is giving us common framing and ways of speaking about what you are trying to do and accomplish. Scott note: this is partly why I created the data mesh community, this podcast, and my company - to make it easier to share information with each other about how to do data mesh, both with other implementers and within organizations. There will be pushback from many parties to many aspects of data mesh. Be prepared for that and help people understand the reasoning behind the changes - what do these changes enable you to do? E.g. you aren't pushing data ownership onto the domains simply because as a central data team you're lazy - it's that they are the ones with the knowledge and skill to actually own their data well. There are many big challenges in data mesh but the biggest is probably getting and then _retaining_ buy-in across many different groups. If you don't have leadership buy-in, what happens when you hit challenges? If teams won't take on ownership, can you move forward? Other Important Takeaways (many touch on similar points from different aspects): Duncan mentioned that in Agile, there is a tendency to focus on a specific problem. But the biggest value of data comes from reuse, not just that focus on one specific problem/use case. So be prepared to adapt methodologies like Agile to data where circumstances are far different. If you aren't able to deliver incremental value - and prove/communicate that incremental value - you are highly likely to lose your credibility and/or sponsorship of your data mesh journey. As is often mentioned, your business colleagues probably - almost certainly? - don't care about data mesh. They care about getting good value from data. Data mesh can help us accomplish that but it's not the point. Talk to them about what you are trying to achieve and why, not about the inner workings of data mesh. What changes for them and why? Everyone already has ways of working with data. Instead of trying to change the way they work, look to intersect and gently nudge them in a better direction. The key to that is helping them understand what is changing and why. If you treat Zhamak's writings - or any other content around data mesh - as gospel, you are headed for failure. Zhamak has said this herself. We are figuring out how to do data mesh and you need to adjust for your own organization to do data mesh well. Be prepared for your organization to be even more messy/varied than you likely expect once you start to dig deeper into your ways of working in general and how those could be better improved by data. Your organization likely grew organically - different parts of a large forest behave quite differently even if it's one forest. ?Controversial Scott Note?: Someone made the point that to get buy-in, you can show people why it isn't sustainable for the organization to continue doing data the way we've been doing it. That generally isn't going to sway most people. They understand that but it's like saying everyone needs to do something like make their own clothes to fight climate change - you won't convince many… Your organization's data culture needs to work like a moderately well-functioning society: you have rules and regulations that people need to abide by or face the consequences. Think about the phrase from "The Good Place": 'What do we owe each other?' Democratization/federation doesn't mean anarchy. Your actual data mesh journey - maybe not the story you or others present at conferences - will be extremely messy. Organizations, people, life, etc. are all messy. Don't be worried about getting it perfect but worry about driving towards value. ?Controversial?: You need a certain level of maturity to be able to tackle data mesh but you also, as Stefan mentioned, need urgency. If there isn't a need to really change your approach to data, is data mesh worth the effort? Probably not… Much like with any transformation project, you need time to see results. That doesn't mean you can't deliver incremental value along the way but this is not a quick change at the overall organization level. That's why you transform pockets - domains/LOBs - and work on the greater organization over time. In Agile transformation we have Agile coaches. Do we need similar in data? Do we need evangelists? It seems you need someone connecting the dots for people. Just because people can self-serve doesn't mean they will really consider the 'art of the possible' with data. Have someone inspiring and coaching them. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#268 Adapting to and Adopting Product Thinking - Transforming Your Org for Sustainable Data Mesh - Interview w/ Iulia Varvara 56:02

۲ years پیش56:02

56:02

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Iulia's LinkedIn: https://www.linkedin.com/in/iuliavarvara/ In this episode, Scott interviewed Iulia Varvara, Advisory Consultant in Digital and Organizational Transformation at Thoughtworks. To be clear, she was only representing her own views on the episode. Some key takeaways/thoughts from Iulia's point of view: If you are greatly changing your general approach to something - which data mesh does in many ways - you need to focus some amount on actual transformation. These approaches are not a switch you flip, it takes time and concerted effort to make lasting changes that work well. If an organization hasn't really broadly embraced product thinking, starting with data as a product/product thinking in data can act as a catalyst for other aspects of the business to embrace product thinking. You don't change the organizational mindset through words - you start using new ways of working that change people's mindset as they see the benefit of those ways of working. At the end of the day, talk is cheap. To do data mesh well and have it work for an organization, it's best to tailor to their existing ways of working. Yes, change is necessary but a revolution is far less likely to work than an evolution. How are teams working and where can we make smaller tweaks? Because you need to tailor your implementation to your own organization, any data mesh blueprint that will supposedly work for all organizations is likely to be snake oil at best. ?Controversial?: The first two principles of data mesh - domain data ownership and data as a product - have the most impact on the organizational operating model. Scott note: federated computational governance might be up there for some organizations too. With data mesh - or really any change to your operating model - you need to make the goal(s) and measures of success visible to your people. That way, people can understand if progress is happening and what to prioritize. In product thinking, everything really centers around user value. It will inform your strategy, vision, and business goals. Create feedback loops to constantly test against serving user value. Create long-lived teams around your data products. If not, there is a significant risk of falling back to project thinking which will mean fewer innovations/experiments with the data and not enough subject matter expertise embedded into the data products to make them truly valuable. Similarly: "stop funding the work, start funding the team." Stop trying to focus on exact tasks to be done instead of outcomes to achieve. That's key to product thinking. Many people believe you have to completely change your company to start having domains own their data. Start with one or two domains, don't change the entire company upfront. When funding the teams, there is an assumption that the team will experiment and find more and more things that are valuable to users. But that ability to _find_ value is crucial - you won't always know what is valuable ahead of time, give the team space to find value. Also, by funding long-lived teams, they can learn more and more about what drives user value and keep an eye out for work that is no longer aligned to user value - you can shut down work that's no longer valuable. This is a rarer occurrence in data than people expect, shutting down work. Iulia started with a few basics about general transformation and digital transformation, whether that includes data or not. To really be able to embrace a digital and data-driven future, organizations need to embrace product thinking across the entire organization. They need to align their strategy and operating model to be adaptable and flexible. If the organization has already embraced product thinking, few have really pushed that to data in her experience. But if the organization is new to product thinking entirely, then starting from the data side could create a strong catalyst because data is probably one of the hardest concepts to apply product thinking to - after taking on data, product thinking is far easier to grasp in other areas of the business. In general, Iulia believes that mindset changes don't come from mandates. Instead, by implementing new ways of working, people's mindsets will start to shift once they see the impacts/benefits of those new ways of working. They see the benefit and change their mindsets. But that will of course take time and concerted effort - the mindset change by decree is faster but doesn't typically stick. Show don't tell. As other guests have noted, Iulia pointed out to maximize the chance of a data mesh implementation succeeding, you have to take into account the existing ways of working, the organizational and team operating models. Yes, certain aspects will need to change but trying to completely change an organization's operating model is going to be too disruptive. Instead, align a transformation paradigm to how the organization already works so people can evolve and adapt. Don't throw them in the change deep end and don't throw the baby out with the bathwater. Of course, this also means there isn't some blueprint for data mesh that will work for all organizations. In Iulia's book, the first two pillars of data mesh - domain-based data ownership and data as a product - are the two that have the biggest impact on the organizational operating model. She said, "When you start thinking about your data in terms of products, and put your user in the center of your attention, you try to organize all your efforts around the user needs. Right? You create this connection between the data team and the value." That is a big change to how most organizations work around data and it will take effort to make it happen. In general, Iulia recommends that for any large operating model change, you really need to clearly communicate multiple things. What are the changes, why are you making them, what is the actual target outcome/goal, what are the measures of success, etc. That way, people can measure how well things are moving forward and more easily prioritize. "Because there would be so many things to be done at the beginning, that team really needs to have a clear understanding what to start with." Transformation will mean tens of changes, understanding where to start and why are crucial. Specifically to product thinking, user value is your Northstar for Iulia. It will inform your strategy, vision, and business goals. Those business goals will be split into hypotheses of value for how you can reach the goals. This is where you start to allocate teams, to the actionable items from the hypotheses of value. But it all comes back to focusing on user value. Steer your work through feedback loops to focus on that user value and you have a great shot at implementing product thinking/focus well. Iulia pointed to something many miss when it comes to treating your data as a product. If you don't have a long-lived data product team, it can cause many issues that significantly undercut the value of building data products. One is that you often lack the subject matter expertise in the data product team, so the information encapsulated is not nearly as deep or as relevant to the topic area for the data product. Another is that if the team isn't long-lived, will they really have the time and psychological safety to run experiments and innovate? Similarly to data as a product, Iulia recommends going small, then sustaining, then scaling when it comes to domain ownership. Basically, start from one to two domains and go broader over time. Trying to reorganize your organization on day one so one or two domains can own their data in that time-frame, that's a TON of effort. Don't try to revolutionize your company to do data mesh, evolve and build the understanding as you go broader. You need to prove out value first too before you go broad. In wrapping up, Iulia returned to the concept of funding the teams, not the work, and especially long-lived teams. When you fund the teams, they are able to focus on finding value. There isn't an expectation of the teams to be prescient, always knowing what will be valuable. And there isn't a need to simply react to tickets instead of finding what will be of value. The other aspect is that you can understand what should be decommissioned. Far too often, data work continues well past when it is valuable. But with a product mindset, teams can constantly be focused on user value and shut down things that no longer drive value. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of November 12, 2023 23:30

۲ years پیش23:30

23:30

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#267 The Developer Experience - How Do We Delight Data Developers? - Zhamak's Corner 30 19:49

۲ years پیش19:49

19:49

Key Points: The current data product developer experience sucks. In software, we have one simple interface that manages all the pieces needed to get the job done, packages those components. In data, we have to jump through hoops and interfaces across many tools. Right now, the developer has to manage everything themselves which is a ton of work. But it's also a big risk because of lifecycle management - if everything isn't packaged and deployed as one, you have dependencies drifting from each other. There are many places from software we can learn from as to how to do this containerization. Ruby-on-Rails and CloudFoundry are good places to look. Sponsored by NextData , Zhamak's company that is helping ease data product creation. For more great content from Zhamak, check out her book on data mesh , a book she collaborated on , her LinkedIn , and her Twitter . Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Data Mesh Radio episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#266 Leveraging Decades of Information Architecture Learnings to Do Data Well - Interview w/ Akins Lawal 1:03:39

۲ years پیش1:03:39

1:03:39

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Akins' LinkedIn: https://www.linkedin.com/in/akinslawal/ Schema for Success: https://www.schemaforsuccess.com/about In this episode, Scott interviewed Akins Lawal, a Data Strategist. To be clear, he was only representing his own views on the episode. Some key takeaways/thoughts from Akins' point of view: Far too often in data, people move to build _something_ instead of focusing on building good information architecture specific to the task at hand and the organizational goals and capabilities. Good information architecture isn't about tech, it's about the principles and practices of how you're going to structure your data/information. Keep going back to good product principles in information architecture: your focus should be on what are you trying to accomplish over what are you trying to build. ?Controversial?: Organizations need to focus far more on hiring for learning capacity instead of only for current skills. The world is changing too quickly to try to focus on specific skills for many data-intensive jobs. Leadership buy-in ends up being the number one determining factor of success for projects and transformation according to many studies. Trying to proceed - even with the greatest plan ever - without that buy-in greatly reduces the chances of success. Maturity models can be extremely helpful but they sometimes don't tell the full story. Look for pockets of maturity in your organization and see what can be copied/replicated and what can't when improving the maturity of other areas of the organization. !Controversial!: There is something different about information exchange in person rather than entirely virtual. It is far more likely to create a deeper understanding if you can collaborate and whiteboard. Scott note: I'm 50/50 on this. A big question is cost benefit of mental energy, travel, etc. - better return doesn't always mean better return on investment. Let people know "that data is more of a tool to empower … that also needs to be woven into the organizational fabric." Reward people for (appropriately) sharing their data. At the start, accolades can be enough but if you really understand incentives, it should become something more deeply rewarded. People, process, tools but it always comes back to people being the most important. You can focus on and improve your data culture no matter the size of an organization from two to 200K. In Akins' view, many more people need to slow down a bit to really consider what they are building, why, and for whom. Far too often, data people - and tech people in general - want to build something but we need to focus on good information architecture. We need to consider a lot about the specifics of why we are building something and also who will use it and how in order to maximize positive outcomes. When starting with good information architecture, Akins recommends asking a lot about what the system will do and why. Map out the user flows even, how will people access information. That way, you can start to back into what kind of systems and approaches will support what you are trying to accomplish - it's not about what you are trying to build, it's about what are you trying to accomplish and building something that can help. As you start to plan out your information architecture, Akins recommends you start finding your champions - you need people to rally around to get things moving forward. Then you plan out your process - there are many different tried and true processes for sharing context/information but it's important to find one that works well with the use case and your organization. You should be looking for happy mediums between all involved because no one will get everything they want if you're doing it well. In Akins' view, if you want to become data driven as an organization, you need to focus on hiring learners, not just for current skill sets. Data - how to analyze it, leverage it, share it, etc. - is a lifelong problem/challenge. New tech and approaches always emerge. You want someone focused on staying up-to-date on best practices, not specifically proficient in one tool that could be close to obsolete in a few years. Leadership buy-in is far and away the most important factor to new initiatives succeeding, according to multiple studies from groups like HBR and McKinsey. So Akins recommends to make sure you are aligning with those leaders and showing them the benefit of improving your data initiatives. A big reason so many data initiatives fail is that lack of buy-in and support. The best laid plans are still far less likely to succeed without support from above. Akins talked about how maturity models can be a very helpful tool for finding what's already working in your organization relative to data work. You can find the patterns and then assess if you can make those into repeatable patterns for the rest of the organization or not. It's all about creating a situation where things can mature. In person collaboration for Akins just kind of 'hits different' - you are better able to exchange context if you're in the same room and able to whiteboard. Potentially that's around driving meaning and trust? Virtual tools still have not caught up to the in-person collaboration capabilities. So if you aren't in person, he believes you will likely need to meet more often just to ensure you really understand each other. Akins then talked about the importance of leveraging data to empower people and weaving that data understanding and empowerment into the fabric of the organization. How do you empower people to make more and better decisions with data? That's how you move towards being data driven. Something Akins likes for driving data sharing and usage is incentivization: how are you rewarding people for sharing and leveraging data? Leaders should be giving teams and people that are sharing and using data well lots of accolades and potentially other rewards so others want in on the action. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of November 5, 2023 11:45

۲ years پیش11:45

11:45

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#265 Are You (Even) Doing Data Mesh™? - Mesh Musings 55 17:03

۲ years پیش17:03

17:03

Key takeaways: If you want to say you are doing data mesh, go ahead. If you really want to know if you are, look to Zhamak's book and her talks - how much are you attempting and are you going with the thin slice model - not ignoring any of the pillars. Have some empathy for yourself. If others want to say they are doing data mesh, it's not really a big deal as long as we aren't trying to emulate the ones fooling themselves or over-marketing. I'd rather welcome more folks into the tent than gatekeep. Maybe that is just my own prerogative but I have empathy for people who see what they think the end journey should look like. But it's like that transformation picture of build a skateboard, then bike, then moped, then motorcycle, then little 2 door car, etc. with the end idea of building a sedan. Please Rate and Review us on your podcast app of choice! Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#264 Will GenAI and Data Mesh Really Mix? - Interview w/ Madhav Srinath 51:09

۲ years پیش51:09

51:09

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Madhav's LinkedIn: https://www.linkedin.com/in/madhavsrinath/ In this episode, Scott interviewed Madhav Srinath, CEO at Nexusleap. Overall, we are super early in the Generative AI cycle and hype is huge. This discussion is one of early impressions, not fully formed answers. It's far too early for that. Also, FYI, there were some technical difficulties in this episode where the recording kept shutting down and had to be restarted. So thanks to Madhav for sticking through and hopefully it isn't too noticeable. Generative AI will mostly be shortened to GenAI throughout these notes. LLM stands for large language models which power GenAI. Some key takeaways/thoughts from Madhav's point of view: ?Controversial?: An emerging best practice seems to be having layers of LLMs - one model where you might ask it complicated questions and the second model is trained specifically to vet the answers for correctness and governance concerns. The cost of running many models in production is typically actually quite low, at least infrastructure wise. Instead of an always-on architecture, most organizations are leveraging a serverless architecture - or leverage APIs from others providing the models - so they essentially only pay a few cents per query. ?Controversial?: Use GenAI as a "scalpel, not a broadsword". Many are trying to use them in overly broad ways and getting not great results. The ability to take a mountain of data and get something out of it in a structured way isn't a new concept. We've been trying to do that with data mining for years. It's just that it is finally maturing into something more widely useful/usable with GenAI. People are generally still only trying to solve pretty shallow problems with GenAI, e.g. writing an article. Scott note: That's probably good because most aren't ready to do the work necessary to have GenAI be usable for much deeper use cases ?Controversial?: We may need human handlers for LLMs to do GenAI well. If we aren't sure of the quality of the answers and we need high quality answers, there needs to be guardrails and probably a human in the loop. Scott note: this might prove to be better than having the human just do the analysis or not, remains to be seen. If you have the right guardrails in place, there isn't really any harm in starting to work with GenAI. But, you have to understand it's early days and there are definite sharp corners for you to cut yourself on - that human in the loop is important for a myriad of reasons and you have to be careful around things like privacy. ?Controversial?: It's better to start at domain level questions and focus on domain-specific problems right now with GenAI. That way, you can more easily control the inputs you feed it and it can help with more specific, targeted questions. Look at machine learning use cases. Creating narrow focuses for each model has been proven to be a far better strategy instead of creating one overarching model to try to solve many problems. Why not try the same with LLMs, creating models specific to topics? Relatedly, you can add more focus areas to an LLM as you train it. Trying to get it to understand everything at the start will likely overwhelm the LLM to a point where the quality of your answers will fall. LLMs can be used to infer relationships between domains or data products. You still have to point them at high quality data and you need someone to check their work but they could be used to more easily find out where data products already are or should be interoperable. A potential good use case is to have GenAI models focused on finding those potential relationships and then use a second GenAI model that's more targeted at finding information based on those relationships. ?Controversial?: You shouldn't be training GenAI models from scratch. Start with one of the many open source models available and train it on your specifics. Leverage work that others have already done for you. You can train them by having your business specialists share information with the LLMs. Since everyone essentially has access to the same models, companies will differentiate on the information - and especially the quality of the information - they feed their LLMs. ?Controversial?: GenAI may be more useful for data producers than data consumers. They still need to focus on the fundamentals but GenAI can really make them more productive. Scott note: imagine being able to get 5 sample data models or be able to ask an LLM to figure out the best way to make your data fit with other data products to be interoperable. For Madhav, a lot of what Generative AI has become is the concept of data mining with a personable interface. We've been trying to create a way to dig into data that is unstructured and get some insights or information - something that is structured - for a while. The concept isn't new but we've finally found something that might actually be able to do it well and make the outputs easy to consume. Right now, in Madhav's view most of the emerging GenAI use cases have been pretty shallow, such as to help write an article. It probably can get far deeper but it's quite early days. However, we probably need to put a human in the loop to actually make sure the answers LLMs are giving are correct. That might be the best option in his view, something like a guide plus guardrails driven by a human to make sure these LLMs don't hallucinate. In a way, this isn't all that different from other machine learning work - black boxes tend to have unexpected consequences. Madhav's view is that it's totally okay to start working with GenAI at your organization as long as you understand GenAI/LLMs have quality issues right now and are really only at the MVP stage in many senses - especially if you are using them internally on your own data. There will probably be good ways to put a wrapper around them to prevent improper data usage/leakage and prevent hallucinations as well. Starting with domain specific questions/problems is where Madhav thinks people should focus their GenAI work. If you try to feed an LLM a ton of information from many sources, you can't really be sure of the logic it uses to generate answers versus keeping the inputs tighter and asking about more specific business areas. Keeping that tighter focus and having many LLMs across the organization gives you an ability to more tightly focus your models on specific topics. You can then attempt to add additional focus areas to those LLMs once you have the model performing well on one topic. While LLMs aren't magic, Madhav is seeing an emerging use case where people point the LLMs at maybe two data products and ask it to infer relationships between them. It might discover something people haven't thought of before. You still need a human in the loop or you end up with something like the Pastafarian 'belief' that global temperature rise since the 1700s is caused by the decreasing number of pirates globally - correlation doesn't equal causation. It's not a magic wand but it could help people find more places where data is already interoperable or should be. Then, he's seeing once those relationships are discovered, a differently tuned GenAI model is used to actually infer some information based on those newly discovered relationships. Again, specialized models. Madhav doesn't believe most organizations should be training their GenAI models from scratch. Instead, go and find the open source models and add your necessary information to the training - basically, why start at zero to train it to one when you can start at 0.7? Leverage the work others are doing so you don't need super expensive LLM training focused engineers. By starting with an existing base model, you can tune it based on your own answers instead of trying to feed it very specific data and supervise the base-level initial training. Leverage your business subject matter experts and get them to share their tribal knowledge with the LLMs as well. Circling back to the idea of layered LLMs, Madhav talked about how some organizations are having models specifically tuned to answer questions about data but then there is a secondary LLM that is focused on checking the answers the first one gives for sanity/correctness as well as around governance, e.g. security and privacy concerns. Again, that separation of the work. And it's not nearly that expensive when it's all done in a serverless way - if your LLMs are becoming cost prohibitive, you are probably not running them in a cost effective way. Madhav has a few strong feelings around what organizations should be doing with LLMs. The first is that most should not be trying to train their own LLMs - the time and cost just don't make that much sense when open source models are advancing at frankly remarkable speeds. The second is that GenAI is probably more helpful for data producers than data consumers. It really can make producers far more productive, e.g. letting them generate insights on their own data or helping them to find good interoperability points with their data and other data products. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of October 29, 2023 14:53

۲ years پیش14:53

14:53

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#263 Panel: Applying Site Reliability Engineering Practices to Data - Led by Emily Gorcenski w/ Amy Tobey and Alex Hidalgo 59:22

۲ years پیش59:22

59:22

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Emily's LinkedIn: https://www.linkedin.com/in/emily-gorcenski-0a3830200/ Amy's LinkedIn: https://www.linkedin.com/in/amytobey/ Alex's LinkedIn: https://www.linkedin.com/in/alex-hidalgo-6823971b7/ Alex's Book Implementing Service Level Objectives : https://www.alex-hidalgo.com/the-slo-book In this episode, guest host Emily Gorcenski, Head of Data and AI for Thoughtworks Europe (guest of episode #72) facilitated a discussion with Amy Tobey, Senior Principal Engineer at Equinix and Alex Hidalgo, Principal Reliability Advocate at Nobl9. As per usual, all guests were only reflecting their own views. The topic for this panel was applying reliability engineering practices to data. This is different than engineering for data reliability which is focused on data quality specifically. The overall concept is taking what we've learned from reliability engineering across disciplines but mostly in software, especially SRE/site reliability engineering, and bringing those learnings to data to make data - especially data production and serving - more reliable and scalable. Scott note: this is probably one of the most frustrating topics in data for me because it feels like it's basic foundational work yet most organizations aren't tackling this well yet if at all really. The best starting point for an organization is simple awareness and starting to have reliability engineering conversations around data. And you will probably feel like you're behind after listening to this. Everyone is behind on this 😅even most orgs aren't doing SRE well so applying it to data, that's no surprise. Scott note: I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views individually. Scott's Top Takeaways: Reliability engineering is one of the majorly overlooked aspects of what we need to bring to data - whether you are talking data mesh or not. People are starting to really move on product thinking and to a lesser degree bringing microservices and CI/CD approaches to data. But reliability engineering is still a somewhat distant afterthought if at all. At the end of the day, reliability engineering comes down to observability first. If you can't observe what's happening with your systems, you can't really start to identify issues and work on fixes. Observability in data CANNOT only be about the data quality. That is observing what is happening with the data versus the data applications/systems/pipelines/etc. The data itself may seem fine - no quality issues - but if your systems storing, moving, transforming, etc. your data are degrading, you are missing the forest for the trees. A broken pipe that delivers 5% of the water but it's clean is still broken even if the water passes quality checks. Relatedly and somewhat in contrast - the measures used in operational systems reliability engineering like availability and latency are potentially the wrong areas to focus on for reliability engineering in data. Don't copy-paste those key metrics, think about what matters when it comes to the data applications/systems as well as the actual information you are sharing. As Amy mentioned, there is too much binary thinking about data - whether it is quality, reliability, availability, etc. The idea of criticality is often missing from data discussions - and trying to measure by who screams the loudest is a really poor approach 😅 As an overall organization, we need to map what data is critical and why. The best way to find out your SLOs for data related work is to have conversations. Just because you can, doesn't mean you should. Not all needs are created equal. Don't take consumers at their first word, dig in and find out what will serve their needs but limit the work to something reasonable. Remember that return on investment is what matters, not simply return. Most data work has value but does the value justify the costs? Keep things simple. As Alex said, focus on "is this doing what my users need it to do?" Look to measure around what your systems need to do to serve customer needs. It's very easy to get caught up measuring the wrong things. Currently, in many organizations, reliability engineering work is much more like archaeology than engineering as Emily noted. We need to focus on providing the context to those trying to ensure reliability and that takes forethought on the systems/architecture side as well as the organizational side. Observability - data or otherwise - isn't a switch you can easily flip, it's a practice you build and get better at over time. Other Important Takeaways (many touch on similar points from different aspects): You can make progress on reliability engineering in data just by making sure you have an incident management process. If you do, then look to improve it. You can iterate towards good, everyone starts somewhere :) As Emily pointed to, can all your SLOs really be covered by SLAs in data? How do you have an SLA around semantic meaning for instance? Do we need to extend the SLO concept to really make it applicable to data? Just starting the reliability concept discussions with people in data is often difficult if people don't have operational systems/plane experience. In data, far too often there is a either 'it's right or it's wrong' approach instead of how acceptable something is. Sometimes imperfect data is acceptable and the same goes for the reliability of your systems. You can iteratively improve your observability and reliability practices. You can start to show value as you build more capability. Much like with anything, unless you have budget and full high-level support, start small. Take Alex's simple definition of reliability: "Is this doing what it's supposed to be doing?" Then work to find ways to observe if your systems are actually doing what they are supposed to be doing. And only focusing on where you think things could go wrong is monitoring, focus on observing - the difference between those two is a rabbit hole you should go down yourselves. Relatedly, you need to really consider expected behavior when building systems. This is of course around data quality but also setting your SLOs - what are you trying to achieve by building the data application/product? As Alex said, "SLAs really exist for lawyers and financial departments, and SLOs exist for engineers actually trying to keep things reliable." Data contracts are often focusing on the SLAs instead of the SLOs but both are important when thinking about data. SLAs engender trust as a guarantee, SLOs are about making sure you're actually thinking about consumer needs. ?Controversial?: Don't start with formal SLAs - and maybe SLOs - unless your organization is mature enough to handle them. People under 18 - at least in many countries - aren't able to enter many legal agreements for a reason. Your maturity level matters. Potentially look to informal agreements or 'best faith effort' as you are starting to think about SLAs and SLOs. People often think about data freshness and quality metrics but there are other important metrics like durability and availability. If you have perfect data but no one can access it or it gets lost, what value is it? Does managing cost fall under reliability engineering? Efficient usage of resources often falls on the SRE team to some degree - how do we think about wrapping in cost management. As Emily noted, many companies have their data management on a different cloud from their applications and that creates a LOT of cost inefficiencies. Cloud economics actually prevent good data usage and practices. Because cloud providers make so much of their money on data movement - especially egress - the economics of using best-in-class tools becomes untenable, especially if those are on a separate cloud. Relatedly, many - most? - cloud data tooling seems to be focusing on driving usage and revenue instead of driving value taken in the greater picture. Instead of focusing on how to play together to create value, each is creating lock-in. That creates additional reliability and observability challenges. In data, many organizations are still at the pre-crawl phase relative to observability. Something as basic as 'if you create an alert, have a runbook for when the alert goes off' is missing. We need to bring maturity in - but at a reasonable pace, not try to fix overnight - so organizations can get to actually reliable systems and data. For a lot of organizations, they still treat data work as a cost center. Even in data mesh to some degree. If we can't prove out the value of the data work in general, it will be MUCH harder to get budget for something like reliability engineering work in data. But reliability is really crucial if you want to become data driven rather than simply say you're data driven. At the end of the day, lack of reliability engineering practices in data are twofold: first, lack of awareness. Second, organizational. There simply needs to be a focus and investment if we believe reliability engineering in data will pay off. Or maybe it's not worth it for most organizations. Time will tell but you can bet that the most advanced organizations that really are 'data-driven' aren't skipping it. Once we tackle the above, hopefully tooling will follow but data tooling in general just isn't in a good spot relative to reliability engineering - it's too hard, too manual, and too expensive in general to do reliability and observability well around data systems. Scott note: this is why I'm an advisor to Masthead Data because they are doing some really cool reliability engineering stuff for data. One positive note is that industry challenges such as GDPR are driving organizations to realize they need to change. It may not be happening that quickly but the cost of keeping up with regulation in legacy systems is becoming greater by the day. We might see a bigger tipping point soon. A key point of SRE is to understand the inherent complexity and interconnectedness of our systems. There's some understanding of that which exists in data but we generally don't look to solve for it by learning about how things are interconnected beyond lineage. Good observability and reliability engineering practices have change as an inbuilt assumption. This is where monitoring often falls short - you have to constantly update your very explicit assumptions versus your understanding. Focus on understanding the needs of your users/customers and observe if the system is actually serving those needs appropriately. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#262 Setting the Groundwork to Become Data Driven - Interview w/ Corrin Shlomo Goldenberg 1:06:52

۲ years پیش1:06:52

1:06:52

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Corrin's LinkedIn: https://www.linkedin.com/in/corrin/ In this episode, Scott interviewed Corrin Shlomo Goldenberg, Senior Product Manager of the Data Platform at BigPanda. It's important to note that BigPanda is not at the stage yet where data mesh makes sense but this is a story of getting production of data into the heads and hearts of the application development team, which is a crucial aspect to doing data mesh well, whether it's done pre data mesh or as part of the journey. Some key takeaways/thoughts from Corrin's point of view: When doing data work, it's easy to fall into the trap of trying to do everything. Go back to product basics - start from the why, why are you doing this? If not, we just are creating new forms data swamps. It's not uncommon for developers to think of data simply as what's in the database, especially in B2B startups. Sometimes you have to work with them to get them to really understand they need to be creating and storing data to be leveraged for analytics. It's not even data exhaust, sometimes the data doesn't even exist! Related, many B2B companies feel they aren't data oriented enough. You can work to change that of course but know that almost everyone else feels the same; we all start our data journey somewhere, get inspired to go forward. It's hard to pinpoint the time for a growing B2B company when it's actually time to start collecting and analyzing a lot of their data versus when it would be overkill/too early. Scott note: for larger organizations, look to have the conversation early in the lifecycle of any product - build a data sourcing strategy even if it's not implemented from day 1. Obviously doing data work isn't free - make sure you have the conversation about when to flip the switch. It's often driven by someone wanting a report or information, prepare ahead of that. An indicator you need to be preparing more data is when the product managers are struggling to answer basic questions. Often it's the 'how many' type questions that shouldn't be hard to know. ?Controversial?: When getting started with development teams even understanding data work, it's far easier to have that data work centralized in a data team. You can decentralize over time but introducing them to the idea of data work and a data platform in general while trying to hand over ownership might be too much. Scott note: this probably isn't really controversial as much as an inconvenient truth. Prioritization is key - not just what you work on but what is the incremental value of different aspects of work. Look to make sure you can justify what work you are doing - circle back to 'the why'. Ownership isn't just about who owns the work but who owns the outcomes. Focusing on the work over the target outcome is not likely to end well. Similarly, ownership isn't always black and white. While a team owns their domain, a central data team will often own the data related to the domain. Partnership is crucial, teamwork makes the dream work. Use good product management practices - just building something won't automatically create usage. Talk to your constituents and help them understand what you've built and why to drive more usage. Corrin started with the tale of BigPanda and how she started building out their data, ML, and analytics capabilities. When she came in, they didn't have the infrastructure or really the focus on a scalable platform for storing and analyzing their internal data. They were doing a lot of this for external clients but hadn't moved to doing it internally, which is pretty common in B2B startups. But BigPanda wanted to do a data driven transformation of their business model so they had to change the situation around their internal data. There is always a balance for when you start collecting data at scale in Corrin's mind. At a B2B startup, you need to ask how early should it be for the company but the same is applicable for an early-stage offering at a larger organization. Most development teams aren't tasked with dealing with creating the necessary data until far later in an offering's lifecycle but it would be nice if you could include it at the start. But it definitely isn't free so there is always a balance and the conversations need happen, hopefully earlier than later. Corrin's tipping point for when you should really start to press development teams on creating necessary data is when it becomes hard to answer simple 'how many' type questions. It is also an easier conversation than a hypothetical one. If it takes more than a day to get basic information on how your customers are using your product, that's obviously an issue that's only going to grow. It's also a pretty tangible place to start. When they started to build out the data platform, Corrin said it just made sense to start centralized. If the R&D team wasn't really thinking about data, trying to upskill them enough to take over the work entirely was probably a bridge too far. Plus, if your data requirements aren't complex enough to require decentralization, decentralization is often just an extra layer of complexity. So they moved to a high communication model where people can see what data work is happening even if it's controlled by the central team. They can slowly upskill the development teams to understand data instead of trying to hand over ownership prematurely. Corrin talked about working with the team to understand the product mindset to data. Start from the why - it's easy to fall into the trap of trying to do everything because it might have value. That's what happened with data lakes that became data swamps. Focus people on the why and you can bring them more and more into working with data. Similarly, while Corrin and team didn't have a lot of pushback on getting things done, she was very cognizant of prioritization and cost/benefit. Again, focusing on 'the why': what is most important and when? Why are the requirements like this? Can we cut the cost down by storing for less time and/or refreshing less often? When you say 'real time', what do you actually mean? Etc. Corrin has been seeing good results from having strong ownership conversations. While the central team still owns the data, they are partnering with the domains as the domains still need to own the concepts and the understanding of the information. While this might not work at a large scale, it's perfectly normal and functional at a 300 person company. Scott note: centralization isn't the enemy until it becomes a bottleneck 😎 As with all global companies, BigPanda has some challenges around communication, per Corrin. Time zone differences and of course differences in focus are just two of them. So she recommends spending a lot of time to communicate to stakeholders about what you are building and why. It's easy to assume that because you build out a data product, people will use it but you have to work with people to ensure they actually use what you built. Corrin pointed to the fact that many companies in the B2B space feel they aren't "data oriented" enough. She gave a few tips for how to become more data oriented but also has empathy for people feeling that - it's pretty common, most B2B companies feels they aren't as data oriented as everyone else. Similar to data mesh, where everyone believes all the other companies are far down their path. It's simply optics - companies project a better image than the reality of their situation with data. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of October 22, 2023 27:17

۲ years پیش27:17

27:17

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#261 Just What the Heck is a Data Product Container? - Zhamak's Corner 29 20:37

۲ years پیش20:37

20:37

Key Points: Containers in software abstracted away a number of very cumbersome tasks and encapsulated a lot of the dependencies software had to its environment. Combined, that meant developers could focus on delivering value instead of focusing on the infra. We need to do the same in data. It's all about sharing data in a responsible and easy way. That means putting all the components together so you don't have to manage many versions. Just like microservices. How do you manage to make this easy for the data product developer - bundle everything together. But centralized data products are creating a lot of potential issues/risk to scalability and flexibility. We just keep trying to centralize in data. Sponsored by NextData , Zhamak's company that is helping ease data product creation. For more great content from Zhamak, check out her book on data mesh , a book she collaborated on , her LinkedIn , and her Twitter . Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Data Mesh Radio episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Rerelease of #65 What's a Data Contract Between Friends - Setting Expectations with Data Contracts - Interview w/ Abe Gong 1:01:37

۲ years پیش1:01:37

1:01:37

Due to health-related issues, we are on a temporary hiatus for new episodes. Please enjoy this rerelease of episode #65 with Abe Gong all about how people are implementing data contracts in the wild. There are so many ways people can just do only defensive data contracts and I think that is such a missed opportunity. Maybe it's where you will have to start but there's a much better way and we talk a bit about why I think that is so distressing that people aren't talking to each other. Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here Abe's Twitter: @AbeGong / https://twitter.com/AbeGong Abe's LinkedIn: https://www.linkedin.com/in/abe-gong-8a77034/ Great Expectations Community Page: https://greatexpectations.io/community In this episode, Scott interviewed Abe Gong, the co-creator Great Expectations (an open source data quality / monitoring / observability tool) and co-founder/CEO of Superconductive. One caveat before jumping in is that Abe is passionate about the topic and has created tooling to help address it. So try to view Abe's discussion of Great Expectations as an approach rather than a commercial for the project/product. To start the conversation, Abe shared some of his background experience living the pain of unexpected upstream data changes causing data chaos / lots of work to recover from and adapt. Part of where we need to get to using something like data contracts is to remove the need to recover in addition to adapting and move towards controlled/expected adaptation. Abe believes that the best framing for data contracts is to think about them as a set of expectations. To define expectations here, this would include not just schema but also the content of data, such as value ranges/types/distributions/relationships across tables/etc. So for instance, a column may be a one to five for rankings and then the application team changes it one to 10. The schema may not be broken - it is still passing whole numbers - but the new range is not within expectations so the contract is broken. At current, Abe sees the best way to not break social expectations is via getting consumers and producers in a meeting to talk about the upcoming changes and prepare, such as with versioning. But, as tooling improves, Abe sees a world where we won't even need a lot of those meetings going forward - either because data pipelines can be "self-healing" and automatically adapt to changes upstream or because metadata and tools for context-sharing will reduce the need for meetings. Abe sees two distinct use cases in general for data contracts or more specifically how people are using Great Expectations to implement data contracts. The first is purely defensively - put some validation on the data you are ingesting to prevent data that doesn't match from blowing up your own work; the second type is when the consuming team shares their expectations with the producers and there is a more formal agreement - or contract - with a shared set of expectations. The first often leads to the second, via an agreement conversation that happens after there was an upstream breaking change. Abe also mentioned there is a third constituent on data contracts in the room: the data. Sometimes the consumers and producers may agree on what they expect, but if that’s different than what’s in the actual data, then it's hard or dangerous to move forward. The data has a veto. There was an interesting discussion on the push versus pull of data contracts - should the producer team create an all-encompassing contract or should we have consumer-driven contracts? Would producer-driven contracts be too restrictive, preventing the serendipity insights data mesh aims to produce? Would consumer-driven contracts mean multiple contracts for each data product that the producer agrees to? Is that sustainable? So, to sum it up, the idea of a set of explicit expectations around a data product that are the result of collaboration between producers and consumers sounds like where we should all head if possible. If the expectation set is only coming from the producer side, it might be overly restrictive and miss a lot of the nuance necessary to actually create consumer trust. And exclusively consumer-driven contracts don't sound sustainable or scalable. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Rerelease of #48 Overcoming Obstinate Organizational Obstacles in Data Mesh - Interview w/ Scott Hawkins 57:27

۲ years پیش57:27

57:27

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here Scott Hawkins' LinkedIn: https://www.linkedin.com/in/scott-hawkins-8934393/ In this episode, Scott interviewed Scott Hawkins, Principal Data Architect at ITV. Scott views data mesh as a mechanism for change. Your company culture and your understanding of it are crucial to establishing data mesh well, driving that buy-in. Ask yourself: what challenges does data mesh actually address and hopefully solve, how will it impact the business not the tech, and what does it change. Your organization might not be ready for data mesh. Or a specific domain might not be ready. And that's okay! As ITV moved forward, they found a "good enough" solution via a global ID. It's not perfect, there might be some overlap - such as one person might have a different global ID for their online subscription versus their broadcast subscription - but it is far better than what they were doing. And it allows for interoperability/joins across the data. This is a big improvement - don't let perfect be the enemy of good or done. One thing working for ITV is deploying a "team-in-a-box" to help domains move forward - similar to an internal consulting team. Each situation is different so each box they are given is different. The team-in-a-box concept also means it is somewhat easier to build common best practices internally. Coming to the table with defaults has really helped ITV. Per Scott, there are 3 good ways to drive buy-in for the domain teams: At the senior level - so it trickles down as the management for the team is bought in. Via a strong carrot - solve a problem for them as a kind of quid quo pro / mutually beneficial solution. Trying to solve an unrelated problem will drive lower buy-in. Work on realigning the team KPIs/OKRs with the senior leaders to actually realign incentives. Continuing on the driving buy-in, Scott recommends working with the domain managers to generate a viable/valuable carrot for the entire team. Explain to those leaders why it matters, work with the leaders to revamp the KPIs if the KPIs are getting in the way of delivering a good data product. This is why exec-level buy-in is so crucial - it is pretty hard to start modifying team KPIs/OKRs without it! Talking to teams to understand who they (the individual and the group) operate is crucial to developing the right path for them. Scott also talked about making failure an option. You can try to work with a domain and if it isn't working, it's okay to move on. You don't need to get everyone onboard on day one or sharing their data on day one. If you design incentives well, people will want to participate eventually. Until then, it's okay to walk away from that team. Due to health-related issues, we are on a temporary hiatus for new episodes. Please enjoy this rerelease of episode 48 with Scott Hawkins. I thought this episode was crucial because it is foundational to understanding a lot of good techniques and practices for getting lots of the organizational pieces of data mesh right. You will learn a ton on this one! Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of September 10, 2023 31:50

۲ years پیش31:50

31:50

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Rerelease of #150 3 Years in, Data Mesh at eDreams: Small Data Products, Consumer Burden, and Iterating to Success, Oh My! - Interview w/ Carlos Saona 1:24:33

۲ years پیش1:24:33

1:24:33

Due to health-related issues, we are on a temporary hiatus for new episodes. Please enjoy this rerelease of episode 150 with Carlos Saona. eDreams' approach is very unique and interesting because it was essentially all on its own so there are a ton of useful learnings to consider if they are the right fit for your own organizations. Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Carlos' LinkedIn: https://www.linkedin.com/in/carlos-saona-vazquez/ In this episode, Scott interviewed Carlos Saona, Chief Architect at eDreams ODIGEO. As a caveat before jumping in, Carlos believes it's too hard to say their experience or learnings will apply to everyone or that he necessarily recommends anything they have done specifically but he has learned a lot of very interesting things to date. Keep that perspective in mind when reading this summary. Some key takeaways/thoughts from Carlos' point of view: eDreams' implementation is quite unique in that they were working on it without being in contact with other data mesh implementers for most of the last 3 years - until just recently. So they have learnings from non-typical approaches that are working for them. You should not look to create a single data model upfront. That's part of what has caused such an issue for the data warehouse - it's inflexible and doesn't really end up fitting needs. But you should look to iterate towards that standard model as you learn more and more about your use cases. ?Controversial?: Look to push as much of the burden as is reasonable onto the data consumers. That means the stitching between data products, the compute costs of consuming, etc. They get the benefit so they should be taking on the burden. Things like data quality are still on the shoulders of producers. You should provide default values for your data product SLAs. It makes the discussion between consumers and producers far easier - is the default good enough or not? ?Extremely Controversial?: At eDreams, you cannot publish data in your data product that you are not generating. In derived domains (e.g., customer history), “generate” includes the derived stitching. NOTE: Go about an hour into the interview - not episode - for more specifics. When starting with data mesh, there must be a settling period - consumers must understand that things are subject to change while a new producer really figures things out for the first few weeks to months. You want to avoid duplicating data. But you REALLY want to avoid duplicating business logic. Be careful when selecting your initial data mesh use cases. If the use case requires a very fast time to market, while it has value, you likely won't have the time and space necessary to experiment and learn. You need to find repeatable patterns to scale in data mesh. Hurrying is a way to miss the necessary learning. Look ahead and build ahead for obvious interoperability. E.g. create foreign keys for data products that don't exist yet but will. Be clear about what early data mesh participation means - what will it net domains that are part of the early implementation? And be specific too about what your early implementation won't include or achieve. Don't over promise and under deliver. Similarly, strongly emphasize that learning is a priority in your early implementation and that you are factoring in learning into promises and estimations. You can't promise you'll find the right solution to a challenge on day one, things need space to evolve as you learn more. It's okay to not have everyone as part of your initial implementation - engagement or even buy-in - but set yourself on a path where their participation is inevitable. Making data as a first class citizen doesn't just happen. There is incremental work to be done by the domains. Make sure you reserve time to actually do that work - the data quanta creation and maintenance. It is not feasible to have your documentation be fully self-describing for everyone. eDreams chose to set the bar at documentation that is self-describing for readers that already know about the domain. For readers that do not know the domain, that introduction must happen somewhere else. At the start of a data mesh journey, your central team will likely control all the use cases being served by the mesh. But at some point, self-serve needs to happen. Consumers need to be able to serve their needs without their use cases going through the central team. With small-size data products and the data combination burden being on consumers, versioning tends to not be as much of a problem because the concepts of domain events don't change that often. Or when they do, the retention window has been typically short, making versioning easier. To make data producers feel a better sense of ownership, 1) look for ways for producers to better leverage their own data; 2) maximize your number of consumers for their data quanta so there is a quicker time to identify issues with the data product - more eyes means more who can spot issues; and 3) create automation to easily/quickly let domains identify sources of data loss rather than searching: with proper setup, you can make it easy to identify if the data pipeline is the problem. If it's not, then the issue is in the domain. When Carlos and team were looking at building out how to tackle their growing data challenges a few years ago, they were looking at request for proposals (RFPs) from a number of data consultancies around building out a data lake but just were not convinced it would work. Then they ran across Zhamak's first data mesh article and decided to give it a try themselves. Until more recently, Carlos was not aware of the mass upswing in hype and buzz around data mesh so their implementation is very interesting because it wasn't really influenced by other implementations. When they were starting out, Carlos said they didn't want to try to create a single, overarching approach. It was very much about finding how to do data mesh incrementally. They started use case by use case and built it out organically, including the design principles and rules - they knew they couldn't start with a single data model for instance. But it was quite challenging iterating towards that standard data model. When choosing their initial use cases to try for data mesh, Carlos and team had some specific criteria. They rejected anything that needed a very quick turnaround because it wouldn't let them have enough time/space to try things, learn, and iterate. They did plan ahead by creating foreign keys to data products that didn't exist to make interoperability down the road when they would exist easier. And they were very honest with stakeholders about what early participation meant - and what it didn't mean; that way, it was clear what benefits stakeholders could expect. According to Carlos, while they had executive support and sponsorship for data mesh, that wasn't enough to move forward with confidence at the start. They needed to have a few key stakeholders that were engaged as well and wanted to participate. It was also okay to have some stakeholders not engaged but just informed of what they were trying to do with data mesh. You don't have to win everyone over before starting. Five things Carlos thinks others embarking on a data mesh journey should really take from their learnings: 1) it's okay to not have everyone really bought in or especially engaged upfront but they will have to participate - make their eventual participation inevitable. 2) Really emphasize that you are learning in your early journey, not that you have it figured out - and factor in learning when doing estimations and promises. 3) Don't try to design your data model from the beginning; you need to learn via iteration - you will start to find your standards to make it easy to design new data products. 4) When treating data as a first class citizen, it's important to understand that will take additional time. Reserve the team's time to create and maintain their data quanta. 5) Let the use cases drive you forward and show you where to go. Carlos' philosophy is, within reason push as much of the burden onto the consumer as you can. Obviously, we don't want consumers doing the data cleansing work - that's been one of the key issues with the data lake - but the costs of consumption should fall on the data consumers as they are the ones deriving the most benefit. So eDreams makes the consumers own stitching data products together for their queries and makes them pay for the consumption. This minimizes the costs - including maintenance costs - to producers. One very interesting and somewhat unique - at least as far as Scott has seen - approach is how truly small Carlos and team's data quanta are. Thus far, they have really adhered to the concept that each data quantum should only be about sharing a single type of domain event and really nothing more in it. This again makes for lower complexity and maintenance costs for data producers. They are considering changes with upcoming BI-focused data products so that is to be determined. Carlos believes - and Scott exceedingly strongly agrees - it is not feasible for your documentation for your data quanta to be fully self-describing. You can't know someone else's context. You need to write good documentation so people can still understand what the data product is and what it's trying to share but if you do not have knowledge of the domain, it would be a considerable amount of effort - essentially impossible to do it right - to fully explain the domain and how it works in the documentation of each data product. Getting to know how other domains exactly work is outside of the scope of the data mesh. At the start of their journey, the data team was in control of all the use cases, who was consuming, and who was producing, according to Carlos. But, as they've gone wider and there is a self-service model for data consumers, more and more of the use cases are directly between the producers and consumers - or the consumers are consuming without much interaction with producers if they already know the domain. It could become an issue with people trying to understand data from lots of different domains for the sake of understanding but it hasn't been an issue so far. To date, Carlos hasn't seen many problems around versioning. They thought they would have many more issues with versioning than they have which Carlos believes is from keeping their data products as small as possible and using domain events. When they have had versioning, the retention window for the data has been relatively short so the versioning has been relatively simple to move to the newer version. And because most people are getting their data from source-aligned data products, changes have a smaller blast radius - they won't affect data products that are downstream of a downstream of a downstream data product. Domain events have been enough because their main stakeholder has been machine learning. They are now working on a different kind of data quanta for consumers such as BI, and they plan to include more governed versioning there. One of the biggest challenges early on according to Carlos was that domains didn't really feel the ownership over the data they shared. So to increase the feeling of ownership, they first looked for ways for producing domains to use their own data - as many other guests have mentioned. Second, they tried to maximize additional consumers of data products by looking for use cases. That led to faster feedback loops if there was a problem - more eyes on the data - so producers discovered issues sooner. And third, the platform team helped identify issues that might be in the system or in the data platform/pipeline process - if there was data loss, there is automation to help identify if it is on the platform side; if it's not on the platform side, then it is an issue with the domain. That one automation has led to a lot less time searching for the cause of data loss rather than fixing data loss. Carlos and team built in a few different layers of governance. The first is a universal layer for standard metadata in each data product, like when something happened, who is the owner, the version of the schema, the existence of a schema, etc. These are enforced automatically by the data platform and you can't put a data product on the mesh without complying. Producers must also tag any PII or sensitive information like credit cards. Then, a second layer is policies for data contracts between producers and consumers. As many guests have suggested, they have found having default values for SLAs in data contracts provides a great starting point for discussions between data producers and consumers. "You can have your cake and eat it too," using domain events per Carlos. You don't want direct operational path queries hitting your data quanta as they are designed for analytical queries - they will have a separate latency profile. But at eDreams, the pipeline that writes data quanta to the analytical repository is implemented with streams that can be consumed in real-time by operational consumers (microservices). Other tidbits: When launching a new data product, there must be a settling period - consumers must understand that things are subject to change while the producer really figures things out. You want to avoid duplicating data. But you REALLY want to avoid duplicating business logic. Data products should have customized SLAs based on use cases. You don't need to optimize for everything. Let the needs drive the SLAs. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Rerelease of #133 Nitty Gritty From the Deployment Committee: Crucial Learnings on Driving Buy-in and Data Product Discovery - Interview w/ Ammara Gafoor 1:19:06

۲ years پیش1:19:06

1:19:06

Due to health-related issues, we are on a temporary hiatus for new episodes. Please enjoy this rerelease of episode 133 with Ammara Gafoor. There is a ton to learn from this one and reflect back on. Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . ammara.gafoor@thoughtworks.com Ammara's LinkedIn: https://www.linkedin.com/in/ammara-gafoor/ Data Mesh in practice article series from Ammara and colleagues: #1: https://www.thoughtworks.com/en-us/insights/articles/data-mesh-in-practice-getting-off-to-the-right-start #2: https://www.thoughtworks.com/en-us/insights/articles/data-mesh-in-practice-organizational-operating-model #3: https://www.thoughtworks.com/en-us/insights/articles/data-mesh-in-practice-product-thinking-and-development #4: https://www.thoughtworks.com/en-us/insights/articles/data-mesh-in-practice-technology-and-the-architecture In this episode, Scott interviewed Ammara Gafoor, Principal Business Analyst at Thoughtworks who has been working on a few client projects related to data mesh including one for well over a year. Before jumping in, it's important to note that much of Ammara's learnings come from an implementation in a 100K+ employee company split into 21 high-level domains. So the definition of domain in this episode revolves around that context of a very large business unit, not a two pizza team size sub domain. Some key takeaways/thoughts from Ammara's point of view: There often is a hangup around data work, especially relative to data mesh, where people want to get it all right, all perfect the first time. That's never going to work. Get something decent out there, test, and iterate. Perfect is the enemy of done. No bikeshedding! If you don't look to change domain's KPIs to align their operational work to data mesh "you won't prioritize it - you cannot prioritize it." Make it easy for domains to prioritize data mesh work if you want it to get done. ?Controversial?: source oriented data products should not be made available to business users within the domain or to almost anyone in other domains - at least by default - as they are difficult to understand for anyone other than the highly data literate people in the domain. "Don't make things that you don't need yet." Build data products for use cases you've identified. Think of the target business outcome and work backwards to create the necessary data products. Far too many people try to build customer 360 solutions. And they very rarely work. No one ends up happy. Sometimes it seems like data mesh literature says get rid of many existing data roles - but the people in those roles are subject matter experts in how the organization and/or the domains do data. You want to leverage them as you evolve their roles and they can pick up new skills related to data mesh. When driving buy-in, be prepared to repeat yourself multiple times. And then repeat yourself multiple times more... Learn to speak the language of the domain as well to drive buy-in with the business people. Look for the 2 most obvious use cases inside a domain. Supporting those use cases, you will need 5-6 foundational source-oriented data products that will support many - most? - of the use cases for that domain. It will appear like you need many more source oriented data products but zooming out, you will see you don't need nearly that many. Typically, using the median, use cases are powered or supplied by two consumer oriented data products and each consumer oriented data product is powered by three to four source oriented data products. They planned for a single approach to work with all domains to start implementing and contributing to the data mesh. That proved to not work well and every domain needed it's own process to really make it all work. There are commonalities of course but there isn't a cookie cutter approach unfortunately. It's very easy for people within the same domain to talk past each other, especially in a functionally siloed org structure. Data mesh can be a forcing function to find a shared, common language to drive to better results. From Team Topologies, there are 3 streams: operating model, product stream, and technology stream. They should all be given equal weight. No, you can't just focus on the tech stream... There may be push-back from both IT and the business side to having the business people collaborate closely in a data mesh implementation. But it's very important to overcome that - that collaboration with the business people is crucial to making data mesh work. Business leaders may have - or at least believe they have - what they already need currently via shadow IT. It may be difficult to convince them to change their ways of working "for the greater good". When looking at driving buy-in with domains, look at how they are incentivized. Often, short-term results are more rewarded than focusing on mid- to long-term results. If next year's funding is based on this year's results, they will focus on this year's results... Typical personas in a data mesh implementation: the IT sponsor, the business owner, the "no stance/sideline", the "says yes to your face...", the product owners, and the data lake builders (aka those feeling the target on their backs). Ammara started off the conversation sharing about how she and her team "had it all laid out" for the plan to standardize how they'd bring each domain up to speed on data mesh - from the introduction of new ways of working to being ready to participate in the data mesh implementation in just six weeks. And then reality struck. Each domain is different and much like trying to explain the benefits or implementation of data mesh, a single approach for all audiences just didn't work well so they adapted. Every domain is unique and required its own unique approach to make implementing data mesh in that domain work. There are of course some commonalities but each of the 13-14 domains that are part of the data mesh implementation thus far has had its own unique challenges. So, Ammara shared some stories about working with different stakeholders. Often, the first stakeholder they encountered was an IT sponsor for the domain itself - either an IT leader in the domain or an IT counterpart for the domain. This persona typically wanted to bring them in and welcomed them with open arms. And while they were often bought in on data mesh, there was a push - from IT and often the business side - to only speak with IT. So Ammara and team had to work to get permission to also include the business people in the conversations about their proposed data transformation. Because without the business support and knowledge, your data mesh implementation is likely to fail. How many episodes have said tie your data strategy to your business strategy? But, the business people often have what they need currently via shadow IT. So why would they want to give that up? It's an emotional response to be asked to give up what you have for the greater good and the long-term. There is the concept of immediate returns - you build a dashboard and there is immediate potential value - versus the mid- to longer-term returns from things like building your data platform and building out your data governance capabilities. Ammara has seen many times there is not any incentive to wait and focus on the mid- to long-term returns - if your funding this year is based on results this year, focusing on your results 2-3 years out is often doesn't feel like an option. They won't get rewarded for that long-term work. And most domains don't even have the capabilities to do said mid- to long-term high-value work. But to do data mesh right, we need to incentivize patience - and incentivize and provide the capabilities to do things right for the long-haul instead of just the short-term, low stakes wins. According to Ammara, as part of a successful data mesh implementation, there is the technical stream - the Team Topologies meaning of work stream - but you must also work on the operational stream at the same time. And the product stream too. If you don't look to change domain's KPIs to align their operational work to data mesh "you won't prioritize it - you cannot prioritize it." You need to put a metric into place to measure progress - it doesn't even have to be a great measure! It's a way to start the conversation. There is too much of a hangup in data mesh around trying to get things perfect the first time. Get it done, measure it, iterate on it, and move forward. Don't let perfect be the enemy of done and/or good. Don't fall to bikeshedding. The cost of change and the cost of failure in data historically have been very high per Ammara. But we have new economic models with cloud that make that no longer true. We now have "the privilege to be able to fail". Failure wasn't an option historically. But that's such a foreign concept to many, it will cause some to push-back. They have lacked the psychological safety to fail. And we have to understand why they are pushing back and work with them to understand that failure in a highly agile environment is incremental learning. After picking the 2 most obvious use cases in a domain - again, the very large business unit concept of a domain -, Ammara believes it will reveal a 5-6 of the foundational source-aligned or "source oriented" data products of the domain that will be able to power most use cases. So just start building the MVP of those source-aligned data products because they will support other use cases down the road as well. On Personas, Ammara laid out a few she and team have run into: The IT sponsor - typically a Data Architect or Data/Analytics Lead; bought in to data mesh, likely after feeling the pain points as Zhamak has laid out. Trying their best to go wide on getting people bought in on data mesh and has some - but not a ton of - social capital to influence. Their social capital is more with the IT/data people and less on the business side of the domain. They are critical to get things moving. The Business Owner - generally supportive of the data mesh initiative but doesn't have the time - or the incentive - to spend time on the data mesh implementation. You're trying to get their support by the promise of making their lives easier. The Sideline Watcher - sees data mesh as probably 'yet another data trend'. Not pushing back but not taking a stance. Waiting for the tide to turn one way or another before making their own waves. The "Yes to Your Face" - will say yes to you and then just go do whatever they were going to do anyway… These are inevitable - try not to take it personally. The Product Owners - they are building the dashboards or the analytical solutions, desperate for the data. They really WANT to work with you but don't know exactly how - how can they get the resourcing and we're asking them to rethink the way they do their work. Help them figure out how they can partner where possible. The data lake (or other historical data paradigm) builders - have spent so much time and effort to build a viable data lake/warehouse/etc. Often fight you because you're going against everything they've built. It's not personal against the data mesh team but it is personal if you put all their hard work aside. But they can build data initiatives very well, try to work with them and let them know you're building off the knowledge they've gained if not their direct work. For Ammara, a lot of the data mesh literature and conversations feel like they say there are new roles and therefore there isn't room for many existing data roles, like the data warehouse or data lake builders/maintainers. But she thinks that's not a great idea - and Scott agrees. They are subject matter experts in how the domain's data flows and systems actually work and can be excellent guides to bringing more people into the data fold as they themselves pick up new skills. Trying to hire your way to a data mesh is not a great idea… No one is redundant, everyone has valuable knowledge for Ammara. You need to make your IT sponsor successful in order for your data mesh implementation to go broad in that domain so that means learning the - and communicating in the - language of the business according to Ammara. That might mean you have to deal with the horror of PowerPoint Presentations. And as many guests have said, the selling points and implementation details of data mesh don't stick with the broader audience the first time. Repetition, reframing, holding of hands, etc. You won't succeed if you try to just message once. Be prepared to repeat yourself. And then repeat yourself again. Ammara gave an example of why data mesh can really help improve communication and drive to common language. In manufacturing, there is the concept of "on time, in full delivery" as a very crucial KPI. And the domain had analytics teams constantly asking to build this for the different manufacturing lines while at the same time, the business side said they didn't have the information. How could that be when there were 10+ completed "on time, in full delivery" projects that had been funded? So once Ammara and team removed the data team from the picture, the business folks were able to talk with the regular IT team and they came to a shared, common understanding of what was actually needed and what was missing. It's pretty easy to lose sight of what the actual need and use case is when people are siloed by function. It is crucial to understand the three streams of work model, per Ammara. The operating stream is "building the cadence for IT and business to communicate" in order to prioritize. This helps identify which data products will be built. The product stream is identifying the actual data products that need to be built, as in what are the scope and boundaries. The technical stream is about building the data product and the platform needs. Each of the three streams should have equal weighting. This is another way to think about your MVP thin slice, you must encapsulate some of each capability, each stream. As previous guests have noted, many domains build data products that benefit themselves first in Ammara's experience. This obviously makes it easier because there is more buy-in and no cross-domain communication and prioritization friction. But that is just the initial stages of a data mesh implementation - still in phase 1 before going truly broad. More domains are moving to support use cases across domains so phase 2 might be up soon. Ammara does not believe source oriented data products, ones that are difficult to understand outside the domain, should not be made freely available on the mesh; they should not be made available to business users within the domain or to other domains. And her reasoning is very sound: if the data products are difficult to understand, it's easy to misuse them and they are more likely to change with the source systems so breaking changes/versions are more common. Other domains can consume the information from those source oriented data products in specially designed consumer oriented data products instead of directly from source oriented data products. Data scientists are a bit of another story as they are data literate enough to do some spelunking but even then, data scientist beware. Ammara is also seeing an interesting pattern relative to source oriented data products. When you really start to map out a lot of obvious use cases for a domain - and remember, the size of a domain in this context is quite large -, it might seem like you need a large number of source oriented data products. But when you zoom out further, it becomes clear that you can actually shrink those into a much smaller number, that 5-6 data products mentioned earlier for that domain. The way things are evolving at Ammara's current client is 3 layers relative to data products and use cases. For each use case, there are one or more - typically two it sounds like - consumer oriented data products. Then each consumer oriented data product is derived from or powered by typically three to four source data products. So the domains are able to create multiple consumer oriented data products off the same set of 5-6 data products. But it's still early days and will likely evolve further :) Encourage people to think business need first instead of data first according to Ammara. Think about what business outcome you are trying to achieve and then work backwards to what data you need to address that. If we are just sharing information without intention, it can lead to misuse of data - will people really...…

D

Data Mesh Radio

1
Rerelease of #177 - Zhamak's Corner 14 - The Data Can't Protect Itself 20:22

۲ years پیش20:22

20:22

Due to health-related issues, we are on a temporary hiatus for new episodes. Please enjoy this rerelease of episode 177. As stated in the original show notes, this is one to revist often as it is a great level-setting on why are we doing what we do in data mesh. Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Sponsored by NextData , Zhamak's company that is helping ease data product creation. For more great content from Zhamak, check out her book on data mesh , a book she collaborated on , her LinkedIn , and her Twitter . This is likely to be an episode to revisit. Zhamak explains a simple concept - data should not be copied unless it is owned by a data product - but the why is multi-layered and important. It might be one of the most important yet underestimated aspect of data mesh because when done right, it truly ensures trust in data - for consumer but also producer. There's a lot of nuance in how Zhamak is thinking about this but the actual application is quite easy :) Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Data Mesh Radio episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of September 3, 2023 37:36

۲ years پیش37:36

37:36

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Rerelease of #52 Data Mesh Data Governance: Getting Out of Your Own Way - Interview w/ Sarita Bakst 1:11:30

۲ years پیش1:11:30

1:11:30

This is a rerelease of a previous episode as we are on a hiatus related to Scott's recent health issues. This is a very important episode for many reasons so please do check it out again :) Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . Data Mesh Learning Meetup Presentation: https://www.youtube.com/watch?v=7iazNKG8XQo Sarita's LinkedIn: https://www.linkedin.com/in/saritabakst/ In this episode, Scott interviewed Sarita Bakst, Managing Director and a leader in Firm-wide Data Management at JP Morgan Chase. Sarita was previously on a Data Mesh Learning meetup and is helping lead the firm's data mesh governance charge. Per Sarita, historically, data governance has meant controls and gatekeeping to most people - typically getting in way of innovation. So there needs to be a focus on changing that narrative, not just through words but actions to show it's not the case. In data mesh, you need to ensure that domains can make good decisions on governance and seek out subject matter experts when it makes sense. Sarita covered that one of the key issues in the way governance has been done is the people making decisions - the central governance team - don't have the real understanding of the data. When those decisions are put in the hands of the people who really know the data, but with guardrails and guidance, the fear is lifted about can we actually use this data and how. This opens up lots of new opportunities to leverage your data. Sarita strongly recommends starting with purpose-built data products. Find a use case and build data products to serve that specific. And data products MUST be about unlocking business value. You don't need to serve up all of a domain's data on day one, in version one of that domain's first data product - make it extendible and reusable so you can find additional consumers and expand over time. Get out of your own way on data governance in data mesh. You are going to learn and your approach will evolve as you learn. It's okay to not know everything upfront, set yourself up to not get in trouble - put the proper guardrails in place - but you won't know everything. Think of designing your risk controls as toll-gates and make sure they aren't bottlenecks. Have standards (not standardization) so people don't have to invent things from scratch. Standards for interoperability, naming, etc. - think of them as guiding principles instead of rules. Make sure domain owners know who to contact and when on governance subject matter expertise. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Rerelease of #44 A Pragmatic Approach to Getting Started with Data Mesh at Northern Trust - Interview w/ Khanh Chau 1:16:11

۲ years پیش1:16:11

1:16:11

This is a rerelease of a previous episode due to my health related issues. New episodes will begin again in a few weeks. Please enjoy this very important episode of Data Mesh Radio. Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Khanh's LinkedIn: https://www.linkedin.com/in/khanhnchau/ In this episode, Scott interviewed Khanh Chau, Lead Architect for the Data Mesh Initiative at Northern Trust. Khanh believes you have to be passionate about making data better to do a good job implementing data mesh. And it is DEFINITELY a journey so you need patience and vision. Also, each journey is unique, you can't just copy/paste from another organization. You need to make failure okay - but you should look to make it easy to fail fast, measure, and adjust. Khanh talked about the need for exec buy-in before heading down the data mesh path. They got that exec buy-in by proving that the total cost of ownership of data was quite high as the consumers had to do a LOT of work to get the data to usable. When speaking internally, the business people were very excited to participate if it meant they could get quality data. Some of the IT/data engineering folks were harder to convince. It was especially hard to get them to shed layers of not-useful technology. Some IT teams were easier to convince - they had felt the impact of a few too many middle-of-the-night data downtime incidents. Other teams hadn't felt that pain so there were harder to win over. There was also the incentive of additional possibilities - data mesh meant they could do things they couldn't do before. Khanh talked about making the platform the easy and right path for 80% of use cases. They focused on making things easy to configure; basically: what transformations do you want to do and then it automatically provisions the pipelines. Their goal was to make it easy to make good progress quickly; their time to initial deploy went from 2-3 months per data service to 2-3 weeks per data product and they hope to drive it down further. Northern Trust has been moving forward with data mesh for about 7 months as part of their high-level digital transformation initiative. On the data side, they had previously focused on data virtualization and data federation but it was not delivering the results they wanted. It was not as scalable as they wanted - it was taking 2-3 months to launch each new data service. They also did not have great information on who was consuming the data and why. For their data mesh proof of concept, Khanh and team set a timeline of 9 weeks. They needed to prove value by then or data mesh would be a very tough sell internally. Khanh talked about the need to sell data mesh as a paradigm shift in order to get people out of technology-focused thinking. Northern Trust decided to take a pragmatic approach e.g. not pushing all aspects of data ownership fully left. Khanh and team were focused on finding a "happy balance" on data product SLAs and quality - improvement was necessary but the team preferred done to perfect. A big focus and a key driver for Northern Trust has been building muscle and learning/evolving along the way. It's important to evolve quickly and not build muscle in the wrong way. Northern Trust is still in the early days on figuring out interoperability between data products. It's more of an art than a science. Khanh believes bi-temporality is more important right now than interoperability. There are a lot of great learnings to takeaway from the Northern Trust journey. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Rerelease of #25 The Most Key Data Mesh Takeaway: It's Not Just You - Mesh Musings 5 21:00

۲ years پیش21:00

21:00

Due to health-related issues, we are on a temporary hiatus for new episodes. Please enjoy this release of episode 25, potentially the most important episode of Data Mesh Radio. Please Rate and Review us on your podcast app of choice! Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of August 27, 2023 16:30

۲ years پیش16:30

16:30

Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
New Episode Hiatus Announcement 1:39

۲ years پیش1:39

1:39

Due to lingering severe health issues and not trying to rush myself back, I will be doing a 3 week hiatus around new episodes. In that break, I will be rereleasing some old episodes that I think people really need to listen to or possibly reflect on if they haven't listened to them lately. Here's to better health in the near future!…

D

Data Mesh Radio

1
#251 Panel: Tying the Business Strategy to the Data Work and Vice Versa - Led by Burce Gültekin w/ Ghada Richani, Beth Bauer, and Michael Toland 1:02:10

۲ years پیش1:02:10

1:02:10

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . #251 Panel: Tying the Business Strategy to the Data Strategy and Data Work (and Vice Versa) - Led by Burce Gültekin w/ Ghada Richani, Beth Bauer, and Michael Toland Burce's LinkedIn: https://www.linkedin.com/in/burcegultekin/ Beth's LinkedIn: https://www.linkedin.com/in/beth-bauer-102449/ Ghada's LinkedIn: https://www.linkedin.com/in/ghada-richani/ Michael's LinkedIn: https://www.linkedin.com/in/mjtoland/ Michael's 'Super Chicken' reference talk by Margaret Heffernan: https://www.ted.com/talks/margaret_heffernan_forget_the_pecking_order_at_work In this episode, guest host Burce Gültekin, Chief Data and Analytics Officer at FrieslandCampina facilitated a discussion with Ghada Richani, Managing Director, Data & Technology Strategy & Project Management Office at Bank of America (guest of episode #206), Beth Bauer, CEO at her own consulting company PosiROI (guest of episode #218), and Michael Toland, Senior Product Management Consultant & Coach at Pathfinder Product Labs. As per usual, all guests were only reflecting their own views. The topic for this panel was how do we tie the business strategy all the way down to the data work via the data strategy and vice versa of making sure the data work is tied into the business strategy. It's a complex topic so a lot of this is about driving to good ways of working and keeping sight of the actual overall business goals. Scott note: As per usual with panels, I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views individually. Scott's Top Takeaways: It's easy to lose sight of the data work when focusing on a data strategy. Make sure to understand and layout your understanding of what work will need to actually happen - at a high level - to deliver on your data strategy. Essentially, make sure you can execute on your data strategy via work and not words. And don't you dare forget your governance 😎 In many organizations, there is some - maybe a LOT of - cultural change necessary before the data strategy influences the business strategy. You need to prove the data work drives business outcomes and is aligned to the business strategy for a while - maybe 2-3 years - before the data strategy can be a key factor in shaping the business strategy. Think about layering your data strategy to include what you can accomplish, what value you can drive today based on past investments and what you are trying to build for the future, where additional investments could drive more value down the line. If all you focus on is areas for additional investment instead of reinforcing the value already generated, it can feel like past investments haven't really paid off. If you don't share about what capabilities you've built and the value, why would someone want to invest in future capabilities? Well executed data work - at its core simplicity - helps people to make better decisions. Lean into how your data work will help people to be better at their job. If you are pitching data as the key decision maker, most business execs will at best laugh it off but probably feel threatened. Show them how data makes them better at making the key decisions, not making the decisions for them. The business strategy and data strategy are frameworks for making decisions - help your organization understand how data will make them better at making decisions. Not just by being more informed but by having more focus on what information matters. There are so so many inputs to consider, people are information overloaded. The data strategy should be about finding what matters and elevating that so leaders can focus on execution, not sorting through and absorbing information from 150 dashboards. To develop an effective strategy, you have to understand where you are as a business. How can you know where you want to go and how you want to get there if you don't? Look to assess your high-level organizational capabilities as well as at the line of business level so you can help people align on the work to be done. A low data literacy team trying to do advanced AI is a bad strategy 😅 To create a good data strategy, you need to actually understand what a data strategy is and how it applies to your specific organization. A data strategy isn't only about desired outcomes but the entire information lifecycle from data acquisition/generation through to management and then leveraging the information gained to drive business decisions and outcomes. And say no to data for the sake of data! Data leaders should help their business leaders to get data fluent to a degree where they can leverage data and know when they are out of their depth relative to data and should call in the experts to help them. We don't need every executive to be highly data literate as long as they recognize where they have weaknesses and lean on the right people to help them with data when it is called for. Other Important Takeaways (many touch on similar points from different aspects): As Burce said in her wrap-up: "Communicate, communicate, communicate and make sure everyone has the right incentives in place so that their behavior is moving [the business and data strategies] forward." Think of data as feedback from the real world about what is happening and what you are trying to do. If we purposefully stop people from thinking of data as this mystifying, all powerful force, what more could we do? How many more people would be willing to use data more frequently? If the data strategy is its own wholly separate plan/approach, you're probably doing it wrong. It must be closely aligned to the business strategy or you are likely headed for trouble. A good data strategy is an enabling function to allow data and data work to support the business strategy. If your data strategy isn't tightly aligned with your business strategy, your data work to execute on your data strategy is far more likely to be seen as a cost center. Make sure the rest of the organization understands how the data strategy supports their work and key business targets for data to be seen as a value generator. Many executives think of data as mostly powering their applications and interactions with customers/partners. It's crucial to get them to understand how data can help make better decisions looking forward and not merely reporting on what happened. Especially because they need to help generate the data to use to make those forward-looking decisions. Before doing data work, you should always ask what is the business justification. Things might not turn out how you expected but it's important to share with others and also make sure you are focusing on work that should drive business value. It also makes it easy to show the value you delivered instead of people believing the value happened elsewhere. Data work enables value, look to show that often. If you want a data strategy of becoming more data driven and more data fluent to succeed, you have to take into account the organizational and cultural change aspects. You don't have to solve for everything but technology isn't going to solve your data literacy challenges for you! When focusing on the data work, think about the small steps. You don't need to map every single step out but you can't try to take giant leaps and expect things to work out well. When building out the data strategy, look at timelines to value and making people understand when value will come. If you only focus on quick wins, you will see your potential for data wins dry up over time. You have to balance near-term wins and building long-term capabilities to create far more capabilities for wins in the future. But only focusing on the 2-year out horizon probably won't win you any followers, friends, or funds. A potential good leverage point to bring business execs in on the data strategy is around risk. What risks are they looking to take and how can data better enable them to understand and quantify those risks to ensure the potential rewards are actually worth the risk? Throughout the data strategy and data work, always tie things back to specific business problems. Not having a good data platform isn't a business problem, not being able to handle your data at scale to reliably and quickly make decisions on certain challenges is a major business problem and not having a good data platform causes that business problem. If you provide data without context, your business partners won't really care. Same for data work. Think about the business context of what you are trying to do and communicate that to drive alignment and buy-in. Data leaders should certainly want to be part of the conversations shaping the business strategy. But unless you've got a proven track record in your company of data driving your business results, it might take time to earn your seat at the table. While unfortunate, that is totally normal. Cultural change unfortunately typically takes a long time. If your company is not heavily focused on data, it may take a long time before the business strategy is influenced by data. You need to get more and more momentum and change people's view of data and their experience with leveraging data to do their jobs better before you can expect data to be integral to the business strategy. So start building the momentum now! The data strategy is just half the battle. You need to incentivize people to actually execute on it, to do the data work. Incentives matter far more than we typically give credence to in data. Find your advocates, your data champions. Those are your best tools to really drive cultural change, the people sitting alongside other business folks and showing them the power of embracing data. If you are tying the data strategy to the data work, you need to understand how you already do data work. Coming up with a grand strategy perfectly aligned to the business strategy that completely changes how data work gets done is going to be a real mess. If the people crafting the business strategy aren't data literate, are they going to create a strategy that relies on data to drive results? Probably not. Try to help them get to a decently data literate level. When building your data strategy and focusing on the data work, you should consider what work ties to which parts of the business strategy. If the company isn't trying to do real-time data interaction and analysis, do you really need that streaming ML solution? If you can tie your investments directly to business objectives, it's easier to get funding and to keep the attention of business leaders around data initiatives. It's okay - and probably a best practice - to iterate on your data strategy as you do more data work. Are you getting more or less aligned with the business strategy? Even if the business strategy is a once a year proclamation from the CEO and Board, you don't have to live with that time table. Test, learn, adapt. The data work alone never adds value. If you deliver to an amazing data strategy and the organization can't act on, can't take advantage of the work, then it was a waste of time and effort. Don't get too far ahead of the organization while you improve what they are doing relative to data. Well-functioning data teams do innovation work that may or may not pay off. Try to keep that in mind when you build your data strategy. If no one is trying and potentially failing, you are probably playing things too safe. Keep your blast radii small if things blow up, but create space for innovation. When talking about data work and value, business value is not always in dollars. Make sure people understand that. Ask your business partners how much they value certain outcomes. The world is changing so much faster than it ever has before. Execs need inputs at a much quicker pace than historically to stay on top of things and ahead of the competition. Work with them to understand their pain and align on data work to address it instead of focusing on the cool tech aspects. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#250 Staying Aligned on the Vision in Data Mesh - Lessons from Volkswagen's Journey - Interview w/ Christoph Spohr 1:11:07

۲ years پیش1:11:07

1:11:07

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Christoph's LinkedIn: https://www.linkedin.com/in/christoph-spohr-2a33b619b/ In this episode, Scott interviewed Christoph Spohr, Lead Architect of Big Data Platforms and the Product Owner of Data Mesh at Volkswagen Group. To be clear, Christoph was only representing his own views on the episode. Quick Note: apologies that Scott's audio is a bit weird, he had yet to build his makeshift sound studio in the Netherlands. Some key takeaways/thoughts from Christoph's point of view: "There are not solutions, only trade-offs." Understand that nothing is perfect. !Controversial!: Your biggest risk to a use case is choosing the wrong abstraction, the wrong architectural paradigm. When working with a domain, look for a way to achieve the required result that is as close to their existing way of working as possible. Scott note: this can become challenging on finding reusable ways of working if you go too far but greatly smooths domain-level adoption. It's all a tradeoff 😜 The second biggest risk is do stakeholders really feel and acknowledge the pain the existing approach/system is causing and want to move? The third biggest risk is about promising things you cannot deliver - obviously look to avoid that. ?Controversial?: A really key aspect of doing data mesh well is having and keeping perspective: doing data mesh is more about "aligning people on a shared vision and not getting lost in the overall process." Don't get so bogged down in exact implementation details if possible. Keep people informed and aligned and start your conversations on clarifying what people believe/understand about data mesh so you aren't talking past each other. For many people in your organization, their understanding of data mesh will be shaped by vendor content/conversations. You need to check alignment many times as it can drift. Advice to former data mesh self: Really focus on keeping that communication between all parties going and making sure people are aligned on the approach, complexities, trade-offs, and risks. And don't think people automatically stay aligned, stay on top of the communication and relationships. In a large organization, if you want to implement data mesh, find sweet spots for it to succeed. Because you will have to make compromises/trade-offs. Everything in data and IT is about finding the right compromises but don't put yourself in a place where your compromises fully undermine your data mesh vision. When putting together your enterprise-level vision for data mesh, you need to break it down into two parts: A) how would it work with and serve existing business processes and B) how to approach the IT architecture and assessing/acquiring the necessary capabilities to do data mesh. Data mesh is one of the most - if not the most - complex data architectures and you need to have a strong understanding of the complexities and not underestimate them. Otherwise, scalability will likely be a significant challenge. Data mesh does not mean all your data problems go away. As Zhamak says, it also creates new ones. It's important to listen to stakeholders' challenges and deliver a solution that addresses those challenges. If you try to deliver to their exact wants, those often aren't scalable ways to address their challenges. Focus to the end result, not what they _say_ they want. When it comes to data mesh, make sure stakeholders understand that there is a cultural change component. If they want things to work at the speed and scale data mesh can provide if done well, approaches and hearts/minds have to change. Data mesh is not a magic wand to wave at your data to make your approaches/solutions scalable. ?Controversial?: Your IT people need to work on their communication skills. And watch out for IT/data team members getting a big head about what they can do, that can create lots of challenges. Plant your data mesh seeds early - have conversations with stakeholders, even when they aren't ready to take advantage of the data mesh approach/your implementation yet. They will be much more likely to come to you for help when they are ready and eager. Try to understand your senior leaders' IT systems background and their favorite - or at least known - approaches. When you frame data work in a way they understand, even if some of the implementation aspects change, they are much more likely to be in favor of it. ?Controversial?: Instead of going to the business leaders, look to talk to the technical people within a domain first to find out the business challenges at the technical level. That way, you can bring the business leaders an idea of an approach that is more fully formed. Scott note: you can take this too far quite easily and try to come in with only a technical solution to the challenge. ?Controversial?: People think data platforms are mostly about doing more and more complex analysis on data but that's rarely the use cases that add the most value. Scott note: strongly agree, reducing friction to leveraging data is rarely about the most complex data and analytics practices. "[My] loyalty to data mesh ends when complexity gets too big." Scott note: take this one to heart. Data mesh for the sake of data mesh helps no one - except maybe a resume 😅 Don't get bogged down by analysis paralysis. Look to work in a simple and speedy manner - if you have to backtrack, it is almost certainly still going to be quicker than trying to answer every question ahead of doing the work. Christoph started the conversation with some perspective into the IT/data landscape at Volkswagen where they have 100Ks of employees and 100s of petabytes of data. So, when you think about what good data mesh practices look like in that environment, at that scale, it's much more about "aligning people on a shared vision and not getting lost in the overall process." And it's easy to lose alignment because so many people are getting their understanding about data mesh in general - or at least aspects of data mesh - from vendors. Those definitions are often very different, especially around how much is centralized or decentralized. So look to start your conversations with colleagues by clarifying everyone's understanding of data mesh or a specific aspect of data mesh to stay on the same page. Your understanding will change over time and so will theirs. In data and IT work, at the end of the day, everything is about compromises and tradeoffs for Christoph. Nothing is the "right" answer, it is about the search for the least bad answer. When you are looking at data mesh in a very large organization like Volkswagen, he recommends looking for your sweet spots - there are places data mesh could thrive and places where it's not a good fit. Don't get too bogged down in trying to go organization wide early on. You couldn't support that anyway! Christoph recommends before getting too deep into your data mesh implementation strategy, break down your high-level vision into two pieces. The first is: how would this align to and serve the existing business processes? You need a business side of your vision - what are we doing data mesh for? You also won't be able to change the way the organization works easily so find ways to align to how things already work. The second is the IT architecture and the necessary capabilities to actually pull off doing data mesh well. How do you get into a data-driven mindset? What are the kinds of technical implementation details you will need to accommodate - not even the use cases, just what types of domains and sources. How will you manage risk, how will you make sure things can scale? Choosing the wrong abstraction, the wrong architectural paradigm is the biggest risk to a use case in Christoph's view. Each team has a different way of working and you should look to, still within the framework of data mesh, find a way of working that achieves the target results but is as close to their existing way of working as possible. This will likely smooth adoption friction and prevent some misunderstandings. His next two biggest risks were first, make sure you have stakeholders that really feel the pain of the existing approach to the use case and systems and are willing to move. If there isn't a desire to move, it's difficult to drag them along and succeed. The other - the third biggest - risk is about don't over-promise; make sure what you say you will deliver is realistic. For Christoph, "there are no solutions, only trade-offs." When discussing approaches, let people know the trade-offs you are making because nothing will be perfect. That honesty will resonate well with most people. In Volkswagen's journey specifically, Christoph expected the biggest pain point for customers to be that data was in silos but it wasn't - they were more concerned with fine-grained access control. So when he talks to stakeholders, he makes sure to understand their pain first instead of trying to pitch the approach they've landed on with data mesh. But it's also important to not ask people what they want in his experience, they will try to just add scaling to their existing approach when a new approach is often necessary. So, listen for the pain, pitch to the pain, but don't pitch to their exact desired approach if it's not a scalable solution. When asked about advice to his former data mesh self, Christoph circled back to the challenges around communication. You need to constantly be staying in touch with people, aligning on vision and sharing trade-offs. You also need to focus on what are people's pains and look to deliver something scalable to that pain rather than asking them what they want - they often want their current approach to magically scale. You have to partner with them and make sure they understand that cultural change will be necessary to accomplish their goals - this isn't a magic mesh wand. And he also recommends watching out for any of the IT/data team getting too big of a head because they can do really interesting things with data. Christoph recommends a few things for getting more buy-in for your data mesh implementation in a large organization. One is to plant your seeds by having lots of conversations about what you are trying to achieve because when people are ready, they will come to you. A second is to talk to a lot of people to understand where big challenges/opportunities are in the organization. That way, you can try to focus a bit more on high-value use cases. And the third is try to frame your approach and choose an abstraction that matches the senior leader/stakeholders. If they like one approach, try to frame things - and hopefully develop your approach - in a way that works well for them so you aren't trying to drive buy-in for the approach as well. Volkswagen is potentially a different kind of organization to many others out there. In Christoph's experience, as a hierarchical organization, it often makes sense to talk to the technical people within a domain first to assess what challenges there are and come to the senior leaders with something more fully formed and targeting their actual business challenges rather than asking them to share their data related challenges. Be opportunistic in selecting use cases to go after. And he is also not loyal to the concept of data mesh - if it gets too complex at scale, he will look for something else. As to ways of working, Christoph recommends to go for simple and speedy because even if you head down a path that isn't working, it will still likely be quicker to pivot and head in a new direction than if you had gotten bogged down with analysis paralysis. Look to converse with other experts in the industry on your journey - Christoph was unlucky in finding people who had done data mesh at Volkswagen scale but that's not the case for most organizations. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of August 20, 2023 29:31

۲ years پیش29:31

29:31

Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#260 Driving the Big Picture Forward - More on Northern Trust's Data Mesh Implementation - Interview w/ Jimmy Kozlow 1:13:56

۲ years پیش1:13:56

1:13:56

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Jimmy's LinkedIn: https://www.linkedin.com/in/jimmy-kozlow-02863513/ In this episode, Scott interviewed Jimmy Kozlow, Data Mesh Enablement Lead at Northern Trust. To be clear, he was only representing his own views on the episode. Also, FYI, there were some technical difficulties in this episode where the recording kept shutting down and had to be restarted. So thanks to Jimmy for sticking through and hopefully it isn't too noticeable that Scott had to ask questions without hearing the full answer to the previous question. There is a lot of philosophical discussion in this conversation but tied to very deep implementation experience. It is hard to sum up in full without writing a small novel. Basically, this is one to probably listen to over just reading the notes. Also, Scott came up with a terrible new phrase, asking people to "get out there and be funky." Some key takeaways/thoughts from Jimmy's point of view: Start your mesh implementation with your innovators. Find the people who are excited to try out something new. You want to spend your early time on innovating and learning, not constantly driving buy-in. Find good initial partners! It's okay to start a bit simplistic - with each domain and on your general implementation - because you better believe there are complexities coming as you scale. This is not going to be simple. But the ability to tackle that complexity effectively is what differentiates data mesh so that unavoidable complexity ends up being where you find lots of incremental value. Data product complexity is often from maintaining interoperability across the data sets within the data product - not just to other data products. You want to build out a data product with more data sets if you can because that means there is much more information a consumer can easily leverage that is highly related instead of stitching things together across many data products. ?Controversial?: There is value in inorganic data product growth. A central team can see where there is a gap in the available data on the mesh. Even when there isn't a use case calling for a specific data product, filling in those important information gaps can create an environment where more use cases will emerge quickly. Scott note: I go back and forth on this a lot. Data for the sake of data is bad but if you identify gaps, it can accelerate a lot of new use cases. Financial services is somewhat unique in that many of the desired use cases are already known - lines of business have had their use cases often for decades so it's about serving them well in new ways rather than inventing new use cases whole cloth, at least at the start of a journey. ?Controversial?: Start from serving existing needs instead of trying to invent those new use cases. That imagination around new use cases will be important but there's probably a lot of things your business wishes they could already do to serve customers and that's where your low hanging fruit is. Go after those first to create adoption momentum. While data mesh might provide more initial friction to use cases, - you need to gather up more information early to embed the necessary regulatory/governance decisions - it means a better overall outcome. But getting people bought in that the initial friction is worth it can be hard. The most important metric - at least for Northern Trust - is shown value. That demonstration creates momentum through excitement and it pushes adoption as more teams want to get some of that value and they can also have the fear of missing out. What success looks like in data mesh will be different for every organization especially through the different phases of your mesh implementation. Scott note: this is so incredibly important. Consider what would make your implementation a success in each phase. It's important to understand the difference between creating the ability for teams to leverage data mesh and actual adoption. One is much more tech focused and is very important but value comes from people participating and using what you build. Just because you build it doesn't mean people will use it. Focus on ensuring adoption. It's crucial to constantly communicate why you are going this route with data mesh. Even if there is friction to getting something completed, it's "the right kind of friction." You are making your data work sustainable and scalable. There is a steep learning curve in getting to critical mass with data mesh. Each domain has a different capability level and trying to keep teams collaborating and communicating well can be a challenge. Don't assume good communication will happen, you have to enable it. ?Controversial?: Leverage your central team to help domains early in their mesh journey. Help from the central team is an easy way to inject speed to getting that initial data product shipped. And at the same time, they help bring that domain's capabilities up to a necessary level. When helping a domain, the central team's goal is to get them to good enough quickly and then give them the autonomy to do what's valuable. Things like internal sharing communities are then extremely helpful as domains can exchange deeper insights with each other and connect, creating potential collaborations. ?Controversial?: When first working with a domain, start simple. Don't go for an advanced use case or data product, build up their capabilities instead of trying to throw them in the deep end. There are so many complexities in data mesh. Constantly consider trading complexity versus value, whether that is speed, a deeper use case, etc. The complexities will still be there when teams are better able to deal with them, no need to try to tackle everything at the start. Take measurements - e.g. time to first data product production for a new domain - with a big grain of salt. If you are close to the domain, you can get a sense of how much friction is in your processes and platform but there are so many factors impacting time to launch an initial data product. There's a hard balance to strike between data modeled to fit the use case and data modeled to fit with the rest of the data that's available on the mesh. It's especially hard when the domain really doesn't understand how to model data well yet. Be prepared to help domains out to make sure they aren't just publishing a data silo in data product form. There are many places in your implementation where you want to reduce time and friction. Getting to a proper level of understanding isn't bad friction. Trying to rush people through their understanding of data and how their data plays into the organization is likely to bite you in the end. Same for trying to rush people through learning your new ways of working. While it might seem obvious, focus is such a crucial aspect to doing data mesh right. Helping people figure out what to focus on when and keeping the lines of communication open to figure out what's most important right now is TOUGH but valuable. Similarly, it's incredibly helpful to help people connect the dots, help them see what could be possible. If you have people focused on making sure others can understand the big picture of your implementation and the available data, they can contribute to that big picture so much more. An organization starting out highly centralized will have a very different journey to one that is starting out highly decentralized. ?Controversial?: Your central team is there to mostly "coordinate, facilitate, and align people to the policies and practices" plus drive adoption. Look to have a guiding hand, a light touch. ?Controversial?: Be very careful cutting corners in your implementation. There are many ways to hurt your scalability. Governance is especially a bad idea to cut in financial services. ?Controversial?: Ask yourself who, at the end of the day, is tasked with making sure your data mesh implementation continues to progress. Not the specific tasks like building a data product but the overall implementation. If there isn't someone focused heavily on that, you might want to reconsider your approach. Jimmy's role is a rather unique one in that he is literally tasked with enabling data mesh to happen and make sure they are focusing on the right things. That means a myriad of different things but a lot of it is ensuring the communication and collaboration happens where it needs to while still focusing on the big picture. It might be like an American football coach that coaches the entire team but is still calling plays and making the minute decisions during the game too. It's a big set of tasks to take on. At Northern Trust, they started their mesh implementation with the innovators according to Jimmy. That's a common pattern for all tech innovation, not just in data mesh - find the people enthusiastic to try something new so you don't have to spend half your time driving buy-in. There obviously also had to be a need where data mesh could work and would be a differentiator. For Jimmy, one big complexity factor he is seeing is around data products. He believes that you really start to drive value of a data product higher the more relevant and interoperable data sets you can include in the data product - within reason of course. But as you add that fourth or fifth data set, it gets complex to maintain interoperability and consumability even at the data product level. But that complexity is where a big part of the value really lies in data mesh - that it pushes organizations to take on that complexity but in a scalable way. There is a bit of a push/pull in data mesh for Jimmy: organic data product growth from new use cases versus the centralized team pushing for the value from more and more interoperable data - basically creating data products that fill the gaps between existing data products to enable additional use cases. New use cases emerge the more data products you have that are well crafted and that link data across domains; but the question becomes do you push for new data products that don't have a specific use case - inorganic growth - in order to create a tipping point where those use cases can quickly emerge? It would mean less work for consumers and faster time to market if the data is already available instead of working with the producing team. But will the data products be made well enough to serve use cases? Are people ready to go and discover pre-made data products instead of ones tailored to their needs? Should we only apply this to new data products - how would a central team spot when additional attributes are needed to take an emerging data product from only serving a use case to being more globally valuable? It's still a question in data mesh as to whether to pursue inorganic data product creation. At the start of their journey - and even though they are two plus years in, it's still relatively early - Jimmy and team are focusing on existing, known use cases as they build out the existing available data and improve their capabilities. Capabilities not just to deliver new data products but how to deliver incremental data that fits well into their already existing set of available data on the mesh. It's about building out the entire picture instead of focusing too much at the micro level. Scott note: balancing that micro and macro level is hard - extremely hard? - but the earlier you get good at figuring out how to add value at the overall mesh level while serving use cases, the more value you will deliver with each incremental data product. Jimmy talked about with data mesh, even though we can deliver scalable data products relatively quickly, there can be more initial friction for new use cases. E.g. teams have to go and collect the necessary information to do the governance well instead of trying to add the governance at the end. And some might be frustrated or not bought in that the upfront friction is worth it. So he's trying to show the value far exceeds that initial extra friction but people will of course resist new ways of working. Such is the nature of working with humans 😅 When asked what does success look like for them, Jimmy pointed to showing value. It's important to note that isn't merely delivering value but being able to show that value. As Jerry McGuire said, "SHOW ME THE MONEY!" Being able to show value generation helps to build momentum and adoption. If you are proving value then it's not nearly as difficult to get incremental investment. Teams want to participate and capture value too. Excitement builds. But it's also important to note that what success looks like will change - maybe not wholesale but at least in part - in different phases of your implementation. For Jimmy, there are two important aspects of your implementation, essentially the setup and the knockdown. You have to set up your implementation for success by building out the platform and capabilities but getting teams to actually adopt is still crucial. Just because you built something amazing, you still have to work with people to understand the shift in mindset and approach to get them to buy-in and adopt. A great platform that no one is using isn't really a great platform… Trying to keep momentum of the whole mesh implementation until you reach critical mass is very challenging. Jimmy talked about how trying to get teams with quite different capability and speed levels to work together can be hard. It's not as though any organization is built to all move together as one - it's not a car that's built to move as one unit, it's more like a group of cars - so you need to really focus on the coordination, collaboration, and especially communication. ABC - Always Be Communicating 😎 Jimmy believes that the central data team in data mesh should be a key point of leverage. They can jump in to help a domain early in their journey to get something delivered while raising that domain's capabilities. A central team can bring repeatable patterns to find easy paths for new domains. That way, the domains still learn by doing but they don't have to learn by repeatedly failing. But once a domain is capable enough, the central team tries to move out quickly to give the domains autonomy to do what's valuable. They are also building a community of practice for practitioners to share insights with each other, providing even more leverage and discovering more repeatable, high value patterns. When asked about bringing a domain up to speed and the question of complexity, Jimmy strongly believes that you should start simple. You don't hand the 7yr old who wants to help you cook the knife and have them go wild on day 1. Get them into a groove, get them confidence and understanding how to deal with data, then you can start to think about adding complexity. But dealing with data is complex enough, keep it simple for them to deliver value initially. When thinking about success and things like a new domain's time to first data product, Jimmy believes you can learn where there is friction but the actual times vary quite a bit. So you might see lengthening times for new domains launch a data product but it's a good thing because you are dealing with domains who really aren't sure what they are doing and have to learn a ton about data in general and their own data - that means you are penetrating the less data savvy parts of the organization. All things equal, you want to go faster but just take things with a grain of salt. There's also the fun of trying to thread the needle of data modeled for the initial use case - so fit for purpose - yet also modeled so it fits well and is interoperable with the rest of the data in your available mesh of data products. Jimmy said this is especially true of domains just getting up to speed with dealing with their data and data modeling so prepare for them to move slower and need more help. When asked about where there is friction in the process of bringing on new domains that we _shouldn't_ try to reduce, Jimmy pointed to learning and understanding. People need to take the time to understand how to do data work and all that but also understand the new ways of working and why the organization is going in this direction. It's that old are you trying to get them to do the steps you say or achieve the target outcome you give them. Learning takes time, don't rush it. Jimmy's role is pretty unique as far as other organizations telling their story. He's focused on helping people focus in the right areas but also on helping them connect the dots. There is such a big picture when you think about the entire information scape of an organization and helping people to connect to each other and see where they could enhance that bigger picture is highly valuable. This drives better value while also reducing miscommunications, duplication of work, and wasted time. Scott note: this is somewhat similar to the 'Data Sherpa' concept I've mentioned repeatedly that just about everyone looks like I'm a madman when I bring up. The question of what to decentralize versus centralize is a tough one for every organization doing data mesh. Jimmy pointed to the fact that where…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of October 15, 2023 21:06

۲ years پیش21:06

21:06

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#259 The Fellowship of the Data Mesh: The Journey to Meshtopia - Mesh Musings 54 14:50

۲ years پیش14:50

14:50

Key points: Every organization starts from a different location when heading to Data Meshtopia. Every journey will look different. Try to understand why they chose to do what when - often by just getting in touch and asking them. Because the journey looks different, where you put your focus when will depend. Look to the fundamentals of what are you trying to achieve, how to achieve short and long-term value, and how are you making progress. You can choose your own journey - there are different paths to getting to Meshtopia, just because you look like another organization doesn't mean you have to tread the same path. Think about the old trope of playing to your strengths while shoring up your weaknesses. This doesn't mean ONLY play to your strengths, that is where I am seeing the most failures… Just because you're bad at governance doesn't mean you can ignore it… Find your fellowship or fellowships. Find people to constantly stay in touch with and exchange information. They are also your best leverage points quite often: "Hey, we want to do XYZ approach, Scott here says this is a very standard approach from these 10 organizations and here are the pitfalls so we aren't making this up out of our butts." Meshtopia isn't some utopia. It's not as if all our problems are suddenly solved - some of the old lingering ones might be but there are new problems to battle too. It's not as if life is blissful the second you get to a better place. Please Rate and Review us on your podcast app of choice! Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#258 Data Mesh on Hard Mode: Learnings From Airtel's Early Data Mesh Journey - Interview w/ Sid Shah 1:16:15

۲ years پیش1:16:15

1:16:15

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Sid's LinkedIn: https://www.linkedin.com/in/siddharthin/ In this episode, Scott interviewed Sid Shah, Head of Product Data and Analytics at Airtel, a large India telecom operator. To be clear, he was only representing his own views on the episode. Before we jump in, it's important to note that Airtel are doing data mesh on 'hard mode'. Because of regulatory requirements/restrictions, they are all on-prem. That means extra challenges when it comes to securing compute resources. Some key takeaways/thoughts from Sid's point of view: Sometimes you have stop/start in data transformation. It's okay, you can rebuild your momentum. It's okay to try to move when you aren't 100% sure. One important thing about data mesh is that it can validate that you aren't the only organization facing a lot of the common challenges that come with high scale and business complexity/velocity around data. You are not alone. Scott note: there's a reason 500+ companies are doing data mesh :) Similarly, you can leverage the stories of other organizations on a mesh journey to get buy-in internally. Their stories can help you explain to your organization that others are using this approach, that this isn't just a problem in your organization. Many data issues in a large organization can probably be traced back to poor ownership in some form or fashion. Can the teams who should own data - the ones who know it best - even own the data if they wanted to? Do they have the tooling and capabilities? What data related pains are universal to your organization? Does data mesh target any of those? Those universal pains in your organization can be a great selling point for doing data mesh. Changing data ownership is very much not trivial. It's not about responsibility, it's about building up the capabilities to allow domains to truly own their data in a safe and scalable fashion. It's crucial to mesh - forgive the pun - data mesh and the way your organization functions and operates. It's crucial for implementing but also driving buy-in. It's a great way to alleviate push back when someone says 'that could never work here' or similar if you align it to your existing ways of working. The business doesn't want a sausage factory tour. Focus on speaking to their business problems and then actually addressing them. Data quality, velocity, reliability, etc. What can they do and what are you fixing, not how you fix them. A potential good mesh selling point for line of business leaders is time to market. If the time between deciding to run an experiment and it going live can go from a few weeks to a few days, maybe even a few hours, what does that mean for them? ?Controversial?: Getting outside in perspective to validate your approach can be a good starting point. Look to bring in outside presenters doing data mesh to talk about how data mesh is working for them and help frame it for your organization. Scott note: Controversial because I know of no one else doing it… ?Controversial?: As long as the eventual ownership lies with the domains, it is okay to co-own data and hand it over gradually. Give domains the time they need to increase their data fluency. If you can, get data mesh explicitly included in the engineering/data strategy for the organization. Scott note: Easier said than done and probably don't go for that until you're ready but it will make your job implementing much easier. As many others have noted, do not try to onboard every domain at the start of your journey, that's a recipe for disaster. Sometimes you have to tell domains you aren't ready for them yet, whether that is capacity or because their use cases are too complex for your platform right now. There's a temptation to go after the most 'value rich' domains but you will be far better served going with the domains that are leaning in, that are really excited to participate. Failure at some scale is inevitable in data mesh. Learn from it and move forward. Try to make sure your stakeholders understand that failure is not the worst thing that can happen if you can iterate based on it and if you limit the blast radius. Really consider what your focus should be in every stage of your journey. At the start, it's not about building the perfect solution, it's about building momentum, getting a few good wins to use to market internally, and learning where your friction points are so you can build a scalable approach. It's easy to try to get every aspect of data mesh 'right' at the start but that will overwhelm you quickly. Make sure to keep all your stakeholders engaged. That might sound obvious but every persona - and oftentimes person - is quite different so you will need to focus on different aspects of data mesh when speaking to them to succeed in maintaining engagement. You'll need to keep a lot of people updated on your data mesh progress and upcoming plans. But very few care about the nitty gritty details. What's changing that will allow them to deliver more value? Consider whether you should immediately start billing lines of business/domains for using the central mesh platform versus making them aware of the costs first for a time. It can be a big shift in thinking and approach so switching the cost to domains if they aren't getting incremental budget could be a real point of friction/loss of buy-in. Scott note: this is a really important point It will be pretty hard to lock down your return on your investment in data mesh or for specific use cases. How does the business value faster time to market, fewer incidents, better quality data, new use cases, etc.? Scott note: you can flip it around and make it something domains need to measure or assign a value to but it's not exact :) Sid started by talking about some of the challenges of data and analytics in a very large organization that has grown through acquisitions - data is siloed across the organization and you can't get sight of the customer journey across lines of business. To address this, a few years ago they started landing data from across the organization into a central data lake. You can probably see where this is going - a central team getting more and more overloaded while there is an ever increasing demand for data. Add to that a data platform where the only teams sophisticated enough to use it being in the central data team and you have major bottlenecks, poor quality governance, the business teams weren't sure what to trust or what data really meant, all combining for a bad analytical experience. So that was the stage for why they started to look at data mesh. Even before coming to data mesh as a potential solution, Sid and team thought the biggest problem was poor data ownership. The teams who knew the data couldn't own the data even if they wanted to - they didn’t have a platform they could actually use or the data fluency to own their data work. The other issues could really be traced back to that - poor quality, bottlenecks, low trust, etc., it all came back to poor data ownership. According to Sid, coming across data mesh and all the other organizations that were implementing - including the stories from this podcast and the community - validated that they weren't the only ones experiencing these data and analytics issues. And it gave them not just validation but hope and a set of talking points and proof points to leverage internally for driving understanding and buy-in. Sid and team understood early on that changing data ownership is not a "trivial exercise". It's not just changing the responsibility, you have to enable domains to actually be able to properly own their data or it's just the same problem of bad data and bottlenecks but just new people to point the finger at. They decided to not try to solve challenges at the small scale but come up with an approach that was going to really target the pain at the enterprise level, so that's why they arrived on data mesh. Two things that helped drive data mesh forward as a concept at Airtel were a general understanding of data mesh and its aims at the senior leadership level in the organization and again feeling that pain for long enough that people were ready for a fix. A third aspect was creating a way for data mesh to actually happen at Airtel specifically - how could data mesh work in their organization without trying to change everything about the organization and how people/processes worked? Essentially, how can you explain how this would all work to people at their level of involvement - senior leadership doesn't worry much about what tech your platform uses - and that your mesh implementation won't disrupt the entire organization in a bad way while addressing a lot of the causes of data pain. As a number of guests have pointed to, when speaking to the business, Sid recommends to speak to their business pains. Data quality issues, not being able to nimbly react to the market based on data, reliability issues of things breaking, etc. The business wants to deliver on their objectives, not talk about how you are building the platform. Think about time to experimentation as well - if you can take the time to deploy an experiment in the market from weeks to days or even hours, what can that do for them? The product and engineering orgs love the sausage factory tour so keep it to just them :) When looking to drive buy-in and understanding of data mesh with the technical teams, the engineering and product teams/leaders, Sid brought in a number of people from other organizations doing data mesh. People got used to the idea of data mesh and also, if they are seeing many internal presentations on data mesh, they understood the momentum was building so they'd have to get on the bandwagon. Scott note: this is part of why Data Mesh Understanding exists - not everyone can easily bring in those outside speakers. When they were just starting out their journey, a number of domains gave the quite fair pushback of they didn't have the skill sets to do data mesh, to own their own data. They also didn't have the resourcing to take on the additional work, both people and compute - recall, Airtel is all on-prem. So Sid and team had central resources - including people - to loan to those domains who wanted to own their data, who were bought in, but needed that help. Another aspect was to find champions and develop hero stories, where someone not that advanced around data was able to successfully build a dashboard or similar to inspire others and lower the perceived bar to doing the data work themselves. Sid and team knew that if the domains couldn't successfully own their data, the work would fall back on his team. So they had a bit of an extra incentive to make it work 😅 So they made sure to only take on the amount of work they could handle instead of onboarding every domain they could/that was interested right at the start. Just make sure you communicate effectively/transparently. Scott note: It's absolutely normal and reasonable to push domains out in time if you don't have the capacity to work with them, whether that is resources or platform capabilities around their use cases. At Airtel, Sid and team didn't succeed on their first attempt at data mesh - in fact they "failed miserably". They weren't fully prepared and didn't have a lot of the issues figured out. Luckily their company didn't throw in the towel but the failure helped them understand what really mattered. It was about making data mesh possible, making it a part of the strategy so teams could feel like it was okay to lean in, and building out the resourcing - platform, compute, and people sides - to make this an actual possibility. Sid didn't see his early data mesh role as trying to convince every domain, win over everyone upfront. It was about getting out the first few use cases and working with a few domains. It was more about finding collaborators that could help drive to a scalable platform. It was about finding the friction points as domains look to own their data and then addressing those friction points. Really focus on what you need to accomplish and don't try to take on the world :) It's going to be important - and somewhat challenging - to keep all your stakeholders engaged at the start of your journey according to Sid, especially those not participating early. It's important to keep engineering and product leaders engaged but also the people doing the actual work in the domains and the data team. And of course, you need to keep the business stakeholders engaged as they are the ones who will do a lot of the work but also get the biggest benefit. But it's also crucial to understand each group will require different approaches to keep them engaged. Focus on that incremental progress and find leverage points instead of trying to tell them about everything. For Airtel, the product and engineering teams were excited about faster product delivery and quicker delivery of insights. For Sid, it was important to communicate to leadership that the data mesh journey would take a few years and to get their buy-in that this was a long-term approach, not a quick quarterly fix. But most teams really only care about what is going to be coming, what value will be delivered, in the next quarter or two. Again, focus on delivering updates about what the audience cares about and understand you will have multiple personas to cater to. At Airtel, budgeting, especially around data, has been done centrally and not in each domain according to Sid. So, for the first year of their data mesh journey, the data team isn't even showing people exact budgets/costs for their domains around their data work but they intend to build a perspective on overall expected costs. This is a big change in thought for the domains from when the costs were all on a central shared data team. The data team didn't want to switch from essentially an afterthought to billing the domains too quickly but they do want the domains to consider costs. The return on investment conversation has been somewhat difficult to nail down in many use cases for Sid. What value do you attribute to faster time to market? If something now takes 1 day instead of 4 weeks, there is a very clear business value BUT what is the measurement of that business value? Time savings is nice but ends up being pretty low in the grand scheme at Airtel. There are also new use cases that could not have been done before. But overall, it's hard to point to an exact return figure. Scott note: this is where you can flip this around on the lines of business and ask them to tell you how much value it generates for them to do a use case or to get to market quicker/easier and do smaller scaled, fine-tuned experiments. Sid and team are still figuring out things like the domain onboarding model and many governance aspects in the platform and out. Should they have an embedded expert in things like data modeling go into domains to help them get up to speed? Now that sharing data is far easier, how do you make sure data is well governed across a number of dimensions like quality, security, privacy, etc.? It's not all figured out and that's okay :) Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of October 8, 2023 21:34

۲ years پیش21:34

21:34

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#257 Panel: Doing Data Mesh Data Governance Well - Led by Andrew Sharp w/ Nicola Askham, Kinda El Maarry, PhD, and Jay Como 1:09:24

۲ years پیش1:09:24

1:09:24

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Andrew's LinkedIn: https://www.linkedin.com/in/andrewsharp27/ Nicola's LinkedIn: https://www.linkedin.com/in/nicolaaskham/ Kinda's LinkedIn: https://www.linkedin.com/in/kindamaarry/ Jay's LinkedIn: https://www.linkedin.com/in/jaycomoiii/ In this episode, guest host Andrew Sharp, Principal Consultant - Data Governance & Data Protection at The Oakland Group (guest of episode #172) facilitated a discussion with Kinda El Maarry, PhD, Director of Data Governance at Prima (guest of episode #246), Nicola Askham, AKA The Data Governance Coach, an independent data governance consultant (guest of episode #129), and Jay Como, Strategic Advisor at Curate Insights (guest of episode #92). As per usual, all guests were only reflecting their own views. The topic for this panel was how do we do data governance well including how do we get started around data governance in data mesh. There's a lot to learn about how to improve your governance but there are no blueprints unfortunately. You have to do the work specific to your organization. Scott note: I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views individually. Scott's Top Takeaways: The single biggest misconception around data mesh is that because we call it doing decentralized data, it means decentralize everything. It's even worse when people think it means decentralize everything on day 1. Find your balances as to how far to decentralize different aspects but a central governance function/team will be crucial to doing data mesh right. They need to focus on leverage points and enablement, especially via guardrails, policies, and standards. Each of those three we want to - at least eventually - automate as much as we can. That's the 'computational' part Zhamak mentions. If all we do is empower the domains, there will be chaos. We've seen this with improperly done decentralization in tech repeatedly. It's 'federated' governance and not merely decentralized. We need guiding hands provided by governance to enable the domains to do the right things. In mesh, you have to balance individuality/freedom and consistency. The information encapsulated and how it is encapsulated must be free but why constantly reinvent the wheel where you don't need to? Try to make conformity easy, simple, and the best decision around those places where differentiation isn't a value add. Data mesh creates a great path to _start_ to federate/decentralize data governance. If you try to rush into decentralizing everything upfront, it will lead to chaos. Domains need training wheels around a lot of governance aspects, don't throw them in the deep end. Data ownership is such a crucial aspect to get right in data mesh, especially for governance. That can feel obvious but it's also one of the most common places mesh implementations are suffering or falling down. Be clear what ownership means and help owners understand and take up ownership. Trying to standardize everything instead of create standards that make things easy but are flexible is something that has held back data governance for so long. Data mesh gives us the opportunity to try so many more things out. Take advantage of that and learn along the way. Communication and culture. In data, we are used to thinking that the data communicates itself: look at this perfectly self-describing data model. It's just not the case. And you can't transform your culture and the way your organization communicates overnight. Invest a lot of time and effort into communication and culture if you want to have sustainable value delivery with data mesh. Business people don't want 'data governance', they want to achieve business outcomes. Share with them how the work will help them achieve their goals instead of the specifics of data governance. Speak to outcomes and you will get them excited to do data governance. Focus on the why - the how is exciting and interesting to data folks but most business people don't care! Other Important Takeaways (many touch on similar points from different aspects): Your mesh governance setup will look different to any other organization's at the very low level and that's to be expected. Your organization is unique - fortunately or unfortunately 😅. It's incredibly easy to get yourself in trouble by trying to be too prescriptive around mesh data governance and following it 'as written'. Zhamak didn't try to write everything as if it was the mesh bible. We're still figuring out how this works. Test what works and be prepared to change what doesn't. Mesh data governance best practices are barely emerging at most. You will need to develop your own theories and practices in some regards. And be prepared for lots of evolution as we as a collective industry figure out how to do this better before we figure out how to do it well. Almost every single organization struggles trying to figure out the mix of responsibilities and roles around data owner, data product owner, data product manager, etc. You will almost certainly not get it perfect to start - test and evolve. Scott note: please, for the love of all that is holy, go talk to say 5 mesh governance leaders and see where they settled on roles and responsibilities so you aren't all repeating the same mistakes. Every domain will have its own quirks. Don't be afraid of changing things, especially roles/responsibilities depending on domain maturity/capabilities. Yes, everything being uniform would be great but it's not realistic. Federate/decentralize at your own pace, don't get ahead of yourself. Some aspects of governance will remain more centralized than you expect as you test what works best for you all along the journey. Again, your equilibriums will be different from other orgs and will change along your journey. Focus on maturity models to understand how to appropriately decentralize aspects of your governance. Not every domain will be as mature either. Again, data mesh is a journey, you don't flick a switch and everything changes. We need mechanisms to at least enable - and potentially enforce - interoperability. If all we have is good data products that don't interoperate, we've only got high quality data silos. Consider how strongly you encourage interoperability and when. Part of a successful data mesh implementation will be about ensuring the data products created are valuable. Highly related, it is important to ask if you are creating the right data products. It's hard to do that ahead of time with governance but governance is key to at least measuring if you are succeeding there. Try to define what good looks like for your domains. It's easier to paint from a reference than from just your mind. It will also encourage more standardization across those things where again freedom and variation don't add value. Nicola said, "You don't go and assign data owners, you identify and engage them." Instead of treating data ownership like a hot potato, you need to figure out who should be the owner and help them step up. That can be different in each domain. Yes, a blueprint that works for every domain would be great but it's not realistic. Many people are excited about what data mesh can bring to their team when they take data ownership. Engage and flame those fires. Help them realize that data ownership can be a great boon, not just more work. Easier said than done of course 😅. On the flip side of course, there are many who will be highly resistant to taking on data ownership. Look to high-level exec support to put the right incentives in place so people will do the right thing. It's also okay to leave some people behind rather than spend all your time trying to convince a brick wall. Data governance in data mesh is harder than traditional data governance. If you are really struggling to do many of the aspects of traditional data governance, really consider if you want to take on doing data governance on hard mode by going the data mesh route. Data mesh isn't for every organization. Conversely, data mesh can be something that forces you to really mature quickly when it comes to data governance. It can be the fire that forges much better practices but that's a risk and it will mean a lot of work if you don't have a great foundation. Scott note: I still go with the previous point that don't try to rush into this if you don’t have a strong foundation but they are interesting points to weigh against each other. An absolutely crucial aspect of good governance is making the implicit explicit. Even make that explicit to your teams; tell them repeatedly to make the implicit explicit. Because that's where you will find friction and misconceptions that will undermine your work. If you are trying to layer in a ton of new roles, it usually means asking for a lot more budget. Think if that will be a hindrance or can you get away with less than perfect ownership to start. It's a risk but if the initial consumers are understanding too that we don't have the budget to nail it perfectly at the start, it can mean you can move ahead with far fewer road blocks. Historically, data work has been often bifurcated with the technical/technology data work in one group's hands and the logic/conceptual/non-technical work in another. Learning to blend the technical and non-technical will probably be harder than you expect. It's a lot about unlearning bad habits and that isn't easy. As often mentioned on this podcast, data literacy in mesh needs to focus on enabling people to actually do the work around data rather than deal specifically with data tools. We need to lower the bar to working with data at the tooling/services/platform and the understanding levels. You may see engineering people get more engaged in governance than they have historically or more than you’d expect. Obviously, invite them into the party to help out. They can add some really great additional expertise and perspective. There is a great misconception around mesh data governance and maturity. Everyone seems to think other organizations are doing governance really well while they are struggling. That's just not true, everyone is struggling. That doesn't mean take it easy but also don't get discouraged. Maintaining mesh momentum is a monumentally material mission. All jokes aside, it is really easy to lose momentum, keep to looking to deliver incremental value and take measured risks - don't swing for the fences at the start. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#256 How to Drive Towards a Data-Driven Culture with Data Mesh - Interview w/ Amy Edwards 1:21:28

۲ years پیش1:21:28

1:21:28

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Amy's LinkedIn: https://www.linkedin.com/in/amy-tang-edwards/ In this episode, Scott interviewed Amy Edwards, Formerly Director of Analytics and Product at Vista. To be clear, she was only representing her own views on the episode. Some key takeaways/thoughts from Amy's point of view: A potential Northstar metric for how data driven you are: how often are people getting, interpreting, and actioning on the data themselves versus waiting for an analyst to tell them? Hard to measure but it's where you want to head, where more and more people are making the effort to interact with data. If you want a data product/use case to succeed, the absolute most important thing is an engaged consumer stakeholder - someone who really, really wants the data for a use case and how they want it. If someone isn't leaning in, consider not building something for them. Scott note: this sounds controversial but it is reinforced in almost every data mesh conversation I have. Similarly, if you are building a data product, you should make sure you are very aligned with your stakeholder. Don't be a request dumping ground, build iteratively together. To understand your progress towards being data driven, you need to actually measure things to track changes over time, your progress. You almost certainly will not find the perfect aspects to measure at first and what you measure will evolve. But it starts with measuring something. There's nothing wrong with starting with a using a success metric that you know isn't going to be something you focus on when a data product matures. As products go through phases, so too should your measurement. So start with number of users as a key early data product success metric - it's easy and somewhat correlated to value driven. ?Controversial?: The adoption curve to not just data mesh but each incremental data product is steeper than you expect. Try to get more people using each data product more so incremental use cases emerge. That will mean you need to do more hand-holding than you likely expect. ?Controversial?: Because that adoption curve is hard, especially across data products, your data team or data product owners might need to do more connective tissue work with other data products to enable people to more easily consume across domains. Always look to drive more and deeper engagement to create a more data-driven culture. ?Controversial?: When thinking about interconnectivity between data products, do you have the consumers create their own bridges or do you pull some of the data from one product into another? Will that create duplicated data and is that the worst thing? What is needed today for where you are in your journey to a truly data driven culture? Sometimes you will probably build something that is pretty customized to a use case and you can't adjust your data product easily to fit a new use case, especially a use case combining data across multiple products. See if you can push some of your data into another data product to again make things easy for consumers. !Controversial!: It can make sense to focus more on driving adoption than creating every data product to be highly reusable. If there isn't demand for data and/or if consumers aren't actually leveraging data, a highly reusable product will still not see much usage. People using the data is your data-driven Northstar metric. Similarly, it's okay to take on data tech debt around your data products if it's a conscious decision. You understand what the work will be to make it - the data product or how it fits in your broader mesh - more scalable in the future and you made trade-offs now. !Controversial!: duplication of work is better than not having the data available to consumers. It's okay to have the same data in multiple places for now as we crawl or walk. When we're running (or even flying!) we won't have the duplication. Scott note: how does this factor in to MDM? Providing a lot of data analysis capabilities and flexibility to people who aren't confident in their data abilities can overwhelm them. Potentially look to offer them more static views/dashboards early as they get comfortable learning to consume analytics-driven insights. You should want your data analysts to move beyond the work of pulling reports and answering basic questions but you might have to leverage them as change agents to teach others to do that. Would be great to not have to use them that way but you probably will need to have some group of people on the ground helping people up their data fluency. A YouTube training course will not be enough :) When you think of managing data as a product, that means data is part of the product too. So managing the data should be part of the general product management/software development responsibilities of the domain. But that's an advanced state - you almost certainly won't start there early in your mesh journey. When your data product team is too separated from the rest of the domain or the general product aspects of your company, there is a risk of that team becoming siloed and not really understanding the domain to the level they should. When you start to build out your foundational data products, your data products might look more like data than products but you want to move more and more towards that product mindset and output if you can. ?Controversial?: When building out a data product team, the first hire should be someone with data engineering capabilities. Scott note: right now, strong agree. If we can get tooling/platforms to a better place, maybe that changes. Because of the lack of a really strong UX, data products can't really drive demand in-and-of-themselves. Therefore, to see good usage that drives strong value, you need that really engaged stakeholder. It's incredibly difficult - impossible? - to create a data product so amazing, people flock to it simply because they have to use it like some consumer products. ?Controversial?: Crawl -> Walk -> Run. Build a solid foundation for your mesh in the long run. Scott note: this would be compared to only building to specific use cases. Should you build out a strong foundation of data products to support many use cases or only those use cases you are targeting? Amy started the conversation around measurement in data. To become data driven, you have to actually measure things. It doesn't have to be the perfect measure but you need to track and understand your progress. In fact, it almost certainly won't be a perfect measurement and your measurements will evolve over time too as you want to focus on different aspects of a data product or mesh journey. In her own journey to get back in shape, she focused early on exercise quantity over quality because it was about forming the habit. After she saw success there, she started to focus more on performance and speed of her runs. It's also important in Amy's mind to understand what are you actually trying to achieve in a certain phase of any implementation. For data products, there is the development phase, once it goes into production, maturity stage 1, etc. What looks like success or is important will change and evolve in each phase and as you progress in your mesh journey. That evolution is not driven only by that your measurement framework gets better, it's that what you want to even focus on changes. So, starting with number of users of a dashboard was a good initial metric = it was early phase, relatively easy to track, etc. So as a data product matures, what you track should change. For Amy, even in a more data literate environment, it can be a pretty hard adoption/learning curve for consumers with each individual data product. So, you want to try to get more users early in a data product's life to encourage developing new use cases. But people don't just generally start using a new data product so someone needs to do more handholding and work than just creating the data product. Part of that might be the data team or data product owners doing more of the interconnection work between data products so users can understand and be confident - otherwise, they will probably stick to what they already know. Focus on increasing and deepening engagement, that's what drives your culture towards being data driven. Amy talked about how there is a bit of a push-pull when thinking about those better interconnections between data products. A product owner or manager should be thinking about how can their data product better connect to the greater data landscape inside the company. And then should that be pushing data from their data product into someone else's or pulling in data from another data product to make their data product more valuable. Find those paths of least resistance for consumers to get useful information to drive increased adoption. For Amy, there is an interesting challenge when it comes to balancing high reusability versus customization for data products. If you do not create the demand for data with the consumers, then even the most reusable data product won't see a lot of use. So they focused more on customizing data products to use cases than is probably general practice in data mesh. But that meant the users had an easier time adopting and leveraging the data products. You can't create too many highly customized solutions in the long run, you just can't support that many data products. But data products that are perfect in theory with very low consumption is probably far worse than a situation where you will need to refactor in the future but you have people eagerly consuming data. It was still manageable to use this approach quite far into their mesh journey. At Vista, there was some duplication of work in Amy's view, where multiple data products might have some of the same data. But, she viewed that as a conscious choice and that it's better to have the data more easy to consume than to really put the hammer down on not have duplication of data. Again, they really focused on driving usage with an eye on improving things later. Scott note: this is an interesting note and balance to strike. I honestly don't know how I feel. If you're aware and it's communicated and people understand, as long as it's not causing issues, I guess it's okay? One key metric for Amy along the early drive to becoming data-driven was how often people were self-serving their analytics. Essentially, how often were they going and looking at dashboards or outputs of some kind to make their decisions rather than leveraging a data analyst to answer questions for them or worse, waiting for a data analyst to come tell them the insights. It can be hard to measure specifically but you can get a decent idea of momentum around that kind of metric. A really interesting insight Amy had was that when they first started trying to work with the broader organization, many of the people were not yet confident with their analytical capabilities and providing them a ton of flexibility around their analytics dashboards overwhelmed them. So, instead, they started by providing more static views to let people learn how to consume data and analytical information before giving them more advanced self-service tooling. And to get the general business users more comfortable, they paired those business folks up with data analysts to walk them through what data existed and how they could query it and leverage it. It let the business users get comfortable in the shallow end, not throwing them in the deep end. Amy talked about how the role of the data analyst might not be the most glorious when you are in a transition phase. They were using data analysts as change agents, helping to train the business people to get better and better with data. It would be great if the data analysts didn't have to do that but there is that necessary transition period where people are improving their data fluency. After getting the general populace to a better data fluency, the analysts could focus more on higher-value, more in-depth analysis. When asked about how a data product team should look and function as part of the greater organization, Amy talked about how an end-state, highly data-driven organization might look versus how you will probably start out. If we want to manage data as a product, we have to think of data management and generation at the domain level as just part of the product function/software development. That may still be a separate data product team within the domain but it's not a completely separate unit/function. But when you are starting your data mesh journey, you almost certainly will not be at that capability level from a platform standpoint or general developers being that data literate - there will probably be a need for a separate data product team that is led more from the data side than the product side. But with that setup, beware of silos even within the same domain, try to make sure the data product teams are integrated into the work of the domain. Amy talked about the number one most important aspect to a successful data product or use case: an engaged consumer stakeholder that is strongly aligned. It's very hard to manufacture demand by creating the most attractive data product in the world. You want a stakeholder who is helping drive the development decisions and is chomping at the bit to get the data. Don't let the data product teams develop in isolation either. Make sure they are iterating with the stakeholder. When building out your data mesh, Amy recommends focusing a lot on the foundation first. You will be building out some unsexy data products, really the core of what you'll build on later, as the early part of your journey. But that means you can focus more and more on building to value later. Of course you need the buy-in, funding, and momentum to go this route. Crawl, then walk, then run. Quick tidbit: When building out a data product team, Amy recommends your first hire being a data engineer. They are the ones who "build the rough plumbing", the pipes in the walls and without that, the water doesn't move anywhere throughout the house, in or out. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of October 1, 2023 33:10

۲ years پیش33:10

33:10

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#255 Zhamak's Corner 28 - Generative AI and Data Mesh: The Start of a Long Road 22:31

۲ years پیش22:31

22:31

Takeaways: This is just scratching the surface of generative AI and data mesh - we will have much deeper discussions in future episodes. Zhamak believes generative AI has a ton of positive real world potential, especially in data mesh. Scott is more skeptical. But if things like GenAI are only able to be leveraged by a few large companies trying to collect as much information - especially sensitive information - as possible, there are some big potential societal issues that might come from that. We need to democratize the ability to leverage these types of tools. ChatGPT set off a frenzy. It can be easy to want to move incredibly fast towards implementing generative AI. But companies don't have the vast amount of data where they can throw moderate - or worse - quality data and get something useful out. Garbage in, garbage out is a real concern. Because they have less data than essentially the sum of the internet like OpenAI used for ChatGPT, companies need to focus on providing quality data into an LLM (large language model) in order for it to actually provide good results. Again, otherwise it is garbage in, garbage out. Sponsored by NextData , Zhamak's company that is helping ease data product creation. For more great content from Zhamak, check out her book on data mesh , a book she collaborated on , her LinkedIn , and her Twitter . Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Data Mesh Radio episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#254 Easing Into a Data Mesh Journey - Ocean Spray's Pre-Data Mesh Preparations - Interview w/ Paul Cavacas 1:06:46

۲ years پیش1:06:46

1:06:46

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Paul's LinkedIn: https://www.linkedin.com/in/paul-cavacas-32a36158/ In this episode, Scott interviewed Paul Cavacas, Senior Manager of Data and Analytics at Ocean Spray. Quick note before jumping in: Ocean Spray is just at the beginning of their journey - in their pre-implementation phase - and there hasn't been a lot of resistance yet internally. That might make a few people jealous 😅 but there's a lot of interesting things Paul is doing to ensure that they are ready to decentralize what makes sense to decentralize at the right time. There is a lot to be gained from not rushing in. Also, apologies that Scott's audio is a bit weird, he had yet to build his makeshift sound studio in the Netherlands. Some key takeaways/thoughts from Paul's point of view: As many have stated, asking the data team - especially one person - to become an expert on many different areas of the business just to complete data work for a project just won't scale. At best it creates incredibly concentrated tribal knowledge. Use this point to drive buy-in for decentralizing data ownership. Having someone who really knows your internal IT application landscape well can really help in choosing which initial teams to start with for a data mesh implementation. That person already has good relationships and a deep understanding of your operational plane so you can pick good problem areas and partners. Similarly, build your early buy-in momentum with people that are more likely to be excited to participate in a data mesh implementation. You don't need to convince the most difficult teams to participate at the start. Central ownership isn't necessarily bad until things stop scaling. Having that central ownership means less flexibility and agility to react quickly to market changes or opportunities but also less cognitive load on teams. It's a trade-off. Many of your domains really won't understand data ownership. Find ways to slowly transition them into understanding what ownership entails e.g. starting with documentation and SLOs. What data are they sharing and why does it matter? This isn't going to happen overnight. If you aren't building overly complex data products, look to find people within the domain that are somewhat technically savvy - especially if they want to advance their careers - and start to prepare them for data ownership. Those might be your data product developers or data product managers. Scott note: Brian McMillan talked about a plan to do that in episode #26. ?Controversial?: Even if you aren't looking to move fast with your data mesh implementation, look to put at least a proto platform in place so people can at least test out ideas if they want to move more quickly. Setting up your testing and data contract framework will probably be a big challenge. Talk to people about what they need and iterate towards it. But it will be difficult to figure out at the start and you'll start with something sub-optimal. Be prepared and go forward anyway :) Focus more on finding the "right" partners - those willing to really partner and learn and try - rather than the "right" domain that has the most valuable data or use case. This comes up in many episodes. If you are looking for the right partners, there are good signals to watch for such as who presents their internal results with more advanced data graphics because they are likely a domain that really cares about the data and want to dig in more. Consider how much need there is to decentralize certain aspects and if there is enough capacity and/or work for a domain. Those might sound different but if there isn't a ton of data work, that's a lot of effort for a domain to learn to own data for a small amount of data work. Scott note: Yushin Son at JPMC mentioned they are creating a central team for small domains that can't justify a full data product team in the panel episode #233. When creating a satisfaction scoring system for data products, consider what is the goal of the satisfaction score and how long that feedback is useful. So when a data product is in beta, harsh but useful scores are valuable but a 2 on data quality before v1.0 might not be relevant once it is actually a released product. It's okay to wipe past scores clean. ?Controversial?: When looking at your first data products, you don't want to lift and shift existing data assets to your platform. But, you can go after existing use cases as your early products if you are providing the same thing in a better/easier format. Don't let perfect get in the way of good/progress. It's okay to put in solutions that won't work in the long run as long as you acknowledge they will need to be replaced and give yourself the room to replace them. It's taking on technical debt through a conscious choice and not locking yourself in. ?Controversial?: When you are getting started, make sure to work with a few different areas to prevent yourself from delivering a solution, even at the mesh level, that is overly targeted to one domain's needs. Paul started off with a bit about why they are headed down the data mesh path. For a large internal project, Paul had to become an expert on so many aspects of the company and that's just not scalable in the long-term or if he's on vacation. So, he's started to decentralize the data capabilities - slowly - as teams understand what data they will likely need to own in the long-run. And he's not playing data ownership 'hot potato', he's making sure they are prepared in the right ways. At Ocean Spray, Paul shared that until recently everything tech - including data - was very centrally owned. In some areas, the IT team knew possibly more about the business processes than even the business people in those domains. So the company is going through all of their software and applications to decide how that should look in the future. Data mesh plays well into that rethink because central ownership scales until it doesn't and limits flexibility. As there isn't a rushed timeline, Paul has been able to put together a complete idea of what the data mesh roadmap will look like. But he also understands that it could look completely different as he learns more and starts trying to actually implement different aspects. There are some existing data sets/assets out there that could pretty easily become data products in the right environment so that is where they are targeting first. They are working with the teams to transfer some part of the ownership, especially around documentation of use cases and SLOs. The central team is pairing to take existing data assets, decompose those into their data products and help people get on the path to real ownership. Paul recommends what Brian McMillan talked about in depth in episode #26: finding people within domains that are at least somewhat tech savvy and want to advance their careers. Work with them to get them more and more up to speed. Ownership is not something that gets transferred in a day - treat it with more respect than that. So that's finding receptive people inside the receptive domains. Yes, it won't always be easy but why make the buy-in complicated at the start if you don't need to? Right now, Paul is building out some of the technical underpinnings of the platform they plan to build. If there are teams that want to move more quickly, they can start to test things out now. As long as those teams understand things aren't fully automated and they may have to change things about what they build now when the company starts to fully move to data products. One big piece he is anticipating is the need for testing and data contract mechanisms. But exactly how to do that is still a challenge and will be learned along the way. He's anticipating a workable but not perfect solution to start. Build to useful and then improve. Paul circled back on the idea of finding the right partners over the right use cases/domains. Having engaged and excited partners, who know you can up their own data capabilities and drive value for them too, will make your early journey far easier than going for the most "valuable" data. You are also likely to get better feedback because they are bought in to collaborating with you! To find those partners, potentially look at how teams present their results internally. If they are presenting with lots of advanced figures and almost a flair around data, that is great sign. How much data ownership/work gets decentralized and when is a key remaining question for Paul. He's aware that he'll have to test what works and iterate as he learns but there are plenty of domains that are too small to justify them learning a ton about how to own data when there just isn't that much data/data work to deal with. There will be a shared ownership model between the central team and the domains. Scott note: this works up to a certain scale and in certain types of organizations. Shared ownership in a very large organization rarely works that well for all that long - too much political infighting and challenges but it's an interesting pattern for smaller orgs that seems to be working well. Paul's plan for assessing the quality of data products is to create a rubric scoring system - asking people to rate them across multiple dimensions like usability, data quality, SLA compliance, etc. And that the scores or how they are measured may change across time. At the start of a data product's life, when it's still in beta, those scores can be invaluable to iterate towards value but then consider throwing the historical scores out once it hits that v1.0. That's because there is a useful aspect of feedback depending on what you are trying to achieve and bad historical scores could hinder the success of a data product when it is now very high quality and valuable. For Ocean Spray, their first few data products are going to be source aligned, combining a lot of important sales information. That way, those people who want raw data can still get at it but then they can build out more and more views/data products for the users on top of those. That way, there is still the scalable/productized underlying production of the raw data and then more fit-for-purpose outputs for the different users. Paul is not letting perfect get in the way of progress. Data contracts have to get to a place where we aren't locked onto schemas as something that can never change. But no one has really come out with a better solution yet, so that's what he's doing to start. It's better than nothing so go with it while you figure out better ways. Paul finished with a bit of advice around working with a few domains at the start of your journey. That way, you can take the learnings and understand the needs from multiple domains to abstract to a better solution for the organization rather than one overly tied to one domain's needs. Scott note: people seem pretty 50/50 split on working with one domain or 2-3 at the start of your journey. It's an interesting question. Other unique factors of Ocean Spray: The corporate structure is a co-op of growers so there isn't some massive pressure to grow at all costs. Domains have been able to get access to other domains' data relatively easily for a long, long time. It hasn't been cleaned and prepared for them but there is an existing culture of sharing. They are moving more and more to 3rd party applications rather than custom-built, which means data isn’t necessarily in an easy to consume format by default. (maybe not all that unique?) Because many domains are quite small, the central team will likely still own most if not all of the data work for those domains. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of September 24, 2023 17:23

۲ years پیش17:23

17:23

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#253 Data Mesh Implementation Success Metrics - Data Quality - Mesh Musings 53 40:58

۲ years پیش40:58

40:58

Key takeaways: As mentioned the last two times, at the start, it's more important to start measuring something than it is to measure the right things. Do NOT let analysis paralysis hold you back. Start measuring early to figure out what actually matters and that will also change over time. Similarly, your success metric measurement framework will probably suck to start. Oh well, get to measuring. Use fitness functions. Episode #95 with Dave Colls covers a lot on this. Data mesh really is a journey and your success measurement will be too. You will need to find small and simple ways to measure. Don't get bogged down. Your measurements will be rough and kinda depressing with the amount of challenges to tackle at the start. Just understand this is about how well you are doing, not how complete you are - there is always more to do! Reflect back on how far you've come, we often forget to do that! When it comes to data quality measurement at the implementation level, you need to think about what are you trying to accomplish. Many people go down the wrong path of trying to measure quality in a vacuum. It's about what are the expectations and why do we care about quality - to improve our decision making around data and to improve trust so more people feel they can rely on data. It's that simple. Now, measuring how well you are achieving those gets a bit harder… :D So, what to measure or consider how to measure regarding data quality at the implementation level: how often are people in compliance with their quality SLAs, whatever those SLAs may be? How quickly are you detecting and resolving/recovering from incidents? How many incidents are you having and what is their severity? Who is actually discovering the issues - are there automated detections and is it the producer or consumers discovering them? How do you actually think about trust and the impact of trust on the success of your implementation? How do you measure and increase trust levels? How does that impact value creation? And finally, what is the quality of your metadata? Please Rate and Review us on your podcast app of choice! Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#252 Designing and Building a Better Data Governance Approach - Interview w/ Lauren Maffeo 59:37

۲ years پیش59:37

59:37

Use code DATAGOV23 for 35% off ebook copies of Designing Data Governance from the Ground Up here: https://pragprog.com/titles/lmmlops/designing-data-governance-from-the-ground-up/ Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Lauren's LinkedIn: https://www.linkedin.com/in/laurenmaffeo/ Designing Data Governance from the Ground Up (Lauren's book): https://pragprog.com/titles/lmmlops/designing-data-governance-from-the-ground-up/ In this episode, Scott interviewed Lauren Maffeo, author of the book Designing Data Governance from the Ground Up and adjunct Lecturer at George Washington University. To be clear, she was only representing her own views on the episode. Some key takeaways/thoughts from Lauren's point of view: In governance, a very easy way to go down a bad path is to not automate your standards. Making governance an easy aspect of data work will go a long, long way. ?Controversial?: As an industry in general, data governance maturity is still at the infancy phase. And the pace of maturation is far lower than other aspects of software like security. The majority of organizations are not mature enough with data governance to get a lot of value from things like ML or NLP. Data governance best practices are hard to come by. There isn't really even a large community specific to data governance for people to easily exchange ideas. If the number one cause of cybersecurity breaches is employees, is the number one cause of bad decisions made on data because of A) bad data or B) employees not being data fluent enough to make the right decisions? Does the second contribute to the first? "You really have to embed data literacy into very strategic ways of communicating with the organization and educating them that way. Without that approach, I think very little progress can actually be made." ?Controversial?: It's not that hard to teach people to be relatively data fluent. Not to a data engineer level but investing a relatively small amount in your people can get them to a level they can at least typically spot low quality data. ?Controversial?: You can't just "do data governance later" and have a good outcome. Scott note: this absolutely shouldn't be controversial but still seems to be a common pattern. If we actually want to treat data as a product, that includes practices around data too. A product manager couldn't decide to put out their first roadmap three months after the product is shipped. It's hard to measure the ROI on data governance work but it's pretty clear that it's necessary when your data work is not driving results. Despite governance being crucial, you can still roll governance out in a measured manner. You can create ways for people to get used to the idea before rolling it out and it isn't as if day 1 'everything changes!', it can be a gradual process that adds value, not (many) complications. Embracing a fail fast culture in data will probably be hard but it's absolutely something everyone should do. Make the 'failures' small and contained - limit the blast radius - but still failing fast "the essence of innovation in tech." Look to leverage sandbox environments to try out new approaches to governance and get data stewards used to anything new you will be asking of them. A new policy or tool doesn't have to be a sudden light switch flip. "We also can't afford for leaders of any department to not know what quality data looks like for their teams, because their success, the success of their teams depends on having quality data that their customers trust." ?Controversial?: Data mesh is a risky endeavor, do it in a small-scale way and keep it very cost contained as you test if it will work in your organization. For driving data literacy/fluency, gamification really can work. Look to make learning about data a fun and gamified experience. Lauren started with a somewhat common refrain for this podcast: the pace of data practice maturation - around governance and data practices as a whole - is just not keeping pace with innovation in other aspects of software. Even the pace of conversation is not maturing as fast. Cybersecurity is maturing very quickly for example but we're just not seeing that in data. So companies are just not ready to really derive a lot of value from things like machine learning (ML) or natural language processing (NLP). One of the big issues around industry maturity and data governance for Lauren is that there isn't even a large community around the topic. So there isn't a larger cohesive conversation around data governance best practices. Scott note: it's really hard to have a broader conversation too because approaches and practices vary widely and data governance has about 15 varied subtopics that each deserve their own focus rather than being lumped under a huge umbrella of 'other' that governance has become. Most cybersecurity breaches, Lauren pointed out, are caused by internal employees making a mistake. So how do we think about that relative to data? Is that people creating low quality data and/or is it people not being data fluent enough to actually make good decisions based on data? In cybersecurity, there is a big emphasis on training people to see what an attack looks like - should we take the same approach in data regarding bad quality data? "You really have to embed data literacy into very strategic ways of communicating with the organization and educating them that way. Without that approach, I think very little progress can actually be made." Lauren talked about how few companies are really going broad with their data literacy programs, training up a large number of their employees. There is a lot of talk about that as part of data governance programs but few are walking the walk. And she believes it's not that hard to get people to a relatively data fluent level - understanding SQL, being able to more easily spot low quality data, etc. "We'll do the data governance later," is something Lauren has seen and heard in conversations. Governance is seen as something that can be layered on like a coat of paint at the end of a car being manufactured. But because good governance is intrinsic to data quality and matching to the actual business use case, trying to do it later rarely leads to good results. When asked about selling the return on investment of data governance work, Lauren admitted that it's often quite nebulous but data governance is so key that people know they need it despite not being super clear on the specific value of the work. And you can roll out your data governance tech, policies, and processes at a reasonable pace, creating some definitions and a sandbox to show people how it will work. She is really big on the idea of a sandbox to get people used to new governance practices and tech. It isn't as though everything changes suddenly, it's that you're working towards better data practices that will drive value for the organization. Fail fast is "the essence of innovation in tech" so we need to embrace it far more - but still safely and sanely - in data. "We also can't afford for leaders of any department to not know what quality data looks like for their teams, because their success, the success of their teams depends on having quality data that their customers trust," Lauren said. So we all need to be in this together and have domains really owning and understanding their data. That can't be on a central data team. Gamification is one thing Lauren is seeing work for improving data literacy/fluency. It is a great pathway to creating a data-driven culture. Make it fun and give out rewards :) Lauren wrapped on a simple message. Automate your standards. It is easy to have your tech and standards/processes quickly lose alignment if you aren't making things easy for people via automation. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of September 17, 2023 14:05

۲ years پیش14:05

14:05

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Rerelease of #130 Making the Data Quantum Leap - Starting from the Data Quantum at PayPal - Interview w/ Jean-Georges Perrin (JGP) 1:23:55

۲ years پیش1:23:55

1:23:55

Due to health-related issues, we are on a temporary hiatus for new episodes. Please enjoy this rerelease of episode #130 with my partner in our weekly data mesh roundtables Jean-Georges Perrin. There are a lot of interesting things to take away from this. A biggie is to have an early thesis about what to drive towards - what will drive value early? Doing data mesh doesn't simply create value. And you need to build momentum. There's a lot here to learn about how to apply good software engineering practices to data with data mesh. Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Data Mesh at PayPal blog post: https://medium.com/paypal-tech/the-next-generation-of-data-platforms-is-the-data-mesh-b7df4b825522 JGP's All Things Open talk (free virtual registration): https://2022.allthingsopen.org/sessions/building-a-data-mesh-with-open-source-technologies/ JGP's LinkedIn: https://www.linkedin.com/in/jgperrin/ JGP's Twitter: @jgperrin / https://twitter.com/jgperrin JGP's YouTube: https://www.youtube.com/c/JeanGeorgesPerrin JGP's Website: https://jgp.ai/ In this episode, Scott interviewed Jean-Georges Perrin AKA JGP, Intelligence Platform Lead at PayPal. JGP is probably the first guest to lean into using "data quantum" instead of "data product". JGP did want to emphasize that as of now, he was only discussing the implementation for his team the GCSC IA (Global Credit Risk, Seller Risk, Collections Intelligence Automation) within PayPal. Some key takeaways/thoughts from JGP's point of view: Data mesh as it's been laid out by Zhamak obviously leaves a lot of room for innovation. For some, that's great. For others, they want the blueprint. And it's okay to wait for the blueprint. But JGP and team are excited to innovate! PayPal's 3 main initial target outcomes from data mesh: A) faster and easier data discovery, B) easier to use the data in a governed way, and C) increase data consumer trust in data. PayPal's initial data consumers are data scientists so their platform and data quanta are built to serve that audience first. Really consider what you want to prove out in your MVP. Is that minimum viable A) data quantum, B) data platform, C) data mesh, or D) something else? Only doing a data quantum probably sets you up for trouble and a platform only won't be tested until it has data quanta on it. Data contracts are crucial to making trustability actually measurable and agreed upon. Otherwise, it's far too easy to have miscommunication between data producers and consumers, which leads to lack/loss of trust. Producers, don't set your data contract terms too strictly when first launching a data quantum. There's no need to over-engineer - despite how interesting that can sometimes be... For too long, we have tried to keep software engineering and data engineering overly separate. They are both just engineering with slightly different focus and data mesh really leans in to that. We've also tried to keep operational and analytical far too separate. We should look to build out tooling where data can live that serves both operational and analytical workload needs. But we aren't there yet. Analytical APIs, at least as far as we've seen them to date, are just not going to do what we need relative to accessing data from data products/quanta according to JGP. Standardizing metadata access APIs across data quanta has made it very simple for data consumers to begin using new data quanta as they are introduced to the mesh. PayPal has observability, discovery, and control APIs. Domain is an overloaded word. It can mean a very large high-level domain like Sales, Finance, or Marketing with hundreds or thousands of people in it or it can mean a smaller, 'two pizza team' level scale. PayPal is doing only one data quantum per domain but that domain is really at the two pizza team scale - they aren't trying to have a single data quantum for Marketing. It's crucial to understand data quanta and the use cases they power both have life-cycles. So really applying product thinking is crucial. Most data engineering teams do work in a waterfall approach and that just doesn't scale well. However, moving to data mesh can mean additional cognitive load as it really requires an Agile mindset to do right and that shift in the ways of working is not trivial. It's good to have smaller delivery requirements so you get faster feedback on what you are creating - a core tenet of Agile. Don't try to deliver everything all at once. Get it in users' hands early to get feedback. JGP started the conversation talking about how in his team, he's really leaning into the idea that software engineering and data engineering are not that different. Zhamak has discussed this too. We should focus on sharing practices so we all create better software and infrastructure. For JGP, data engineering work in most organizations has followed a very waterfall approach. However, his team has been mostly working in an Agile manner. Therefore it wasn't a huge switch to their ways of working - like it is at many organizations - once they started doing data mesh. And luckily, there was already an appetite for changing the way they were tackling data challenges. In the spirit of being agile and capital A Agile as well, PayPal set out on their data mesh journey. They wanted to do an MVP but what was the P? Minimum Viable Data Product/Quantum? Minimum Viable Platform? Both? Minimum Viable Mesh? JGP recommends looking at what you want to deliver as a minimum unit of value. PayPal already had extensive data platform expertise so they were able to focus on delivering data products/quanta (plural of data quantum) but they worked in parallel to build out their initial data quantum and mesh capabilities. As many guests have noted, it's dangerous to only do a minimum viable data product/quanta. PayPal has been building data platforms for a long time. As mentioned by JGP, they were one of the pioneers of the self-service data platform concept. But data mesh offered a path to faster and easier data discovery, to making it easier to use data in a governed way, and to increased trust in data by the data consumers - their first consumers being data scientists. A big benefit of addressing those needs is those data scientists are able to better tell if the data they access is the right data for their use case. One thing JGP emphasized that's significantly helping PayPal move forward is standardizing APIs across data quanta. Those are not data access - or analytical - APIs as JGP thinks those will just never work all that well. Instead, as their audience is data scientists only to start, everything anyone needs other than the actual 1s and 0s of the data is accessible via Python APIs. The metadata, the observability/trust data, etc. Then, the data scientists use notebooks to work with the data. But standard APIs means data consumers only have to learn one interface. This is similar in concept to what many are doing with data marketplaces - one standardized way to interact with the information about the data quanta. PayPal is using the terms data product and data quantum as two separate things. A data product is simply a product powered by data and analytics. Those have been around for quite some time. But PayPal is looking at data quanta like side cars, used specifically to power more and more of their data products going forward. PayPal have invested heavily in making data contracts work well per JGP and earlier PayPal guest Jay Sen. They've been building APIs to make it far easier to consume data contracts as people learn more about a data quantum. And as mentioned before, they can consume observability metrics via API as well. When asked about how are they setting their actual contractual terms, the data producers initially put out some contractual terms and then may adjust those terms as data consumers request. It's important for data producers to not set their data contract obligations too strictly unless there is a user-based need. JGP made the good and often unspoken point: the term domain has lost a lot of its meaning. It can mean a very high-level domain like Marketing, Finance, Sales, or HR. Even in software companies, a domain could be Product. But at PayPal, they are being quite strict about what they mean for domain in data mesh: it is a small scale sub-domain - think two pizza team size - and they enforcing a strict 1:1 relationship of one data quantum per domain; and of course, not cross domain source data quanta too. That way, each small domain can focus on creating a great data quantum instead of worrying too much about how big each data quantum should be. The scope should never get that huge at a two pizza team size. Back to APIs, PayPal is implementing an API-first approach. APIs for the data quantum control plane, observability APIs, and data discovery APIs. It's the preferred way of working for their initial consumers - data scientists. However, as mentioned previously, JGP does not believe analytical APIs - that is APIs designed to do things like filtering and returning many hundreds to thousands or more results - are really feasible. Definitely not now and possibly ever. So APIs are great for getting at the metadata but not the data for analytical use in his view. JGP wrapped up in sharing how our tooling must evolve so we don't have to think about such a hard wall between analytical and operational. There will always be analytical and operational workloads but our systems can evolve to support both. We aren't there yet though. Quick tidbit: If you are just delivering data, the 1s and 0s, you are not delivering the necessary trust. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#249 Zhamak's Corner 27 - Creating Truly Scalable Interconnectivity for Data 21:30

۲ years پیش21:30

21:30

Takeaways: A data product by the data mesh definition cannot by definition be a silo because it is meant to be consumed by the rest of the organization as a greater part of the picture of the organization. Whether that is true in practice remains to be seen :) APIs are the way we've learned in software to interconnect between services, so we need to learn how to leverage the same approaches in data. Within a single organization, we may be able to get by without having a great set of industry-wide interconnectivity standards - the company can fully control their own sharing. But, we know that to truly unlock the value of our organizations, cross organization data sharing - in a scalable and useful way - is necessary. So we need better ways to do that - standards and tooling/technology. It remains to be seen which will emerge first: the standards or the tooling/technology - a chicken and egg issue there - but they must really must be intertwined to work well. Sponsored by NextData , Zhamak's company that is helping ease data product creation. For more great content from Zhamak, check out her book on data mesh , a book she collaborated on , her LinkedIn , and her Twitter . Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Data Mesh Radio episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#248 Doing Data Quality Right by Building Trust - Interview w/ Ale Cabrera 1:10:11

۲ years پیش1:10:11

1:10:11

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Ale's LinkedIn: https://www.linkedin.com/in/alejandracabre/ In this episode, Scott interviewed Ale Cabrera, Senior Data Quality Product Manager at Clearbit. To be clear, she was only representing her own views on the episode. Some key takeaways/thoughts from Ale's point of view: A key part of understanding what data work will be impactful is a simple phrase: "Is my understanding correct?" Putting out there what you took in and making sure you're on the same page will save a ton of time and headaches! Her advice to her past self: In data, far too often, we try to jump to solutioning instead of really taking the time to understand the problem. Start from understanding the problem and assessing it first. It's very easy to make data say something that it's not actually reflecting. Quality isn't just about accuracy or similar metrics, sometimes there are intangible aspects around correctness that people get but usually can't measure. In data work, many people miss two crucial aspects - the voice of the customer and the why. If you build the greatest thing ever but it isn't what the customer wants, it won't be used. Similarly, if you focus on the work and not the target outcome, your results are likely to be subpar. If you want to prove data work return on investment, try to associate it to a key company metric and talk about how improving that metric will drive better business outcomes. When you want to prove out the value of data quality, attach quality issues to direct business challenges or goals. It’s easy if you are a company selling data but you have to understand why bad quality data negatively impacts the company in order to gain influence to improve your data quality. Far too often in data work, people try to exchange data - the 1s and 0s - and forget to exchange information. Get people together and make sure you are in alignment, get people actually talking to make sure you understand and have a good forward path. The most crucial aspect of data quality is trust. If you can't drive trust, then all the 'quality' in the world doesn't mean anything because people won't use it. As a data person, your job is not to do data work. It is to unblock teams/projects/people and add value. Yes, that is through data work but the work isn't the point. The way to think about trust is a combination of "credibility, reliability, and intimacy by self-orientation." Measuring trust is hard but a good way is through interviews with users. Nothing will ever be perfect so consumers need to understand that. A mature and healthy conversation is to ask them to tell you if they see any data quality issues because issues always can happen. If they won't consume from data that might not be perfect, they can’t consume any data. When you are quick to react to data quality issues, your consumers can get more value from the data because they can be more sure sooner whether the shape of the data changing is an issue or a genuine reflection of something changing. In data, typically one-size-fits-all really ends up being that it fits none. You don't want to over customize a data product but still, your consumers are likely going to want something that fits their needs rather than something generic that they have to do the heavy lifting to be able to use. How people consume the same data can have a large impact on trust too. While data people hate Excel, sometimes you have to deliver something they can use in Excel. Otherwise, they will struggle to trust the data enough to rely on it. Is what makes something great the output or the impact? When applying product thinking, something doesn't really matter unless it is used to generate value. A key aspect of product thinking is prioritization. Yes, it would be great if we could build to every ask but focusing on what will most likely deliver value and why will keep you more focused on generating value through your data work. Ale started with a bit about her career and what led her to focus on data quality, including what led her to her current role. On advice to her past self: in one of her past roles, a lead engineer gave her enduring advice: think before you code 😎 Oftentimes, in data, people jump to trying to solution instead of really taking the time to understand the problem and look at the various potential solutions to choose one that will work better in the long run. She also said to respect the data when considering quality. You can get data to tell a story that it really isn't reflecting the reality of what is happening so you have to give it the respect to not shape it to tell "statistical lies". Another important aspect of data work for Ale is focusing on the voice of the customer and the why. What is the user telling you, what are the pain points. You can build the best solution but if it isn't what the customer wants, they probably won't use it. Look at all the amazing data platforms that barely anyone uses. Focusing on the why, why are people looking at this data or trying to address this challenge, will keep you from trying to solution instead of meet the customer where they are and help them solve the business challenge, not just solve a challenge about data. As a data person, the point of your job is not to do data work. It is to unblock teams/projects/people and add value. Yes, that is through data work but the work isn't the point. When trying to prove return on investment for data work, Ale believes it's important to tie it to a key business metric. Something like 'Our data quality improvement from 98% accuracy to 99% accuracy will mean a decrease in churn by 5%, netting the company X amount of revenue.' In her case, she is attaching the data quality work to reducing churn because it is a key metric that is intrinsically linked to data quality in a company selling data. For Ale, an important aspect of establishing yourself as an internal expert on data quality is tying the data work to value - what actually matters for the company. Improving data quality for something that drives 1% of revenue so it grows 25% is not as much of an increase as improving something that is 10% of revenue so it grows 3%. But the business might favor one over the other. What matters most to the business and why? Ale shared a story about why it's so crucial to get people actually in a space exchanging context and information instead of just exchanging the 1s and 0s of data. She was working on a project involving two teams and neither side had ever met each other despite being months into the project. The data quality was terrible, it was seeming like half the data was wrong or missing. So the other side asked about what was happening when they were pulling data from both servers. Her team only knew about one of the servers. So get in the room - virtual or physical - and actually communicate! Trust is the most crucial aspect of data quality work for Ale. If you have the best quality data but people don't trust it enough to use it, then it's still not of value. And trust is often built more through relationships than anything else (see Beth Bauer, episode #218). What is the purpose of the work if not to drive an outcome. If there isn't trust, can you really contribute significantly to an outcome? The way to think about trust is a combination of "credibility, reliability, and intimacy by self-orientation." She uses interviews as a good way to measure trust as time goes along. And that also encourages users to actually say something if they see an issue, which also increases trust after an issue. Listening to people is just a fundamental building block of driving trust. When asked about if you lose credibility when telling data consumers to immediately flag any issues, Ale was clear that in data, mistakes happen. Nothing is ever perfect. So it's crucial that we can have conversations and tell people to speak up if they see quality issues. Recognizing that there will be data issues doesn't mean you are sloppy with data, only that you are realistic. Being aware and reacting quickly to issues also means consumers can be more sure that an anomaly or change in the data is actually something real and react to that more quickly. Recognizing you will make mistakes and working quickly to rectify those mistakes builds trust. For Ale, there is a tendency in data for delivering overkill and/or something the other party has difficulty trusting. Providing a dashboard for someone who wants to see the actual data is just not that helpful. Meet people where they are while you upskill them to be better, even if that where is in Excel or Google Sheets. Ale talked about switching to the product mindset in data. If you are building 'amazing things' but no one is using them, are they really all that great? Is what makes something great the output or the impact? A good way to get to impact is focusing on acceptance criteria - what would make the user happy and what are they actually expecting? In wrapping up, Ale shared a bit about how to become a good product manager and her thoughts on how crucial prioritization is to actually applying data product thinking. How specific should you get with a solution is a very tough question but you can start by asking what is the expected impact versus a more generic solution. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of August 13, 2023 13:11

۲ years پیش13:11

13:11

Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#247 Data Mesh Implementation Success Metrics - The Data Platform - Mesh Musings 52 28:26

۲ years پیش28:26

28:26

Key takeaways: As mentioned last time, at the start, it's more important to start measuring something than it is to measure the right things. Do NOT let analysis paralysis hold you back. Start measuring early to figure out what actually matters and that will also change over time. Similarly, your success metric measurement framework will probably suck to start. Oh well, get to measuring. Use fitness functions. Data mesh really is a journey and so will be how you measure success. You will need to find small and simple ways to measure. Don't get bogged down. Your measurements will be rough and kinda depressing with the amount of challenges to tackle at the start. Just understand this is about how well you are doing, not how complete you are - there is always more to do! Good metrics to consider for the platform success metrics: satisfaction, time to deploy new products, time to update existing ones, ease of use - which kind of blends into all the others -, searchability/discoverability, ease of interconnection, mean-time-to-detect and mean-time-to-recovery from issues, governance, guardrails, and automation/helpful artifacts. Look to measure friction and how you reduce it and if possible - I have no great ideas here - what is the extra value add to the business, something that the platform was the genesis of? Maybe that's suggesting data products? Lastly, reflect back on how far you've come, we often forget to do that! Please Rate and Review us on your podcast app of choice! Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#246 Making Federated Data Governance Approachable and Effective - Interview w/ Kinda El Maarry, PhD 1:16:27

۲ years پیش1:16:27

1:16:27

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . #252 Designing and Building a Better Data Governance Approach - Interview w/ Lauren Maffeo Kinda's LinkedIn: https://www.linkedin.com/in/kindamaarry/ In this episode, Scott interviewed Kinda El Maarry PhD, Director of Data Governance at Prima. To be clear, she was only representing her own views on the episode. Some key takeaways/thoughts from Kinda's point of view: Always be assessing data literacy gaps and work to address them. Gaps don't have to be a bad thing, they are a chance for your people to grow! But ignore them at your own peril :) A key aspect of your job as someone on the central governance team - or anyone managing a governance function - is to reduce friction, not add overhead. It can be hard to do that at the start of a journey but it should be a north star focus. The phrase 'data governance' still has a stigma for many people from what often happens with a central-only approach to governance - processes and rules divorced from the actual business processes and the data which creates bottlenecks and headaches. The federation of governance in data mesh helps address the bottlenecks but it introduces a new layer of complexity which can also cause some fear. ?Controversial?: If you are asking people to take on ownership of data, you must make sure their line of business/part of the org has the actual capabilities to do so. Otherwise, they will greatly struggle to own and are likely to push back. Don't talk about governance in the weeds to people, the toolbox talks, the policies and standards. Focus on what are you trying to achieve with governance and then lean back on the toolbox if you need to explain the how. But many don't care about the how. There is a sliding scale of how much is federated versus centralized based on domain capability and maturity level. You don't just push all responsibilities on to each domain at the start, it's a gradual process finding the right balance. When you find a domain that has skill gaps necessary to overcome, set up a roadmap and have as positive of a conversation as possible to start along that roadmap - it's not what they lack but where they'll grow. !Controversial!: Where possible, get rid of the central data governance committee. Collaboration is more key than trying to keep every stakeholder across the organization informed and involved in most decisions. ?Controversial?: Data governance people should stop thinking about federation as losing control. It's about enabling the decisions to be made by those that can best make them. That way, the governance team gets to focus on the guardrails, the standards, the blueprints - the things that make stuff easier for everyone. No one is saying hand over control of very sensitive decisions without reason. It's about making sure the day-to-day decisions can be made so when there is an important/sensitive decision to be made, that's where the central team can step in to assist. When trying to get teams to take ownership of their data, there isn't a silver bullet - every discussion and domain is different. But start with a clear understanding of what ownership means and start the discussion with what will you do to make them capable and ease the burden of ownership. Just dumping responsibilities in their lap won't go well. Data culture is an often overlooked aspect of data governance. But it provides your scale - getting people to evangelize and share what's working instead of all knowledge dissemination coming from a central team/training is how you can get scale. It's not exactly the data governance team's role to find the right reporting structure and career management aspects of a data mesh implementation but the team can help by creating good spaces for people to share and communicate to create a sense of comradery. Centralized capabilities and coordination of certain aspects are crucial to doing data mesh right. You want to decentralize the decisions and the application of those decisions but centralize the enablement. A core, central data team is pretty important to making that happen. ?Controversial?: Interoperability between data products is important but interoperability often can get in the way of delivering data products. Don't focus as much on it as you probably think. Kinda started off discussing the traditional view of centralized data governance and the stigma around the phrase. Because the legacy way were a team that didn't really understand many of the key business aspects around the data itself, the policies and guardrails were not designed to make creating data assets or projects easy. The decoupling of understanding from decisions led to bottlenecks. And data mesh is specifically designed in part to eliminate those bottlenecks. However, as Kinda pointed to, the federated governance model of data mesh brings a whole new layer of complexity to data governance. So talking federated governance can create anxiety that there will be even more bottlenecks. But really, for many things, it makes governance far easier. A crucial point of data mesh is to unlock the ability to quickly react at the domain level by giving them the ability to make (most of) their own decisions - and easily lean on expertise when they need to. Kinda talked about good communication around governance: focus on the why, what are you doing to drive value via governance, instead of the toolbox angle of policies, standards, etc. If they understand the why, they are more likely to lean in to the extra work data mesh puts on their plates. So start from listening and talking pain points - theirs and other people within the organization. If you are showing they are heard and that you're addressing their pain points to drive value, governance isn't so scary. One aspect of data mesh that people seem to get a bit wrong in Kinda's view (and Scott STRONGLY agrees) is the federation versus centralization balance. You need to understand each domain's capabilities and overall maturity level. It's okay to start with most of the governance centralized if a domain isn't ready yet. It will be a bottleneck but it's a known bottleneck that is taken as a conscious choice. Sometimes, in Kinda's experience, domains will want to move on pushing out a data product before they and/or the organization is ready. That's okay, just make sure they understand what they own and that they will be on their own - at least it's okay as long as it isn't breaking regulatory compliance or anything. Kinda believes that you should look to do away with the central data governance committee where you can. You want stakeholders engaged of course but you can move faster and with better decisioning by collaborating, not committee-ing (blame Scott for that phrasing, not Kinda 😅). For example, instead of creating a global data quality standard herself and pushing it out, she created a hackathon around data quality measurement and the best solution - or really a combination of solutions - won. Governance people should not look at federation as a 'loss of control' according to Kinda but instead, it's freeing the governance people up to focus on where they can add the most value, creating those blueprints, standards, guardrails, etc. It's helping drive data literacy and data culture. And it's acting as an advisor when necessary - you don't want each domain to implement GDPR compliance for instance. At the end of the day, the people on the ground in the lines of business know the data better than someone with only a centralized view really could so make it possible for them to make the right decisions and apply them easily. A big misconception here is that it's handing over all control but that's just not the case - when there are sensitive decisions or things that are of a bigger scale - e.g. regulatory related decisions - that's when the central team should be focusing on partnering the most. There are a few parts to Kinda's approach for driving buy-in for a domain to take data ownership. The first is that you have to understand each domain and person is different. There isn't a single script that just works. These are people you're dealing with. Be prepared to customize the conversations and solutions. Second, have an understanding of what ownership actually means so you can communicate expectations clearly. Hard to understand exactly what's needed at the very start of a journey but get as close as you can. And third, when actually having the conversation, don't start from the expectations and new demands on them, start from what you're doing to make it possible so they actually can take ownership of the data. That includes tooling, training, and realigning their KPIs/OKRs so the data work is actually a key part of their role and measured results. For any data governance effort, Kinda believes you should make sure to focus on all three of technology, process, and people. The people aspect is the most overlooked but your people are what provide scale by enabling the application of whatever governance work you are doing. In general, driving data culture is crucial for data governance in Kinda's view. Data literacy and culture is part of data governance of course but in general, to find your internal key enablers and to inject data as part of the business processes, you can't just create mandates or platforms. You need to make creating spaces for people to share ideas and celebrate each other part of your culture. For Kinda, when asked Zhamak's provocative question of "Will we actually need a CDO in an organization doing data mesh?", she and Scott agree the answer is yes. Because, at least at the start, there needs to be someone running that central coordination and helping to make all the decisions around what to decentralize and when and a very important one: doing the exec to exec buy-in and communication. So strategic direction and maintaining momentum and engagement is crucial, thus a data leader is necessary. Kinda believes in both a top-down and bottom-up approach to doing data governance in data mesh. The top down is that executive buy-in and helping to change people's prioritization and then the bottom-up is focusing on the use case and making sure they have the necessary guardrails and processes as you continue to build those out. In wrapping up, Kinda shared a somewhat controversial viewpoint on interoperability. While it is important, trying to focus on it too much will likely slow you down. She believes you shouldn't let interoperability get in the way of your work and you can't try to solve for all interoperability at the start. Quick tidbits: If you are doing data governance right, it will remove friction instead of add additional overhead. At the start of a journey, the scales might not be as far in favor of removing friction but always look to build that out - that's a key success metric. For every project or initiative always, always, always start by assessing your data literacy gaps. If you don't, you are setting people up for failure. You can show success through your data governance program by measuring things like time to actionable insights. Metrics in general will play a big role in proving out success so try to measure what is changing and you will probably be bad at measuring at first. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of August 6, 2023 20:10

۲ years پیش20:10

20:10

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#245 Panel: Lessons From Doing Data Mesh Again: The Second Timers Club - Led by Samia Rahman w/ Khanh Chau and Sheetal Pratik 1:17:32

۲ years پیش1:17:32

1:17:32

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Samia's LinkedIn: https://www.linkedin.com/in/samia-r-b7b65216/ Khanh's LinkedIn: https://www.linkedin.com/in/khanhnchau/ Sheetal's Links: Sheetal's LinkedIn: https://www.linkedin.com/in/sheetalpratik/ First Paper at ICEB Conference: https://easychair.org/publications/preprint/qZ3m Interview in Harvard Business Review: https://hbr.org/resources/pdfs/comm/BeyondTechnology_CreatingBusinessValuewithDataMesh.pdf Medium Blog on Saxo Bank Implementation: https://blog.datahubproject.io/enabling-data-discovery-in-a-data-mesh-the-saxo-journey-451b06969c8f In this episode, guest host Samia Rahman, Director of Enterprise Data Strategy, Architecture, and Governance at life sciences company Seagen (guest of episode #67) facilitated a discussion with Sheetal Pratik, Director Engineering and leading India Data Integration Platform at Adidas (guest of episode #24), and Khanh Chau, Director of Cloud Data Architecture at Grainger (guest of episode #44). As per usual, all guests were only reflecting their own views. The topic for this panel was a bit different with reflections and learnings from doing a second (or more) data mesh implementation from a practitioner standpoint - it's people who have the experience to reflect back on multiple implementations to give advice to their past selves and those earlier in their journeys after seeing multiple organizations implementing data mesh up close. Khanh Chau was the leading data architect for Northern Trust's data mesh implementation before taking up the same role at Grainger for their implementation. Sheetal was the head of data integration at Saxo Bank as part of their data mesh implementation before moving to Adidas where she is leading the India Data Integration Platforms and has also played integration lead for moving an on prem centralized integration platform to federated cloud integration platform. And Samia worked deeply on two implementations at Thoughtworks - including one closely with Zhamak - before starting at Seagen. Scott note: I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views individually. Scott's Top Takeaways: Prepare to take many concepts from abstract to something concrete and explain it to many people. Repeatedly. And your definitions will change over time too. A big part of leading an implementation is about communication and keeping people on the same page and informed. Similarly, prepare for confusion. People will go to different sources for information - including a lot of vendor content - to learn about data mesh. So keeping people aligned and understanding key aspects of data mesh are crucial. "What is data mesh?" can be a dangerous and difficult question when it shouldn't be. It's important to learn from other organizations' implementations but your journey - if you are going for success and not simply copying - will be unique in what matters most when. It's easy to get overwhelmed trying to manage every aspect perfectly. So look to focus on learning and iterating along the way instead of getting it perfect upfront. Stepping back and looking at the 7 journeys across the panelists, they really don't look that similar. When choosing which domain(s) to partner with, highest value or ROI can look like the most important metric but really, finding someone who is willing to take a risk and will also champion your solution when it succeeds is the most important. You do need to find use cases that have reasonable ROI but having a true partner is the most important part. Data mesh isn't a solution to the business people - and it really shouldn't be to data people probably too. It's a way to address an ever growing challenge and issue within the organization. The business people feel the pain, talk to the pain. They don't care nearly as much about the how as you assume. Make them a tasty sausage, not give them a sausage factory tour. Don't underestimate the fear of change and loss of control. Data mesh is a new approach and many are sick of new data approaches 😅 but people's fear of getting left behind or not being in control is a natural human reaction. Make sure people understand how this makes things better not just for the organization but also for them. Defining data mesh success metrics - at the micro level of data products and at the macro level of the entire implementation - is still hard. There are a lot of measures you can use that are helpful - and do look into fitness functions - but it will take some effort to find good success metrics. And those metrics will change over time. As some past guests have also noted, a good way to drive buy-in for those actually doing the data work is to make it faster to produce data for themselves, not just for others. If you can make getting access to data and creating something useful for the data producing domain quicker, many will lean in to leveraging the platform quickly. Other Important Takeaways (many touch on similar points from different aspects): Faster time to business outcomes is likely attractive to business partners to get buy-in AND a success metric you can actually somewhat reliably measure. For success metrics for your implementation itself, look at time to deliver new products. Slow delivery might be because a domain isn't ready or leaning in but repeated slow delivery is probably a problem with the platform in some way. It's tempting to try to start from the infrastructure aspects first but you have to focus on the value of doing something like data mesh. What is the value proposition for the organization? What is it for the specific person you are talking to? The needs of your organization will look different than other organizations. You probably can't know exactly what will end up being your big focus ahead of time. A lot of the learnings and realizations can only happen when you're in the midst of your journey. Because every part of an organization matures differently, each domain generally has developed a way of working that fits to their domain but makes it very hard to then standardize the ways of working or sharing data. You need to build central frameworks for collaborating but still allowing domains to work (mostly) in their existing ways. Otherwise, you are probably fighting a losing battle. It's easy to lose sight of how important the business partners are to your data mesh success. They can provide a ton of leverage to driving buy-in, budget, etc. Or, they can provide a ton of headwinds/resistance. Find your good partners and make sure you're aligned on making your overall data mesh approach a success. Similarly, at its core, data mesh is about business transformation. Yes, transformation driven by changing how we interact with data and do data work but business transformation nonetheless. To get funding/sponsorship for something beyond the first few data products, you have to show the value proposition, especially in a soft economic environment. You aren't going to do that by talking technology. It's important to show a good return soon for sure but also a broader vision. Technological solutions don't equate to business value to most business people. Talk to them about what they care about. With every success, you can build momentum. But that momentum only builds if you're selling the successes, telling people and proving out the value. Data leaders need to be as much about showing off the results as achieving them unfortunately. It can be effective to mesh buy-in to point out how decentralized efforts will probably happen - e.g. shadow IT - but they become their own little IT fiefdoms if not at least coordinated in some way through a central mechanism. It doesn't need to be ownership but if you don't have a central framework and ruleset, you end up with silos, spending more money on many platforms, and worse quality data. As Khanh said regarding governance, "So how do we bring this whole ecosystem of technologies [together] in such a way that is safe for people to do things faster and meet all the regulatory compliance?" Two key words in there are safe and faster, don't make governance about gates, make it about getting to data production more quickly but still safely. While data mesh may not itself be cheaper than doing nothing, it often results in cost savings relatively quickly by removing duplication of work. If everyone is at least broadcasting what they are working on, then there is far less overlap that might cause confusion and two teams doing essentially the same thing. Trying to build the entire platform ahead of time is a really bad idea but you also can't come to the business with an inferior, clunky, hard-to-use, immature solution. Make sure your organization has some platform capabilities to not make it a huge hassle to build out early data products. Don't try to onboard every domain at the start of your journey. Start to seed the conversation with them, talk to the pain points, but you can't handle 30 domains coming onboard at once. When driving buy-in, don't only focus on executives. You need people throughout the organization leaning in. If you need new skills to accomplish data mesh, you need people who will acquire those skills and level up their data fluency. Talk to the people doing the actual day-to-day work. You potentially don’t need a huge budget to start a data mesh journey. You can start lean - yes, you will have leaner results but if you can prove faster time to business outcomes, you can generate more and more momentum. Build momentum around your mindset shift. You aren't going to convince people overnight. It's important for domains to take over ownership of data but that also doesn't happen overnight. And your initial ownership will probably not be as strong as you'd like. And it probably won't involve the operational plane system experts/developers as much as you'd like. But that will evolve and get better. You have to get started with okay, not perfect. Crawl, walk, run. You probably need to apply DevOps principles to your data products where the data product developers also support the data products, do the operations aspect of them. But it might be hard to get people really bought in on that when your platform is still pretty nascent. You want to build feedback mechanisms around data products to drive domains to make them better. But there are many concepts around data product feedback being touted by vendors but few people are happy with what they've built or are getting from vendors. The best feedback mechanism is probably a culture shift towards providing more/better feedback. As Samia said, "It's not really on us as governors to govern and … force it on you but to empower you so that you are doing the right things for yourself and for your organization." There's a ton of interest in how generative AI can play in data mesh but it's still far too early for it to have a big impact. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Data Mesh Roundtables - Join Us! 3:05

۲ years پیش3:05

3:05

You can simply use the link here https://us06web.zoom.us/j/87380091930?pwd=NVhUS3ZDOS9hSFFBbU1sQm9HQUJHZz09 or find a link on LinkedIn events: https://www.linkedin.com/events/weeklydatameshopenroundtable-da7091373106551734272/comments/ Link to past recordings playlist: https://www.youtube.com/playlist?list=PL9tZROTS_Pi7TjjhLR-sCtwQVeth6Jv7e…

D

Data Mesh Radio

1
#244 Leading a Data Transformation the Empathetic (and Right) Way - Going Far Together - Interview w/ Benny Benford 1:04:36

۲ years پیش1:04:36

1:04:36

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Benny's LinkedIn: https://www.linkedin.com/in/bennybenford/ Benny's Substack: https://www.datent.com/p/elevating-data-to-a-profession-why In this episode, Scott interviewed Benny Benford, the former CDO at Jaguar Land Rover (JLR) and who is currently building out a community around data transformation. To be clear, he was only representing his own views on the episode. Some key takeaways/thoughts from Benny's point of view: !Controversial!: As a data leader, if you want to be effective, you will have to focus much more on what role data work plays in the organization rather than the data work itself. It's about partnering and communication, not SQL and LLMs. That can be a hard lesson but it's crucial to being successful. ?Controversial?: It's easy to try - and common to want - to separate data culture and organizational culture but it doesn't really work. "Data culture is … an aspect of your organizational culture that you're trying to build and create." A guidepost for your transformation journey: "If you want to go fast, go alone; if you want to go far, go together." The data team can be leaders in transforming the organization but they must do it in collaboration, not just going out as pioneers 'to set an example' or something. Absolutely partner with your transformation org if you have one. When you align your data culture transformation to the overall organization transformation messaging, you aren't fighting for attention to a different message. You are using the momentum of the overall org's transformation strategy. Don't try to fight the tide if you don't have to. Try to get that data transformation as part of the org's strategy and messaging. Given the emerging attention to data in many orgs, it can be tempting to try to go off and try to set culture. But that's almost certainly not what the company is asking a data leader to do. ?Controversial?: "Agile and data culture are two sides of the same coin." They are very close just approaching things from different angles. Scott note: I mostly agree. The big challenge is when people are overly rigid with their approaches to Agile. If you try to force people to share data, it often stokes fears of 'can this data be used against me?' And that is a rational fear in an overly competitive org. You need assurance from senior management/the board of directors that trying to undermine other lines of business/domains will not be tolerated. If you want an open data sharing culture, you need to create clear consequences for internal political misuse. Specifically at JLR, when they opened up a lot more access in their Tableau environment, it felt risky. But because of their well-defined and broad reaching initiative, it was nothing but positives, leading to far more/better collaboration. Being frightened of change is totally normal. Don't try to solve every aspect ahead of time. Oftentimes you have to move with uncertainty and have a good framework for adjusting as you learn more. 'Bravery isn't a lack of fear, it's moving ahead despite your fear.' Scott note: this should be at the top of every data mesh strategy document… Very few data leaders have strong transformational backgrounds and it can be hard to drive transformation without that. Look to educate yourself on how to drive effective transformation. Champion's forums for your data efforts are very effective. It can inspire others but often, it can also draw out existing challenges you might not have known. Leverage those who will share their experiences to make it less scary and improve processes for those that follow in their footsteps. All organizational choices essentially have puts and takes. Don't try to hide the weaknesses or challenges of whatever route you are taking. Be honest. When decentralizing data work and ownership, many people in the centralized team may worry about their role and their future with the organization. Provide clarity and information on happy paths for them inside the organization including their career development. Try to explain how the transition is happening "with you" not "to you". Easier said than done. When thinking about data transformation, it can be helpful to think of it as splitting into projects and program. You need someone minding each aspect. One person trying to own both - the day-to-day execution and the long-term change in how your data org works - can be pretty difficult. It's a LOT. ?Controversial?: Transformation happens when a few people's main focus is leading that transformation. If you have it as a bit part of many people's roles, it's unlikely to amount to a cohesive change effort. You don't need a huge training budget or permission to train everyone in the organization. As a data team, you can start a small training program and as that sees success, you can build more and more. There is a lot of great free content out there, make sure to organize it for people but you can get going now. Get people used to the idea of imperfection in data - and then give them permission to do imperfect things. Don't be cavalier and sloppy but it's okay that things provide value without being 'perfect'. EVERY organization is far more immature with data than they show publicly. Everyone feels like they are behind. Meet your peers and embrace sharing reality instead of the Instagram view of your data transformation journey. It will lead to better information sharing and probably also better sleep at night. It's crucial for data leaders to find allies across the organization. You can't transform a 5K+ person organization by yourself. Other teams can provide you leverage and support. Partner with those willing to partner. Benny started off talking about data culture and organizational culture. He doesn't believe you can separate the two but it is a common desire among data leaders to do so. Data culture is something you build and create as an aspect of your organizational culture. Otherwise, you will always be fighting against the tide of your organizational culture. It all needs to be integrated change/transformation. Otherwise the organizational changes will be in conflict with your data culture changes. And people are employees of the organization, not only the data team, at the end of the day. "If you want to go fast, go alone; if you want to go far, go together." For Benny, that emphasizes the long-term transformation efforts you need to drive to a better data culture. You can go somewhere quick but if everyone is going somewhere quick, there is no cohesion, no real concerted change to the organization and thus the culture. If you have a general transformation team, align with them. Align on the overall corporate goals and leverage them to drive your data culture transformation. Yes, slightly harder than just creating your own but far more effective and long-lasting :) Yes, you want to get your data transformation initiatives as part of the org strategy but it doesn't have to be the main focus. And it might take a bit of time to make it a core pillar of the strategy but that's how you do effective change management: building to better over time. Benny made the excellent point that while it can be tempting to leverage the attention data is getting in many orgs to try to drive major organizational culture change, it's almost certainly not what the org is asking a data leader to do. Just because there is some 'juice' there, it's not where a data leader should focus. Ask if your job is really leading the organizational change or is it about delivering on data related objectives? While pressing for an open data sharing culture is great in spirit, Benny has seen it can stoke a lot of fears - what happens if someone looks deeply into my line of business/domain to try to find mismanagement or incorrect decisions? And that could happen in a very competitive organization. That is why the cultural guidance needs to come from senior management and the board of directors around what would be considered unethical internal use. Assuring people there will be consequences for trying to undermine instead of help other lines of business can reduce a lot of fears. When JLR opened up their data sharing, it felt like a risky proposition but really, because the guidelines were clear and it was about opening up much more information access, it went very smoothly and significantly improved collaboration. Benny pressed on the idea that we can solve every eventuality, every challenge in a data transformation ahead of time. Of course, we all know that's not possible. People will be afraid of change, that's normal. But having a good transformation strategy incorporates that fear and addresses it by having good ways to uncover and address emerging challenges. Assure people that while you don't have every answer, you have their back. It's also very common for data leaders to really lack experience in transformation initiatives according to Benny. While data leaders are really good at data, that lack of transformation experience can be hard. Lean into this isn't about going it alone, we're in this together :) Benny is a big proponent - a champion you might say - of champion's forums. Essentially having a place where champions of your data initiatives can exchange information with each other and your data leadership team are crucial because you can find your existing challenge points much easier. If you take in their feedback, they feel seen and heard and lean in even more. You also have great points of leverage to inspire others based on the success of your champions. When moving from a centralized to a decentralized structure for the data team, Benny saw a lot of fear from those in the centralized team. Would they even have a role in the changed organization? What was their career path? You don't want to lose your data talent - it's so expensive and hard to replace - so be clear about the path forward for them. There is always uncertainty but try to show them they are and will continue to be valued. Another experience from JLR that worked well for Benny and team was that when he was hired as a data leader, he realized he probably couldn't lead all of the transformation of focusing on both the day-to-day execution on data projects and the overall change program. So he got in a second leader to lead the projects while he focused on the program aspect. Transformation doesn't happen in a vacuum, people need to be focused on that specifically. That's the program aspect. Just executing well won't transform your team and culture. Similarly, Benny shared how they built a transformation flywheel around data - at first, it was training a few people as part of the central data team's time, training them to act on their own in their lines of business. They didn't even have budget for this training specifically in year one, two, or even three as it continued to grow to 100s of people. They started seeding the organization with data capable champions that pushed others to take the data training. And after a few years, the training was so much in demand, it was too big for the data team to own so they brought in external trainers. But you can start small and have a big impact. Benny recommends when getting started with - or really any point along the way of - a data transformation initiative, a big benefit is to get people used to and comfortable with the idea of imperfection in data. You can capture great value even around something that isn't perfect. And it will never be perfect. That's okay. Nothing in this world that is complicated ever gets to perfect! Deep partnering with other parts of the organization might seem obvious but in Benny's experience, it isn't all that commonly done by data leaders. The data team should have partnerships with HR, Finance, Sales, Marketing, etc. And if you have a transformation office, that should be your number one best friend internally. You need partners to move the organization forward. And that also means data leaders need skills that aren't just data skills. Benny wrapped on that point - what is a data leader's role in a large organization? Their role is rarely to focus on the data work itself anymore and that can be a bit of gut punch for those who love data. But it's about building the bridges to the rest of the organization and helping them do their work better. That means you're still doing the data work, just with the lens of bringing it into your business partners' context. Still, it can be frustrating and hard to give that data work up. That's not unusual or unexpected, it's just part of what it means to take on that data leadership role. Quick tidbit: Data culture and Agile culture, Agile transformation end up being pretty similar in many respects. Look to how organizations are successfully implementing Agile transformations to inform your data transformation strategies. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of July 30, 2023 33:44

۲ years پیش33:44

33:44

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#243 Zhamak's Corner 26 - The Fundamental Data Need: Autonomy with Interconnectivity 21:51

۲ years پیش21:51

21:51

Takeaways: It's important to understand that we need enablement to do data mesh well. That is enablement through technology and enablement through organizational approaches/behavior changes. Doing only one will likely not work. "...they need to move fast, they cannot be bogged down by centralization of any kind, organization or technology." Scott note: I will say we discuss later the need for centrally provided enablers but central bottlenecks are the speed and flexibility killer, look to prevent and remove them where possible. People want to simply produce data as a normalized process of doing their job and make it consumable for the rest of the organization. How can we enable that? Why is it so tough? How do we make it interoperable - and more importantly interconnectable - too? Right now, the missing core component to do data mesh well is an easy ability to create and manage data products. Everyone is having to cobble things together and then trying to layer on the observability, the access control, the interconnectivity, etc. But it's built on a shaky foundation. Sponsored by NextData , Zhamak's company that is helping ease data product creation. For more great content from Zhamak, check out her book on data mesh , a book she collaborated on , her LinkedIn , and her Twitter . Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Data Mesh Radio episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#242 Making Data Accessible Makes Your Data Work Successful - More on PayPal's Data Mesh Journey - Interview w/ Kim Thies 1:17:32

۲ years پیش1:17:32

1:17:32

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Kim's LinkedIn: https://www.linkedin.com/in/vtkthies/ Gemba Walk explanation #1: https://kanbantool.com/kanban-guide/gemba-walk Gemba Walk explanation #2: https://safetyculture.com/topics/gemba-walk/ PayPal Data Contract Template OSS: https://github.com/paypal/data-contract-template/tree/main/docs Start with why -- how great leaders inspire action | Simon Sinek | TEDxPugetSound: https://www.youtube.com/watch?v=u4ZoJKF_VuA In this episode, Scott interviewed Kim Thies, at time of recording a Leader on the Enterprise Data Team at PayPal and now SVP, Client Innovation & Data Solutions at ProfitOptics. To be clear, she was only representing her own views on the episode. Some key takeaways/thoughts from Kim's point of view: When talking about data mesh to execs, it's helpful to go back to basics: "these are the four main principles, and this is what we've built and why." Scott note: I recommend you slightly alter the phrasing, especially around "Federated Computational Governance" ;) Look to Simon Sinek and "Start with the Why". Always investigate the why for the other party. What would be enticing to your business execs to lean in on data mesh? But data mesh for the sake of data mesh is not going to win over the business. The communication and relationship building aspects of data work are often overlooked and will serve you better than just about any architectural or technology decision. Build the relationships so your great data work will address actual business challenges and be leveraged! ?Controversial?: Similarly, learn 'the art of the conversation' so you can extract from people their needs/wants and then see how you can help them meet those. What isn't going well where you can help is a great question to find new use cases where the stakeholder will be engaged - you're directly addressing a business challenge for them. It's probably better to start from needing a solution and data mesh is a good fit rather than using data mesh as a hammer and looking for nails. You shouldn't want to do - or at least shouldn't propose doing - data mesh for the sake of doing data mesh. If you have a clear business use case, it's much easier to get people engaged and keep them engaged/involved - around data mesh or data work in general. Look to a tangible benefit - e.g. cost savings is a pretty easy first use case to go after. ?Controversial?: Data teams - especially data engineers - need to spend more time "experiencing" how customers/users actually use data to deliver on their business objectives. It will lead to better outcomes and better relationships with the business leaders/teams. Leverage the Gemba Walk philosophy: walk the 'factory floor' and talk to people far more often. Ask them how they get their work done. It doesn’t need to be overly formal, just collect information to help others do their job better with data. You don't get to rest once you've gotten initial approval. Execs' attention will not last, they will start to focus on other challenges. Keep pointing to the business challenges you are addressing - not the data work itself - to stay relevant and near top-of-mind. It's not as if every aspect of your business starts doing a data mesh approach when you start your journey. There will be compromises and other parts of the business will likely choose other approaches. That's okay and normal. Build your momentum and successes around data mesh but accept it won't be right for everyone, especially at first. Data mesh will be received differently by every person or at least every persona. Each of the pillars might resonate differently. So be ready when speaking to focus on the aspect that is getting them to lean in. You need to balance the four pillars in your journey but not every conversation :) There is probably far more transformation needed in your data practices than you expect. Even after hearing that, there is still probably far more than you expect. Processes and hearts + minds especially. In the current business environment - likely headed for recession - you might be able to get people bought in on data mesh simply for time-savings for the data team. In downturns, cost cutting becomes far more attractive. Once you get your mesh journey going and you have some interesting capabilities to offer, it's important to go out and find additional use cases. Even if you're proving a lot of value, people still probably need a little convincing or at least some additional understanding of what you're doing, they won't all come to you. To speak the same language as your business partners "you have to listen first." It's pretty easy to assume you mean the same thing but even foundational phrases like data product and data contract often have completely different meanings for people within the same organization. It's incredibly easy to overlook user experience in data. Don't fall into that trap! Scott note: we did a data user experience panel if you want to dig deeper, episode #190. ?Controversial?: Domain ownership is probably the most important aspect of data mesh because it puts the data back in the hands of the people who really know it best. Context loss is such a prevalent problem in data and data mesh solves for that quite well. Kim started off with a bit about the PayPal journey to data mesh. They weren't looking to do data mesh, they had a specific business problem of disparate data sets across multiple domains that needed to be combined with decent governance and observability. It just so happened that data mesh was a great fit, so they went the data mesh route. And there actually was an engineering-led conversation of the engineering team looking for a use case to use data mesh but that didn't end up being the data mesh initiative that went to production. There was also a bit less scrutiny on a business-led journey because it was within a larger line of business/domain rather than it was the core data team strategy. One thing Kim believes is that data teams, especially data engineers, do not spend enough time really understanding and "experiencing" how customers use data. They should go and pair closer to the business teams to deliver better solutions. That will lead to better outcomes but also better relationships. This is that aspect of data UX that is often overlooked. People want something to make their jobs easier or make them perform better, not just data. So how does data actually intersect with that? To actually do that 'experiencing', Kim and team leveraged the Gemba Walk philosophy (see links above for a deeper dive). It’s a Japanese concept of going and 'walking the factory floor' to collect information. You don't need to do super formal interviews, meet people where they are and ask them questions about their day-to-day. Get a sense of what really matters and how they do work. If you just ask people what they want, they might give you the Henry Ford 'faster horse' answer versus you discovering their points of friction. Kim discussed the challenges of not just keeping but retaining buy-in and attention to your data mesh implementation - or really any data work. Once you get the approval, there are probably things that are more top of mind for your business partners, especially compared to the specific data work parts. So keep circling back to the business challenges you are addressing in conversations to stay near top of mind. Kim and team accomplished that by getting to a really rapid, tangible proof of concept. They quickly had something to show from the work that made it clear this could work at scale. Starting from entirely inside one business unit meant Kim and team had a lot of autonomy. As soon as they were brought more into the central data team, things changed and there was much more focus on communication and sharing - and compromising too - especially around technology choices and approaches. Many teams were taking different approaches to similar challenges with the same technology so how do you get to best practices? One thing that really stuck out to Kim about data mesh and driving buy-in is that different pillars resonate for different personas. Data scientists or even data engineers embedded into a domain often love the self-serve aspect. Not even for consumption but being able to actually self-serve their own data production needs. They aren't afraid of the ownership principle of data mesh because they can own their own timelines and not get stuck in centralized bottlenecks. They might not even realize they are struggling pre data mesh because the central bottleneck/friction is so ingrained to their way of working, you can really make things far better for them but they wouldn't have considered asking. Kim believes you should be prepared to market your mesh capabilities once you have them up and running. That doesn't have to be a sales-y approach but you want to go and find additional challenges people couldn't solve but now could with mesh to gain more momentum, converts, and funding. Learning the art of the conversation is crucial. Is there a way that you can advance their business goals, address their challenges? If yes, they are more likely to lean in and stay bought-in to the use case. Don't be afraid of a little sales and marketing tactics to get to better business outcomes for all. It's very easy to use the same term and mean something different. Just look at all the what is data mesh or what is a data product content out there. So Kim believes when talking to your internal partners, start from listening first so you understand how they are using different terms. It's easy to jump to solutioning instead of understanding but your solutions will be lacking without the understanding :) It's pretty easy to lose track of the customer experience in data mesh in Kim's view. There are so many things to focus on and honestly, UX design in data hasn't really been a thing. As an industry, we haven't talked about UX design for the platform or the data product much before data mesh started to force the conversation. If you have all the capabilities in the world but a bad user experience, are people really going to use what you've built? Circling back to communication, Kim talked about the challenges of communicating with execs and getting bogged down in the work done. She said it's important to start from baseline communication around data mesh: "these are the four main principles, and this is what we've built and why." If you start there, then people can really start to connect the work to what might be beneficial for them. And as she stated earlier, still look to start from the listening and understanding aspects first and then always start from the why. Scott note: see the YouTube link above. In wrapping up, Kim talked about the through-lines of the conversation. Establish and build relationships, not just data products. Learning the actual problems/challenges is key. Talk to people where they are before you try to build something for them. Understand what's actually going on and where their friction points actually are. And then also look for scalable business cases - the best way to discover those is by establishing the relationships :) Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of July 23, 2023 15:42

۲ years پیش15:42

15:42

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#241 Data Product Success Metrics - A Kinda Deep Dive - Mesh Musings 51 17:31

۲ years پیش17:31

17:31

Key summary points: At the start, it's more important to start measuring than it is to measure the right things. Do NOT let analysis paralysis hold you back. Similarly, your success metric measurement framework will probably suck to start. Oh well, get to measuring. Create a framework and tooling/platform capabilities - where necessary/useful - to make measuring and reporting against success metrics simple. That framework should be about defining the metrics and especially how to measure, not what success looks like for individual data products. Use fitness functions Good metrics to consider in order of usefulness: user satisfaction, user value, data quality, time to business decision, delivery to expectations, time to update (can be squishy), and usage Please Rate and Review us on your podcast app of choice! Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#240 Driving to Better Healthcare Patient Outcomes Through Data - Interview w/ Smriti Kirubanandan 59:59

۲ years پیش59:59

59:59

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Smriti's LinkedIn: https://www.linkedin.com/in/smritikirubanandan/ Smriti's HLTH Forward Podcast: https://hlthforward.buzzsprout.com/ In this episode, Scott interviewed Smriti Kirubanandan, a Healthcare and Public Health Data Expert at a large consulting firm. To be clear, she was only representing her own views on the episode. Much of the challenges and opportunities discussed in this episode are more on the US side because of the not-so-well-functioning healthcare system there. Some key takeaways/thoughts from Smriti's point of view: In healthcare, it's easy to lose sight of the patient in the data - focusing solely on a condition, an area of the body, or a set of data instead of a person. It's vitally important to be focused on the data through a lens of treating the patient as an entire person. !Controversial!: It can sound time consuming to interact with data "in a much more intimate format" much like a 1:1 conversation but it's very important to drive to better outcomes. Instead of automated decisioning, we can point our tooling to compile the relevant information better to make decisions faster without removing the care or the person. Machines making automated decisions leads to worse patient outcomes. "Obviously, privacy is important. Ethics is important. How do we interconnect this data and how do we get to communicate amongst" the payers and providers? So physicians can look at a much more complete picture of the patient to treat them better. There are many organizations collecting important health data about people. We need to rally around the patient outcomes instead of the financial outcomes and combine the data. Easier said than done of course. ?Controversial?: Companies with important health data need to lean forward at the table, to buy-in on collaborating around sharing data or we will continue to have suboptimal patient outcomes. More organizations should make it possible to 'act local' relative to individual health. Instead of every decision being a very complex one, can we make things easier to simply make health progress if not 'fix' everything for someone's health. Basically, make it easier to make small decisions around more concrete and focused areas, much like a domain in data mesh. It's very important to empower people to leverage their own health data so we have to focus on getting them access and then giving them the power to do something with their data to drive better outcomes. There are 3 big issues we need to tackle simultaneously: 1) How do we give access to relevant and useful data to caregivers? 2) How do we ensure digital equity? And 3) how do we share data ethically? Think about interoperability - can I pull data from one system to integrate into my system - and interconnectivity - a more two-way interoperability/integration. We need to focus on interconnectivity far more. Especially in something as important and complex as healthcare, it's crucial for the data and engineering people to stay focused on target outcomes and not get lost in the code/work. A shared vision at the project and organization level are key. Many data projects go wrong because we still struggle with communication. Not that we aren't communicating but keeping all parties to data work aligned and in sync as learnings emerge is very hard. And data work needs to allow gray areas, which it often doesn't do well currently. Value-based care is a really important aspect of getting people the best care and data can help support that well. But it requires a lot of ethics and transparency to get there. ?Controversial?: Digital twins of actual people could change healthcare in major ways. If done right, it could greatly improve the ability to treat patients because of the ability to test against negative health outcomes and find more optimal treatment plans. Smriti started out the conversation with a bit about her background and then jumped right into a key challenge in data around healthcare today: treating the patient as a person and not a set of data points and measurements. How do we look holistically at a person and focus on what would be best for their health AND life - the two are intrinsically linked. We still want to drive insights but personalized care, at least in the US, seems to be on the out and we can bring it back better with better data. Can we actually interact with healthcare data at the person level instead of at the billing code level? Smriti believes we can - that instead of letting the automated tooling make the calls on important health decisions - such as if key procedures or tests are authorized by insurance - we can use the tooling to allow a more "intimate" interaction with what the person is going through and how can we serve them best. That we can better leverage tools to make more humane decisions for folks. For Smriti, it's still a very tough challenge for how do we get data to interact across the various healthcare data silos, how do we smoothly exchange this data. And how do we tackle the governance of making the data interconnect? Right now, physicians cannot see a large amount of crucial patient data; but is that on patients to connect the data between offices and facilities so doctors have a more complete view? How do we maintain privacy if we are sharing information across systems? What about ethics, do we really want to give a lot of these companies very intimate health data? Scott note: see the recent acquisition of One Medical by Amazon - they are now supposedly requiring patients to waive their HIPPA rights to get care. Because there are so many challenges around integrating healthcare data across so many systems/silos, Smriti believes that one company itself can't make that big of an impact to the overall system. BUT each company being better about doing their part can help achieve a data-driven aspect to healthcare that leads to better patient outcomes. There needs to be more of a concerted effort to collaborate in the right ways. In Smriti's view, it's very important to empower people to make better health decisions for themselves driven by data. That means giving them access to more of their data, giving them the capabilities to leverage that data to make decisions, and then empowering them to actually act on those decisions. There are some pretty basic things we can do to improve the health of our fellow citizens and it's on a number of people to keep the pressure on to move forward on that. Not one entity can do it alone but we should all be pressing for better solutions. Smriti talked about 3 big challenges to sharing our data and driving better patient outcomes. The first is how do we actually get data in front of our care-givers? How can we empower the individual to share that data and how can the physicians or other care-givers access it and drive better patient decisions? The second is how do we ensure digital equity? Many people don't have access to good internet. Many are not digitally literate enough to actually participate in data sharing. How do we empower them to participate in better health outcomes? And the final challenge is how do we actually share this data ethically and with empathy? All of these are being worked on but it takes a very large cross-org contingent to move things forward. Everyone can play a part but it will take a lot of collective work. Interoperability versus interconnectivity is something Smriti is passionate about because interoperability doesn't really ensure that two systems can share information all that well - it might be that your data is in a proper format but your definition is way different than mine. Interconnectivity is about a two-way collaboration and easy integration between systems around data in her definition. That interconnectivity is necessary to really supercharge our health data revolution :) Smriti talked about a challenge in data that many past guests have touched on - how do you keep people focused on target outcomes instead of the minutiae of the work, keeping them from getting lost in the code instead of what you are trying to achieve. It's key to have a shared vision about what is the goal and why are you doing the work. And if people lose sight of that, you need to bring them out of the weeds or you'll get very interesting solutions that don't solve actual important problem. When asked why data work seems to not net expected results so often - the 80%+ of analytics initiatives don't meet expectations statistic - Smriti pointed to difficulties in communication. Not that we aren't communicating but the challenges around how do we quickly iterate together and share small-scale incremental learnings so reality and expectations are not constantly drifting apart. Basically, communication is hard and we need to place more focus on getting it right - and having a wider tolerance range initially of what 'right means - but there's no silver bullet. We need to be able to be vulnerable with each other and operate in gray areas :) Smriti really believes in the concept of value-based care. But to get there, we need transparency around price and care. Individuals need to have access to their information but also need to equip themselves with the knowledge of how to leverage their data to get better care. It isn't all on the care workers. Digital twins in healthcare is something Smriti is really excited about. A digital twin of a person gives providers an ability to potentially test reactions to different treatment protocols, optimizing positive outcomes and hopefully minimizing negative outcomes. Physicians can test a number of treatments simultaneously without experimenting on the patients themselves :) Healthcare digital twins are in their very early stages but she is quite excited about the possibilities. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of July 16, 2023 15:52

۲ years پیش15:52

15:52

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#239 Panel: The Role of Data Product Management in Data Mesh - Led by Frannie Helforoush w/ Alla Hale and Jill Maffeo 1:04:04

۲ years پیش1:04:04

1:04:04

Please Rate and Review us on your podcast app of choice! Get involved with Data Mesh Understanding's free community roundtables and introductions: https://landing.datameshunderstanding.com/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Frannie's LinkedIn: https://www.linkedin.com/in/frannie-farnaz-h-a7a11014/ Jill's LinkedIn: https://www.linkedin.com/in/jillianmaffeo/ Alla's LinkedIn: https://www.linkedin.com/in/allahale/ In this episode, guest host Frannie Helforoush, Technical Product Manager/Data Product Manager at RBC Global Asset Management (guest of episode #230) facilitated a discussion with Alla Hale, Senior Data Product Manager - Digital Capabilities at Ecolab (guest of episode #122), and Jill Maffeo, Senior Data Product Manager at Vista (guest of episode #151). As per usual, all guests were only reflecting their own views. The topic for this panel was broadly data product management and the role of the data product manager in a data mesh implementation. Data Product Manager is still a very nascent role so there is still a lot of confusion around it :) If I were to sum up the feeling of the conversation very succinctly, it would be: it's early days, have patience. Scott note: I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views individually. Scott's Top Takeaways: The role of the data product manager is pretty wide-ranging. It's easy to get overwhelmed and not focus on what really matters. We have to be patient as we learn best practices around data product management because it's still a nascent space. It's crucial to focus on who the user of a data product will be - whether that is an intermediate user creating a report for another or someone directly consuming from the data product. It's very easy to lose sight of what is the exact use case and how will people use the data product - to consume information. Data for the sake of data is just expensive 1s and 0s. There isn't even a common understanding of what a data product is, no standard definition. So data product management is even tougher to define in many ways. It is still a very emerging practice so everyone is still figuring it out. It's okay if things are a bit messy and muddled, they are for everyone else too. Because there isn't _really_ a tangible UI (UX is also a bit muddled) to a data product, it's really hard to get a good understanding of the boundaries of a data product. People don't have an experience with data products like they do with most types of products, digital or physical, so you have to have some patience as people figure it out. There are a ton of learnings we can bring from physical and digital/software product management to data products. Some things only need small tweaks to work well. But be prepared for lots of trial and error so make more room for experimentation than you would in software. What is a data product team - or is there a data product team - is a question every organization has to ask. And the answer for each org probably evolves too. At first, as you are figuring out how to build and manage data products and your platform is immature, you probably have a team specific to data products. But eventually, for many data products/domains, there will likely simply be a data product developer that is part of the product team or data product development will be among the general team's developer duties and product management gets a bit more fuzzy. Product marketing is a relatively foreign concept in data products. But it seems there needs to be far more interaction with existing and potential users of a data product to add value to existing use cases and create new use cases. As of now, that responsibility probably falls on the data product manager unfortunately. The best path to developing a very valuable data product is 'consuming stakeholder' engagement. If the consumer isn't engaged, if they aren't giving the necessary information to really develop the data product to solve a use case, consumption seems to be below expectations across every org I'm seeing. Other Important Takeaways (many touch on similar points from different aspects): User Experience (UX) is such a crucial part of essentially all product management - how does that play in data products? Is it on the platform team to create the UX and the data products just plug into that UX? Is that just the UI? How do documentation, number and types of APIs, interoperability/interconnectivity to other data products, etc. factor in? We need more adoption of forward-thinking/leaning product management practices in data. Instead of only being reactionary to requests, how are we going to extract what people want next? Data products are so new and so many people have preconceived notions of what the phrase should mean or does mean. Be prepared for lots of work on alignment just around the definition of data product. Be prepared for friction when introducing good software product management practices to data. Many data people aren't used to really interacting with product management/managers so they will require some time and hand-holding. Getting buy-in for the budget to put the proper amount of data product management/managers in place might be a little difficult. Especially when you start to ask where that budget comes from. Make sure to have these conversations early in your journey and adjust as needs arise. Similarly, a common issue in software product management is a PM is too overloaded to talk to customers or potential customers. That's even more likely to happen in data product management if you aren't careful. Really focus on making sure there is enough time for product discovery or you will be doing development by request only. Defining data product success metrics - broadly and then specifically for each data product - is going to be a struggle. Usage is an indicator but not comparable across data products for instance. That doesn't even get into the challenge of then measuring the success metrics. But you just have to start and starting with mediocre metrics is better than not starting at all. How the heck do we think about A/B testing relative to data and data products? Do we have enough consumers to test? What does it even mean to A/B test data? Similarly, experimentation is key to doing data better. But can we experiment around the data itself? Or are we just using data to power and measure experimentation? Maybe experimentation around interfaces to data? Data products have a lifecycle. If you are unsure of the value of a data product to users and you can't really get them to articulate it, sometimes shutting down the data product - at least temporarily to do the 'did anyone scream' test - is going to be the right call. It's crucial to balance long-term product sustainability with time to market. As you learn more about building data products, many organizations are seeing decreasing time to market of new data products but especially at the start of a data mesh journey, it is really easy to create non-sustainable/maintainable products to match requests instead of needs in a productized way. We can create data products that answer questions that 'should' be asked, but should we? If we are sharing insights that are not ready to be leveraged or are not what the stakeholders care about, what will adoption be like? Thus far, it seems like most data mesh implementers are saying adoption is below expectations for proactive data products not created to specific stakeholder-defined use cases. Hopefully that changes as we increase data literacy but important to consider. It's crucial to consider interoperability and cross-domain usage/queries as part of your potential user flow but it's also very hard to try map ahead when you aren't sure of those use cases. We can easily lock ourselves in and reduce flexibility, which then might become like a data warehouse, just micro-warehouses everywhere… It will be interesting to see where data product managers come from - if they come from traditional product management, they have to learn a lot of the hidden nuances of data. But coming from data, it's easy to get too tied into the data work instead of what is the data product supposed to do. Figuring out where data product management reports will be important for organizations. It could be into the data team, it could be into product org, into the domain, or maybe even the CTO. There are definitely puts and takes to each. Personally, I think they should start reporting into the data org and eventually move to being part of the product org or reporting into the domain itself because data ownership and data products are just a part of a domain's mandate. Frannie asked a great question: should the data product manager own the data product strategy? And what even is a data product strategy? I feel like the answer is yes but that data product strategies will be very immature as people learn how to build and manage data products. Who should own the overall strategy across all data products? It all starts from the user and use case. To build a good data product, you need to understand how people are going to use it. But then, are we building one-off instead of universally usable data products? I think you start from a use case and build out, expand out but maintain as much flexibility as possible. Product discovery - the process of finding out what products you should build and what are the useful features to add to those - is really important in product management. But it's far more challenging in data because everyone always says yes to 'do you want more data?' How can you determine what will be used and valued? Where is a real use case versus a 'that would be nice to have'? Data product management should extend into what data do we want, what incremental data should we generate, not only what data do we have. As Stephen Galsworthy talked about, especially with hardware, you often don't get to change what info you are collecting. Data sourcing is going to be part of data product management. Maybe good data product marketing is just being highly available and willing to explain - having office hours and show and tells. If trying to generate new uses cases from unengaged potential users isn't driving great value, maybe it's all about inbound marketing, not outbound. Much of the value in data mesh is cross domain use cases and that value is captured by combining information from different data products. But it seems we are still early days in figuring out how to communicate some of the potential questions out there to design good data products to answer them. Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#238 Bringing Software Testing Best Practices to Data - Interview w/ Sofia Tania 1:09:47

۲ years پیش1:09:47

1:09:47

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Tania's LinkedIn: https://www.linkedin.com/in/sofia-tania/ Presentation: "Data Mesh testing: An opinionated view of what good looks like": https://www.youtube.com/watch?v=stNZQESndAA In this episode, Scott interviewed Sofia Tania (she goes by Tania), Tech Principal at Thoughtworks. To be clear, she was only representing her own views on the episode. Scott asked her to be on especially because of a presentation she did on applying testing - especially important for data contracts - in data mesh. Scott note: I was apparently getting extremely sick throughout this call so if I ramble a bit, I apologize. Tania's dog also _really_ wanted to be part of the conversation so you might hear us both chuckling a bit about her antics. And Tania has some really great insights so I probably asked her probably the hardest questions of any guest to date. She did a great job answering them though! A lot of the takeaways are about are we actually ready to do a lot of the necessary testing to ensure quality around data, which I don't think has a clear answer yet :) Some key takeaways/thoughts from Tania's point of view: We have to bring software best practices to data but we should do it smartly and not make the same mistakes we made in software, let's start from a leveled up position. Zhamak has said the same. The question becomes how but looking at how practices evolved in software should bring us a lot of learnings. Just pushing ownership of data to the domains won't suddenly solve data quality challenges. The new owners - the domains - have to really understand what ownership means and what quality means for use cases leveraging their data. A reasonably good way to measure if your data product is 'good enough' regarding data quality is to look at your SLOs (service level objectives) and SLIs (service level indicators). If you are constantly hitting the SLIs, you can probably focus more on new features. If not, you need to improve your quality. !Controversial!: Consider almost a zero-trust approach to testing for data. Test as data is flowing through the systems, as data lands. Test in development as to what changes might impact data. And then consumers should be writing tests against source data to prevent issues. Scott note: that's a lot of tests but how important is certain data to your org? In a decentralized ownership model, many data consumers are less likely to trust data - at least at first - so you need to show them why they can trust at the data product level. Leveraging proper testing and data contract strategies is crucial to being able to prove out data quality. You should look to build out a robust testing and observability framework as part of your platform. Data product owners/domains shouldn't have to build it out manually themselves. If you only have detection of data quality issues once data hits production, once something or someone is potentially already using it, that's an issue. Look to create ways to test data at the data product development stage, as part of the CI/CD. We can't rely only on lagging quality indicators if we want to up our data quality game. Data for analytics and AI is even more complicated than on the operational plane. Generally, data hasn't been transformed multiple times on the operational plane so if there is an issue, it's either in the source application or an issue with the API call that was made. In data, we have to develop smarter tests as data flows through your pipelines. Data producers need to define quality data in terms of what consumers actually want/need. Instead of arbitrarily setting quality levels, what do the consumers want? Consumer-driven testing in data sounds wonderful. But it's hard to see teams being willing to do it :) We need better tooling and ways of working to make this easier. Data quality surveys of data consumers are important for a number of reasons but are lagging indicators. They should be used to help develop appropriate SLAs/SLIs for data products and monitor if data products are generally meeting customer needs. ?Controversial?: Can a data producer really develop a custom test for their data product for each consumer or do the consumer owes it to the producer to develop tests to ensure the data product continues to serve their use case well? Scott note: this could start a LinkedIn war but it's an important question to ask! If you push for consumer driven testing, don't be surprised at a lot of pushback. That happens still even in the API world where it's been more accepted for years :) Are consumers ready and able to programmatically define what good data quality means for them for each use case? There are some tools that can help but practices and tooling are still mostly nascent. ?Controversial?: Many consumers still have the 'give me all you have and I'll sort through it' mindset. Trying to get them to lock into what they are consuming will be hard. There can be a real chicken and egg scenario around data products, especially testing. Consumers don't know precisely what they want and what will best suit them until they see the data/data product. But building out a data product and having to change it a lot to customer feedback is also tough - producers want to build it once instead of 10 iterations. Just be prepared for this to be an ongoing issue in data and for it to lengthen times to data product release. ?Controversial?: Having your transformations handled by low-code/no-code solutions can easily hurt you more than it will help you. Be very wary. Scott note: this is coming up in A LOT of conversations recently and was featured in the Thoughtworks radar released in early May In software development - including development for data - abstractions are crucial but can get you in trouble. Really think deep about your abstractions because it's easy to lose sight of what underlies the abstraction. And abstractions of abstractions of abstractions just compounds the issue :) Tania started with a bit of her background, especially related to data mesh. She worked on one of the clients that was an inspiration for Zhamak's original data mesh blog post and spent 2+ years as the lead on the technical side of a data mesh implementation at another client. Her background as a developer and tech generalist have shaped her thoughts around bringing good software practices to data. For Tania, the reason she originally put together the presentation on software testing practices in data was a client question (paraphrased): 'if we already have data quality issues in the centralized setup with clear ownership and people who really know data, how the heck are we going to _improve_ data quality by pushing data ownership into the domains?' It's a very fair question - just pushing ownership without the capability and buy-in to own the data is possibly (likely?) going to lead to worse quality. So we need tests that work and can be shown to consumers to help ensure quality and trust in that quality. Showing people your kind of data quality certification goes a long way towards trust. In Tania's view, much of the existing data observability tooling and practices, while valuable, only really alert when there is already a problem that's hit production. Is there a way where we can shift testing left, not just in ownership but testing earlier in the flow of data? Earlier in the development timeline of a data product? So that is 3 potential ways to shift left, to test earlier. Think about detect versus protect - can we prevent data quality issues instead of only better identify and resolve them? Tania talked about how data product producers need to start to shift their thinking around data quality. What specifically do my consumers want - and why? Quality is inherently subjective so extract from them what their needs are and look to serve those. And we should look to stop using _only_ lagging quality indicators like surveys. They are valuable in reshaping what SLAs should be and is a data product meeting needs and expectations but they are certainly not designed to quickly detect issues. But do consumers actually know what would make data 'high quality' for them? Consumer-driven data quality testing is a good idea for many reasons in Tania's view. When we think about a single data product having 5 known, regular data consumers, does the data producer need to develop 5 different sets of tests to specifically protect against breaking changes or issues specific to each use case? Do they have to define quality metrics differently for each use case of the same data product? Do they have to be so familiar with each use case that they evolve their tests as use cases evolve? How much can we reasonably ask the data product consumers to do in the testing space to ensure quality? But Tania admitted she hasn't led a client in doing consumer-driven testing for data. It's really hard to get data testing right in general, are people really ready for doing consumer-driven data testing? We don't really have the tooling or the general best practices to do it well yet. And there is also just philosophical pushback - being forced to programmatically say what good quality means instead of saying 'the data quality isn't good enough' is a tough pill to swallow for consumers. Do consumers really know precisely what they want? Things like the tools Great Expectations or Soda Core are a good start here but we need more. And many consumers are still in the 'give me all the data you have' kind of mindset so reducing the possible scope of data they get is not an easy mindset shift. Tania also pointed to a persistent challenge in data that is a chicken and egg problem: data producers can't build exactly what consumers want until they get feedback from the consumers. But the consumers don't know exactly what they want until they've seen an early iteration of the data product. So you have lengthening time between conception to release because both sides need more from the other to move forward but can't until they get at least some information. A good way to press on consumers might be to ask them about bad-case scenarios - what has to be there and why? That will _possibly_ prevent kitchen sink feature requests. As the conversation transitioned into low-code/no-code tooling, Tania lamented about the difference between ease of use and simplicity. While low-code/no-code tools can be very easy to use at the start, as scale/complexity of use cases increases, they often become extremely difficult to manage. They are focused on ease-of-use, their architecture isn't about maintaining simplicity of managing the solution as it scales. As you add more and more views, you might actually have 30-40 joins across many data products and performance comes to a halt. This was also mentioned in the Thoughtworks Radar that was released in early May 2023 (Tania contributed to that). In wrapping up, Tania shared what she believes is a good way to measure if you are doing well enough with your data product, especially in regards to data quality. Look at your SLOs (service level objectives) and SLIs (service level indicators) - are you hitting those regularly? Then maybe you can focus more on new feature development. But if not, you might need more/better monitoring/observability. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of July 9, 2023 31:02

۲ years پیش31:02

31:02

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#237 Zhamak's Corner 25 - We Don't Have to Jerk the Wheel - Making Smaller Correction Decisions to Get to Our Data Destination 20:31

۲ years پیش20:31

20:31

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Sponsored by NextData , Zhamak's company that is helping ease data product creation. For more great content from Zhamak, check out her book on data mesh , a book she collaborated on , her LinkedIn , and her Twitter . Takeaways: We should be thinking about how we can get out of the batch mode into the streaming mode. Yes, technologically but also think about How can we get to making decisions based on smaller amounts of data more frequently - both automated systems like AI but also for our people. Instead of making adjustments or decisions based on big batches of data, we can make smaller course corrections. "Data mesh is about building responsibility into data and the quality of the data you share and being explicit about that quality." Make the cost of mistakes that much smaller by creating smaller decisions that add up to the bigger decisions - it's not one giant leap, it's many steps that can avoid more hazards as you come across them. "Make decisions at the speed of the market" is crucial to being nimble, being able to react to opportunities or new challenges. To do that, we need to put data in the hands of those closest to the market, the domains. Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Data Mesh Radio episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#236 Driving Buy-in For Decomposing the Monolith; and Then Actually Doing It - Interview w/ Brenda Contreras 1:10:44

۲ years پیش1:10:44

1:10:44

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding . Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Brenda's LinkedIn: https://www.linkedin.com/in/brenda-contreras-9649a47/ In this episode, Scott interviewed Brenda Contreras, VP of Engineering and Architecture at Self Financial. Some key takeaways/thoughts from Brenda's point of view: "Iterate small and sell your solutions on a practical level." It's kind of funny how often people in tech try to skip the communication. If you really align on communication and understanding, your business partners are far more likely to empower you to drive business value for them through engineering and data work. ?Controversial?: As an engineering/data leader, don't dictate: set the vision, explain the vision to business partners, but try to let your technical team leverage patterns that will work for them instead of only your favorite way. Similarly, make sure your team understands which aspects of target outcomes drive value and why. They might have an approach you didn't expect but if they aren't focused on the key aspects of the outcome, even amazing feats of engineering won't create value if it's not tied to business needs. Fail fast is very important to doing microservices right. How can we learn to adopt it in data and AI? "We need we need to be … able to experiment more, we need to be more flexible" to really drive to business value quicker and easier. Before you start to decompose anything, it's crucial to understand what you already have. That can sound a bit obvious but if you start trying to do the work before understanding the 'before' picture, getting to a good 'after' picture is going to be very hard. You have to understand the landscape of your systems but it's equally important to understand how those systems power the actual business. Not just services or products, but the business. What matters and why? Talk with and carefully listen to your business partners to find those. It's crucial to align with your business counterparts on big picture visions and then what aspect of the big picture you will address and get clear on why and when. This can feel obvious but it's often overlooked. Over-communicate, share your vision often and far and wide. Your business line leaders don't care about the how of tech - most of the time at least. They care how can you support them in tackling their goals and challenges. Make your communication about changes focused on that. They don't care how the sausage is made. It's absolutely valid, even in a data mesh or microservices journey, to have a "sustain bucket". Things don't need to change for the sake of change. 'Legacy' doesn't have to mean bad/a headache. When you start to talk about breaking down the monolith internally, it's important to sell people on the reality: managing smaller systems is easier. Having one group that owns a set of systems - and that it's clearly defined - makes it much easier to find the information you need. Microservices isn't easy but it does make things much easier to manage at the micro level. Save talking deep tech or data for the people doing the tech or data work - when you talk to business partners, abstract that for them. They don't care about the particulars of your system for sending emails - they care that the emails are reliably sent and tied to the business aspect. When you first start moving towards microservices, look to start setting up your domains around your products. Then you can start to break your systems down into microservices from there. You should think about data mesh the same - don't start building the data products without knowing the exact owner and why they own it. Don't try to toss out the old in favor of the new. Look to net new on your new platform or approach. Just because something has been around for a while doesn't mean it isn't valuable as is. Look for where changes will make a big positive impact. Be very ambitious in your vision for where you could go. But take it at a reasonable pace - don't be in so much of a rush that you create more challenges than you address. You can sell the value of experimentation at the experiment level. It's hard to just get an overarching experimentation budget but constantly showing the value of fast experimentation will make it easier to get the time to do more experimentation in the future. Brenda started with a little about her history, mostly on the app development and software engineering side before moving to Self where she's added data/analytics to her architectural focus. She learned a lot working in the office of the CTO at Charles Schwab about how to decompose a monolith and not become overly rigid around the philosophy of microservices at the same time - how do you still have shared services? How do split into product domains? How do you make sure everything plays nicely together internally and for the customer? That learning about how to share data well on the operational plane is serving Brenda well in general at Self. When she first arrived, creating an inventory of what already existed and how it worked together was crucial to start decomposing appropriately. What were the systems yes but especially how they powered the actual business. What really matters and why? What are the key strategic initiatives and how can you positively impact them by improving systems? A 99% improvement in latency on something that drives no change to how people use systems is wasted if impressive work. It's okay for things to live in a "sustain bucket" as you focus on key business drivers. And you can only really find those key drivers by listening to your business partners. When Brenda started to consider where to start with decomposing the operational systems at Self, she first looked to where they might "run out of runway". What were the systems that were likely to cause trouble driven by growth? Because you don't want to have to tell your business partners they have to stop growth 😅 A key aspect of decomposing a monolith is getting buy-in. Brenda had a lot of success with showing people the benefits of smaller systems that are easier to manage. Deploys are easier. Maintaining a smaller database is easier. It's easier to figure out who owns what - if someone needs information, they know who to go to. Etc. And extracting the information and then explaining your plan for change should be done in the language of the business. They don't care if you are switching your backend to Cassandra, they care about what impact changes will have on the business and what changes for them. Brenda recommends that when you start to look at decomposing a monolith into microservices - so on the application side - you should start by breaking into domains first. Without clear owners and sets of microservices, you start to split things off without a clear forward path. Once you have the domains identified, you can start to move necessary capabilities into your microservices. Differentiating baby and bathwater - what should get tossed out and what should be kept - is crucial in Brenda's mind when you are doing any kind of transformation. There are existing solutions that you don't need to migrate immediately because they're fine as is. Again, don't look to make changes for the sake of changing something. Look to net new use cases as you start to decompose and eventually those legacy use cases will get reimagined and replaced. It's very easy to focus on the wrong things in Brenda's experience - instead, focus on the conversations with business partners and work backwards towards what really matters. Then the work follows from that. E.g. in an acquisition, integrating systems and/or migrating the acquired company to your systems seems to be a major focus instead of integrating what will drive value and keeping things again in that "sustain bucket". Brenda's secret sauce to aligning with business partners comes back again to communication and understanding the big picture of the organization, how their part of the organization fits in, and what are their priorities. Essentially, it's taking the vision for where we want to go and breaking that into manageable projects to move forward on. Always be communicating priorities and timelines and working with people to realign at the small and large scale vision level. And, a biggie is to _really_ listen to your business partners and C-level execs when they talk challenges. "We need to be … able to experiment more, we need to be more flexible," Brenda said relative to figuring out how to drive more business value around data. The fast fail approach in software is crucial to getting to business value fast - how can we adopt that in data? Partly, it's again going back to communicating well with your business leaders through good storytelling. She said, "iterate small and sell … your solutions on a practical level." In wrapping up, Brenda again emphasized the importance of good communication and collaboration with your business partners. It's so easy in data and tech in general to let that drop to focus elsewhere. But when you are all at least reading from the same book, even if there is some disagreement in strategy, people can at least make informed decisions. Don't try to skip the communication with your business partners! Learn more about Data Mesh Understanding: https://datameshunderstanding.com/about Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of July 2, 2023 12:56

۲ years پیش12:56

12:56

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#235 Decom-mesh-ioning - Appropriately Decommissioning Existing Platforms in Your Data Mesh Journey - Mesh Musings 50 15:06

۲ years پیش15:06

15:06

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ So here are the summation points of this episode: Your legacy data platforms probably aren't going anywhere anytime soon. Those can have long lives but look to shut down unused capabilities Your new projects, where applicable, should be data mesh data products. But a large percent aren't going to be that at the start. Figure out some incentives for people to push their new data products to your mesh platform early if possible You should not look to lift and shift data projects/assets/products unless it is truly easy for all users - producers and consumers - lift and shift sounds great but it doesn't work well in all but the rarest of cases Be prepared for people to get concerned about having to migrate early - communicate strongly that you aren't forcing migrations of existing data work Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#234 Doing Data Work That Matters: Perspective From a Line of Business Head - Interview w/ Iryna Arzner 1:03:07

۲ years پیش1:03:07

1:03:07

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Iryna's LinkedIn: https://www.linkedin.com/in/irinakukleva/ Mobey Forum: https://mobeyforum.org/ In this episode, Scott interviewed Iryna Arzner, Head of Group Customer Growth, Retail Banking at Raiffeisen Bank International (RBI). To be clear, she was only representing her own views on the episode. Scott note: I mostly use the phrase line of business or LOB instead of domain in this write up but they are mostly interchangeable. Some key takeaways/thoughts from Iryna's point of view: As a line of business head, data has value but only in so far as they can use it. If it's not aligned to a use case or business need, data work can be more of a distraction than a benefit. It can be very interesting for a line of business owner to know how much their data is worth to other parts of the organization - that could drive funding for additional data work inside their LOB or even more funding than that because the LOB is core to driving business value at the organizational level. "You cannot be successful in your data strategy if there are no business leaders that understand the value of the data and are very much determined to uncover this value." Scott note: couldn't put it better A good way to get your business leaders more data fluent is to very closely pair with them. Sitting side-by-side on a project will up their fluency far better than any training course ever could. "How do we get these data insights that we actually need to fuel the business strategy?" It's crucial to understand the LOB business strategy and focus data work around that. Start from the business needs and work to the data, not the other way around. Scott note: PREACH! {For many senior business execs, they've operated on gut and limited data for literally decades. You need to partner with them to show why data makes their decisioning better/quicker instead of that the data will now make the calls. Data is a tool for them to be better, not their AI robot replacement :) You can't force a senior exec into leaning in on data. Start from their actual business goals and work backwards towards what could be better or what can't they do unless they have better data. It's easy to say your data work should have business value but actually get specific around driving to how your data work supports the business strategy - corporate and line of business strategy. Scott note: if it doesn't have business value in some respect, why do it?! Far too often, especially in advanced areas like AI, data people/teams want to do really cutting edge projects. But if they aren't attached to business priorities - or even if they are but just looking at the business challenge in a way that doesn't align to business leaders - it will likely be wasted effort. ?Controversial?: Please, please, please stop building platforms for the sake of platforms. LOB leaders don't care about what cool tech you use, they care about capabilities that drive business value. ?Controversial?: If your LOB execs aren't willing to share their strategies and use cases necessary to develop planned data work, you might have to resort to bringing things to them you believe are aligned and hope they bite. Scott note: if you can skip these execs at first, they'll come around if others are driving more value through data Related but very different, talk to LOB leaders about if they are interested in those potential insights that aren't directly tied to current business strategy/challenges. Some (e.g. Iryna) like to be opportunistic and have the cognitive load for potentially interesting if tangential insights. The key is to ask. !Controversial!: The "time is past" where business leaders can get away with not understanding tech or data. Business leaders need to understand the necessary data capabilities to drive to their use cases. They can't just wait for insights to be dropped in their laps. Data people need to help business leaders understand the value of a platform approach. Just because something can be addressed 'tomorrow' doesn't mean that it's the right call. What is sustainable and scalable? It's crucial to communicate that. ?Controversial?: It's okay to focus your data work on the pioneers, the data advanced domains. Yes, they might get even further ahead of the pack but they will be easier to work with. Scott note: this is actually a pretty big debate in data mesh as it can lead to building a platform only for advanced teams. Really closely partner between the business and data teams on use cases. If you requirements dump and the data team does work divorced from the business context, it will rarely drive good value. Part of delivering a good product or service is giving your customer something relevant to them. In data, that seems to be missing from lots of work - what is actually relevant to someone's needs and wants? There is a tendency by many data people to defer to the data first. Instead, think about leveraging your subject matter experts more to help define what data you need or what reasonable hypotheses to test :) Iryna started out with a bit about her background and then jumped right into it. She said "it takes two to tango" relative to the business side and the data side collaborating. She understands the power and value good data work can drive for her line of business so she and her team are often pairing with the data team to figure out how to drive value. For Iryna, a data strategy cannot be successful unless there are at least some business leaders that are leaning in - that understand the value of data and are also determined to actually capture that value. Specifically in her area, Iryna gave an example that while in the past, banking was often done via relationship with a banker - so insights about customers could essentially be managed in the head of the banker - as people move to mostly digital, there needs to be a lot more data insights to be able to offer them the best services to their needs at the right time. Starting from the business strategy and the use cases and working backwards to what data is necessary to support those use cases is crucial in Iryna's view. It's proving to be the case over and over in data mesh that starting from the use case is key to getting value out of the data work because you need to know what would actually drive business decisions and actions and then drive into the fine details. But it's easy to get lost in the data work AND it's just as easy to get lost in the business strategy - you need to actually get specific about the data products necessary to support use cases. Iryna discussed that most senior execs have been operating much more on gut than on data for years to decades - you aren't going to change that overnight and you need to partner with them to make their lives easier via data if you want them to lean in. And the best way to do that is to partner on their business challenges, looking for opportunities or challenges they can't address right now without better data. And don't start with the biggest challenges first, think small and quick to prove out value as you build to bigger and better. A common if understandable failure point Iryna mentioned was data people's desire to do really challenging work - fairly often irrespective of if that really matters from a business priority standpoint. Or even if some AI work is aligned to a key business objective or challenge, if it isn't aligned to the way the business leader thinks about the business, it often won't leveraged because they have to change their mental model of their business just to "get" it. So again, start from the conversation with the business leaders and work backwards to significantly increase your chances of your data work driving large-scale business value. Iryna mentioned she's "allergic" to platforms built for the sake of building a platform - 'it's what everyone is doing, so we should too!' Or wanting to use fancy tech instead of focusing on capabilities. Those are red flags that the data team aren't aligned to business value or learning from past mistakes - past projects didn't fail because the platform just wasn't cool enough, it was because the platform didn't make it easy to do what was needed! Many execs in Iryna's view won't want to engage much on data that isn't tied to their strategic initiatives/challenges. But, there are still many like her that are happy to be opportunistic. Actually have the conversations and ask the leaders where they come down on that. Some want people laser focused on what they are trying to do but you can build interesting and highly valuable partnerships with the flexible and opportunistic ones like Iryna. "This is the past," is something Iryna said about business leaders having no clue about technology or data. Leaders need to understand at least the basics and upskill to be better able to leverage the new data capabilities from new tools being developed all the time. The data team will be needed to identify - "in a nice way" - business leader skill gaps and help them close those gaps. And it's important for business leaders to understand the benefit of platform and product thinking in data. Just because a challenge can be 'solved' tomorrow doesn't mean it will stay solved! Sustainability and scalability of solutions in data are crucial. Iryna believes it's good to at least start your data initiatives working with the "front-runners and pioneers". That's because then you will save time and hassle - they are already bought in and capable. Seeing those front-runners drive lots of value with data will lead others to want to follow. Scott note: there is a lot of controversy around this topic - will you only build for teams that are already advanced? How far do you go with this strategy? As to where lots of data projects go wrong, as past guests have also noted, Iryna pointed to the requirements dump pattern - the business stakeholders and data team meet, information is exchanged and then the data team goes off and works on the use case and comes back a month or two later and it's not really what the business side wanted and it doesn't deliver the value. Instead, as Ghada Richani also pointed to, the business and data side need to pair closely throughout the process as the use case and needs evolve and as incremental value is created. The data team can actually get a big boost to happiness from seeing the actual business value get created from their work. Make this whole process mutually tied and mutually beneficial. When thinking about products and good customer interactions, Iryna says it's important to bring your customer something that is actually relevant to them at that point in time. We should look to do the same in data. And that should honestly be easier in data because your customer can - and should - literally tell you what they care about and want. And you can prototype together instead of having to deliver a final product - you need to communicate that but it's more of a partnership than just a customer relationship. Start to work together on the vision of what success looks like. Iryna gave a great recent example of when the data isn't going to necessarily give you the best information. They are looking at a specific financial product and how to sell it better based on certain life event triggers. So, instead of making guesses and random hypotheses, why not ask the people on the ground for what they look for, what are their hypotheses as to why they are selling when and what are the best triggers to use to sell the financial product? This is how data and business processes can build off each other - you don't have to treat your data as if it should tell you the information - extract context from subject matter experts! While she might not be the most typical line of business head in a data mesh implementation - she's more altruistic than what we're often hearing - Iryna and her team are simply excited to understand what value their data has for the rest of the organization. That might even lead to insights that get shared back to her team for additional use cases. But proving out the value of her domain's data is also likely to lead to more funding for their data work so it should have a positive impact all the same. In wrapping up, Iryna urged listeners to tie their data work more closely to the business strategy - yes at the corporate level but even the line of business objectives. Every bit of your work should be tied somehow to driving business value and you should have visibility to that. Yes, platform work can have that but you should understand how that platform work actually drives value. And also, you should sit side-by-side with your business leaders in your data work - that will up their data fluency far more than any formal training program ever could. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of June 25, 2023 18:58

۲ years پیش18:58

18:58

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#233 Panel: A Head Data Architect's View of Data Mesh - Led by Khanh Chau w/ Balvinder Khurana, Yushin Son, and Carlos Saona 1:06:17

۲ years پیش1:06:17

1:06:17

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . #233 Panel: A Head Data Architect's View of Data Mesh - Led by Khanh Chau w/ Balvinder Khurana, Yushin Son, and Carlos Saona Khanh's LinkedIn: https://www.linkedin.com/in/khanhnchau/ Balvinder's LinkedIn: https://www.linkedin.com/in/balvinder-khurana/ Yushin's LinkedIn: https://www.linkedin.com/in/yushin-son-30362b1/ Carlos' LinkedIn: https://www.linkedin.com/in/carlos-saona-vazquez/ In this episode, guest host Khanh Chau, Director of Cloud Data Architecture at Grainger (guest of episode #44) facilitated a discussion with Balvinder Khurana, Technical Principal and Global Data Community Lead at Thoughtworks (guest of episode #135), Carlos Saona, Chief Architect at eDreams ODIGEO (guest of episode #150), and Yushin Son, Chief Architect of Data Platform & Data Products Engineering at JPMorgan Chase. As per usual, all guests were only reflecting their own views. The topic for this panel was an architect's view of data mesh, especially from an architecture lead standpoint. There are many challenges architects face in data mesh, managing the micro level minutiae, down to the data product output and input port decisions but balance that with crucial high-level decisions. Balancing the near-term and long-term vision and roadmap/North Star. Scott note: I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views individually. Scott's Top Takeaways: Balvinder said, "…data mesh expects a lot more out of architects, there would be a lot of people and process and operation management that you will have to understand." It's important to understand that architects aren't just building out the systems but the entire org capability to do decentralized data well. It's a lot of responsibility but also a lot of interesting new challenges. As keeps coming up in many episodes, doing decentralized data/data mesh doesn't mean everything is decentralized. That's what federated means in data mesh: building core building blocks that make the work far easier but not trying to chase into every corner on every use case - that's not scalable/repeatable. Creating ways for people to collaborate and interoperate easily. Doing the glue work so people can focus on the specific value add of the use case/data product. It's absolutely crucial to understand that data mesh is not a complete vision just yet. If you are expecting to pick it up and simply run with it like it's a playbook, you're in for a bad time. The tooling isn't exactly there yet to do this easily and even if it were, we are still in the early days of learning the patterns to do it well. It's like microservices in 2013 more than 2023. Every team will interpret data mesh differently - and many will interpret it in a way that lets them do what they want most :) Be prepared to step in to prevent people building everything themselves and also be prepared for teams that still expect a central data team to own everything. Communication and balance will be key. Multiple years into your journey 'we're figuring it out' will still be a common refrain. Don't make flippant decisions but you'll learn and improve your processes so leave yourself maneuvering room but don't worry about getting things perfect. Optimize for learning and iterating to better, especially on the culture side. Be prepared to compromise. You probably won't be able to decentralize major core systems - e.g. SAP ERP - immediately or maybe ever. While a perfect setup might be what you want, driving to value sooner is where to focus. Be practical and prepared for 'better but not good yet' kind of outcomes as you implement, especially early. It's easy, especially early in your journey, to get overly focused on one domain or use case. That can be at the architecture level, the ways of working level, all sorts of governance aspects, etc. You MUST keep a balance between the micro - e.g. down to the use case or data product level - and the macro of can this decision scale and serve the broader organization. Your organization is not a greenfield. Be prepared to not look exactly like any other orgs. You'll have lots of constraints. Frustrating yes, but every single implementation is messy behind the scenes. Everyone is trying to figure it out and cutting some corners and/or making missteps. Give yourself a break as you learn and iterate towards value. Other Important Takeaways (many touch on similar points from different aspects): We just plain don't really have data mesh best practices yet. They are starting emerge in certain aspects but if you aren't ready for some ambiguity and experimentation, you aren't ready to do data mesh. Every organization will find different balances between centralization and decentralization for certain aspects of a data mesh journey. It's okay if your balance looks quite a bit different from others'. Your balance between centralization and decentralization for many aspects of your implementation will shift over time. Constantly be measuring and assessing if things are 'good enough' and be prepared to change. But that also gives you the freedom to experiment :) Pre data mesh, it was hard to conceive of doing decentralized data similar to how microservices works on the operational plane. It seemed it would cause chaos and data silos. But if done well, you can give teams significant autonomy and still have a larger internal data ecosystem where data from across the organization plays nicely and can be more easily used. A centralized platform offering means more complexities than just one team owning a massive data lake setup - e.g. managing backwards and forwards compatibility across many domains - but also has lots of advantages. A biggie can be that with teams owning their own infra, there is no resource contention - especially important in financial services around month-end processing. The old paradigm of central data ownership and control prevent fast releases so you don't really have any autonomous work by the producing domains. And you can't keep up with things changing - they all go through a backlog that means the time between something happening in the real world and the change to the table in the data lake can be weeks to months. That just doesn't work for the speed of change in the real world. It's crucial to understand that data mesh is about building and using new muscles, ones around decentralization. Much like training, you will probably strain or pull some muscles. That's normal and to be expected. No one gets it perfect the first time around. Building out the interoperability at the tooling level - not even the standards to interoperate the data but the tooling to make it possible for teams to combine data across data products or source data from upstream data products - is going to be harder than you expect. Not even policy/security governance, just the way data flows independently throughout the mesh rather than one giant pipeline per use case. Early in your journey, you probably need to find domains that are excited to give data mesh a try. Rather than dragging a domain along, look for a true partner. Some degree of failure is inevitable in a data mesh journey. But the cost of failure - and our ability to learn and iterate from 'failure' to 'not failure'/value - is so drastically different in data mesh. Fast failure, while possibly scary, is a crucial concept to embrace. Quickly learning and iterating is key. Once you've built up the buy-in and excitement for data mesh, your job might become hype management :D People are used to data projects being built as fast as possible but with not as much focus on sustainability - data-as-a-product - or interoperability. Make sure to keep initial expectations manageable and share why doing it right is better than doing it right now. When building out initial expectations, maintaining flexibility for breaking changes is very helpful. You don't want to make breaking changes carelessly but as you learn how to do data mesh, it's key to be able to scrap something that you find won't work rather than support a non-scalable platform for years to come. Right now, you'll probably have to build a lot more tooling than you'd want, including some custom glue between services. Hopefully that gets far better but right now, there is still a lot of building to do if you want to do data mesh well. If you weren't part of the microservices revolution and evolution, really study what worked and didn't. There are many lessons learned that you don't have to relearn :) Yushin said, "Architects are not wizards. So we make a lot of choices and some of those choices will be wrong. And we need to experiment. And we need to pivot quickly once we find out that something's not going to work." That flexibility is necessary. If you don't have any flexibility to experiment and change, your org is probably not ready for data mesh. It's still very difficult to abstract away the necessary capabilities for managing data from the tooling. It's very easy to get overly tied to specific tools. That is costly and difficult to find specialists for many tools and becomes a blocker to scaling. Look to build good abstractions where possible. Tools are hopefully coming soon. Your room for experimentation is much broader, especially at the architecture/platform level, early in your journey. You probably don't only want to go with more 'legacy' tools, it's okay to try some bets on more 'cutting edge' offerings. There's balance of course but you can be a bit adventurous if you manage risk from that experimentation well. Really focus on enabling self-service for the domain teams - there are certain aspects of governance that should be enforced automatically by the platform so teams don't have to reinvent everything and then it all becomes a mess because everyone is handling things differently. And it's okay to delay tackling use cases because you aren't ready to meet all their needs just yet. It's going to take far longer than you'd like to sunset existing platforms and the data assets/data products using them. Don't press for lift and shift. Potentially look to the Strangler Fig Pattern as you replace existing use cases. And REALLY assure people you aren't going to decommission existing data platforms overly quickly. Look to prevent copies of data that don't have a very clear reason to exist. If you can always get the data, do you need to persist it? There's storage costs yes, but a bigger cost might be the headaches around governance - including bad actor access risk - of keeping track of all those copies… As Carlos recommended, look to push the cost of that governance onto teams that want to keep copies and see if they still want to make copies of data :) Potentially look to leverage tools that use open formats so you don't have to do custom integration around tools. Scott personal note: this is where I think data mesh can actually push the biggest industry changes. Imagine no more crazy proprietary formats and tools just work together! You can drive data mesh participation buy-in from consumers by showing the general friction and pain points of your existing data setup. Almost every team has a lot of pain points. Can you really trust upstream data right now or do you want a contract that provides you a lot of assurances? Cost will be a major factor in a lot of your architectural choices. Would that things were relatively equivalent but pricing across vendors is often extremely different. Be prepared to use more tools than you might want. Different teams will have different capability levels regarding data. There is often going to be some needs that fall back to a centralized team, whether that is the platform team or another shared central team. In an ideal world, it would all be decentralized ownership but sometimes the cost of creating a full data function inside a domain isn't worth it. That's okay because that team shouldn't be a bottleneck if it's only handling a few teams but be careful 😅 Right now, a real value differentiation of data mesh is the ability to scale with nimbleness and agility while managing data quality. But for most organizations, at least right now, their data mesh implementation isn't super cost optimized or performant. Cost optimizations are typically second order concerns as you start out but will become bigger and bigger topics. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#232 It's About the Value, Not the Data - Effectively Partnering With the Business - Interview w/ Aaron Wilkerson 1:12:11

۲ years پیش1:12:11

1:12:11

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Aaron's LinkedIn: https://www.linkedin.com/in/aaron-wilkerson-81bb21a/ In this episode, Scott interviewed Aaron Wilkerson, Senior Manager of Data Strategy and Governance at Carhartt. To be clear, he was only representing his own views on the episode. Apologies for the lawn work sounds around the middle of the episode :) Before we jump in, this episode contains a lot of really good framing on how data leaders can actually partner with business people to drive to what matters for them. How do you extract what matters to the organization and to each specific business partner? And then how do you tie the data work to that? So while this episode is not heavy on data mesh specifics, it's really important to really considering the business partner's point of view and how to work with them to drive value for the organization. Some key takeaways/thoughts from Aaron's point of view: ?Controversial?: Aaron (and Scott) laid out a challenge for data leaders: have a conversation this week with a stakeholder and never mention data. We get too wrapped up in the data instead of listening and understanding stakeholder business challenges. When thinking data strategy, you should first think business strategy. At the end of the day, it all comes down to how data can support the business in its objectives, not about doing data work for the sake of data work. What are the key business goals and target outcomes? Business people very rarely care about the how of data, the sausage making. Don't try to communicate to them about the how, focus on the what and the why. Really drive towards what are they trying to accomplish and work backwards towards what data work you can do to support them. When working with business partners, yes you can try to teach them to be more data driven. But most of the conversation should still be in the language of the business. Data work itself doesn't drive value, how it supports business partner value creation is crucial. Make sure you understand what matters and then communicate how you will help them. Dig into what business people are measured on, their KPIs, to best figure out how to support them with data work. How others are measuring their success is a very important indicator. ?Controversial?: Data leaders really need to understand and take to heart that data just doesn't play that big a role in the day-to-day machinations for most business people. It can play a big role in supporting their work, it's just never going to be a top-of-mind focus for most. ?Controversial?: Coming to business people with the data first instead of starting from their business challenges and goals first - you are essentially asking them to do the work of figuring out if you've brought them something of value. Start from asking them what they'd value and work towards creating that. Focus on the data work supporting your business partners where they're at - not where you wish they were - to move forward. Speak with them and _listen_ to their challenges. Then find ways to bring data in to support and improve their work-streams. When tying data work to the business strategy, while many people might have the same target outcomes, they all have different responsibilities. Really dig into what aspect the person you are working with needs to happen to best support them. Scott note: really important nuance to consider. When talking to an exec, be concise. "… here's the challenge we're facing, here's the outcome we want to get to, and here's next steps. And then are you all aligned to us?" Try to focus your data work on taking away work from your stakeholders instead of adding more things to their plate. Just because data might drive incremental benefit, it might not be worth it. Think return on investment, not just return. Data people have to work with business people to ask the question of what would actually get them to act. A 10% improvement on one aspect of their work probably won't get buy in. Find what would get them excited to move forward and focus on supporting those initiatives with data work. By starting from the problems people have, their pain points, your business partners will be bought in that any insights you find are likely valuable. Otherwise, if it is an incremental opportunity they hadn't considered, you need to read them in much more on the details of why the problem matters before even getting to the insight. That's again more work for them. You will probably have to build a track record of quick wins before business partners look to rely on you for bigger and bigger projects. And that's okay, you can get to know their general problem areas well through the quick wins. While data people often want to share everything they find, it can be a double-edged sword. Just letting people know about something can be okay but if you identify a new challenge or opportunity that isn't closely aligned with their current focus areas, you are asking them to do more work to identify if it should be a new focus. Think carefully about whether and how to bring those up. To be able to partner well with people down the road, first make sure you "stabilize, build the trust, and get everyone kind of on the same page around … what we need to do to fix [any] leaks." Lay that groundwork and start to build off a good foundation before trying to do large data projects. Learn to say no and to prioritize. Don't stretch yourself too thin or you won't deliver as well as you could and you will likely not build the trust necessary to really drive the value you want. Communicating what you're doing - really marketing your work and success stories - is crucial in data leadership. If all you are focused on is doing the work, you won't build the momentum and have people coming to you to partner on work. Focus on learning the language of the business. Yes, it would be great if everyone were data fluent but the main value drivers for most companies happen at the business level. RACI matrices are a very useful and important tool. Scott note: this comes up A LOT - get specific on responsibilities and ownership. Overcommunicate on both aspects. It's amazing how important making your business partners feel seen and heard is, not even 'fixing' their pain points. People are far more likely to work with people that take the time to understand them. It sounds simple - and it kind of is - but it's crucial to laying the groundwork for a good partnership. ?Controversial?: When focusing on being a data leader, when you aren't in a conversation with a fellow data person, you need to lean in to talking in the language of the business. Too many data leaders are still entrenched in the ways of doing data instead of supporting the business via data work. Aaron started with a bit about his background and how he's been upping his game around better partnering with the business and focusing more on the strategy aspects - data and non-data strategy. He was hitting a career ceiling from focusing so much on the data work itself and communicating by showing people the data. Since he started digging deeper into data strategy, everything points to the business strategy, so the last few years have been about getting smarter on aligning the data strategy to the business strategy and doing data work to support key business objectives. What are the target business outcomes and how can data / data work support those? For Aaron, when he looks at companies' business strategies, it's pretty rare to even see the word data featured. Maybe it's to become data-driven but it's certainly not 'build a data warehouse' or anything at the tactical level. It's drive revenue, reduce costs, etc. So a key approach is doing the logic gap work - what do they care about and then think how data might support that. At the end of the day, they care that they get help on their key initiatives, not whether you built a data warehouse or leveraged some new paradigm or tool to drive that help. To best partner with business people, Aaron recommends digging into their incentivization. Not just their goals but what are the measures of their success, their KPIs? How can you support them in doing a better job at what is most important to their role? Understanding what drives them will help you best understand how data work can support them, which in turn should support the business strategy. And it's crucial to really take to heart the fact that data just isn't that big of a part of the day-to-day work of most people inside the business. We want data to make that work more impactful but trying to get everyone to be a citizen data scientist or whatever is just not going to work or drive good business value. Aaron talked about two different ways of bringing data to business people - starting from a data first approach or starting from the business pain points and working to the data. If you come with the data first approach, where you've done some analysis and ask them if this is of value, you are asking them to do more work to evaluate if it's useful to them. That's probably not going to win you many friends. That's the tail wagging the dog. The business people care about certain things, start from listening to them and then working to support them via data. You want the data to make their lives easier, not to add more work to evaluate what you've found and for them to figure out if it even matters to their strategic objectives and KPIs. In a lot of the same vein but from a different angle, it's really important to meet your business partners where they are with data. Look at their work-streams and their goals/challenges and find ways for data work to better support what they are doing instead of trying to reinvent the wheel or push them in an entirely new direction. The more friction to change, the more likely even potentially very valuable insights won't drive value because it won't drive actual action. Aaron made a really good point about while the organization may have target outcomes, each person working to drive to those outcome has different responsibilities - and measures of success - and it's important to figure out what aspect they own and how you can help them specifically with data. And be concise! He said a good way to think about it is in 3 slides to get your point across: "… here's the challenge we're facing, here's the outcome we want to get to, and here's next steps. And then are you all aligned to us?" Try to focus on making sure you are reducing workload for people in most instances rather than again making them do more work to get to value. Find ways to make it easier for them to focus on more and more valuable work and automate the important but rote stuff. Understanding business partners' goals is only one aspect of partnering well according to Aaron. You have to understand the magnitude of an improvement that would get them to move. There is friction to change so a 10% improvement is often not going to be enticing enough. Think about Aaron's example of a rival cable company offering you a $2/month discount to switch. And you also need to understand how would you communicate opportunities to your specific business partners to spur them to action. How do we market our data-driven insights and improvements? While we would probably all like to make major impacts with data, Aaron has seen it's more likely you need to put together a track record of small quick wins to gain the trust of many business partners. Once you've done that, they will be more ready to partner with you on bigger projects. And while you get to those quick wins, you can more easily understand what drives them and where there pain points lie so you are better prepared for a bigger project when they bring it to you. Aaron talked about the difference between bringing up new challenges you've found and planting seeds for something that might be important down the road. Planting seeds - letting people know a few tidbits of information along the way - is great to lay the groundwork for future work together. But most people will _not_ thank you for bringing a new challenge to their attention if it's not a major unknown issue or opportunity. They are focused on what they are most aligned to, they have to do more work to assess if the new opportunity or challenge you brought to them is worth their time. So consider carefully whether to bring something like that up and if yes, how. Again, try not to give people more work unless it's of significant value to them. To get to a place where you can work on high-impact, large data work, Aaron recommends making sure you build from a solid core. Look to stabilize what data work is already occurring, look for and address value leaks, and build trust. Then you will get to a better partner position. And look to stay away from constantly trying to generate ideas instead of listening to your business partners' ideas. They know their business better than you do and will almost certainly have better intuition about where to focus. Aaron also believes it's really important to prioritize and learn to say no. Don't stretch yourself too thin. Again, focus on stabilizing and then start to work with people on future state but don't add too much to your plate. Saying no is an incredibly powerful tool. Another powerful tool Aaron recommends is marketing your work internally. Many data people fall into the trap of trying to focus simply on doing good work and not sharing what they are doing widely enough. A lot of data people got in to data to focus on doing interesting work instead of heavy communication work :) But it's important for data leaders to constantly be sharing what's going on and just communicating in general. To understand where to focus your work, you need to understand people's needs. And you need people aligned with what you are doing and why. Talk to them about what are the top challenges, what you are doing to address them, etc. Eventually, you can increase the data fluency level of your business partners to a degree where they can help you make strategic decisions too but that only comes with constant communication. On the topic of data fluency, Aaron believes it's probably better in most organizations for the data people to learn the business language and what drives the business rather than trying to teach a lot of deep data concepts to hundreds or thousands of business people. The business processes are what drive value - yes, data can significantly improve the business processes but without understanding which ones to focus on and how to improve them, does the data work really have much value? Aaron, as many past guests have also noted, really recommends leveraging RACI matrices - responsible, accountable, consulted, and informed. Get really specific on who is responsible for what - get explicit and over-communicate to make sure there aren't misunderstandings or gaps. While it gets said a lot, making people feel seen and heard ends up having a large impact on partnering in Aaron's experience. "You're more likely to work with folks who you feel understand you versus people who were talking about something … you don't at all care about." If you learn people's pain points and talk to them, even if you can't address them right now, they are far more likely to lean into the conversation and future partnership. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of June 18, 2023 36:18

۲ years پیش36:18

36:18

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#231 Zhamak's Corner 24 - Can We Change Mistakes to "Happy Little Accidents" in Data? 24:19

۲ years پیش24:19

24:19

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Sponsored by NextData , Zhamak's company that is helping ease data product creation. For more great content from Zhamak, check out her book on data mesh , a book she collaborated on , her LinkedIn , and her Twitter . Takeaways: We can do better in data than what we did learning decentralization in services/the operational side: "We have to level up. We can't repeat the past mistakes. Let's not be silly and fool ourselves just because we have a schema, now we have an amazing system." The services world has learned good ways of communicating between producers and consumers. We should look to learn more from them and look to adapt then adopt what works well. We need to change our approach to measuring and reflecting on past decisions - we might have made a decision based on not great information, does that mean the decision was bad simply because it didn't work out? Probably not, but as Ari Gold said in Entourage "There are no asterisks in life, only scoreboards…" Can we really get to a place where we allow those asterisks? Zhamak believes we can adopt many software development practices across data - that's pretty key to data mesh - but one area people seem to be skipping over are things like decision records - what were you thinking when you made a past decision, what did you know and what were your hypotheses. It's easy to judge results but it's better to judge the judgment :) Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Data Mesh Radio episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#230 Getting Real About Data Product Management in Data Mesh - Interview w/ Frannie Helforoush 1:15:48

۲ years پیش1:15:48

1:15:48

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Frannie's LinkedIn: https://www.linkedin.com/in/frannie-farnaz-h-a7a11014/ Post on the Product Trio concept by Teresa Torres: https://www.producttalk.org/2021/05/product-trio/ In this episode, Scott interviewed Frannie Helforoush, Technical Product Manager/Data Product Manager at RBC Global Asset Management. To be clear, she was only representing her own views on the episode. Some key takeaways/thoughts from Frannie's point of view: There is a difference between the product mindset and creating/maintaining data products but both are very important to exchanging value through data. We should be looking to apply the product mindset to all aspects of our data work, not just how it applies to data products specifically. To do data product management well, you should look to software product management practices and recontextualize those to data. Many map well but some don't. It's not a copy paste, think through what should be applied to data differently. The data product manager needs to serve as the bridge between data producers and consumers, making sure consumer requirements are satisfied much like with a software product manager where they are the bridge between software engineering and software users. If product management is the intersection of business, tech, and user experience (UX), how should we think about that for data product management? Tech and business are easy but there isn't a user interface (UI) so think of UX in terms of data fluency, access, and documentation. Relatedly, documentation around data products is more important than traditional software products because there isn't really a tangible UI for data products. Data discovery platforms are very useful for data consumers because they are not only for discovering that data products exist but they are also great for discovering information about the data products - they provide a good initial understanding of data products before digging deeper. In data mesh, you shouldn't think of one single data platform to rule over everything. Users care about capabilities and their workflows/UX, not about exactly how things are stitched together on the back-end. Don't have 100 platforms but don't also create a monolithic beast you can't evolve. ?Controversial?: Product discovery - the act of discovering what product(s) you should build - is crucial and often overlooked in the data product space. Many organizations are waiting for requests instead of data producers discovering what data products users might want. While we should contextualize product management functions and needs to data product management, oftentimes the terminology gets all jumbled up. Be very explicit in defining terms to other stakeholders to prevent as much confusion as possible - but there will still be confusion :) To help guide you in creating the right data product(s) to serve a use case, you really need to work closely with consumers - what are the day-to-day business problems of the use case? Then you can start to work backwards towards creating the right data products to serve the use case. Make sure to dig into who will actually use a data product and how will it drive value for them. There are far too many business processes that are wildly inefficient or even unnecessary simply because people aren't asking - or asking again - those questions. Don't let your data product be inefficient, be intentional in asking. Leverage the 'Product Trio' concept for data product management. While you won't have a design lead, find someone that can still help to optimize the data user experience (DUX) instead of the UI as you would with software. Again that UX is data fluency, access, and documentation. Use product management frameworks to deep dive into potential data users' actual challenges. Get specific so you can find points of leverage to improve the process. To actually drive value, you have to drill down. A 'fast fail' culture/approach is crucial to getting the best value from data. Producers and consumers can quickly test what works and iterate towards better instead of making data work all or nothing. Failure in data has to be something we are okay with. Potentially the hardest aspect of data product management is defining success and the metrics to measure your success. Don't feel bad if you are struggling here too :) It's okay to start with less than great metrics and iterate towards better. Enabling self-serve for consumers of data products should fall on the platform rather than on the data product owner. If the platform isn't robust enough, you shouldn't make every data product manager build out those capabilities to make up for it. Frannie started off with a bit about her background and how that has helped her adopt the data as a product mindset, applying product thinking, when looking at creating and maintaining data products. A lot of it is about mapping traditional software product management to data product management as many things translate pretty well 1:1 but lot of things _don't_. And knowing the difference is a crucial point of leverage. The data product isn't the point, it's merely a way to exchange information to support a use case. In software product management, the product manager "serves as a bridge between software engineering and software users". In data product management, the product manager should serve as the bridge between data producers and consumers to ensure consumer requirements are satisfied. Martin Eriksson's idea that product management is the intersection of tech, business, and user experience is foundational to Frannie. However, what actually is a user experience (UX) relative to a data product? Much of the user interface (UI) is actually owned by the platform itself rather than the data product so that makes UX a little more nebulous. She believes we should break down the UX into three categories: data fluency, access, and documentation. Access is the easiest because it's about making sure people can request access and can grant access easily - yes, automatic access is great too but that's not always possible. Data fluency is about how well the data product communicates its information to target audiences. Can people understand what the data product contains, why it exists, how things are organized, etc.? Lastly, unfortunately it seems documentation is harder and also more important in data product management than software product management. In software there is a tangible user interface but that's not really feasible for data products so the documentation is the way to guide people to understanding. For Frannie, data discovery platforms are really valuable for consumers to understand what data products exist. And once they find data products that might be of interest, the platform allows them to learn more about the data product. Companies should make it easy to find the most useful information about a data product all in one place, e.g. the lineage, the purpose of the data product, sample queries, sample data, etc. as well as request access. Scott note: this is a partly a trapped metadata challenge. Many tools trap their metadata so it's hard to bring in things like SLA observability directly into the data discovery platform. There is a big difference between data discovery and product discovery - or in this case data product discovery - in Frannie's terminology. In software product management, product discovery is about finding the needs users have and looking to fill that with product{s). Basically, discovering what products or features of products should exist. This is often overlooked in data product management because so much of the genesis for data products is defined use cases where the consumers come to the producers with their use case at least somewhat well scoped and understood. But data product discovery is a very useful practice to at least investigate if not lean more heavily into. Scott note: I've been saying this for literally over a year - communicate and spark the art of the possible with data consumers! Frannie talked more about the process of discovering the right data product(s) to create. You want to dig in deep with the potential users. They might have ideas about what data they want but the most important aspect is what is the day-to-day challenge or opportunity they are looking to address? What is the actual business process and how do they want it improved? That will help tell you what data product(s) you need to create for the use case and how you need to shape it to address the business process they want to make better. Essentially, consumers are experts in what they are trying to fix, don't rely on them to be experts in what data to share and how, that's for the producers to own. The 'Product Trio' concept from Teresa Torres has really been helpful to Frannie and team around data product management. While it is coming from the product management space where the trio is the product manager, the technical lead, and the product designer, it needs to be altered for data products since that designer role doesn't make as much sense when there isn't a direct UI. So you still have a product manager, a tech lead of some kind, and then someone that is more aligned to the user, maybe a data scientist or a data architect, to help ensure a positive data user experience (DUX). Again, Frannie breaks that DUX down into 3 components: data fluency, access, and documentation. Frannie recommends looking at product management frameworks for leading product discovery sessions, e.g. the value proposition framework. Frameworks can help you get to actual tangible information about business processes and their challenges. Think about concrete ways to qualify and potentially even quantify aspects of the business challenges. Get pretty specific instead of a laundry list of potential use cases, drill into one or two and figure out how to help. Scott note: this seems to be a recurrent issue in data - lack of specificity around use cases and needs. It's part of why requirements gathering fails. Do not skimp on this process :D Failing fast is a concept Frannie believes we need to adopt - and maybe adapt - in data. That's what prototyping needs to be better able to quickly get to value. Failure has historically been somewhat of a catastrophe in data because so many things have been essentially all or nothing with huge budgets. But with things like data mesh and in general with collaborative iterative data work between producers and consumers, failure doesn't mean forever fail, it leads us to places to improve. So while we might need new terminology around fast fail - data people seem to hate the concept of failing at all - it's a practice that will help us quickly iterate towards better solutions and more value. A fast fail culture with smaller blast radii can really highlight how people will use a data product instead of the pre-planning phase where you try to come to how both sides _think_ a consumer will use a data product. Frannie shared what probably all data product managers feel but few hear: it's very difficult - and probably a bit squishy - to define success for a data product and then also define the metrics to measure success. E.g. just pointing to number of users doesn't necessarily equate to value. User satisfaction is a useful measure but obviously it doesn't cover every aspect. How well are you doing at hitting your SLAs? It's okay to start with non-perfect metrics as long as they are directionally valuable. Scott note: I recommend listening to episode #95 with Dave Colls on fitness functions if you want to dig deeper into success metric tracking. Quick tidbit: It's important to build out your data consumer self-serve capabilities to give consumers a better feeling of being able to actually get necessary access. And it's even more important to make sure that is handled by the platform and not the individual data products. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of June 11, 2023 16:54

۲ years پیش16:54

16:54

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#229 Making 'Agile' Work in Data - Mesh Musings 49 13:18

۲ years پیش13:18

13:18

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts (most interviews from #32 on) here Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. My overall point here is that why do so many folks in data hate Agile? Because in data it's so rarely done well basically. Things are done because 'that is the way they are supposed to be done' instead of 'because this will make our teams happier and more efficient'. And quite honestly, Agile isn't for every organization. The spirit of Agile probably should be for every organization so maybe go read the Agile manifesto but in data, the one size fits all approaches are obviously breaking more and more. So work with your teams and talk about what you want to achieve and collaborate with them to get there. Yes, easier said than done but I believe in you. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him at community at datameshlearning.com or on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#228 Keeping Your Eyes on the Prize: The Data Value Chain - Interview w/ Tina Albrecht 1:15:23

۲ years پیش1:15:23

1:15:23

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Tina's LinkedIn: https://www.linkedin.com/in/christina-albrecht-69a6833a/ In this episode, Scott interviewed Tina Albrecht, Lead Coach for Data-Driven Transformation at Exxeta. Some key takeaways/thoughts from Tina's point of view: Always start from your value chain - how do you actually generate value from data work? Any process or other tool you attempt to leverage that isn't focused on improving your data value chains will likely be ineffective in generating value. And why do data work if not to generate value? Your two most likely reasons you are losing value in your value chain are lack of clear ownership/responsibility and bottlenecks. Look to regularly assess both. When measuring if things are good enough, generally the DORA KPIs are good measures of data process maturity. But also look at two aspects: A) how happy are people (from customers, decision takers up to the team) with the current process. Satisfaction is a great measuring stick because it is highly correlated to effectiveness. And B) how much effectiveness is lost to bottlenecks and constraints. The two ways most data mesh implementations seem to be going wrong are a misinterpretation of Team Topologies and lack of teams owning responsibilities. On the first, there are often breakdowns in how teams collaborate together and on the second, we need the platform team to own enabling domains but the domains keep trying to push work back to the central platform team. It's important to regularly assess if aspects of your data transformation are good enough for now. But it's also very important - and easy to lose sight of - how are your teams feeling during the transformation. If you significantly improve capabilities but everyone is miserable, will they leverage those new capabilities? Scott note: interesting perspective and it factors into the rate of transformative change: constant seismic shifts are probably not great for morale. In data mesh, you should consider rotating your data engineers between domains so you can have a larger group of data capable people that truly understand domain challenges. Scott note: we have to still make sure we give embedded data engineers a career trajectory or they will leave. Many domains will truly not understand what data ownership really entails. Work with them and embed highly data fluent people that can raise the domain's capability and understanding of data ownership. !Controversial!: Just having a simply community or guild for your data engineers and having them become part of the domain is not enough. There should be a central organizational home for data engineers and you should embed them into domains but the central team should handle their career trajectory. Team Topologies is very helpful in data mesh but if you don't understand your actual value chain, it won't work well. You need to start from how you generate value. When considering if data mesh is right for an organization, if the organization doesn't have a clear vision of what would be better if they had better data capabilities, that's a major red flag. Do they understand how data can drive value for the organization specifically? Would data mesh align to their business strategy? Clarity - getting deep into how things actually interconnect - is crucial to doing a large scale transformation like data mesh right. Too often, things are left up to vague interpretation and balls get dropped. Drive to specifics. ?Controversial?: Overcommunicate. Set literal mandatory information exchange mechanisms - e.g. at the person-to-person level in workshops - between domains. Get explicit that domains need to be regularly exchanging information, not just the 1s and 0s of data. Scott note: this is an emerging pattern from teams who have cleaned up problem data mesh implementations. It seems like it really works. ?Controversial?: When trying to change your data processes, confusion is actually a positive sign. It indicates people are listening and trying to figure it out. If everyone thinks they understand all the changes without any confusion, they probably have wrong implicit assumptions. Scott note: human communication is silly sometimes but this is an excellent point. While it can be hard to exactly measure the value of data, you should always be asking what is the purpose of the data or data work. What value does this provide, why are we doing this work? Tina started with some of the big questions you will need to consistently ask yourself in a data transformation: is this good enough? Is this good enough for now? What is changing and how is it changing? What are the milestones we want to hit and, looking back, that we have hit? And often overlooked: how is/was the team feeling during the transformation? Those questions can help you narrow in on how well your transformation is going - and let's face it, data transformation doesn't stop. A good three layer system to think about when breaking this conversation down is: the surface layer of what happened in the past and where things are now; one layer deeper is where could they go if they had stronger data capabilities; and the bottom layer being okay, but how can we get there? When actually measuring whether a current solution is "good enough", Tina recommends two measures or questions to consider. The first is: how happy are people with the current process? Happiness is a decent measure of effectiveness. If people are happy, working hard to improve the process probably doesn't make financial sense, the return isn't there. The other aspect is to think about how much effectiveness of the process is lost to constraints and bottlenecks. This analysis will give you a good insight into the process and provide the perspective on if it is currently good enough. Tina talked about two ways most data mesh implementations seem to be going wrong. The first is how teams are interacting with each other: the team and domain setup. There are often breakdowns in how teams collaborate together so things that should be explicitly owned and bridged between the teams just keep getting dropped. The other is data ownership where domains won't take data ownership and/or don't understand data ownership. We need the platform team to own enabling domains to leverage the platform but the domains keep trying to push work back to the central platform team or not owning data well enough so the central team has to step in to help. If that happens, the platform team can't get the necessary platform work done to do data mesh and they become that centralized bottleneck again. In data mesh, Tina believes there is still a lack of understanding by domains of what data ownership means, what they are actually supposed to be responsible for. You can help domains better understand data ownership by making sure they have necessary embedded data engineering talent within the domain to actually be capable of owning data as more domain members learn how to own data. And you need strong governance capabilities to help teams understand how to interoperate data between domains easily. Tina talked about with one client, they are rotating embedded data engineers between domains - and the central platform team - so the domains become more data capable and you have a wider knowledge base about data across the organization as the data engineers share that knowledge with each other and the organization. And just having a simple community or guild for the data engineers wasn't enough, they needed to go with the embedded model instead. That central hub and a central team managing the data engineers' careers has been very important to keeping people happy. Similarly, Tina talked about how while Team Topologies is a great tool for organizing your teams in data mesh, it's only a tool. If you don't understand your value chain, if you don't really focus on how you create value via data, it won't save your data mesh implementation. Start from value first. Is data mesh right for your organization? When Tina is assessing that question for clients, a great starting question is simply what changes, what value would come from doing data mesh? If there isn't a clear vision as to what would be better and how that would drive clear value - and a large amount of value too, data mesh is not a light undertaking - then will data mesh really align with the business strategy and drive value? Scott note: I think these questions are REALLY crucial to answer. If you don't know what will change if you do this well and how that ties to business value, you shouldn't do it :) For Tina, in general in data work, there are two big areas where value is blocked or lost along the value chain. As she mentioned earlier, lack of clear ownership and responsibility is a big one. People understand how value is generated but it's unclear who owns what and so major needs along the value chain aren't met - basically, no one thinks they own crucial aspects and so they don't get done. The other is again simply bottlenecks - where are things blocked or where are dots not connected? Once you identify issues in either aspect, you should have your areas to target for change to drive more value related to those specific processes. A key aspect of transformation for Tina is deep clarity. There are so many things that are changing, who owns what and what outcomes do they own? What is actually being done and why? There needs to be strong governance leadership that lays out many aspects rather than leaving things to chance. It doesn't have to be heavy-handed governance but ensuring things will work together - and that someone _owns_ making them work together well - is the best way to ensure a successful data transformation, data mesh or otherwise. And you have to stay on top of things, it's not a one-and-done kind of transformation. Intentionality around communication is also crucial to successful data transformation for Tina. Overcommunication is a virtue. Data isn't about the 1s and 0s, it's about sharing information. So you need to be explicit in setting expectations and creating mechanisms for people to exchange information, especially across domains. Have regularly scheduled workshops to actually get people exchanging crucial context - it's like a good relationship, you need to continue to work on your communication. Many think about that information exchange at the actual 1s and 0s level but we need people to exchange information with each other constantly too. Otherwise there are too many incorrect implicit assumptions and again, balls get dropped and value is needlessly lost. Tina made a somewhat comical but very true point: when you are doing a large-scale change to how you do data work, if there isn't anyone saying they are confused, it's probably a bad sign. Because large-scale change is difficult and inherently change will be at least a bit confusing for most so if no one is speaking up, they probably have some bad implicit assumptions you need to address but you don't know what they are. At least with confusion, you can drill into where they don't get it. Lean into confusion because it creates the perfect situation to actually exchange context and drive people to the same page. While it is incredibly difficult to provide an exact value of data and data work - it will be valued differently by different people - Tina still asks people what is the purpose of doing the work. Why do we care about this data? That will tell us the general value of it if not a specific dollar figure. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of June 4, 2023 15:54

۲ years پیش15:54

15:54

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#227 Panel: Creating a Data Mesh Platform (1st Iteration) - Led by Paolo Platter w/ Manisha Jain, Jean-Georges Perrin (JGP), and Max Schultze 58:46

۲ years پیش58:46

58:46

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Paolo's LinkedIn: https://www.linkedin.com/in/paoloplatter/ Paolo's Medium (multiple data mesh articles): https://medium.com/@p-platter Agile Lab's website: https://www.agilelab.it/ Manisha's LinkedIn: https://www.linkedin.com/in/evermanisha/ 'A streamlined developer experience in Data Mesh' blog post by Manisha: https://www.thoughtworks.com/insights/blog/data-strategy/dev-experience-data-mesh-platform 'A streamlined developer experience in Data Mesh (Pt. two)' blog post by Manisha: https://www.thoughtworks.com/insights/blog/data-strategy/dev-experience-data-mesh-product 'Data Mesh Accelerate Workshop' blog post by Thoughtworks: https://martinfowler.com/articles/data-mesh-accelerate-workshop.html Max's LinkedIn: https://www.linkedin.com/in/max-schultze/ Max's Data Mesh Learning meetup presentation: https://www.youtube.com/watch?v=QwtTdP2wKFo (he has many more on YouTube! https://www.youtube.com/results?search_query=max+schultze+data+mesh ) Data Mesh in Practice ebook he co-authored (Starburst info gated): https://www.starburst.io/info/data-mesh-in-practice-ebook/ JGP's LinkedIn: https://www.linkedin.com/in/jgperrin/ JGP's 'Data Mesh for All Ages' book: https://jgp.ai/2023/01/20/data-mesh-for-all-ages/ JGP's website (lots of data mesh content): https://jgp.ai/ JGP's Blog Post 'The next generation of Data Platforms is the Data Mesh': https://medium.com/paypal-tech/the-next-generation-of-data-platforms-is-the-data-mesh-b7df4b825522 In this episode, guest host Paolo Platter, CTO and Co-Founder of Agile Lab (guest of episode #3) facilitated a discussion with Manisha Jain, Data Engineer at Thoughtworks (guest of episode #220), Jean George Perrin (AKA JGP), Intelligence Platform Lead at PayPal (guest of episode #130), and Max Schultze, Associate Director of Data Engineering at HelloFresh (guest of episode #21). As per usual, all guests were only reflecting their own views. Scott note: I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views individually. Scott's Top Takeaways: As with every aspect of data mesh, you will need to focus your data platform work on what best drives value - doing so incrementally and both in the short and long-term. Look to start at common friction points in your org's data work, not just data mesh work, e.g. automated provisioning. That will give users a good first experience that is incrementally better to existing data work, potentially driving buy-in. It's _really_ crucial to understand user personas and necessary capabilities for each persona. Who is using it, why are they using it, and what are their points of friction? That experience plane will become incredibly important as you move along but it probably shouldn't be your initial focus. The word platform, especially in data mesh, can be thought of as a plural. Don't focus on only creating a 'single platform to rule them all', focus on delivering scalable capabilities that reduce friction and support scalability and reliability/trustability of data work by producers. Users don't care how it fits, create a system that can evolve and scale! You must treat your platform as a product itself - think about how well it satisfies user needs, how good is the user experience, etc. But that also means your platform has a lifecycle where you add and eventually prune features. From the start, apply product thinking to your platform :) Prioritization of platform capabilities will probably always be a challenge once you have more than a few domains using the platform. Balance being reactive and proactive with the need to generalize solutions to fit many use cases. Again, look to product management for advice on how to manage your roadmap and priorities. REALLY think about how to handle breaking changes. They are likely inevitable and that's okay but you need to focus strongly on communication and limiting the impact / providing a gentle migration path. Don't break without really needing to and do so sparingly. Data integration, especially in a highly regulated industry, is going to be a challenge you will face relatively early on. It's likely not going to be easy, be prepared for that. You need to understand where there is friction in the data product creation/management process - that's where your platform should focus. That might feel obvious but it's not how data platforms have been built - basically, look to automate unnecessary friction first, even if that means focusing on things like templates and blueprints instead of cool tech. Other Important Takeaways (many touch on similar points from different aspects): There are many ways to potentially get started - probably too many to list. But think about getting to early necessary capabilities that deliver value. It's easy to get bogged down in technical aspects - instead start specifically by asking what creates value quickly. If you aren't ready to build your platform incrementally, you probably aren't ready to do data mesh. You need to be comfortable with demonstrating value and building as you learn and as your needs progress. When you are early in your journey, discoverability/usability is a characteristic many might overlook - you are building data products to support a use case. But if you want to drive buy-in and get incremental users of those data products from additional domains, discoverability might be a very important early capability. What is your adoption leverage point? There's an interesting balance between MVP for the platform and easily enabling MVPs for data products. It's hard to say exactly where the line falls. Scott note: Glovo said in episode #139 that they wish they had focused a bit more on making it easier to launch and initially manage data products over other more advanced capabilities. Consider when to start saying you actually have a platform - that can be a bit of a political statement. Potentially wait until you have started to build out the data experience plane for data producers. If there isn't a tangible way to interact, the platform may exist but users don't really know what it is and what it enables them to do. But the experience plane shouldn't be among the first capabilities you build either. :) Your data platform work needs to focus on capabilities and enabling value delivery. But that value delivery needs to be visible to business users. Basically, find your value leverage points that are visible to users and focus on satisfying those when you can to drive buy-in that you're delivering value. It can be a political game unfortunately. You can't treat domains as if they are the same or, especially early, often even similar. There will be different value drivers/needs and often very different capabilities so you have to make sure your platform has the necessary capabilities to drive value for that domain. And if it doesn't yet or won't in the near future, it might not be time to partner with that domain yet - and that's okay! At the start of your journey, you will probably be focused on building capabilities to serve specific needs - of course, try to build those in generalized ways to serve many needs but still. As you evolve and grow the platform, you need to focus more on your overall suite of capabilities and look for where you have gaps. Don't only wait for capabilities requests - but also don't just build cool things because they are fun to build :) Start to build out data product specifications based on early data products. Those will provide a much easier path for later data products, whether built by new domains or existing domains using the platform. Charging models and who should own what costs are going to play a bigger role than you'd probably like. It's going to be a challenge you need to address at the platform level even while (presumably) others decide who should actually own the costs. Do not focus too much on building capabilities to _launch_ a data product. The launch is just a single day in the life of a product. You need to build capabilities that also help domains maintain and evolve their data products. Should the bounded context of a data product drive all the way down to the resourcing? Basically should there be zero shared resources between data products. I know this one is a bit controversial as it's a key aspect of Zhamak's vision but people worry about costs. If you believe in that complete separation, how do we make sure there aren't overlaps on computational resources and storage? It won't be super easy because it's not how most data services have been built historically. Many data engineers will still be thinking waterfall. Be prepared. Agile can work very well if it's done well - but it's not done all that well in data all that often… SAFe seems to be less hated in doing Agile for data - it's pretty widely panned in software - but it _might_ provide a bridge to doing Agile in data. Something to investigate but definitely not advice/guidance. Paolo essentially said (heavily paraphrased/edited for flow) 'We really need to pay attention, to optimize the ratio between the value that we bring with new features against the satisfaction of various users. This drives prioritization across phases, whether the focus of that phase is on adoption, productivity, satisfaction, etc. What is the outcome we want to achieve in each phase? Then we use that to prioritize features.' Every mesh platform build out will be different, especially depending on what capabilities are most valued in your organization. For some, that might be security/privacy governance re regulators. For others, it might be usability. Find you value leverage points and also find what the people with the purse strings will find the most valuable. Measuring the return on investment, even just the success, of a data mesh platform will be tough but it's important to start early and _start_ to get your arms around it. Productivity gains are a pretty easy measurement to consider as you get more sophisticated. Look to measurements of return on investment in software to get some ideas but be comfortable that things will always be a bit squishy. If you are building your platform as a product, products focus far more on user experience (UX) than most aspects of data infrastructure have historically. Make sure you really understand the value drivers - if users don't adopt your platform, nothing else you do with it really matters :) Producers being willing to own their data - and then understanding what that ownership really means - will probably be one of your biggest challenges. Hard to tackle that at the platform level but look to support those efforts as best as you can. Data producers and consumers still don't generally talk to each other in many organizations. Look to potentially put more mechanisms in to foster communication. Scott note: see my mesh musing - episode #188 - where I believe all consumers MUST register their use case with producers. You will have to decide things like do you isolate compute and storage resources at the domain or data product level. Data producers don't care, make that invisible to them. They key aspect to focus on is to enable and not block - basically, give domains the agency to get their work done but find ways to reduce their friction and obfuscate decisions like that from them. They are important decisions but irrelevant to users. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#226 Learnings From Implementing Data Mesh at a Large Healthcare Company - Interview w/ Mike Alvarez 1:19:55

۲ years پیش1:19:55

1:19:55

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Mike's LinkedIn: https://www.linkedin.com/in/2mikealvarez/ In this episode, Scott interviewed Mike Alvarez, Former VP of Digital Services leading the data mesh implementation at a large healthcare distribution company. He's now working on his own startup. Some key takeaways/thoughts from Mike's point of view: Lean in to the new value-creating possibilities that can come from empowering thousands of your colleagues to leverage data. As an industry, we have to learn to do data work in an incremental fashion. It's not been the norm and it can break people's perception of data work but it's crucial to get where we want to go. You can drive data mesh buy-in from domains by showing them the freedom they will have. Autonomy, empowerment, going at their own speed, etc. can get many to lean in. Advice to past data mesh self: Early in your journey, you can share your vision until the cows come home and people will say they understand - and probably think they understand - but it's incredibly easy to get misaligned. Really focus on what you are trying to achieve. What are the target outcomes? Similarly, it will be harder than you expect to drive buy-in. Many people say that but it's still going to probably be harder than you expect after hearing that :) We need to move away from old approaches to data for large companies because the sheer scale of initiatives ends up creating bloat and risk factors unto themselves. Small and nimble gives us much quicker time to value delivery and builds to much greater outcomes. Shadow IT develops to try to move at the speed of business for domains. But it's rarely scalable or robust enough to even support the domain in the long-run and it certainly isn't built to integrate well with the rest of the organization. Try not to hold past shadow IT decisions against domains. Most teams - especially pre data mesh - don't truly understand the data they are ingesting. It's on consumers to get more information but if the producers aren't helping them, teams will ingest what they can even if they don't fully understand it. Data they don't understand well drives value but could be driving so much more value. Start from the problem first: what am I trying to solve? Do I need a new approach or can I use something I already have? Don't reinvent the wheel but we might just have to reinvent doing data at scale a la data mesh. Collect stories of past attempts internally with negative outcomes. What were the common reasons, the common patterns for things failing or not delivering expected value? They are useful for perspective and to drive buy-in. Treating data as a product makes more and more sense the deeper you dig into it. But just doing data as a product can't survive on its own as an approach to doing data. When trying to share information about data mesh, it's not like everyone will instantaneously understand or be on board. It will likely take a while in most organizations to build up the momentum to even consider starting on a data mesh journey. Have patience. Data mesh really enables teams closest to the customer, closest to the day-to-day business, to drive more value through data. It allows them to react much more quickly as the world evolves and focus on the problems of the customer. ?Controversial?: The operating model change with data mesh is what drives the real value. And lots of domains can get bought in that they get to own their own destiny but be empowered to manage their data like a product instead in non-scalable and quickly deteriorating ways. Scott note: I think we do need better tech to fully leverage the potential value of data mesh but right now, I agree that most of the value is driven by operating model changes. A shared vision of what you are trying to achieve is important. It lets people rally around something and start to build a community, which is crucial to delivering on a data mesh approach. ?Controversial?: Don't try to force your domains, your lines of business to leverage your centralized tooling and comply with optional governance (there is non-optional governance of course). In exchange for those who leverage central tooling, pay them back via automating away toil where possible. Community is about give-get. Be a good member of the community. The three crucial dimensions of a product: viability, feasibility, and desirability. When adopting product thinking, you should think about does your product satisfy all three factors. You need to communicate when something isn't feasible. Too often in data, people have just said no instead of 'no, and here is why…' Let people in on your thinking and prioritization process around what work to do when. Good product management skills are necessary to understand data as a product - to transition us from creating and sharing data sets to sharing high value information exchanged via a data product. You need to delve into and understand the domain to figure out what would be most useful to share via a data product. !Controversial!: It might be time to completely let go of the concept of "single source of truth." We've been chasing it in data for so long but the cost/benefit is starting to look it doesn't make sense. What are we trying to achieve - perfect data or a strong understanding of the world and how it's changing? Scott note: strongly agree and so does Zhamak. New, more correct information about aspects of the business is not always welcome. Unfortunately, you might have pushback if you attempt to tackle a problem that changes people's view of their business. So choose use cases, especially early, well :) Mike started off with the general need for large companies to change their approach to analytics at scale. We've been doing a lot of the same things for the last 30 years and they aren't quick enough to respond to changing business needs - 6+ months and $1M+ to get to your first query just doesn't make sense anymore - did it ever? And we can do better now. The business side of companies shouldn't have to wait for data and see the world change well before a solution is delivered. We need to move at the speed of business. Regarding shadow IT, despite leading a central data/IT organization, Mike doesn't hold it against domains. The lines of business can't deal with the bottlenecks of going through a central team and try to build things themselves. However, it's rarely all that scalable and certainly isn't built with sharing to the rest of the organization in mind. Shadow IT just isn't built with a product mindset so it becomes brittle and dilapidated quickly. So the central team is a bottleneck but the decentralized approaches don't scale. Add in the teams generally not really truly understanding the data they are ingesting or even often producing and it's a recipe for data underperforming expectations. Of course, this is what Zhamak identified and why she created data mesh. Mike talked about when considering a new approach to data, he didn't want to do data mesh for the sake of it. What was the problem they wanted to solve? Could an existing approach or platform do what was necessary? What were the organization's past failure modes or times when things didn't meet expectations and what were the common through-lines or patterns? And then he took those past unmet expectations and used them for understanding as well as driving buy-in. The definition of insanity is trying the same thing over and over and expecting different results. So if data projects were constantly not meeting expectations, shouldn't we change the way we approach data? And treating data as a product seemed like a great start, which led to selecting data mesh :) While data mesh can feel like the right call to some immediately, it's not likely to be the universal reaction at any organization. Mike and team spent a number of months driving to how this could work and building up the buy-in and momentum to even start on their data mesh journey. This isn't an overnight approach, you really need to think deeply about how it could work and - back to those potential failure modes - how it could go wrong so you can prevent heading down bad paths as best as possible. But what really drove Mike's interest in data mesh as a possible solution was how it could enable the teams closest to the customer to react to customer and market needs, especially changes in customer demands/wants/challenges. It is about empowering the teams to move at the necessary pace to stay ahead of the competition instead of waiting for a centralized team to give them access to leverage their own data or the data of teams close to them in the organization. For Mike, the value of data mesh isn't about the technology shifts, at least not yet. It's about the operating model shift - giving domains the capabilities and empowerment to handle data. We are trusting them to own their data and giving them the ability to do so in a scalable way. We are giving them the ability to react in a much quicker and more meaningful way. All of these can get people leaning in to doing data mesh. But they don't care whether it's data mesh or any other paradigm. That's where data people need to connect the dots for them, how can this work and what benefit does it have for the domain. And what are the actual changes for them? To get the most out of data mesh, Mike believes you have to have a strong vision of what are you actually trying to achieve. It's not an approach to take on lightly. You need to really think about aligning everyone around that shared vision and build as a community effort. How do you take the principles and new approaches and focus on delivering business value - for their own domain and the broader organization too? Mike believes a big part of doing data mesh is kind of the social contract around enablement and empowerment. Sure, teams can go off in their own direction but if they give up some of their autonomy to stick to the centrally provided tooling - which makes governance far easier -, you need to give them something in return. In their case, Mike and team gave the gift of automating away a lot of the toil work :D On advice to his past data mesh self, Mike talked about early in a mesh journey, people believe they are aligned on vision but they probably aren't. Holding all of data mesh as a concept and then contextualizing it to your specific organization is a massive amount of work and cognitive load. Trying to get someone to fully understand that upfront without seeing the progress, you will almost certainly have some misalignment and misunderstandings. Instead of the specifics, focus on the target outcomes, what are you trying to achieve? If people align on the benefits, you are more likely to gain and retain momentum. And it will take a lot of effort to get most people committed to the vision, just be prepared for that. Mike talked about the three key aspects of a product: viability, feasibility, and desirability. Feasibility it a crucial aspect to consider in data - especially data as a product thinking - because often, something just isn't likely to work for a number of reasons. And when there is the desirability but not feasibility, you really need to communicate why it's not going to happen. With data mesh, there can be a misconception that the switch has been flipped and we can do any data work we can think of - and that we should! But that prioritization process and understanding - and then communicating - what is the current art of the possible is important. Always be communicating about what you are doing when and why. Domain understanding is crucial to really understanding data as a product for that domain in Mike's view. How do we move from trying to serve data sets as if that is the product to creating the information that will be most useful to consumers about the domain in a productized way? And then iterating towards more and more value as you improve the data product or suite of data products representing the domain. Easier said than done of course. Mike asked the provocative question of do we still want to seek the fabled "single source of truth." It can be a bit like the dog chasing its tail - when you catch the tail, then what? Are we trying to perfectly clean data or are we trying to drive value from data? Is the juice worth the squeeze or can we drive better value - and especially nimbleness - by taking a slightly different view? Scott note: Zhamak urges people to consider "the most relevant source of truth" because there are multiple perspectives on the same things that can all have value, you have to decide what is best. Mike warned that some use cases are politically untenable or even toxic. Especially early in your journey, consider will participants actually want to know the information. Yes, in the abstract, we want everyone to be perfectly data driven but humans aren't and won't ever be. Don't ignore that and tackle something that will be more hassle than it's worth. In wrapping up, Mike had two points. The first is learn to work incrementally. That has been somewhat of the antithesis to how data work has historically been done but it's incredibly important. The second point is to really lean into empowerment and the art of the possible. We don't really know what might happen when we empower thousands of our colleagues to be better able to leverage data. Be excited and open to the journey of finding out what value they create. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of May 28, 2023 34:32

۲ years پیش34:32

34:32

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#225 Zhamak's Corner 23 - Driving to Resilient Data Value Now and in the Future 22:29

۲ years پیش22:29

22:29

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Sponsored by NextData , Zhamak's company that is helping ease data product creation. For more great content from Zhamak, check out her book on data mesh , a book she collaborated on , her LinkedIn , and her Twitter . Key Takeaways: Postel's Law: Be conservative in what you do, be liberal in what you accept from others. We can do better in data than what we did learning decentralization in services: "We have to level up. We can't repeat the past mistakes. Let's not be silly and fool ourselves just because we have a schema, now we have an amazing system." The services world has learned good ways of communicating between producers and consumers. We should look to learn more from them and look to adapt then adopt what works well. Zhamak believes we have to learn to prepare our data for future use cases. Scott note: If she means reuse of data being generated for current use cases, most agree. If she means creating data that doesn't currently serve a use case, almost everyone else seems to disagree. Time will tell. More on Postel's Law: https://ardalis.com/postels-law-robustness-principle/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Data Mesh Radio episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#224 Building Out Scalable Automated Access for Data Mesh at Disney Streaming - Interview w/ Himateja Madala 1:06:55

۲ years پیش1:06:55

1:06:55

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Himateja's LinkedIn: https://www.linkedin.com/in/himatejam/ Himateja's AWS ReInvent presentation (link starts at her part): https://youtu.be/y1p0BGsPxvw?t=1991 In this episode, Scott interviewed Himateja Mandala, Senior Data Engineering Manager and Head of the Data Mesh Data Platform at Disney Streaming. To be clear, she was only representing her own views on the episode. Some key takeaways/thoughts from Himateja's point of view: ?Controversial?: Your existing data platform(s) might not be able to serve data mesh well, even with reasonable augmentation - especially if your data platform has become hard to change. You might have to build from scratch. When the data platform's key users aren't part of the centralized team, you need to think about enabling automated capabilities by default, e.g. security the second data lands or easy to leverage and understand monitoring/observability. ?Controversial?: Data products serving different use cases often end up looking relatively different. Is your data product for dashboards and reporting/analytics; is it for serving a recommendation engine or machine learning model; or is it more for internal usage? Be okay with data products not being uniform. Even if your data mesh platform operates outside the traditional paradigms, many data producers - especially data engineers - will still be thinking data pipelines. Be prepared for that, it's an ingrained way of thinking for many. Data contracts are very helpful in defining and maintaining quality. If you set up good observability on your data products, owners can quickly identify when there are quality challenges. When building out your platform, user conversations are crucial. Go and focus on pain points. The coolest capabilities in the world won't lead to good adoption if you aren't addressing real pain/needs. Automation and blueprints are key to scalability in a data mesh data platform. Teams need to be empowered to easily do their work. Don't only focus on creating the tooling to process and share data in the abstract, dig into how teams will share information with each other, how they will communicate. That isn't only exchanging data via data products. Even if you have domains inside your organization that want to share data/information with each other, it is hard to get to a place where consumers can actually trust the data without a lot of explicit enablement of trust at scale. Enabling trust at scale is a key role of the platform. Enabling teams to go at the speed of their business through owning their own infrastructure really drives good buy-in. It might take slightly longer to get something spun up the infrastructure for the first data product but they quickly learn and will often strongly prefer the visibility and control they now have. When requests for new capabilities come to the data platform team, you need to consider how to generalize the capability to be applicable to more use cases if possible. And sometimes the right answer is the platform can't support that one-off need. Centralize your governance capabilities in the platform but federate the decision making. There should be standard approaches to access control to make it easy for people. !Controversial!: Disney Streaming has very strong RBAC (role-based access control) policies to make it very easy to delineate who should have access to what but if you have a certain clearance level for one domain, you have that clearance level for all domains. Scott note: this is about 35min into the interview, it's a really interesting approach. I couldn't see it working for highly regulated industries but it's working very well for them. ?Controversial?: Preventing data from leaving the mesh under any circumstances is an effective risk control - if someone somehow gets access to something they probably shouldn't have access to, the blast radius is quite contained. If you have any data sharing agreements with partners/vendors, make sure to keep their access heavily contained. Create specific spaces - or cloud accounts - with strong rules to prevent them getting improper access to any other parts of your data. Himateja started the conversation with the situation Disney Streaming was in that matched many organizations right now: many data platforms but not one that will really fit with data mesh, even with augmentation. So she and her team decided that because the existing platforms were too hard to change to meet the needs of a data mesh implementation, they'd need to build their data mesh platform from the ground up. When you have new key personas leveraging the data platform, even if those are data engineers embedded into the domains, Himateja recommends rethinking how data work is done. What do people need automated and by default, like security? How do you create monitoring/observability that helps people easily pinpoint issues as they come up? How do you make data accessible by default at the data product and greater mesh level? Etc. In a decentralized, federated data approach, ways of working and needs will be different so dig into what are the actual pain points instead of solving the same pain points of previous implementations. Himateja shared that while people may think data products are pretty similar, they end up relatively different based on use case. Audience also really mattered when trying to figure out what capabilities people required early in the journey - execs were often more focused on data privacy and security and data scientists were focused on data quality. It's hard to focus on the business context at the platform level because many people are used to doing that via request. The data products themselves need to own business context. Data contracts are crucial to maintaining data quality in Himateja's view. While they are certainly helpful to data consumers, they are also very helpful to data producers because - with proper observability - data product owners can quickly identify and address quality issues as they emerge instead of waiting until consumers complain and downstream data is wrong. That proactive alerting and then response helps everyone better trust the data. However, data contracts are still a work in progress because not everything is easy to define in a contract, there are definitely gray areas that are improving but not great yet. Scott note: and that's okay, we can't get everything perfect upfront, we have to iterate towards better :) Himateja then shared a lot about what the data platform team that she leads set out to do at the start of their data mesh journey. One aspect was to create a center of excellence approach, standardizing how data engineering work is done to create data products across the 15+ teams running on the platform now. They did that by starting to drill into pain points and doing lots of listening to potential users. They needed to take a different approach rather than just yet another data platform. Preventing the central data platform team from becoming a central data engineering team was a worry for Himateja: how do you prevent being a bottleneck and empower teams to do what they need to do? Especially at the start of a journey? As many guests have pointed to, automation and blueprints have been crucial. Teams pushed back initially at the thought of managing their own infrastructure but they realized it gave them the ability to move at their own pace - no more waiting in a prioritization queue for necessary infra. Another key milestone was developing tools to make it easy for cross domain communication and data sharing. Domains at Disney Streaming actually wanted to share their data with each other and the data mesh platform/implementation made that possible - it was previously very difficult to trust data but now that quality metrics were clearly defined and tracked, data sharing and usage between domains increased significantly. Prior to doing data mesh, Himateja shared that data engineers in domains had no real visibility into data infrastructure - provisioning timeline or any other aspect. They'd push a ticket and wait for things to happen. But with data mesh, since they own the infrastructure, they can go at their own pace and can understand much more about any delays. So they understand and better control their own timelines, which makes them far happier. And then once they’ve gone through the process of spinning up infrastructure the first time, the next time they can be that much faster. They are better able to move at the speed of their business. Himateja shared about the process of evaluating new platform capabilities requests. As many past guests have noted, you need to establish a process to abstract away the requirements from individual use cases to find a generalized approach. Otherwise, you end up with yet another overburdened platform that you can't evolve. A specific example at Disney Streaming was enabling their Apache Kafka clusters to better communicate across domains which were leveraging individual cloud accounts. Instead of building a solution to share data for each technology they use in the platform, they built a system to better enable sharing across accounts with proper access control and privacy. And sometimes, the answer is that you can't support a unique requirement via the platform - that's okay and often is the right call. At Disney Streaming, Himateja and team implemented a very interesting approach to access control via RBAC (role-based access control). There are a few levels of data usage clearance but if you are at access to PII level or access to financial information level in one domain, it's the same clearance level for all domains. This might not work for heavily regulated industries but it's working very well for them. The work to decide who has access to what data is done ahead of time instead of constant requests. They just think very carefully about each use case and who should have access and why but there isn't a need to manually grant access. And there is of course oversight to see how people are using data to make potential changes. Scott note: this is a really interesting approach and I'd love to hear people's feedback. In wrapping up, Himateja shared how they are strongly limiting their blast radius around sharing data with partners/vendors. They have accounts that are not able to get access to any other accounts where they give those partners/vendors access so there is not a way for them to access data they shouldn't be able to see. It's a simple security pattern but others should consider adopting it in her view. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of May 21, 2023 17:10

۲ years پیش17:10

17:10

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#223 What Does Your Data-Driven Org Look Like - Mesh Musings 48 11:05

۲ years پیش11:05

11:05

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts (most interviews from #32 on) here Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Important points to consider about being a data driven organization: If your execs can't envision what would change if the organization were significantly better at data, you have some work on your hands at understanding their challenges first and then also evangelizing Your domains should be leaning in and understand what being data driven means for their domain specifically. If they aren't you won't be able to really help them move forward. You can't drag a team to being data-driven. Your execs should be aligned on what being data driven means for the organization, especially at the macro level - how does it all fit together instead of highly competent data silos? How do we focus on incremental value delivery via concrete use cases? You should absolutely make sure your data strategy and your vision of the data driven organization ties to the actual business strategy and supports crucial priorities. You need to make sure you have - or can build - a test and learn culture. If not, can you really be data driven? Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him at community at datameshlearning.com or on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#222 "Not All Governance is Created Equal": Data Governance as a Data Mesh Value Lever and Driver - Interview w/ Lynn Noel 1:15:08

۲ years پیش1:15:08

1:15:08

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts (most interviews from #32 on) here Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Lynn's LinkedIn: https://www.linkedin.com/in/lnoel/ DAMA New England: https://damanewengland.org/ Correlation/Causation XKCD: https://xkcd.com/552/ In this episode, Scott interviewed Lynn Noel (pronounced Knoll), Data Governance Lead at AIM Consulting. To be clear, she was only representing her own views on the episode. Some key takeaways/thoughts from Lynn's point of view: There is a strong data governance value proposition in data mesh around making data broadly "safer, better, and easier to use and share." Lean into that value and communicate it widely. ?Controversial?: Many organizations doing data mesh are moving from a very centralized approach and they have certain concerns and approaches they should take. While other organizations are overly decentralized and need to be using different approaches to migrating. Really consider these as two different approaches that land on a similar end approach. You should look to return governance to its original meaning: guiding and steering. In data mesh, governance shouldn't be about control. ?Controversial?: "Not all governance is created equal." There are non-negotiables: infrastructure security and data classification. Don't over-index on sharing data at the expense of crucial governance. Shift your governance left. Shift left the responsibilities - where appropriate - but also shift governance left in your timelines - it needs to be part of the build processes. If you are doing data mesh, you have to understand the existing organization governance structure - whether that is called data governance specifically or not - and align with it. Legal, risk management, IT, etc. Make sure they are aware of your work and give them confidence you will keep things governed well. Every member of a project team on a consulting engagement is obligated to deliver value. That means you need to translate your work to business value. Governance isn't only risk mitigation, it should have incremental value. In data, we too often "confuse the pipes for the water" - meaning we focus too much on the plumbing instead of the resource people want: understandable, usable, trustable data. Consider telling business users you are providing them an ecosystem of capabilities instead of a platform. They can understand that they have their own "habitat" and can customize aspects to meet their own needs but that it all needs to work well together too with other habitats. Federated computational governance is an overly scary term. Break that down for users and what it means for them and why it actually makes their lives easier/better. ?Controversial?: "If you automate something that users aren't expecting to have automated, they will be extremely surprised and disoriented by the platform." Make sure to communicate and ask people what they actually want automated. What you see as friction might be value-add work for them. Communicate! You can plan out a lot of things but they just won't survive "first contact" with the data model and/or the business model. Be prepared to iterate, don't plan too much and be flexible; iteration is great, you are implementing what new you've learned. If your organization is coming to data mesh from a highly decentralized approach, it's very likely there are pockets of data that never were put into the central data infrastructure. It will probably be hard to get people to trust sharing that data and giving up control of it because the previous approach couldn't handle what they were doing. ?Controversial?: Part of data mesh or any transformation initiative, you need people who have ownership over, the responsibility for, organizational change management. Driving buy-in at the personal and organizational level. Scott note: I kind of like this but it's also scary. Who has the responsibility of driving buy-in specifically? Use the phrasing capability instead of feature when talking data platform. What capabilities are you offering to enable your use cases? And your capabilities should be very use-case agnostic. Lynn started with her being the lead data governance person for a data mesh implementation at a client. It's allowed her to have governance be part of the value drivers for all aspects of mesh. She said data governance in mesh is about making data "safer, better, and easier to use and share." The value proposition of doing governance well in data mesh starts to emerge when you think that to really get the most out of data, you need teams to feel like they can rely on data, not just use it. It's crucial in all aspects of data, but especially with data mesh, that governance is an enabler instead of a blocker according to Lynn. As a governance leader, you can't be trying to put up blockers but it's crucial for the team to understand why aspects of governance are helpful and add value. You should be building alongside - especially the building aspect, being part of the work - to keep things moving while also making sure you aren't setting yourself up for trouble later, whether that is quality, regulatory, or otherwise. Lynn said "not all governance is created equal" - it's important to establish where you can and cannot compromise. Infrastructure security and data classification are both non-negotiables. Both wrap in to external threats and internal access control, including to regulatory requirements. Risk management and compliance just cannot be ignored so make them part of the build process and ensure everyone understands why they matter. It's a core of product thinking around data. There are multiple ways to 'shift left' in governance in Lynn's view. The first is shifting governance left in the project timeline - it should be part of every aspect of building your data mesh from the start - platform, data products, etc. You don't layer governance on at the end. We also should be shifting some of the responsibilities left to the developers - while giving them the capabilities and knowledge to handle what they can and come to experts for what they can't. Lynn has had some success with talking to business users about not providing them a platform but an "ecosystem". It lets them envision something on a bit more of a tangible level, that they will build out their own habitat but it plays into a larger ecosystem. So they are comfortable in their own habitat which lets them better deliver value to themselves and others. Of course, your ecosystem can't infringe on other ecosystems so it is a good analogy for how domains all interact. When looking at what to automate as part of your platform, Lynn recommends strong communication with users. You can surprise and disorient them if you automate the wrong things. E.g. there are certain aspects of discovery analysis that look a lot like standard tests but one is incremental work where you want to dive deeper into the data and one is part of daily work without value add from doing manually. Automate the second but not the first :) Lynn brought up that your organizational setup before moving to data mesh will have a very large impact on what you should focus on when in your journey. If you are coming from an overly centralized approach, it will mean that you understand the benefits of integration but might have some challenges with letting go of central control. If you are coming from a very highly decentralized world, you might need to focus much more on alignment because domains may have had freedom to do as they please with little thought for users in other domains. In organizations doing data mesh coming from a very highly centralized approach, Lynn says be prepared for a lot of dark data. Because the officially blessed IT systems were overly rigid, many people were doing work outside of the systems, often in Excel or similar. Getting people to trust you that you'll take good care of their data and they can still maintain access will be hard. There are many fun governance challenges including naming conventions to tackle with data like this. For Lynn and her team, it's been very important to focus specifically on answering what's in it for me from users. Me in the personal sense and me in the organizational and/or domain sense too. And having people whose role involves owning organizational change management is very helpful as part of a product team. There is literally a person or people responsible for getting alignment and driving buy-in. Quick Tidbits: In data, we too often "confuse the pipes for the water" - losing the forest for the trees. We focus too much on the plumbing instead of the resource people want: understandable, usable, trustable data. Focus on the outcomes and business value. It's easy to scare people by using the 'federated computational governance' term. Work with teams to show them where we can automate the past pains of governance so everyone can work on what drives incremental value not manually applying policies. Regarding a data warehouse: "I never saw a plain vanilla implementation survive anybody's business data model." Your plan will not survive first contact with the business model as is. Be prepared to iterate on your plans in data mesh. But now, that doesn't break everything like it did for the enterprise data warehouse. Similarly "no taxonomy survives first contact with the data." You have to build so you can update and iterate, prevent brittleness. Don't use the phrase feature for your platform, use 'capability' instead. Make sure you are building the capabilities to support multiple use-cases and start to prioritize capabilities based on what use cases they can unlock. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him at community at datameshlearning.com or on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of May 14, 2023 13:07

۲ years پیش13:07

13:07

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#221 Panel: Building Your Data Mesh Roadmap - Led by Eric Broda w/ Elizabeth Calloway and Phill Radley 1:02:55

۲ years پیش1:02:55

1:02:55

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Eric's LinkedIn: https://www.linkedin.com/in/ericbroda/ Eric's Medium: https://medium.com/@ericbroda Phill's LinkedIn: https://www.linkedin.com/in/redshirts/ Liz's LinkedIn: https://www.linkedin.com/in/elizabeth-negrotti-calloway/ In this episode, guest host Eric Broda, an Executive Consultant in the financial services space (guest of episode #38) facilitated a discussion with Liz Calloway, a Data Governance expert in Financial Services (guest of episode #92), and Phill Radley, Principal Data & AI Strategy Consultant at Thoughtworks. As per usual, all guests were only reflecting their own views. Scott note: I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views. This will be the standard for panels going forward. Scott's Top Takeaways: It’s important to understand that when you decentralize, different aspects can - and should - move at different paces. Your roadmap needs to account for that but it should also take advantage of that. Domains can go at their own pace, including ones looking to quickly drive towards significant data value. Everyone's roadmap, by their inherent nature, will be unique based to the organization's context. Trying to copy-paste from another organization will end badly. That said, there are some pretty core capabilities that all mesh roadmaps should have. A roadmap should point you towards your target endpoint. Of course with data mesh, there isn't really an exact endpoint - you don't stop evolving, much like with microservices - but the idea is the same: how do you want the organization to operate relative to data production and sharing? Set your North Star to guide you. Your roadmap really should have time built in for "socialization". If you don't, then there is no real extra collaboration or communication between teams and everyone executes on what they expect is needed - you optimize at the micro level, not the macro. It's crucial to human cognition to not only be 'doing' but thinking and relating to other people. That's how you can drive sustainable change management. Be very clear on your value proposition for your data mesh journey but don't try to get your budget and set your value proposition as this long-term delivery of value. Look to the incremental value delivery and find value propositions for incremental work. Basically, deliver value along the way and don't tie your budget and value to something that is a massive delivery 3 years down the road. For data mesh, absolutely be prepared to make compromises on when you plan to deliver capabilities. If you aren't prepared to make compromises, you unequivocally are not ready for data mesh. Focus more on business capabilities over purely underlying technical capabilities in your platform roadmap. The tech is the most tangible part but it's also not the thing that really ends up driving value or being overly important to most users. Governance might actually be the most crucial aspect to your roadmap. If you don't map out your planned capabilities around governance, you don't really know when you'll be able to tackle certain types of use cases. And it's also very easy to leave off crucial governance issues to later that will hurt you as you try to scale like policy as code, data product blueprints, and interoperability standards. Other Important Takeaways (many touch on similar points from different aspects): Try to delay your key architectural decisions until you know more. At some point, you have to make a call but don't try to lock in your architecture right away. However, this will REALLY frustrate many (most?) data engineers because they often care more about the tech than the capabilities. Overall, focus more on what are you trying to do to support and advance the business strategy in your roadmap. Data people want to focus on the data work but it's very easy to diverge from the business strategy. So make sure you are keeping your roadmap aligned to business goals. If you are going to create a roadmap, you should get clear on the definition of your roadmap. What are you actually trying to achieve with your roadmap? What is important to include and what isn't? Don't create a roadmap to create a roadmap - at the start of your process, define the goal around creating the roadmap. A roadmap at the high level is a strong communication tool to drive alignment across many parts of the organization. It is a vision to share and work towards. You will have your top roadmap but you might have roadmaps for aspects of your data mesh like your platform too. The roadmap isn't rigid. The target vision 'destination' is but focus a lot on what is our next 'pit stop', what is our next value delivery instead of trying to plan everything out years in advance. If you're spending too much time to ensure you are 'right', being 'right' might not be actually valuable - you spent too much of your "time, talent, and/or treasure" as Liz said where the return on investment isn't there. Learn to fail fast and iterate to value with smaller bets. Even when it comes to a roadmap, prepare people for one of Zhamak's mantras: think big, start small, move fast. Your roadmap shouldn't be too ambitious and you should roadmap out your organizational changes too. Far too many try to focus purely on the technical and platform aspects. Do not try to escape Conway's Law - look to reflect its impact in your roadmap. Don't focus overly much on exactly where you are going without reflecting on how far you've come. Think about your successes - showcase/celebrate and try to measure them as that will drive further buy-in and momentum. You can highlight the work of the team and inspires others. Make this part of your formal roadmap. You want to think about when you will add your self-service capabilities. You have to make some compromises at first in a data mesh journey, especially around the platform. It can't do everything you want it to on day one or you will have spent far too much time building instead of doing. Be honest about the importance of capabilities and look to automate where you find the most friction. Governance on your roadmap will really set a tone for your journey. Is this about better data processing or about better business outcomes through AI and analytics? Does your definition of minimum viable product for a data product evolve as you mature in your data mesh journey? It's an interesting question to dig deeper into. A simple, compelling value proposition could be 'the work we do to satisfy regulatory compliance should be reused for our analytics and AI. We should drive the cost and complexity down for both by focusing on reuse.' Your roadmap could possibly even include developing a better decisioning process for future work. Focusing on tying business use cases back to data work. Even something like measuring the value of your data work. It won't be great from day one but think about business processes around data work you want to mature. In your platform, potentially err towards building the business capability first and then building out the self-service aspect to take care of that. You need to understand what you actually want to build and an excellent (the best?) way to do that is to get real-world feedback on what are the actual pain-points of putting something into production. It's okay to have either (or both) an org-wide, large-scale transformation roadmap or a much smaller roadmap focused on that incremental value delivery. Both could be a valid approach depending on your organization's situation. Be real with assessing where you are in your data maturation journey, your data governance journey. No one is judging you on that. It's better to be honest to yourself than going for bravado. Again, no one externally is looking at you and especially not at your roadmap :) There are multiple successful and sensible ways to start a data mesh journey. You shouldn't be overly focused on how others are getting going. Yes, learn from them but you can - and should - tweak to your organization. Don't go for easiest, go for most likely to succeed in the short and long run. Yes, easier said than done. Getting your first mesh data product to production is a "momentous occasion" as Eric said. You had to really get a lot of things in place to achieve it. There are a few parallel value delivery tracks in your data mesh journey. There is business value delivery - delivering on the use case. And there is also mesh capability delivery - delivering on the ability to do data mesh. When you first start out, the business value delivery will outstrip the capability delivery but the capability delivery is where you have your long-term value leverage, your ability to scale value delivery. It isn't easy and can sound cheesy, but make sure to build empathy into your roadmap - how do we exchange context with each other and learn what drives value for each other so we can do better work that delivers more business value? Good empathy will lead to stronger communication and collaboration when it's valuable/necessary. If you see something, say something. Over communicate. Think about what others might want to know and let them know, don't only wait for someone to ask. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#220 Building Your Early Mesh Data Platform and Data Product Capabilities - Interview w/ Manisha Jain 1:26:20

۲ years پیش1:26:20

1:26:20

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts (most interviews from #32 on) here Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Manisha's LinkedIn: https://www.linkedin.com/in/evermanisha/ 'A streamlined developer experience in Data Mesh' articles by Manisha: Part 1 - Platform: https://www.thoughtworks.com/insights/blog/data-strategy/dev-experience-data-mesh-platform Part 2 - Product: https://www.thoughtworks.com/insights/blog/data-strategy/dev-experience-data-mesh-product Article on Lean Value Tree: https://rolandbutler.medium.com/what-is-the-lean-value-tree-e90d06328f09 Blog post on mentioned data mesh workshops: https://martinfowler.com/articles/data-mesh-accelerate-workshop.html In this episode, Scott interviewed Manisha Jain, Data Engineer at Thoughtworks. Some key takeaways/thoughts from Manisha's point of view: Manisha's top 3 pieces of early mesh journey advice: A) put together a specification of what a data product is, make it clear. Your definition will evolve/improve but if people don't understand the building blocks, it's going to be hard to build value. B) start to create standardized input and output ports because that is how data products - your units of value - actually exchange value. C) make it easy to discover your important SLOs to create trust; partner with consumers to serve what they need. When working with your first domain, it's crucially important to make sure they have strong data engineering talent - whether that is existing people or someone embedded into the team. That way, someone inside the team building the data product can better interact with the platform team to communicate basic needs at the infrastructure level. This isn't as necessary for later domains - the platform already exists. Both the platform and product teams need to really understand and align on responsibilities and necessary capabilities. That will help you streamline your developer experience, which is crucial to scaling data mesh. People very often confuse data and data product. You should look to get crisp on what a data product means internally. Scott note: we still don't have a simple, on-paper explanation of a data product you can put in front of a team of non-data folks… It's crucial to understand how data mesh can align with how your organization thinks and works. That will make it far easier to drive buy-in. "…only when they're comfortable with that concept … will [it] make sense to go ahead and explore more." The platform team needs to focus on delivering capabilities to domains, not technologies. But they also need to think about mesh-level capabilities, e.g. supervision capabilities. Think about what capabilities are needed when, don't boil the ocean. Maybe certain use cases are too difficult to tackle right now. Scott note: that is the mark of a good org approach, when you can say "we aren't ready yet" and it's okay It's important to remember that the data platform is there to make it easier to create, deploy, and manage/evolve data products. Your data products are the unit of value exchange in data mesh. Use that as a guide when deciding what the platform should offer. ?Controversial?: Boundaries around data products and data product teams are more important than most realize. We really do need teams to be able to act independently and not deal with the hassles of shared infrastructure. Scott note: I know people get worried about cost but time to market of information matters too as well as time spent dealing with untying infrastructure knots by the platform team. Consider doing a series of workshops with a small group closely aligned to each domain to drive understanding and alignment. Don't try to do all the domains at once, each has a different set of needs and capabilities. The Lean Value Tree is an effective method to break down what you are trying to accomplish into actionable pieces of work with explicit assumptions. You state the bets you are making, the hypotheses you are testing, etc. so people are on the same page. When you start working with a domain in data mesh, really drive down to specifics. Don't just identify a use case and maybe the necessary data products. What skills and tooling are necessary to create and maintain those data products? What would a team look like to own those data products? How are you going to put that all together? Doing data mesh, every new domain is different so the onboarding plan for each incremental domain will have to be adjusted. That doesn't mean everything starts from scratch but you need to assess gaps in each domain's capabilities to figure out how best to enable them. To understand what capabilities domains need, the platform team needs to have strong communication and partner with the domain teams to build what they need, what helps them get the job done, not build the coolest platform :) The platform team should focus on addressing three questions in the initial build: a) how do users create value? b) how can we ensure users trust the data? And c) how do you make data products usable and discoverable? Your platform team probably won't recognize what all will be reusable components until they've brought multiple domains on to the platform. And that's okay. ?Controversial?: Most aspects of concepting and then building/deploying a data product are reusable, the data modeling and data transformations are the things that are unique to every data product. Interoperability standards won't happen magically but you can do more custom mapping between data products very early in your journey to get use cases out. Start to look for places to create simple standards used to integrate data products. ?Controversial?: Offering automated modeling or sample data models is a double-edged sword. They can be helpful to get teams to something passable but it's rarely going to create a good data product without more work anyway. Basically, at best it's an outline but shouldn't be published like it's the article. Look to build to the thinnest slice that delivers good value. Don't get ahead of yourself. But don't try to get by ignoring one of the data mesh principles. As you build data products in a domain, you will learn more and more about that domain which will lead to enhancing/evolving the data products you do have and potentially creating new ones as use cases emerge. It's important to stay curious and be prepared to share new insights via data. ?Controversial?: Data people have to really learn to speak in the language of the business. Otherwise, we will continue to talk past each other instead of learning and iterating together towards value. Manisha started the conversation with her thoughts on how to get going with data mesh, on-boarding any domain but especially your first domain. Work with a small team aligned with that domain to find how data mesh can align with how the organization works and thinks - this will be different for every organization. That alignment is crucial to getting people comfortable and driving buy-in. People have to be comfortable with how it will work and what are their responsibilities. As Manisha said, "…only when they're comfortable with that concept … will [it] make sense to go ahead and explore more." According to Manisha, when you are bringing teams up to speed, it's really crucial to get on the same page on what you mean and what you expect from them. They often confuse data and data product for example. The differences can be subtle but are important to understand. As Chris Haas also stated in his episode, they are using the Lean Value Tree method to break down target outcomes into explicit assumptions and more manageable aspects of work. What are the bets you want to make and what are the hypotheses you are testing? Your initial workshop(s) with a domain can also be a lesson in how to deliver value using a data mesh approach and prioritization. Manisha talked about how when working with a domain, you might identify multiple potential use cases. But you need to choose what is a priority to do now and why. This can surface what are the top one or two use cases and also show the domain how to prioritize as use cases continue to emerge in the future. The use case(s) they select to prioritize then directly lead to discovering the data products that needed to support the use case. And then you identify what skills and tooling are needed to actually execute and build and then maintain the necessary data products. Then, you can start to back into what a team working on the necessary data products (and potentially platform) look like. You can use that Lean Value Tree concept to really get specific because far too often in data work, things are left too vague. Scott note: Get specific, get explicit, chase away vagueness - but of course leave LOTS of room for experimentation and iteration as you learn and build. When asked more about workshop dynamics, Manisha shared how they try to keep them from being too heavy on the domain - get a few people, maybe 2-4, who really understand the domain and can represent the business aspects, not just the data and/or technical aspects. Each workshop has its own goal as an outcome but it's important to first align data mesh to organizational goals, the business strategy. Then you can get into data mesh specifics. They call their workshops 1) accelerate, 2) discovery, and 3) inception. Manisha shared some crucial dynamics when working with your first domain that do get easier as you bring on additional domains. In the first domain, it's crucial to really narrow in on understanding and definitions including roles and responsibilities. Data product owner is a new role, what does it actually mean? And there's the initial platform work too. But as you bring on your third, fourth, fifth, etc. domain, there is internal learning to share with the new domains. There is more clarity around what a data product is - they can even see already built data products - and roles/responsibilities. But you will need to definitely do a gap analysis to figure out how to best enable each domain as each domain is unique. So there is a balance - look to maximize reuse of platform, processes, organizational changes, etc. but don't look to force new domains to adhere to exactly how previous domains went through the journey. For Manisha, it's very important for the platform team to think in terms of capabilities. Deliver capabilities, not technology, to the domains. Work with early data product teams closely and focus on what they are trying to do instead of how you want to solve the technical aspects. Focus on specifically what are they trying to achieve? Also, the platform team needs to consider what mesh-level capabilities are necessary when. Don't try to deliver a complete platform at the start - your platform is a product and minimum viable product, make sure you understand what minimum means and don’t go overboard. The platform team can focus on a few simple things to drive to a good initial outcome/partnership with domains in Manisha's view: 1) how does the work create business value? What do the domains need to do to actually drive value? 2) How will users trust data, what does trust mean and what's needed? 3) How do we make it possible for domains to create and manage a data product that is usable and discoverable? By focusing on the task at hand and then mapping to capabilities to support that task, you can prioritize and deliver something useful and valuable without boiling the ocean. You don't need to try to include every capability at the start, that is a bad anti-pattern. Get close to the use case and find friction. You will also learn to recognize reusable components of the platform but some reusable components might not be evident at the start. Manisha then went further into finding and identifying reusable components. The things that are most unique to each data product are the data modeling and data transformation in her experience. Almost every other aspect of spec-ing out and building a data product are reusable, merely customized to the data product itself. Finding the necessary SLAs and SLOs by working with consumers, that is a reusable process. How your SLAs are actually measured, the definitions around those SLAs are reusable. The infrastructure and CI/CD is reusable. The overall data product blueprints are reusable. So look to make these reliable as your organization learns how to build data products to make for easy reuse. On data modeling and interoperability, Manisha shared that it's crucial to let domains evolve how they model their data as they learn. And interoperability, especially to support a use case, is of course important; but you will likely see a need for interoperability standards emerge when it's needed - basically, don't try to build all your standards ahead of time. That might be creating an enterprise data model with a different name :) When asked specifically about sample data models and automated data modeling tooling, Manisha pointed to them being a double-edged sword. While they can be helpful, most (all?) data products need more custom data modeling to maximize their value. Essentially, the tools can get to a decent initial data model but domains should look to improve them. If Platform teams offer automated modeling tools, they should make sure there is a big caveat to their usage . Manisha recommends you make sure your initial domain has strong enough data talent - whether existing or embedded - to communicate the basic needs to the platform team. Regular developers are often not going to be data fluent enough at the start to drive to exact data infrastructure needs like a data engineer could. But be careful not to over index towards tech too. Every domain will need people skilled in creating value through data modeling but you probably won't need people as advanced in data infrastructure later - the platform is already built by that point :D It's important to differentiate what the platform should offer and what the data product developers should handle according to Manisha. The platform, at least the aspects around data product creation, should be focused on making it quicker, easier, and more reliable to create, deploy, maintain, and evolve data products. It sounds easy but it's actually easy to lose focus on that. Look for friction points in the creation and management lifecycle and automate what doesn't add incremental value. E.g. a data product developer shouldn't have to manually add data to the catalog so look to automate it - and yes, not everything should be built upfront :) Scott note: she added some good flavor around data product boundaries but it's very hard to summarize Within the platform, Manisha believes it's very important to maintain team boundaries because shared resources become a bottleneck and pretty quickly can become very hard to manage. This is why Zhamak has been so clear on the data product as an independently deployable unit of architecture. Manisha gave the example of even the namespace for data products in the data catalog should be reserved for that one team so teams have a dedicated space to put all their data products. Manisha gave some early mesh journey advice: 1) back to data product specification, you should create something that gives teams a very clear idea of what a data product is and encompasses. Scott note: still waiting for someone to open source their data product creation template… 2) if, as Zhamak says, data products are our unit of value exchange in data mesh, then making it easier to exchange value is crucial. Start to create standardized input and output ports so you can easily ingest and serve data. ETL shouldn't be a concept, it's ingest or serving only. 3) really focus on making it easy to discover and then implement SLOs and SLAs. Being able to understand and trust data is crucial to being willing to rely on it. That trust comes from good communication around SLAs. Manisha believes learning the language of the business is crucial for data people. You need to extract the actual business value drivers and build to those so you have to be talking the same language - unfortunately for data people, the language that aligns to business value is usually the business language :) Look to ask more business user-focused questions than trying to get technical. Quick Tidbits: "… the data product spec should [at a] minimum talk about the data set ports, domain, service level agreements, how do I share my data, what does data sharing look like…" - Make your data product specification easy to understand what someone will create and what a consumer will receive. Again, focus on a streamlined developer experience that keys in on autonomy. That's the way…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of May 7, 2023 32:38

۲ years پیش32:38

32:38

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#219 Zhamak's Corner 22 - Increasing Resilience of Data Processes Through Software Best Practices 21:01

۲ years پیش21:01

21:01

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Sponsored by NextData , Zhamak's company that is helping ease data product creation. For more great content from Zhamak, check out her book on data mesh , a book she collaborated on , her LinkedIn , and her Twitter . Key Takeaways: We need to be better about getting on the same page regarding some semantics in data mesh. Otherwise, it's hard to work together internally and across organizations to move the industry forward. There are so many things we've learned about how to break systems into smaller components on the services side, about preventing tight coupling but the data world has yet to apply those learnings. We're heading down some paths that we don't need to if we follow past learnings. As an example of above, early data contract approaches are too tightly coupled around schema. We need to be a little less rigid there but how feels to be determined. Postel's Law: "Be conservative in what you do, be liberal in what you accept from others." Learn it and think about how to apply it to data so we create more resiliency across our internal data ecosystems. Right now, there isn't much out there on the how. Resiliency at scale is possible on the operational plane, why not the data plane? We need to be "very mindful and not naïve" around how we integrate in the data world to not make the same mistakes we made on the services side. Postel's Law: https://ardalis.com/postels-law-robustness-principle/ Semantic Diffusion article Zhamak mentioned: https://www.martinfowler.com/bliki/SemanticDiffusion.html Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Data Mesh Radio episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#218 Building the Right Data Strategy: Why Are We Even Doing This - Interview w/ Beth Bauer 1:31:13

۲ years پیش1:31:13

1:31:13

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts (most interviews from #32 on) here Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Beth's LinkedIn: https://www.linkedin.com/in/beth-bauer-102449/ Beth's Website: https://posiroi.com/ Harvard Business Review article on 'The 3 Elements of Trust': https://hbr.org/2019/02/the-3-elements-of-trust In this episode, Scott interviewed Beth Bauer, Founder and CEO, PosiROI. FYI, there are lots of nuggets in this one for people creating a data strategy or trying to tie your data work to value creation. Beth's ADEPT^2 Framework (covered briefly near the last 10min of the episode): Analytics - Acuity - Data - Decisions - Engagement - Enablement - People - Processes - Technology - Trust Some key takeaways/thoughts from Beth's point of view, much of which she helped craft: To do data right, we need shared responsibility. There is the technical piece of course but the business aspect is just as important. "…we need to realize that nobody's anything without each other" across the units and enterprise. ?Controversial?: Really good data management can cause some challenges to power structures, especially "how it's always been done" power structures. Try to work with people to give them sight to how they are important in a changed organization. ?Controversial?: Don't think data or digital _transformation_. A transformation is something that completes. This is a journey, an ever evolving journey of improving your data practices. Data fluency is crucial - not just giving people the ability to work with data but the trust, especially in themselves, to leverage and even rely on data. People need guiderails to know where they can safely create insights independently, and where they need to ask for more expert guidance. Trust is made of relationships, judgment, and consistency (from HBR article linked above). Over half (half!) of trust is driven by relationships. If you want people to trust your data, you have to form and build relationships. In the same vein, if data itself is the consistency + judgment, that doesn't even get people half way to trust. It's crucial to think about delivery timelines for data strategy work. There are things that will take much longer and deliver considerable value but you need to break them down into manageable pieces that have incremental value as you deliver. As needs shift, you can react because it isn't a locked long-term strategy. Scott note: that project with a payoff only starting 3yrs down the road, it's always 2.5yrs too late. Deliver value incrementally building to the desired outcome, which can shift. ?Controversial?: Sometimes it's necessary to go from the high-level data strategy vision all the way down into the weeds and vice versa. Oftentimes in data, those weeds really do matter and you should look to connect the strategy to specifics when it's of value. It’s really important to recognize that gaining alternate lenses from outside your usual project team can help, regardless of your data function. There's no shame, and it shouldn't be for everything, but expertise outside your normal team is often of great value. Data is a team sport. "A continuous process of gap analysis is absolutely critical." If you aren’t continually assessing your competencies, your data practice will deteriorate. Unfortunately, we are never done, new competency needs emerge and existing ones can easily atrophy. When people ask for data, it's important to simply ask "what are you going to do with it?" If they can't answer that - what would it change for them? - then you want to figure that out before doing significant work. Scott note: as Alla Hale mentioned in episode #122 "what would having this unlock for you?" To do data well, you need to create a high-level vision and strategy that is flexible and can evolve. And then you need to break down your target milestones and start working out what needs to be worked on when. It's okay to make long-term bets but you need incremental value delivery as well. ?Controversial?: Collaboration in data can be a double-edged sword. It can add significant value but coordination can cause friction and delays. Look for ways to avoid coordination slow-downs so you can deliver value faster and more easily but still collaborate to drive towards mutual benefit/value. You have to trust that others will deliver, give them the room to do so. Data without context, without the proper metadata and owner shaping it, can often be more harmful than beneficial. ?Controversial?: Data people have to get comfortable with giving up control and enabling teams to work autonomously to deliver value. It's important to prevent silos, yes, but you still need to let go of some control to create more value as an organization. Look to the Agile philosophy and find ways to achieve the goals of Agile. Orgs doing Agile often get caught up in the ceremony and in the small picture work. Don't focus on ticket closure achievement, work in a nimble way to build incrementally towards much bigger pictures. Data isn't the point. Data is a driver to do business better. Data for the sake of data is far too common of a trap - avoid it and focus on delivering better business results through data. To do data right, producers and consumers need to be able to talk transparently about goals, expectations, limitations, etc. That goes back to trust - both sides have to be able to have a genuine, high context conversation with strong trust. The business people don't need to know "how the sausage is made" relative to data processing but they do need to know what type of sausage it is, what goes into it, what's the flavor, is it a patty or a link, etc. Scott note: this is sharing the information, not merely the 1s and 0s of data. We need to focus on sharing information, not pure data. You need to build out a data sourcing strategy - both external sourcing and internal sourcing. Internal sourcing then further splits into preparing existing data for usage and creating new sources where you lack the data entirely. It can be easy to lose focus on how data work, especially execution on data work, supports the business strategy. But that's where we constantly see the statistics on data work not meeting expectations - when the work gets disconnected from the business value and business strategy. All parties understanding where information actually comes from, how it was generated or sourced, will drive far more trust in the data. ?Controversial?: Much like it's crazy to set a data strategy that doesn't reflect the business strategy, it's similarly crazy to set a business strategy not backed by data. You might not be overly sophisticated in the data you initially provide to a business strategy team but if they aren't leveraging data, they are missing crucial context and understanding. Minimum viable products (MVPs) are very important to innovation and collaboration. They are "… your first iteration of … did I hear you properly? And are we beginning to create what we needed to together?" But too many try to rush them or overengineer them. Focus on both what minimum and what viable mean. Use this as pilot-testing. Beth started the conversation with a bit about her background and then got to swinging a bit at current practices :) She talked about the need to have high-level data strategy vision but also be able to understand things in the weeds. Oftentimes those details in the weeds are important to being able to execute. Just don't get lost in them! Setting a flexible, evolvable strategy and vision is crucial in Beth's view. You need to have a vision of where you want to go before you can figure out what actions you want to take. And then you need to look at milestone goals and break down what needs to be done when. Look to find incremental value delivery instead of putting all your value eggs in the basket that won't pay off until 2-3 years down the road. No one is willing to wait that long and needs will probably shift along the way. It's definitely okay to make long term bets that will require work to deliver value years down the road but focus on delivering value before that too. Priorities will change and so will your understanding of what will deliver the best value. So set out what needs to be done and when and do get going and be ready to reprioritize as you learn! Beth talked a little about the cost of collaboration. If that is loosely coupled collaboration but not necessarily coordination on most to every step, great. But there is always a friction cost to closely controlled collaboration. Try to avoid work that requires too many dependencies - get aligned and work together towards a common vision. We really need to have trust that other parties can and will deliver. If you need to be working that closely, do you really trust each other? Agile can be as much of a hindrance as it can be a help in Beth's view. If it's about taking the bigger picture and constantly breaking it down into achievable pieces, great. But if the focus is on completing work instead of driving to bigger picture value - if the work doesn't have an extremely tangible connection to value - you get lost in the cogs of the machine. You need a strategy that can handle how the cogs work but the cogs aren't the point of the machine - what are they actually driving? Trust is made up of 3 elements according to an HBR article (linked above): relationships, judgment, and consistency. Relationships are over 50% of that trust. Beth believes data is essentially the judgment and the consistency but to get people to use the data we have, the data we can provide, we need to build relationships. You can't just show them the data, it doesn't even get them halfway to trust! Scott note: one comment I make is the difference between someone simply using the data and someone relying or depending on it. It seems like a small differentiation but it's not. One is using the brick as an accent to the building and one is building with the brick as a key element of structural support. Beth pointed to how while the business folks don't need to know exactly how "the sausage is made" relative to data, they do need to understand more than just 'here's some data for you.' It's about sharing the necessary context. If you want sausage analogies, what flavor, what are the ingredients, what is the shape as in patty versus link, etc. They don't care about the data processing techniques but they should care about - and can gain value from - how was the data transformed from a business perspective. Scott note: this is where I talk about sharing information versus data. Just the 1s and 0s of data have no value without context as Beth said. So embed the context, focus on sharing the context, otherwise it is just values in a more complicated spreadsheet :D Creating data sourcing strategies - at the micro and macro level - are important in Beth's view. Don’t overly rely on external data, that's costly. What data do you already have internally that you should leverage? What data could you be generating internally that you aren't? Dive into specifics and create a scalable way for lines of business to figure out good paths with sourcing data - internally and externally - going forward. Make sure to look at things from a cross domain lens too and also think about privacy, regulatory, etc. For Beth, many organizations have trouble keeping the data work aligned to the business strategy. So there needs to be a specific focus on making the work matter, driving to business value. Yes, at the micro level but also on the whole - what business objectives and business outcomes is the data work supporting? Getting "down in the weeds" can also be very helpful, the details do matter as they make it “fit-for-purpose”. On business strategy and data, Beth echoed the view many other guests have shared that creating a data strategy not aligned to the business is not a smart practice. But almost as egregious is not using data to help power your business strategy but this is extremely commonplace. Data is synthesized knowledge of what is going on in the world, often how the organization is interacting with the world. Why wouldn't you want to leverage that for shaping your strategy?! In data, Beth has seen the benefit of the MVP (minimum viable product) methodology and they are wonderful if used correctly. However, they are often not used correctly :) Innovation doesn't have a steady timeline - it's messy. MVP timelines are tough so focus on getting to something viable instead of hitting a deadline - and communicate that to stakeholders. MVPs are about making sure you are on the same page and then iterating to better from there. Beth talked about the need for continuously doing gap analysis with your data and business capabilities. The world is ever changing and new challenges and needs will constantly come up. Plus current capabilities can atrophy. You might be hindered in projects because you need some special capability - especially think legal/regulatory compliance - and you should know that _before_ doing the work :) Two key questions Beth uses when people ask for data/data work: 1) what are you going to do with this? And 2) How much value do you think this will generate? You don't need to get super specific but people need to at least have a good idea of what the work will unlock and the value of that. If it won't cause any action, why do the work? If the cost outweighs the value, that should be known so you can work to balance that equation by cutting costs and/or finding more value. For Beth, to do data right, we need shared responsibility. There is the technical piece of course but the business aspect is just as important. "…we need to realize that nobody's anything without each other." We need to drive to address current gaps and we need short, medium, and long-term strategies that drive necessary work in the short, medium, and long-term. Don't get overly focused on the near or the long-term. But it's not just about doing the data work, especially the technical data work. What value and change does this work actually drive? "And what I found is that largely, a lot of organizations, the challenge is, with really good data management comes really good transparency into how things work. And that really causes pushback on the power structures, and particularly in the 'how it's always been done' power structures. Because if it now points to a way that things can be done better, you start to get into things. Things are happening behind the scenes that have nothing to do with data, and everything to do with people's perceived value of themselves to the organization - without (them) thinking about how they can evolve to actually move from what they're doing today to doing it better." Scott note: it's crucial to help people see how they can move to doing more valuable things - their time of toil is behind them and we can unleash the value creation :) Quick Tidbits: In data, there is often a rush to get things done instead of get to automation. It's often - but not always - the right call to slow down to do things right and set yourself up to do them faster/better/more scalably as you move forward. Lineage, especially for how data was generated and transformed from external sources, is really crucial to increasing trust in that data. "Data for the sake of data is useless." Scott note: PREACH! "The digital transformation, to me, is the biggest misnomer and misguidance that we've ever created." A transformation has an end. This is a journey. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of April 30, 2023 19:28

۲ years پیش19:28

19:28

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#217 Pt 2 of Another One on Buy-In: Flipping the Script on Working with Your First Domain - Mesh Musings 47 20:06

۲ years پیش20:06

20:06

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts (most interviews from #32 on) here Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Here are 8 more leverage points to consider when trying to work with your initial domain Make it EXTREMELY tied to their top priorities. Find additional funding sources. Find a true executive sponsor - think about being the one that brought a whole new approach that is driving far better data results… Having their data be leveraged by other teams can mean more funding flows for their data projects and potentially their domain as a whole. They control how things work for everyone in the data mesh implementation. They aren't just the trendsetter, the data mesh implementation leads will make it work for everyone but they are the secret favorite child and initial shaper. Similar to moving at the speed of business, MUCH smaller experiments. Better data quality. Potentially easier regulatory reporting. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#216 What is Your Part in Doing the Right Thing in Data: Value, Ethics, Literacy, and More - Interview w/ Guy Taylor 1:10:40

۲ years پیش1:10:40

1:10:40

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts (most interviews from #32 on) here Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Guy's LinkedIn: https://www.linkedin.com/in/guytaylor/ Deep Work by Cal Newport: https://www.youtube.com/watch?v=xJYlhhT7hyE In this episode, Scott interviewed Guy Taylor, Director of Data Science and Analytics, as well as the Director of Experimentation at Booking.com. To be clear, Guy was only representing his own views on the episode. Some key takeaways/thoughts from Guy's point of view: In general, "people want to do the right thing." Look to reward people that do the good, ethical things as part of their work product. Always be asking "what is my part in this?" Ask for the expectations and try to be clear in your expectations of others. Literacy is not only about an ability to read but also write. So with data literacy/fluency, we need to be able to use data but also create and share it. It's about learning how to share information - not just 1s and 0s - and share it well. There are still major communication gaps between producers and consumers in many cases with data. Part of that is just not getting on the same page, really making sure both sides close that gap. Scott note: as Andrew Pease said, both parties should go more than half way to ensure you've covered everything. If you don't align on expectations, you're far more likely to have a bad time :) Data people need to stop trying to jump to the tooling to address a challenge first. Get the information necessary - what are people trying to accomplish to create value? - then look to how tools can drive towards capturing that value. In tech and especially in data, there is a strong tendency or leaning towards taking action now. Sometimes no action or no action right now is the right answer. Often times, you need more information to make a better decision instead of patching the hole in the bottom of the boat with bread. ?Controversial?: The way many organizations are trying to leverage data contracts won't lead to significantly better outcomes. There should be technology handling the interface but consumers still need to speak with producers to align on expectations and share their use cases. ?Controversial?: There needs to be more accountability and responsibility on data consumers to actively participate in data work to drive towards the most business value. Ethics in data is always a challenging but interesting issue. Start by creating a set of principles and try to frame potential choices within those principles. Your company ethics will evolve, update your practices as they do. Try to "do the right thing" and encourage others to check in on past decisions to reevaluate too. It's crucial to consider cognitive load - people are most productive when they have time to spend on thinking. Work to give teams, especially those new to owning data, the time to learn, not just the information they should learn. When people know there are expectations on them but they don't know what they are, that's unnecessary cognitive load. Look to make expectations more clear on domains about what data ownership means. On data ownership, be very clear about how far it extends. Data producers should own sharing the information but the consumer has responsibility of ownership too. Scott note: for me, you need to be clear in every relationship but it can be the producer owning the data, the insight, or the insight and the 'so what' - you just have to be very clear! In most organizations, teams are overloaded with work and with cognitive load. You can't just easily pull apart the complexity and fix that overnight. And in a lot of cases, it will be hard to do it at all. Look to prioritize instead of do everything. Saying no can be your ultimate productivity tool. Be like Marie Kondo - 'does this spark value for your organization.' Don't be afraid to shut things down that don't spark value and refurbish your orphaned data generating processes that are valuable. Time for innovation is crucial. It's on leaders to enable their teams to prioritize innovation and experimentation. Always be delivering value. Look to use incremental value delivery methods - how do you break down a big potential project and deliver over time instead of the high risk way of a big project? Guy started off by talking about data literacy and how the analogy of literacy - it's not only the ability to read but also write - carries over to data well. Data literacy or data fluency, it's not just can someone consume data but can they also produce data, can they share information in a way that can be 'read' by others? After all, we aren't trying to share data, we are trying to share information but via data. When Guy embeds people from his data team into domains it "is with the express purpose of doing education, making sure that we are having the conversations around what things mean, what our expectations of those things are." Instead of embedding people into domains to do most of the work, they are focused on helping other people get to a level they can handle far more of the necessary data work. Which is quite often not the deep data work but bridging the communication gaps and getting on the same page. That is especially important for expectations - mismatched expectations is one of the most prevalent and damaging challenges to data work. So Guy is asking data team members to spend a lot of time making sure the producers know how to manage those conversations and drive to what is actually of value instead of what was initially requested. In his experience, there is tendency for data people to try to jump to the tooling to solve issues according to Guy. Going back to expectations, if you try to solve without the expectations setting and leveling conversation, you will likely not deliver what consumers expect. You may see it as solved but they sure don't. That's where you get the dreaded "the data is bad" feedback because there aren't clear metrics and expectations. If you align - and as Ghada Richani mentioned in her episode (#206) stay aligned through collaborative prioritization - then there is a much better chance of delivering value and making all parties happy. A comment Guy made was that there is an over tendency towards action in tech and especially data. People see a problem and they want to jump to trying to fix it instead of getting the necessary information first. And it may be no action is the best answer too. Just because there is pain, that doesn't mean action is necessary immediately. Right now, Guy sees the industry conversations around data contracts and data sharing agreements as slightly naïve: people seem to be thinking this is about data integration between systems instead of data sharing between two parties. And that producers should declare every aspect of what they are producing instead of consumers being part of the conversation. Consumers need to share what they are trying to achieve, how they will use the data, etc. so producers understand the value and what would disrupt that value creation. There needs to be accountability and responsibility falling on consumers too. The contract portion can serve as the technology interface but that doesn't replace the need for conversation. Ethics in data is always going to be an interesting but challenging problem in Guy's book. A good place to start is the social contract aspect: how would this be viewed by society? As an organization, start down the ethics path by creating and agreeing to a set of principles. Create good ways for people to seek and receive useful feedback regarding ethics. And honestly, your company ethics will change and it's important to reevaluate your ethical choices, especially as you learn more - your organization will have made mistakes and that's typically not something to lose sleep over, fix it now and know you’re better. Basically look to "do the right thing." While it can feel good to 'make progress', Guy believes in the Deep Work by Cal Newport type philosophy. People have the greatest impact when they have the time to really think and process. Yet, in today's work world, that is a rarity for anyone. If we are asking teams to really take on data ownership, we have to work to prioritize the time to learn - and that includes processing time. Yes, people learn by doing but not only doing :) Guy talked about trying to clear the space for teams to learn something new, including the impact to the cognitive load capacity of teams, especially when it comes to data ownership on the domains. When people know there are expectations of them and their work but those expectations aren't explicit or clear, that's unnecessary cognitive load - domains need to have crisp and clear expectations - and if the expectations of them by consumers and the data team aren't super clear yet, communicate that. But try to get to more detail to make the implicit very explicit. Another common friction point Guy pointed to is the lack of understanding of your impact on others. He believes again that most people want to "do the right thing" but they don't always know there is even a problem to address. How are your actions impacting downstream data consumers? Again, there is a responsibility on those data consumers to generate the conversation! Sharing that context allows people to do the right thing because they are aware. Currently, most teams in most organizations seem to be overloaded with work and cognitively overloaded in Guy's view. It would be lovely to wave a magic wand and fix that but it's not possible. So, we have to work to pull out the complexity and give our teams the ability to do their best work but high value work has interconnected complexity so you can't take a machete to it and expect good results. Look to prioritize what really matters when you can and break things down into manageable chunks. In talking again about data ownership, Guy believes that producing teams should not have to own too much of the downstream consumption. They should own the transformation and sharing of data after it's been transformed but the consumer should own the insight or metric - they asked for the data so they need to have some responsibilities and accountability too. Scott note: I don't entirely agree it should be that for every use case but I do like a very crisp line of ownership if it works. Guy uses the Marie Kondo approach when looking at orphaned systems or processes - evaluate it and is this "sparking joy", is it creating value. If it is, great, let's get it into a good shape and put it into the right hands ownership wise. Not shove something to someone while it is still in data disrepair but get it functioning well and hand it over. But if something isn't creating value, shut it down. For too long in data, there has been a hesitancy to shut things down because at some point they might create value. Don't fall into that trap. It's important to get to an experimentation - a test, learn, and then iterate - approach in Guy's view; he is the Director of Experimentation after all! A good way to get people to see the value of experimentation is how it provides far better incremental value delivery. Instead of huge projects with big budgets that take years and rarely deliver the value expected, what if instead you took the overall goals and broke it into manageable pieces and delivered value over time as you get closer and closer to the project vision. You'll be more nimble and get a return on investment far quicker. A few tidbits from the end of the conversation: Skunkworks can be a great approach to trying things out and seeing if there is value. Don't try to move the skunkworks directly to production but you can do some fun and useful innovation that way. Good leaders set their teams up to innovate. They prioritize the time to try new things and let their people explore, let them go off the "paved road". Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of April 23, 2023 18:09

۲ years پیش18:09

18:09

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#215 Panel: Leading a Data Mesh Implementation - Led by Kim Thies w/ Omar Khawaja, Ferd Scheepers, and Mike Alvarez 51:49

۲ years پیش51:49

51:49

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts (most interviews from #32 on) here Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Kim's LinkedIn: https://www.linkedin.com/in/vtkthies/ Mike's LinkedIn: https://www.linkedin.com/in/2mikealvarez/ Ferd's LinkedIn: https://www.linkedin.com/in/ferdscheepers/ Omar's LinkedIn: https://www.linkedin.com/in/kmaomar/ In this episode, guest host Kim Thies, Director of Intelligence Automation at PayPal facilitated a discussion with Ferd Scheepers, Chief Information Architect at ING, Mike Alvarez, Former VP of Digital Services at a large healthcare distribution company (guest of episode #236), and Omar Khawaja, Head of Business Intelligence at Roche (guest of episode #96). As per usual, all guests were only reflecting their own views. Scott note: I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views individually. Before we jump in, I think the main takeaway here would be a data mesh implementation leader's journey can be a lonely one. Find peers and exchange information. You can reach out to me (Scott) but there are also many leaders that want to exchange information with each other. The other is the meaning of journey: it's never done; be prepared to continue to push - it can feel Sisyphean but it's important to keep moving forward and expect to continue to drive buy-in. Scott's Top Takeaways: Everyone sees the 'Instagram photos' version of other organizations' data mesh journeys - it's not the reality. Everyone is struggling with certain aspects of data mesh because if this were easy, people would read Zhamak's book and be done with it. It's just not realistic to expect that, give your leaders (yourself?) a break :) It's incredibly important to understand you will get things 'wrong'. Scratch that, you will get MANY things wrong. But 'wrong' in data doesn't have to mean wrong for good/all time. It's about trying, learning, and iterating to better. Really look into fast fail practices. If you feel the need to get everything right, data mesh is definitely not right for you right now. A champion early adopter is crucial to drive a data mesh implementation. There needs to be someone who will partner with you that is "brave enough to try new things" as Omar said. It's easy to focus on so many aspects of data mesh and forget that the mindset shift and cultural change are the biggest challenges for most organizations. They are the most squishy/least tangible but that doesn't mean they aren't absolutely crucial. Teams need the autonomy and empowerment to move at the right speed for them. The more friction in their data work, the more time - and effort - it takes to deliver value and the world can have moved on. You have to give teams the capability to strike while the iron's hot. That's a good way to drive buy-in: we're going to give you the ability to move at the speed of business. Be very clear on expectations. What do we owe each other? Focus on how data mesh drives value for them but also how it drives value for the organization. What is the output of their work, what value comes from what they did or will do? It will be quite difficult to change this kind of thinking overnight. Similarly, be very clear on responsibilities and be very clear on target outcomes. It's easy to get lost in the work instead of what are we trying to achieve. A good success story doesn't have to be a massive win. Time to delivery is really valuable so showing getting something into MVP in a few weeks will get a number of teams excited. Driving that lower is of significant value - and it's a tangible value for many business partners where they can reasonably share about the impact to their business. Other Important Takeaways (many touch on similar points from different aspects): There's a very good reason why we refer to them as data mesh journeys :) You can't treat this as a project. Think of your physical fitness and health. It's an ongoing journey with plenty of trials and tribulations. It's okay if data mesh is not a fit for your organization. There's no shame in that. Be honest when considering if it's a fit or if you just want to do it because it sounds like a good solution in the abstract. In general, look to collect reasons for past failures prior to doing data mesh. Not to throw others or past ways under the bus but to emphasize that the current ways aren't working as well as we might like. Proof points really help convince others. It's always a challenge to maintain autonomy and empowerment without it leading to silos. There isn't a magic formula. It's something you will have to constantly keep a watch on but you're not the only one. There is a balance between autonomy and interoperability. You'll probably get it 'not great' at first and THAT'S OKAY. Work to find the right balance - or at least an acceptable balance - as you move forward. And that balance will likely shift. The focus of the platform should be on making users' lives simpler so they can actually own the data. It's easy to fall into focusing on the tech but the value is in friction reduction for users and the abstractions you offer them. Be prepared to need to continually drive buy-in. Gravity around historical - legacy? - data management practices is STRONG. You will have people wanting to use an enterprise data warehouse or a data lake setup or not wanting to focus on integration with the rest of the organization. Announcing you are doing data mesh doesn't magically mean everyone is bought in. Similarly, be prepared to explain concepts and terms repeatedly. Data mesh - and data work in general - is not the main focus of most people in your organization. Do your best to have the patience to work with people to drive towards a better understanding. Don't focus overly on terminology - see my unicorn farts theory… - focus on what the change in work means for people and what outcomes you are trying to achieve. Business leaders and users don't care that you are doing data mesh - data mesh is a shared language for the data people working on the implementation. Especially don't try to explicitly define federated computational governance :D As Ferd said, look to "bring governance back to the essence of what we want to do." Don't get overly focused on governing everything instead of the right things. That can mean not taking on really difficult challenges right at the start of your journey by passing on use cases. That's okay, think about your capabilities and honestly assess them. Don't bite off more than you can chew. Incremental progress is the best approach to most (all?) aspects of data mesh rather than trying to take giant leaps. Early in your journey, likely focus more on quick wins that will give you some good internal marketing material - market those successes to build desire for additional domains to join, build the momentum of success. Also, think about limiting chances for failure early. Fast fail in general is crucial but you don't want a full 'fast fail' on your first use case. Many things will not work or go great in a data mesh journey. That's the reality of being on the bleeding edge, being an explorer. It's very important to reflect not just on wins but take the learnings from what didn't work. If you don't reflect on your mistakes, you're far more likely to repeat them. Your early champions and business partners can come from unexpected corners of the business. Look for bravery and a well-defined - and valuable - use case with limited initial scope. Be open to the idea of surprising initial use cases and working with unexpected lines of business/domains. Try to directly tie business value generation to your data mesh work. Understand the value drivers for the organization and prioritize the work based on those value drivers. To get buy-in, you don't have to offer domains the perfect solution, just one that is a better option than whatever else is available internally for their use case. Another good potential hook for potential business partners is that data mesh - specifically handling your data as a product - has less technical debt so it can quickly pay for itself in saved time especially for use cases that are mandatory like regulatory compliance. Be very diligent in putting teams together, especially early - throwing bodies at the problem won't solve anything but not having the right capabilities in the teams sets those teams up for failure. I recommend reading Zhamak's book on this topic re necessary capabilities in teams. Incentivization to become a data producer can be quite hard but it's extremely important to get right. How can we think of value creation through data without the perception of monetization? Because that can feel like you are selling data externally. For your first use case, you need to have the business and technology/data team working as one team - one _actual_ team, not just a team in name. That doesn't mean a huge re-org but you need people constantly sharing context and partnering to drive to value. Otherwise, you have a high risk of falling back to the same old ways of doing things. Make failure not a catastrophe. As Vanya Seth says, "contain your blast radius." It's crucial for people to understand that doing decentralized data means a lot of changes. Get people comfortable - or at least acutely aware - that we'll need new ways of working and delivering value. It's probably easier in most organizations to get funding/buy-in if you focus deeply on the business use case. Yes, building data mesh capabilities has an overhead to the initial use case or two as you spend time and effort learning how to do data mesh and it's okay to talk about that. But if you aren't talking direct, tangible business value, getting buy-in - and more importantly funding - will likely be harder. Make sure you have buy-in from consumers on use cases. If they aren't bought in, will they even consume? If you don't have users, the best product in the world is useless. Budgetary issues - why would one domain create value for another if they aren't getting compensated for it? - are extremely common. There isn't an easy answer here unfortunately. As stated above, in general, it's often lonely being the person leading a data mesh journey. Who do you talk to? There's me (Scott) but find other leaders and network. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#214 Is Core Data the Way to Achieve MDM's Goals in Data Mesh - Interview w/ Marcie Stoetzel 1:02:38

۲ years پیش1:02:38

1:02:38

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts (most interviews from #32 on) here Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Marcie's LinkedIn: https://www.linkedin.com/in/marcie-stoetzel/ DGIQ Conference (Marcie and Samia Rahman will be speaking in June; in San Diego, CA, USA): https://dgiq2023west.dataversity.net/registration-welcome.cfm Square hole video Scott mentioned: https://www.youtube.com/watch?v=cvwN_O5ypTE Article about the "leaf sheep" sea slug Marcie mentioned: https://www.bbc.com/travel/article/20210324-the-odd-sea-creature-powered-by-the-sun In this episode, Scott interviewed Marcie Stoetzel, Principle Product Manager of Enterprise Core Data at Seagen. To be clear, she was only representing her own views on the episode. Before we jump in, Seagen might be a bit of a special case in how much the domains can leverage each other's data around a key record type. There is a lot to learn but you might not be able to find use cases that are as broadly impactful to many domains at once. Some key takeaways/thoughts from Marcie's point of view: ?Controversial?: In data mesh, building a data culture focused on engagement, learning, and upskilling might be as important (more?) as doing the data work - if teams aren't willing to engage, what's the point of doing the work? Scott note: Marcie doesn't explicitly say this but it sure feels like an undercurrent. It's crucial to make your data culture something that can embrace data mesh. When going to domains, come with a target value proposition. Why would they get value from participating in your data mesh initiative? Scott note: If there isn't a value prop for the domain, you will almost certainly struggle with incentivization. There are very few 'good Samaritans' willing to do a lot of data work with no discernible direct benefit to that domain. Seeing is believing. Don't just tell people what you are going to do, get demos in front of them and show them more about what you plan to do. Make sure to solicit feedback as you build. It's far easier to pivot after a short period of work than after a longer period. You can find misaligned expectations far earlier and work on appropriate prioritization. Basically, you spend less time building things that don't matter and more time on things that do. Look for ways for domains to share about what information they have that might be useful in the organization. Oftentimes, domains operate so independently, they aren't in the habit of finding ways to collaborate. Scott note: Communication is the easiest path to high-value, cross domain use cases - the mesh won't find them for us. To actually get to cross-domain use cases, you need domains to consider things from another domain's point of view. So extract more information about each domain to literally show to other domains to spark conversations. Be someone that connects the dots for domains and then connect people across domains. Once there is more visibility to cross-domain use cases, business goals can be realigned to focus on building to common goals. That can help with some incentivization and alignment. Scott note: I hadn't previously heard this one articulated but it really does map with goal realignment where many organizations are seeing success with data mesh. When you are early in your journey, look to create a "thin slice of an end-to-end experience for [a] domain." Really don't try to do too much. Scott note: this is what pretty much everyone who has worked on at least two data mesh journeys circles back to. It's VERY important to understand the thin slice concept. Interoperability requires a certain level of trust. You can decide by your use-case how strong that trust has to be. If you need it to be essentially perfect for regulatory reasons, that's far different from the needs of many other use cases. Focus, especially early in your journey, is very useful. Again, thin slice. Don't take on too much while you build initial momentum and buy-in. ?Controversial?: Find ways to humanize your data work, add a bit of levity and humor. It will make people connect to it more and internalize it more. Plus, it's just more fun. ?Controversial?: Similarly, "explore, discover and mature data together." It's okay to be vulnerable, especially as we learn. Transparency is also crucial. Community and things like workshops are crucial to see wider adoption and engagement in a data mesh implementation. Consider if you want to use the phrase master data management (MDM) at all. Master has slavery connotations and MDM feels far too heavy as a phrase. It can scare off the exact people you want leaning in to dealing with Enterprise Core Data. Marcie started off with a bit about her background as a teacher and as well on the commercial side of the healthcare space which has shaped her view of teaching/learning and also healthcare data needs. Part of Seagen's goal in using Enterprise Core Data instead of Master Data Management (MDM) as a phrase was the moving away from the connotations with slavery (master) and that this data is core to the enterprise, that this is crucial to the organization, not just a data management practice or task. Enterprise Core Data at Seagen is about creating a way to make data that many domains leverage the same to prevent lots of domains doing the same work and make interoperability FAR easier. MDM also is typically managed centrally instead of enabled centrally and managed at the domain level (federated governance) so you have to rethink a lot to do Enterprise Core Data instead in a data mesh setup. Trying to map 1:1 to MDM in a historical data approach won't work well. At Seagen, Marcie and team decided to tackle one type of core data record first - healthcare professionals - rather than trying to unify every type of record across healthcare - don't boil the ocean or bite off more than you can chew. They are working on creating the platform for domains to manage their core data records which creates more sharing opportunities and even higher quality data - teams can better cross-reference information. While they are still pre-production - it's early days - even bringing this to domains' attention is sparking conversations between domains about potential collaboration and new use cases. Marcie and team are winning converts by showing them what the platform will be able to do for their own domain but also keeping an eye on that interoperability and leverage provided to other domains. That means that domains get some value from participating even if no other domains participate. If other domains do participate, then everyone gets more value from each other. They started with a simple value proposition - this will make handling your own data easier - and then created a group collaboration incentive - the more domains that participate, the better the information and the less work everyone has to do to get to better outcomes :) When asked about incentivization complications around domains wanting to focus on their own goals, Marcie mentioned that as the domains are starting to find cross-domain use cases, the organization can realign goals to be about focusing on those common goals where everyone wins. What drives the most value for the business and how do we incent that kind of outcome/behavior? Scott note: this is an interesting nuance that I haven't heard before. How they found the cross domain use cases was also interesting. Marcie and team met independently with different business domains to extract what each team felt could be a good output of working with a common enterprise core data platform, as well as tying these individual value props back to the larger goal of helping more patients. The data team then literally took all those outputs and showed each business domain that uses HCP (healthcare professional) data the value across Seagen of having an enterprise core data platform. This sparked collaboration ideas between domains for how to drive even more value from the core data platform. Marcie is seeing that domains have to spend a LOT of time to cleanse and match data. And downstream consumers of their data have to do the work too as they don't know it's already been done upstream. The quality requirements for most use cases are pretty high. So creating a way for domains to much more easily interoperate data will save them a LOT of time and effort. The core data platform will hopefully prevent lots of domains from having to do a lot of that quality checking work and it will also increase quality by having more sources of information to verify data is high quality / correct - basically, the checking of quality becomes far less arduous. One thing that is working well at Seagen for Marcie and team is doing lots of demos and small proofs of concepts. Similar to doing sprint demos, teams are buying in because seeing is believing. They are showing the business real, realized value that will come from their participation. So teams are leaning in. Similar to what Karolina Henzel mentioned in episode #104, there can be a LOT of value in addressing data quality issues for domains. Marcie talked about the value of maintaining focus on a thin slice. Instead of trying to get many domains bought in on the data mesh concept, there is a specific use case and they are only working with a few domains at the start. There are clearly defined and scoped benefits. Again, thin slice. And focusing on the end-to-end solution to working with this data for the domains has also helped to get and keep everyone on the same page. Quick Tidbits: Look for ways to share knowledge in fun and interesting ways. Upskilling can be a bit intimidating, make it more gamified and less high pressure. Humanize (or in Kye's case, dog-ize) it a bit. Really embrace an attitude of learning - be vulnerable and transparent. "Explore, discover and mature data together." "Engaging conversations, exploration, curiosity, and a safe space" are crucial. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of April 16, 2023 30:22

۲ years پیش30:22

30:22

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#213 Zhamak's Corner 21 - Reinventing Data Development, Not Data Processing and Storage 20:26

۲ years پیش20:26

20:26

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Sponsored by NextData , Zhamak's company that is helping ease data product creation. For more great content from Zhamak, check out her book on data mesh , a book she collaborated on , her LinkedIn , and her Twitter . Key Takeaways: We need to make the data product the first class primitive of information sharing - make them the basic building block of how we create our internal data/AI ecosystem. (repeat from ZC20) "Tools reshape behavior" - if we want developers to change their relationship to data, we need to give them the capability to do so easily. Or even if not easily, at least not arduously :) The industry sees the value of data mesh but we need to find much lower effort ways to create sustainable, large-scale change. We all need to be finding catalysts. There is no reason to try to reinvent a lot of the technology in data at the physical layer. Lake storage, streaming technologies, ML libraries _at the physical layer_ are great. But we need new ways of accessing and leveraging them to make it far easier to create and manage data products. While everyone seems to be talking data products, there seems to be so many different definitions. This has led to a weaker market pull on vendors to improve their tooling to make data mesh more easily possible. Semantic Diffusion article Zhamak mentioned: https://www.martinfowler.com/bliki/SemanticDiffusion.html Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Data Mesh Radio episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#212 Reflections on Building a Data Mesh Platform from Scratch - Interview w/ Jyotshna Karki 59:42

۲ years پیش59:42

59:42

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts (most interviews from #32 on) here Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Jyotshna's LinkedIn: https://www.linkedin.com/in/jyotshna-karki-81a24038/ In this episode, Scott interviewed Jyotshna Karki, Data Engineer at Novo Nordisk. To be clear, she was only representing her own views on the episode. Some key takeaways/thoughts from Jyotshna's point of view: In data, especially in data engineering, people need to be curious. There are so many new innovations that may really be majorly beneficial. Look to try out more approaches and technologies. You can have happy data producers and consumers with a centralized data lake setup and still have data mesh be the right evolution. In the long-run, at scale, it isn't efficient to have a centralized data team coordinating all data use cases. For many domain teams, the centralized data processing and storage can be a black box. Data goes in, it gets transformed and stored by the central team and then served out. This can create a high dependency on experts and technology. ?Controversial?: If your domain team consists of their own data engineers and data scientist with domain knowledge experts to manage their own data products, it's okay to work with multiple teams at the start of a mesh journey. Scott note: if you don't need to drive buy-in and your org can do this, I don't see it as a major risk. But probably at most a few hundred (tens maybe even) organizations are like this worldwide. Don't try to enable every tool as part of your platform. You should focus and create a good experience on the most widely used tools rather than trying to support every tool available out there. ?Controversial?: Probably don't try to automate processes at the proof of concept stage. Wait until the need and impact is greater. But non-automated processes are typically tech debt you should look to pay down when it makes sense, don't ignore that or it will hurt more at later stage. Around best practices or things like reusable components and data pipeline blueprints, look to create centralized community sharing mechanisms but with decentralized ownership and contribution. Try to enable that sense of community knowledge sharing and trust. Create a process to assess if you need to make a change to your platform. What are the business needs and are you meeting them? Constantly look to evolve and improve your platform. KPIs for your platform are important - it's a product - but it's okay to start out pretty simple and use low tech monitoring signals like number of support tickets and customer feedback. Similarly, try to be more data-driven around building your platform. Even if that data is pretty raw and unsophisticated to start. Scott note: try to stay away from vanity metrics but it's okay to / you will probably start with vanity metrics until you understand what drives business value from the platform. Democratizing data, especially doing data catalog well, has led to less data deduplication. Because people know where to find data and can reliably get access again, they don’t copy or build something similar to ensure they have data they need. "…make sure that it is reliable enough for people to depend on this data." Trust is crucial, give them visibility to see how data is handled to give them enough trust to really depend on it, not just use it. Look to community events like hackathons to drive additional experimentation and value. If you make innovation a part of your culture, good things will likely come. Jyotshna started off the conversation with a bit about her background, especially in data engineering and the need to be and stay curious. There are so many new approaches and technologies that could provide significant benefit to consider. Think in that product mindset and look to evolve your approaches and tech stack to create more value. Specific to Novo Nordisk's data mesh journey, Jyotshna and team saw the writing on the wall for their data lake setup. While their centralized data lake was doing well and people were happy with it, there were increasing consumer and producer demands and the central data team was still required to help teams create their data products. Having a centralized team in the middle of every use case just wouldn't be efficient. Then, they hit some cloud service limits which caused some major headaches as well. All this led to looking to decentralize via data mesh. At Novo Nordisk, many domains already had significant data capabilities and there were people building data products anyway according to Jyotshna. What they really needed was a way to empower and enable teams to more easily create and manage those data products in an interoperable way and lower the bar. So the central data team was to focus on the platform but there wasn't a huge need to upskill all the domains. There is still another centralized team of data experts to help domains that aren't as data fluent. Scott note: while this is not super uncommon, most organizations are not this lucky :D Specifically to the pharma industry, Jyotshna shared some of the pre data mesh compliance/regulatory issues that were better addressed with data mesh. Domains needed to work with regulators but it was hard for them to really see exactly how the data was stored as it was managed by the central team, which is part of compliance. It was all part of the central data lake AWS account and those teams didn't have the ownership or visibility they needed. But with data mesh, the teams now have the visibility to their own data storage and access to audit logs and data governance. Jyotshna shared that at Novo Nordisk, there was so much demand to participate in their data mesh, the data platform team and any centralized data capabilities - to assist the domains that didn't have high data fluency - worked with multiple teams to start. This helped them to define the requirements for their data mesh platform to support multiple data domains. While this is a data mesh anti-pattern, it went well for them as many of the domains again were quite capable with data engineering and data analysis. There were also many domains that wanted to contribute aspects to the platform so there were good feedback loops between the platform team and many domains. Scott note: Don't go this route unless your domains are already highly data fluent/capable. Working with many domains at the start can create a high-risk scenario instead of thin slicing. Jyotshna and team are focused on enabling proof of concepts more than trying to automate everything right at the start. She noted they are focusing on understanding the problem deeply and moving fast to get proofs of concept into people's hands and then circling back to automate when there is more need and things are slightly more stable. Basically, they are being agile. It also has led to more modular components and reusability - they can get things out in a prototype phase and then think bigger picture how to deal with similar problems instead of point solutions. In order to prevent tight coupling and keep modularity, Jyotshna and team started to actually remove things like data pipeline blueprints, reusable components, and bootstrapping accounts from the data mesh platform. While that might feel counterintuitive, they wanted to create a community specifically around things like blueprints so the central team wasn't managing them, community members were managing them. Look to create central sharing mechanisms but with a decentralized ownership and contribution model. Community-led innovation is more scalable than centralized knowledge ownership. When thinking about platform maturity and if they need to pay down any tech debt, especially around certain features, Jyotshna and team benchmark quality levels and compare those to the actual business needs. Being in a heavily regulated industry, some aspects of compliance just are non-negotiable, you must meet them. But there are places where 'good enough for now' is a completely acceptable and correct answer. Some signals they use are support tickets and direct feedback around different aspects of the platform. They are also starting to build KPIs but it's a work in process. One interesting aspect of doing data mesh has been less duplication of work per Jyotshna. This is a target goal of data mesh of course but it came about naturally as now that people can reliably find and access data, they don't feel a need to build it themselves. Jyotshna said "make sure that it is reliable enough for people to depend on this data". Part of your platform and your overall mesh is to make it easy for consumers but also producers to trust the data. If you have a black box process, can producers really trust it? And evolution of your data products plays a part in trust too - a consumer can trust that the way data is presented is still relevant to the business. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of April 9, 2023 14:41

۲ years پیش14:41

14:41

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#211 Another One on Buy-In: Flipping the Script on Working with Your First Domain Pt 1 - Mesh Musings 46 20:17

۲ years پیش20:17

20:17

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts (most interviews from #32 on) here 3 use case aspects or patterns to lean away from when trying to work with your initial domain: What I call the domain type 'Everyone else in the organization wants your data so give it to us'. Who is going to fund this? Not having a clear path to exec sponsorship and what the value expected will be is not great. How do I add this to my priorities - essentially 'great, but that's more work I can't take on.' And the 4 you should look to lean into: Find your initial use case that is self-contained in the domain. Move at the speed of business. Find a use case that is easy enough with enough return to pay for itself as you also invest in the other aspects. We will be investing heavily in making your domain better with data. Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#210 Organizational Scalability in Data Mesh - Interview w/ Chris Haas 59:50

۲ years پیش59:50

59:50

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts (most interviews from #32 on) here Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Chris' LinkedIn: https://www.linkedin.com/in/christopher-andr%C3%A9-haas/ Article on Lean Value Tree: https://rolandbutler.medium.com/what-is-the-lean-value-tree-e90d06328f09 In this episode, Scott interviewed Chris Haas, Advisory Consultant at Thoughtworks. To be clear, he was only representing his own views on the episode. Some key takeaways/thoughts from Chris' point of view: When starting a data mesh journey, you have to find teams that are actually willing to participate. If they won't truly take ownership of their data, it's essentially a non-starter. You can help convince many domains by showing them the investment you'll put into their teams on upskilling and other capabilities enhancements - it's not just new responsibilities and work. In data mesh, look to drive domain-wide understanding and buy-in, at least to the vision of what a more data capable domain means, the benefits to the domain. Not everyone will or has to care about data mesh specifically but you shouldn't have data mesh by decree from upper management. Consider lean value tree mapping - it helps everyone align on a single mission with goals to support the mission, what bets you have to make for each goal, and on down. It can help you stay focused on what are you trying to achieve. For use case prioritization within a domain, look to 1) what are the business goals, 2) what use case will have the highest value, and 3) what is the easiest to execute on. Don't get overly focused on value if you aren't ready to actually deliver it. Scott note: don't forget risk too ?Controversial?: Choose a use case with a single domain if possible for your organization's first data mesh use case. It just limits the amount of challenges and potential failure modes. It's common to want to first focus on the technical aspects of data mesh - it's the tangible part and easier to see changes - but you should be focusing on the organizational aspects just as much if not more early in your journey. You can start your data mesh platform team in a central data team, that's a normal pattern. And it doesn't have to be huge, 4-5 people building and a product owner - the platform is a product too. A domain-based data product team doesn’t have to be large or require a big reorganization of the domain. Make sure they have the capabilities - data fluency and domain subject matter expertise - to execute but it can be a small team carved out of the domain. Data product teams tend to be long standing teams but they are typically responsible for developing multiple data products, not just a singular data product. Most data products don't need that level of day-to-day tending :) ?Controversial?: You should consider your data product team a new team in the domain and a new team means budget. If you don't have budget and you don't rework priorities, you are putting additional work on a domain and the likelihood of success falls - look to avoid that risk. Doing decentralized data, doing data mesh doesn't mean everything is decentralized. Scott note: this is a persistent myth around data mesh. Not everything becomes decentralized or you will just create silos. ?Controversial?: Once you've proved out you can do data mesh and drive value, when thinking about adding new domains, consider setting up a transformation office to coordinate, upskill, and prioritize work across domains. Look to build an internal community to encourage cross domain communication and hopefully collaboration. Finding cross-domain use cases is likely to be more difficult. Leverage that internal community to create more conversations that result in new, valuable use cases. When finding your data products for a specific use case, start from what would be the specific data product to support this use case. Then start to work backwards towards what source-aligned data products you need. And of course, look at existing data products for potential reuse first. !Controversial!: Source aligned data products should be - or at least start off - "very specific and very small." Chris started out with a rather blunt but crucial statement: when thinking about data mesh, you have to identify at least one domain that will actually take ownership of their data. A successful data mesh implementation can't be entirely IT driven. And domain data ownership and coupling that with data as a product are typically very much not the natural order for most large organizations. You will need to invest into domains to make them capable of owning their data - showing that you will invest in their success can help win them over - so you want to make sure it will be money/resources/time well spent. For Chris, it's very important for there to be at least a domain-wide understanding - and hopefully buy-in too - for what the domain is trying to do around data. That can be about data mesh or simply how they are changing their relationship to data. It won't work well to do data mesh from just upper management buy-in and pushing that down as a mandate. And that requirement can lead to deciding that data mesh isn't right for a domain and that is perfectly okay and reasonable - not every data management challenge is a nail for a data mesh hammer. Lean value trees are an important tool for Chris and team when speaking with a domain about data mesh. What are they actually trying to achieve - the mission - and work backwards from there. Break down the goals, then figure out the assumptions or bets around those goals. This helps you stay focused on what you are trying to achieve - is it deliver a data product or is it address the use case and create business value? So the output of a lean value tree helps align the team on a mission or missions which align to the business goals. Then, when thinking about use case prioritization, you need to balance those business goals, the amount of expected value of a use case, and the expected amount of work to deliver that use case*. * Scott note: it is kind of included in the amount of work and expected value but I'd also factor in risk - what is the risk of this use case not being valuable and what is the risk of the team not being able to execute well on the specific use case. Probably don't take a big gamble on your first use case ;) At the start of a journey, Chris recommends to find a use case that benefits the producing domain if at all possible. Yes, we want domains to publish data to benefit the entire organization but if a domain is going to be the test subject and invest their money, their people's time, their people's cognitive load, etc., it will be hard to find a domain willing to do that for another domain without decently strong incentivization. And at the start of your journey, those incentivization and community mechanisms are likely hard to come by. If you don't have these challenges and domains are all happy to help each other, consider yourself very lucky. Based on interactions with a number of clients and prospects, many organizations would rather focus on the technical aspects of data mesh first over the organizational and Chris wishes that weren't the case. While building out the technical aspects is no easy task, if you aren't ready to actually do the day-to-day work, what are you building the tech to support? Scott note: this is extremely common and is also a very common comment from consultants. The tech feels more tangible and it's easy to say yes/no than squishy operating model discussions. But they are crucial to doing data mesh right. Chris believes an early data mesh alignment on the organizational model doesn't have to be - and shouldn't be - disruptive. You can have a domain start to carve out new ways of working without doing a major reorganization. There should be a team building the platform and a product owner for the platform but 4-5 people building is reasonable and they can live in a central data team, that's a normal pattern, no reorg needed. As for the data product team, you want them to be part of the domain if possible and it should be a mix of highly data fluent people and domain subject matter experts but again, it can be a small team. So a small team carve out but not realigning the entire domain. Scott note: remember domain is an overloaded term. Some domains are sub-domains that are 3-5 people but typically we mean the line of business which can be 10K+ people. For Chris, when asked about data product teams, he said you should look to have them be long-standing teams. You need an owner per data product but often the development teams will be charged with creating multiple data products rather than only one. Especially as you build the knowledge of how to build good data products, that team can become more efficient. But do not treat your data products like projects, they should be managed/evolved by long-standing teams, otherwise they are more likely to fall to data disrepair. As a consultant, Chris recommends working with a consultancy :D but the point is that your data product team in the domain should be considered a new team and that new team needs a budget. It might be reallocated budget but hopefully it's new budget. But the consultancy angle is that if the implementation doesn't deliver value, it's easier to cut the consultancy than reassign or lay-off employees. Scott note: I think this is a valid concern but many can't get the budget and many organizations are doing this work entirely internally. Once you've proven value from and a capability to do data mesh, a typical pattern is to set up a transformation office that will assist other domains in their journey according to Chris. That way you can maintain coordination, find reusable patterns - tech, architecture, people, process, etc. patterns -, prioritize, upskill, etc. That office is helpful in getting teams up to speed but also setting expectations - domains won't magically transform their data capabilities overnight. And look to build an internal community function to encourage cross domain communication and hopefully collaboration. For Chris, that community aspect is really crucial to identifying cross-domain use cases. The data product owners should be communicating with each other to discuss what they've created in case that sparks new ideas for new use cases. For finding what data products you need to support a use case, start from the mission, then the consumer-aligned data product that would best serve that mission. Once you know what you'd want to have as the end product, you can start to find the necessary source-aligned data products. Chris wrapped up with a few useful tidbits: Don't worry about trying to serve future, unknown use cases with data products you are building now. Build for reuse and build for evolution and then evolve when those new use cases emerge. Source-aligned data products should be - or at least start out - "very specific and very small." Don't try to cram too much in. Scott note: I think this can create some discoverability issues but it is a pattern I am seeing more. See Carlos Saona episode #150 for how that looks in an implementation Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of April 2, 2023 19:46

۲ years پیش19:46

19:46

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#209 Panel: What is BI's Place in Data Mesh - Led by Ammara Gafoor w/ Elif Tutuk and Ryan Dolley 59:30

۲ years پیش59:30

59:30

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Ammara's LinkedIn: https://www.linkedin.com/in/ammara-gafoor/ Ammara's Articles on data mesh Part 1: https://www.thoughtworks.com/insights/articles/data-mesh-in-practice-getting-off-to-the-right-start Part 2: https://www.thoughtworks.com/insights/articles/data-mesh-in-practice-organizational-operating-model Part 3: https://www.thoughtworks.com/insights/articles/data-mesh-in-practice-product-thinking-and-development Elif's LinkedIn: https://www.linkedin.com/in/elift/ AtScale article on data mesh: Principles of data mesh and how semantic layer brings it to life: https://www.atscale.com/resource/data-mesh-principles-semantic-layer/ General AtScale article: The Semantic Layer’s Critical Roles in Modern Data Architectures: https://www.atscale.com/resource/the-semantic-layers-critical-roles-in-modern-data-architectures/ Ryan's LinkedIn: https://www.linkedin.com/in/ryandolley/ Super Data Bros Blog: https://superdatablog.substack.com/ Super Data Bros YouTube: https://www.youtube.com/@superdatabros In this episode, guest host Ammara Gafoor, Principal Business Analyst at Thoughtworks (guest of episode #133) facilitated a discussion with Elif Tutuk, Global Head of Product at AtScale, and Ryan Dolley, Independent Data Consultant and one of the SuperDataBros (guest of episode #183). The focus topic area was what role does Business Intelligence (BI) have in data mesh - e.g. where does it sit and who owns it - and how do we enable BI to really drive significant value in a data mesh implementation. A few other episodes that would be good to get a broader picture here on related topics in addition to Ammara and Ryan's episodes are #199 with Brent Dykes and #192 with João Sousa. Scott note: I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views. This will be the standard for panels going forward. Scott's Top Takeaways: It's really important to actually define what business intelligence even means to your organization. If everyone doesn't have a clearly defined picture, it's one of the easiest things to have a major mismatch on expectations - everyone thinks they know what we mean by BI but we often mean different, sometimes vastly different, things. In data mesh, it's okay to have multiple org groups/layers of BI across the org - say BI teams embedded in the domains and a central BI team too - as long as responsibilities are relatively clear and teams _communicate_. If everyone is working on similar or overlapping goals, it's going to create BI sprawl - e.g. dashboard and report sprawl with weak ownership. BI must focus on enabling business users, not just data analysts. How do we make it so regular business users can actually drive their own analysis and then own the output? How can they then share their insights back to others. It's important to focus on enabling BI capabilities over tooling. In BI that feels especially hard because the interface to the data for most people is literally the tools themselves so of course tools feel like they are most important. How we make that distinction, especially to users, is hard and seems to be to-be-determined. Really consider if BI is only a consumer or, more likely, is it also a crucial producer to the mesh. Are data analysis outputs going to be made available in something like an analytics catalog? Whether you call that part of the mesh or not, how are people reliably sharing the insights they create and how are others discovering those? Ammara brought up the complexities of how do we go from providing raw data in these fundamental data products to actually doing BI - the tools are not designed to ingest from many sources and the BI professionals are typically not super technical. Do we create domain data marts or mesh aggregated data products? I like the recipe angle Mahmoud Yassin mentioned in episode #103. Data virtualization technologies also probably play a key role here. Elif made a good point about there is a difference between making the data analytics ready and business ready. How do we bridge that gap? How do we even define that gap so we can recognize it before addressing it? Design thinking and product are truly crucial to doing BI right, whether data mesh or not. For too long, all aspects of data but especially BI have been focused on outputs - kind of like # of widgets produced - as the metric. It can feel like accomplishing something even if it's not creating value. We need to focus much more on how people will use what we build to drive specific value. How do we create "delightful experiences" as Zhamak always says for consumers of business intelligence. Other Important Takeaways (many touch on similar points from different aspects): BI is about "translating the things that happen in the business … into the format that's going to enable people to make informed decisions on it…" as Ryan said. It's easy to get bogged down in tools and techniques but at the end of the day, we're trying to provide intelligence to make better decisions, that enables people to understand and take reasonable actions. Data mesh gives us a chance to reinvent how we produce BI dashboards and reports. Instead of creating single tables to support each dashboard, we can easily source reliable data without creating rigid and quickly deteriorating or abandoned data assets. Balancing user experience - especially letting users use their tools of choice - and reliability/scalability is very hard in BI. How do you track lineage into Excel? How do you _govern_ data that's been put into Excel? But if you try to get everyone into Power BI, will you have user disengagement? Scott note: you won't even be able to pry Excel from my dead hands. It's not for reporting but it's the best data poking tool around for small to moderate data sets We need to start to think about how our mesh data products interface with business users. At the mesh experience plane level, how can they have what Zhamak calls in her book "a delightful experience"? In general in data, we often overlook the rank and file users to focus on the highly technical ones - those are more 'our people' - but we will miss out on a ton of business value if we do that with data mesh. Is a centralized BI team a mesh anti-pattern? I agree with Ammara and her colleague Emily Gorcenski: nope! You need someone really focused on generating and owning cross-domain use cases or we focus on preventing data silos for nothing. If we don't actually combine the data across the domains, what's the point of data mesh?! Just make sure to share information with other BI teams to prevent knowledge silos. There is a lot of focus on who sits where and exactly how things interact, which is important in some respects. But it's more important to focus on what are we trying to actually accomplish and then who owns that. Clear responsibilities win, the org chart doesn't! :) Governance is a key enabler in BI. Not just defining who does what but creating common and easy to leverage interfaces to our data. That can't be done individually for each data product - it would be a ton of work on data producers and consumers would have an awful time. So you have to consider how far does the mesh experience plane extend. Is it into BI or not? If not, how do we achieve scale if business consumers have a disjointed experience and can't easily share with each other. Mesh consumer-aligned data product owners need to consider if BI users are a target consumer. What output ports are you creating to make it easy for BI users to leverage your data product? How do we think about the semantics and metrics layers relative to BI in data mesh? BI users want to be able to trust data without deep-diving into transformations and lineage. How do we make that simple and easy? I really don't have good answers here. Zhamak has talked about her disdain for layers specifically for data in a data product and the large-scale data pipeline approach. I don't believe this extends to the concepts of a semantics layer or a metrics layer if you use this as something data products need to produce their metadata into. It's another aspect powering data discoverability. It's kind of a complicated analogy for how this might work and I am working on the wording… Governance has to be focused on enabling BI usage of data more than restricting. Of course, controlling who has access and what people can do with it is crucial but with mesh, if we are so focused on producing great quality data across the org, we have to think about how we can enable usage at scale. That's table stakes for data mesh. We need to create good and low friction ways for people to go from generating an insight to sharing that insight. It's often that insights get locked in at the BI layer. How do we share those insights back to others on the mesh to leverage? Can we easily push insights into the mesh to create a new data product? This is a cultural AND a technology issue that needs to be addressed. In BI, there is often too much of a bias towards 'give me the complete picture and I'll go from there' which leads to complicated and non-effective customer 360 approaches. Just sourcing a bunch of data without a specific business objective. Humans are curious creatures! Do we need to focus on what information are you trying to understand and why first rather than produce all this data and I _might_ have an interesting insight? I think yes. One thing that _feels_ like it is often missing from a BI strategy is bringing the business knowledge to the data instead of only bringing the data to business users. How to actually go about bringing the business knowledge to the data, I have no idea. Maybe this is the consumer-driven data modeling and feedback involved in data product creation but it feels deeper than that. It's a concept that is brewing in my mind… Do we want to have some org-wide defined metrics? Taxonomies and ontologies can easily become overly rigid but having optional adherence can make things much easier when you think about standardized metrics and meanings. It's all a balance and it feels like you should be constantly assessing it because it can easily go the route of too many metrics and your mesh becomes littered with metrics that all mean slightly different things. BI people are likely to be skeptical of data mesh. Trying to get access to tons of systems and cross-correlate and consume data from multiple sources has historically been a pain - probably an insane pain to be honest. Be ready for pushback even if this is a better solution in the long-run - and likely short-run - for them. Show them how BI in data mesh can be better, that their focus is on value. We need to get BI people on board with data mesh and what it offers and how it will be better for them in the long-run. BI people are often the data gateways to execs. If the BI people aren't on board, are your executives going to see the benefit of data mesh? Not just buy-in wise but are they literally going to see an improvement in their business operations from data work if the BI people aren't leveraging it? In general, we need to get better at translating the business need into the BI need then into the data need. Data work is often not as valuable as expected because it gets divorced from what people actually care about. Far easier said than done of course but think about user experience more relative to the entire data consumption and analysis / BI process. Churning out reports and dashboards is not a good end-consumer user experience. What do business users get out of it to drive their business? We need to tie more BI work directly to what would drive change, what would drive action if we found out new information. We can start by just looking at who is using reports/dashboards before getting more complicated. Product thinking in BI isn't just about the consumer experience, we need consumers to be part of the conversation and giving more feedback on usage and importance of the BI work. Instead of just having information pushed at them, it needs to be a collaborative push/pull. Ammara asked about data mesh and BI teams: "…how does data mesh not add to the complexity? How do we not add more strain to the teams? … How do we not add to more to their cognitive load?" How do we make BI feasible in data mesh? It's not as obvious of an answer as it feels initially. We want to get more value leverage, not more work, from the teams. Data virtualization might be a key aspect of doing BI well. Ghada Richani (episode #206) raved about what data virtualization has allowed them to do as well because it's crucial to easily be able to ingest data you are supposed to have access to. And many tools aren't designed to take data from lots of sources. I have lots of places where I am seeing data virtualization misused but this one seems safe :) Some people are better served by feeding them reports and dashboards than giving them advanced self-serve analytics capabilities. Do we need to level them up? Do we want to move past "first wave" type practices? I honestly don't know. Some people like to cling to ways of the past but it might also be serving them well. If it's scalable and reliable, do we need to push to 'modernize'? Probably? A bigger BI question than just data mesh related :) Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#208 Making Prioritization a Priority and Focusing on Delivering Value - Interview w/ Srinivas Paluri 1:14:19

۲ years پیش1:14:19

1:14:19

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Srinivas' LinkedIn: https://www.linkedin.com/in/spaluri/ In this episode, Scott interviewed Srinivas Paluri, CTO at Rentbase. Srinivas was previously part of a data mesh implementation as the Senior Director of Data Engineering at Zillow. Some key takeaways/thoughts from Srinivas' point of view: Data mesh advice to former self #1: Ambiguity is inevitable. Don’t be afraid of ambiguity - it's often better than binary thinking - but also be as clear as possible on responsibilities even if the target outcome is ambiguous. Clear responsibilities at least drive things forward. Data mesh advice to former self #2: Involve product management way earlier. Every product owner needs to understand product ownership, prioritization, and what is the value to the business. And then communicate the value of the work to the business. If that value's not clear, why are you doing the work? Data mesh advice to former self #3: Create a small team, maybe 5-6 people, focused on enabling new domains to learn how to own their own data and create data products. Scott note: see episode 48 for how ITV is implementing this pattern Prioritization - and communication around prioritization - is probably one of your most useful tools in data mesh. If you get good at that, teams will often buy-in more quickly. Data producers see changing priorities, not more work. Consumers have a clear understanding of what work is done when and why instead of silence or a link to a Confluence or JIRA page. Good data mesh product management isn't only focused at the data product level or the platform. You need to look at the bigger picture of how everything fits together to drive even more value. Make sure you have thought coverage of mesh-level product management. Make sure everyone is aligned on how data engineering work supports business value and the organization's goals. The exec team should understand it of course but don't skimp on informing the data engineers - if they know more, it can help with better prioritization. Data ownership - and buy-in on that ownership - is crucial and can have immense value. You can drive that buy-in by showing people the consequences of centralized data ownership creating bottlenecks. What more value could have been created if things weren't stuck in the JIRA logjam? To communicate the cost of data issues, it's very hard to try to put dollar amounts on them but you can show people the business impact and they can start to calculate it themselves - 'this report takes 5 people 3 days to clean up at the end of the quarter' - that has a big impact on business people who can calculate costs far better than the data team. Work with teams to assess and document wins with data. You might not get to exact numbers but you can show value with a little bit of partnering. Then share those wins and attract others that want some of that data-enabled value creation :) Look to business partners for setting prioritization around data work. See if you can get them to seek the budget to do the work with you. What are the company's priorities and where should you be investing to support those? How does data work play into that? Set yourself up to learn and iterate when it comes to architecture. Prepare yourself to fail, and fail fast. It's way too hard to try to get everything right up front, you'll expend too much energy. Embrace that and get moving. Engineering teams need to be part of key business decisions. Not necessarily as a decision maker but engineering finding out about changes once systems break - this is a pattern that has to change. If you want to be data-informed, data-driven you have to consider the impact to data as part of the product decisions/changes. You should make sure you support new data owners with technology and capability building. You need to make them capable of owning data, not just give them the responsibility. And that won't be free. In data mesh, seek out teams that will give you blunt and honest feedback and are willing to work with you to improve your data mesh implementation. You want people who will push you to do better and share in the pain and the success. Focus on communicating what data mesh changes for your organization, what value it unleashes. Really talk with your central data team and explain what it means for them and that it's not a threat to their jobs, they get to focus on cooler things than building pipelines. ?Controversial?: Don't look to rebuild your entire tech stack to do something like data mesh. See how far you can get with a lot of what you have already. Srinivas started by saying how important architecture is, yes, but how it's too hard to try to get everything right up front. Instead, set yourself up to be able to change your architecture as you learn more. Fail fast is crucial, you need to be able to get moving sooner rather than later. Otherwise, there is too much risk in building something that doesn't fit needs or even more likely, never building anything because you're always waiting for the perfect technologies. To do engineering - especially data engineering - well, Srinivas believes engineering needs to be part of the decision making process. If not providing input, at least then being aware of strategic shifts and working with key stakeholders to shift systems to better align to new changes. You can't change your business model and not expect it to not require a shift in what data you need and how you work with it! And you could have started collecting data sooner for the business model shift too :) As data mesh aficionados know, centralized data engineering can create bottlenecks - and almost certainly will at scale. Srinivas recommends you show people the impact of these bottlenecks through specific examples. You can use that to drive better buy-in for decentralized, domain-based data ownership. It can be hard to quantify the exact impact but it's important to try to at least communicate the outcome of those bottlenecks, how did it impact day-to-day business operations and capabilities. When trying to push data ownership to domains, it's crucial to make sure you give them the support to actually own the data per Srinivas. That's technology and that's capabilities/understanding. If you are saying data ownership has value and you can show the value, then the organization should support that value, it's not free, it's worth investing in. Srinivas believes you should focus on making gradual changes rather than sudden shifts in a data mesh or other large-scale data implementation. There needs to be commitment to making change to do it right. And as part of that, look to create and foster close collaboration with users. You need teams to be blunt and honest to help you get to where you need to be, to get to a valuable outcome. That feedback will help you improve your processes and platforms. If Srinivas could give 3 pieces of advice to his former self about data mesh: #1 On ambiguity: it can be helpful rather than binary right or wrong type thinking. So get comfortable with ambiguity, the world is rarely black and white. But ambiguity should be more about the ways of achieving or the expected end result - make sure to set responsibilities as clearly as possible because ambiguous ownership rarely works out well. #2 Get product management involved earlier. They should be there from the start. Understanding product ownership is so crucial to getting data mesh right. You need people who are focused on prioritization and focusing on things that are valuable to the business. If it doesn't have a clear value, should you be doing it? And then when you establish value, communicate it! #3 Look to create a data mesh/domain enablement function. A team that is specific to helping additional domains figure this out. It's not easy and you will find great advocates that can really help teams get moving quickly. A team of 5-6 people would probably be a good enough size. Scott note: see episode 48 with Scott Hawkins for a great example of this internal consultant team in a box approach Another thing Srinivas learned looking back on his time at Zillow is to focus on communicating to people what does data mesh change for the organization and especially for them. There is a vague sense of data mesh changing the way we work but what is the actual value we expect to drive. Not a specific dollar amount but what's the vision of the organization of once you reach a relatively data-informed, data-driven state? Faster reactions to market changes? Better identification of new opportunities? Significant cost savings? And again, talk to people about what changes for them, driving value and also responsibility/role wise. If you want a somewhat visceral approach to showing people why the central data engineering team has become a bottleneck, Srinivas recommends asking how long would it take to clear your current backlog with your current team if no new tickets came in. What about how big of a team would you need to actually clear your backlog based on the number of tickets coming in - is anyone's backlog actually decreasing? What could your organization be doing if you didn't have that bottleneck? What more value could you be creating by really enabling teams to understand and leverage the organization's data? Yes, there will be a cost but we have to invest to create value :) When driving data mesh buy-in, Srinivas again went back to the need for product management in general. Talking with the producers about why this matters; how you are shifting their prioritization, not just adding additional work; how they can actually achieve this and what are meaningful milestones / incremental value deliveries; etc. Prioritization - and communication around prioritization - are likely to lead to your easiest, happiest paths. Data mesh product management should not only focus at the micro level - the data products and the platform - but also at the mesh level according to Srinivas. To really drive value at scale, you need interoperability and things working in harmony. Product management is not just about managing the product but the entire suite of products. Think about your data products as an overall suite of information to serve many use cases and drive a huge amount of value. For Srinivas, when looking at setting your priorities for data work, start with what are the overall organization's priorities, what are the priorities of your business partners? If you want to do data work that isn't valuable to them, will it be valued even if it drives the expected value? Look to find data work that are both valuable and valued and that support the overall organization's priorities. Scott note: this is especially key as you are getting going on a data mesh journey. Everyone on the data engineering team should understand how their work supports the company's priorities and drives value for Srinivas. And the exec team should understand how the data engineering work maps to value creation too. Sometimes, that can be harder with platform work and the like, but it's important for everyone involved to understand how the data engineering work drives value. That helps map to prioritization decisions as well. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of March 26, 2023 33:10

۲ years پیش33:10

33:10

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#207 Zhamak's Corner 20 - Crossing the Data Value Chasm 19:19

۲ years پیش19:19

19:19

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Sponsored by NextData , Zhamak's company that is helping ease data product creation. For more great content from Zhamak, check out her book on data mesh , a book she collaborated on , her LinkedIn , and her Twitter . Takeaways: People are seeing early value from data mesh but we can see where it could be a much greater amount but we are still held back by the organizational challenges and even more by the tooling. In many cases, the tooling isn't good enough yet to change developer behavior - if we remove friction, they will likely want to lean in more on data mesh. We have to find catalysts at the micro level in data mesh to make massive shift at the macro level. We can't try to change everything through pure force of organizational will. But we haven't found these catalysts yet. It's easy to get lost in the vastness of change in a data transformation around data mesh. Try to focus more at the micro with a goal of creating cascading reactions to drive the macro. We need to make mesh data products the first class primitives of information sharing - make them the basic building block of how we create our internal data/AI ecosystem. Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Data Mesh Radio episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#206 Learnings from Delivering and Then Measuring Value of Data Mesh Work - Interview w/ Ghada Richani 1:17:49

۲ years پیش1:17:49

1:17:49

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Ghada's LinkedIn: https://www.linkedin.com/in/ghada-richani/ In this episode, Scott interviewed Ghada Richani, Managing Director, Data, Analytics, and Technology Innovation at Bank of America. To be clear, Ghada was only representing her own views on the episode, not that of the company. Some key takeaways/thoughts from Ghada's point of view: "What I know today is going to change tomorrow." Data mesh is a journey, don't try to get too comfortable, we should always be trying and iterating. Be an explorer. Scott note: strong agree and so does Zhamak Speed is always a challenge with data mesh - some want to move too fast but others want to boil the ocean to make everything that comes after extremely fast. Work with people to take them along the journey and be part of the decisioning process, don't get ahead of yourself. Take your stakeholders lockstep along your journey, keep them very informed. Let them control prioritization. That way, they can see what changes are happening and why delivery timelines are extending - they made those calls! Expose the evolution of the data product to stakeholders. They can then understand tradeoffs, especially in regulatory or other governance challenges. And again, they own the prioritization :) Don't start with generic requirements, start with stakeholder deliverables. The requirements will emerge from that conversation - stakeholders don't necessarily know what is required technically or as a product structure, but for sure they know what they want to achieve from a business perspective. In data mesh, data producers should be treated as a stakeholder as well. Make sure they are engaged and that they are getting something from the process. That can be credit/visibility for value creation, true ownership instead of demands/requirements, additional insights about their own domain, lots of information about why this matters, etc. When driving buy-in and/or getting approval, you have to know your audience. That might seem obvious but it's really not. What do they care about? Have you actually talked to them about what they care about? How can you win them over? How can you make it make sense to them and excite them? When trying to get approval for a big project, break it down into tangible pieces. If there are 20 things that could be improved around a high-value process, look at them discreetly and deliver more and more of the 20 over time. You get a budget to fix 1 or a few and then prove out value and get a budget for more. If something is a priority for your team but it isn't for another team that you need to partner with, lean into that friction. Why is it a priority for your team and not their team? Should that be pushed up the chain to align better? You can't make it a priority by pressing them, you need to work the right levers. ?Controversial?: One lever you can pull to drive data producer buy-in is the opportunity cost of not acting. There is a reason for a use-case, something that will improve a process, drive value. If you ask the reluctant data producer to own the risk, the cost of not acting to improve that process, most will say no way and will participate. Measuring the value of data work is very hard and pretty imprecise. It must be a collaborative process with stakeholders - they are the ones who derive the direct value from the work. How much value is there in speeding up time to deliver data products by 40%? Your data/platform team can't know. It's crucial to build an environment where data failures fall on multiple stakeholders. And that a data product or other data work not meeting the expected value isn't necessarily a bad failure - your hypothesis maybe simply didn't prove true. Limit the costs and size of those failures but if you aren't failing, you probably aren't taking on enough risk. ?Controversial?: For high profile, high visibility projects/data product builds, check-ins 2-3 times a week is normal and is often helpful. You can identify and attempt to mitigate challenges and risks as they emerge. ?Controversial?: Create a highly visible accountability model for stakeholders and make sure they are aware. It's not about calling people out but holding them accountable and if they aren't doing what's necessary, the executive sponsor should know and can address it. Virtualization at the query layer has allowed Ghada and team to mature the underlying data products over time while still presenting a mature and complete data product experience to users. Without virtualization, data consumers would not be nearly as happy with the data mesh implementation. A data virtualization layer has made interoperability easier as well. Connections between data products still need to be found but then they can be codified and offered as a view for others to use. Discovering and establishing domain boundaries is crucial in data mesh. Sometimes, "orphaned" data that isn't really owned by a data domain might live there temporarily but you should always be looking to find/create a true owner if people are relying on that data. Your domain boundaries will change and that's okay. Be ready, be vigilant for measuring if it's time for boundaries to change. There isn't a silver bullet way to approach this but with experience, necessary boundary changes will start to become more obvious. It's okay to have more than one data product per data domain but make sure they are truly incremental to each other. The boundaries and the governance are more important than the number of data products in a domain. Every data product should have a specific purpose. Not serve only a single use case, we need data reusability. But don't add too much scope to a data product. Ghada started discussing balancing speed, structure, and control in your data mesh implementation. There are those that want to build everything upfront and boil the ocean but there are also those that want to get to value as soon as possible without taking the product mindset to heart. Work with both sets of people to keep them deeply informed and show them why a balanced approach works better. If stakeholders are very close to the journey, they won't be pushing back on timeline - they can see where prioritizations are changing and the learning is happening as data products - or other aspects of your mesh - are being built. In fact, let them control prioritization where it makes sense so they are the ones causing timelines to stretch and they made the tradeoff decisions. Keeping stakeholders closely informed also has benefits around control in Ghada's experience. They can understand the tradeoffs relative to governance challenges like regulatory compliance. Exposing the actual evolution of the data product itself to stakeholders helps stakeholders feel comfortable with the process and that compliance/regulatory concerns are addressed. For Ghada, starting from requirements for a use case doesn't work well, people aren't sure and they get stuck in the details instead of the big picture. Instead, work with them to focus on what they are trying to achieve, what their deliverables are, and then work backwards to figure out what they need to meet their own deliverables. And those deliverables better be tied to value somehow :) While driving buy-in from data producers, Ghada recommends making them a clear stakeholder in the process. She's found that really deeply informing them of how their data will be used and the value it will drive often gets them excited to participate. Of course, you need to work with them to prioritize the work but showing them the value - or potential value - of a great use case often helps set that prioritization. You also want to make sure to highlight their work, either for them or preferably making the stage for them to present the value delivered from their data, giving them credit and visibility. When you do those things and give people true ownership, not just requirements, many data producers are far more willing to get involved. In general, when trying to get approval for data work, while Ghada recognizes it can be hard, she has a few good approaches. One is to look at different aspects of what you are trying to improve. Say a process or product line drives significant value for the company. What could you do to tangibly improve the value it delivers? Not as one giant project, break it down into more tangible improvements and seek a budget to tackle one or a few so you can prove out value, getting a budget for additional improvements. Another is to know your audience. This might seem simple but really, you have to learn what drives your counterparts and find a way to communicate the benefits in their own language and address something that matters to them. Make it digestible and hard to resist wanting to tackle the challenge. It's definitely more art than science. One way Ghada has found to drive buy-in from reluctant data producers is to assign the cost of not doing something to them. Essentially, there is a benefit, a value to doing the proposed work - whether that is increased revenue, decreased cost, decreased risk, increased speed, etc. So, there is a negative of not doing the work the proposed work and you ask the reluctant data producer to officially own the cost of not doing that, own that business risk. Many have become far less reluctant to participate :) For Ghada, there are two ways in general to measure the value of data work - economic value and impact value. Economic value is slightly easier to conceive if not that easy to measure - if you make improvements to a process or say create a new product line, you measure the incremental revenue it drove or the amount of cost savings. Impact value, the team(s) impacted by the changes have to give the value measurement - what is the value of speeding up a process, improving the data quality, lowering the associated risk, etc. Neither are exact measurements so it's crucial for stakeholders to understand that it's about triangulating and assessing value, not an exact amount of return. And the stakeholders again have to be the ones that assess value. Only they can say what an impact would mean for them, the data team doesn’t have the context to do that. And you need an organizational environment where the forecasts are seen as forecasts, not commits. It's okay to have failures in data work, according to Ghada. As many past guests have also noted, experimentation is about trying, learning, and iterating. Sometimes the learning is that this won't work or isn't worth the effort. Getting to that learning quickly and iterating to value - or stopping work when that's the right call - is crucial to driving significant value from data work. Your culture must allow for failure or you just won't take on initiatives that are higher risk but higher reward and where the reward justifies the risk. You need to see getting to value and getting something directionally right as a win so you can iterate towards more value. In Ghada's experience, for high profile, high visibility, high intensity projects/data product builds, it's not unusual to check in 2-3 times every week with all the stakeholders. While it may feel like overkill, you can find miscommunications or friction early and even more importantly you can identify and work to address challenges and risks as they emerge, e.g. if someone is disengaging. Instead of the data team going off and doing a bunch of work to deliver at the end of a months-long project, it's tight feedback loops and iteration and changing priorities through close collaboration. And have a highly visible accountability model - if someone isn't delivering, that should escalate to the executive sponsor to figure out prioritization and an appropriate response. On the platform side of things, Ghada is very happy with their use of data virtualization for their virtual query layer. As teams have learned how to build and mature their data products, data virtualization has meant they can expose what a mature data product looks like even when the underlying data product is not yet mature. The underlying data creation and curation process is not fully productized or robust in many instances but consumers don't have to care. The views presented to users are controlled by subject matter experts and serve as a type of interface or output port of a sense. More on data virtualization: 1) Sometimes, that virtualization layer can lead to query performance challenges but usually, that's tied to someone trying to do too large of a query all at once instead of breaking it down appropriately. 2) Data virtualization has made exposing connections between data products much easier. It's just creating another virtualized view. Connections need to be discovered/surfaced manually but beyond that, it's quite easy to do interoperability if the data fits well together. Discovering and mapping domain boundaries is really crucial in data mesh according to Ghada. And it will get easier as you go along. You really want to consider what are you trying to accomplish with a data product and not have many things loaded into one data product, or it will become overloaded and be hard to evolve/improve well. Data owned by a team that are not the subject matter experts is a likely occurrence but you should look to rectify it quickly. Teams building data products that consume information from upstream data products should not take unnecessary dependencies. At BofA, they create a virtual view that combines the upstream data from the source data product with the data product rather than that downstream data product taking a dependency. They are also having many domains that are represented by one data product but some have more than one data product. The boundaries and the governance are far more important to get right than trying to match a certain number of data products to domains. Every data product should have a defined purpose in Ghada's view, that's how you can find your data product boundaries. But a data product should also not take on additional purposes, that's scope creep. That doesn't mean it can only serve a single use case, reusability is crucial but when someone tries to find the right source for accomplishing a goal with data, it's best if they have to consider fewer options but still get all the data they want/need. Yes, easier said than done :) Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of March 19, 2023 21:54

۲ years پیش21:54

21:54

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#205 The Gartner Drama Files - Mesh Musings 45 19:49

۲ years پیش19:49

19:49

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#204 Driving Towards Data Driven - How to Add More Data to Your Org's Decisions - Interview w/ Stephen Galsworthy 1:07:57

۲ years پیش1:07:57

1:07:57

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Stephen's LinkedIn: https://www.linkedin.com/in/galsworthy/ In this episode, Scott interviewed Stephen Galsworthy, former Head of Data at TomTom. Obviously, given he's no longer with the company, he was only representing his own views. Some key takeaways/thoughts from Stephen's point of view: Just because the products you sell are made from data, that doesn't mean you necessarily have a good process for leveraging data to make your products better. You need to embed information collection into how you create products, how customers interact with your offerings. The AI Flywheel: If you can get good information on product user experience, you can feed that into AI systems that generate incremental insights to improve your user experience even more. Hopefully, that generates more users which generate even more information to improve even further. If data collection on usage is not part of your business model - for a number of reasons - it can be hard to convince customers and/or partners to enable that data collection. Even if it's simply to improve the user experience. Try to add it in to your product development as early as possible. If your organization is data hesitant, look for existing success stories from data. Look for something that couldn't have happened without the data. And then share that success internally to drum up more interest. ?Controversial?: Data should rarely be THE deciding factor. Data should be a touch point that can strongly inform or give clarity. Use it for giving clarity or measuring as you iterate. Help execs understand it's not magic and that it's not all right or wrong. It's easy to trust data when it confirms your intuition. How do you use it when it doesn't? How much credence should you give the data? ?Controversial?: High performing companies tend to be those that use data to help make adjustments to their product strategy, using it as a tight feedback loop. It's not as though the data decides but it constantly informs and confirms (but yes, not absolute confirmation). Don't fall into the trap of collecting all the data you can. But do think about what you could do if you had additional data, what might that inform. Work ahead, you don't get past data the day you implement. There is a big difference between producing data and producing data as a product. But if we don't incentivize and assist teams to produce data as a product, few will and then your data practices will remain fragile. Data producers need to be able to understand who has taken a dependency on their data but it's too hard in general right now for them to understand. Better technology offerings here can help. A great - possibly the best - incentive for a team to produce their data as a product is what their data consumers can provide back to them in the form of insights. How can a team managing an app get more insight to improve their app by sharing their data? That's a cohesive org-wide data practice instead of domains only focusing on themselves. 'In production' should not be seen as the end or the key goal, that's a project mindset. In production just means the start of a new phase for your data product. To get a team to data-driven, find a data ambassador within a team - and if there isn't one, deploy one into that team. Really encourage someone to be the lead of wrangling their data. Work to develop a passion for data in someone in the team to spread to the rest of the team. Scott note: staying away from infectious analogies The most data-driven teams have a mindset of always be experimenting. It's not about being right at the start, it's about constantly trying, learning, and getting better. Stephen started off with a bit about his background from working for companies consolidating information into a product they sell, such as TomTom taking in tons of data to create maps to sell. However, using data as a key aspect of your products doesn't necessarily translate into collecting and analyzing information about how your products are actually used which would allow you to drive improvements like new features or even new products. Organizations that are focused on obtaining and analyzing information of usage and other market dynamics are likely to be those that win in the market more often going forward. Their user experience knowledge is going to be a much tighter feedback loop. The AI flywheel is something Stephen mentioned that can create a virtuous cycle. You are creating data around interaction points with your products that feed AI to make the products better. The better they are, the more people interact with them generating more interaction points, meaning you can make your product even better. Essentially, collecting and analyzing data to make your product better -> you make your product better -> more people use the product -> more data to analyze to make your product better. However, if it's not inherently part of your business model, trying to pivot to that information gathering practice can be a tough sell for customers and/or partners. Oftentimes, partners aren't even allowed to hand over data. Think about how you'll effectively collect data as early as possible even if you don't start collecting it then. A good way to build momentum around the importance of data to an organization is start small according to Stephen. It might sound great to try to convert the entire organization to being data driven at once but Rome wasn't built in a day. Show some successes from working with data, find something that couldn't have been done without leveraging data like a new product, and then share those successes internally and encourage more parts of the organization to try leveraging data. Stephen believes that data should rarely be THE deciding factor in decisions. The human in the loop is crucial. And it's crucial to make that clear to execs - you don't want them to think data is a magic wand / silver bullet but they also don't want the data making the decision. Data is a touch-point in decisions, it's used to create better feedback loops as you iterate towards good solutions. Leveraging data well often isn't really about making the right call, it's about finding better and better ways. It's very important to frame the role of data in your organization well - it's what differentiates high performing organizations in many cases based on Stephen's history. Again, it's not a silver bullet but data makes it so you can make smaller bets to quickly get to a better solution via tight test, learn, and then iterate loops. Again, you want to make data a companion to execs instead of an either/or to their intuition and experience. Data can help them arrive on the right decision. Stephen laid out a data maturity journey for an organization or a team: you need to start with collecting data - it seems obvious but you need mechanisms to actually collect data as noted earlier. Once you start collecting data, you don't need to become the most data driven team in the world overnight; start to process and analyze the data - find bits of information and insights to assist as you decide if there should be a formal data analyst/data scientist or not. From there, drive to faster experimentation and improve. The faster you can get solid information and iterate, the better. That will naturally lead to more data democratization as people see the value of the data and want to get involved. That can easily build from a team level up to larger organization level as well. But it's a journey, it's not a switch to flip. Being cognizant of the cost of data acquisition has been relatively easy for Stephen since his past employers have been involved in hardware/IoT and there was a distinct cost to pipe the data back into the organization. But it's easy to lose sight of the cost of data collection for many organizations. He recommends looking at what data you want to collect by specifically what you want to figure out and why it might inform a decision. Just collecting data for the sake of data isn't good. Scott note: you really want to have these conversations as early as possible because the earlier you have the data, the more you can shape your decisions. If XYZ metric will be crucial for a decision in 6 months, you don't want to start collecting the data for XYZ in 6 months. For Stephen, the challenge of data producing teams understanding downstream dependencies on their data is getting better but is still not fixed. But we shouldn't focus on just understanding who is using our data - it's the difference between producing data and producing data as a product. If you are producing data as a product, you should be actually interacting with your consumers so it's inherent that you should know who is consuming - and crucially, you know why they are consuming. But there is of course a cost to producing data as a product, lots of engineering time, especially if you don't have the tooling and capabilities, so that should be incentivized whether you are doing data mesh or not. The best incentive, the 'best carrot' rather than stick, for getting teams to really work on sharing their data as a product has been the insight flywheel from Stephen's point of view. The team sharing their data, if the consumers then generate many useful insights that are helpful back to that producing team, that's how data at an organization-wide scale hopefully works. The 1+1+1+1 = 10 kind of approach but it's also hard to make sure that will happen and that teams make sure to give information back to the producing team as appropriate. You want to try to foster an org culture of 'if another team wins from my data, that is a win for our team too' but easier said than done :D In wrapping up, Stephen talked about how can you get a team that isn't data driven to start heading down that pathway. There was the data maturity journey mentioned earlier but this is about developing a passion in someone on that team for data and then letting their enthusiasm for - and results from - data get everyone else on board. You may have to deploy someone into that team but make sure there is someone in the team driving them to be more data driven because of passion instead of simply try to use the stick. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of March 12, 2023 14:41

۲ years پیش14:41

14:41

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#203 Panel: Making Privacy Practical and Scalable in Data and Data Mesh - Led by Debra Farber w/ Samia Rahman and Katharine Jarmul 58:53

۲ years پیش58:53

58:53

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Debra's LinkedIn: https://www.linkedin.com/in/privacyguru/ Debra's Shifting Privacy Left Podcast: https://shiftingprivacyleft.com / Katharine's LinkedIn: https://www.linkedin.com/in/katharinejarmul/ Katharine's book: https://www.oreilly.com/library/view/practical-data-privacy/9781098129453/ Samia's LinkedIn: https://www.linkedin.com/in/samia-rahman-b7b65216/ Quick acronyms to know: PETs - privacy enhancing technologies; SMEs - subject matter experts Scott Note Warning: there is some nerding out about how awesome it could be if some advanced privacy approaches and PETs were implemented at a broad scale across the industry to protect individual's privacy. It's pretty early days so warning about getting your hopes up :) In this episode, guest host Debra Farber, privacy expert and host of the Shifting Privacy Left podcast facilitated a discussion with Katharine Jarmul, the author of the upcoming book Practical Data Privacy and Principal Data Scientist at Thoughtworks (guest of episode #157) and Samia Rahman, Director of Data and AI Strategy and Architecture at life sciences company Seagen (guest of episode #67). Scott note: given this is a newer area, I wanted to share my takeaways rather than trying to reflect the nuance of the panelists' views. This will be the standard for panels going forward. Scott's Top Takeaways: Privacy has been dominated by risk compliance historically but it's starting to move past the defensive governance aspects. Privacy has mostly been at the tail-end of the development cycle across systems and data but is starting to shift left across the board much like aspects of data mesh. Regarding data mesh, given how many additional aspects of development we are asking domains to own, is it fair to ask them to own privacy as well? How can we train people to understand when to use privacy enhancing technology and then make it easy to implement those decisions? Much like other aspects of the self-serve platform. Privacy tech is emerging and maturing at a very significant rate. What was once a pipe dream or was prohibitively expensive is much closer to being available to the masses. Much like data mesh in general couldn't really have been done before cloud-native tooling/technologies started to mature, privacy is in a similar wave forward. If you want to hear more about specific tech, there's some talk in this episode and Katharine's interview (episode #157). With the explosion in upcoming privacy-focused legislation across the world - much of which is at least slightly different from each other - we will see a large increase in the need for organizations to do privacy well/better. Shifting left is really the only way to do this scalably or we'll potentially see organizations _stop_ doing a lot of currently valuable data work because the cost of privacy and risk compliance becomes prohibitive. Upcoming legislation may be the thing pushing privacy forward more than anything else. Privacy is extra crucial when dealing with data leaving the organization, whether in partnership or for those selling their data on a marketplace. Mozhgan Tavakolifard, PhD (episode #154) talked about this where many companies are opting to merely package and sell insights because they can't track usage deep into those partner or data purchaser systems. It's a bit like what Zhamak discussed re data going into the data science area of an organization right now and all governance and visibility at best gets hazy in Zhamak's Corner 18 (episode #195). Will cross-org data mesh help address that? Probably but it's 5+ years away. Paraphrasing Debra: At the end of the day, privacy is not about compliance. Privacy is about respecting the humans behind the data, not just the data itself. Protecting the data itself is about risk to the organization, that's compliance. We need to encourage a mindset of how do I respect these human's choices and their desires in this context of collecting the data, that's essential. If you don't do privacy well, there are risks to the company of course. But a big one is that people will still look for - and usually find - ways to get access to sensitive data. People will seek out the value. If you make data easily accessible with the right privacy levels, you can unlock many high-value new use cases in compliant and low-risk ways. Organizations should start to look at the rewards of doing privacy well, not only the risks of doing it poorly. Other Important Takeaways (many touch on similar points from different aspects): New ways of doing privacy are going to mean "measurable, quantifiable, verifiable, and auditable tools and capabilities." We need to think of privacy like any other tooling in the development lifecycle. It's about providing abstractions to make the easy/right calls to the domain experts and having a central support structure when things are more tricky. Privacy isn't just about the data, it's not simply a metadata-like concept people are trying to add back privacy to data at rest. Much of the important aspects of privacy are about how the data flows through systems and privacy in those flows and each of the systems, not just the end place it gets stored. In data mesh, the self-service platform will need - currently needs? - to provide privacy-as-code capabilities and so people can easily build data products with privacy built-in instead of added at the end of the process. We can't expose the tech, that's far too complex. How do we provide the good abstractions to make this easy and thus scalable? We need an ability to almost have a privacy capability as an ingest mechanism - point at a data source and say "we need this anonymized" and it's not a super custom build by the producers. We're just at the start of developing those types of capabilities but we need to make it so it's not all on the producer, consumers can consume with privacy on demand. We need policies as code or other easily digestible forms of policies - and compliance - and need to train our people well on what privacy means, why it's important, when to apply, etc. We need tooling to help with federated privacy because otherwise, there is too much privacy context/knowledge AND technology to learn and it won't be scalable. It appears there are some tools emerging but it's still seemingly early days. Anonymization is often pretty easy to overcome if you just add additional datasets. This is especially a risk in sharing data with other organizations. Anonymization isn't a wand you wave and all your privacy risks get taken care of. How can we still derive the value of anonymized datasets? It's often much harder so will companies do the ethical privacy aspects or only the required aspects of privacy? We need better, easier PETs to make it easier to still extract value from anonymized data. How do we balance enough privacy training and not hit information overload? It's hard to get people to learn what's necessary because privacy is such a big topic. We need global and domain-level policies that can again be actually digestible. Can we measure time to compliance, time to privacy, time to 'doing the right thing' ethically? That would be best to understand where we need to improve but we're probably just at the start of that. This is an interesting fitness function area. Subject matter experts (SMEs) have so much specific knowledge that you need to leverage them to discover privacy risks and privacy rewards too. Much like any aspect of governance, trying to have the central team make decisions just isn't scaling so we need to make the people in the domain capable enough to handle privacy. Privacy rewards: in many organizations, there are very high value sets of data that cannot be leveraged for specific use cases due to privacy and other compliance restrictions. Getting to a place where we can easily leverage that high risk but high value data will potentially unlock large amounts of business value. There are lots of instances of teams finding those high value data sets and using shadow IT to get at them. If that's the only way people can get access, many will completely skirt any compliance and privacy. So getting to a place where they have access but it's according to policy and tracked is crucial to lower organizational risk - from compliance and ethics wise. From Katharine: "But if you build easier ways to get access and safer, more responsible, more ethical ways to get access, then you have a win-win situation and people are not going to find shortcuts." Companies are starting to loosen the shackles on data and focus on maintaining privacy but also enabling innovation around privacy-sensitive data. That's a great mindset but there are still many questions on how to do that specifically. Too many, especially in blockchain, conflate privacy and confidentiality. Keeping something confidential is a security aim. So if you focus on confidentiality, you can't actually use the data you have - no one is allowed access, it's on lockdown. We need to get far better at risk modeling for privacy. What are the potential harms by the humans? We need to move beyond only thinking about if there might be a breach, what data might they get. We can free up data for far more uses if we do this right but ethics around data usage is just not a common thought. We need to train people to think ethically and about potential harm. There are multiple issues with anonymized data. Are you taking away the utility? Are you fooling yourself into thinking it can't be de-anonymized? That's a pretty common outcome with adding additional data sets - especially a risk if sharing data externally. Don't treat anonymization as your hammer and everything looks like nails. We need to teach developers about differential privacy which is about "bounding the probability of someone learning" a specific thing. Differential privacy "got a bad reputation" but we can add noise and maintain accuracy now. It is the "gold standard for anonymization". Healthcare patient data is one of the biggest challenges in privacy because you want to maximize the efficacy of care but also maximize privacy. And then how do companies take the data of the individual to extrapolate further to see broader trends? We need to get people upskilled so they can understand when to transform data in privacy preserving ways - and then the self-serve platform needs to make it easy for them to do that. But we don't have great industry-wide understanding on how to do either of those that well yet. Self-sovereign identity, while very interesting, is probably a long way away from being widely adopted. There needs to be a lot of industry collaboration and agreement and it's not really a big benefit in many areas based on the legal requirements versus cost. It would be great for company-to-company interoperability with privacy but who will build it? The 3 panelists were very excited about it though :) Privacy and data sovereignty are going to be intermingled in interesting ways in data mesh. Querying data where it is instead of piping it all over the world* will help maintain privacy and comply with laws - many countries don't allow data to be exported as is. * see Zhamak's Corner 13 episode #173 that covers some of what querying data where it is means and that's not necessarily about source systems but it does mean not moving it without necessity Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#202 Creating a Balanced, Sustainable Approach to Your Data Mesh Journey - Interview w/ Kiran Prakash 1:22:35

۲ years پیش1:22:35

1:22:35

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Kiran's LinkedIn: https://www.linkedin.com/in/kiran-prakash/ Kiran's article on the 'Curse of the Data Lake Monster': https://www.thoughtworks.com/insights/blog/curse-data-lake-monster In this episode, Scott interviewed Kiran Prakash, Principal Engineer at Thoughtworks. Some key takeaways/thoughts from Kiran's point of view: ?Controversial?: You MUST have exec sponsorship to move forward with your data mesh implementation. You need the top-down push for necessary reorganization when the time comes. Scott note: only kinda controversial, really more often ignored :D ?Controversial?: Data mesh, if done well, doesn't need to have a huge barrier to entry. That's a misconception. If you think about gradual improvement/evolution, you'll be on the right track. "The Curse of the Data Lake Monster" was like the data field of dreams - there was expectation that if you build a great data lake, value will just happen. If you ingest and process as much as you can, the use cases will just happen. And it really wasn't the case. So we should apply product thinking to data to focus on what matters. The 'Curse' was a manifestation of Conway's Law - the strong separation between IT and the business led to mismatched goals and subpar outcomes. With microservices, that started to be much less of an issue on the operational plane so why not try with data? It's easy to lose sight of Conway's Law and aim for distributed architecture first but the organizations doing data mesh well are changing their architectural and cultural approaches and patterns together. Don't try to do the architecture first, you really don't know your key challenges yet. It's very important to have a target operating model and get clear on your organizational vision and purpose around data - how will you use data? Once you have an organizational vision and purpose, domains should start setting goals aligned to that vision and purpose. As others have noted, don't get ahead of yourself, work in thin slices for your data mesh implementation. Stay balanced at an overall level between the data mesh principles as you add more and more thin slices but don't try to solve all problems up front. If you modernize your legacy software but don't change the organization, expect to do the same type of modernization in about 5 years. To really get to a scalable approach to data mesh, you should look for organizational and process reuse as much as tech/architectural and data reuse. Move from measuring outputs to measuring value outcomes. Sounds simple - it isn't - and it's crucial to changing your mindset about how you approach data. ?Controversial?: Use the 4 key metrics from DORA (https://dora.dev/) to measure how well you are doing in your software engineering. And a key aspect of data mesh is about applying good software engineering practices to data after all. If you want to measure the value of your data work, you need to break it down into tangible objectives. Ask the owners of those objectives to provide the value of meeting the objectives. Then look to measure how much data work contributed to achieving those objectives. Think of a use case as a value hypothesis: you are making a bet that something will have value. It's okay to be wrong, that's the nature of betting. But limit the scope of your mistakes so you learn and adjust towards value instead of making big mistakes. ?Controversial?: If you don't have a culture where it's okay to fail, it will be very hard to do data mesh well. Scott note: it will essentially be impossible in my opinion Many times what people consider minimum viable product is neither minimum nor viable. This is often due to a culture where you can't test things with users when they are still very rough. That will limit your success with data mesh. However, most people are reasonable so if you read them in that this will be a rough sketch/first iteration, they usually are on board to help you iterate to good. "Architecture is about the important stuff. Whatever that is." - Ralph Johnson (via Martin Fowler) Always think about necessary capabilities and build to those. The most important are those capabilities you need now. Don't get ahead of yourself. The 'data platform' is really a misnomer, there will be multiple platforms. Users care about services, not if you have one platform or five or more. Don't have platform sprawl but don't over-centralize, that usually leads to scaling and flexibility challenges. Kiran started off by talking about a blog post of his with a colleague from 2019 called "The Curse of the Data Lake Monster" - lots of clients were building big data lakes and it wasn't providing the expected value. There was an expectation that if you ingested and processed as much of your data as you could, it would create great use cases and lots of value. But it didn't happen. Value doesn't just happen without concerted and concentrated effort. So Kiran asked why aren't we applying product thinking to data to figure out and focus on what matters to drive value. What would happen if we focused on outcomes instead of platforms? What if we measured value not how many terabytes were processed and stored? A key reason Kiran feels the 'Curse' happened was the strong separation between business and IT. That separation meant both were seeking different goals instead of collaborating. IT was focused on building things instead of solving business problems and the business side was focused on doing what they could, not building out a scalable and robust data practice. Conway's Law in action. We saw microservices really help to tackle those same issues on the operational plane so the data side of the house was definitely ripe for some product thinking-led innovation. Data mesh avoids a lot of the issues of past data approaches by not leading with the technology - the first two principles are not tech focused. For Kiran, many (most?) of the organizations that are getting data mesh right are respecting Conway's Law and shifting their architecture and organizational approaches together. But in thin slices so as to not put too many eggs in one basket and make reasonable progress. And they are getting the exec sponsorship because while you don't want to reorganize your entire company upfront to do data mesh, you do need some top-down pushing to actually drive the necessary org changes when appropriate. According to Kiran, while many people think data mesh has a high barrier to entry, that shouldn't be the case. There should definitely be a target operating model at the organizational level and organizations need to keep that in mind but it's not as though, again, you reorganize the organization all upfront. Organizations also need to really answer the question of what are they trying to do with data and what would doing data mesh well drive for them - if that's not crisp, they probably aren't ready to do data mesh because their business strategy isn't aligned to or contingent on doing data well. Once the organization has the target operating model and vision down, Kiran recommends that domains should start to set their own specific goals aligned to the broader organizational vision. They should work on some hypotheses on how to achieve those goals and how they plan to measure their progress towards those goals. Start to build out your thin slice approach to making progress towards your vision and goals. Don't get super far ahead of yourself, look to progress at a meaningful but reasonable pace and tackle as little as is necessary now while still making sure you are aligning with the organizational vision. Keep your eyes on the prize, don't take on too much now. And yes, easier said than done. Kiran pointed to a quote he read about if you modernize your legacy software stack but don't change your organization, you will need to do the same modernization in about five years. The same goes for data - if you are taking on data mesh from a tech-first approach, you'll just have to do all the same modernization again and you won't get a lot of the potential benefit from data mesh. Decentralizing the architecture will only really change things if you change the organizational aspects too. People, process, technology. For Kiran, we need to really start to focus more on measuring value outcomes instead of inputs - how many terabytes or operations per second isn't directly tied to value. Teams need to have it made clear what is actually valued and valuable. In many large organizations, there are often less clear links between data work and business value so you have to educate and incentivize teams to do the high-value data work. When thinking about trying to measure the return on investment in data work, especially data mesh, Kiran recommends starting by breaking it down into more tangible goals and measuring the value of achieving those goals. Then you can start to say how did the data work contribute to achieving those goals. But a data team can't really know the value themselves, whether inside the domain or not. And by breaking things into smaller goals and objectives, you can more quickly iterate towards value with tight feedback loops. Instead of large-scale projects, you build to larger and larger objectives through breaking things down and achieving meaningful micro progress that leads to large macro value. Kiran talked about thinking of use cases as value hypotheses: you believe it will have value and thus you are making a bet. And it's okay to get things wrong, just limit the scope of the mistake so you can use the missteps as learning so the larger macro bet has a much higher chance of paying off. This is iterating to value, those tight feedback loops. If you don't have a culture where it's okay to be wrong, okay to fail, then data mesh is potentially (Scott note: almost definitely) not right for you. Minimum viable product is often neither minimum nor viable in Kiran's experience. If you can't put something pretty rough in front of stakeholders, you waste far more time and effort building in wrong directions and are less likely to hit on success. But that's often out of the control of the product team. So it's a catch-22: do you put in a lot of effort to get it well past MVP or do you risk losing face? So we need a culture where we can actually do thin slicing well to really derive the most value out of building software, whether that is apps or data products. That incremental value delivery is really crucial to maintaining nimbleness as you scale. If you can't actually deliver in thin slices, it can significantly increase risk as you are making larger bets. But Kiran's seen that if you spend the time to explain the need for thin slicing and that what they are looking at is the 'sneak peak' and you just want feedback, most people get it and are reasonable. But you need to communicate about it :) Kiran used a phrase Martin Fowler uses often from Ralph Johnson: "Architecture is about the important stuff. Whatever that is." When thinking about decisions that are hard to reverse, spend a lot more time and care, but the ones where it's easy to reverse, those probably don't make up the core of your architecture. When building your architecture, it's again important to build incrementally instead of trying to get it perfect from the start. Think about necessary capabilities, not technologies. In data mesh, that is about data product production and then monitoring/observing, data product consumption, mesh level interoperability/querying, etc.* Start to map out what you need and then think what level you need from a capability standpoint and when. You don't need to build out capabilities for when you have 20 data products when you have 1-2 data products. * Scott note: Kiran went on for quite a bit here about necessary capabilities for data mesh late in the episode if you want to listen. Quick tidbit: Leverage the 4 key metrics from DORA to measure how well you are doing your software engineering as applied to data - 1) Lead Time to Changes (LTTC); 2) Deployment Frequency (DF); 3) Mean Time To Recovery (MTTR); and 4) Change Failure Rate (CFR) https://dora.dev/ Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of March 5, 2023 33:11

۲ years پیش33:11

33:11

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#201 Choose Your Blast Radius and Other Lessons Learned Across 10s of Data Mesh Implementations - Interview w/ Vanya Seth 1:21:12

۲ years پیش1:21:12

1:21:12

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Vanya's LinkedIn: https://www.linkedin.com/in/vanyaseth1809/ In this episode, Scott interviewed Vanya Seth, Head of Technology for Thoughtworks India and Global 'Data Mesh Guild' Lead for Thoughtworks. To be clear, Vanya was only representing her own views on the episode. Some key takeaways/thoughts from Vanya's point of view: Data mesh is at a similar inflection point to where microservices was a decade ago. Let's not relearn all the hard lessons they already learned. We should adapt/contextualize to data of course but we can skip a lot of the anti-patterns. Similarly, many people are stuck thinking "there's no way that could work" regarding data mesh like they were when people suggested development and operations be combined in DevOps. It's understandable - it's hard to imagine a post monolithic world when all you've known is monoliths. ?Controversial?: We should try hard to prevent creating the fear of missing out (FOMO) for those not doing data mesh. If data mesh isn't right for your org, especially if it isn't right at this time, that's perfectly okay. Don't take on the overhead cost of data mesh if it won't bring more value than cost. Scott note: PREACH! ?Controversial?: Some CDOs or CAOs, their organizations don't really get the value of data so they are implementing data mesh to try to prove out value and make their mark. That can obviously create issues if their organizations aren't ready. A few indicators an org is ready for data mesh (see below for expanded context): A) data/AI investments are not delivering the promised/expected returns and/or it's hard to point to the value delivered in general from data/AI investments; B) the organization is attempting to throw more people at centralized data management and it's not working (platform included); and C) there's extremely unclear ownership around many aspects of data, especially who owns aspects of hand-offs or who owns the end data asset - how can a consumer actually ask a question about the data with no clear owner? "Innovation in queue syndrome" = your innovation agenda is "in queue" and keeps getting deprioritized because you are dealing with everything else first just to keep your data practice flowing. Use value stream mapping to understand how your organization drives value from business processes and where there is value leakage. Especially useful if data work isn't driving value. We should take a lot of learnings in how microservices service discovery evolved, especially the tooling, for data mesh. There is no need to reinvent the wheel on this. Some existing tooling from the microservices space is just fine for data mesh too. We don't need to invent new tools when existing ones - which are already robust and mature - can be extended or even used as is. Platforms aren't about the tooling, they are about the holistic user experience - how do you stitch things together to automate the toil and let users focus on what matters? The tooling is under the hood, not the main interface. Users of your various data platforms should not be directly interacting with tools for the most part. It should be about abstracting away the tools and making it easy for them to interact with data, not the tools of the platform. !Crucial!: "Choose your blast radius." Far too many are looking to change the entire organization at the start of a data mesh journey instead of limiting scope to a reasonable level. Find one "courageous" domain to move forward. "Nothing succeeds like success itself." Get to a data mesh win that you can tout quickly so others will get bought in and see the value and want to participate. Incremental value delivery builds interest and momentum. Build your platform at the same time as you're building your initial data products. Far too many platforms are built with tools as the focus instead of automating away toil and focusing on necessary capabilities. !Crucial!: Evolvability should be a first class concern when building your platform, just like with any product. You must be able to continue to improve and change to meet needs. Focus on the abstractions and the ubiquitous language - e.g. business people don't care what the technical underpinnings of a data product are, they care about what it means for them and how they can access/leverage it. When starting your data mesh journey, look at the use cases to decide how much of each pillar you really need. Don't overbuild early. If you only need a minuscule amount of governance, great. If you don't actually need the producing team to be overly involved in ownership, awesome. Don't go for full data mesh at the start. What you should focus on relative to your early journey is unique to your own situation and use case. Don't worry about competitors or how others are starting - their circumstances are their own. Vanya started with a bit about her background and how deeply entrenched she's been in the microservices space - that played into the overall conversation a lot. Both Vanya and Scott agree if we want to do data mesh right, we really should take learnings from microservices and DevOps so we don't have to relearn what they already did the hard way. For Vanya, data mesh is at a similar inflection point to where microservices was a decade ago - people were extremely skeptical that developers and operations could even work together, much less around combining them in a singular approach with DevOps. It's hard to imagine a post monolith world when all your career and experience are with monoliths. We have to be somewhat kind to those people in understanding that change is hard and scary :) But, as a counter, for data mesh Vanya believes (and Scott agrees) we must try to prevent creating the same fear of missing out (FOMO) that microservices had. For many, if your organization wasn't doing microservices, it wasn't seen as a cool place to work and that all the best developers were at companies doing microservices. We don't want that in data mesh because it will lead to lots of wasted effort for companies that shouldn't be doing data mesh now or potentially ever. According to Vanya, there are a few really good indicators an organization might be ready for data mesh. Before we get into the 3 she listed, a few things that might be indicative of indicators (Scott note: I know, I know, silly Scott phrasing) are constant displeasure of the kinds of initiatives they've been doing in the data and AI space - there is a constant pressure to prove the value of data and AI investments but really, an inability to do so. Long and lengthening cycles to return on data work/projects. A biggie is an ever-growing platform that is trying to do too much and hasn't been delivered - trying to boil the ocean. So the 3 indicators data mesh could be a good fit that Vanya listed were: 1) Investments in data and AI aren't delivering expected value and it's hard to actually point to the value that is being delivered. Users aren't getting "the right data at the right time with the right quality". 2) Large and growing central data teams where trying to scale is done by throwing more people at the problem and it just isn't working. When automation would be better, they add people. 3) Confusion around who owns data when and why. Who owns the handoff between systems? Who owns the documentation and metadata around data? When someone has a question, how hard is it to find who owns the data? Vanya highly recommends using value stream mapping to understand how you drive value with business processes and especially where are value leakages; this can be data or not, and should be applied to both analytical and operational data processes. You can understand better your business processes and expected outcomes - if something didn't meet expectations was that because expectations were wrong or did something happen along the way to lose value? Value stream mapping gives you an objective and neutral starting point and helps identify problem areas - value leakage - where you can prioritize what to tackle first. In microservices, Vanya pointed to how challenging service discovery started to become until tooling came along - specifically mentioned Consul - so we really don't have to reinvent everything in data mesh. The tools out there, especially those in the open source space, are really making nice progress - specifically mentioned DataHub - compared to where they were 2 years ago at the infancy of bleeding edge data mesh adoption. Overall, we should 1) look to existing tools to see if we can use them as is; 2) look to extend existing tools where possible to cover incremental needs specific to data mesh; and then 3) look to create new tooling that is required for data specific challenges. Again, don't reinvent the wheel. For Vanya, one thing many organizations struggle with in data mesh is the self-serve platform - what is the goal? Circling back to an earlier point, it's not about building the most amazing, ocean-boiling platform. It's about stitching tools together to automate the toil away - how can you create a holistic user experience to focus on doing the value-add? The value of the platform to the users is the abstractions away from the tools that make it easy to focus on what needs to be done to drive value from data, not play with the shiny tools. Focus on enabling interacting with the data, not the tools of the platform. "Choose your blast radius" is a key phrase for Vanya. Think about scope appropriately and don't try to bite off more than you can chew. You don't have to reorganize your entire organization on day one to do data mesh, that is far too much of an upfront cost and makes failure a massive cost. Look at how it was done well in microservices: thin slices, not taking a sledgehammer to the monolith. Gradual evolution is sustainable, a revolution either succeeds or it doesn't - don't take on risk that isn't actually beneficial! "Nothing succeeds like success itself," was another line from Vanya. It's crucial to get to an early win or two to show off to the rest of the organization proving data mesh delivers value and getting them interested in participating. 'Hey, we did this and it was a big win, who's next?!' It's not just about showing value, it's about showing there was a reasonable encapsulated timeline, not just promises. That incremental value delivery creates momentum and the more momentum you have, the more you can get people on board. As many past guests have noted, it is a pretty bad (Scott note: fully terrible) idea to build the platform and then bring it to the users when it's done in Vanya's view. There are far too many unexpected friction points and finding those and tackling/automating away the actual friction is where the platform adds value, not bells and whistles. You want to find those as they emerge and work on tight feedback loops - that's product thinking! And if you don't make evolvability a first class concern, you are not building your platform as a product either. For Vanya, it's pretty easy for tech people to focus on the tech, whether that is in data or not. But the overall organization doesn't care about the tech, they care about what they can do. So it's crucial to find the ubiquitous language and make your implementation and platform about what are people trying to do. The user isn't accessing S3, they are accessing the Inbound Marketing Conversion data product. S3 is simply a mechanism to accessing the data and insights. When considering your thin slice early in your data mesh journey, it's okay to have a very unbalanced slice in Vanya's view. This has been mentioned before but it's important to reiterate. If you only need a bit of one of the pillars but you do need more capability in another of the pillars, that's absolutely okay. Don't build today for all the problems of 6mo from now. You want to focus on tackling the toil of today. Quick tidbits: Vanya's phrase "innovation in queue" is when an organization keeps putting off their innovation agenda for more immediate concerns - everything innovative ends up getting deprioritized in the queue. Most data mesh journeys are taking six to seven months to really prove out doing data mesh and value. Scott note: this seems to be standard for larger organizations but a complete POC means faster follow-on for additional use cases. It's a balance! In some organizations, if data is not really valued the CDOs or CAOs are looking to implement data mesh to show the value of data but their organizations often aren't ready and trying to do data mesh just creates more challenges than benefits. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
{Bonus} Zhamak's Corner 19.5 - How Does AI/ML Change When Trust is Automatic? 11:03

۲ years پیش11:03

11:03

Sponsored by NextData , Zhamak's company that is helping ease data product creation. For more great content from Zhamak, check out her book on data mesh , a book she collaborated on , her LinkedIn , and her Twitter . This episode is part of the greater AI/ML conversation I had with Zhamak but it's super important to emphasize the importance of trust - enough so that I created a separate quick episode on it. Not just trust in the data itself but that there is easy access and there will be going forward. A lot of the things we have done in data historically has been defensive in nature - especially grabbing a copy of the data now because who knows when you'll get access to it again. What if we can implicitly trust that there has been care and foresight in preparation of the data I find, that there is an owner I can ask if I'm confused or curious, that my access won't suddenly go away or that what's there won't suddenly change without my knowledge? In ML/AI, the data scientists have done things in ways that made sense to their situation and challenges. What happens when we make trust inherent? What incremental value does that drive? Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Data Mesh Radio episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#200 Zhamak's Corner 19 - An AI/ML Future Without So Much (Needless?) Complexity 16:59

۲ years پیش16:59

16:59

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Sponsored by NextData , Zhamak's company that is helping ease data product creation. For more great content from Zhamak, check out her book on data mesh , a book she collaborated on , her LinkedIn , and her Twitter . This episode is part of the greater AI/ML conversation I had with Zhamak. To start, Zhamak recognizes we aren't where we want to be in terms of capabilities - ways of working or tooling - to make this a reality just yet. But, if we can make it so data scientists can trust and easily consume from data products - that we create data products that don't care what use case type - regular analytics or AI/ML - can we remove a lot of the complexity they face? Do they need feature stores for data they aren't transforming? If they can get continued access and know the quality, why create a separate process that has fragility instead of trust the data product owners upstream? I wasn’t smart enough in the moment to talk about do we need to have a copy of the training data itself for reproducibility but folks smarter on ML than I am can answer that one, probably in the affirmative. But overall, there is a lot of complexity in the way we do AI/ML because data scientists can't trust the sources of their data and they feel the need to take control because if they don't, their models break. So we need to earn their trust and show them a better way. But again, we aren't there yet, so let's work to make this a reality in the future. Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Data Mesh Radio episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#199 Finishing Your Data Marathon - Driving to Action from Data - Interview w/ Brent Dykes 1:08:00

۳ years پیش1:08:00

1:08:00

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Brent's website and book: https://www.effectivedatastorytelling.com/ Brent's LinkedIn: https://www.linkedin.com/in/brentdykes/ Brent's Data Analytics Marathon Forbes article: https://www.forbes.com/sites/brentdykes/2022/01/12/data-analytics-marathon-why-your-organization-must-focus-on-the-finish/?sh=2af698743c3b In this episode, Scott interviewed Brent Dykes, Chief of Data Storytelling at his own firm, AnalyticsHero. Scott asked Brent to be on after João Sousa pointed him to Brent's content. Some key takeaways/thoughts from Brent's point of view: Focus on the so-what, what should people take away and do from the insights, not the sausage making of the insights. Execs want to eat the dang cake, not hear about how you made it! !Controversial!: You want to get to a place where you remain more neutral until the data informs your view. It can cause more cognitive load to update our views instead of waiting for the data to speak first. Many organizations lose steam in actually driving action on analytics, they don't drive change with data - they fail at least one of the following: generating actual insights, communicating their insights well enough to drive action, and/or actually acting on the insights. The analytics marathon: data collection -> data processing -> data visualization/reporting -> data analysis -> insight communication -> take action. Many companies stop the data marathon after getting to the visualization/reporting step. They aren't driving the results they want so they focus more time on collection and processing instead of finishing the race and get caught in a loop. Really consider why are you doing data work. It's not to simply do analytics, to build the dashboards and reports, it's to take action on the data and affect change through more informed decisions. It's easy to like the data when it supports your narrative. A strong analytics culture bends decisions and thinking to the data instead of the other way around. Companies with good analytics practices, that take action on data, typically have at least an executive sponsor around analytics if not buy-in from the entire leadership team. There is an understanding that actions are driven by data when possible. And there is a test and learn culture. Executive support and a test and learn culture are what drive results from analytics - many companies buy the same tools and have vastly different degrees of success with data. If your organization isn't doing analytics well, the best way to drive towards doing analytics well is get to wins from analytics and build momentum to drive to higher and higher exec sponsors. ?Controversial?: Sometimes to drive necessary understanding, in documentation - or other ways of bringing data users up to speed - you really need to show the lineage all the way back to how the data is even collected. The more you share data, the more likely information is to be misunderstood. Beware the difference between what a metric means and what people _think_ it means. ?Controversial?: To get to scale, we use passive communication - read mostly documentation - about our data. But to truly drive to understanding, we also can't shy away from active - read person-to-person - communication. Scott note: Episode 150 w/ has some interesting insight on how far documentation should go. ?Controversial?: Storytelling is often the easiest way to sway people with data because human brains have evolved to accept information via stories. We've been doing it for 1000s of years. Mature organizations understand the data can be wrong and prepare for that. Move fast and make incremental moves instead of a big bang approach. You learn more and can do better the next time even on actions that weren't as valuable as expected. ?Controversial?: Business analyst roles should evolve to be more like personal trainers - helping people learn how to do good analysis and then communicate their insights. They won't work themselves out of a job, merely get to a place where they focus on the bigger, harder, deeper, more valuable questions. Brent started with a bit about his background and why he titled his book "Effective Data Storytelling: How to Drive Change with Data, Narrative, and Visuals." There are a few places many organizations fall down in driving change via their analytics whether that is a failure to generate actual insights, a failure to communicate insights well enough to drive action, or a failure to actually take action on the insights - he's most focused on the communication of insights, a place often overlooked. You can find the best insights in the world but if you can't communicate those insights well enough, no one will understand them and/or understand the potential impact of acting on them. Communicate well enough to drive change! The analytics marathon is one of Brent's big analogies for explaining where organizations fail along the path to taking action on their insights. There is data collection, which pretty much all organizations do. Then data preparation into data visualization. But this is where many orgs fall off because they are simply reporting on what's happening, the descriptive analytics and not actually driving to diagnostic analytics. Instead of doing deeper analysis, they believe their problems lie in what data is collected so they try to collect more data, thinking it's simply a lack of information instead of a lack of analysis. And then of course, once you do the analysis, you still have to communicate and take action. For Brent, a few common indicators an organization will likely have a good analytics practice include: 1) an executive sponsor for being or becoming data driven. Possibly the entire leadership team. 2) a general commitment to driving actions from data where possible. It's "how we do things." 3) A test and learn culture in the organization that's supported by data. If an organization isn't yet data driven, isn't doing analytics that well, Brent recommends getting to wins and slowly moving your executive sponsorship up the ladder. It might start at a Director level and then after you build momentum, people will take notice and you can climb to VP level and then C-Suite level. It's about showing the value of analytics and plugging along so you have proof points when you move the conversation higher in the organization. Rome wasn't built in a day and neither is a good, organization-wide analytics practice. As data initiatives have become more ambitious, it's often meant ownership has become more murky according to Brent. What was once data that was essentially only for the generating team is now a potential core value asset and driver for the organization. And that opens you up for much more misunderstanding. Focusing on making sure information is understood - not just data is made available - is crucial to making good decisions with your data. There's often what the metric means and what others assume it means. Brent shared his views that we need both active and passive ways of sharing context around data. Passive is the metadata, the documentation and the like. If we want to scale, passive is crucial. Self-service can't just be a pipe dream. But too often, people in data want to only do passive and ignore the people-to-people conversation. But often that's key to nuanced data or crucial to working with key people making big decisions based on data. For 1000s of years, humans have been passing information via stories - human brains have evolved to share information via stories. We inherently want to know where the story goes. For Brent, mastering that storytelling with data and about data is the best way to convey the information we generate and discover with our data. If you don't communicate insights to those who can take actions, in a way they can understand, they won't take those actions :) For Brent, execs rarely want to hear how the sausage was made via data. You want to show them what you've discovered and what they should do with that, not how you came upon that. It can be important to show people the sausage making isn't that hard though, especially trying to enable a team to do self-serve analytics. Really consider which is more appropriate to the situation. It's pretty easy to like what the data is saying and point to the data as backing you up when you agree with it in Brent's experience. But a truly data-driven culture will focus on updating their thoughts and processes based on what the data says, improving their understanding via the data instead of trying to bend the data to support our hypotheses. It's about getting to a place where you try to remain more neutral until you hear what the data says and shape your vision around that. On data-driven versus data-informed as semantics of what we're trying to get to, Brent likes the idea of data-driven. For him, it means really leaning into the data. Part of that is understanding and accepting that sometimes the data is wrong or we didn't ask the question in the right way - that's just getting to data maturity. And data-driven companies recognize the value of learning - there is still value from experiments and moves that didn't have as much benefit as expected when digging into the why. You have incremental understanding even if not direct incremental business value. And it sets you up to do better on the next iteration. When asked about where will business analysts fit in data storytelling in the future, Brent sees them like personal trainers - they won't be doing the work but showing people how and assisting them until they can get to a level the BAs aren't as needed. That's for the analysis and especially the insight communication. Pure self-service analysis is nice in theory but you need a way for people to get help and make sure they aren't hurting themselves :) If more people are far more capable, that means the BAs can focus on the more valuable, large-scale questions. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of February 26, 2023 31:54

۳ years پیش31:54

31:54

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#198 How Do We Make Data Contracts Easy, Scalable, and Meaningful - Interview w/ Ananth Packkildurai 1:23:25

۳ years پیش1:23:25

1:23:25

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Ananth's LinkedIn: https://www.linkedin.com/in/ananthdurai/ Schemata: https://schemata.app/ Data Engineering Weekly newsletter: https://www.dataengineeringweekly.com/ In this episode, Scott interviewed Ananth Packkildurai, Author of Data Engineering Weekly and the creator of Schemata. Scott note: we discuss Schemata quite a bit in this episode but it's an open source offering that I think can fill in some of the major gaps in our tooling and even ways of working collaboratively around data. Some key takeaways/thoughts from Ananth's point of view: !Important!: Collaboration around data is crucial. The best way to get people bought in on collaboration around data is to integrate into their workflow, not to create yet another one-off tool in yet another pane of glass. ?Controversial?: There is so much friction between initial data producers - the domain developers - and data consumers because they are constantly speaking past each other. The data consumers have to learn too much about the domain and the data producers rarely really understand the context of most analytical asks. Data creation is a human-in-the-loop problem. Autonomous data creation is not likely to create significant value because the systems can't understand the context well enough right now. As Zhamak has also pointed out, there is far too much tool fragmentation. It made sense with lots of readily available VC money and finding how to approach things with cloud but we need holistic approaches, not spot approaches to things like data quality, observability, lineage, etc. !Open Source Product!: Schemata was created to enforce certain rules in a cooperative platform around data sharing specific to data schemas to help alleviate much of the above friction. Data needs to really take a lot of learning from the platform engineering for microservices space. They make it easy for teams to test new services or changes, deploy, etc. In data, we are asking domains to own their data without giving them the tooling to easily do so. It's too much of an ask. Scott note: PREACH! ?Controversial?: In general, we need better ways to share about what data is already available and what data we expect. This is where data contracts as a platform instead of tooling only becomes important. !Scott Controversial Note!: Many get data contracts woefully wrong. Data contracts aren't about _only_ the contract. They signify a relationship that has contractual terms but think about a vendor - is your only interaction, communication, set of expectations, etc. the contract or is a part of the relationship with strong guarantees? Consumer-driven data contract testing is important. It is defensive in a way - if my upstream changes, I don't necessarily want to consume from it. But consumer-driven testing can also be a great part of the conversation around how consumers are actually using the data - it's a programmatic way to describe usage to producers. Scott note: if we can make consumer-driven testing easy, great. But we need to reduce the burden on producer and consumer to ensure data contract compliance. We need to be able to take consumer requests and translate those to producers as well as give guidance to producers about effective - cost or otherwise - ways of meeting those requests. E.g. including a user ID might seem easy to a consumer but it could be very expensive for producers in their standard way of calling user ID. How can we put a prescriptive way in front of data producers to make it easy to meet requests? ?Controversial?: Pull requests > requirements gathering. A consumer can show exactly what they want and a producer can either approve or deny but it generates a better conversation. Teams need to figure out coordination and communication in a decentralized data modeling world. That is the federated aspect of data mesh - if it's all decentralized, nothing works with each other and you end up with garbage data at the organization level despite domains having well modeled data for themselves. !Controversial!: Schemata believes there needs to be a core data domain for data that links most other domains. Scott note: While not rare in data mesh, a core domain can become a bottleneck and may not give you the flexibility required. Adevinta (episode #40) discusses leveraging a core domain in depth. It's very, very valuable to provide automated feedback via your platform to people considering creating a new data asset - whether that will become a data product or not - including how well it fits in the organization's data landscape. Are you creating something that can actually be leveraged for other use cases? Does it integrate well with existing data assets/products? It's crucial to have something that gets all the parameters of a data contract on paper. Think about negotiating an agreement with a vendor - is it all just verbal or are you starting from something concrete and working from there? Have your platform provide the basic parameters that people can adjust. Far too often, the first conversation a data producer has with their consumer is once something breaks for the consumer. These silent or stealth data consumers create expectations without ever telling the producer. That causes many, many issues. Schema should be immutable - the only way to change your schema is by creating a new version. Ananth started by sharing a bit about his background. Despite writing the Data Engineering Weekly newsletter, he sees his experience as somewhat between a data engineer and a data analyst. That gave him the ability to see the full end-to-end journey of how data was handled at many different organizations. He consistently saw that analytical data outside of the application scope was an afterthought because developers were focused singularly on their application, not how it fit into the greater scheme, especially on the analytics side. For Ananth, the data marketplace is a useful concept for many organizations when thinking about data contracts. It might be a bit more of a data bazaar than like Amazon in certain ways as there can be a bit of collaborative negotiation - 'oh, you have XYZ to offer, what about ABC, could you do that?' We need standardized ways to discuss/document data to make it far easier to share data, or at least start the conversation off from an informed standpoint when collaborating to get the most useful data created and shared. We need programmatic ways for producers to share what data they have available including their expectations like SLAs and consumers to request data they want with their expectations. Scott note: It's crucial to understand that data contracts are less about the actual contractual terms and more about the establishment of a relationship that is covered through the contract terms. There are expectations but the contract isn't the entire relationship between the data producer and the data consumer. Essentially, the relationship includes the contract but just having SLAs will not resolve many of the issues people have around data contracts/sharing. Similar to something Chris Riccomini mentioned in episode #51, Schemata is looking to provide feedback to producers about what broke downstream when they made a change. Or more valuably what will break before a commit is deployed. Data producers haven't had much of this feedback historically - e.g. "if you make this change, it will break your data contract expectations on the schema front because of…". But, Schemata is also designed for producers to see how well what they are offering fits with what other domains are offering when thinking about how well does my domain or potential new data product integrate into the overall organizational data sharing landscape. On consumer-driven testing in data contracts/agreements, Ananth thinks there are two aspects: structural and behavioral. Structural is what you'd expect and what most people discuss in data contracts - mainly schema validation, is it backward compatible, is it strongly typed, is the required metadata complete, is there a registered owner, are the SLAs defined and complete, etc. The behavioral is similar to what Abe Gong talked about in episode 65 about what are the expectations, does the data behave the way people expect such that it can actually be leveraged for their use case. A key, widespread reason we need consumer-driven testing is producers rarely really understand how data consumers will use their data or are using their data already. Thus, that behavioral testing can inform the producer - along with actual human to human conversations - about how consumers will be/are leveraging data. One general issue many teams have according to Ananth is the consumer doesn't really understand the cost or complexity of doing something around data creation. E.g. the producer of one domain might not store the user ID so to get every user ID is an expensive database call. So a consumer creating a pull request instead of a demand/request for data means you can start from a deeper conversation about what the data will be used for and why it's structured like it is proposed in the PR. This is also much more in a developer in the domain's workflow of using git. It's all just far less vague even if the initial proposal is infeasible - a producer has far more information about how the data might be used to start to iterate towards a workable solution. According to Ananth, many people looking at Schemata have seen the need for years but there hasn't been a great way to implement what Scott calls "making the implicit explicit" around data sharing/data contracts. And this isn't a typical problem at a small company but once you get to a certain scale, the need for decentralized data modeling starts to become very evident. But with decentralized data modeling, it's pretty easy to put yourself in a bad spot because there is no collaboration layer so you create data silos / things that just don't interoperate well. Much like thinking federated governance versus decentralized governance in data mesh. Schemata has a concept of a core domain that then every incremental entity or event you model, it will automatically assess how well the new event or entity is connected to the core domain. The theory is to quickly figure out how well what you are building will connect into the greater whole of the organization through the core domain. It gives you quick feedback on what is in process and you can easily add more fields to better match the core domain if a producer wants. It isn't a blocker, it's giving feedback to someone creating a pull request - data producer or consumer - about how well the resulting data model would fit in the organizational data landscape. Ananth discussed how data creation is really a human-in-the-loop challenge - autonomous data creation is just not very valuable now and might never be. We need a collaborative platform to create data that is truly valuable and understandable but especially usable. The crucial aspect is to make a tool that integrates into people's workflows instead of yet another screen that further fractures the data management experience. Schemata is trying to be like Snyk - automatically scanning and giving people actionable advice but with little effort on their part. Where are your likely pain points? How could you address them? You can more easily set a goal of remediation/improvement and figure out how well you are doing. What are the top 2-3 things you could focus on to make the data you share that much better/more valuable? A big thing many overlook in creating data contracts is about defining the value and/or cost of something happening according to Ananth. It's about getting people to the table to discuss something concrete and make sure people are on the same page. Instead of requirements, it's a collaborative discussion. Alla Hale in her episode #122 talked about every conversation, you should have something to show the other party, whether a full prototype or a post-it note with a little drawing. So getting to clear contract/agreement is far easier if you have a system that defines an owner, defines the parameters you need, makes sure the implicit aspects are explicit so both parties can fully agree, etc. One thing Ananth - and Scott - keep running across are stealth data consumers creating one-sided data contracts. Essentially, the consumer has created their consumer-side testing and is consuming but the data producer has no idea they are consuming their data. Or many don't even really do the testing/contract model to protect themselves at all. The first the producer hears about their consumption is when something breaks for the consumer. With Schemata, at least there is a contract in place and stealth data consumers just have to inherit existing contractual bounds. Scott note: I hate stealth anything in data, let the producer know or they will potentially make breaking changes that could be prevented if they were just aware. According to Ananth, we can really learn a LOT from the DevOps movement that has become more the platform engineering movement on the microservices side. If we try to push ownership to domains/data producers without the tooling to help them verify they comply with governance and that things are working okay, that's a lot of extra work on the data producer end. It's why we are seeing so damn much pushback from domains about not wanting to own their data - it's just way too much of an ask. Data producers just don't have enough information about what might be an issue when they try to make a change and it causes unnecessary friction. We need to make both the producer and consumer more productive, so that people can develop and deploy without tons of manual intervention. Far too many teams are using tooling to solve single problems and while that one-off tool helps address a singular issue, it creates an even more disjointed data management workflow in Ananth's view. It's easy to focus too much on the spot challenge instead of the overall challenges in data management, the holistic process. Tooling fragmented with cloud and it made sense as we figured out new approaches and patterns - and VCs were quite free with their money - but we need to think about the whole process as one again now. Zhamak has mentioned this multiple times as well. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#197 Explorers Needed, Experts Not (Yet) - Mesh Musings 44 14:54

۳ years پیش14:54

14:54

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ All about why we need more explorers in our data mesh implementations and that it's too early for experts :) Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#196 Data is a Team Sport - Learning to Collaborate Through Data - Interview w/ Andrew Pease 1:09:57

۳ years پیش1:09:57

1:09:57

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Andrew's LinkedIn: https://www.linkedin.com/in/andrewpease123/ In this episode, Scott interviewed Andrew Pease, Field CTO of North Europe at Salesforce. To be clear, he was only representing his own views on the episode. Some key takeaways/thoughts from Andrew's point of view (mostly written by him): Sensitizing people to data and improving their data fluency can be a challenge. Lots of people have had some less than perfect past experiences - perhaps a dry, abstract class has given them "statistics trauma". It's important to make it digestible for them to get started. Organizations typically evolve into silos so IT systems/approaches often evolve into silos too - Conway's Law. The bigger those organizations and silos are, the harder they are to bridge / the deeper the divides. Much as we'd like one, there is not a single silver bullet architecture for all organizations to overcome these silos. Without relevant IT architectures and processes, it can be challenging to put relevant and timely data and actionable insights into the business people's workflows. You won't get it "perfect" the first time, but get started and learn to improve through experience. You should reiterate to people that data is there to augment their role, not to replace it. It's there to help them be more efficient and successful in their work. That's a key part of data fluency, not just understanding how to use data but where data can help. Feedback loops are very important to increase data quality levels and data value. It's important to build in these loops to make end-users feel like they are a part of a constant and never-ending improvement exercise. It shouldn't be a big burden but data quality is a team sport. It's important for data consumers to understand not only the potential of data and analysis, but also the limitations. E.g. you can't reliably score lead quality from simply a person's name and their email address. The data needs to be representative enough to find useful patterns. AI should be perceived in the enterprise as augmented intelligence - it is there to make the human in the loop better, not replace them. It is crucial to inform operational teams, the data producers, about what data might be needed in the future, not just now. And then incentivize those data producers to actually create and maintain quality data products. If all we do is ask, it likely gets lost in the operational "priorities". Anecdotal feedback on what data is being used and is useful is great. But it's not going to tell the actual full story. Make sure to create ways to track usage and measure impact of data work. Data hackathons can be great ways to set up some cross domain collaboration and improve data fluency but also knowledge of other domains and the organization as a whole. It's vital to figure out how to get people excited about data, in combination with incentivizing them to do so in appropriate contexts. IT and the business side need to meet and collaborate in order to make data a crucial and embedded aspect of everyone's roles. As always, communication is crucial, especially around reorganization of data teams and competencies. A clumsy reorg will certainly alienate - and possibly infuriate - people. "The most complex system that we have in our organizations isn't a computer, it's the people who are operating the computers." When we think about the composable enterprise, we need to think about the humans in the loops and how and where they interface. Look to have a standardized way to bring people to better data fluency. Many different roles have budgets for ongoing training in their field, everyone should have that for data and it should be part of any organization's new employee ramp period too. Andrew started off by discussing the general way that organizations evolve. It's pretty natural for most to evolve into silos and the larger the organization, the deeper the divides between the silos and the harder it is to bridge those divides. With Conway's Law, IT systems/approaches also then often develop into silos. There is a lot of required intentionality to prevent evolving into silos or lessen divides that have already formed. And there is no "silver bullet architecture" to overcome the challenges silos create or undo the silos. One of the big dreams of being data driven is putting timely and actionable data - "what do you want to tell them?" - in the workflows of business people. But, according to Andrew, many organizations attempting to do that look at it as an all-or-nothing kind of goal and that's just not reasonable. You won't get it "perfect" at the start. And that's okay, it doesn't make it not worth doing. As part of that process, it can be very important to reiterate that data is there to help not replace people - AI should mean augmented intelligence, it's there to help the human in the loop be better. There are two major opposing forces re data quality in Andrew's view. First, you never get a second chance to make a first impression so your data quality has to be up to a certain level before showing to potential consumers. But conversely, the only way to get to actual quality data - essentially what matters, why it matters, and what quality levels are acceptable - is to get data in front of consumers and then iterate towards the required quality. Feedback loops are crucial to actual data quality so you can optimize for what matters. Your data consumers must understand that data quality is a team sport so they need to participate too. Andrew brought up his concept of "statistics trauma" when discussing improving people's data fluency - essentially, many have a bitter taste from past statistics/math and/or data related work/school. So to get execs more data driven, you need to sensitize them to data but in a careful approach. That falls to the CDO and it can be challenging but is quite rewarding when it works. It's as much about communication as anything else in data. In data, Andrew believes there needs to be far more bi-directional conversations. Data consumers need to tell data producers what they need and that can include data that doesn't exist yet so the producers need to start capturing it. So the earlier a data consumer can tell a data producer about their needs, the more likely they will get what they want down the line. Data mesh helps there because it's not the central team trying to understand and take requests to the producers. By cutting out the data team in the middle, you have a better chance to get to what data consumers want more quickly. But we can't lose sight of something that many seem to overlook - we can't just inform data producers of what we want them to produce and maintain, we need to properly incentivize and enable them to do so. In Andrew's view, there is obviously value in collecting feedback on what data is viewed as valuable. But it's going to have bias - essentially it's valued but might not be valuable - so you should develop more concrete ways to measure what data work is useful and valuable. We should track what is being used but also how well what we thought would be valuable actually performed - that way we might better know what additional data might drive incremental value. Your feedback loops should include both quantitative and qualitative measurement where possible. If you make ultimatums around data usage - you're either with us as a data user or you're against us - you won't get buy-in per Andrew. Mandates just don't get the buy-in some people believe. So you need to work to figure out why someone is not leveraging data. Again, make it less intimidating and make it rewarding. If you threaten people - do this or we'll fire you - you will simply get people adhering to the letter instead of the spirit - we want to make using data useful AND fun. Gamifying learning about data and data hackathons are two great ways to accomplish that. Around data, if we want a "yin yang synergy" between business and IT, both parties have to meet the other _more_ than halfway in Andrew's experience. Both sides have to be willing to partner to improve. There isn't a silver bullet way to accomplish it but embedded IT in the business and vice versa can certainly help. You could rotate people across different business units. Etc. However, it's very important if you are in a decentralized organization to make sure you share best practices. Andrew said, "the most complex system that we have in our organizations isn't a computer, it's the people who are operating the computers." There is a major change in the way our brains work between learning something and trying to get a point across. Some people are good at switching between those quickly - e.g. in a meeting - but many aren't and it's important to not leave them behind. So communication is crucial to get right and think about the broad group you are trying to work with. Sometimes data should be brought in to the discussion to make a point but sometimes it should purely be about increasing data fluency. It's easy to try to focus on hiring for data skills in many roles but really, every organization should at least consider data training as part of a new employee training according to Andrew. Obviously don't forget existing employees but immersing people in data, especially the data of the organization, from the start pays off in the long run. Quick tidbit: Data consumers need to understand what is actually possible with data. E.g. lead scoring based on a person's name and email address is not a reasonable request. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of February 19, 2023 26:35

۳ years پیش26:35

26:35

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#195 Zhamak's Corner 18 - Fixing Unnecessary Complications in Serving Data to AI/ML 18:49

۳ years پیش18:49

18:49

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Sponsored by NextData , Zhamak's company that is helping ease data product creation. For more great content from Zhamak, check out her book on data mesh , a book she collaborated on , her LinkedIn , and her Twitter . So, continuing the conversation about AI and ML's place in data mesh, we start the episode with Zhamak discussing an unnecessary complication we've created in data - why do data sets/assets only have to serve one user or even user persona? Yes, product thinking is about creating reuse but are we thinking reuse across regular analytics and ML/AI at the same time? We need to make it easy to give access in the language of, that native mode of access of, the data consumer. We shouldn't have to care what it is used for, regular analytics, ML, or anything in between. There's also this very painful bifurcation between upstream data production and data science where the second data enters the data science realm of influence, it's copied over and you lose sight of it for discoverability, governance, security, quality, etc. They pull it in and then it's essentially impossible to track. That creates all kinds of problems. So why don't we extend data mesh into what they are doing? Do they need to make copies of the data in the feature store? If they have a trusted source of access to the data, do they care? Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Data Mesh Radio episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#194 The One Where Scott Goes off About Data Contracts (Part 1) - Mesh Musings 43 18:10

۳ years پیش18:10

18:10

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#193 The Hidden, Pesky Persistent Challenges in Data-Intensive Applications/Service/ML - Interview w/ Ebru Cucen 1:07:42

۳ years پیش1:07:42

1:07:42

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Ebru's Twitter: @ebrucucen / https://twitter.com/ebrucucen Ebru's LinkedIn: https://www.linkedin.com/in/ebrucucen/ In this episode, Scott interviewed Ebru Cucen, Lead Consultant at Open Credo. To be clear, Ebru was only representing her own views on the episode. Some key takeaways/thoughts from Ebru's point of view: It's far too hard for data producers to actually reliably produce clean, trustworthy, and well-documented data. We need to give them a better ability to do that, whether that is tooling or ways of working remains to be seen. Scott note: It's no wonder it's been hard for many teams to get their domains to own their own data ;) There is a hidden challenge in data-intensive service/application development. The version of the data - the schema, the API, and the data itself version - need to be understood and coordinated as the developers don't control their own data sources unlike software development of the past. But we don't have good ways of doing that right now on the process or tooling front - data product approaches help but fall short. We are lacking the tooling to easily manage data quality for producers. While there are so many data related tools, there is a real lack of things that make it easy to manage the quality. We are getting there on observing or monitoring quality, but not managing and maintaining quality. Fitness functions can help you measure if you are doing well on your data quality/reliability. As the speed to reliably ship changes on the application side increased - microservices and DevOps -, that just made the data warehouse, the data monolith that much harder to deal with. Instead of slow-changing inputs and gentle evolution, it simply became more and more of a data exhaust model that breaks the warehouse. Large data monoliths are just far too hard to maintain, especially as the speed of change of application and the world increases. However, monoliths aren't 'the enemy', even microservices advocates say sometimes a monolith is the right call. Look to figure out what's the right solution for the now and future. Don't distribute, don't decentralize without a specific reason. ?Controversial?: Data really needs much better version control systems and practices. Yes, there is the versioning of the data product but the actual versioning of the data - that immutability factor, when did this data change and what was it before that - is the most important versioning for data/analytics. Versioning means safety - safety for consumers but especially for producers to be able to roll back. We need those better safety features so we can test much more thoroughly in data but right now, we don't have great ways to do that. It's hard to fight Conway's Law. If we don't fix our ways of working together, it is extremely difficult for consumers and producers to align well enough to get the most value from our data. Communication issues will be reflected in the data as well. The tools we have for data are so specialized, you might need to use 5+ tools just to properly manage a simple ingestion process - it's just not there to support the producers well enough. How can we observe and validate data before writing very specific testing - testing shouldn't be the only line of defense. We need a way to define and create our quality gateways much more easily. With fast feedback cycles and close collaboration around data, especially with data science, it makes everyone so much more productive. E.g. people aren't building on deprecated data sources and you can get to initially testing a hypothesis in a day or days instead of weeks. It's important to think of your data like a garden instead of a single project - you must tend to it and improve it further. Your garden is never "done" and weeds can creep in quite easily. Get that green thumb. To build to good models in data science, you have to ask what questions can we ask of the data, can we get enough data, is it high enough quality, etc. You need to answer will we be able to achieve a likely positive outcome and then iterate towards good - and then make it better - instead of making things all or nothing and making a static model. Not all questions can be asked of the data you have and you need to measure how well what data you have can answer the questions you want to ask. Be realistic about what you are trying to do and what you actually can do based on what you have. How do we create psychologically safe environments where people can fail safely and learn from that - we need iterative communications, interactions, learning, and development. Inject more empathy into your teams and communications. We need a better way of understanding the challenges, what are we achieving together, instead of what is each person's role. The sum of the parts is the purpose. ?Controversial?: As we increase the amount of data we have and the number of people attempting to leverage that data - and let's not forget the increasing complexity of the world at large - we are likely to see it get harder to communicate relative to data. We have to try harder than ever to get it right, or at least to an acceptable end outcome. ?Controversial?: Similarly, our understanding of certain questions or sets of data will change more frequently than historically and communicating that - and why our understanding has evolved - is going to get more complex as well. Ebru started by sharing her background where she was a software engineer and trainer including training people on SQL before moving into data/data science. As a software engineer, it was crucial to at least model and understand data well enough to ingest and store it for the application side. The big challenges for software engineers really came in integrating that data into the monolithic data warehouse and then keeping it well integrated as the application evolved. The monoliths were bottlenecks on the software side and the integration into the data monolith was just becoming too much of a major bottleneck for all. As the DevOps and microservices movements picked up steam, the speed to reliably changing the application significantly increased. That increased speed created more and more challenges in integrating into a monolith - the applications drifted far too quickly to easily work with the data monolith. But monoliths aren't necessarily the wrong choice for all, just at scale and especially scale of complexity, they become a massive bottleneck. When talking about versioning, Ebru talked about the many copies of data challenge - which one is the right one to use and can I trust it? There are people doing incredibly important work on data where they can't reliably trace it to source and know they are working on the right version. And with no clear ownership of data, nothing ever gets cleaned up so finding a reliable, repeatable source of data is very hard. So people copy the data they do find to their work area lest the source goes away, creating more copies. So we've figured out how to do versioning relatively well on the software/microservices side with APIs but we haven't figured it out for data - whether that is versioning the analytical API or the data itself. It's far too hard to make our data assets maintainable right now, thus the big push to data mesh. For Ebru, when asked specifically what is the most important aspect of versioning in data - code, schema, API, or the data itself - she chose the data itself. This is a somewhat controversial choice but her reasoning was traceability - what actually happened to the data and when did it change? She expects that the code versioning, we'll have more version control systems and many people already manage their data related code work in git or other systems. Another point Ebru made was that software development hasn't really had a focus on aligning itself to what version of data it is using. When you do a production deployment, the database is the database, it's tied to the application. But when we start to think about how we actually deploy software going forward, if it is referencing external data as part of that, the version of the data source it's leveraging obviously matters far more and we need far more coordination to ensure the software is referencing what we need it to. There is not enough tooling out there to easily manage this coordination and it's causing far too many issues. Scott note: this is a really incremental thought here but VERY hard to explain. Historically, most services have been more or less wholly contained in what data they use or they access information from other services via a versioned API on the operational plane. So the coordination is less challenging. We have not really figured out well how to do that for data intensive applications - this is partly why everyone is building data products, whether data mesh or not, but it's still challenging if you don't think about providing a steady access mechanism and a way for a consumer to know what they are accessing hasn't suddenly changed without their knowledge. See the episode on my rant on data contracts and how it's not just schema and constraints. We just can't escape Conway's Law according to Ebru. While many people have applied it to the operational plane, we really need to think about how Conway's Law applies to data. The way we exchange information can't only be the data itself, we need to get better at how we actually communicate and collaborate internally or gaps in how we communicate will be reflected in the data and our data integrations. Without fixing the way we work together and communicate, the producers and consumers will not collaborate well enough to leverage our data to the fullest extent. Ebru believes that right now, it's still far too hard for producers to reliably publish clean, trustable, and understandable data. We haven't developed great ways of working and the tools are definitely not there yet. So if we try to push ownership on them too quickly, it will not go well. They have historically published what they want and we need to make it far easier to publish what consumers specifically want or they won't likely want to participate. Data mesh is a sociotechnical approach but for Ebru, there is a lot of talk about the social and the technical is still lacking. There are so many tools but they don't work together that well natively and most only do a few very specific things - you could need 5+ tools to accomplish just the ingestion part of a use case. There is also a major challenge on the testing side - can you observe what changes would occur before writing the tests? In general, we need to change our ways of working in data to enable much faster feedback cycles in Ebru's view. She was working on a project where everyone was in close collaboration and you could try things out and get feedback in the same day, meaning there was far less time spent building toward a solution only to find out the data wasn't available or there were other challenges. With better data ownership, we can go from idea to ingestion to testing in a short period of time, significantly improving how productive data science team members are. Scott note: if you listen to early data mesh presentations from Zhamak, she talks more about data science/machine learning/AI than regular old analytics. This is that data bazaar/data marketplace kind of concept in action. Ebru believes we need to take more learnings from microservices, especially the concept of Lego pieces. In data, we haven't really built incrementally to really achieve good value - it's often been all or nothing. But cloud means we have a chance to do things differently. That iteration means we can fail faster too - if we have an idea but we can't get the right data or even get enough of the right data, instead of building for weeks, we can change course. It's important to realize you can't ask any question to any data as well - sometimes you have a question that just can't be answered with the data you have or can get and that's okay. To do data well/better, Ebru believes we need to create psychological safety and an ability to fail safely. That means we will have to train data consumers far better on how we work with data - a 95% confidence interval doesn't mean what most believe. And our understanding of data evolves too so consumers must learn to evolve their understanding. Human interaction is far more crucial than many want to believe in doing data well. In data, as Zhamak has mentioned this trend towards super fractional roles, Ebru believes there is far too much focus in many organizations on what specifically is "my role" instead of what is the team's role and how can we make sure we accomplish our objectives. This fractional thinking of course creates more friction and challenges and handoffs - handoffs are always a place of lost context. So work to have teams focused on accomplishing team goals instead of individual ones. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of February 12, 2023 19:17

۳ years پیش19:17

19:17

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#192 Diagnosing the Analytics Gap - All About Diagnostic Analytics - Interview w/ João Sousa 1:07:48

۳ years پیش1:07:48

1:07:48

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . João's LinkedIn: https://www.linkedin.com/in/joaoantoniosousa/ João's Medium: https://joao-antonio-sousa.medium.com/ Brent Dykes' LinkedIn: https://www.linkedin.com/in/brentdykes/ In this episode, Scott interviewed João Sousa, Director of Growth at Kausa.ai. To be clear, he was only representing his own views on the episode. The "four types" will often be throughout this summary. The four types refers to the types of analytics: descriptive - what is happening; diagnostic - why is it happening; predictive - what might happen in the future; and prescriptive - what actions should we take. Some key takeaways/thoughts from João's point of view: Of the four types of analytics, diagnostic analytics is VERY underserved. The other three - descriptive, predictive, and prescriptive - are where most organizations are focusing more so there's a "diagnostic analytics gap." ?Controversial?: Of the four, diagnostic analytics requires the most domain/business expertise. Tips for improving your diagnostic analytics: 1) show the value of drilling down in to the why - find a few use cases and communicate the value well; 2) promote a closer collaboration between data and business people; 3) improve your definitions around data roles; 4) very clear communication of expectations and who does what; 5) don't get into firefighting mode, have a structured approach to diagnostic analytics; and 6) automate the repetitive parts. There are 3 levels of diagnostic analytics immaturity: getting "stuck in the what" instead of the why; "the usual suspects" where you look at things from the same angle, same slice and dice and only leverage a small portion of the data available; and "need for speed" where you have the right data culture of drilling down to the why bet often get stuck in the trade-off between speed and analysis thoroughness. Signals you are "stuck in the what": 1) in reviews, teams are just reviewing what happened instead of data-driven recommendations or data-driven hypotheses. 2) You lack impactful stories when asked "what real data-driven insights have you shared recently that drove action?" Signals you are stuck in "the usual suspects": 1) few to no real new major insights or hypotheses. 2) Lack of incremental data work / no new slices and dices to further analyze data. 3) Indirectly, data teams become more disconnected from the business. Signals you are stuck in "need for speed": 1) continuously cutting corners on thorough analysis in the name of speed. 2) Juggling too many tasks / priorities with constant 'hair on fire' type requests. Culture and people are the biggest levers in data but the hardest to change. We need better processes and tooling to enable accelerating diagnostic analytics. Tooling and processes specifically for diagnostic analytics are few and far between. Many companies do not put much value and/or effort in to diagnostic analytics - that is highly correlated to analytics maturity. Diagnostic analytics work can be seen as boring compared to predictive and prescriptive work. It's typically not as technically challenging and many data people are not as interested in the business aspects. ?Controversial?: The best teams segment their diagnostic analytical questions into strategic, tactical, and operational. High performing teams also adjust thoroughness versus speed to best suit the specific need. They also automate as much as possible to reduce burden on the human in the loop. !Very Important!: Decentralization presents a big potential risk to diagnostic analytics. Analytics within the domain seems covered but many questions are cross domain... Scott note: There isn't a clear owner of diagnostic analytics in data mesh - if domains know their own data well, they should be able to do diagnostic analytics on information internal to the domain but it will be far harder cross domain. And that is end-state, not mid mesh journey. Diagnostic analytics likely falls to where you have your business analysts, whether that is embedded only, centralized only, or a mix. Insight definition criteria (from Brent Dykes): 1) provides a shift in understanding; 2) is something unexpected where the organization was not previously aware; and 3) it's relevant and/or aligns to what stakeholders care about. João added a 4) it must be delivered on time and communicated effectively. João started the conversation discussing the four types of analytics: descriptive - what is happening; diagnostic - why is it happening; predictive - what might happen in the future; and prescriptive - what actions should we take. Most of analytics work over the last 30 years has been the descriptive and both descriptive and diagnostic are typically owned by the analytics team. Data science, ML, and AI have moved the needle for doing predictive and prescriptive analytics the last few years. But diagnostic analytics remains underserved. That diagnostic analytics gap exists for a number of reasons in João's view. On the people side, diagnostic analytics requires two sets of skills/knowledge: the analytical + technical and the business + domain. Without the domain knowledge, it is far harder to connect the dots around the why - a key concept in data mesh in shifting data ownership left*. Yes, we know sales in this region are falling, but why, what changed? João believes diagnostic analytics requires the most domain knowledge of any of the four types. * Scott note: I always think of the Pastafarian - or Church of the Flying Spaghetti Monster - figure as to why they dress like pirates and how the number of pirates is strongly inversely correlated to global temperatures. See here . Correlation doesn't mean causation, relevant XKCD here On the tools and processes side, João believes diagnostic analytics is far less developed than any of the other four types. Dashboards are great for descriptive analytics - what is happening - including some exploration but they are difficult to use to actually understand the why, drilling down to the root cause. Culture around diagnostic analytics is another large issue for many organizations - there are many varied approaches and lots of differing views on the actual value of doing deep diagnostic analytics. João has three different diagnostic analytics immaturity stages before getting to a well-functioning approach. The least mature is "stuck in the what," where the business stakeholders are the ones trying to do diagnostic analytics with low data fluency* to drive to the why. They are only reporting the what, the descriptive analytics. The second maturity level is "the usual suspects" - essentially, the team builds lots of slice and dice dashboards and then just monitors things using that, they don't think to keep adjusting their angles and dig in. The third maturity level is "need for speed." The domains have the capabilities - usually via embedded analysts - to analyze their own data but are almost always siding with speed versus comprehensiveness of analysis. The world and business are changing fast but it takes time to do good analytics well to generate an actual insight. Scott note: this brings up the question of where diagnostic analytics lives in a data mesh implementation. If domains have high data fluency, then presumably they can do their internal analysis but what happens if the information to drive to the why is cross domain? This is why I believe many domains are likely to have their own business analysts but organizations will still have a centralized business analyst team too. On the question of who does the diagnostic analytics in most organizations - an empowered and highly data literate domain or a centralized analytics team - João said it depends. In a low data maturity organization, it's typically the analytics team - hopefully pairing closely with the business. In a higher data maturity team, it's about upskilling the subject matter experts in data and providing the right tools so they can do the analysis themselves. João shared two signals you might be "stuck in the what", that you need more diagnostic analytics maturity. The first is in your weekly or monthly review meetings, you are talking about what is happening and there are only some high-level guesses as to why - "oh, that's _probably_ because we changed the website" - and not much more. Nothing is data-driven answers or even hypotheses. The second is reflecting that you haven't taken any real data-driven actions with a large impact recently. If you aren't driving your actions from your data, it's likely you aren't answering the "why" questions. It's a lot harder to detect if you are in "the usual suspects" phase of maturity per João. The data teams aren't getting lots of additional requests. The business people are generally happy because they have dashboards that show a lot of information sliced and diced in how they typically look at things. But they are only testing existing hypotheses and not really coming up with fresh/new insights. So two signals are that there aren't really any new insights or hypotheses and there aren't many requests from the domain to the data team. The third signal, one that's indirect, is that because there is that lack of incremental data work and requests, the data teams start to become more disconnected from the business. Teams that are stuck more in the "need for speed", João said while it's a better place to be, it's still frustrating. Teams are always trying to balance thoroughness of analysis versus speed. So some signals you are there is the pressure to cut corners on thoroughness of analysis in the name of speed - that actually happening is another signal - and constant high-priority interruptions for diagnostic analysis, juggling too much and putting aside the long-term work to take care of the fast turnaround requests. When teams break past the immaturity stages for diagnostic analytics, João pointed to a few things high performing teams do well. The first is to segment questions/requests into tactical, strategic, and operational. Strategic questions are typically more big picture so they change less frequently and thus are typically less urgent than operational or tactical requests. Strong teams also adjust their thoroughness versus speed depending on what the situation calls for. Lastly, they automate as much as possible - there is still a human in the loop but repetitive tasks aren't value-add tasks for someone to do. João shared Brent Dykes' definition of an insight, which is probably much more strict than many use. First, it must provide a shift in understanding - so not "we found this anomaly", it changes what people know. Second, it must be unexpected - so those teams stuck in "the usual suspects" won't meet this because they are only testing against the expected. And third, it must actually matter, it must be relevant and/or aligned to what stakeholders care about. João added his own criteria of it must be on time and communicated effectively. These are all necessary to actually drive the right action. João wrapped with a few tips for improving your diagnostic analytics: First, show the value of drilling down in to the why - find a few easy initial use cases to really show the value, not your most difficult questions that will take months to really answer. Second, have the data and business people collaborate more closely so the data people can better understand requests and business people can start to think about new analytical approaches. Third, really get clear around your data role definitions: who does what and why and what _aren't_ they supposed to do. Fourth, start to get very clear on expectations, improve that communication so everyone is on the same page. Fifth, plan ahead and don't get stuck in firefighting mode grasping for straws - it's too easy to approach diagnostic analytics in an unstructured, reactive manner. Finally sixth, look to automate away the repetitive parts as much as possible. Quick tidbit: Beware the 'boring' label for diagnostic analytics. Many data people want to focus on the more technically challenging predictive or prescriptive analytics. Show people diagnostic analytics is valued. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#191 Zhamak's Corner 17 - AI/ML's Place in Data Mesh 16:42

۳ years پیش16:42

16:42

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Sponsored by NextData , Zhamak's company that is helping ease data product creation. For more great content from Zhamak, check out her book on data mesh , a book she collaborated on , her LinkedIn , and her Twitter . Humans by our very nature categorize things - otherwise how can we really differentiate? How can we learn about new ideas and experiences if not finding a way to store them in our mental models. And in data, we've been treating diagnostic and descriptive analytics as an entirely different category to the predictive analytics of AI and ML. The way we partition the world in data is around how data will be used and then prepare the data as such, to be very fit for purpose. What if instead we partition around the data domain and don't really care about who or how things are used - we want to serve all consumers - what changes? Can we create data that is simply usable by many? Does that actually reduce complexity overall by not owning data production designed to specific purposes? Do we really need to treat AI/ML as if their consumption is all that different or special? Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Data Mesh Radio episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#190 Data User Experience (DUX): An Introductory Panel - Led by Karen Passmore with Alice Parker and Wannes Rosiers 1:23:44

۳ years پیش1:23:44

1:23:44

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Karen Passmore (CEO at Predictive UX) led this discussion with Wannes Rosiers (Product Manager at Raito) and Alice Parker (Data Engineer at DNB). This panel was held in partnership with Data Mesh Learning - you can see a link to the video here: Panel: Data User Experience - An Introduction (Data Mesh Learning and Data Mesh Radio) Alice's LinkedIn: https://www.linkedin.com/in/aliceparker/ Wannes' LinkedIn: https://www.linkedin.com/in/wannes-rosiers/ Blog post 'The Importance of UI/UX - and why Raito’s first hire was a designer': https://www.raito.io/post/the-importance-of-ui-ux-and-why-raitos-first-hire-was-a-designer Raito blog: https://www.raito.io/blog Karen's LinkedIn: https://www.linkedin.com/in/karenpassmore/ Predictive UX: https://www.predictiveux.com/ Some key takeaways from panelist Wannes Rosiers: DUX handles user experience using data for all your users (data producers, data engineers, data analysts, data scientists, report users, ...), hence it is not - and certainly not restricted to - data platform user experience. Data products are typically chained from producer oriented data products up until consumer oriented data products. Downstream product and UX requirements are completely different than Upstream, yet you should be considering all of them. This friction between global UX and local UX is the biggest challenge to scale Data User Experience. Much more than typical digital products, you have no idea how people interact with your data products. We are moving more and more to self-serve analytics, which means that while developing a data product, you don't know what will end up on the screen of your user. You can wireframe fixed dashboards, you can't for certain other data products. As always, accessibility of your data product is a huge part of your user experience. Next to this, it is important to monitor usage to continuously evolve your product. (Not surprisingly the two key elements we work on at Raito) Domain thinking exceeds source thinking in your product. You should abstract away the concept of data sources, users are interested in valid and bounded insights, typically domain-bounded. And last but not least. You need top-level decisions on data product granularities. To make sure you can connect certain data products, they should be interoperable and preferably on the same granular level. Which level to pick is mainly guided by company KPIs, hence you need management support to pick these. Scott note: I am pretty new to thinking and dealing with UX in data so I took an opportunity to write down some of my own takeaways that may or may not agree with any or all the panelists. Hopefully you'll find them useful. The most key takeaways from Scott's view and learnings: If data is not usable, is it useful? How many data projects fail simply because no one really made the data usable? UX KPIs are really important because it's such an easy thing to overlook. If you have a bunch of fragmented UXs in your data value chain, your overall UX will suffer. Think about a dish - you might prepare all the ingredients perfectly but if they were all cooked separately, will there be the right flavor harmony? Probably not, it will be disjointed flavors that don't blend well. Sometimes we think about the target business process in user experience so we don't always have to think about directly using a platform - we can think about how things integrate into a business process, a workflow. If I don't have a UI but interacting with data is part of my day-to-day role, that's a user experience. Data projects and data work need three things: business requirements, technical requirements, and user requirements. Many skip the user requirements at their own detriment. The best way to improve user experience is to actually talk to the users about their wants and needs 😅 Empathy is crucial to data UX - make sure empathy is baked in to what you do :) stop prioritizing your technical requirements and business requirements over your user requirements. It's very hard to figure out the user experience path between 'I create a data product' and 'user self-service drives an action from the data product'. It can feel a bit underpants gnome-like of step one steal underpants, step two ???, step three profit. So get really in depth talking to your team about what is necessary to actually get to an actionable insight from a data product. UX technical debt, while not insurmountable, will be a bigger hindrance than many expect because the UX is typically intrinsically tied to the underlying implementation. So to improve the UX to a great degree, it can also require improving a lot of the underlying implementation. Definitely not always the case but it's something to watch out for. A lot of additional takeaways from Scott: User experience is often overlooked in data because historically, most data manipulation or analysis has been performed by experts in their own domain - e.g. a business analyst is an expert in SQL so all they need is SQL or data engineers have handled ingestion and transformation so there hasn't been a great UX. It's easy to get trapped in the idea that a "data user" is just a data consumer but if we aren't designing and creating a good user experience for producers and anyone else involved in dealing with data, we are looking for trouble. Each user persona has different requirements and it's important to not try to design one-size-fits-all experiences because it will probably be one-size-fits-none. User experience is about lowering the bar to working with data to make it actionable, whether for a producer or consumer or anyone else. How do we remove friction from data to action? Data consumer user experience spans a lot of things from finding then accessing then understanding data and then often transforming into their own purposes. Data user experience can be improved at any point in a data project's lifecycle but it's always best to start at the beginning if possible. How can we make it easier/quicker/more reliable/etc. to get to an initial answer on is data valuable? Part of ensuring a good data user experience is enabling the data consumers to easily communicate their expectations/requirements to data producers. It's not just the UX at point of interaction but also the UX to make the data better. It's important to understand focusing on UX isn't about right or wrong, it's about improving the user experience over time. The data producers and consumers are often an afterthought when designing data architecture. The UX is something at best smeared on top to try to make it a little easier to deal with when it should be a crucial aspect of the architecture itself. When many people think about data UX, they think about someone consuming from a dashboard, not all the steps that data had to go along to get into that dashboard. Metadata is crucial to helping people understand the data they are accessing - especially when you consider being able to trust data as part of the user experience. If you can't trust it, you won't use it! It's important to think about UX as a lifecycle because there is a data lifecycle - information is created and then acted upon creating more information of a sort. So we can't think of it as "data is consumed" and that is the end of the line for the business when it comes to broad user experience of data. It's crucial to think about user experience at the data product level but even more so at the informational/question level. How can data be combined from multiple sources? Does UX extend to data interoperability standards even? Yes, but that shouldn't be managed by the UX team necessarily 😅 A very big part of UX is just encouraging conversations because then UX ideas will emerge. 'Oh, you're trying to do X, that makes so much sense, let me make this change to make that less painful.' But too often users take what is given and don't pipe up - give them room and encouragement to share feedback. Similarly, open communication seems really integral to getting to a good UX. You need people interviewing the users but users also need to have space to discuss too. UX knowledge transfer shouldn't only happen because someone asked. Information transfer can be really hard, especially if we do it all as pure documentation. Look for ways to enable better sharing of information - lower the friction to creating simple information sharing mechanisms, whether that's sharing the data or the metadata. A huge benefit of data mesh is reliable and scalable data sharing yes but it's also about the art of the possible - 'oh, we have this data, what about XYZ use case, can we do that?' How can we embed curiosity and an ability to explore more in our data systems while still protecting privacy/PII/etc.? It's easy to get myopically focused on the platform UX and not the total data UX. Some of that is cultural too. You want to try to design for it but the platform team can't own developing the culture. But making it loud and clear what would drive more value might help you design a better overall data UX. It's very hard to know just how a user will interact and experience a data product. So have some conversation with potential users and maybe give them a private tutorial so they don't go off the rails. It's hard to embed 'understanding guardrails' into data we share. It's easy to lose sight of cost when considering user experience. How do we make it so we can enable great experiences in a cost-effective way? It would be great to have data usability design patterns and methodologies emerge for creating usable data products but we are VERY early days there. Usability testing is pretty crucial because while they aren't your enemy, the saying "no plan survives contact with the enemy" is still apt with data work in general. Don't design in a vacuum, work on constant communication and information flow. It's crucial to think about UX at the micro and the macro level. If you only think about it at each data product, you are missing the information of the organization and how it all interoperates to tell stories. But it's also pretty easy to focus on the grand platform and then dealing with the data products is a bit of a nightmare. So again, it's not easy :) Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of February 5, 2023 33:17

۳ years پیش33:17

33:17

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#189 Our Data is In the Cloud… Now What? - Interview w/ Vikas Kumar 1:16:42

۳ years پیش1:16:42

1:16:42

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Vikas' LinkedIn: https://www.linkedin.com/in/vksnov9/ Vikas' Twitter: @vikaskumar9 / https://twitter.com/vikaskumar9 Vikas' email: vikaskumar9 [at] gmail In this episode, Scott interviewed Vikas Kumar, AVP and Head of Data, AI, and ML at CNA Insurance. To be clear, he was only representing his own views in this episode. Some key takeaways/thoughts from Vikas' point of view: In data mesh, make sure to keep focused on bringing the business domains along. You aren't building for the sake of building. If users can't derive value from the data work being done, why is it being done? The 2010s through the early 2020s have been about moving data to the cloud but we are starting to see people really leverage that data to generate value. The cloud unlocks many new possibilities around data due to flexibility, scalability, and unit economics. With moving to cloud, there is much less focus on specifically managing the data and more focus on getting value from the data. SaaS data product offerings really unlock people's time to focus on driving value. Cloud gives us the scale and data availability but there is still a long way between having the data available and leveraging the data for significant value. Cloud can be a double edged sword - it gives you flexibility and scalability but without good controls, you are likely to do a lot of duplicate work. Be careful that ease of data product creation - or at least PoC creation - doesn't create chaos and data product overlap. Make sure to have good governance here including strong communication. ?Controversial?: We aren't very good yet at making it easy for business domain users to leverage data in many of their decisions. Where do we fall on the spectrum between we need to teach them how to do everything data and we need to curate everything for them? ?Controversial?: It's easy to focus too much on the short-term quick wins in data. You need to think about your overall data landscape and build a foundational approach so you can go after big picture, big impact bets with your data work. You should think about building every data product from a foundational approach too to make them more extensible. We need to get people out of their functional silos with business people only speaking business and data people only speaking data. To do data mesh well, we have to focus on the operating model of the organization around creating and maintaining data products. There is too much focus on the technical aspects instead of how does this actually get done in a way that fits with the organization's ways of working. Data producers must assess data consumers' data fluency levels. If they aren't very strong with data, should you really be delivering them raw data instead of curated insights? For any data product, you should start by mapping it to a target outcome. But it shouldn't stop there because with reuse, new outcomes may emerge that drive additional value. Data product owners are crucial to building good data products. It's their job to identify and then satisfy the objective of building the data product. What are you trying to achieve? We shouldn't focus only on the data product - the work to create that data product is what makes it valuable, the data product is merely the vehicle for delivering the value, the output of real product work around data. ?Controversial?: Many companies doing data mesh appear to be trying to leave data governance until 'later' and that is likely to bite them. The governance meaning the security/access control but also the interoperability. You might not need to implement all of your data governance upfront but you should plan out your general governance strategy very early in a data mesh journey. Access control is a really hard problem. Many organizations don't have good communication or visibility into who is using what data and especially how/why. We need to be asking these questions and then setting access policies that expire too - we should check in to see if people still need access, that's just good governance. According to Vikas, 2010 through the early 2020s the focus has been on moving the data to the cloud to better drive value. And now that more and more of our data is in the cloud, we are starting to see much broader adoption of things like ML and AI. The cloud gives us the promised but under-delivered scalability of the "big data" technologies along with the flexibility to move quickly and experiment. Cloud can also mean it's easier to bring non-data people into the mix to drive better collaboration between the data people and the business people/domain. So cloud gives us this massive scale and data availability but we still have to learn to better leverage our data, drive value from it - we are still in pretty early days there as an industry. A big outcome of the mass movement of data to the cloud is how much time is spent on data management versus getting value from the data according to Vikas. DBAs used to spend 60%+ of their time just managing the data but data people's time is now focused on getting value and probably only 10-20% is spent managing the data specifically. But cloud can be a double-edged sword too - if it's very easy to create new data products or beta data products, you have to be very careful to not create overlap/duplicate work/data products. It all comes down to governance and your operating processes to prevent that. As an industry, we are getting much better at serving data reliably at scale according to Vikas but we still struggle with the gap between the data is available and the data is able to be used by consumers in the business domains. We are still working on figuring out where to meet in the middle between handing people reports and maybe dashboards - a kind of old school approach - versus upskilling them to very high data fluency so they can build everything themselves. When asked that question - do the data people have to learn all the business context or vice versa - Vikas gave the very data mesh answer of "it depends." But that makes sense because there shouldn't be a single prescribed method, you have to look at how your organization works and fit with that model. And you probably want to meet somewhere around the middle. Otherwise, you will cause unnecessary friction. So look to your general ways of working, cross train people, get people exchanging context about what they are trying to achieve and instill a culture of feedback and collaboration. That's how you can actually execute well on a data mesh strategy. Vikas talked about your data strategy north star being about getting value from your data, reliably and at scale. So, you need to be realistic about where you are in that capability journey right now. As a data producer, you need to assess can your data consumers do everything necessary if you give them raw data or should you be curating it for them so they can actually leverage the insights. Work to find the high value return data work early instead of trying to do the most complicated aspects of data. It's okay to start small, no shame there. A data product should always map to a target business outcome according to Vikas. But that shouldn't be the only factor. The reason for creating a data product should be trying to achieve that outcome so use that as the north start for the data product but we must build in a way where data products can be reused - sometimes with some additional work - for additional use cases. And it's really crucial to have a data product owner that is discovering and focusing on the objective of the data product. How can you provide the business meaningful data that meets their objectives, that should be a key objective of every data product. When asked how do we balance focusing on the long-term wins instead of the quick - but typically small - wins, Vikas talked about the need to create a holistic view of your data and build a very strong foundation for how you will deal with data in general. That makes it so you can jump on the quick wins when you find them but you also have a steady foundation for making much bigger bets going after long-term big wins. But with a shaky foundational layer for your data, those long-term big wins are much less likely to pay off. And that foundational aspect comes in at the data product level too - build data products that can be easily extensible when it makes sense because they are built to be extensible from the start. Kent Graziano in the recent data modeling panel railed against having to rebuild every time you extend a data product, don't do that :) For Vikas, there are many value streams for a data product - most people focus on the data set itself but it could be the governance work or the collaboration conversations between producer and consumer. We need to focus less on the data product as the exact output instead of the data product being the vehicle for delivering value but the overall product work itself significantly enhances the value of the data product. Data governance seems to be the part of data mesh that confuses a fair number of organizations so they ignore at their significant peril according to Vikas. While you might not have to build every aspect of your governance upfront, it's crucial to think about how you will apply governance. And to truly get to the ideal of a self-serve platform, governance needs to be a simple part of the ways of working. Saving that for later is not going to end well for many organizations. And while access control is hard, we need to get far better at understanding who is using what and _why_. How long should someone get access to data? Forever access should be a non-starter. And how do we make it easy to grant that expiring access? Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#188 Controversial Opinion: You Must Register Your Use Case With the Data Producer - Mesh Musings 42 15:38

۳ years پیش15:38

15:38

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. I share my strongly held belief that every data product consumer MUST register their use case with the producer and why it is so crucial and important. I sum up with these points: Data isn't ever going to be self-describing, at least not deep, context rich data and it definitely won't be as we focus on data democratization. So, you need to register your use case to make sure you and the producer are on the same page for what is truly being shared Data producers don't know what data consumers want - and especially don't know what potential data consumers want - without more information/conversation. Data producers kinda suck at doing that right now so help them out and improve the information flow. Who knows, they might have something way better for ya You might actually be able to get at a heck of a lot more useful information if the producer 1) understands your use case and knows they can give you access to the PII/sensitive stuff because your use case is fine on ethical/legal/regulatory checks and/or 2) you are taking on the risk around if you violate ethical/legal/regulatory bounds. You need to communicate what would most effectively help you achieve a target outcome and have a reasonable, collaborative negotiation between data producers and data consumers. Just consuming from a data product as-is might not meet your needs well enough. Kinda similar to #1 but just getting a much better understanding of what you are really consuming. Does it mean what you think? Did you create an amazing insight, how can you collaborate with the producer to show that off? How can you make sure you will continue to get what you need as the data product evolves. I'll be digging somewhat deeper on the next episode with a rant on data contracts. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#187 Maximizing the Value of Your Data Through Data Products - Interview w/ Bruno Aziza 1:04:00

۳ years پیش1:04:00

1:04:00

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Bruno's LinkedIn: https://www.linkedin.com/in/brunoaziza/ Bruno's Medium: https://medium.com/@brunoaziza Bruno's YouTube (Carcast videos): https://www.youtube.com/@brunoaziza In this episode, Scott interviewed Bruno Aziza, Head of Data and Analytics at Google Cloud. Some key takeaways/thoughts from Bruno's point of view: The end goal of your data strategy should be to reliably and scalably turn data into value. The best way to do that is by creating data products. How you get there might be different but don't lose focus on turning data into value. "The number one barrier to your ability to drive value of data is not your technology, it's your people and how you organize your team." Focus on the point of what you are trying to deliver, not the actual output. It's not about delivering a dashboard, it's about creating a sustainable way to explore, share, and consume information/insights, whatever form that takes. !Controversial!: There are 3 phases to getting to data driven; 1) is building a data lake or ocean, 2) is data mesh, and 3) is getting to a data product factory equivalent. It's easy to try to put the cart before the horse in data. Before doing something like data mesh, you have to think how you will develop data as a function in your organization. Understanding the data product manager role and leveraging data product managers well is crucial to building an effective data product strategy and practice. They are your data product CEOs. A CDO's effectiveness depends on if they have a true seat at the exec table - can they create the necessary change - and how many people in the organization are "devoted to the data opportunity." ?Controversial?: "Really smart data leaders hire for the business department." Understanding if someone cares about data and if you can work with them is important. If that potential head of marketing hire doesn't care about data or has low data fluency, will it be possible to work together? The companies doing the best on data literacy are making data a crucial part of their culture. Their daily practices increase data fluency across the entire organization rather than in a centralized data team. It's not about just training, it's about changing habits. Data driven companies are "162% more likely to surpass their revenue goals." "We're seeing people kind of rushing into migrating and not thinking about governance." ?Controversial?: There are two components that significantly increase the chances of successfully transforming to being a data driven organization: 1) a true organizational mandate to become data driven and work on data products and 2) does your organization have the attitude and aptitude to drive towards being data driven. There isn't a clear pattern yet for the best way to find your data product managers - teach the business aspects to the data people or vice versa. But it's clear that understanding what value data products drive - not just the ins and outs of the data product itself - is crucial. More and more, the centralized data team model is getting swamped in large organizations. But too many are too happy to fully decentralize, which also causes many issues. Federated - decentralized control and work but centralized collaboration and practices - is the approach seeing the most success. ?Controversial?: There are 3 types of data products: 1) internal domain-focused (data on the inside, not very reusable), 2) core, centrally managed, and 3) everywhere in between :) A major issue in the organizations that focus on empowering domains without interoperability is that - surprise, surprise - there are different semantic meanings and so the data becomes very difficult to integrate/interoperate. Focusing too much on quick wins will mean you miss out on the places where data can add a lot of value. Quick wins are typically not big wins or you find the big wins fast and then all your following quick wins are moderately sized at best. Bruno started off with something you don't often hear a vendor say: "The number one barrier to your ability to drive value of data is not your technology, it's your people and how you organize your team." So while you can't buy your way to a data mesh, you also can't just flip a switch and be doing data mesh. You need to build your organization's capabilities to a degree they actually can derive value from their data. It's also not easy for a data leader to necessarily create the necessary change per Bruno's conversations with data leaders. Many don't get a true seat at the executive table. And even if they do, if there aren't enough people "devoted to the data opportunity," it will be a very hard road to drive the data function to where it can add significant value. Bruno also dove into what he's seeing that makes for a high data literacy rate at customers - changing the day to day interaction, the habits of working with data. Making data part of many more people's roles and making it an intentional part of the company practices/habits builds an incredibly deep bench of data talent across the organizations. So his three components to positive change in your data approaches as a company are a strong data leader, the proportion of people committed to data work, and daily practices involving data. While we know being data driven has an advantage - data driven companies are 162% more likely to surpass their revenue goals per a study - Bruno sees a few reasons why only 27% of companies are actually data driven now. To be data driven, you need to reliably produce data at scale, hence creating data products. And to do that, you need to build out the capabilities to handle data at scale - and not skip the governance :) But the end goal is to provide a reliable way to create value from data. That's really it. The best way to reliably do that is via data products in his view. Bruno is seeing people go through three phases in getting to a reliable, scalable way to turn data into value. Phase 1 is the data ocean - it's not a lake, that's landlocked. The second stage is data mesh, allowing people to autonomously innovate with data but relying on central resources. And the third stage is a data factory. Scott note: the factory analogy might be rough because 1) feature factories are a very bad software pattern and 2) factories are notoriously about producing the same things at scale. And while we want scalable ways of creating data products, they should be more fit for purpose to use cases (but of course reusable as well) in my view. The data product manager role is crucial to getting data products right according to Bruno. You need someone to be the CEO for you data product, that is focused on the actual value the data product drives and how reliable is the data product creation/maintenance. What more should be added to the data product? How is it used? To drive that cultural shift, you need a strong leader of the data organization that is empowered to make the right changes. For Bruno, there are two factors that significantly increase the chance of an organization successfully becoming data driven. The first is an organization-wide mandate that data matters and that people must participate in the change and leverage data. Especially if the CEO is bought in on the data opportunity and the need for more and better data for themselves especially and the organization more broadly. The other is the attitude + aptitude to actually go out and build a scalable capability to build data products. And that's far easier said than done. That can be driven centrally or in a distributed way but you need people to step up and own the data. The centralized data team model is becoming harder and harder for companies to scale according to Bruno. The team needs to be constantly ahead of the curve and they don't have the ability to learn all the necessary context so they quickly get overwhelmed by requests. This was a key factor in Zhamak creating data mesh as a concept. But the teams that are just fully decentralizing are creating data silos and making it increasingly hard to answer cross domain questions. So the organizations that are doing a federated approach with a strong sense of overall collaboration are winning - there are things that are centralized and things that are decentralized and each organization needs to figure out what works for them but balance is crucial. Find the right approach for the job. Bruno talked about those companies that are focusing more on empowering domains than on the bigger picture of how domains can also work together. No major surprise but it creates data silos because everyone has different definitions so nothing is easy to integrate/interoperate. This is leading to the rise of the idea of the universal semantic layer. Quick tidbits: Data leaders should be involved in the hiring process for business people. That way, you can start to build a relationship early and help select someone who values data and has a decent data fluency. You don't want to be left out of the process. It's absolutely okay to have domain-only data products that are very specialized to that domain - basically data on the inside in a data product. It's also - per Bruno - okay to have very centralized data products that are pretty core across the organization. But look for places to build reusable data products to get the most leverage from your data work. To do data products and data product management right, you can't only focus on the data product launch. Maintenance and growth/evolution are crucial aspects of product thinking. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of January 29, 2023 23:04

۳ years پیش23:04

23:04

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#186 Zhamak's Corner 16 - An Interpretation of Zhamak's Call-to-Action 1:07:31

۳ years پیش1:07:31

1:07:31

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Sponsored by NextData , Zhamak's company that is helping ease data product creation. For more great content from Zhamak, check out her book on data mesh , a book she collaborated on , her LinkedIn , and her Twitter . We missed our window to record so I am interpreting what Zhamak is saying in her Medium post about why did she creator her company and the general state of the tooling market around data mesh. I also added on a the full 50min recording of our second recording which I had broken up into episodes 4-7. Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Data Mesh Radio episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#185 How the Heck Do We Do Federated Computational Governance Part 1 - Mesh Musings 41 20:47

۳ years پیش20:47

20:47

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. I will have much more to say on federated governance in data mesh but to reiterate my points: Federated doesn't mean decentralize everything. Really focus on enabling fast decisions with the people who have the understanding Shared ownership of responsibilities or oversight but not owned by a central team can work. Stop thinking that this shoves all the ownership to the domain on day 1, they aren't ready for that Start to plan out your governance roadmap pretty concretely at the start. But don't be worried about trying to have your ideal governance setup upfront. I've talked about CYA or cover your butt - think about what makes sense and what you can build as a skeleton upfront rather than trying to build a fully functioning body Start to think about interoperability at the concept level, that semantic level but you don't need to have your ultimate interoperability plan from the start. You can enhance and develop it as you go. And go talk to people about what they are doing! Blueprints. Do all the blueprints. Please, find friction and find a way to make easy buttons on this. There are 10s of episodes touching on this. Go look at what AgileLab is doing especially. Leave automated access control stuff for later. Focus on fast requesting and granting access Create data quality standards of measurement. That way everyone can understand how much they can trust a data product and why. Data contracts are about a producer/consumer relationship. This ain't a transaction. Get to know both sides and build to an understanding of what information is actually shared. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#184 Ontologies Don't Have to Be Scary: An Ontology Primer - Interview w/ Neda Abolhassani, PhD 1:14:20

۳ years پیش1:14:20

1:14:20

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Neda's LinkedIn: https://www.linkedin.com/in/neda-abolhassani-ph-d-61354329/ OSDU Ontology: https://github.com/Accenture/OSDU-Ontology In this episode, Scott interviewed Neda Abolhassani PhD, R&D Manager at Accenture Labs. To be clear, she was only representing her own views in this episode. There's some very specific language about ontology in this episode but I think it's quite approachable for most people as a good understanding of ontology, the difference with taxonomies, and some specific insight into developing and applying an ontology. Some key takeaways/thoughts from Neda's point of view: When starting developing an ontology, it's best to start from the business questions you want to answer. It is okay to choose bottom up or top down, but the business applicability is the main point. You can convince people ontologies and knowledge graphs aren't scary or that hard to learn and leverage with a small demo of what they do and how to use them. Look for open ontologies that have already been created around your domain or area you are trying to model. They can usually be easily augmented and extended but there's no reason to reinvent the wheel. Data people need to learn enough about the domain to build the right ontologies and data models but data people learning domain knowledge can "discombobulate" them :) Get the data people with the subject matter experts to learn what's necessary. Try to keep your ontology as generic as possible but still encapsulate what you need; that way it is much easier to apply the ontology to other domains/departments. Set your ontology up to evolve as you learn more about your organization and as your organization itself evolves too. You don't have to change your ontology simply because there are new use cases. If you designed it generically enough, it should be able to handle most new use cases whether as is or with a few additions. A good way to measure if your ontology is good enough and is still meeting your needs is to look at the business questions you want to answer. Are you able to answer them with your current setup? It's okay to have a global ontology and then ontologies that are more specific inside the domain if that is necessary to extract more value from the domain. ?Controversial?: Knowledge graphs need an ontology as well as a data model. So if you plan to leverage a knowledge graph for data mesh, you will need an ontology. When is best to actually start to deploy a knowledge graph and develop your ontology for data mesh - if you go that route - is still somewhat up in the air. The earlier the better in general but it will mean more work as things are unsettled early in your data mesh journey. Basically, it depends. Constant schema changes can make designing and updating your ontology more challenging. Whether that means you want to delay adding something immature to your ontology versus waiting until the schema settles a bit more, it's hard to say. Just, it can create additional challenges. ?Controversial?: To really have high-value interoperability in a data mesh, you need a way to capture not just the metadata of the data products but also the semantic meanings in the domains. And that should be done via a knowledge graph. Ontologies are richer than taxonomies because while both have definitions, ontologies also have description logic. This gives people an ability to define how data technically fits together across different aspects. OSDU, or open subsurface data universe, is a specific open source data platform for the oil and gas subsurface data. Neda developed an ontology to go along with that. Neda started off with a definition of ontology: "So literally, an ontology is a formal explicit specification of a shared conceptualization. I know that it has lots of jargon, but I'm going to explain it to you. So it is an abstract model of concepts, properties, relationships, and it is standardized, it is machine readable. And it is not just the instance level data, it doesn't include the instance level data, but it includes the schema and the type level information and how stuff should be connected in your domain." When asked if it's best to start top down or bottom up when thinking about building an ontology, Neda said either is acceptable but the main advice is to start from the business questions you want to answer. After all, this isn't an exercise for fun, there needs to be a business purpose. And look for open ontologies for your problem statement or industry. There are a number of ontologies that have already been created that you can leverage, extend, and/or use for inspiration. There is no real reason to reinvent the wheel. When building your ontology, Neda recommends keeping it as generic as possible. That way, you can apply it to multiple domains with no conflict. But it still has to meet your needs obviously. There are ontology editors to make things easier as well but it's important to set your ontology up to evolve as your understanding of your organization evolves and as your organization itself evolves. You can even do version control of your ontologies to make collaboration far easier as multiple parties look to improve the ontology simultaneously. For Neda, ontologies and knowledge graphs go hand-in-hand. It's okay to have an ontology for the global organization and another one for a specific domain if that's of value. Ontologies are typically about communicating externally from the domain or whatever grouping you are representing. And knowledge graphs are for integrating data from different sources or domains. And for knowledge graphs, you need an ontology and a data model for it. Ontologies are richer than taxonomies because while both capture the definitions, ontologies also have description logic. That description logic gives you a better ability to define things like unions, intersections, restrictions, and equivalences. So ontologies are broader than just concepts and terms. Neda then discussed the OSDU or open subsurface data universe - an open source data platform for subsurface data in the oil and gas space. She specifically saw a gap in OSDU where companies loading their own data into the OSDU format was pretty challenging. It required a lot of subject matter expert time to match schemas to the OSDU format. So Neda and team developed a technique using a knowledge graph and AI techniques to try to automatically match and map data in a company's own schema to the OSDU format. And as stated earlier, a knowledge graph needs an ontology :) As Neda worked to build out the OSDU ontology, she looked at the OSDU canonical data format and reviewed the schemas to understand what embedded choices were made so she could ensure she added that in to the ontology. She looked at the ontologies specific to related spaces or even some that were part of the OSDU area of interest like seismic data. However, the ontologies that existed for oil and gas were mostly outdated and didn't really cover what was really useful and interesting. An aspect of OSDU that made developing the ontology easier was that the schemas were not changing very often so there wasn't a constant remapping and versioning challenge. So, circling back to data mesh, Neda believes it's important to leverage a knowledge graph to really ensure good interoperability between domains and data products. A data catalogue - or other mechanism for discovering data in data mesh - that only has information about the individual data products and not how they interconnect won't have as much value.* When should you actually start to develop and deploy your knowledge graph is again something that requires more study and feedback. All else equal, the earlier the better, but of course as things are developing and changing rapidly early in your data mesh journey, trying to _also_ update your ontology will be a ton of extra work. Time will tell. *Scott note: yup. Isn't that just high quality data silos? Even if they interconnect, if people can't easily understand and find the interconnections, you likely lose a LOT of the value of data mesh. Whether knowledge graph and ontologies are the best approach remains to be seen however. Neda covered an aspect that is really important for all things data mesh: how to measure when things are good enough versus they need updating :) for her, it's about what she said at the start: what are the business questions you are trying to answer? If you are still able to answer those well enough, you probably don't need to change your ontology. But if those questions have changed considerably and your current implementation is not able to answer those questions well, your ontology will need to be updated - maybe some new concepts will be added and some old concepts deleted. You do want to be careful to try to keep things backward compatible as you deploy a new version of your ontology. Evolving ontologies is a challenging thing but if you designed your ontology well enough at the start, you probably don't need to do it all that often according to Neda. You should design your ontology in a generic enough way so it can handle new use cases without every little new aspect needing a whole new ontology version. However, that doesn't mean your ontology should never evolve. Things change or need clarification and you should be willing to be adaptable. Scott note: this is where Zhamak sees challenges with ontologies: if they are overly centralized and overly rigid, they prevent people from expressing real meaning at the data quantum level because they are trying to fit the definitions of the data quantum into the ontology. In wrapping up, Neda shared her views on how to really get started on building out a good ontology and knowledge graph. It will require your data people to learn enough about domains from the subject matter experts to develop the ontology. Be prepared for that to be a bit confusing as sometimes learning a lot of domain knowledge can "discombobulate" your data people. And it won't be a super quick exercise. But Neda believes it will pay out in the end and add a lot of value. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of January 22, 2023 13:44

۳ years پیش13:44

13:44

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#183 Business Intelligence's Place in Data Mesh - BI-gin With the End in Mind - Interview w/ Ryan Dolley 1:17:52

۳ years پیش1:17:52

1:17:52

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Ryan's LinkedIn: https://www.linkedin.com/in/ryandolley/ In this episode, Scott interviewed Ryan Dolley, an Independent Business Intelligence (BI) Consultant. Before we jump in, lots of things in the key takeaways are marked as potentially controversial. Because much of what Ryan covered hasn't really been stated by a lot of people. So until we have more consensus, of course things _could_ be controversial. Some key takeaways/thoughts from Ryan's point of view: Begin with the end in mind. It's easy to lose focus on what you are trying to accomplish instead of what steps you are taking. Focus on what the target outcome is and use that as a North Star to measure if you need to course correct. BI people need to brace themselves for a wave of innovation coming. There is so much - hopefully positive - change coming up the stack and BI people can embrace it or get washed over by the wave. Embrace and ride that wave and upskill! ?Controversial?: Data mesh - and just about every other paradigm - does not focus enough on the last mile of analytics, at least not explicitly. So we need to get a lot more specific about what is necessary to actually take advantage of upstream improvements in data to deliver better analytics. !Important!: We need to get more specific on who does cross-domain BI in a decentralized world. Otherwise, we have interoperable data but no one specifically leveraging that interoperability for improving our understanding of the overall organization. BI as a practice needs to be much better at understanding and implementing iterative feedback. It's not been part of the playbook to date. ?Controversial?: If domains develop BI capabilities, how will they interact with a central BI team if one exists? Historically, they haven't gotten along well... ?Controversial?: Your central BI / Data warehouse team should get converted into an insight generation engine team. We need to figure out if BI is a primary role inside the domain or if it is a skill people inside the domain need to develop as part of their roles. Which is preferable is likely dependent on the size, complexity, and data value of the domain. BI can't be a data mesh afterthought - we'll get to it when we get to it isn't going to cut it. It might not be literally part of the data mesh implementation but it must be part of your overall data strategy. Otherwise, why are you doing data work if not to generate and take action on insights? !Controversial!: The semantic layer is likely to play an important role in data mesh. Scott note: depends what you mean by semantic layer. See Zhamak's Corner #12 for more on the dangers of layers. BI teams must become part of the insight generation and resulting action conversations. That means implementing product thinking instead of simply responding to requests. But most are not ready for that mindset shift just yet. ?Controversial?: Domains should own an element of BI and not just share their data via data products but should generate and share their insights as part of data mesh. Product oriented thinking is not widely prevalent in the BI world - e.g. most BI teams do not understand the usefulness/usage of their BI assets or even have monitoring setup to track usage. !Controversial!: The BI analyst in the board room as merely a live question answer mechanism is a bad approach. It often leads to wrong answers with no meaningful quality control. BI people need to adopt product thinking and create meaningful relationships with users so they can evolve what they produce to continue to suit user needs. Blindly producing the same report for 4 years into the void needs to go the way of the dodo. !Important but Annoying!: We will probably have to meet some people where they are when it comes to sharing insights. Trying to upskill everyone to being an SQL whiz isn't realistic. Allowing people to export into Excel for further manipulation isn't going anywhere... Part of product thinking is designing to the user. That's pretty rare in data. What are they - the data consumer - specifically trying to do and how can the producing team support that? Instead of what does the producing team want to share, which is the common current method. Dashboards aren't dead, they are the best way to answer known questions in an oversight and management kind of way. ?Controversial?: BI is possibly the most important part of your data culture simply because for the vast majority of your organization, they will interact with data mostly at the BI level. But too many organizations leave BI as an afterthought in data culture because it will be where the most people need the most help and thus is the most difficult aspect. People change is hard :) ?Controversial?: In a successful data mesh implementation, there will be less work on the BI team because a lot more will be pushed down to the data product. The more the data product can share the insights inherently without an interpreter - the BI team - in the middle, the better. Ryan started with a bit of inspiration and/or a call to action to fellow business intelligence people: data mesh and a lot of other movements in data are creating a massive wave of innovation further up the data stack / data and analytics process. So BI people can either get with the wave and ride the changes to greater value or get washed over by it. Ride that wave :) Upskilling can be painful but it can mean a major upgrade in capabilities and value. And being highly tech-capable in BI is a great career move right now. According to Ryan, the last mile of analytics where BI really fits has been somewhat ignored in most modern data approaches and technical innovations. Tableau defined an era of BI and while Tableau isn't going away, that era is coming to a close. So he believes things like data mesh need to focus more on the last mile of analytics - or at least specifically call things out more often as to what changes and how in BI. Whether we call that self-serve or anything else. What does the upstream changes from data mesh mean for BI? How can we take advantage of these improvements in data processing, ownership, quality, etc. to deliver actual insights to those that matter? Ryan asked how should we actually do corporate BI in a data mesh type setup? Right now pre data mesh, most organizations have centralized BI teams. And while domains will probably need some BI capabilities, a recurrent theme in the podcast is that most exec questions are not single domain questions. So we build this huge amount of interoperable data, who is in charge of bringing that together at the corporate level? Who actually owns creating and maintaining C-suite and board-level dashboards and reports? Are we expecting execs to build those themselves? 🤣 And historically, central BI / Data warehouse teams haven't really loved embedded BI teams, often calling them "shadow IT". Can they play nicely together? If yes, how? There is often too much of a 'we'll cross that bridge when we come to it' relative to BI as part of most modern data strategies according to Ryan. While BI may not actually be part of the exact data mesh implementation at some organizations - there's all kinds of definitions and each org is extending the self-serve platform to encompass different aspects of analytics - BI definitely should be part of your data strategy. Otherwise, what is the point of doing data work if not to, you know, generate actionable insights? This is especially an issue when trying to embed insights and analytics in general into other applications. An important aspect of a data mesh implementation for Ryan is what is left to the BI tool and what is pushed down into the data product*. There isn't a lot of guidance or discussion happening exactly on what is a best practice or how people are implementing embedded insights, just that it's the future. There needs to be far more conversation about who owns data throughout the data and analytics process lifecycle. Is it all on domains? *Scott note: episode #40 w/ Xavier Gumara Rigol covers this well For Ryan, there are a few reasons why the general data warehouse has not been as successful as it could be. To actually interact with the data warehouse, people need a decent level of technical and BI skill. So when it comes to data mesh, we don't want there to be an even higher skill level required. So domains likely need to own some of their own BI, sharing insights from the domain in a consumable way to the rest of the organization. We can't only have domains sharing data, we need to be explicit about sharing information to a purpose. Is that to get the data, the insight, or the action that should be taken based on the insight? Scott note: there will be people in your organization that are only information consumers - taking info from dashboards and reports made for them. It's up to the organization to figure out if that is acceptable or to try to upskill those people against their will. While product oriented thinking is picking up in more areas of data, it's still not very common in the BI world according to Ryan. Most BI teams have no real idea of how useful any of what they produce is to the organization. Sometimes that is they don't have the systems set up to even monitor it but if they do, they typically don't understand how usage translates to value or honestly even look at the usage metrics. That means the BI teams lack the insights to make their BI assets as valuable as possible - what are the SLAs people need to make the most use of this data? How timely or accurate or complete does this need to be to drive the most value and be trustable? If BI people develop those relationships and that information flow, they can evolve what they produce to continue to meet needs. We need to stop producing work into the void; who is using this BI asset and why and how can we make it better and/or less costly to produce? A really important aspect of designing usable BI is thinking about the actual user in Ryan's view. It's often overlooked too and is part of product thinking: what is the product market fit for the BI asset / data product you are developing? How do people want to interact with it? How far along the pathway from raw data to insight to "so what" - the action that should be taken - is it the producer's responsibility to own and do the actual work? Does the consumer understand and agree? How can we generate things in data to create understanding and trust with the minimal amount of effort? Dashboards are great if you're monitoring a known question but we also need to get far better at answering how do we persuade someone with data. Ryan believes BI people need to move past the idea that the output is the end of the process. That dashboard or report or whatever is a communication mechanism about what is currently known. But it's not as if there is a single answer that stays the same. How many customers do we have is not a one-time answer question. There needs to be a lot of work to bring more iteration into BI, especially collaborating with consumers as both sides learn more about questions they are trying to answer. So BI teams need to move past being order takers and move to being part of the productization process around data and insights. Iterating in BI is not common practice but needs to be! BI is probably the most important aspect of your data culture according to Ryan. That's simply because it's the place where the vast majority of your organization will interact with your data, at the BI level. Whether that is in generating insights or merely consuming them, BI should not be left until the end of your data strategy. But it often is left to the end specifically because it is where there is the most human interaction and change management and that's the hardest part of change management, dealing with those pesky humans. Don't try to put it off until the end as it will just create more pain than is necessary. Ryan said "begin with the end in mind." It's quite easy in data work to get lost in the how instead of the why. The how is the fun/interesting technical challenges part. What are you trying to achieve? BI at the end of the day is not about generating insights, it's about supporting actions that help the business. So how do you build your BI strategy into your data mesh implementation strategy so you are not just making data available and trustable? How can you move past just generating insights? How do you think about the end - making smarter business decisions driven by trustable data - and work backwards? Don't lose sight of what you are trying to achieve. Ryan wrapped up the conversation by returning to the crucial points of integrating in product thinking to BI and upskilling. We can go further than we ever have before with data and BI but we have to embrace new ways of working, both on the strategy/data culture, and the technology. But just trying to advance your BI practice with technology is going to miss a whole lot of the value. Quick tidbits: We have to figure out in the domain if BI is a primary role or is it part of the necessary skillsets involved but not a separate role. It will probably be domain dependent, not even organization dependent, based on the complexity, size, and data value of the domain. Right now, most organizations want to embed insights into their general flow applications - e.g. the CRM for the sales team - but they aren't doing the back-end work to make that possible. Putting a business analyst in the boardroom simply as a vehicle to live answer questions usually causes more issues than it solves. It leads to a high pressure environment where you can't be sure of quality. Scott note: fully agree. There's a long rant I have about this… Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#182 Zhamak's Corner 15 - A Post Pipeline World? How to Build Data Products - and a Product Creation Ecosystem - That Inherently Create Trust 15:09

۳ years پیش15:09

15:09

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Sponsored by NextData , Zhamak's company that is helping ease data product creation. For more great content from Zhamak, check out her book on data mesh , a book she collaborated on , her LinkedIn , and her Twitter . We wrap up another recording of Zhamak's corner talking about how do we actually start to look to build data products in a post pipeline data world. Data tools right now are kind of duct taped to each other and duct taped to the pipeline - how do we rethink starting from the end product - that mesh data product - and hook the tools to that to make interacting with it better. If you build a system that truly focuses on intentionality and responsibility that people can see, it creates trust. Away with the data black box! Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Data Mesh Radio episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#181 Learnings from BlaBlaCar's Early Data Mesh Journey: Positive Transformation for the People and the Organization - Interview w/ Kineret Kimhi 1:12:48

۳ years پیش1:12:48

1:12:48

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Kineret's LinkedIn: https://www.linkedin.com/in/kineret-kimhi/ Kineret's Blog Post 'Do’s and Don’ts of Data Mesh': https://medium.com/blablacar/dos-and-don-ts-of-data-mesh-e093f1662c2d In this episode, Scott interviewed Kineret Kimhi, Analytics Lead at BlaBlaCar. Some key takeaways/thoughts from Kineret's point of view: !Interesting Decision!: BlaBlaCar reorganized their data organization but did not fully decentralize by embedding people into domains. Instead, they kept a central team but combined multiple functions into a squad around domains - a key domain might have a data engineer, data analyst, data scientist, and a software engineer. !Scott Mantra Too!: Sharing your experience - data mesh or otherwise - early and often with the broader data community means better and quicker feedback, not just internal experience. It's okay to be vulnerable about what didn't go well, you can get better info and help save others the same pain. ?Crucial?: It's very important that when you split up your teams from functional data role teams, people keep in contact with functional role peers. If not, it can be very lonely as the only data engineer inside a domain. There is a significant turnover risk and a risk to not having scalable learning and knowledge transfer of data work if not handled well. Data mesh will lead to a lot of potential changes to people's ways of working, especially with each other. Don't shy away from that, people need to know you aren't forgetting they need career development and that you'll support them as they learn and get used to new ways of working. Communication really is the most important aspect of getting data mesh right. You need to get feedback and keep people informed, aligned, and in sync. Change is less painful if people are told why it is happening and are informed instead of discovering the results of change. Look to align as much of your data team at the start of a data mesh journey - get everyone involved in the plan as best as possible. But don't completely change your data team structure at the beginning, start small. On that note, starting small means there isn't a huge disruption to your ways of working and data org so a team/domain can get comfortable with new ways of working. You can get great feedback to make it easier/better for the domains that follow if you keep in close contact. When doing your data mesh PoC, look for simple use cases. Set your PoC up for success and focus on learning how to do data mesh rather than tackling the hardest challenges first. Reorganizing your data teams can be frustrating to non-data team stakeholders. You can't have everything drop to the wayside while you learn to do data mesh. There needs to be a balance between learning a new way of working and continuing at least some semblance of business as usual around data work so you don't cause major disruption to exec's ways of working. It's really easy to go wrong with governance early in a data mesh journey. Getting people on the same page and on the same tooling is crucial so there is a better shared understanding, e.g. lineage, observability, catalogue, etc. If you do not have key core aspects of data governance in place, get those in place before starting a data mesh journey or you will make it much harder on yourself than it should be. ?Controversial?: Data documentation needs to get disrupted because it is still far too manual and too difficult for consumers to really understand without just experimenting with the data themselves. ?Often Overlooked?: It's important to recognize data mesh isn't the right fit for every organization. Why are you looking to do data mesh? Does data mesh even address your data challenges? !Important!: Make certain to give your PoC the resources - including people - needed to succeed but also set it up to succeed. Give the space and limit the pressure on your PoC to really learn if data mesh is for you and if your organization is ready to do data mesh. As you get domain teams to start owning their data, it's not a switch to flip. It's a process, work with them to get them capable and don't ask them to do overly complex things as they learn. Crawl, walk, jog, run. Make the team participating in your PoC feel like the pioneers and the potential heroes. Make it as easy on them as is practically possible and try to keep as much load off them as possible. Look to make participating in the PoC as beneficial to them as possible and not a burden. Set up a strong governance process to prevent schema changes from causing unnecessary downstream pain. But that can't be on the centralized team to make the changes, that's a bottleneck and doesn't scale. Don't expect data mesh to suddenly solve all your data challenges :) Kineret started off the conversation saying she was previously running data engineering at BlaBlaCar but with their move to data mesh, that isn't really a necessary role anymore so now she is the Analytics Lead. This was part of their greater reorganization of their data org where they are organized around domains into squads instead of by functions like data engineering. So crucial domains might have a data engineer, software engineer, data analyst, and data scientist in one squad focused on their data. So BlaBlaCar has a central data team but each squad is essentially attached to a domain. They kept their chapters around functional roles to keep knowledge sharing high and promote more camaraderie between similar roles attached in a squad to the different domains. BlaBlaCar sees the value in sharing their experiences early and often with the general data community so they can take in external feedback and also help out others looking to do similar things with data. Scott note: If I had a nickel for every time I tried to preach this… :) On advice as to how to start a data mesh journey, Kineret was relatively insistent that you need to form a group of people to partner with on the transformation. No matter your title, you need to have people to lean on and get feedback from. You can't drive it simply by force of will. Some things BlaBlaCar did that she believes helped make their journey (more) successful were: getting broad alignment around their data mesh journey, including around planning. It wasn't just a small team of people, all the data team worked together. Second aligns well with Zhamak's advice too: start small. It allowed them to get people used to a new way of working instead of trying to shift the entire data approach of the organization at once. They focused on collecting feedback from everyone involved in the PoC so they could see how well it worked and so that future domains could replicate the successful parts and avoid the ways of working that didn't go as well. As part of their 3mo PoC, Kineret and team took in a LOT of feedback from that single team. The reorg made a number of the data squad attached to the domain feel lonely and disconnected from others in a similar functional role. Again, back to the chapters approach to keep people connected around their role functions. Kineret said she believes if they tried to move all the domains at once to data mesh, their journey would have failed and they likely wouldn't have kept their data people nearly as happy because they might not have implemented the chapters approach early enough. To keep your overall organization on board with your data mesh journey, it's important to think about how stakeholders interact with the data team and keep that stable while you are in transition according to Kineret. If key stakeholders across the organization have to go through an entirely new and different process with each domain, those stakeholders are not going to be happy. So plan ahead and communicate what changes and have things well documented, not just at the data product level but at the business process level. If only you could do data mesh as if it were a separate thing unto itself but it's part of the business strategy, you can't keep it in a bubble. For Kineret, there have been some data mesh transition pain points especially around people moving into different roles or day-to-day responsibilities but communication is the key to keeping everyone aligned and limiting the unnecessary pain that often comes with change. If someone is used to talking with Alice in the central team about challenges with data and then Alice is suddenly in another domain, there is some frustration and concern by that data consumer as to who to talk to and how can they get to a good relationship with their new contact. But a key goal of data mesh is to make data consumers' overall experience better. So while it might be a bit challenging as things change, keeping data consumers informed of change and making sure there are low friction processes to get what they need are crucial. Kineret believes data documentation as a whole needs to be disrupted. We generally have some necessary pieces but it's still overly manual to properly document and consumers still generally can't really understand the data without digging into it themselves. The documentation generally isn't capable of doing what's necessary to get someone up to speed on a data product. Even though her team is doing great, it's still a challenge to find the right mix of important to in-depth. And it's still very manual work to create the documentation. Scott note: very much true. Data documentation is still an incredibly difficult task to get right and is probably far more tedious than it should be. When asked for some general data mesh getting started advice, Kineret had some beginning questions instead. What does your general data governance look like? If it's not robust, you should look to set that up before decentralizing. You'll save yourself a lot of unnecessary pain. Second, what is your buy-in for data mesh like and what is the reason for thinking data mesh is the right choice to solve your challenges?* Data mesh isn't right for every organization. Third, are you really ready to do your PoC and give it the resources necessary to succeed? Are you setting your PoC up to succeed by not putting too much pressure or trying to tackle too hard of a problem? Lastly, can you find a PoC use case that is relatively contained so this doesn't have too much outside influence and too many stakeholders? Can you clear the space to make it possible to succeed? *Scott note: so much this; if centralization isn't you bottleneck, decentralizing is far more likely to cause more issues. Be realistic about what data mesh can change and what it can't. Don't use an excavator to dig a 3 inch deep trench in your garden… A non-standard approach Kineret and team took was by separating ingestion and putting that on the data platform team. So the data domain teams could focus on the cleaning and transforming data instead of setting up the extracting from databases or other data stores. This was part of BlaBlaCar's capability building and data ownership transition strategy. They didn't ask or expect the world from the data domains as they were learning. Find relatively simple things for them to do instead of the most complex data engineering tasks. Crawl, walk, jog, run. Heading into the end of the conversation, Kineret really emphasized how important it is to get the people aspects of something like data mesh right. Make sure people can feel seen and heard, keep people informed, keep people in touch with those doing the same functional roles so your technical folks don't get too lonely, etc. Really make sure you focus much more than most technical people probably want to - the tech is cool! - on making this a positive transformative experience for the people in your organization, not just the organization itself. Quick tidbits: Governance can be really difficult but very crucial early in a journey. You want everyone on the same page relative to ways of working but also on the shared governance tooling, e.g. observability, cataloging, and lineage. Make sure to treat your initial PoC domain teams like pioneers and give them the support and guidance necessary. Keep undue burden off them as best as possible, keep in constant contact for feedback, and look to make it as beneficial as possible to them. Celebrate the PoC domain team because they went and did the big, kinda scary thing that could transform your organization. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
Weekly Episode Summaries and Programming Notes – Week of January 15, 2023 31:28

۳ years پیش31:28

31:28

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#180 Shared Understanding Leads to Data Value That's Outstanding - Interview w/ Chris Dove 1:13:07

۳ years پیش1:13:07

1:13:07

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Chris' LinkedIn: https://www.linkedin.com/in/charles-dove-b4715723/ In this episode, Scott interviewed Chris (Charles) Dove, Data Architect at Endava. To be clear, he was only representing his own views. Some key takeaways/thoughts from Chris' point of view: Data used by only one use case in one way is not how you make money by leveraging data, it's too expensive. Set yourself up to reuse data and make sure the organization is aware of what data is available. ?Controversial?: Tooling around data, especially metadata, has gotten better. But is it good yet? There are still some major fundamental gaps that seem like basic blocking and tackling around sharing data. Data isn't the point, it's merely a vehicle for exchanging information. Far too often there is an implicit understanding of a taxonomy/shared terms in different business units that is actually incorrect which leads to misunderstandings and mismatched data being treated as the same. But it's not easy to make all aspects of all parts of data explicit and easily understandable, we have to invest and find good ways to do that. !Incrementally Important!: Business people in domains often don't understand their own data because it's embedded in an application. So they only experience their data in a context that is already framed for them by the application. So they don't think about someone else not understanding the data inherently when those others _aren't_ experiencing it through the same application. Getting to a 'good enough' level of documentation is crucial to prevent misuse of data based on misunderstandings. But every organization has to figure out what is good enough and how to get there, what level of documentation is required, there isn't a blueprint. A constant challenge in data is implicit assumptions by producers around 'they work here so they know this' instead of documenting nuance. This leads to misunderstandings based on incomplete documentation. Beware the TLA - three letter acronym - in data documentation. It might have a lot of embedded context for those with domain knowledge but it's not helpful for those without the understanding. Focus on explaining the concepts at a level an outsider can understand. ?Controversial?: The point of data literacy isn't to teach everyone technical skills, it's to get to an understanding of their own data and how to share the context of their data so others can get a decent understanding of their data. If a good use case emerges, we don't need everyone to be able to create and maintain a data product but we need an organizational understanding of what data might be available and what it means. Tribal knowledge is a double edged sword. It's great your organization has that knowledge but it's a massive risk point. Get it out of people's heads as much as possible. Scott note: find low friction ways like interviewing people to extract their context. Like this podcast... Most organizations don't have a great data documentation strategy or practice. It's better to get going with learning how to share information about your data than try to make your data documentation or documentation strategy perfect upfront. Get something in place and recognize technical debt but something is better than nothing if it's not the end state. Trapped metadata - where tools try to enforce a closed system instead of easily offering up critical information for ingestion by other systems - is a persistent problem that doesn't seem to be getting better. It's even worse when you write custom code to do transformations because most people aren't creating the necessary metadata at all. We need vendors to be bought in that publishing their metadata is the right move and head towards metadata standards to make creating a more complete picture via metadata easy/feasible. Some vendors are moving in that direction, notably Atlan and data.world. Truly getting people to change the way they think and feel - not just do work - is an incredibly difficult challenge that most companies don't ever really address. Make the change in ways of working a value-add to actually change hearts and minds. To do data mesh well, we need to figure out scalable and highly effective ways of communicating changes - just offering data product versions won't cut it. To do that, data producers need to know how their data is used as well. Companies need to really see and understand the business benefit from their data before they are likely to change their ways of working around data. That can be a chicken-and-egg issue though. Chris started with his view that while tooling is getting better in general, most tools are still very lacking in how their metadata plays into the greater organizational view of data which means we can't do some pretty basic things. Or at least there aren't comprehensive tools that make easy sharing of context easy across teams because of many metadata incompatibilities/challenges. So we need to get to a way to show the semantic context-related metadata as well as the transformation metadata in one place that is also understandable by the business users. A hard order to be sure but fundamental to enabling the vision of companies being actually data-driven. There is a very common problem in organizations that comes from an implicit taxonomy and homonym problem according to Chris. The classic example is the definition of a customer but it goes far deeper than that. Some bit of data often has a very specific meaning in a source system and/or domain but then a different business unit looks at it with their own interpretation of what it means and misses the nuance, the differences. So do you have an enterprise taxonomy or do you try to document the exact meaning differences or do you not let people have access to data in case they misunderstand? Not as easy of a choice as many would like to prevent these misinterpretations or misaligned data mixing. An interesting and very crucial nuance Chris mentioned about data sharing: the business people in a domain are often consuming their own data through a different lens. The data is embedded into an application for them so the interface makes much of the nuance, the meaning explicit. But that meaning isn't included in the data by default - the column title in the table doesn't have that meaning. So those business people - the data producers - often struggle to understand why people are confused or don't get the nuance. So it's important to make sure the data producing domain understands the interface others use to consume their data. Scott note: this is benefit I hadn't considered of having domains consume from their own data products. If nuances about the data aren't explicit, if the documentation isn't good enough, will they get confused about their own data? Will that force them to do better in building their data products? Chris hit on a common problem many are having in data mesh - and data in general: what depth of documentation and explanation is necessary for data to be useful and not misused? People automatically assume some level of knowledge of the domain simply because others are in the organization. 'You work here so you understand my domain' type of attitude. So we need to make sure people can at least understand what they don't know and give them a way to get up to speed on what they need to know about a domain. Self-service can be a recipe for disaster if people can't understand when they are missing the necessary understanding/meaning. While domain-specific acronyms can have a lot of embedded information in them for people with knowledge of a domain, they are often a major hindrance to those trying to learn about the domain in Chris' experience. Instead of focusing on exactly what your team calls everything, focus on the concepts and why they matter. As Shakespeare said "what's in a name", don't be enamored with sharing context via domain specific language. Referring back to Vlad Khononov's DDD episode, the internal domain language is the ubiquitous language but the published language is what is used to share with the rest of the organization. Focus on that published language - how can things be understood easily by those outside the domain? For Chris, the point and meaning of data literacy isn't what most think - it's about getting people to understand what data they have and the general meaning/context so it can be communicated with the rest of the organization. It's understanding how data can be used and shared, not the exact technical aspects. It's about getting to a capability to share context and understand other's context around data without getting overly technical. When there is a use case that emerges, not every single person in the company needs to be able to create and maintain a data product. Basically, the concepts matter far more than everyone learning SQL. In Chris' view, tribal knowledge is a very dangerous place to be. You have amazing and extremely valuable knowledge but it's trapped in people's heads. What happens if they leave? We all know about tribal knowledge but it's especially important in data because again the context and nuance, not just the column name, matters :) So extract that valuable tribal information, get it into a consumable format for the entire organization. It frees up the time of your most knowledgeable people too as they aren't answering questions all the time - extract once but leveraged by many :) Good documentation, good knowledge sharing isn't about anticipating every challenge and writing out the fix or entirely preventing it according to Chris - that's not feasible. It's okay to get things into a knowledge base rather than the perfect metadata tool at first. You want to improve but if you are waiting for the perfect solution, you won't ever move forward with your data documentation. So get something out, put it in front of others, ask for feedback, and improve. And as stated, documentation doesn't have to answer all questions - it's something to make sure people generally understand a certain set of data and if they have a deeper question, they have a clear question escalation path for who to ask. It's easy to get lost in data by focusing on 'data as the point' in Chris' experience. Data is merely a vehicle for exchanging information. But there are lots of interesting technical challenges in dealing with data so data people often lose the plot. Without the context around the data, it's useless too, so we have to focus on delivering it as one packaged unit. Scott note: this is what Zhamak keeps referring to as a data product container or a unit of data - that it isn't merely the 1s and 0s but the context, the user experience, the lineage, etc. wrapped in one package so it is usable as is. For Chris, the biggest issue right now in data, especially for something like data mesh, is the trapped metadata problem. It's something Scott has mentioned repeatedly: most tools that touch your data in some way at best generate metadata that is trapped in that tool or it's extremely difficult to extract and integrate that metadata into other tools. And when people write custom code to do transformations, they often don't generate the metadata at all! So trying to get the full necessary picture of what's happening around our data is extremely time-consuming and difficult. Chris called out the need for specification around metadata so we can at least bring it all into one place. Only a few vendors have moved to making it possible to even extract most of the metadata they create - he noted Atlan and data.world - but hopefully more vendors are pushed - or dragged - into doing the same. OpenMetadata or other early projects may provide a good way to start developing some standards for how things are described, shared, and/or stored. But again, trapped metadata is a lock-in pain that vendors are seemingly unwilling to let go of unless their hands are forced. To really move forward with how we all approach data - as an industry and at the organizational level - we need to change the way people think and feel about data according to Chris. But change forced upon people only _might_ change their way of working and usually doesn't. So we have to focus on changing hearts and minds or the behavioral changes won't actually net the necessary care changes to the ways of working and understanding we need in data. Easier said - to change hearts and minds - than done but actually changing how we work requires empathy not mandates. Chris finished on two points: the first is to really change the way an organization does data, they have to understand how data fits into their overall strategy and how treating it as a product impacts the work. Something may be valuable now but that value might fade. It's okay - and even very healthy - to end of life any data work that is no longer valuable. And the second point is that reuse is really key to generating strong business benefit from data. The cost of getting data to a point you can leverage it is typically high, look to make it reusable and find valuable ways of reuse as much as possible. Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#179 Reliability Engineering for Data and Data Mesh - Mesh Musings 40 19:05

۳ years پیش19:05

19:05

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

D

Data Mesh Radio

1
#178 Data Modeling in Data Mesh Panel - w/ Juha Korpela, Kent Graziano, and Veronika Durgin 1:07:00

۳ years پیش1:07:00

1:07:00

Sign up for Data Mesh Understanding's free roundtable and introduction programs here: https://landing.datameshunderstanding.com/ Please Rate and Review us on your podcast app of choice! If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here Episode list and links to all available episode transcripts here . Provided as a free resource by Data Mesh Understanding / Scott Hirleman. Get in touch with Scott on LinkedIn if you want to chat data mesh. Transcript for this episode ( link ) provided by Starburst. See their Data Mesh Summit recordings here and their great data mesh resource center here . You can download their Data Mesh for Dummies e-book (info gated) here . Juha Korpela (Chief Product Officer at Ellie Technologies) facilitated this panel on data modeling in data mesh with Veronika Durgin (Head of Data at Saks) and Kent Graziano (The Data Warrior, former Chief Technical Evangelist at Snowflake). This panel was hosted by the Data Mesh Learning Community in partnership with Data Mesh Radio. Veronika's Links: Veronika's LinkedIn: https://www.linkedin.com/in/vdurgin/ Data Vault North America User Group: https://www.meetup.com/dvnaug/ Kent's Links: Kent's LinkedIn: https://www.linkedin.com/in/kentgraziano/ Kent's Website: https://kentgraziano.com/ Kent's Twitter: https://twitter.com/KentGraziano Data Vault Alliance: https://datavaultalliance.com/ Juha's Links: Juha's LinkedIn: https://www.linkedin.com/in/jkorpela/ Ellie Technologies' website: https://www.ellie.ai/ This write-up is from Scott Hirleman's point of view: As someone without a ton of depth in the data modeling concepts, here are some of my key takeaways that should be taken with a grain of salt :) I decided not to write up everyone's opinions but more what are my takeaways: Start from the business concepts instead of the technical (a data mesh theme). Focus on enabling people to do the data modeling instead of trying to do it centralized but do have a centralized understanding. It's crucial to not only consider data modeling at a data product level, that's the route to data silos. Communication is the most important data modeling skill. Share early and often to get feedback and work together on that fast feedback cycle to quickly iterate. Limit your blast radius, only bite off what you can chew and limit what can go sideways early as you learn - do not cause lots of downstream damage. If you don't stay connected to each other and communicate well, you'll likely have MDM (master data management) style nightmares in data mesh. Centrally define the standards and rules and have a clear way to get help and settle questions/disputes. Starting data modeling from technical integration is about what can we do right now. We should focus on what we need to do to drive business value, not what is possible based on the existing solution. In data modeling, far too often people don't look at what data consumers want. User requirements, business requirements, and technical requirements all must be met. Not everyone needs to be a data modeler but everyone should understand how information is communicated internally via a data model. Alla Hale method (episode #122): show up to every meeting with something to discuss. Don't show up to the first meeting with it built but constantly show something to get feedback on. Shared understanding is crucial and can ONLY be reached via strong communication. Stop trying to shy away from communication. Make the implicit explicit - "what do you mean by that?" is one of your most crucial tools to doing data modeling well. Data modeling in data mesh will probably be far more similar than dissimilar to data modeling in a more centralized world. The focus on the business concepts is crucial. Far too often we try to start from the technical instead of what are we trying to achieve and it's crucial to not fall into that trap. Getting the technical aspects for interoperability wrong can be a pain but if things work together technically but not at the business level, that's a lot of sound and fury signifying nothing - essentially that's a lot of cost for work and compute that doesn't lead to actual business value. One thing I'll note is every one of the guests is a Data Vault proponent. I'm not sold that it's the right way in the long run for data mesh - I feel like we need to evolve data modeling concepts for a more distributed, federated organizational approach. But from what they said, Data Vault does sound like a very solid base to start from - start from the business concepts first and focus on what you are actually trying to accomplish. Data modeling for the sake of data modeling is not something anyone should want to do. As with just about everything else in data mesh, data modeling should be about limiting your blast radius of potential negative impacts as you get to fast initial and incremental feedback. Get to that iteration, take on things that matter but don't make it a big bang. Fail fast and all that :) This is not about taking requirements and going off to your own world at the domain level. It's even more crucial to have overall communication/cohesion as we enable more and more people/domains to own their data. The most important aspect for developing lasting interoperability via data modeling is the business concepts. It's not that hard to do the technical interoperability once you figure out how things should work together. Starting from the technical feels easier but is a recipe for losing the value of the bigger picture. Typically, technical-based implementations are less extensible because the technology decisions are embedded into the solution instead of an enabling factor. Veronika said something like "focus on the words and meanings and not the data types." She also said that starting with technical integration focuses far too much on what can be done with the existing implementations instead of what do we need to drive value and how can we improve the existing implementations to create more value. As with most things in data and software engineering, it's okay to build your data model in an opinionated way but maintaining flexibility, especially as you are doing your initial development, is crucial. Major change can be quite costly - especially if you have to change the entire foundation of what you are doing. Kent railed against the need for rework. Build your data model and subsequent data products so providing another view or angle isn't nearly as difficult and requires only the work to do that, rather than changing everything else you've done to also accomplish the new data view - in data mesh, this would often be a different API or a new table sharing the data for a different use/perspective. Again, look to prevent rework and ensure flexibility. A massive concern with data mesh is data silos, the worry that if you have a bunch of domains doing their work separated and not in communication, nothing will interoperate. So you probably do need some kind of centralized group - whether that is their main role or part of their other responsibilities - helping domains do their work in the context of the greater organization. Note, that is what loose coupling from microservices means. Fully decentralized would be no connections versus things work together but can scale independently - that is decoupling. Having people who are there to help is the key to federated instead of fend-for-yourself so data architects are a crucial aspect of data modeling in data mesh. While there is no centralized data model in data mesh - they aren't flexible and mean you lose a ton of context from trying to force things to comply with that model - there obviously can be centralized guidance and direction, a standard set of data models, etc. Think about a well-functioning federated government - maybe not the US… There are people doing work in the centralized function but it's about enabling those at the more local level to do things the right way. Juha quoted someone with something like "governance is not about leading people to do things right, it's about setting them up to do things the right way". That centralized team can't know what's right for specific situations - because they lack the localized context - but they can specialize in enabling doing things the right way. Kent claimed there is an enterprise data model and that can quickly go the wrong direction but if you interpret it well - that there are clear relationships across the business that are crucial to model well in your data, that are fundamental business truths you should reflect in your data - it can mean much less learning of deep domain specific context because you understand how domains fit the organization. A number of people believe data modeling must be all about one view or perspective to rule them all. That is where data mesh fundamentally pushes back. You can have one view you agree on as an organization - such as revenue - but others should still be free to publish something that is similar in meaning but from another view. Much like in Domain Driven Design (listen to Vlad's episode for more, #171), there should be a 'language' (broad definition, think interface and terminology) of the domain to maximize the context of information shared in the domain and a separate 'language' used to communicate to the rest of the organization. That way we can still maximize context for business value locally but communicate globally in order to also maximize global data interoperability, which is a crucial organization-wide business value driver. Kent mentioned another worry of data mesh that is often closely aligned with the data silo worry: master data management-related nightmares. While we absolutely have to reinvent MDM for data mesh - look for a few panels on that in the near future? - it's pretty clear it's bad to potentially have 10 different definitions that might filter to someone who doesn't understand the nuance and differences. Especially if that exec asks a simple-seeming business question and gets 5 different answers. Data trust is gone. So we have to be clear in tackling that problem and strongly communicating. Maybe not mastering data but mastering ways of answering typical questions? Overall, I think you will learn a ton just like I (Scott) did :) Data Mesh Radio is hosted by Scott Hirleman. If you want to connect with Scott, reach out to him on LinkedIn: https://www.linkedin.com/in/scotthirleman/ If you want to learn more and/or join the Data Mesh Learning Community, see here: https://datameshlearning.com/community/ If you want to be a guest or give feedback (suggestions for topics, comments, etc.), please see here All music used this episode was found on PixaBay and was created by (including slight edits by Scott Hirleman): Lesfm , MondayHopes , SergeQuadrado , ItsWatR , Lexin_Music , and/or nevesf…

Amazon Basics Dog and Puppy Pee Pads, 5-Layer Leak-Proof Super Absorbent, Quick-Dry Surface, Potty Training, Regular (22x22"), 100 Count, Blue & White

Pluto TV - Watch Free Movies, Shows & Live TV

Minecraft

پادکست هایی که ارزش شنیدن دارند

Data Mesh Radio « » Weekly Episode Summaries and Programming Notes – Week of August 27, 2023

بایگانی مجموعه ها ("فیدهای غیر فعال" status)

Weekly Episode Summaries and Programming Notes – Week of August 27, 2023

بایگانی مجموعه ها ("فیدهای غیر فعال" status)

پادکست هایی که ارزش شنیدن دارند

Data Mesh Radio « »
Weekly Episode Summaries and Programming Notes – Week of August 27, 2023