S1E8 Data quality on Modern Data Stack

Data engineering and analytics for leaders

In a modern data stack, data is collected from various sources, such as databases, APIs, and third-party applications. This data is then processed and transformed into a usable format for analysis. However, data quality can suffer at every stage of this process, leading to unreliable insights and flawed decision-making.

One of the biggest challenges of maintaining data quality in a modern data stack is the sheer volume and variety of data. With so much data coming in from different sources, ensuring that all data is accurate, complete, and consistent can be challenging.

Another challenge is data lineage. With data flowing through multiple systems, it can be difficult to track its origin and how it has been transformed over time. This lack of transparency can make it challenging to identify and address issues with data quality.