با برنامه Player FM !
پادکست هایی که ارزش شنیدن دارند
حمایت شده


1 Why You Need to Grow Your Customer’s Decision Confidence with Brent Adamson and Karl Schmidt 36:25
Airflow in Action: Powering Instacart's Complex Ecosystem
Manage episode 477521274 series 2948506
The evolution of data orchestration at Instacart highlights the journey from fragmented systems to robust, standardized infrastructure. This transformation has enabled scalability, reliability and democratization of tools for diverse user personas.
In this episode, we’re joined by Anant Agarwal, Software Engineer at Instacart, who shares insights into Instacart's Airflow journey, from its early adoption in 2019 to the present-day centralized cluster approach. Anant discusses the challenges of managing disparate clusters, the implementation of remote executors, and the strategic standardization of infrastructure and DAG patterns to streamline workflows.
Key Takeaways:
(03:49) The impact of external events on business growth and technological evolution.
(04:31) Challenges of managing decentralized systems across multiple teams.
(06:14) The importance of standardizing infrastructure and processes for scalability.
(09:51) Strategies for implementing efficient and repeatable deployment practices.
(12:17) Addressing diverse user personas with tailored solutions.
(14:47) Leveraging remote execution to enhance flexibility and scalability.
(18:36) Benefits of transitioning to a centralized system for organization-wide use.
(20:57) Maintaining an upgrade cadence to stay aligned with the latest advancements.
(23:35) Anticipation for new features and improvements in upcoming software versions.
Resources Mentioned:
https://www.linkedin.com/in/anantag/
Instacart | LinkedIn
https://www.linkedin.com/company/instacart/
Instacart | Website
https://www.instacart.com
https://airflow.apache.org/
https://aws.amazon.com/ecs/
https://www.terraform.io/
https://www.astronomer.io/events/roadshow/london/
https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/
https://www.astronomer.io/events/roadshow/san-francisco/
https://www.astronomer.io/events/roadshow/chicago/
Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
64 قسمت
Airflow in Action: Powering Instacart's Complex Ecosystem
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
Manage episode 477521274 series 2948506
The evolution of data orchestration at Instacart highlights the journey from fragmented systems to robust, standardized infrastructure. This transformation has enabled scalability, reliability and democratization of tools for diverse user personas.
In this episode, we’re joined by Anant Agarwal, Software Engineer at Instacart, who shares insights into Instacart's Airflow journey, from its early adoption in 2019 to the present-day centralized cluster approach. Anant discusses the challenges of managing disparate clusters, the implementation of remote executors, and the strategic standardization of infrastructure and DAG patterns to streamline workflows.
Key Takeaways:
(03:49) The impact of external events on business growth and technological evolution.
(04:31) Challenges of managing decentralized systems across multiple teams.
(06:14) The importance of standardizing infrastructure and processes for scalability.
(09:51) Strategies for implementing efficient and repeatable deployment practices.
(12:17) Addressing diverse user personas with tailored solutions.
(14:47) Leveraging remote execution to enhance flexibility and scalability.
(18:36) Benefits of transitioning to a centralized system for organization-wide use.
(20:57) Maintaining an upgrade cadence to stay aligned with the latest advancements.
(23:35) Anticipation for new features and improvements in upcoming software versions.
Resources Mentioned:
https://www.linkedin.com/in/anantag/
Instacart | LinkedIn
https://www.linkedin.com/company/instacart/
Instacart | Website
https://www.instacart.com
https://airflow.apache.org/
https://aws.amazon.com/ecs/
https://www.terraform.io/
https://www.astronomer.io/events/roadshow/london/
https://www.astronomer.io/events/roadshow/new-york/ https://www.astronomer.io/events/roadshow/sydney/
https://www.astronomer.io/events/roadshow/san-francisco/
https://www.astronomer.io/events/roadshow/chicago/
Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
64 قسمت
همه قسمت ها
×



1 Embracing Data Mesh and SQL Sensors for Scalable Workflows at lastminute.com with Alberto Crespi 30:09




1 Modernizing Legacy Data Systems With Airflow at Procter & Gamble with Adonis Castillo Cordero 22:13






1 From ETL to Airflow: Transforming Data Engineering at Deloitte Digital with Raviteja Tholupunoori 27:42

1 A Deep Dive Into the 2025 State of Airflow Survey Results with Tamara Fingerlin of Astronomer 23:26
به Player FM خوش آمدید!
Player FM در سراسر وب را برای یافتن پادکست های با کیفیت اسکن می کند تا همین الان لذت ببرید. این بهترین برنامه ی پادکست است که در اندروید، آیفون و وب کار می کند. ثبت نام کنید تا اشتراک های شما در بین دستگاه های مختلف همگام سازی شود.