Optimizing Large-Scale Deployments at LinkedIn with Rahul Gade

The Data Flowcast: Mastering Airflow for Data Engineering & AI

محتوای ارائه شده توسط The Data Flowcast. تمام محتوای پادکست شامل قسمت‌ها، گرافیک‌ها و توضیحات پادکست مستقیماً توسط The Data Flowcast یا شریک پلتفرم پادکست آن‌ها آپلود و ارائه می‌شوند. اگر فکر می‌کنید شخصی بدون اجازه شما از اثر دارای حق نسخه‌برداری شما استفاده می‌کند، می‌توانید روندی که در اینجا شرح داده شده است را دنبال کنید.https://fa.player.fm/legal

20d ago 27:47

MP3•خانه قسمت

Scaling deployments for a billion users demands innovation, precision and resilience. In this episode, we dive into how LinkedIn optimizes its continuous deployment process using Apache Airflow. Rahul Gade, Staff Software Engineer at LinkedIn, shares his insights on building scalable systems and democratizing deployments for over 10,000 engineers.

Rahul discusses the challenges of managing large-scale deployments across 6,000 services and how his team leverages Airflow to enhance efficiency, reliability and user accessibility.

Key Takeaways:

(01:36) LinkedIn minimizes human involvement in production to reduce errors.

(02:00) Airflow powers LinkedIn’s Continuous Deployment platform.

(05:43) Continuous deployment adoption grew from 8% to a targeted 80%.

(11:25) Kubernetes ensures scalability and flexibility for deployments.

(12:04) A custom UI offers real-time deployment transparency.

(16:23) No-code YAML workflows simplify deployment tasks.

(17:18) Canaries and metrics ensure safe deployments across fabrics.

(20:45) A gateway service ensures redundancy across Airflow clusters.

(24:22) Abstractions let engineers focus on development, not logistics.

(25:20) Multi-language support in Airflow 3.0 simplifies adoption.

Resources Mentioned:

Rahul Gade -

https://www.linkedin.com/in/rahul-gade-68666818/

LinkedIn -

https://www.linkedin.com/company/linkedin/

Apache Airflow -

https://airflow.apache.org/

Kubernetes -

https://kubernetes.io/

Open Policy Agent (OPA) -

https://www.openpolicyagent.org/

Backstage -

https://backstage.io/

Apache Airflow Survey -

https://astronomer.typeform.com/airflowsurvey24

Thanks for listening to The Data Flowcast: Mastering Airflow for Data Engineering & AI. If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

37 قسمت

#Tech #Airflow