27 subscribers
با برنامه Player FM !
پادکست هایی که ارزش شنیدن دارند
حمایت شده
Running Kafka on Kubernetes, across three clouds at Adobe (DoK Day EU 2022) // Adi Muraru
Manage episode 329958508 series 2865115
https://go.dok.community/slack
From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)
Adobe runs dozens of Kafka clusters spread across both public (AWS and Azure) and private clouds to power the Adobe Experience Platform message bus.
In this talk, we share the experience of running Kafka as a stateful Kubernetes workload in a cloud-native environment using the open-source Koperator and the tools we built to automate the whole lifecycle.
While running a stateful service like Kafka on Kubernetes may be intimidating at the first glance, we share our thought process, the tools, and the results that can make this a reality in any organization. The Kubernetes Operator pattern helped us automate all the operational aspects for the lifecycle of the cluster; abstract away the cloud specifics allowing us to focus on Kafka; achieve increased resilience and elasticity; implement automated Kafka rebalancing using CruiseControl, and harness all the metrics to implement an observable environment. We also plan to demo how these all come together.
Join us as we share our team's learnings on:
- Considerations for running a stateful service in a multi-tenant Kubernetes cluster:
- Pod sizing in terms of memory and CPU
- Scheduling
- Persistent Volume Claims (PVC) tuning
- Resiliency topics
- How to handle regular Ethos upgrades and maintenance
- How to bring custom monitoring solutions
- Advanced network topics such as using dedicated VIPs
If you or your team ever wanted to learn how to run your own stateful service in Kubernetes and/or see what's possible, come join us!
Computer Science Ph.D. and Principal Scientist at Adobe having experience in complex, large scale distributed systems.
My focus currently is on designing and building the message streaming platform for Adobe Experience Platform using Apache Kafka. The endeavor is unique as it spans both Amazon AWS and Microsoft Azure clouds and handles billions of messages a day.
243 قسمت
Manage episode 329958508 series 2865115
https://go.dok.community/slack
From the DoK Day EU 2022 (https://youtu.be/Xi-h4XNd5tE)
Adobe runs dozens of Kafka clusters spread across both public (AWS and Azure) and private clouds to power the Adobe Experience Platform message bus.
In this talk, we share the experience of running Kafka as a stateful Kubernetes workload in a cloud-native environment using the open-source Koperator and the tools we built to automate the whole lifecycle.
While running a stateful service like Kafka on Kubernetes may be intimidating at the first glance, we share our thought process, the tools, and the results that can make this a reality in any organization. The Kubernetes Operator pattern helped us automate all the operational aspects for the lifecycle of the cluster; abstract away the cloud specifics allowing us to focus on Kafka; achieve increased resilience and elasticity; implement automated Kafka rebalancing using CruiseControl, and harness all the metrics to implement an observable environment. We also plan to demo how these all come together.
Join us as we share our team's learnings on:
- Considerations for running a stateful service in a multi-tenant Kubernetes cluster:
- Pod sizing in terms of memory and CPU
- Scheduling
- Persistent Volume Claims (PVC) tuning
- Resiliency topics
- How to handle regular Ethos upgrades and maintenance
- How to bring custom monitoring solutions
- Advanced network topics such as using dedicated VIPs
If you or your team ever wanted to learn how to run your own stateful service in Kubernetes and/or see what's possible, come join us!
Computer Science Ph.D. and Principal Scientist at Adobe having experience in complex, large scale distributed systems.
My focus currently is on designing and building the message streaming platform for Adobe Experience Platform using Apache Kafka. The endeavor is unique as it spans both Amazon AWS and Microsoft Azure clouds and handles billions of messages a day.
243 قسمت
Tüm bölümler
×

1 Unsticking Ourselves from Glue: Migrating PayIt’s Data Pipelines to Argo Workflows and Hera | DoKC Town Hall 23:17

1 Repel Boarders! How to find a Kubernetes operator that really protects your data | DoKC Town Hall 19:22

1 DoK @ Comcast - Deliver Business Outcomes & Improved DevX with Data Services on K8s | DoKC Town Hall 16:43

1 DoK Talks - What is Kafka? The rise of one of the world's most used streaming data technologies // Abbey Russell 15:28

1 DoK Talks - (almost)Everything you need to know about stateful cloud native network applications // W Watson 43:39

1 The Outer Nerd #001 - Dungeons & Dragons - Why should you care? // Abhi Vaidyanatha, Fabian Met & Chase Christensen 58:25


1 Data-driven Diversity, Equity, and Inclusion // Lisa-Marie Namphy, Melissa Logan, Tiffany Jachja, Audra Montenegro & Cortney Nickerson (DoK Day North America 2022) 19:50

1 Formula 1 telemetry processing using Apache Kafka on Kubernetes // Paolo Patierno (DoK Day North America 2022) 15:36

1 Choosing Kubernetes for Stateful Applications // Akshay Ram & Peter Schuurman (DoK Day North America 2022) 18:31

1 Kubernetes 360º - Data driven observability - from Secrets to logs // Ben Hirschberg (DoK Day North America 2022) 17:11

1 Shifting Left Stateful Applications In Kubernetes // Viktor Farcic (DoK Day North America 2022) 15:52

1 Medical - Healthcare Data on Kubernetes // Olyvia Rakshit & Prasad Dorbala (DoK Day North America 2022) 13:41

1 Highly Available Postgres Clusters In Kubernetes // John Long & Jonathan Gonzalez (DoK Day North America 2022) 15:04


1 Open Source Databases on Kubernetes- Best Practices // Peter Zaitsev (DoK Day North America 2022) 16:04


1 Databases on Kubernetes: Why are they important? // With Bhavin Shah, Xing Yang, Gabriele Bartolini & Patrick McFadin (DoK Day North America 2022) 34:51


1 Architecting Your First Event Driven Serverless Streaming Applications on K8 // Timothy Spann (DoK Day North America 2022) 13:29

1 Fybrik - A Kubernetes based platform for governed data use // Flora Gilboa-Solomon, Alexey Roytman, Maryna Strelchuk & Barry Hijkoop (DoK Day North America 2022) 20:59

1 The Challenges of Data Processing On Kubernetes - A look at Spark, Flink, Dask, and Ray // Holden Karau (DoK Day North America 2022) 20:09

1 Scaling our SaaS offering to thousands of clusters // Dax McDonald (DoK Day North America 2022) 21:04

1 Why we decided to migrate our Jaeger storage to ClickHouse on Kubernetes // Arul Jegadish Francis (DoK Day North America 2022) 13:48

1 Building a Digital Factory for the Sheet Metal Industry // Elie Assi (From the DoK Day North America 2022) 20:48

1 How we built our Big Data Stack (almost) entirely on top of Kubernetes // Neylson Crepalde (From DoK Day NA 2022) 16:00


1 Dok #152-Running PostgreSQL in Kubernetes:from day 0 to day 2 with CloudNativePG // Gabriele Bartolini 1:03:50


1 Dok Talks #150 - Building a Simple Postgres Async Streaming Cluster // Julian Fischer 1:04:45

1 DoK Talks #149 - Overcoming challenges with protecting and migrating data in multi-cloud K8s environments // Sebastian Glab & Martin Phan 47:40

1 DoK Talks #147 - Evaluating Cloud Native Storage Vendors // Dinesh Majrekar 1:00:03

1 Dok Talks #146 - OpenFeature - Making feature flags a commodity // Oleg Nenashev 1:01:30


1 DoK Talks #144 - We will Dok You! - The journey to adopt stateful workloads on k8s // Guy Menahem 1:06:30

1 DoK Talks #142 - Kubernetes Cluster Upgrade Strategies and Data: Best Practices for your Stateful Workload // Peter Schuurman 58:45

1 DoK Talks #144 - Mastering MongoDB on Kubernetes, the power of operators // Arek Borucki 1:00:50


1 DoK Talks #141 - Dossier: multi-tenant distributed Jupyter Notebooks // Iacoppo Colonnelli & Dario Tranchitella 1:00:10

به Player FM خوش آمدید!
Player FM در سراسر وب را برای یافتن پادکست های با کیفیت اسکن می کند تا همین الان لذت ببرید. این بهترین برنامه ی پادکست است که در اندروید، آیفون و وب کار می کند. ثبت نام کنید تا اشتراک های شما در بین دستگاه های مختلف همگام سازی شود.