Distributed Systems Engineering with Apache Kafka ft. Guozhang Wang

Streaming Audio: Apache Kafka® & Real-Time Data

Tim Berglund picks the brain of a distributed systems engineer, Guozhang Wang, tech lead in the Streaming department of Confluent. Guozhang explains what compelled him to join the Stream Processing team at Confluent coming from the Apache Kafka® core infrastructure. He reveals what makes the best distributed systems infrastructure engineers tick and how to prepare to take on this kind of role—solving failure scenarios, a satisfying challenge.

One challenge in distributed systems is achieving agreements from multiple nodes that are connected in a Kafkacluster, but the connection in practice is asynchronous.

Guozhang also shares the newest updates in the Kafka community, including the coming ZooKeeper-free architecture where metadata will be maintained by Kafka logs.

Prior to joining Confluent, Guozhang worked for LinkedIn, where he used Kafka for a few years before he started asking himself, “How fast can I get value from the data that I’ve collected?” This question eventually led him to begin building Kafka Streams and ksqlDB. Ever since, he’s been working to advance stream processing, and in this episode, provides an exciting preview of what’s to come.

EPISODE LINKS

• Join the Confluent team
• Diving into Exactly-Once Semantics with Guozhang Wang
• In Search of an Understandable Consensus Algorithm
• The Curious Incident of the State Store in Recovery in ksqlDB
• From Eager to Smarter in Apache Kafka Consumer Rebalances
• KIP-595: A Raft Protocol for the Metadata Quorum
• Join the Confluent Community Slack
• Learn more with Kafka tutorials, resources, and guides at Confluent Developer
• Live demo: Kafka streaming in 10 minutes on Confluent Cloud
• Use 60PDCAST to get an additional $60 of free Confluent Cloud usage (details)