Using Apache Kafka and ksqlDB for Data Replication at Bolt

Streaming Audio: Apache Kafka® & Real-Time Data

What does a ride-hailing app that offers micromobility and food delivery services have to do with data in motion? In this episode, Ruslan Gibaiev (Data Architect, Bolt) shares about Bolt’s road to adopting Apache Kafka® and ksqlDB for stream processing to replicate data from transactional databases to analytical warehouses.

Rome wasn't built overnight, nor was the adoption of Kafka and ksqlDB at Bolt. Initially, Bolt noticed the need for system standardization and replacing the unreliable query-based change data capture (CDC) process. As an experienced Kafka developer, Ruslan believed that Kafka is the solution for adopting change data capture as a company-wide event streaming solution. Persuading the team at Bolt to adopt and buy in was hard at first, but Ruslan made it possible. Eventually, the team replaced query-based CDC with log-based CDC from Debezium, built on top of Kafka. Shortly after the implementation, developers at Bolt began to see precise, correct, and real-time data.

As Bolt continues to grow, they see the need to implement a data lake or a data warehouse for OTP system data replication and stream processing. After carefully considering several different solutions and frameworks such as ksqlDB, Apache Flink®, Apache Spark™, and Kafka Streams, ksqlDB shines most for their business requirement.

Bolt adopted ksqlDB because it is native to the Kafka ecosystem, and it is a perfect fit for their use case. They found ksqlDB to be a particularly good fit for replicating all their data to a data warehouse for a number of reasons, including:

• Easy to deploy and manage
• Linearly scalable
• Natively integrates with Confluent Schema Registry

Turn in to find out more about Bolt’s adoption journey with Kafka and ksqlDB.

EPISODE LINKS

• Inside ksqlDB Course
• ksqlDB 101 Course
• How Bolt Has Adopted Change Data Capture with Confluent Platform
• Analysing Changes with Debezium and Kafka Streams
• No More Silos: How to Integrate Your Databases with Apache Kafka and CDC
• Change Data Capture with Debezium ft. Gunnar Morling
• Announcing ksqlDB 0.17.0
• Real-Time Data Replication with ksqlDB
• Watch the video version of this podcast
• Join the Confluent Community
• Learn more with Kafka tutorials, resources, and guides at Confluent Developer
• Live demo: Kafka streaming in 10 minutes on Confluent Cloud
• Use PODCAST100 to get an additional $100 of free Confluent Cloud usage (details)