Practical Data Pipeline: Build a Plant Monitoring System with ksqlDB

Streaming Audio: Apache Kafka® & Real-Time Data

Apache Kafka® isn’t just for day jobs according to Danica Fine (Senior Developer Advocate, Confluent). It can be used to make life easier at home, too!

Building out a practical Apache Kafka® data pipeline is not always complicated—it can be simple and fun. For Danica, the idea of building a Kafka-based data pipeline sprouted with the need to monitor the water level of her plants at home. In this episode, she explains the architecture of her hardware-oriented project and discusses how she integrates, processes, and enriches data using ksqlDB and Kafka Connect, a Raspberry Pi running Confluent's Python client, and a Telegram bot. Apart from the script on the Raspberry Pi, the entire project was coded within Confluent Cloud.

Danica's model Kafka pipeline begins with moisture sensors in her plants streaming data that is requested by an endless for-loop in a Python script on her Raspberry Pi. The Pi in turn connects to Kafka on Confluent Cloud, where the plant data is sent serialized as Avro. She carefully modeled her data, sending an ID along with a timestamp, a temperature reading, and a moisture reading. On Confluent Cloud, Danica enriches the streaming plant data, which enters as a ksqlDB stream, with metadata such as moisture threshold levels, which is stored in a ksqlDB table.

She windows the streaming data into 12-hour segments in order to avoid constant alerts when a threshold has been crossed. Alerts are sent at the end of the 12-hour period if a threshold has been traversed for a consistent time period within it (one hour, for example). These are sent to the Telegram API using Confluent Cloud's HTTP Sink Connector, which pings her phone when a plant's moisture level is too low.

Potential future project improvement plans include visualizations, adding another Telegram bot to register metadata for new plants, adding machine learning to anticipate watering needs, and potentially closing the loop by pushing data back

to the Raspberry Pi, which could power a visual indicator on the plants themselves.
EPISODE LINKS

• Apache Kafka at Home: A Houseplant Alerting System with ksqlDB
• GitHub: raspberrypi-houseplants
• Data Pipelines 101
• Tips for Streaming Data Pipelines ft. Danica Fine
• Motion in Motion: Building an End-to-End Motion Detection and Alerting System with Apache Kafka and ksqlDB
• Watch the video version of this podcast
• Danica Fine's Twitter
• Kris Jenkins’ Twitter
• Streaming Audio Playlist
• Join the Confluent Community
• Learn more on Confluent Developer
• Live demo: Intro to Event-Driven Microservices with Confluent
• Use PODCAST100 to get an additional $100 of free Confluent Cloud usage (details)

فصل ها

1. Intro (00:00:00)

2. Practical Kafka data pipeline (00:02:24)

3. The architecture (00:04:14)

4. Raspberry Pi (00:07:46)

5. Data modeling (00:11:08)

6. Avro schema (00:16:38)

7. Integrates, processes, and enriches data (00:19:11)

8. Windowing (00:21:55)

9. HTTP sink connector (00:24:27)

10. What's next (00:28:49)

11. It's a wrap (00:32:00)