Building Real-Time Analytics Systems

Gain deep insight into real-time analytics, including the features of these systems and the problems they solve. With this practical book, data engineers at organizations that use event-processing systems such as Kafka, Google Pub/Sub, and AWS Kinesis will learn how to analyze data streams in real time. The faster you derive insights, the quicker you can spot changes in your business and act accordingly.

Author Mark Needham from StarTree provides an overview of the real-time analytics space and an understanding of what goes into building real-time applications. The book’s second part offers a series of hands-on tutorials that show you how to combine multiple software products to build real-time analytics applications for an imaginary pizza delivery service.

You will:

• Learn common architectures for real-time analytics
• Discover how event processing differs from real-time analytics
• Ingest event data from Apache Kafka into Apache Pinot
• Combine event streams with OLTP data using Debezium and Kafka Streams
• Write real-time queries against event data stored in Apache Pinot
• Build a real-time dashboard and order tracking app
• Learn how Uber, Stripe, and Just Eat use real-time analytics

Table of Contents

Chapter 1. Introduction to Real-Time Analytics

Chapter 2. The Real-Time Analytics Ecosystem

Chapter 3. Introducing All About That Dough: Real-Time Analytics on Pizza

Chapter 4. Querying Kafka with Kafka Streams

Chapter 5. The Serving Layer: Apache Pinot

Chapter 6. Building a Real-Time Analytics Dashboard

Chapter 7. Product Changes Captured with Change Data Capture

Chapter 8. Joining Streams with Kafka Streams

Chapter 9. Upserts in the Serving Layer

Chapter 10. Geospatial Querying

Chapter 11. Product ion Considerations

Chapter 12. Real-Time Analytics in the Real World

Chapter 13. The Future of Real-Time Analytics

This book is a practical guide for implementing real-time analytics applications on top of existing data infrastructure. It is aimed at data engineers, data architects, and application developers who have some experience working with streaming data or would like to get acquainted with it.

In Chapters 1 and 2, we give an introduction to the topic and an overview of the types of real-time analytics applications that you can build. We also describe the types of products/tools that you’ll likely be using, explaining how to pick the right tool for the job, as well as explaining when a tool might not be necessary.

In Chapter 3, we introduce a fictional pizza company that already has streaming infrastructure set up but hasn’t yet implemented any real-time functionality. The next seven chapters will show how to implement different types of real-time analytics applications for this pizza company. If you’re interested in getting your hands dirty, these chapters will be perfect for you, and hopefully you’ll pick up some ideas (and code!) that you can use in your own projects.

The book will conclude with considerations when putting applications into production, a look at some real-world use cases of real-time analytics, and a gaze into our real-time analytics crystal ball to see what might be coming in this field over the next few years.

About the Author

Mark Needham is an Apache Pinot advocate and developer relations engineer at StarTree.

As a developer relations engineer, Mark helps users learn how to use Apache Pinot to build their real-time user-facing analytics applications. He also does developer experience, simplifying the getting started experience by making product tweaks and improvements to the documentation. Mark previously worked in Developer Relations, Product Engineering, and Field Engineering at Neo4j, where he co-authored O’Reilly’s Graph Algorithms book.