Designing Event-Driven Systems

I’m hopeful that the ideas outlined in this book are another step forward in how we think about sharing and exchanging data, helping us change how we build microservice architecture. The ideas may well seem odd at first, but stick with them. Ben is about to take you on a very interesting journey. —Sam Newman

Many forces affect software today: larger datasets, geographical disparities, complex company structures, and the growing need to be fast and nimble in the face of change. Proven approaches such as service-oriented and event-driven architectures are joined by newer techniques such as microservices, reactive architectures, DevOps, and stream processing. Many of these patterns are successful by themselves, but as this practical book demonstrates, they provide a more holistic and compelling approach when applied together.

Author Ben Stopford explains how service-based architectures and stream processing tools such as Apache Kafka can help you build business-critical systems. You’ll learn how to apply patterns including Event Sourcing and CQRS, and how to build multi-team systems with microservices and SOA using patterns such as “inside out databases” and “event streams as a source of truth.” These approaches provide a unique foundation for how these large, autonomous service ecosystems can communicate and share data.

•  Learn why streaming beats request-response based architectures in complex, contemporary use cases

•  Understand why replayable logs such as Kafka provide a backbone for both service communication and shared datasets

•  Explore how event collaboration and event sourcing patterns increase safety and recoverability with functional, event-driven approaches

•  Build service ecosystems that blend event-driven and request-driven interfaces using a replayable log and Kafka’s Streams API

•  Scale beyond individual teams into larger, department- and company-sized architectures, using event streams as a source of truth

Contents

Part I. Setting the Stage

Chapter 1. Introduction

Chapter 2. The Origins of Streaming

Chapter 3. Is Kafka What You Think It Is?

Chapter 4. Beyond Messaging: An Overview of the Kafka Broker

Part II. Designing Event-Driven Systems

Chapter 5. Events: A Basis for Collaboration

Chapter 6. Processing Events with Stateful Functions

Chapter 7. Event Sourcing, CQRS, and Other Stateful Patterns

Part Ill. Rethinking Architecture at Company Scales

Chapter 8. Sharing Data and Services Across an Organization

Chapter 9. Event Streams as a Shared Source of Truth

Chapter 10. l ean Data

Part IV. Consistency, Concurrency, and Evolution

Chapter 11. Consistency and Concurrency in Event-Driven Systems

Chapter 12. Transactions, but Not as We Know Them

Chapter 13. Evolving Schemas and Data over Time

Part V. Implementing Streaming Services with Kafka

Chapter 14. Kafka Streams and KSQL

Chapter 15. Building Streaming Services

About Me

I work as an architect and engineer with most of my experience focussed on distributed data technologies. I manage teams and work with larger organisational issues too.

I’m currently at Confluent leading the Office of the CTO (a research function) although my role has spanned most departments at one point or other over a range of projects from developing a strategy for streaming system design through to engineering a new version of Kafka’s replication protocol.

Previously I led the central data platform team at RBS where we built the event-driven database: ODC. I also worked at Thoughtworks and a few financial institutions. I live in London.