Distributed Tracing in Practice

Since most applications today are distributed in some fashion, monitoring their health and performance requires a new approach. Enter distributed tracing, a method of profiling and monitoring distributed applications—particularly those that use microservice architectures. There’s just one problem: distributed tracing can be hard. But it doesn’t have to be.

With this guide, you’ll learn what distributed tracing is and how to use it to understand the performance and operation of your software. Key players at LightStep and other organizations walk you through instrumenting your code for tracing, collecting the data that your instrumentation produces, and turning it into useful operational insights. If you want to implement distributed tracing, this book tells you what you need to know.

You’ll learn:

• The pieces of a distributed tracing deployment: instrumentation, data collection, and analysis
• Best practices for instrumentation: methods for generating trace data from your services
• How to deal with (or avoid) overhead using sampling and other techniques
• How to use distributed tracing to improve baseline performance and to mitigate regressions quickly
• Where distributed tracing is headed in the future

Distributed tracing is a method to understand distributed software. That’s a lot like saying that water is wet, though—not terribly helpful, and reductive to a fault. Indeed, the best way to understand distributed tracing is to see it in practice, which is where this book comes in!

In the coming chapters, we’ll cover the three major things you need to know to get started with implementing distributed tracing for your applications and discuss strategies that you can apply to solve the problems caused by distributed architectures. You’ll learn about the different ways to instrument your software for distributed tracing and the styles of tracing and monitoring available to you. We’ll discuss how to collect all of the data that your instrumentation produces and the various performance considerations and costs around the collection and storage of trace data. After that, we’ll cover how to generate value from your trace data and turn it into useful, operational insights.

Finally, we’ll talk about the future of distributed tracing. By the end of this book, you should understand the exciting world of distributed tracing and know where, how, and when to implement it for your software.

Ultimately, the goal of Distributed Tracing in Practice is to allow you to build, operate, and understand your software more easily. We hope that the lessons in this text will help you in building the next generation of monitoring and observability practice at your organization.

About the Author

Austin Parker is an Open Source Software Engineer at LightStep, where he works as a core contributor and maintainer to the OpenTracing project. Prior to LightStep, he was a Software Architect at Apprenda building enterprise platforms using Kubernetes.

Daniel Spoonhower is a co-founder at LightStep, where he’s building performance management tools for modern software systems. Previously, Spoons spent almost six years at Google where he worked on developer tools as part of both Google’s internal infrastructure and Cloud Platform teams. He has published papers on the performance of parallel programs, garbage collection, and real-time programming. He has a PhD in programming languages from Carnegie Mellon University but still hasn’t found one he loves.

Jonathan Mace is a tenure track faculty member at the Max Planck Institute for Software Engineering. His areas of research include cloud systems, distributed systems, networking, and operating systems.

Ben Sigelman is the CEO and co-founder of LightStep, co-creator of Dapper (Google’s distributed tracing tool that helps developers make sense of their large-scale distributed systems), and co-creator of the open-source OpenTracing API standard (a project within the CNCF).

Rebecca Isaacs is a software engineer currently focused on the performance tuning and debugging of large-scale data center services. She was previously a research scientist, most recently at Google. She first started thinking about tracing for distributed systems over 15 years ago while at Microsoft Research, which she joined after obtaining a PhD from Cambridge University and a BSc from the University of Glasgow.