Software Telemetry

Software Telemetry shows you how to efficiently collect, store, and analyze system and application log data so you can monitor and improve your systems.

Summary

In Software Telemetry you will learn how to:

• Manage toxic telemetry and confidential records
• Master multi-tenant techniques and transformation processes
• Update to improve the statistical validity of your metrics and dashboards
• Make software telemetry emissions easier to parse
• Build easily-auditable logging systems
• Prevent and handle accidental data leaks
• Maintain processes for legal compliance
• Justify increased spend on telemetry software

Software Telemetry teaches you best practices for operating and updating telemetry systems. These vital systems trace, log, and monitor infrastructure by observing and analyzing the events generated by the system. This practical guide is filled with techniques you can apply to any size of organization, with troubleshooting techniques for every eventuality, and methods to ensure your compliance with standards like GDPR.

About the technology

Take advantage of the data generated by your IT infrastructure! Telemetry systems provide feedback on what’s happening inside your data center and applications, so you can efficiently monitor, maintain, and audit them. This practical book guides you through instrumenting your systems, setting up centralized logging, doing distributed tracing, and other invaluable telemetry techniques.

About the book

Software Telemetry shows you how to efficiently collect, store, and analyze system and application log data so you can monitor and improve your systems. Manage the pillars of observability—logs, metrics, and traces—in an end-to-end telemetry system that integrates with your existing infrastructure. You’ll discover how software telemetry benefits both small startups and legacy enterprises. And at a time when data audits are increasingly common, you’ll appreciate the thorough coverage of legal compliance processes, so there’s no reason to panic when a discovery request arrives.

What’s inside

• Multi-tenant techniques and transformation processes
• Toxic telemetry and confidential records
• Updates to improve the statistical validity of your metrics and dashboards
• Revisions that make software telemetry emissions easier to parse

About the reader

For software developers and infrastructure engineers supporting and building telemetry systems.

About the author

Jamie Riedesel is a staff engineer at Dropbox with over twenty years of experience in IT.

Table of Contents

1 Introduction

PART 1 TELEMETRY SYSTEM ARCHITECTURE

2 The Emitting stage: Creating and submitting telemetry

3 The Shipping stage: Moving and storing telemetry

4 The Shipping stage: Unifying diverse telemetry formats

5 The Presentation stage: Displaying telemetry

6 Marking up and enriching telemetry

7 Handling multitenancy

PART 2 USE CASES REVISITED: APPLYING ARCHITECTURE CONCEPTS

8 Growing cloud-based startup

9 Nonsoftware business

10 Long-established business IT

PART 3 TECHNIQUES FOR HANDLING TELEMETRY

11 Optimizing for regular expressions at scale

12 Standardized logging and event formats

13 Using more nonfile emitting techniques

14 Managing cardinality in telemetry

15 Ensuring telemetry integrity

16 Redacting and reprocessing telemetry

17 Building policies for telemetry retention and aggregation

18 Surviving legal processes

Review

“The telemetry bible!”—Sander Zegveld, Developers.nl

“An in-depth guide to operating software telemetry systems.”—Sushant Bhadkamkar, Lyft

“A must-have tome of knowledge written by one of the leaders in software telemetry.”—Andrew Bovill, CACI International Inc

“Something for every level of distributed systems, from hardware to networking to operating systems to software.”—Lokesh Kumar, Urgently

About the Author

Jamie Riedesel is a staff engineer at Dropbox. She has over twenty years of experience in IT, working in government, education, legacy companies, and startups. She has specialized in DevOps for the past decade, running distributed systems in public clouds, getting over workplace trauma, and designing software telemetry architectures.