Delta Lake: The Definitive Guide

Ready to simplify the process of building data lakehouses and data pipelines at scale? In this practical guide, learn how Delta Lake is helping data engineers, data scientists, and data analysts overcome key data reliability challenges with modern data engineering and management techniques.

Authors Denny Lee, Tristen Wentling, Scott Haines, and Prashanth Babu (with contributions from Delta Lake maintainer R. Tyler Croy) share expert insights on all things Delta Lake–including how to run batch and streaming jobs concurrently and accelerate the usability of your data. You’ll also uncover how ACID transactions bring reliability to data lakehouses at scale.

This book helps you:

• Understand key data reliability challenges and how Delta Lake solves them
• Explain the critical role of Delta transaction logs as a single source of truth
• Learn the Delta Lake ecosystem with technologies like Apache Flink, Kafka, and Trino
• Architect data lakehouses with the medallion architecture
• Optimize Delta Lake performance with features like deletion vectors and liquid clustering

Table of Contents

Chapter 1. Introduction to the Delta Lake Lakehouse Format

Chapter 2. Installing Delta Lake

Chapter 3. Essential Delta Lake Operations

Chapter 4. Diving into the Delta Lake Ecosystem

Chapter 5. Maintaining Your Delta Lake

Chapter 6. Building Native Applications with Delta Lake

Chapter 7. Streaming In and Out of Your Delta Lake

Chapter 8. Advanced Features

Chapter 9. Architecting Your Lakehouse

Chapter 10. Performance Tuning: Optimizing Your Data Pipelines with Delta Lake

Chapter 11. Successful Design Patterns

Chapter 12. Foundations of Lakehouse Governance and Security

Chapter 13. Metadata Management, Data Flow, and Lineage

Chapter 14. Data Sharing with the Delta Sharing Protocol

About the Authors

Denny Lee is a Staff Developer Advocate at Databricks. He is a hands-on distributed systems and data sciences engineer with extensive experience developing internet-scale infrastructure, data platforms, and predictive analytics systems for both on-premise and cloud environments. He also has a Masters of Biomedical Informatics from Oregon Health and Sciences University and has architected and implemented powerful data solutions for enterprise Healthcare customers. His current technical focuses include Distributed Systems, Apache Spark, Deep Learning, Machine Learning, and Genomics.

Tristen Wentling works in machine learning, data engineering, and statistical analysis using Python, Apache Spark, and Scala. He is a machine learning advocate loves the flexibility of neural networks. Tristen holds an M.S. in Mathematics and B.S. in Applied Mathematics.

Scott Haines is a Databricks Beacon and has been working with data systems and distributed systems and architectures for over 15 years. He recently wrote a book encapsulating his journey called Modern Data Engineering with Apache Spark: A Hands-on guide for building mission-critical streaming applications. He enjoys teaching people how to simplify data systems and data-intensive services and takes to the snow in the winter to pursue his love of snowboarding.

Prashanth Babu is a Databricks Certified Developer who helps guide design and implementation of customer use cases by building out reference architectures, best practices, frameworks, MVP, and prototypes, which enables customers to succeed in turning their data into value.