Delta Lake: Up & Running

With the surge in big data and AI, organizations can rapidly create data products. However, the effectiveness of their analytics and machine learning models depends on the data’s quality. Delta Lake’s open source format offers a robust lakehouse framework over platforms like Amazon S3, ADLS, and GCS.

This practical book shows data engineers, data scientists, and data analysts how to get Delta Lake and its features up and running. The ultimate goal of building data pipelines and applications is to gain insights from data. You’ll understand how your storage solution choice determines the robustness and performance of the data pipeline, from raw data to insights.

You’ll learn how to:

• Use modern data management and data engineering techniques
• Understand how ACID transactions bring reliability to data lakes at scale
• Run streaming and batch jobs against your data lake concurrently
• Execute update, delete, and merge commands against your data lake
• Use time travel to roll back and examine previous data versions
• Build a streaming data quality pipeline following the medallion architecture

Table of Contents

Chapter 1. The Evolution of Data Architectures

Chapter 2. Getting Started with Delta Lake

Chapter 3. Basic Operations on Delta Tables

Chapter 4. Table Deletes, Updates, and Merges

Chapter 5. Performance Tuning

Chapter 6. Using Time Travel

Chapter 7. Schema Handling

Chapter 8. Operations on Streaming Data

Chapter 9. Delta Sharing

Chapter 10. Building a Lakehouse on Delta Lake

The goal of this book is to provide data practitioners with practical instructions on how to set up Delta Lake and start using its unique features. This book is designed for an audience that fits any of the following profiles:

• Data practitioners with a Spark background
• Data practitioners unfamiliar with or new to Delta Lake needing an introduction to the technology, the problems it solves, its main features and terminology, as well as how to get started using it
• Data practitioners looking to learn about the features and benefits of modern lakehouse architectures

It is important to note that this book and the features discussed apply to the Delta Lake open source framework (Delta Lake OSS). Proprietary features and optimizations that some companies offer around Delta Lake are considered out of the scope of this book.

First, we discuss why Delta Lake is an important tool for building modern enterprise data platforms and data science and AI solutions, followed by instructions on how to set up Delta Lake with Spark. Each of the subsequent chapters will walk you through the fundamental functions and operations of Delta Lake using step-by-step instructions and real-world examples.

The code examples in the book range from snippets that can be used in a PySpark shell to those designed to be run with a complete end-to-end notebook. In this book, all code snippets will be in Python, SQL, and, where necessary, shell commands.

A GitHub repository is provided to aid readers in following along throughout the book. Datasets, files, and code samples are provided in the repo and referred to throughout the book.

About the Author

Bennie is a principal architect with Insight Digital Innovation-a Microsoft and Databricks partner. As Principal architect with Insight, Bennie’s primary focus areas are Modern Data Warehousing, Machine learning, AI, and IoT on various commercial cloud platforms. Bennie has overseen many Data + AI projects in different application domains such as health care, the public sector, oil & gas, and financial applications. Bennie has architected and delivered real time streaming Data Lakehouse applications with Databricks, Spark Structured Streaming, Delta Lake, and Microsoft Power BI for various application domains. Bennie brings a wealth of practical experience in implementing secure, enterprise-scale Data Lakehouse-based solutions to support business intelligence, data science and machine learning. Bennie has also been a frequent speaker at Databricks events at Microsoft Technology Centers around the country, and was a speaker at the Data+AI 2021 summit.

Dan Davis is a Cloud Data Architect with a decade of experience delivering analytic insights and business value from data. Using modern tools and technologies, Dan specializes in designing and delivering data platforms, frameworks, and process’ to support data integration and analytics for on-premises, hybrid, and cloud environments on an enterprise scale.