Learning Apache Drill

Get up to speed with Apache Drill, an extensible distributed SQL query engine that reads massive datasets in many popular file formats such as Parquet, JSON, and CSV. Drill reads data in HDFS or in cloud-native storage such as S3 and works with Hive metastores along with distributed databases such as HBase, MongoDB, and relational databases. Drill works everywhere: on your laptop or in your largest cluster.

In this practical book, Drill committers Charles Givre and Paul Rogers show analysts and data scientists how to query and analyze raw data using this powerful tool. Data scientists today spend about 80% of their time just gathering and cleaning data. With this book, you’ll learn how Drill helps you analyze data more effectively to drive down time to insight.

• Use Drill to clean, prepare, and summarize delimited data for further analysis
• Query file types including logfiles, Parquet, JSON, and other complex formats
• Query Hadoop, relational databases, MongoDB, and Kafka with standard SQL
• Connect to Drill programmatically using a variety of languages
• Use Drill even with challenging or ambiguous file formats
• Perform sophisticated analysis by extending Drill’s functionality with user-defined functions
• Facilitate data analysis for network security, image metadata, and machine learning

Table of Contents

Chapter 1. Introduction to Apache Drill

Chapter 2. Installing and Running Drill

Chapter 3. Overview of Apache Drill

Chapter 4. Querying Delimited Data

Chapter 5. Analyzing Complex and Nested Data

Chapter 6. Connecting Drill to Data Sources

Chapter 7. Connecting to Drill

Chapter 8. Data Engineering with Drill

Chapter 9. Deploying Drill in Production

Chapter 10. Setting Up Your Development Environment

Chapter 11. Writing Drill User-Defined Functions

Chapter 12. Writing a Format Plug-in

Chapter 13. Unique Uses of Drill

Appendix A. List of Drill Functions

Appendix B. Drill Formatting Strings

About the Authors

Paul Rogers is an Apache Drill committer at MapR where he focuses on Drill’s execution engine. Paul has worked as a software architect at a number database and BI companies such as Oracle, Actuate and Informix. Paul was the early architect of the Eclipse BIRT project. His interests include making Drill even easier to use for end-users and plug-in developers.

Charles Givre is an Apache Drill committer and has worked as a Senior Lead Data Scientist for Booz Allen Hamilton for the last six years where he works in the intersection of cyber security and data science. Mr. Givre is passionate about teaching others data science and analytic skills and has taught data science classes all over the world at conferences, universities and for clients. Most recently, Mr. Givre taught a data science class at the BlackHat conference in Las Vegas and the Center for Research in Applied Cryptography and Cyber Security at Bar Ilan University. He is a sought-after speaker and has delivered presentations at major industry conferences such as Strata-Hadoop World, BlackHat, Open Data Science Conference and others.