Hadoop: The Definitive Guide

Get ready to unlock the power of your data. With the fourth edition of this comprehensive guide, you’ll learn how to build and maintain reliable, scalable, distributed systems with Apache Hadoop. This book is ideal for programmers looking to analyze datasets of any size, and for administrators who want to set up and run Hadoop clusters.

Using Hadoop 2 exclusively, author Tom White presents new chapters on YARN and several Hadoop-related projects such as Parquet, Flume, Crunch, and Spark. You’ll learn about recent changes to Hadoop and explore new case studies on Hadoop’s role in healthcare systems and genomics data processing.

• Learn fundamental components such as MapReduce, HDFS, and YARN
• Explore MapReduce in depth, including steps for developing applications with it
• Set up and maintain a Hadoop cluster running HDFS and MapReduce on YARN
• Learn two data formats: Avro for data serialization and Parquet for nested data
• Use data ingestion tools such as Flume (for streaming data) and Sqoop (for bulk data transfer)
• Understand how high-level data processing tools like Pig, Hive, Crunch, and Spark work with Hadoop
• Learn the HBase distributed database and the Zookeeper distributed configuration service

Table of Contents

Part I. Hadoop Fundamentals

Chapter 1. Meet Hadoop

Chapter 2. MapReduce

Chapter 3. The Hadoop Distributed Filesystem

Chapter 4. YARN

Chapter 5. Hadoop 1/0

Part II. MapReduce

Chapter 6. Developing a MapReduce Application

Chapter 7. How MapReduce Works

Chapter 8. MapReduce Types and Formats

Chapter 9. MapReduce Features

Part Ill. Hadoop Operations

Chapter 10. Setting Up a Hadoop Cluster

Chapter 11. Administering Hadoop

Part IV. Related Projects

Chapter 12. Avro

Chapter 13. Parquet

Chapter 14. Flume

Chapter 15. Sqoop

Chapter 16. Pig

Chapter 17. Hive

Chapter 18. Crunch

Chapter 19. Spark

Chapter 20. H Base

Chapter 21 . ZooKeeper

Part V. Case Studies

Chapter 22. Composable Data at Cerner

Chapter 23. Biological Data Science: Saving Lives with Software

Chapter 24. Cascading

Appendix A. Installing Apache Hadoop

Appendix B. Cloudera’s Distribution Including Apache Hadoop

Appendix C. Preparing the NCDC Weather Data

Appendix D. The Old and New Java MapReduce APIs

About the Author

Tom White is one of the foremost experts on Hadoop. He has been an Apache Hadoop committer since February 2007, and is a member of the Apache Software Foundation. Tom is a software engineer at Cloudera, where he has worked since its foundation on the core distributions from Apache and Cloudera. Previously he was an independent Hadoop consultant, working with companies to set up, use, and extend Hadoop. He has spoken at many conferences, including ApacheCon, OSCON, and Strata. Tom has a BA in mathematics from the University of Cambridge and an MA in philosophy of science from the University of Leeds, UK. He currently lives in Wales with his family.