Architecting Data and Machine Learning Platforms

All cloud architects need to know how to build data platforms—the key to enabling businesses with data and delivering enterprise-wide intelligence in a fast and efficient way. This handbook is ideal for learning how to design, build, and modernize cloud native data and machine learning platforms using AWS, Azure, Google Cloud, or multicloud tools like Fivetran, dbt, Snowflake, and Databricks.

Authors Marco Tranquillin, Valliappa Lakshmanan, and Firat Tekiner cover the entire data lifecycle in a cloud environment, from ingestion to activation, using real-world enterprise architectures. You’ll learn how to transform and modernize familiar solutions, like data warehouses and data lakes, and you’ll be able to leverage recent AI/ML patterns to get accurate and quicker insights to drive competitive advantage.

This book shows you how to:

• Design a modern cloud native or hybrid data analytics and machine learning platform
• Accelerate data-led innovation by consolidating enterprise data in a data platform
• Democratize access to enterprise data and allow business teams to extract insights and build AI/ML capabilities
• Enable your business to make decisions in real time using streaming pipelines
• Move from a descriptive analytics approach to a more predictive and prescriptive one by building an MLOps platform
• Make your organization more effective in working with data analytics and machine learning in a cloud environment

Table of Contents

Chapter 1. Modernizing Your Data Platform: An Introductory Overview

Chapter 2. Strategic Steps to Innovate with Data

Chapter 3. Designing Your Data Team

Chapter 4. A Migration Framework

Chapter 5. Architecting a Data Lake

Chapter 6. Innovating with an Enterprise Data Warehouse

Chapter 7. Converging to a Lakehouse

Chapter 8. Architectures for Streaming

Chapter 9. Extending a Data Platform Using Hybrid and Edge

Chapter 10. Al Application Architecture

Chapter 11. Architecting an ML Platform

Chapter 12. Data Platform Modernization: A Model Case

Who Is This Book For?

This book is for architects who wish to support data-driven decision making in their business by creating a data and ML platform using public cloud technologies. Data engineers, data analysts, data scientists, and ML engineers will find the book useful to gain a conceptual design view of the systems that they might be implementing on top of.

Digitally native companies have been doing this already for several years.

As early as 2016, Twitter explained that their data platform team maintains “systems to support and manage the production and consumption of data for a variety of business purposes, including publicly reported metrics, recommendations, A/B testing, ads targeting, etc.” In 2016, this involved maintaining one of the largest Hadoop clusters in the world. By 2019, this was changing to include supporting the use of a cloud-native data warehousing solution.

Etsy, to take another example, says that their ML platform “supports ML experiments by developing and maintaining the technical infrastructure that Etsy’s ML practitioners rely on to prototype, train, and deploy ML models at scale.” Both Twitter and Etsy have built modern data and ML platforms. The platforms at the two companies are different, to support the different types of data, personnel, and business use cases that the platforms need to support, but the underlying approach is pretty similar.

In this book, we will show you how to architect a modern data and ML platform that enables engineers in your business to:

• Collect data from a variety of sources such as operational databases, customer clickstream, Internet of Things (IoT) devices, software as a service (SaaS) applications, etc.
• Break down silos between different parts of the organization
• Process data while ingesting it or after loading it while guaranteeing proper processes for data quality and governance
• Analyze the data routinely or ad hoc
• Enrich the data with prebuilt AI models
• Build ML models to carry out predictive analytics
• Act on the data routinely or in response to triggering events or thresholds
• Disseminate insights and embed analytics

This book is a good introduction to architectural considerations if you work with data and ML models in enterprises, because you will be required to do your work on the platform built by your data or ML platform team. Thus, if you are a data engineer, data analyst, data scientist, or ML engineer, you will find this book helpful for gaining a high-level systems design view.

About the Author

Marco Tranquillin is a seasoned consultant who helps organizations make technology transformations through cloud computing.

Valliappa Lakshmanan is a renowned executive who partners with C-suite and data science teams to build value from data and AI.

Firat Tekiner is an innovative product manager who develops and delivers data products and AI systems for the world’s largest organizations.