Python Data Cleaning and Preparation Best Practices

Take your data preparation skills to the next level by converting any type of data asset into a structured, formatted, and readily usable dataset

Key Features

• Maximize the value of your data through effective data cleaning methods
• Enhance your data skills using strategies for handling structured and unstructured data
• Elevate the quality of your data products by testing and validating your data pipelines

Book Description

Professionals face several challenges in effectively leveraging data in today’s data-driven world. One of the main challenges is the low quality of data products, often caused by inaccurate, incomplete, or inconsistent data. Another significant challenge is the lack of skills among data professionals to analyze unstructured data, leading to valuable insights being missed that are difficult or impossible to obtain from structured data alone.

To help you tackle these challenges, this book will take you on a journey through the upstream data pipeline, which includes the ingestion of data from various sources, the validation and profiling of data for high-quality end tables, and writing data to different sinks. You’ll focus on structured data by performing essential tasks, such as cleaning and encoding datasets and handling missing values and outliers, before learning how to manipulate unstructured data with simple techniques. You’ll also be introduced to a variety of natural language processing techniques, from tokenization to vector models, as well as techniques to structure images, videos, and audio.

By the end of this book, you’ll be proficient in data cleaning and preparation techniques for both structured and unstructured data.

What you will learn

• Ingest data from different sources and write it to the required sinks
• Profile and validate data pipelines for better quality control
• Get up to speed with grouping, merging, and joining structured data
• Handle missing values and outliers in structured datasets
• Implement techniques to manipulate and transform time series data
• Apply structure to text, image, voice, and other unstructured data

Who this book is for

Whether you’re a data analyst, data engineer, data scientist, or a data professional responsible for data preparation and cleaning, this book is for you. Working knowledge of Python programming is needed to get the most out of this book.

Table of Contents

1. Data Ingestion Techniques
2. Importance of Data Quality
3. Data Profiling – Understanding Data Structure, Quality, and Distribution
4. Cleaning Messy Data and Data Manipulation
5. Data Transformation – Merging and Concatenating
6. Data Grouping, Aggregation, Filtering, and Applying Functions
7. Data Sinks
8. Detecting and Handling Missing Values and Outliers
9. Normalization and Standardization
10. Handling Categorical Features
11. Consuming Time Series Data
12. Text Preprocessing in the Era of LLMs
13. Image and Audio Preprocessing with LLMs

About the Author

Maria Zervou is a Generative AI and machine learning expert, dedicated to making advanced technologies accessible. With over a decade of experience, she has led impactful AI projects across industries and mentored teams on cutting-edge advancements. As a machine learning specialist at Databricks, Maria drives innovative AI solutions and industry adoption. Beyond her role, she democratizes knowledge through her YouTube channel, featuring experts on AI topics. A recognized thought leader and finalist in the Women in Tech Excellence Awards, Maria advocates for responsible AI use and contributes to open source projects, fostering collaboration and empowering future AI leaders.