Foundations of Machine Learning

A new edition of a graduate-level machine learning textbook that focuses on the analysis and theory of algorithms.

This book is a general introduction to machine learning that can serve as a textbook for graduate students and a reference for researchers. It covers fundamental modern topics in machine learning while providing the theoretical basis and conceptual tools needed for the discussion and justification of algorithms. It also describes several key aspects of the application of these algorithms. The authors aim to present novel theoretical tools and concepts while giving concise proofs even for relatively advanced topics.

Foundations of Machine Learning is unique in its focus on the analysis and theory of algorithms. The first four chapters lay the theoretical foundation for what follows; subsequent chapters are mostly self-contained. Topics covered include the Probably Approximately Correct (PAC) learning framework; generalization bounds based on Rademacher complexity and VC-dimension; Support Vector Machines (SVMs); kernel methods; boosting; on-line learning; multi-class classification; ranking; regression; algorithmic stability; dimensionality reduction; learning automata and languages; and reinforcement learning. Each chapter ends with a set of exercises. Appendixes provide additional material including concise probability review.

This second edition offers three new chapters, on model selection, maximum entropy models, and conditional entropy models. New material in the appendixes includes a major section on Fenchel duality, expanded coverage of concentration inequalities, and an entirely new entry on information theory. More than half of the exercises are new to this edition.

Table of Contents

1. Introduction

2. The PAC Learning Framework

3. Rademacher Complexity and VC-Dimension

4. Model Selection

5. Support Vector Machines

6. Kernel Methods

7. Boosting

8. On-Line Learning

9. Multi-Class Classification

10. Ranking

11. Regression

12. Maximum Entropy Models

13. Conditional Maximum Entropy Models

14. Algorithmic Stability

15. Dimensionality Reduction

16. Learning Automata and Languages

17. Reinforcement Learning

About the Authors

Mehryar Mohri is Professor of Computer Science at New York University’s Courant Institute of Mathematical Sciences and a Research Consultant at Google Research.

Afshin Rostamizadeh is a Research Scientist at Google Research.

Ameet Talwalkar is Assistant Professor in the Machine Learning Department at Carnegie Mellon University.