Machine Learning

Machine Learning: From the Classics to Deep Networks, Transformers and Diffusion Models, Third Edition starts with the basics, including least squares regression and maximum likelihood methods, Bayesian decision theory, logistic regression, and decision trees. It then progresses to more recent techniques, covering sparse modelling methods, learning in reproducing kernel Hilbert spaces and support vector machines. Bayesian learning is treated in detail with emphasis on the EM algorithm and its approximate variational versions with a focus on mixture modelling, regression and classification. Nonparametric Bayesian learning, including Gaussian, Chinese restaurant, and Indian buffet processes are also presented. Monte Carlo methods, particle filtering, probabilistic graphical models with emphasis on Bayesian networks and hidden Markov models are treated in detail. Dimensionality reduction and latent variables modelling are considered in depth. Neural networks and deep learning are thoroughly presented, starting from the perceptron rule and multilayer perceptrons and moving on to convolutional and recurrent neural networks, adversarial learning, capsule networks, deep belief networks, GANs, and VAEs. The book also covers the fundamentals on statistical parameter estimation and optimization algorithms.

Focusing on the physical reasoning behind the mathematics, without sacrificing rigor, all methods and techniques are explained in depth, supported by examples and problems, providing an invaluable resource to the student and researcher for understanding and applying machine learning concepts.

• Provides a number of case studies and applications on a variety of topics, such as target localization, channel equalization, image denoising, audio characterization, text authorship identification, visual tracking, change point detection, hyperspectral image unmixing, fMRI data analysis, machine translation, and text-to-image generation
• Most chapters include a number of computer exercises in both MatLab and Python, and the chapters dedicated to deep learning include exercises in PyTorch

New to this edition

• The new material includes an extended coverage of attention transformers, large language models, self-supervised learning and diffusion models

Table of Contents

1 Introduction

2 Probability and stochastic processes

3 Learning in parametric modeling: basic concepts and directions

4 Mean-square error linear estimation

5 Online learning: the stochastic gradient descent family of algorithms

6 The least-squares family

7 Classification: a tour of the classics

8 Parameter learning: a convex analytic path

9 Sparsity-aware learning: concepts and theoretical foundations

10 Sparsity-aware learning: algorithms and applications

11 Learning in reproducing kernel Hilbert spaces

12 Bayesian learning: inference and the EM algorithm

13 Bayesian learning: approximate inference and nonparametric model:

14 Monte Carlo methods

15 Probabilistic graphical models: part I

16 Probabilistic graphical models: part II

17 Particle filtering

18 Neural networks and deep learning: part I

19 Neural networks and deep learning: part II

20 Dimensionality reduction and latent variable modeling

About the Author

Sergios Theodoridis is professor emeritus of machine learning and data processing with the National and Kapodistrian University of Athens, Athens, Greece. He has also served as distinguished professor with the Aalborg University Denmark and as professor with the Chinese University of Hong Kong, Shenzhen, China. In 2023, he received an honorary doctorate degree (D.Sc) from the University of Edinburgh, U.K. He has also received a number of prestigious awards, including the 2014 IEEE Signal Processing Magazine Best Paper Award, the 2009 IEEE Computational Intelligence Society Transactions on Neural Networks Outstanding Paper Award, the 2017 European Association for Signal Processing (EURASIP) Athanasios Papoulis Award, the 2014 IEEE Signal Processing Society Carl Friedrich Gauss Education Award, and the 2014 EURASIP Meritorious Service Award. He has served as president of EURASIP and vice president for the IEEE Signal Processing Society. He is a Fellow of EURASIP and a Life Fellow of IEEE. He is the coauthor of the book Pattern Recognition, 4th edition, Academic Press, 2009 and of the book Introduction to Pattern Recognition: A MATLAB Approach, Academic Press, 2010.