Alice’s Adventures in a differentiable wonderland

Neural networks surround us, in the form of large language models, speech transcription systems, molecular discovery algorithms, robotics, and much more. Stripped of anything else, neural networks are compositions of differentiable primitives, and studying them means learning how to program and how to interact with these models, a particular example of what is called differentiable programming.

This primer is an introduction to this fascinating field imagined for someone, like Alice, who has just ventured into this strange differentiable wonderland. I overview the basics of optimizing a function via automatic differentiation, and a selection of the most common designs for handling sequences, graphs, texts, and audios. The focus is on a intuitive, self-contained introduction to the most important design techniques, including convolutional, attentional, and recurrent blocks, hoping to bridge the gap between theory and code (PyTorch and JAX) and leaving the reader capable of understanding some of the most advanced models out there, such as large language models (LLMs) and multimodal architectures.

The book is supplemented by a companion website where I will publish additional chapters and coding exercises (https://www.sscardapane.it/alice-book). The book is self-published to keep the price as low as possible, feedback on possible imprecisions is welcomed and rewarded by a (much Italian) coffee!

Table of contents

I Compass and needle

Chapter 1: Foreword and introduction

Chapter 2: Mathematical preliminaries

Chapter 3: Datasets and losses

Chapter 4: Linear models

Chapter 5: Fully-connected layers

Chapter 6: Automatic differentiation

II A strange land

Chapter 7: Convolutional layers

Chapter 8: Convolutions beyond images

Chapter 9: Scaling up the models

III Down the rabbit-hole

Chapter 10: Transformer models

Chapter 11: Transformers in practice

Chapter 12: Graph layers

Chapter 13: Recurrent layers

Appendix A: Probability theory

Appendix B: Universal approximation in 1D

About the Author

Simone Scardapane is a researcher at Sapienza University of Rome, where he teaches neural networks and machine learning. In his free time, he (endlessly) talks about machine learning at the intersection of the no-profit, academic, and industrial worlds.