Deep learning is probably one of the most exciting technologies of this decade. Machine and deep learning frameworks such as PyTorch make it surprisingly straightforward and accessible to train a deep learning (probabilistic) model from data, whether to test a theory, experiment with a new idea, or simply implement a published paper and try to reproduce its results (most of PyTorch is written in C++ and and tensor operators do not require holding the global interpreter lock while executing).
A typical piece of code for training a deep neural network is very simple and is listed below. First, we load the data as tensors. We then define the model architecture (i.e layers that the input will cascade through), the loss function (that will drive the optimization), and an optimizer (that will update the parameters after a backward pass). Finally, we execute the training loop. At each step, a batch of data is loaded stochastically and passed through the model during the forward pass. The batch is used to approximate the gradient of the loss, which is then computed during the backward pass (reverse automatic differentiation). The optimizer subsequently traverses the model parameters and updates them according to gradient descent, or one of its many variants.
dataset = ... model = ... optimizer = ... for i in range(n_epochs): # Forward pass y = model(...) error = loss(...) # Zero the accumulated gradients optimizer.zero_grad() # Backward pass error.backward() # Update the parameters optimizer.step()
Basically, almost every programmer can write and train a deep neural network today, from students finishing their bachelor’s or master’s degrees to software engineers who have only recently developed an interest in deep learning.
PyTorch, however, hides a tremendous amount of complexity behind this seemingly simple interface, including automatic differentiation, gradient propagation, tensor allocation, and the execution of forward and backward operations. The goal of this book is to bridge the gap between conceptual understanding and practical implementation by revealing the machinery hidden behind modern deep learning frameworks.
The idea is to explore deep learning from the algorithmic level all the way down to the hardware and microarchitecture. Many topics are already covered, while others are yet to come, including how PyTorch does the auto differentiation, latent variable modeling, generative models like LLMs, reinforcement learning, MoE, distillation, hardware multi threading, vector hardware, GPU kernels/CUDA, inference engines, and, of course, security.
This is the beginning of a long journey. I am committed to releasing a new version of this book every month.
Let’s make deep learning fun!
If you spot an error or have a suggestion, please feel free to reach out to me at kondah.mouad@gmail.com