13: Backpropagation & MLP
In this lesson, we dive into backpropagation and the creation of a simple Multi-Layer Perceptron (MLP) neural network. We start by reviewing basic neural networks and their architecture, then move on to implementing a simple MLP from scratch. We focus on understanding the chain rule and backpropagation in the context of neural networks, and demonstrate how to calculate derivatives using Python and the SimPy library.
We also discuss the importance of the chain rule in calculating the gradient of the mean squared error (MSE) applied to a model, and demonstrate how to use PyTorch to calculate derivatives and simplify the process by creating classes for ReLU and linear functions. We then explore the issues with floating point math and introduce the log sum exp trick to overcome these issues. Finally, we create a training loop for a simple neural network.
Concepts discussed
- Basic neural network architecture
- Multi-Layer Perceptron (MLP) implementation
- Gradients and derivatives
- Chain rule and backpropagation
- Python debugger (pdb)
- PyTorch for calculating derivatives
- ReLU and linear function classes
- Log sum exp trick
log_softmax()function and cross entropy loss- Training loop for a simple neural network