Lesson 3

Introduction and survey
“Lesson 0” How to fast.ai
- Where is Lesson 0 video?
- What does it to do with the book ‘meta learning’ and fastai course?
How to do a fastai lesson?
- Watch with note
- Run the notebook and experiment
- Reproduce the notes from the codes
- Repeat with a different dataset
How to not self-study?
- physical and virtual study group
- study with people on forum
- Learn with social interactions is better than self-study
Highest voted student work
- Many interesting projects to check out
Jeremy’s Pets breeds detector
- Jeremy’s Pets repository
- What you should do with this App example?
Paperspace: your DL workstation in cloud!
- Does Jeremy speak highly of it? and Why?
JupyterLab: real beginner friendly
- Why JupyterLab is so good for beginners to take advantage of?
Make a better pet detector
- After training, we should think about how to improve it
Comparison of all (image) models
- Did anyone compared most of the image models and shared the finding?
- Where to find the notebook for comparison?
- Which 3 criteria are used for comparison?
Try out new models
- How to select and try out models with high scores
- Where is the train.ipynb file?
- How to try models on TIMM?
- How to compare them by loss?
- Why this model is actually impressive?
- What can the name of a model tell us?
- Why Jeremy only train 3 epochs? 18:58
Get the categories of a model
- How to get labels or categories info from the model?
- The rest is we learnt from last lecture.
What’s in the model
- What two things are stored in the model?
What does model architecture look like?
Parameters of a model
- How to zoom in on a layer of a model?
- How to check out the parameters of a layer?
- What does a layer’s parameters look like?
The investigating questions
- What are the weights/numbers?
- How can they figure out something important?
- Where is the notebook on how neuralnet work
Create a general quadratic function
- How to create a general function to output any specific quadratic function by changing 3 parameters?
- How to generate result from a specific quadratic function by changing 1 parameter?
- Why do we create such a general (quadratic) function with multiple unknown parameters rather than directly writing a particular quadratic function with specific coefficients?
Fit a function by good hands and eyes
- What does fit a function mean? (search better parameters based on dataset)
- How to create a random dataset?
- How to fit a general quadratic function to the dataset by changing 3 parameters with jupyter widgets by hand?
- What is the limitation of this manual/visual approach?
- where is this notebook
Loss: fit a function better without good eyes
- Why do we need loss or loss function?
- What is mean squared error?
- How does loss help the hand/visual approach to be more accurate and robust?
Automate the search of parameters for better loss
- How do we know which way and by how much to update parameters in order to improve on loss?
- Can you find enough derivative material on Khanacademy?
- What exactly do you need to know about derivative for now according to Jeremy? 34:26
- What is the slope or gradient?
- Does pytorch do derivative or calc slope/gradient for us?
- How to create a function to output sme loss on a general quadratic function? 35:02
- What do you need to know about tensor related to derivatives for now according to Jeremy? 36:02
- How to create a rank 1 tensor (a list to store numbers) to store parameters of the quadratic function? 36:49
- How to ask pytorch to prepare the calculation of gradients for these parameters? 37:10
- How to actually calculate gradients for each parameter based on the loss achieved by this specific function (3 specific parameters) against the whole dataset? 37:38
- In other words, this time when we calculate loss we can easily get the gradient for each parameter as well.
- What does the gradient value mean for each parameter? 38:34
- How to update parameters into new values with the gradients produced by the loss? 39:18
- How to automate the process above to find better parameters to achieve better loss? 41:05
- Why this automation is called gradient descent?
- notebook
The mathematical functions
- Besides dataset, loss function, derivative, what is also very crucial in finding/calculating those parameters?
- Why we can’t simply use quadratic functions for it?
ReLu: Rectified linear function
- Real world powerful models demands complex parameters and also complex functions, how complex a function can we come up?
- Is it possible to come up an infinitely complex function by simply doing addition of extremely simple functions?
- What could such extremely simple function look like?
- What is rectified linear function? How simple it is? What is linear and which part is rectified?
- What does rectified linear function look like in plot?
- How to adjust the 2 parameters of the function by hand with widget?
- What the function could look like under different parameters? 44:46
Infinitely complex function
- How powerful can the addition of extremely simple functions be?
- How to create a double rectified linear function (double relu) and adjust 4 parameters by hand with widget?
- How much more flexible does this double relu function look compared to a single rectified linear function?
- Can you imagine how complex can a function be when millions of rectified linear functions are added?
2 circles to an owl
- a very concise summarization of sewing fundamental ideas together for deep learning
A chart of all image models compared
- Can it be done with brute force computation with simple code?
- Does Jeremy look for the model comparison chart for best models?
- What is the wrong way of using the comparison chart by students? 50:45
- How does Jeremy use the chart?
- how does Jeremy decides which models to try out step by step?
Do I have enough data?
- Did you already build a model and train on your own dataset?
- Is the result good enough for you?
- What is the mistake the DL industry often make on this issue? 52:55
- What is Jeremy’s suggestion?
- How and what could semi-supervised learning and data augmentation be helpful?
- What about labeled and unlabeled data?
Interpret gradients in unit?
- How much does the loss go down when parameter a increase by unit of 1? 55:24
Learning rate
- Why we don’t update parameter values in large steps?
- Why does Jeremy draw a quadratic function to refer to the model when zooming in very close into the complex function?
- What would happen when update parameters with large values? 57:19
- Does large drop on loss necessarily demand large value increase of parameter according to the quadratic nature?
- What is learning rate? Why we need it to be small? How to pick a good value of it? 58:07
- What would happen if your learning rate is too big?
- What would happen when too small?
break
Matrix multiplication
- When the model requires millions of rectified linear functions, how to calculate fast enough?
- What is actually needed from linear algebra to do DL 1:01:33
- How easy it is to do matrix multiplication? 1:01:51
- What are the dataset and parameters in the matrix multiplication?
- Does matrix multiplication do the rectified part for you?
- What GPU is good at? 1:03:49
Build a regression model in spreadsheet
- Intro to Titanic Competition on Kaggle 1:05:01
- What is the dataset 1:05:18
- What to do with the train.csv file?
- How to clean the dataset a little bit?
- How to transform the dataset for matrix multiplication? 1:07:17
- How to prepare parameters for matrix multiplication? 1:08:50
- What’s wrong with the much larger value of the column ‘Fare’ compared to other columns? 1:09:35
- What to do with the values of ‘Fare’ and similarly the values of ‘Age’?
- What is normalizing the data?
- Does fastai do all these normalizations for us? Will we learn how fastai does it in the future?
- Why to apply log to values of ‘Fare’? 1:10:59
- Why do we need values to be evenly distributed?
- How to do mmult on dataset and parameters in spreadsheet? 1:11:56
- How to use mmult instead of addition to add a constant?
- What does the result of our model look like? 1:13:41
- Does Jeremy simply use a linear regression for the model, not even a relu?
- Can we solve regression with gradient descent? How do we do it?
Build a neuralnet by adding two regression models
- What does it take to turn a regression model into a neuralnet?
- Why we don’t add up the results of two linear functions?
- Why we only add the results together after they are rectified?
- What does the model prediction look like?
- Now we need to update the parameters for two linear functions, not just one.
Matrix multiplication makes training faster
- How to make the training to do mmult rather than addition of linear multiplications in spreadsheet?
Watch out! it’s chapter 4
- Please do try out Titanic competition
- Why chapter 4 drove away most of people?
- Ways to work out the spreadsheet yourself
Create dummy variables of 3 classes
- Do we only need 2 columns/classes for a dummy variable with 3 classes?
Taste NLP
- What do Natural Language Processing models do?
- What project opportunities do non-En-speaker students have?
- What tasks can NLP do? 1:25:57
fastai NLP library vs Hugging Face library
- How do these two libraries differ?
- Why we use transformer library in this lecture?
Homework to prepare you for the next lesson