Deep Learning Course (NYU, Spring 2020) - Week 2 – Lecture: Stochastic gradient descent and backpropagation

Course website: http://bit.ly/DLSP20-web Playlist: http://bit.ly/pDL-YouTube Speaker: Yann LeCun Week 2: http://bit.ly/DLSP20-02 0:00:00 – Week 2 – Lecture LECTURE Part A: http://bit.ly/DLSP20-02-1 We start by understanding what parameterized models are and then discuss what a loss function is. We then look at gradient-based methods and how it's used in the backpropagation algorithm in a traditional neural network. We conclude this section by learning how to implement a neural network in PyTorch followed by a discussion on a more generalized form of backpropagation. 0:00:29 – Gradient Descent Optimization Algorithm 0:17:16 – Advantages of SGD, Backpropagation for Traditional Neural Net 0:38:08 – PyTorch implementation of Neural Network and a Generalized Backprop Algorithm LECTURE Part B: http://bit.ly/DLSP20-02-2 We begin with a concrete example of backpropagation and discuss the dimensions of Jacobian matrices. We then look at various basic neural net modules and compute their gradients, followed by a brief discussion on softmax and logsoftmax. The other topic of discussion in this part is Practical Tricks for Backpropagation. 0:49:49 – Basic Modules - LogSoftMax 1:05:53 – Practical Tricks for Backpropagation 1:21:31 – Computing gradients for NN modules and Practical tricks for Back Propagation