Deep Learning Course (NYU, Spring 2020) - Week 11 – Lecture: PyTorch activation and loss functions

Course website: http://bit.ly/DLSP20-web Playlist: http://bit.ly/pDL-YouTube Speaker: Yann LeCun Week 11: http://bit.ly/DLSP20-11 0:00:00 – Week 11 – Lecture LECTURE Part A: http://bit.ly/DLSP20-11-1 In this section, we discussed the common activation functions in Pytorch. In particular, we compared activations with kink(s) versus smooth activations - the former is preferred in a deep neural network as the latter might suffer with gradient vanishing problem. We then learned about the common loss functions in Pytorch. 0:00:15 – Activation Functions 0:14:21 – Q&A of activation 0:33:10 – Loss Functions (until AdaptiveLogSoftMax) LECTURE Part B: http://bit.ly/DLSP20-11-2 In this section, we continued to learn about loss functions - in particular, margin-based losses and their applications. We then discussed how to design a good loss function for EBMs as well as examples of well-known EBM loss functions. We gave particular attention to margin-based loss function here, as well as explaining the idea of “most offending incorrect answer. 0:53:27 – Loss Functions (until CosineEmbeddingLoss) 1:08:23 – Loss Functions and Loss Functions for Energy Based Models 1:23:18 – Loss Functions for Energy Based Models