Deep Learning

NPTEL and Indian Institute of Technology, Ropar via YouTube

Overview

COURSE OUTLINE: Deep Learning has received a lot of attention over the past few years and has been employed successfully by companies like Google, Microsoft, IBM, Facebook, Twitter etc. to solve a wide range of problems in Computer Vision and Natural Language Processing. In this course, we will learn about the building blocks used in these Deep Learning based solutions. Specifically, we will learn about feedforward neural networks, convolutional neural networks, recurrent neural networks and attention mechanisms. We will also look at various optimization algorithms such as Gradient Descent, Nesterov Accelerated Gradient Descent, Adam, AdaGrad and RMSProp which are used for training such deep neural networks. At the end of this course students would have knowledge of deep architectures used for solving various Vision and NLP tasks

Syllabus

Deep Learning - Course Introduction.
Deep Learning(CS7015): Lec 1.1 Biological Neuron.
Deep Learning(CS7015): Lec 1.2 From Spring to Winter of AI.
Deep Learning(CS7015): Lec 1.3 The Deep Revival.
Deep Learning(CS7015): Lec 1.4 From Cats to Convolutional Neural Networks.
Deep Learning(CS7015): Lec 1.5 Faster, higher, stronger.
Deep Learning(CS7015): Lec 1.6 The Curious Case of Sequences.
Deep Learning(CS7015): Lec 1.7 Beating humans at their own games (literally).
Deep Learning(CS7015): Lec 1.8 The Madness (2013-).
Deep Learning(CS7015): Lec 1.9 (Need for) Sanity.
Deep Learning(CS7015): Lec 2.1 Motivation from Biological Neurons.
Deep Learning(CS7015): Lec 2.2 McCulloch Pitts Neuron, Thresholding Logic.
Deep Learning(CS7015): Lec 2.3 Perceptrons.
Deep Learning(CS7015): Lec 2.4 Error and Error Surfaces.
Deep Learning(CS7015): Lec 2.5 Perceptron Learning Algorithm.
Deep Learning(CS7015): Lec 2.6 Proof of Convergence of Perceptron Learning Algorithm.
Deep Learning(CS7015): Lec 2.7 Linearly Separable Boolean Functions.
Deep Learning(CS7015): Lec 2.8 Representation Power of a Network of Perceptrons.
Deep Learning(CS7015): Lec 3.1 Sigmoid Neuron.
Deep Learning(CS7015): Lec 3.2 A typical Supervised Machine Learning Setup.
Deep Learning(CS7015): Lec 3.3 Learning Parameters: (Infeasible) guess work.
Deep Learning(CS7015): Lec 3.4 Learning Parameters: Gradient Descent.
Deep Learning(CS7015): Lec 3.5 Representation Power of Multilayer Network of Sigmoid Neurons.
Deep Learning(CS7015): Lec 4.1 Feedforward Neural Networks (a.k.a multilayered network of neurons).
Deep Learning(CS7015): Lec 4.2 Learning Paramters of Feedforward Neural Networks (Intuition).
Deep Learning(CS7015): Lec 4.3 Output functions and Loss functions.
Deep Learning(CS7015): Lec 4.4 Backpropagation (Intuition).
Deep Learning(CS7015): Lec 4.5 Backpropagation: Computing Gradients w.r.t. the Output Units.
Deep Learning(CS7015): Lec 4.6 Backpropagation: Computing Gradients w.r.t. Hidden Units.
Deep Learning(CS7015): Lec 4.7 Backpropagation: Computing Gradients w.r.t. Parameters.
Deep Learning(CS7015): Lec 4.8 Backpropagation: Pseudo code.
Deep Learning(CS7015): Lec 4.9 Derivative of the activation function.
Deep Learning(CS7015): Lec 6.6 PCA : Interpretation 3.
Deep Learning(CS7015): Lec 4.10 Information content, Entropy & cross entropy.
Deep Learning(CS7015): Lec 5.1 & Lec 5.2 Recap: Learning Parameters: Guess Work, Gradient Descent.
Deep Learning(CS7015): Lec 5.3 Contours Maps.
Deep Learning(CS7015): Lec 5.4 Momentum based Gradient Descent.
Deep Learning(CS7015): Lec 5.5 Nesterov Accelerated Gradient Descent.
Deep Learning(CS7015): Lec 5.6 Stochastic And Mini-Batch Gradient Descent.
Deep Learning(CS7015): Lec 5.7 Tips for Adjusting Learning Rate and Momentum.
Deep Learning(CS7015): Lec 5.8 Line Search.
Deep Learning(CS7015): Lec 5.9 Gradient Descent with Adaptive Learning Rate.
Deep Learning(CS7015): Lec 5.9 (Part-2) Bias Correction in Adam.
Deep Learning(CS7015): Lec 6.1 Eigenvalues and Eigenvectors.
Deep Learning(CS7015): Lec 6.2 Linear Algebra : Basic Definitions.
Deep Learning(CS7015): Lec 6.3 Eigenvalue Decompositon.
Deep Learning(CS7015): Lec 6.4 Principal Component Analysis and its Interpretations.
Deep Learning(CS7015): Lec 6.5 PCA : Interpretation 2.
Deep Learning(CS7015): Lec 6.6 (Part-2) PCA : Interpretation 3 (Contd.).
Deep Learning(CS7015): Lec 6.7 PCA : Practical Example.
Deep Learning(CS7015): Lec 6.8 Singular Value Decomposition.
Deep Learning(CS7015): Lec 7.1 Introduction to Autoncoders.
Deep Learning(CS7015): Lec 7.2 Link between PCA and Autoencoders.
Deep Learning(CS7015): Lec 7.3 Regularization in autoencoders (Motivation).
Deep Learning(CS7015): Lec 7.4 Denoising Autoencoders.
Deep Learning(CS7015): Lec 7.5 Sparse Autoencoders.
Deep Learning(CS7015): Lec 7.6 Contractive Autoencoders.
Deep Learning(CS7015): Lec 8.1 Bias and Variance.
Deep Learning(CS7015): Lec 8.2 Train error vs Test error.
Deep Learning(CS7015): Lec 8.2 (Part-2) Train error vs Test error (Recap).
Deep Learning(CS7015): Lec 8.3 True error and Model complexity.
Deep Learning(CS7015): Lec 8.4 L2 regularization.
Deep Learning(CS7015): Lec 8.5 Dataset augmentation.
Deep Learning(CS7015): Lec 8.6 Parameter sharing and tying.
Deep Learning(CS7015): Lec 8.7 Adding Noise to the inputs.
Deep Learning(CS7015): Lec 8.8 Adding Noise to the outputs.
Deep Learning(CS7015): Lec 8.9 Early stopping.
Deep Learning(CS7015): Lec 8.10 Ensemble Methods.
Deep Learning(CS7015): Lec 8.11 Dropout.
Deep Learning(CS7015): Lec 9.1 A quick recap of training deep neural networks.
Deep Learning(CS7015): Lec 9.2 Unsupervised pre-training.
Deep Learning(CS7015): Lec 9.3 Better activation functions.
Deep Learning(CS7015): Lec 9.4 Better initialization strategies.
Deep Learning(CS7015): Lec 9.5 Batch Normalization.
Deep Learning(CS7015): Lec 10.1 One-hot representations of words.
Deep Learning(CS7015): Lec 10.2 Distributed Representations of words.
Deep Learning(CS7015): Lec 10.3 SVD for learning word representations.
Deep Learning(CS7015): Lec 10.3 (Part-2) SVD for learning word representations (Contd.).
Deep Learning(CS7015): Lec 10.4 Continuous bag of words model.
Deep Learning(CS7015): Lec 10.5 Skip-gram model.
Deep Learning(CS7015): Lec 10.5 (Part-2) Skip-gram model (Contd.).
Deep Learning(CS7015): Lec 10.6 Contrastive estimation.
Deep Learning(CS7015): Lec 10.7 Hierarchical softmax.
Deep Learning(CS7015): Lec 10.8 GloVe representations.
Deep Learning(CS7015): Lec 10.9 Evaluating word representations.
Deep Learning(CS7015): Lec 10.10 Relation between SVD and Word2Vec.
Deep Learning(CS7015): Lec 11.1 The convolution operation.
Deep Learning(CS7015): Lec 11.2 Relation between input size, output size and filter size.
Deep Learning(CS7015): Lec 11.3 Convolutional Neural Networks.
Deep Learning(CS7015): Lec 11.3 (Part-2) Convolutional Neural Networks (Contd.).
Deep Learning(CS7015): Lec 11.4 CNNs (success stories on ImageNet).
Deep Learning(CS7015): Lec 11.4 (Par-2) CNNs (success stories on ImageNet) (Contd.).
Deep Learning(CS7015): Lec 11.5 Image Classification continued (GoogLeNet and ResNet).
Deep Learning(CS7015): Lec 12.1 Visualizing patches which maximally activate a neuron.
Deep Learning(CS7015): Lec 12.2 Visualizing filters of a CNN.
Deep Learning(CS7015): Lec 12.3 Occlusion experiments.
Deep Learning(CS7015): Lec 12.4 Finding influence of input pixels using backpropagation.
Deep Learning(CS7015): Lec 12.5 Guided Backpropagation.
Deep Learning(CS7015): Lec 12.6 Optimization over images.
Deep Learning(CS7015): Lec 12.7 Create images from embeddings.
Deep Learning(CS7015): Lec 12.8 Deep Dream.
Deep Learning(CS7015): Lec 12.9 Deep Art.
Deep Learning(CS7015): Lec 12.10 Fooling Deep Convolutional Neural Networks.
Deep Learning(CS7015): Lec 13.1 Sequence Learning Problems.
Deep Learning(CS7015): Lec 13.2 Recurrent Neural Networks.
Deep Learning(CS7015): Lec 13.3 Backpropagation through time.
Deep Learning(CS7015): Lec 13.4 The problem of Exploding and Vanishing Gradients.
Deep Learning(CS7015): Lec 13.5 Some Gory Details.
Deep Learning(CS7015): Lec 14.1 Selective Read, Selective Write, Selective Forget.
Deep Learning(CS7015): Lec 14.2 Long Short Term Memory(LSTM) and Gated Recurrent Units(GRUs).
Deep Learning(CS7015): Lec 14.3 How LSTMs avoid the problem of vanishing gradients.
Deep Learning(CS7015): Lec 14.3 (Part-2) How LSTMs avoid the problem of vanishing gradients (Contd.).
Deep Learning(CS7015): Lec 15.1 Introduction to Encoder Decoder Models.
Deep Learning(CS7015): Lec 15.2 Applications of Encoder Decoder models.
Deep Learning(CS7015): Lec 15.3 Attention Mechanism.
Deep Learning(CS7015): Lec 15.3 (Part-2) Attention Mechanism (Contd.).
Deep Learning(CS7015): Lec 15.4 Attention over images.
Deep Learning(CS7015): Lec 15.5 Hierarchical Attention.