CS229: Machine Learning - The Summer Edition!


Course Description   This is the summer edition of CS229 Machine Learning that was offered over 2019 and 2020. CS229 provides a broad introduction to statistical machine learning (at an intermediate / advanced level) and covers supervised learning (generative/discriminative learning, parametric/non-parametric learning, neural networks, support vector machines); unsupervised learning (clustering, dimensionality reduction, kernel methods); learning theory (bias/variance tradeoffs, practical ); and reinforcement learning among other topics. The structure of the summer offering enables coverage of additional topics, places stronger emphasis on the mathematical and visual intuitions, and goes deeper into the details of various topics.

Full playlist (YouTube)


Syllabus and Course Schedule


EventDateDescriptionMaterials and Assignments
Introduction and Pre-requisties review (3 lectures)
Lecture 1 [YouTube] 6/24
  • Introduction and Logistics
  • Review of Linear Algebra
Class Notes
  • Introduction [pptx]
  • Linear Algebra (section 1-3) [pdf]
Lecture 2 [YouTube] 6/26
  • Review of Matrix Calculus
  • Review of Probability
Class Notes
  • Linear Algebra (section 4) [pdf]
  • Probability Theory [pdf]
  • Probability Theory Slides [pdf]
Lecture 3 [YouTube] 6/28
  • Review of Probability and Statistics
  • Setting of Supervised Learning
Class Notes
  • Supervised Learning [pdf]
  • Probability Theory [pdf]
Supervised Learning (8 lectures)
Lecture 4 [YouTube] 7/1
  • Linear Regression
  • [Stochastic] Gradient Descent ([S]GD)
  • Normal Equations
  • Probabilistic Interpretation
  • Maximum Likelihood Estimation (MLE)
Class Notes
  • Supervised Learning (section 1-3) [pdf]
Lecture 5 [YouTube] 7/3
  • Perceptron
  • Logistic Regression
  • Newton's Method
Class Notes
  • Supervised Learning (section 5-7) [pdf]
Lecture 6 [YouTube] 7/5
  • Exponential Family
  • Generalized Linear Models (GLM)
Class Notes
  • Supervised Learning (section 8-9) [pdf]
Lecture 7 [YouTube] 7/8
  • Gaussian Discriminant Analysis (GDA)
  • Naive Bayes
  • Laplace Smoothing
Class Notes
  • Generative Algorithms [pdf]
Lecture 8 [YouTube] 7/10
  • Kernel Methods
  • Support Vector Machine
Class Notes
  • Kernel Methods and SVM [pdf]
Lecture 9 [YouTube] 7/12
  • Bayesian Methods
  • Parametric (Bayesian Linear Regression)
  • Non-parametric (Gaussian process)
Class Notes
  • Gaussian Processes [pdf]
Optional
  • The Multivariate Gaussian Distribution [pdf]
  • More on Gaussian Distribution [pdf]
Lecture 10 [YouTube] 7/15
  • Neural Networks and Deep Learning
Class Notes
  • Deep Learning (skip Sec 3.3) [pdf]
Optional
  • Backpropagation [pdf]
Lecture 11 [YouTube] 7/17
  • Deep Learning (contd)
Theory (2 lectures)
Lecture 12 [YouTube] 7/19
  • Bias and Variance
  • Regularization, Bayesian Interpretation
  • Model Selection
Class Notes
  • Regularization and Model Selection [pdf]
Lecture 13 [YouTube] 7/22
  • Bias-Variance tradeoff (wrap-up)
  • Empirical Risk Minimization
  • Uniform Convergence
Class Notes
  • Bias Variance Analysis [pdf]
  • Statistical Learning Theory [pdf]
Reinforcement Learning (2 lectures)
Lecture 14 [YouTube] 7/24
  • Reinforcement Learning (RL)
  • Markov Decision Processes (MDP)
  • Value and Policy Iterations
Class Notes
  • Reinforcement Learning and Control (Sec 1-2) [pdf]
Lecture 15 [YouTube] 7/26
  • RL (wrap-up)
  • Learning MDP model
  • Continuous States
Class Notes
  • Reinforcement Learning and Control (Sec 3-4) [pdf]
Unsupervised Learning (3 lectures)
Lecture 16 [YouTube] 7/29 Unsupervised Learning
  • K-means clustering
  • Mixture of Gaussians (GMM)
  • Expectation Maximization (EM)
Class Notes
  • K-means [pdf]
  • Mixture of Gaussians [pdf]
  • Expectation Maximization (Sec 1-2, skip 2.1) [pdf]
Lecture 17 [YouTube] 7/31
  • EM (wrap-up)
  • Factor Analysis
Class Notes
  • Expectation Maximization (Sec 3) [pdf]
  • Factor Analysis [pdf]
Lecture 18 [YouTube] 8/2
  • Factor Analysis (wrap-up)
  • Principal Components Analysis (PCA)
  • Independent Components Analysis (ICA)
Class Notes
  • Principal Components Analysis [pdf]
  • Independent Components Analysis [pdf]
Miscellaneous Topics (3 lectures)
Lecture 19 8/5
  • Maximum Entropy and Exponential Family
  • KL-Divergence
  • Calibration and Proper Scoring Rules
Class Notes
  • Maximum Entropy [pdf]
Lecture 20 8/7
  • Variational Inference
  • EM Variants
  • Variational Autoencoder
Class Notes
  • VAE (Sec 4) [pdf]
Lecture 21 8/9
  • Evaluation Metrics
Class Notes
  • Evaluation Metrics [pptx]
Recap and wrap-up (2 lectures)
Lecture 22 8/12
  • Practical advice and tips
  • Review for Finals
Class Notes
Lecture 23 8/14
  • Review for Finals
Class Notes
Final 8/16
Other Resources
  1. Advice on applying machine learning: Slides from Andrew's lecture on getting machine learning algorithms to work in practice can be found here.
  2. Previous projects: A list of last year's final projects can be found here.
  3. Data: Here is the UCI Machine learning repository, which contains a large collection of standard datasets for testing learning algorithms. If you want to see examples of recent work in machine learning, start by taking a look at the conferences NeurIPS (all old NeurIPS papers are online) and ICML. Some other related conferences include UAI, AAAI, IJCAI.
  4. Viewing PostScript and PDF files: Depending on the computer you are using, you may be able to download a PostScript viewer or PDF viewer for it if you don't already have one.
  5. Machine learning study guides tailored to CS 229 by Afshine Amidi and Shervine Amidi.