Overview
When, how, and why do machine learning algorithms
work? This course answers these questions by studying the theoretical
aspects of machine learning, with a focus on statistically and
computationally efficient learning. Broad topics will include:
PAClearning, uniform convergence, and model selection; supervised
learning algorithms including SVM, boosting, kernel methods; online
learning algorithms and analysis; unsupervised learning with
guarantees.Special topics may include: Bandits, active learning,
semisupervised learning and others.
Requirements: Coursework will include
 5 homework assignments involving proofs and algorithm design, 50% of course grade.
 A midterm exam, 20% of course grade.
 A researchbased project, 30% of course grade. Project guidelines are available here.
Grading will be based on performance on the coursework where above 90% earns an A, above 80% earns a B, above 70% earns a C, and so on.
Prerequisites: CS 689 (Machine Learning) or CS 589 with instructor approval. No programming experience is required for the class but strong mathematical ability will be necessary.
Readings There is no required textbook for this course. However you may find the following useful.
Homeworks
 Homework 1. Released 9/5, due 9/19. (Solutions)
 Homework 2. Released 9/19, due 10/3. (Solutions)
 Homework 3. Released 10/3, due 10/17. (Solutions)
 Homework 4. Released 10/17, due 11/2. (Solutions)
 Homework 5. Released 11/2, due 11/16. (Solutions)
Feel free to you this latex template and style file.
Projects
Project guidelines are available here. Important dates are:
 Project Proposals. Due 10/5 by email.
 Project Presentations. On 12/12 in class.
 Project Writeup. Due 12/19 by email.
Lecture Schedule
Date 
Lecture Topics 
Readings 
Assignments 
9/5 
Probabilistic Prediction, PAClearning 


9/7 
Statistics background 


9/12 
Agnostic learning, BiasComplexity tradeoff 


9/14 
VC theorem 


9/19 
Rademacher complexity 


9/21 
Covering numbers, Chaining 


9/26 
Nonparametric classification/regression 


9/28 
Model selection, SRM 


10/3 
Boosting 


10/5 
Margin Bounds 


10/10 
NO CLASS  Columbus Day 
10/12 
Perceptron, SVM, Kernel SVM 


10/17 
Surrogate losses, calibration 


10/19 
Gradient Descent, convex optimization 


10/24 
MIDTERM  in class 
10/26 
Online learning: Halving, Hedge 


10/31 
Online Learning: Hedge, FTRL, OGD 


11/2 
Online Mirror Descent and FTPL 


11/7 
Adversarial Bandits 


11/9 
Stochastic Bandits 


11/14 
Unsupervised learning  Kmeans clustering 


11/16 
Unsupervised learning  GMMs, EM 


11/21 
NO CLASS  Thanksgiving 
11/23 
NO CLASS  Thanksgiving 
11/28 
Unsupervised learning  Spectral methods 


11/30 
Minimax theory 


12/5 
NO CLASS  NIPS 
12/7 
NO CLASS  NIPS 
12/12 
Project presentations 


