Machine Learning Theory

CMPSCI 690M, Fall 2017

Akshay Krishnamurthy

When: TuTh 2:30-3:45
Where: CS 140
Office Hours: TuTh 3:45-5 (CS 258)


When, how, and why do machine learning algorithms work? This course answers these questions by studying the theoretical aspects of machine learning, with a focus on statistically and computationally efficient learning. Broad topics will include: PAC-learning, uniform convergence, and model selection; supervised learning algorithms including SVM, boosting, kernel methods; online learning algorithms and analysis; unsupervised learning with guarantees.Special topics may include: Bandits, active learning, semi-supervised learning and others.

Requirements: Coursework will include

  • 5 homework assignments involving proofs and algorithm design, 50% of course grade.
  • A midterm exam, 20% of course grade.
  • A research-based project, 30% of course grade. Project guidelines are available here.
Grading will be based on performance on the coursework where above 90% earns an A, above 80% earns a B, above 70% earns a C, and so on.

Prerequisites: CS 689 (Machine Learning) or CS 589 with instructor approval. No programming experience is required for the class but strong mathematical ability will be necessary.


There is no required textbook for this course. However you may find the following useful.


  1. Homework 1. Released 9/5, due 9/19. (Solutions)
  2. Homework 2. Released 9/19, due 10/3. (Solutions)
  3. Homework 3. Released 10/3, due 10/17. (Solutions)
  4. Homework 4. Released 10/17, due 11/2. (Solutions)
  5. Homework 5. Released 11/2, due 11/16. (Solutions)
Feel free to you this latex template and style file.


Project guidelines are available here. Important dates are:
  1. Project Proposals. Due 10/5 by email.
  2. Project Presentations. On 12/12 in class.
  3. Project Writeup. Due 12/19 by email.

Lecture Schedule

Date Lecture Topics Readings Assignments
9/5 Probabilistic Prediction, PAC-learning
9/7 Statistics background
9/12 Agnostic learning, Bias-Complexity tradeoff
9/14 VC theorem
9/19 Rademacher complexity
9/21 Covering numbers, Chaining
9/26 Nonparametric classification/regression
9/28 Model selection, SRM
10/3 Boosting
10/5 Margin Bounds
  • Project Proposals due
10/10 NO CLASS -- Columbus Day
10/12 Perceptron, SVM, Kernel SVM
10/17 Surrogate losses, calibration
10/19 Gradient Descent, convex optimization
10/24 MIDTERM -- in class
10/26 Online learning: conversion, Halving, Hedge
10/31 Online convex opt: FTRL, OGD, OMD
11/2 FTPL
11/7 Adversarial Bandits
11/9 Stochastic Bandits
11/14 Unsupervised learning -- K-means clustering
11/16 Unsupervised learning -- GMMs, EM
11/21 NO CLASS -- Thanksgiving
11/23 NO CLASS -- Thanksgiving
11/28 Unsupervised learning -- Spectral methods
11/30 Minimax theory
12/12 Project presentations