Machine Learning Theory

CMPSCI 690M, Fall 2017

Akshay Krishnamurthy

When: TuTh 2:30-3:45
Where: CS 140
Office Hours: TuTh 3:45-5 (CS 258)

Overview

When, how, and why do machine learning algorithms work? This course answers these questions by studying the theoretical aspects of machine learning, with a focus on statistically and computationally efficient learning. Broad topics will include: PAC-learning, uniform convergence, and model selection; supervised learning algorithms including SVM, boosting, kernel methods; online learning algorithms and analysis; unsupervised learning with guarantees.Special topics may include: Bandits, active learning, semi-supervised learning and others.

Requirements: Coursework will include

5 homework assignments involving proofs and algorithm design, 50% of course grade.
A midterm exam, 20% of course grade.
A research-based project, 30% of course grade. Project guidelines are available here.

Grading will be based on performance on the coursework where above 90% earns an A, above 80% earns a B, above 70% earns a C, and so on.

Prerequisites: CS 689 (Machine Learning) or CS 589 with instructor approval. No programming experience is required for the class but strong mathematical ability will be necessary.

Readings

There is no required textbook for this course. However you may find the following useful.

Understanding Machine Learning: From Theory to Algorithms by Shai Shalev-Shwartz and Shai Ben-David
Prediction, Learning, Games by Nicolo Cesa-Bianchi and Gabor Lugosi
Some papers that may be of interested, related to the course, are listed here
Some other papers that may be of interest, focusing on interactive learning, are listed here

Homeworks

Homework 1. Released 9/5, due 9/19. (Solutions)
Homework 2. Released 9/19, due 10/3. (Solutions)
Homework 3. Released 10/3, due 10/17. (Solutions)
Homework 4. Released 10/17, due 11/2. (Solutions)
Homework 5. Released 11/2, due 11/16. (Solutions)

Feel free to you this latex template and style file.

Projects

Project guidelines are available here. Important dates are:

Project Proposals. Due 10/5 by email.
Project Presentations. On 12/12 in class.
Project Writeup. Due 12/19 by email.

Lecture Schedule

Date	Lecture Topics	Readings	Assignments
9/5	Probabilistic Prediction, PAC-learning	SSBD: Ch 2-3 Notes	Hw 1 out
9/7	Statistics background	SSBD: App B Siva Balakrishnan's Notes Notes
9/12	Agnostic learning, Bias-Complexity tradeoff	SSBD: Ch 4-5 Notes Optional: SSSSS paper
9/14	VC theorem	SSBD: Ch 6, 28 Notes Optional: Sample-optimal PAC learning
9/19	Rademacher complexity	SSBD: Ch 26 Notes Optional: Explaining NN generalization, Rademacher complexity	Hw 1 due Hw 2 out
9/21	Covering numbers, Chaining	SSBD: Ch 27 Notes Optional: Sara van de Geer's notes
9/26	Nonparametric classification/regression	SSBD: Ch 19 Notes Optional: Tsybakov's book
9/28	Model selection, SRM	SSBD: Ch 11 Notes
10/3	Boosting	SSBD: Ch 10 Notes Optional: Schapire and Freund book	Hw 2 due Hw 3 out
10/5	Margin Bounds	Notes Optional: Schapire and Freund book	Project Proposals due
10/10	NO CLASS -- Columbus Day
10/12	Perceptron, SVM, Kernel SVM	SSBD: Ch 9, 15 Notes
10/17	Surrogate losses, calibration	SSBD: Ch 12 Convexity, Classification, and Risk Bounds Notes	Hw 3 due Hw 4 out
10/19	Gradient Descent, convex optimization	SSBD: Ch 12, 14 Notes
10/24	MIDTERM -- in class
10/26	Online learning: Halving, Hedge	SSBD: Ch 21 Notes
10/31	Online Learning: Hedge, FTRL, OGD	SSBD: Ch 21, CBL: Ch 1 Notes Optional: OL Survey
11/2	Online Mirror Descent and FTPL	OL Survey Notes	Hw 4 due Hw 5 out
11/7	Adversarial Bandits	Bandit Survey Notes
11/9	Stochastic Bandits	Bandit Survey Notes
11/14	Unsupervised learning -- Spectral Clustering	von Luxburg tutorial Notes
11/16	Unsupervised learning -- Spectral Methods	Tensor decompositions paper Notes	Hw 5 due
11/21	NO CLASS -- Thanksgiving
11/23	NO CLASS -- Thanksgiving
11/28	Minimax Theory	Notes Bin Yu's notes John Duchi's notes
11/30	Minimax theory	Notes
12/5	NO CLASS -- NIPS
12/7	NO CLASS -- NIPS
12/12	Project presentations