Machine Learning Theory

CMPSCI 690M, Fall 2017

Akshay Krishnamurthy

Reading List

Concentration of Measure

• Bousquet, Boucheron, Massart book

PAC Learning

• An Almost Optimal PAC Algorithm
• The Optimal Sample Complexity of PAC Learning

Generalization

• Learnability, Stability and Uniform Convergence
• Stability and Generalization
• Understanding deep learning requires rethinking generalization

Empirical Process Theory

• Sara Van de Geer's Notes
• Anthony and Bartlett book

Nonparametric Estimation

• Tsybakov's book
• Larry Wasserman's book
• Nonparametric von Mises Estimators for Entropies, Divergences, and Mutual Informations
• Rates of convergence for nearest neighbor classification
• Fast learning rates for plug-in classifiers
• k-NN Regression Adapts to Local Intrinsic Dimension

Model Selection/Cross Validation

• Optimal oracle inequalities for model selection
• Oracle inequalities for computationally budgeted model selection
• Structural Risk Minimization over Data-Dependent Hierarchies
• Exact post-selection inference, with application to the lasso
• Exact Post-Selection Inference for Sequential Regression Procedures
• Generalization in Adaptive Data Analysis and Holdout Reuse
• Algorithmic Stability for Adaptive Data Analysis

Boosting

• Schapire and Freund book
• Learning Deep ResNet Blocks Sequentially using Boosting Theory
• Optimal and Adaptive Algorithms for Online Boosting
• Margins, Shrinkage, and Boosting

Margin-Bounds

• Theory of classification: A survey of some recent advances
• Empirical Margin Distributions and Bounding the Generalization Error of Combined Classifiers
• For valid generalization, the size of hte weights is more important than the size of the network
• Spectrally-normalized margin bounds for neural networks

Linear Prediction

• Smola, Scholkopf book
• ell_1 regularized neural networks are improperly learnable in polynomial time

Surrogate Losses

• Convexity, classification and risk bound s
• Minimizing The Misclassification Error Rate Using a Surrogate Convex Loss
• Multiclass Classification Calibration Functions

Convex Optimization

• Nesterov's book
• Ben-Tal and Nemirovski's lectures
• Boyd and Vandenberghe's book
• Bubeck Survey
• Lots and lots of new papers on nonconvex optimization

Online Learning

• Cesa-Bianchi and Lugosi Book
• Rakhlin and Sridharan course notes
• Shalev-Shwartz Survey
• Hazan Survey

Follow-the Perturbed Leader

• Efficient algorithms for online decision problems
• Adaptive Online Prediction by Following the Perturbed Leader
• Efficient algorithms for adversarial contextual learning
• Online Linear Optimization via Smoothing

Adversarial Bandits

• Regret Analysis of Stochastic and Nonstochastic Multi-armed Bandit Problems
• The non-stochastic multi-armed bandit problem
• Explore no more: Improved high-probability regret bounds for non-stochastic bandits
• Refined Lower Bounds for Adversarial Bandits
• Open Problem: First-Order Regret Bounds for Contextual Bandits

Stochastic Bandits

• Improved Algorithms for Linear Stochastic Bandits
• Analysis of Thompson Sampling for the multi-armed bandit problem
• lil' UCB : An Optimal Exploration Algorithm for Multi-Armed Bandits
• Taming the Monster: A Fast and Simple Algorithm for Contextual Bandits

K-Means clustering and learning Gaussian Mixtures

• k-means++: The Advantages of Careful Seeding
• Streaming k-means approximation
• Learning Mixtures of Gaussians
• Settling the Polynomial Learnability of Mixtures of Gaussians

Expectation Maximization

• Statistical guarantees for the EM algorithm: From population to sample-based analysis

Spectral Methods

• Tensor decompositions for learning latent variable models

Minimax Theory

• Tsybakov's book
• Assouad, Fano, and Le Cam