Course Schedule (Evolving)

Lecture recordings from Echo360 can be accessed here. Handwritten lecture notes courtesy of Stephen Scarano can be accessed here.

Lecture      Day Topic Materials/Reading
1. 9/5 Tue Course overview. Probability review. Linearity of expectation. Slides. Compressed slides. Reading: MIT short videos and exercises on probability (go to Unit 4). Khan academy probability lessons (a bit more basic). Chapters 1-3 of Probability and Computing with content and excersises on basic probabiliy, expectation, variance, and concentration bounds.
Randomized Methods, Sketching & Streaming
2. 9/7 Thu Estimating set size by counting duplicates. Markov's inequality. Random hashing for efficient lookup. Collision-free hashing. Slides. Compressed slides. Reading: Chapters 1-3 of Probability and Computing with content and excersises on basic probabiliy, expectation, variance, and concentration bounds.
3. 9/12 Tue More random hashing: 2-level hashing. 2-universal and pairwise independent hashing. Slides. Compressed slides. Reading: Chapter 2.2 of Foundations of Data Science with content on Markov's inequality and Chebyshev's inequality. Exercises 2.1-2.6. Chapters 1-3 of Probability and Computing with content and excersises on basic probabiliy, expectation, variance, and concentration bounds. Some notes (Arora and Kothari at Princeton) proving that the ax+b mod p hash function described in class in 2-universal.
4. 9/14 Thu Hashing for load balancing. Chebyshev's inequality. The union bound. Slides. Compressed slides. Reading: Chapter 2.2 of Foundations of Data Science with content on Markov's inequality and Chebyshev's inequality. Exercises 2.1-2.6. Chapters 1-3 of Probability and Computing with content and excersises on basic probabiliy, expectation, variance, and concentration bounds.
5. 9/19 Tue Exponential concentration bounds and the central limit theorem. Slides. Compressed slides. Reading: Chapter 4 of Probability and Computing on exponential concentration bounds. Some notes (Goemans at MIT) showing how to prove exponential tail bounds using the moment generating function + Markov's inequality approach discussed in class.
6. 9/21 Thu Finish up exponential concentration bounds. Bloom Filters. Slides. Compressed slides. Reading: Chapter 4 of Probability and Computing on exponential concentration bounds. Chapter 4 of Mining of Massive Datasets, with content on bloom filters. See here for some explaination of why a version of a Bloom filter with no false negatives cannot be achieved without using a lot of space. See Wikipedia for a discussion of the many bloom filter variants, including counting Bloom filters, and Bloom filters with deletions. See Wikipedia again and these notes for an explaination of Cuckoo Hashing, a randomized hash table scheme which, like 2-level hashing, has O(1) query time, but also has expected O(1) insertion time.
7. 9/26 Tue Finish up Bloom filter analysis. Slides. Compressed slides. Reading: Chapter 4 of Mining of Massive Datasets, with content on bloom filters. See here for full Bloom filter analysis.
8. 9/28 Thu Min-Hashing for Distinct Elements. The median trick. Slides. Compressed slides. Reading: Chapter 4 of Mining of Massive Datasets, with content on distinct elements counting.
9. 10/3 Tue Distinct elements in pratice: Flajolet-Martin and HyperLogLog. Start on Jaccard similarity and motivation for the fast similarity search problem. Slides. Compressed slides. Reading: The 2007 paper introducing the popular HyperLogLog distinct elements algorithm. Chapter 3 of Mining of Massive Datasets, with content on Jaccard similarity, MinHash, and locality sensitive hashing.
10. 10/5 Thu Fast similarity search via locality sensitive hashing. MinHashing for Jaccard similarity. Slides. Compressed slides. Reading: Chapter 3 of Mining of Massive Datasets, with content on Jaccard similarity, MinHash, and locality sensitive hashing.
10/10 Tue No Class. Monday class schedule followed.
11. 10/12 Thu Frequent elements estimation via Count-min sketch. Slides. Compressed slides. Reading: Notes (Amit Chakrabarti at Dartmouth) on streaming algorithms. See Chapters 2 and 4 for frequent elements. Some more notes on the frequent elements problem. A website with lots of resources, implementations, and example applications of count-min sketch. Linear Algebra Review: Khan academy.
12. 10/17 Tue Dimensionality reduction, low-distortion embeddings, and the Johnson Lindenstrauss Lemma. Slides. Compressed slides. Reading: Chapter 2.7 of Foundations of Data Science on the Johnson-Lindenstrauss lemma. Notes on the JL-Lemma (Anupam Gupta (CMU). Sparse random projections which can be multiplied by more quickly.
13. 10/19 Thu Midterm Review. Slides.
10/24 Tue Midterm (In Class) Study guide and review questions.
Spectral Methods
14. 10/26 Thu Finish up the JL Lemma proof. Example application to clustering. Slides. Compressed slides. Reading: Chapter 2.7 of Foundations of Data Science on the Johnson-Lindenstrauss lemma. Notes on the JL-Lemma (Anupam Gupta (CMU).
15. 10/31 Tue Intro to principal component analysis, low-rank approximation, data-dependent dimensionality reduction. Orthogonal bases and projection matrices. Slides. Compressed slides. Reading: Chapter 3 of Foundations of Data Science and Chapter 11 of Mining of Massive Datasets on low-rank approximation and the SVD. Some good videos for linear algebra review. Some other good videos overviewing the SVD and related topics (like orthogonal projection and low-rank approximation).
16. 11/02 Thu Dual column/row view of low-rank approximation. Best fit subspaces and optimal low-rank approximation via eigendecomposition. Slides. Compressed slides. Reading: Proof that optimal low-rank approximation can be found greedily (see Section 1.1). Chapter 3 of Foundations of Data Science and Chapter 11 of Mining of Massive Datasets on low-rank approximation.
17. 11/07 Tue Finish up optimal low-rank approximation via eigendecomposition. Eigenvalues as a measure of low-rank approximation error. Slides. Compressed slides. Reading: Chapter 3 of Foundations of Data Science and Chapter 11 of Mining of Massive Datasets on low-rank approximation.
18. 11/09 Thu The singular value decomposition and connections to low-rank approximation. Applications of low-rank approximation beyond compression. Matrix completion and entity embeddings. Slides. Compressed slides. Reading: Notes on SVD and its connection to eigendecomposition/PCA (Roughgarden and Valiant at Stanford). Notes on matrix completion, with proof of recovery under incoherence assumptions (Jelani Nelson at Harvard). Levy Goldberg paper on word embeddings as implicit low-rank approximation.
19. 11/14 Tue Spectral graph theory and spectral clustering. Slides. Compressed slides. Reading: Chapter 10.4 of Mining of Massive Datasets on spectral graph partitioning. For a lot more interesting material on spectral graph methods see Dan Spielman's lecture notes. Great notes on spectral graph methods (Roughgarden and Valiant at Stanford).
20. 11/16 Thu The stochastic block model. Slides. Compressed slides. Reading: Dan Spielman's lecture notes on stochastic block model, including matrix concentration + David-Kahan perturbation analysis.. Further stochastic block model notes (Alessandro Rinaldo at CMU). A survey of the vast literature on the stochastic block model, beyond the spectral methods discussed in class (Emmanuel Abbe at Princeton).
21. 11/21 Tue Computing the SVD: power method. Krylov methods. Bonus material: connection to random walks and Markov chains. Slides. Compressed slides. Reading: Chapter 3.7 of Foundations of Data Science on the power method for SVD. Some notes on the power method. (Roughgarden and Valiant at Stanford).
11/23 Thu No Class. Thanksgiving recess.
11/28 Tue No Class. Professor traveling.
Optimization
22. 11/30 Thu Start on optimization and gradient descent. Slides. Compressed slides. Reading: Chapters I and III of these notes (Hardt at Berkeley). Multivariable calc review, e.g., through: Khan academy
23. 12/05 Tue Gradient descent analysis for convex Lipschitz functions. Slides. Compressed slides. Reading: Chapters I and III of these notes (Hardt at Berkeley).
24. 12/07 Thu Constrained optimization and projected gradient descent. Course conclusion/review. Slides. Compressed slides.
12/14, 10:30am - 12:30pm Final Exam. Study guide and review questions.