1. 1/21 
Tue 
Course overview. Probability review. Estimating set size by counting duplicates. 
Slides. Compressed slides. MIT short videos and exercises on probability. Khan academy probability lessons (a bit more basic). 
Randomized Methods, Sketching & Streaming 
2. 1/23 
Thu 
Concentration Bounds: Markov's inequality and Chebyshev's inequality. Random hashing for efficient lookup and load balancing. 2universal and pairwise independent hashing. 
Slides. Compressed slides. Some notes (Arora and Kothari at Princeton) proving that the ax+b mod p hash function described in class in 2universal. 
3. 1/28 
Tue 
Union bound. Exponential tail bounds (Bernstein and Chernoff). Example applications. 
Slides. Compressed slides. Some notes (Goemans at MIT) showing how to prove the Chernoff bound using the moment generating function + Markov's inequality approach discussed in class. 
4. 1/30 
Thu 
Hashing continued. Bloom filters and their applications. Hashing for distinct elements. 
Slides. Compressed slides, with Bernstein bound argument fixed. See here for full Bloom filter analysis. See here for some explaination of why a version of a Bloom filter with no false negatives cannot be achieved without using a lot of space. See Wikipedia for a discussion of the many bloom filter variants, including counting Bloom filters, and Bloom filters with deletions. 
5. 2/4 
Tue 
Distinct elements continued. FlajoletMartin and HyperLogLog. Jaccard similarity for audio fingerprinting, document comparision, etc. The median trick. 
Slides. Compressed slides. The 2007 paper introducing the popular HyperLogLog distinct elements algorithm. Chapter 4 of Mining of Massive Datasets, with content on bloom filters, distinct item counting. 
2/6 
Thu 
No Class, Professor Away. 

6. 2/11 
Tue 
Jaccard similarity search with MinHash. Locality sensitive hashing and nearest neighbor search. 
Slides. Compressed slides. Reading: Chapter 3 of Mining of Massive Datasets, with content on Jaccard similarity, MinHash, and locality sensitive hashing. 
7. 2/13 
Thu 
Finish up MinHash and LSH. SimHash for cosine similarity. Start on frequent elements problem. 
Slides. Compressed slides. Reading: Chapter 3 of Mining of Massive Datasets, with content on Jaccard similarity, MinHash, and locality sensitive hashing. Notes (Amit Chakrabarti at Dartmouth) on streaming algorithms. See Chapters 2 and 4 for frequent elements. Some more notes on the frequent elements problem. 
2/18 
Tue 
No Class, Monday Schedule. 

8. 2/20 
Thu 
The frequent elements problem. MisraGries summaries. Countmin sketch. 
Slides. Compressed slides. Reading: Notes (Amit Chakrabarti at Dartmouth) on streaming algorithms. See Chapters 2 and 4 for frequent elements. Some more notes on the frequent elements problem. A website with lots of resources, implementations, and example applications of countmin sketch.

9. 2/25 
Tue 
Countmin sketch analysis. Start on dimensionality reduction and lowdistortion embeddings. 
Slides. Compressed slides. Reading: Chapter 2.7 of Foundations of Data Science on the JohnsonLindenstrauss lemma. Notes on the JLLemma (Anupam Gupta CMU). Linear Algebra Review: Khan academy. 
10. 2/27 
Thu 
The Johnson Lindenstrauss Lemma proof. 
Slides. Compressed slides. Reading: Chapter 2.7 of Foundations of Data Science on the JohnsonLindenstrauss lemma. Notes on the JLLemma (Anupam Gupta CMU). Sparse random projections which can be multiplied by more quickly. JL type random projections for the l1 norm using Cauchy instead of Gaussian random matrices. 
11. 3/3 
Tue 
Finish up the JL Lemma. Applications to clustering, classification, etc. Connections to highdimensional geometry.

Slides. Compressed slides. 
12. 3/5 
Thu 
Finish up highdimensional geometry and connection to the JL Lemma.

Slides: Slides. Compressed slides. Reading: Chapters 2.32.6 of Foundations of Data Science on highdimensional geometry. 
Spectral Methods 
13. 3/10 
Tue 
Midterm Review. Intro to principal component analysis, lowrank approximation, datadependent dimensionality reduction. 
Slides. Compressed slides.. Reading: Chapter 3 of Foundations of Data Science and Chapter 11 of Mining of Massive Datasets on lowrank approximation and the SVD. 
3/12 
Thu 
Midterm (In Class) 
Study guide and review questions. 
3/17 
Tue 
No Class, Spring Recess. 

3/19 
Thu 
No Class, Spring Recess. 

14. 3/24 
Tue 
Intro to lowrank approximation. Projection matrices and best fit subspaces. 
Slides. Compressed slides. Zoom Recording.
Reading: Some notes on PCA and its connection to eigendecomposition (Roughgarden and Valiant at Stanford). 
15. 3/26 
Thu 
Optimal lowrank approximation via eigendecomposition. Principal component analysis. 
Slides. Compressed slides. Zoom Recording. Reading: Some notes on PCA and its connection to eigendecomposition and singular value decomposition (SVD) (Roughgarden and Valiant at Stanford). Chapter 3 of Foundations of Data Science and Chapter 11 of Mining of Massive Datasets on lowrank approximation and the SVD. 
16. 3/31 
Tue 
The singular value decomposition and connections to eigendecomposition and PCA. Applications of lowrank approximation beyond compression.

Slides. Compressed slides. Zoom Recording. Reading: Notes on SVD and its connection to eigendecomposition/PCA (Roughgarden and Valiant at Stanford). Chapter 3 of Foundations of Data Science and Chapter 11 of Mining of Massive Datasets on lowrank approximation and the SVD. 
17. 4/1 
Thu 
Linear algebraic view of graphs. Applications to spectral clustering, community detection, network visualization.

Slides. Compressed slides. Zoom recording. Reading: Chapter 10.4 of Mining of Massive Datasets on spectral graph partitioning. Great notes on spectral graph methods (Roughgarden and Valiant at Stanford). 
18. 4/7 
Tue 
Spectral graph partitioning. 
Slides. Compressed slides. Zoom recording. Reading: Chapter 10.4 of Mining of Massive Datasets on spectral graph partitioning. For a lot more interesting material on spectral graph methods see Dan Spielman's lecture notes. 
19. 4/9 
Thu 
The stochastic block model. 
Slides. Compressed slides. Zoom recording. 
20. 4/14 
Tue 
Computing the SVD: power method, Krylov methods. Connection to random walks and Markov chains. 
Slides. Compressed slides. Zoom recording.
Reading: Chapter 3.7 of Foundations of Data Science on the power method for SVD. Some notes on the power method. (Roughgarden and Valiant at Stanford).

Optimization 
21. 4/16 
Thu 
Finish power method and Krylov methods. Start on continuous optimization. 
Slides. Compressed slides. Zoom recording. Reading: Chapters I and III of these notes (Hardt at Berkeley). Multivariable calc review, e.g., through: Khan academy. 
22. 4/21 
Tue 
Gradient descent and analysis for convex functions. 
Slides. Compressed slides. Zoom recording. Reading: Chapters I and III of these notes (Hardt at Berkeley).

23. 4/23 
Thu 
Finish gradient descent analysis. Constrained optimization and projected gradient descent. 
Slides. Compressed slides. Zoom Recording. Reading: Chapters I and III of these notes (Hardt at Berkeley). 
24. 4/28 
Tue 
Online gradient descent and application to the analysis of stochastic gradient descent. Class wrapup. 
Slides. Compressed slides. Zoom Recording. Reading: Short notes, proving regret bound for online gradient descent. A good book, (by Elad Hazan) on online optimization, including online gradient descent and connection to stochastic gradient descent. Note that the analysis is close to, but slightly different than will be covered in class. 
5/6 
Wed 
Final (2:00pm4:00pm on Zoom) 
Study guide and review questions. 