9/3 
Tue 
Course overview. Probability review, Markov's inequality. Estimating set size by counting duplicates. 
Slides. 
Randomized Methods, Sketching & Streaming 
9/5 
Thu 
Chebyshev's inequality. Random hashing for efficient lookup and load balancing. 2universal and pairwise independent hashing. 
Slides. Some notes (Arora and Kothari at Princeton) proving that the ax+b mod p hash function described in class in 2universal. 
9/10 
Tue 
Union bound. Exponential tail bounds (Bernstein and Chernoff). Example applications. 
Slides. Some notes (Goemans at MIT) showing how to prove the Chernoff bound using the moment generating function + Markov's inequality approach discussed in class. 
9/12 
Thu 
Hashing continued. Bloom filters and their applications. Hashing for distinct elements. 
Slides. I've added a sketch of the correct Bloom filter analysis. Also see here. See here for some explaination of why a version of a Bloom filter with no false negatives cannot be achieved without using a lot of space. 
9/17 
Tue 
Distinct elements continued. FlajoletMartin and HyperLogLog. Jaccard similarity for audio fingerprinting, document comparision, etc. The median trick. 
Slides. The 2007 paper introducing the popular HyperLogLog distinct elements algorithm. Chapter 4 of Mining of Massive Datasets, with content on bloom filters, distinct item counting. 
9/19 
Thu 
Jaccard similarity search with MinHash. Locality sensitive hashing and nearest neighbor search. 
Slides. Reading: Chapter 3 of Mining of Massive Datasets, with content on Jaccard similarity, MinHash, and locality sensitive hashing. 
9/24 
Tue 
Finish up MinHash and LSH. SimHash for consine similarity. 
Slides. Reading: Chapter 3 of Mining of Massive Datasets, with content on Jaccard similarity, MinHash, and locality sensitive hashing. 
9/26 
Thu 
The frequent elements problem. MisraGries summaries. Countmin sketch. 
Slides. Reading: Notes (Amit Chakrabarti at Dartmouth) on streaming algorithms. See Chapters 2 and 4 for frequent elements. Some more notes on the frequent elements problem. A website with lots of resources, implementations, and example applications of countmin sketch.

10/1 
Tue 
Randomized dimensionality reduction and the JohnsonLindenstrauss lemma. Applications to regression, clustering. 
Slides: Compressed/cleaned up, Raw from class. Reading: Chapter 2.7 of Foundations of Data Science on the JohnsonLindenstrauss lemma. Notes on the JLLemma (Anupam Gupta CMU). Linear Algebra Review: Khan academy. 
10/3 
Thu 
Finish up JL Lemma. 
Slides. The Fast JL transform: speeding up random projection with the Fast Fourier transform. Sparse random projections which can be multiplied by more quickly. JL type random projections for the l1 norm using Cauchy instead of Gaussian random matrices. 
Spectral Methods 
10/8 
Tue 
Principal component analysis, lowrank approximation, dimensionality reduction. 
Slides: Compressed/cleaned up, Raw from class. Reading: Chapter 3 of Foundations of Data Science and Chapter 11 of Mining of Massive Datasets on lowrank approximation and the SVD. 
10/10 
Thu 
Eigencomposition and application to PCA and lowrank approximation. 
Slides: Cleaned up, Raw from class. Reading: Some notes on PCA and its connection to eigendecomposition (Roughgarden and Valiant at Stanford). 
10/15 
Tue 
No Class, Monday Schedule. 

10/17 
Thu 
Midterm (In Class) 
Study guide and review questions. 
10/22 
Tue 
The singular value decomposition and its connection to eigendecomposition/PCA/lowrank approximation. Applications of lowrank approximation beyond compression. 
Slides (raw from class). Unannotated slides. Reading: Chapter 3 of Foundations of Data Science and Chapter 11 of Mining of Massive Datasets on lowrank approximation and the SVD. Some notes on the SVD and its connection to PCA (Roughgarden and Valiant at Stanford) 
10/24 
Thu 
Linear algebraic view of graphs. Applications to spectral clustering, community detection, network visualization. 
Slides (raw from class). Unannotated slides. Reading: Chapter 10.4 of Mining of Massive Datasets on spectral graph partitioning. Great notes on spectral graph methods (Roughgarden and Valiant at Stanford). 
10/29 
Tue 
Spectral graph theory, spectral clustering, and community detection continued. Stochastic block model 
Slides (raw from class). Unannotated slides. Reading: Chapter 10.4 of Mining of Massive Datasets on spectral graph partitioning. For a lot more interesting material on spectral graph methods see Dan Spielman's lecture notes. 
10/31 
Thu 
Finish up stochastic block model. Computing the SVD: power method, Krylov methods. 
Slides (raw from class). Unannotated slides. Reading: Chapter 3.7 of Foundations of Data Science on the power method for SVD. Some notes on the power method. (Roughgarden and Valiant at Stanford).

11/5 
Tue 
Class Cancelled. 

11/7 
Thu 
Finish up power method. Connection to random walks and Markov chains. 
Slides (raw from class). Unannotated slides. 
Optimization 
11/12 
Tue 
Gradient descent and analysis for convex functions, example applications. 
Slides (raw from class). Unannotated slides. Reading: Chapters I and III of these notes (Hardt at Berkeley). Multivariable calc review, e.g., through: Khan academy. 
11/14 
Thu 
Finish gradient descent. Projected gradient descent. 
Slides (raw from class). Unannotated slides. Chapters I and III of these notes (Hardt at Berkeley). 
11/19 
Tue 
Stochastic gradient descent for large scale learning. Analysis via online gradient descent. 
Slides (raw from class). Unannotated slides. Reading: Short notes, proving regret bound for online gradient descent. A good book, (by Elad Hazan) on online optimization, including online gradient descent and connection to stochastic gradient descent. Note that the analysis is close to, but slightly different than what was covered in class. 
11/21 
Thu 
Finish up SGD. Gradient descent for least squares regression. Connections to advanced techniques: variance reduction, accelerated methods, adaptive gradient methods. 
Slides (raw from class). Unannotated slides. 
11/26 
Tue 
No Class, Thanksgiving Recess. 

11/28 
Thu 
No Class, Thanksgiving Recess. 

Assorted Topics 
12/3 
Tue 
Highdimensional geometry, curse of dimensionality. 
Slides (raw from class). Unannotated slides. Reading: Chapters 2.32.6 of Foundations of Data Science on highdimensional geometry. 
12/5 
Thu 
Compressed sensing, sparse recovery. 
Slides (raw from class). Unannotated slides. 
12/10 
Tue 
Finish up sparse recovery and basis pursuit. Class wrapup. 
Slides (raw from class). Unannotated slides. 
12/19 
Thu 
Final (10:30am12:30pm in Thompson 104) 
Study guide and review questions. 