University of Massachusetts Amherst
College of Information and Computer Sciences

COMPSCI 589

Machine Learning

Fall 2019

Course Description:

This course will provide an overview of Machine Learning concepts and techniques. For details on the material, please see the class schedule below.

Lectures: Monday & Wednesday 2:30-3:45pm in Hasbrouck Laboratory Addition (HASA) 20.

Credit: 3 units

Instructors:

Ina Fiterau, mfiterau at cs dot umass dot edu, (413) 577-0064. Office hours: Wednesday, after class.
Robert James Hall, roberthall at cs dot umass dot edu, Office hours: by appointment.

Teaching assistants:

Iman Deznabi, iman [at] cs.umass.edu. Office hours: Thursday 2:30-3:30pm, LGRT EDLAB T223.
Deep Chakraborty, dchakraborty [at] cs.umass.edu. Office hours: Monday 1-2pm, LGRT 220.
Purva Pruthi, ppruthi [at] cs.umass.edu. Office hours: Tuesday 2-3pm, LGRT 220.
Yi Fung, yfung [at] cs.umass.edu.

Junior TAs:

Akhila Josyula, ajosyula [at] umass.edu. Office hours: Wednesday 10.30-11.30am, CS207 Cube 1.
Akshit Tyagi, atyagi [at] cs.umass.edu. Office hours: Wednesday 12-1pm, CS207 Cube 1.
Bharath Narasimhan, bnarasimhan [at] umass.edu
Ritwik Anand, ritqikanand [at] umass.edu. Office hours: Monday 4-5pm, LGRT 220.

Graders:

Danyang Liu, danyangliu [at] cs.umass.edu
Robin Wu, rwu [at] cs.umass.edu
Vaishnavi Kommeraju, vkommaraju [at] cs.umass.edu

Textbooks:

(Required) Trevor Hastie, Robert Tibshirani, Jerome Friedman, The Elements of Statistical Learning , 2nd Edition, Springer Series in Statistics.
(Required) Chris Bishop, Pattern Recognition and Machine Learning, Springer 2006.

Grading:

Homeworks: 50%
Midterm: 20% [21/10/2019, in class]
Final: 30% [12/13/2019, 3:30PM - 5:30PM, Friday, Hasbrouck Laboratory Addition (HASA) 20]
Bonus: participation, in class and on piazza.

Class materials will be posted to the Moodle course.

Discussions will happen on Piazza or over Moodle.

Class schedule and content:

Date	Presenter	Title	Reading (announced after each lecture). Textbook reading is mandatory, any papers are optional.
09/04/19	Rob	Introduction and review of prerequisites Definition of Machine Learning Relationship to other fields Course overview Review: linear algebra Review: probability and random variables	Bishop, Section 1.2.1-1.2.4 Probability Theory
09/09/19	Ina	Classification: k-NN and decision trees Learning problem formulation Regression vs classification; supervised vs unsupervised; parametric vs nonparametric models K-NN classifiers Decision trees	ESL Section 2.3.2 ESL Section 2.5
09/11/19	Ina	Probability and estimation Random variable independence Bayes rule Estimators Maximum likelihood estimator (MLE) Maximum a posteriori estimator (MAP)	Bishop, 2.1 Binary Variables Bishop, 2.2 Multinomial Variables Optional: Mitchell, Estimating Probabilities
09/16/19	Ina	Probabilistic classification Conditional Independence Naive Bayes Gaussian Naive Bayes	Optional: Mitchell, 3.1 and 3.2, Naive Bayes
09/18/19	Ina	Linear Classifiers. Linear Discriminant Analysis (LDA) Fitting linear responses Fitting by least squares Maximizing conditional likelihood LDA - model class conditional densities as multivariate Gaussians	ESL 4.1-4.3 (p. 101-102, 106-110) Bishop 4.1.1-4.1.4 Discriminant Functions Bishop 4.2 Probabilistic Generative Models
		Homework 1 out.
09/23/19	Ina	Logistic Regression (LR) Generative vs discriminative classifiers Classification using the logistics function Gradient methods to solve LR: gradient descent, stochastic gradient descent MLE and MAP estimates for LR Generalization and Evaluation Training error and generalization error Hypothesis space, model capacity Generalization, overfitting, underfitting, bias-variance trade-off Regularization, model selection, cross-validation	ESL Section 4.4 (p. 119-120, 127-132) Optional: Mitchell, 3.3, Logistic Regression
09/25/19	Rob	Machine Learning Theory Theoretical model of ML Generalization bounds Consistent learning PAC learning Anostic learning. Relationship to bia/variance tradeoff Infinite hypothesis space. VC dimension. Sauer's lemma	Optional: Nina Balcan, Notes on generalization guarantees.
09/30/19	Ina	Support Vector Machines Maximizing the margin Hinge loss vs logistic loss Basis expansions and kernels The kernel trick	ESL Section 12.3 ESL Section 12.3.6 (p. 434-438) Bishop 6.1, 6.2 (p. 291 - 299)
		Homework 1 due. Homework 2 out.
10/02/19	Ina	Ensemble Methods I Introduction to ensembles Bagging Random forests	ESL Chapter 15 (p. 587-601)
10/07/19	Rob	Ensemble Methods II Boosting. Adaboost Stacking	ESL Chapter 16 (p. 605-622) Bishop Sections 14.3,14.4 (p. 657 - 665)
10/09/19	Ina	Regression I Linear regression Ordinary least squares	ESL Sections 3.1, 3.2.1 (p. 43-51) ESL Sections 3.4.1-3.4.3 (p. 61-73)
		Homework 2 due. Homework 3 out.
10/14/19		Holiday - Columbus Day. Class will be held on Tuesday.
10/15/19	Ina	Regression II Regularization: ridge regression and the lasso	None in addition to the ones from the previous lecture.
10/16/19	Rob	Revision for Midterm Exam.	Before class, please attempt to solve past midterms (posted to Moodle)
10/21/19		Midterm exam. Everything up to and including ensemble methods, not including regression.
10/23/19	Rob	Regression III Regression trees Feature selection Kernel smoothing	ESL 6.1 and 6.2 (p. 191-200) ESL 9.2.1, 9.2.2 (305-308)
10/28/19	Ina	Deep Learning I: Intro and CNNs The Multilayer Perceptron (MLP) Convolutional Neural Networks (CNNs) for vision	Optional: ESL 11.3 Neural Networks
10/29/19		Homework 3 due. Midterm grades out and re-grading requests open.
10/30/19	Ina	Deep Learning II: Learning NNs Training Neural Networks Backpropagation	Optional: ESL 11.4 Fitting Neural Networks ESL 11.5 Some Issues in Training Neural Networks
11/04/19	Ina	Deep Learning III: Sequential Deep Learning recurrent neural networks (RNN) long-term short-term memory (LSTM)	None.
		End of midterm re-grading request period. Homework 4 out.
11/06/19	Ina	Dimensionality reduction I Linear dimensionality reduction Singular Value Decomposition (SVD)	ESL Section 14.15.1 (p.534-536)
11/11/19		Holiday – Veterans’ Day.
11/13/19	Ina	Dimensionality reduction II Principal Component Analysis (PCA) Connection between PCA and SVD	Bishop 12.1 Principal Component Analysis (p.559-569)
11/18/19	Ina	Dimensionality reduction III Sparse coding Nonnegative matrix factorization Independent Component Analysis (ICA)	ESL Section 14.6 (p.553-557) ESL Section 14.7 (p.557-570)
11/20/19	Ina	Dimensionality reduction IV Kernel PCA Spectral clustering
11/22/19		Homework 4 due. Homework 5 out.
11/25/19		Thanksgiving break.
11/27/19		Thanksgiving break.
12/02/19		Snow day. UMass was closed.
12/04/19	Rob	Clustering K-Means Mixture models Expectation Maximization (EM)	ESL 14.3.4 - 14.3.11 (k=means) ESL 8.5 (EM)
		Homework 5 due.
12/09/19	Students, Ina	Revision I for Final Exam.
12/11/19	Students, Ina	Revision II for Final Exam.
12/13/19		Final Exam. Everything up to and including the lecture on 12/04/19. Does not include the lecture on 12/11/19.

Exam exception policy: If you have any special needs/circumstances pertaining to an exam, you must talk to the instructor at least 2 weeeks before the exam.

Late homework policy: If you cannot turn in a homework on time, you will need to discuss with the instructor at least one day in advance.

Regrade policy: Any requests for regrading must be submitted within a week of receiving the grade and preferably discussed during office hours. Each TA will be responsible for a different part of the homework, as indicated when the assignment is issued, so please direct questions appropriately. Only contact the instructors after discussing the issue with the TAs.

Copyright/distribution notice: Many of the materials created for this course are the intellectual property of the course instructors and of the professors whose courses served as a basis for some of the lectures. This includes, but is not limited to, the syllabus, lectures and course notes. Except to the extent not protected by copyright law, any use, distribution or sale of such materials requires the permission of the instructor. Please be aware that it is a violation of university policy to reproduce, for distribution or sale, class lectures or class notes, unless copyright has been explicitly waived by the faculty member.