COMPSCI 650 (Applied Information Theory)


Arya Mazumdar


TuTh 11:30 - 12:45 CS 140

Office Hour

Tu 15:00-16:00 CS 222


This course will introduce the basic concepts of Information Theory: entropy, relative entropy, mutual information, channel capacity, and rate distortion (data compression). Applications of these concepts will be emphasized. The notions of sparse-graph codes, message passing and belief propagation will also be introduced and studied in detail. Applications in inference algorithms and machine learning will be highlighted; in particular, topics of neural networks, statistical estimation, community detection and clustering will be covered.


No required textbook. The following references are useful.

  1. Thomas M. Cover, Joy A. Thomas, Elements of Information Theory, 2nd Edition, Wiley, Link.

  2. David MacKay, Information Theory, Inference, and Learning Algorithms, Cambridge. Download free HERE.

Grading Plan

There will be 4 home assignments, 1 midterm and 1 final exam.

Homework Plan

A group of two students will submit a single homework that will reflect the collaborative effort of that group of students. Assignments must be submitted at the beginning of the class on the deadline. Late submission by a day at the instructors office will incur a 20% penalty. Submissions will not be accepted if there is any more delay without substantial reason (such as a doctor's note).

Scribing Plan

A group of two students will scribe the notes for one (or two) lectures in TeX. Template for lecture notes will be provided. Scribing for two lectures will automatically earn full points for that group of students. Scribed notes must be submitted by email to the instructor within a week of the lecture. Late submission by a day will incur a 20% penalty. Any more delay without substantial reason will incur a penalty that will depend on the instructor's discretion.


Lecture Date Topics Notes
1 Tu Jan 19 What is information, what to expect from this course, entropy Lecture 1
2 Th Jan 21 Joint entropy, conditional entropy, relative entropy and mutual information Lecture 2
3 Tu Jan 26 Properties of mutual information, data compression, uniquely decodable codes, Kraft's inequality, fundamental limit Lecture 3
4 Th Jan 28 Instantaneous codes, Huffman tree, Shannon coding, Huffman code Lecture 4
5 Tu Feb 2 Huffman code, Shannon-Fano-Elias code, Arithmetic coding, competitive optimality Lecture 5
6 Tu Feb 9 Lempel-Ziv coding, basis of source coding, asymptotic equipartition property, typical set, large deviation Lecture 6
7 Th Feb 11 Chernoff bound and KL-divergence, chain rules, Jensen's inequality, data-processing inequality Lecture 7
8 Th Feb 18 Fano's inequality, multiple-hypothesis testing, binary hypothesis testing, Le Cam's identity, total variation distance, Pinsker's inequality Lecture 8
9 Tu Feb 23 Pinsker's inequality, Binary hypothesis testing, Neyman-Pearson test, optimality, Bayes’ test Lecture 9
10 Th Feb 25 Application of binary hypothesis testing, Gaussian noise, binary symmetric channel, codes; Application of Pinsker's inequality: Multi-Arm Bandit problems Lecture 10
11 Tu Mar 1 Finding biased coin, Multi-Arm-Bandit regret lower bound Lecture 11
12 Th Mar 3 Multi-Arm-Bandit intuitions, differential entropy, quantization, maximal entropy, parameter estimation Lecture 12
13 Tu Mar 8 Secret sharing, parameter estimation, Fisher Information, score function, Cramer_Rao bound Lecture 13
Midterm Th Mar 10 Midterm Midterm
14 Tu Mar 22 One time pad, the big picture, Shannon's communication model, discrete memoryless channel, BSC, codes, capacity, Shannon capacity converse Lecture 14
15 Th Mar 24 Channel capacity, binary erasure channel, Gaussian channel and SNR, random codes achieve capacity Lecture 15
16 Tu Mar 29 BSC and BEC, linear codes, a note on finite fields, parity check code and parity check matrix Lecture 16
17 Th Mar 31 Midterm review, Hamming code, decoding, random linear codes achieve capacity Lecture 17
18 Tu Apr 5 Linear codes, correcting errors and erasures, Singleton bound, iterative decoding on Tanner graph, stopping set, ML and bit-MAP decoding Lecture 18
19 Th Apr 7 Iterative decoding, belief propagation, message-passing rules for LDPC codes, expander graphs Lecture 19
20 Tu Apr 12 Expander graph and codes, bit-flip decoding, network coding, max-flow-mincut, multicast Lecture 20
21 Th Apr 14 Network coding multicast: main theorem, linear codes achieve multicast capacity via the sparse-zeros lemma, introduction to compressed sensing, single pixel camera Lecture 21
22 Tu Apr 19 Compressible signal (Kolmogorov complexity) and compressed sensing, spark condition, Vandermonde matrix, RIP condition and sparse recovery Lecture 22
23 Th Apr 21 Mixture of Gaussians, ML clustering, Newton-Raphson and soft K-means Lecture 23
24 Tu Apr 26 Information theoretic clustering, review of the class Best Wishes




Solution is in Moodle.


Solution is in Moodle.