Information Theory II
Spring 2023
⇒ to time table and download of class material
Aim
This course builds on Information Theory I. It introduces additional topics in single-user communication, connections between Information Theory and Statistics, and Network Information Theory.
The course has two objectives: to introduce the students to the key information theoretic results that underlie the design of communication systems and to equip the students with the tools that are needed to conduct research in Information Theory.
Contents
- Differential entropy, including maximum entropy principle
- AEP for continuous RVs
- The Gaussian channel
- The multiple-access channel (MAC)
- The broadcast channel (BC)
- Channels with states
- Method of types, Sanov's theorem, Stein's lemma
- Soft covering and wiretap channel
- Fisher information
Prerequisites
- Undergraduate studies
- Solid foundation in probability and calculus
- Pleasure with mathematics
- Information Theory I
Instructors
| Prof. Amos Lapidoth | Prof. Stefan M. Moser | |
| Office: | ETF E107 | ETF E104 |
| Phone: | 044 632 51 92 | 044 632 36 24 |
| E-mail: |  |
 |
Teaching Assistant
Yiming Yan
Office: ETF D107
Phone: 044 632 65 87
E-mail: 
Time and Place
- Lecture: Thursday, 14:15–16:00, ETZ E9
- Exercise: Thursday, 16:15–18:00, ETZ E9
Textbook
We use the book by Thomas M. Cover and Joy A. Thomas:
T.M. Cover, J.A. Thomas, Elements of Information Theory,
2nd Edition, John Wiley & Sons, Inc., New York, NY, USA.
(Here is a link to an
electronic copy of it.)
Supplementary Notes
Stefan M. Moser: Information Theory (Lecture Notes), 6th edition, Signal and Information Processing Laboratory, ETH Zürich, Switzerland, and Institute of Communications Engineering, National Yang Ming Chiao Tung University (NYCU), Hsinchu, Taiwan, 2018.
Stefan M. Moser: Advanced Topics in Information Theory (Lecture Notes), 5th edition, Signal and Information Processing Laboratory, ETH Zürich, Switzerland, and Institute of Communications Engineering, National Yang Ming Chiao Tung University (NYCU), Hsinchu, Taiwan, 2022.
A. El Gamal, Y.-H. Kim, Network Information Theory, Cambridge University Press
G. Kramer, Topics in Multi-User Information Theory, available here.
Exercises
Printed copies of the exercises will be handed out in class. See time table below.
Examination
Oral exam (30 min.).
Testat requirements
None.
Special Remarks
The lecture is held in English.
Time Table
| W | Date | Topic | Handouts | Exercise | Solutions | Lecturer |
| 1 | 23 Feb. | Differential entropy, relative entropy and mutual information for continuous RVs; max-entropy principle; AEP for continuous RVs (IT: 16.1–5, 3.4; CT: 8.1, 8.3–6, 12.1) | Infos Contents | — | — | Stefan M. Moser |
| 2 | 2 Mar. | AEP for continuous RVs; typical sets, jointly typical sets, and their propoerties; Gaussian channel: definitions, channel coding theorem, and intuition (IT: 20.13, 17; CT: 8.2, 9.1) | Exercise 1 | Solutions 1 | Stefan M. Moser | |
| 3 | 9 Mar. | Channel coding theorem for Gaussian channel: definitions, achievability via random coding, and converse via Fano's inequality; joint source and channel coding (IT: 17, 15; ATIT: 11.8; CT: 9.1–2) | Exercise 2 | Solutions 2 | Stefan M. Moser | |
| 4 | 16 Mar. | Multiple-Access Channel: channel model and definition of capacity region; examples: Independent BSCs, binary multiplier MAC, binary erasure MAC; time sharing and convexity of capacity region; convex hull and Caratheodory theorem; proof of achievability (ATIT: 21; CT: 15.3) | Exercise 3 | Solutions 3 | Amos Lapidoth | |
| 5 | 23 Mar. | Multiple-Access Channel: q-pentagons and Q-region (i.e. second form of capacity region: coded time-sharing), equivalence to time-sharing*; proof of converse; Gaussian MAC: capacity region, achievability via rate splitting (onion piling) (ATIT: 21; CT: 15.3) | Exercise 4 | Solutions 4 | Amos Lapidoth | |
| 6 | 30 Mar. | Broadcast channel: channel model and definition of capacity region; observations (ATIT: 23.2); physically degraded and stochastically degraded BC; achievability via superposition coding (ATIT: Thm 23.23); converse for degraded BC; Gaussian BC: capacity region (ATIT: 23; CT: 15.6) | Converse Proof BC | Exercise 5 | Solutions 5 | Stefan M. Moser |
| 7 | 6 Apr. | Channels with states: channel model and definition; achievability via random coding; Gel'fand-Pinsker rate and capacity: convexity of GP rate and bound on cardinality of auxiliary RV (ATIT: 18.1–18.3, 4.3 [Markov Lemma]) | Exercise 6 | Solutions 6 | Stefan M. Moser | |
| — | 13 Apr. | holidays | — | — | ||
| 8 | 20 Apr. | Gel'fand-Pinsker problem: converse; examples: Channels that Sometimes Get Stuck, Writing on Dirty Paper; different types of side-information (ATIT: 18.4–18.8) | Exercise 7 | Solutions 7 | Stefan M. Moser | |
| 9 | 27 Apr. | Method of types: introduction, number and size of type classes, probability of observing a type; large deviation theory: universal source coding, Sanov's theorem (CT: 11; ATIT: 3, 5) | Exercise 8 | Solutions 8 | Amos Lapidoth | |
| 10 | 4 May | Minimizing relative entropy subject to expectation constraints; Chernoff Information; Stein's Lemma (CT: 11; ATIT: 3, 5) | Exercise 9 | Solutions 9 | Amos Lapidoth | |
| 11 | 11 May | Soft covering: proof of direct part; comparison to channel coding theorem and rate-distortion theorem; outline of converse part (ATIT: 19) | Exercise 10 | Solutions 10 | Amos Lapidoth | |
| — | 18 May | holiday | — | — | ||
| 12 | 25 May | Wiretap channel (ATIT: 20) | Exercise 11 | Solutions 11 | Amos Lapidoth | |
| 13 | 1 Jun. | Fisher Information (ATIT: 8.1–3) | Exercise 12 | Solutions 12 | Amos Lapidoth |
-||- _|_ _|_ / __|__ Stefan M. Moser 
[-] --__|__ /__\ /__ Senior Researcher & Lecturer, ETH Zurich, Switzerland
_|_ -- --|- _ / / Adj. Professor, National Yang Ming Chiao Tung University (NCTU), Taiwan
/ \ [] \| |_| / \/ Web: https://moser-isi.ethz.ch/
Last modified: Tue Oct 17 12:51:30 UTC 2023