Back to overview over all lectures...

Information Theory II
Spring 2025


Note: This homepage is still under construction! Please be aware that some details might change!

⇒ to time table and download of class material

News

  • New room: We are in a new lecture room: ETZ H91.

Aim

This course builds on Information Theory I. It introduces additional topics in single-user communication, connections between Information Theory and Statistics, and Network Information Theory.

The course has two objectives: to introduce the students to the key information theoretic results that underlie the design of communication systems and to equip the students with the tools that are needed to conduct research in Information Theory.

Contents

  • Distributed lossless data compression (Slepian–Wolf).
  • The multiple-access channel (MAC).
  • The broadcast channel (BC).
  • Channels with states.
  • Method of types, Sanov's theorem, Stein's lemma.
  • Distributed hypothesis testing.
  • Soft covering and wiretap channel.
  • Fisher information, Jeffreys prior.

Prerequisites

  • Undergraduate studies
  • Solid foundation in probability and calculus
  • Pleasure with mathematics
  • Information Theory I

Instructors

Prof. Amos Lapidoth Dr. Stefan M. Moser Dr. Ligong Wang
Office: ETF E107 ETF E104 ETF E104
Phone: 044 632 51 92 044 632 36 24 044 632 36 24
E-mail:

Teaching Assistant

Baohua Ni
Office: ETF E105
Phone: 044 632 28 99
E-mail:

Time and Place

  • Lecture: Thursday, 14:15–16:00, ETZ H91
  • Exercise: Thursday, 16:15–18:00, ETZ H91

Textbook

We use the book by Thomas M. Cover and Joy A. Thomas:
T.M. Cover, J.A. Thomas, Elements of Information Theory, 2nd Edition, John Wiley & Sons, Inc., New York, NY, USA.
(Here is a link to an electronic copy of it.)

Supplementary Notes

Stefan M. Moser: Information Theory (Lecture Notes), 6th edition, Signal and Information Processing Laboratory, ETH Zürich, Switzerland, and Institute of Communications Engineering, National Yang Ming Chiao Tung University (NYCU), Hsinchu, Taiwan, 2018.

Stefan M. Moser: Advanced Topics in Information Theory (Lecture Notes), 5th edition, Signal and Information Processing Laboratory, ETH Zürich, Switzerland, and Institute of Communications Engineering, National Yang Ming Chiao Tung University (NYCU), Hsinchu, Taiwan, 2022.

A. El Gamal, Y.-H. Kim, Network Information Theory, Cambridge University Press

G. Kramer, Topics in Multi-User Information Theory, available here.

Exercises

Printed copies of the exercises will be handed out in class. See time table below.

Examination

Oral exam (30 min.).

Testat requirements

None.

Special Remarks

The lecture is held in English.

Time Table

Note that some details of the program might change in the course of the semester. Also note that some linked documents in this table can only be downloaded within ETHZ.

W Date Topic Handouts Exercise Solutions Lecturer
1 20 Feb. Distributed lossless data compression: Slepian–Wolf coding Infos Contents Amos Lapidoth
2 27 Feb. Multiple-Access Channel: channel model and definition of capacity region; examples: Independent BSCs, binary multiplier MAC, binary erasure MAC; time sharing and convexity of capacity region; convex hull and Caratheodory theorem*; proof of achievability (ATIT: 21; CT: 15.3   Exercise 1 Solutions 1 Amos Lapidoth
3 6 Mar. Multiple-Access Channel: q-pentagons and Q-region (i.e. second form of capacity region: coded time-sharing), equivalence to time-sharing*; proof of converse; Gaussian MAC: capacity region, achievability via rate splitting (onion piling) (ATIT: 21; CT: 15.3)   Exercise 2 Solutions 2 Amos Lapidoth
4 13 Mar. Channels with states: channel model and definition; achievability via random coding; Gel'fand–Pinsker rate and capacity: convexity of GP rate and bound on cardinality of auxiliary RV (ATIT: 18.1-18.3, 4.3 [Markov Lemma])   Exercise 3 Solutions 3 Stefan Moser
5 20 Mar. Gel'fand–Pinsker problem: converse; examples: Channels that Sometimes Get Stuck, Writing on Dirty Paper; different types of side-information (ATIT: 18.4-18.8)   Exercise 4 Solutions 4 Stefan Moser
6 27 Mar. Broadcast channel: channel model and definition of capacity region; observations (ATIT: 23.2); physically degraded and stochastically degraded BC; achievability via superposition coding (ATIT: Thm 23.23); converse for degraded BC; Gaussian BC: capacity region (ATIT: 23; CT: 15.6)   Exercise 5 Solutions 5 Stefan Moser
7 3 Apr. Method of types: introduction, number and size of type classes, probability of observing a type; large deviation theory: universal source coding, Sanov's theorem (CT: 11; ATIT: 3,5)   Exercise 6 Solutions 6 Ligong Wang
8 10 Apr. Minimizing relative entropy subject to expectation constraints; Chernoff Information; Stein's Lemma (CT: 11; ATIT: 3,5)   Exercise 7 Solutions 7 Ligong Wang
9 17 Apr. Distributed hypothesis testing   Exercise 8 Solutions 8 Ligong Wang
24 Apr. holidays    
1 May public holiday    
10 8 May Soft covering: proof of direct part; comparison to channel coding theorem and rate-distortion theorem; outline of converse part   Exercise 9 Solutions 9 Ligong Wang
11 15 May Wiretap channel (ATIT: 20)   Exercise 10 Solutions 10 Stefan Moser
12 22 May Fisher Information; Jeffreys prior   Exercise 11 Solutions 11 Amos Lapidoth
29 May public holiday    

-||-   _|_ _|_     /    __|__   Stefan M. Moser
[-]     --__|__   /__\    /__   Senior Researcher & Lecturer, ETH Zurich, Switzerland
_|_     -- --|-    _     /  /   Adj. Professor, National Yang Ming Chiao Tung University (NCTU), Taiwan
/ \     []  \|    |_|   / \/    Web: https://moser-isi.ethz.ch/


Last modified: Sat Jan 11 20:35:23 UTC 2025