Back to overview over all lectures...

Information Theory I
Fall 2025

Please be aware that the program is preliminary and subject to change.

⇒ to time table and download of class material

News

  • Prof. Lapidoth is on sabbatical leave this semester. The course will be taught by myself and my colleague Dr. Ligong Wang.

Aim

This course teaches the fundamentals of Information Theory including the basic source coding and channel coding theorems.

Contents

  • Information theoretic quantities and their properties: entropy, conditional entropy, mutual information.
  • Data compression: efficient coding of a single random message:
    • prefix-free codes
    • Kraft inequality
    • trees with probability
    • Shannon-type codes, Shannon codes
    • Huffman codes
    • nonsingular codes
  • Data compression: efficient coding of a memoryless random source:
    • block–to–variable-length coding
    • source coding theorem: achievability and converse
    • variable-length–to–block coding: Tunstall codes
  • Data compression: efficient coding of a stationary source with memory:
    • entropy rate
    • block–to–variable-length coding
    • universal source coding: Elias–Willems coding
  • Typicality and AEP
  • Data transmission over a noisy channel:
    • discrete memoryless channel
    • channel coding theorem: achievability and converse
    • computing capacity: weakly and strongly symmetric channels
    • Karush–Kuhn–Tucker conditions
  • Joint source and channel coding
  • Continuous random variables and differential entropy
  • Gaussian channel and coding theorem for Gaussian channel

Prerequisites

  • Undergraduate studies
  • Solid foundation in probability and calculus
  • Pleasure with mathematics

Instructors

Dr. Stefan M. Moser Dr. Ligong Wang
Office: ETF E104 ETF E104
Phone: 044 632 36 24 044 632 36 24
E-mail:

Teaching Assistant

Joel Neeser
Office: ETF E105
Phone: 044 632 28 99
E-mail:

Time and Place

  • Lecture: Wednesday, 14:15–16:00, ETF C1
  • Exercise: Wednesday, 16:15–18:00, ETF C1

Textbook

We use two textbooks:

  • Stefan M. Moser: Information Theory (Lecture Notes), 6th edition, Signal and Information Processing Laboratory, ETH Zürich, Switzerland, and Institute of Communications Engineering, National Yang Ming Chiao Tung University (NYCU), Hsinchu, Taiwan, 2018.
  • T.M. Cover, J.A. Thomas: Elements of Information Theory, 2nd Edition, John Wiley & Sons, Inc., New York, NY, USA. (Here is a link to an electronic copy of it.)

Note that both reference books contain more material than what we will cover in class. For the exam only the covered material (in class and exercises) are relevant, see below.

Supplementary Notes:

Exercises

Printed copies of the exercises will be handed out in class. See time table below.

Examination

Written exam (3 hours).
Material: All topics discussed in class according to the program below and the referenced chapters. Exception: The achievability proof of the rate distortion problem and strong typicality are not part of the exam.

Testat requirements

None.

Special Remarks

The lecture is held in English.

Time Table

Note that some details of the program might change in the course of the semester. Also note that some linked documents in this table can only be downloaded within ETHZ.

See also IT1, and Exercises IT1 and Handouts IT1.

References marked with IT point to Information Theory (Lecture Notes); references marked with ATIT point to Advanced Topics in Information Theory (Lecture Notes).

SW Date Topic Handouts Exercise Solutions Lecturer
1 17 Sep. Shannon's measure of information: entropy, mutual information and their main properties; ata compression: codes and trees Course Info
IT 1, 4.1–4		 Exercise 1 Solutions 1 Stefan Moser
2 24 Sep. Data compression: codes and trees, Kraft Inequality, fundamental limitations, good codes (Shannon-type codes, Fano codes), relative entropy, optimal codes (Huffman codes), McMillan Theorem IT 4 without 4.8.2, 3.1 Exercise 2 Solutions 2 Stefan Moser
3 1 Oct. Data compression: discrete memoryless source, block–to–variable-length coding, variable-length–to–block coding, Tunstall coding; stochastic processes, discrete stationary source, Markov source IT 5.1–2, 5.4–7, 6.1–2 Exercise 3 Solutions 3 Stefan Moser
4 8 Oct. Entropy rate, entropy rate of discrete stationary source (DSS) and of Markov source; Data compression: DSS and Markov source; Elias–Willems universal block–to–variable-length coding; Jensen Inequality IT 6.3, 7, 2.6 Exercise 4 Solutions 4 Stefan Moser
5 15 Oct. Optimal gambling: betting on horses; doubling rate; bookie's perspective; optimal gambling for subfair odds; Karush–Kuhn–Tucker conditions; gambling with side-information; horse races with memory IT 10.1–3, 10.5–8, 9 Exercise 5 Solutions 5 Stefan Moser
6 22 Oct. Weakly typical sets; Asymptotic Equipartition Property; high-probability sets; block–to–block source coding IT 20.1–6 Exercise 6 Solutions 6 Ligong Wang
7 29 Oct. Binary hypothesis testing; Neyman–Pearson Lemma; Chernoff–Stein Lemma, divergence typical set ATIT 7.1, 7.5 Exercise 7 Solutions 7 Ligong Wang
8 5 Nov. Log-sum inequality; convexity/concavity of relative entropy and mutual information; Data Processing Inequality for relative entropy and mutual information; channel capacity: setup and first examples ATIT 1.1, 2.4
IT 12.1–4 		Exercise 8 Solutions 8 Ligong Wang
9 12 Nov. Discrete memoryless channels; direct part of the channel coding theorem; Fano Inequality Achievability Proof
IT 11.1–3, 11.5, 11.8; 11.6 		Exercise 9 Solutions 9 Ligong Wang
10 19 Nov. Converse part of the channel coding theorem; feedback; joint source-channel coding IT 11.7, 15.1–5 (IID case only, no ergodic sources)
ATIT 17.5 		Exercise 10 Solutions 10 Ligong Wang
11 26 Nov. KKT conditions for channel capacity; rate distortion problem: setup and converse IT 12.4–5
ATIT 12.1–4 		Exercise 11 Solutions 11 Ligong Wang
12 3 Dec. Rate distortion problem: achievability ATIT 12.1–4 Exercise 12 Solutions 12 Ligong Wang
13 10 Dec. Continuous random variables and differential entropy; Gaussian channel IT 16, 17.1–3 Exercise 13 Solutions 13 Stefan Moser
14 17 Dec. Gaussian channel IT 17.1–3 Exercise 14 Solutions 14 Stefan Moser

-||-   _|_ _|_     /    __|__   Stefan M. Moser
[-]     --__|__   /__\    /__   Senior Researcher & Lecturer, ETH Zurich, Switzerland
_|_     -- --|-    _     /  /   Adj. Professor, National Yang Ming Chiao Tung University (NCTU), Taiwan
/ \     []  \|    |_|   / \/    Web: https://moser-isi.ethz.ch/


Last modified: Thu Dec 11 14:41:44 UTC 2025