The declaration of the copyright is at the bottom of this page. Please, don't hesitate to contact me at if you have any questions or if you need more information.

A Student's Guide to Coding and Information Theory

This textbook is thought to be an easy-to-read introduction to coding and information theory for students at the freshman level or for non-engineering major students. The required math background is minimal: simple calculus and probability theory on high-school level should be sufficient.

Link to Cambridge University Press:

List of Typos and Corrections:

  • On p. 15, in Footnote 1: Unfortunately, the way we have written the statement, it is not true. For a finite number of events {A_l} to be independent, one needs that for any subset of events it holds that

    independence of events

    We should have focused on the case of two events only: two events A and B are independent if, and only if,

    independence of two events

  • Unfortunately, Lemma 5.22 does not hold. Instead Eq. (5.68) should read

    bound on Fano codeword length

    This then means that the derivation of the upper bound on L_av in (5.74)—(5.78) will result in the less tight bound

    bound on average Fano codeword length

    Nevertheless, the statement of (5.74) is true! It just turns out to be much harder to prove... Thus, also Theorem 5.23 does hold the way it is stated.

Information Theory

I use these lecture notes in my course Information Theory, which is a graduate course in the first year. The notes intend to be an introduction to information theory covering the following topics:

  • Definitions in information theory: entropy, mutual information, relative entropy, etc.
  • Data compression: lossless source coding including Shannon-type coding, Shannon coding, Fano coding, Huffman coding, Tunstall coding, arithmetic coding, Elias–Willems coding, and Lempel–Ziv coding.
  • Karush–Kuhn–Tucker conditions.
  • Gambling and horse betting.
  • Data transmission: coding theorem for discrete memoryless channels, computing capacity, convolutional codes, polar codes.
  • Joint source and channel coding: information transmission theorem, transmission above capacity.
  • Gaussian channel: differential entropy, channel coding theorem and joint source and channel coding theorem for the Gaussian channel, bandlimited channels, parallel Gaussian channels.
  • Asymptotic Equipartition Property and weak typicality.
  • Short introduction to cryptography.
  • Review of Gaussian random variables, vectors, and processes.

Download current version:

These notes are still undergoing corrections and improvements. If you find typos, errors, or if you have any comments about these notes, I'd be very happy to hear them! Write to . Thanks!

Advanced Topics in Information Theory (New 4th Edition!)

I use these lecture notes in my course Advanced Topics in Information Theory, which is an advanced graduate course. Based on the theory introduced in the introductory notes Information Theory, it continues to explore the most important results concerning data compression and reliable communication over a communication channel, including multiple-user communication and lossy compression schemes. The course covers the following topics:

  • Method of types.
  • Large deviation theory (Sanov's theorem, conditional limit theorem).
  • Strong typicality.
  • Hypothesis testing, Chernoff–Stein Lemma.
  • Parameter estimation, Fisher information, Cramér–Rao Bound. (New!)
  • Duality, capacity with costs, capacity with feedback.
  • Independence and causality: causal interpretations.
  • Error exponents for information transmission.
  • Context-tree weighting algorithm. (New!)
  • Rate distortion theory.
  • Error exponents in rate distortion theory.
  • Multiple description.
  • Rate distortion with side-information (Wyner–Ziv).
  • Distributed lossless data compression (Slepian–Wolf).
  • Multiple-access channel (MAC).
  • Transmission of correlated sources over a MAC.
  • Channels with noncausal side-information (Gel'fand–Pinsker).
  • Broadcast channel.
  • Multiple-access channel (MAC) with common message.
  • Discrete memoryless networks and cut-set bound.
  • Interference channel.

Download current version (New 4th Edition!):

These notes are still undergoing corrections and improvements. If you find typos, errors, or if you have any comments about these notes, I'd be very happy to hear them! Write to . Thanks!

-||-   _|_ _|_     /    __|__   Stefan M. Moser
[-]     --__|__   /__\    /__   Senior Scientist, ETH Zurich, Switzerland
_|_     -- --|-    _     /  /   Adjunct Professor, National Chiao Tung University, Taiwan
/ \     []  \|    |_|   / \/    Web: http://moser-isi.ethz.ch/

Last modified: Sat Aug 24 16:15:43 CEST 2019