
My ITTextbooks
The declaration of the copyright is at the bottom of this page. Please, don't hesitate to contact me at if you have any questions or if you need more information.
I use these lecture notes in my course Information Theory, which is a graduate course in the first year. The notes intend to be an introduction to information theory covering the following topics:
 Informationtheoretic quantities for discrete random variables: entropy, mutual information, relative entropy, variational distance, entropy rate.
 Data compression: coding theorem for discrete memoryless source, discrete stationary source, Markov source.
 Lossless source coding: Shannontype coding, Shannon coding, Fano coding, Huffman coding, Tunstall coding, arithmetic coding, Elias–Willems coding, Lempel–Ziv coding, and informationlossless finite state coding.
 Karush–Kuhn–Tucker conditions.
 Gambling and horse betting.
 Data transmission: coding theorem for discrete memoryless channels, computing capacity.
 Channel coding: convolutional codes, polar codes.
 Joint source and channel coding: information transmission theorem, transmission above capacity.
 Informationtheoretic quantities for continuous random variables: differential entropy, mutual information, relative entropy.
 Gaussian channel: channel coding theorem and joint source and channel coding theorem, averagepower constraint.
 Bandlimited channels, parallel Gaussian channels.
 Asymptotic Equipartition Property (AEP) and weak typicality.
 Short introduction to cryptography.
 Review of Gaussian random variables, vectors, and processes.
Download current version: (new version 6.9)
 Stefan M. Moser: “Information Theory (Lecture Notes)” (version 6.9 from 10 December 2020, PDF), 6th edition, Signal and Information Processing Laboratory, ETH Zürich, Switzerland, and Institute of Communications Engineering, National Chiao Tung University (NCTU), Hsinchu, Taiwan, 2018.
 Teacher's material: all figures and tables (PDF).
To link to the most current version of these notes, use
https://moserisi.ethz.ch/cgibin/request_script.cgi?script=it
These notes are still undergoing corrections and improvements. If you find typos, errors, or if you have any comments about these notes, I'd be very happy to hear them! Write to . Thanks!
I use these lecture notes in my course Advanced Topics in Information Theory, which is an advanced graduate course. Based on the theory introduced in the introductory notes Information Theory, it continues to explore the most important results concerning data compression and reliable communication over a communication channel, including multipleuser communication and lossy compression schemes. The course covers the following topics:
 Method of types.
 Large deviation theory (Sanov's theorem, conditional limit theorem).
 Strong typicality.
 Hypothesis testing, Chernoff–Stein Lemma.
 Parameter estimation, Fisher information, Cramér–Rao Bound. (New!)
 Guessing random variables. (New!)
 Duality, capacity with costs, capacity with feedback.
 Independence and causality: causal interpretations.
 Error exponents for information transmission.
 Contexttree weighting algorithm. (New!)
 Rate distortion theory.
 Error exponents in rate distortion theory.
 Multiple description.
 Rate distortion with sideinformation (Wyner–Ziv).
 Distributed lossless data compression (Slepian–Wolf).
 Multipleaccess channel (MAC).
 Transmission of correlated sources over a MAC.
 Channels with noncausal sideinformation (Gel'fand–Pinsker).
 Broadcast channel.
 Multipleaccess channel (MAC) with common message.
 Discrete memoryless networks and cutset bound.
 Interference channel.
Download current version (new version 4.5):
 Stefan M. Moser: “Advanced Topics in Information Theory (Lecture Notes)” (version 4.5 from 3 December 2020, PDF), 4th edition, Signal and Information Processing Laboratory, ETH Zürich, Switzerland, and Institute of Communications Engineering, National Chiao Tung University (NCTU), Hsinchu, Taiwan, 2019.
 Teacher's material: all figures and tables (PDF).
To link to the most current version of these notes, use
https://moserisi.ethz.ch/cgibin/request_script.cgi?script=atit
These notes are still undergoing corrections and improvements. If you find typos, errors, or if you have any comments about these notes, I'd be very happy to hear them! Write to . Thanks!
This textbook is thought to be an easytoread introduction to coding and information theory for students at the freshman level or for nonengineering major students. The required math background is minimal: simple calculus and probability theory on highschool level should be sufficient.
Link to Cambridge University Press:
List of Typos and Corrections:
 On p. 15, in Footnote 1: Unfortunately, the way we have written the statement, it is not true. For a finite number of events to be independent, one needs that for any subset of events it holds that
We should have focused on the case of two events only: two events and are independent if, and only if,
 Unfortunately, Lemma 5.22 does not hold. And it is not correct even if we replaced Eq. (5.68) by
This then means that the derivation of the upper bound on in (5.73)—(5.78) does not hold either. Nevertheless, the statement of (5.79) in Theorem 5.23 is true! It just turns out to be much more complicated to prove...
(The interested reader can find more details to this issue in Chapter 4 of my lecture notes Information Theory, although I have not included the complete derivation yet, because I have only succeeded in proving it for binary and ternary codes and not in general. I hope that I will be able to fix this one day.)
Copyright
You are welcome to use the IT and ATIT lecture notes for yourself, for teaching, or for any other noncommercial purpose. If you use extracts from these lecture notes, please make sure that their origin is shown. The author assumes no liability or responsibility for any errors or omissions.
 __ __ / ____ Stefan M. Moser
[] ____ /__\ /__ Senior Scientist, ETH Zurich, Switzerland
__   _ / / Adjunct Professor, National Chiao Tung University, Taiwan
/ \ [] \ _ / \/ Web: https://moserisi.ethz.ch/
Last modified: Thu Jan 7 08:00:40 CET 2021
