Back to overview over all lectures...

Advanced Topics in Information Theory
Pythagoras and Relative Entropy:
A Glimpse into Large Deviation Theory

28 April 2014


If there is something to be announce, it will be published here...

Course Abstract

The method of types is a powerful tool that has been successfully used in information theory for many years already and that has very interesting applications in advanced probability theory. In this course we will focus on one or two rather exotic such applications. We will start with a quick review of types and their properties and then explore the connection to Large Deviation Theory and in particular focus on Sanov's Theorem.

While I claim boldly that quite likely you will never actually be using Sanov’s Theorem in your future career, I still hope to motivate you to have a closer look at it. It is a gorgeous example of how an information theoretic quantity (i.e., the relative entropy) shows up unexpectedly in probability theory and gives the correct answer to a probabilistic problem. This connection then serves us as a path to a deeper understanding of the actual meaning of relative entropy. In particular, we will be discussing the famous Pythagorean Theorem in the context of relative entropies.

Depending on the time we might also have a quick look at the Conditional Limit Theorem.


  • Probability
  • Information Theory


Prof. Stefan M. Moser
Engineering Building IV, IT-Lab 716A

Time and Place

The course will take place on:

  • Monday, 28 April, 2014, 14:20–16:20 in Engineering Building IV, Room 824 (ED824)


The participants will receive a handout that is an extract of my lecture notes:


I will distribute some exercise problems for the interested student. Solutions will then be available a week later for download here.

Special Remarks

The lecture will be held in English.

-||-   _|_ _|_     /    __|__   Stefan M. Moser
[-]     --__|__   /__\    /__   Senior Researcher & Lecturer, ETH Zurich, Switzerland
_|_     -- --|-    _     /  /   Adj. Professor, National Chiao Tung University (NCTU), Taiwan
/ \     []  \|    |_|   / \/    Web:

Last modified: Mon Apr 14 15:49:01 UTC+8 2014