Back to overview over all lectures...

Advanced Topics in Information Theory
Autumn 2006/2007


News

  • Class Evaluation: The class evaluation takes place on January 3 during the normal lecture hours. I would very much appreciate your feedback, so please come to class at this date! Thanks!
  • Final Exam: The final exam will take place on
    • Monday, January 8, 15:40-18:30 (Note that this is one hour longer than usual!)
    Regulations:
    • open book: any book is allowed
    • not allowed are: any telecommunication devices like mobile phones, any laptop with wireless capabilities, any "friend", or any other help from outside...
    • covered material: everything covered in class
  • Mid-Term Exam: The mid-term exam will take place on
    • Monday, November 6, 15:40-18:30 (Note that this is one hour longer than usual!)
    Regulations:
    • open book: any book is allowed
    • not allowed are: any telecommunication devices like mobile phones, any laptop with wireless capabilities, any "friend", or any other help from outside...
    • covered material: everything covered in class until October 30 (not including error exponents)
  • Note: On October 9 there will be no lecture!

Instructor

Stefan M. Moser
Engineering Building IV, Room 727
phone: 03-571 21 21 ext. 54548
e-mail:

Teaching Assistant

Zheng Yan-Xiu
Engineering Building IV, Room 811
phone: 03-571 21 21 ext. 54571
e-mail: <non2000.cm88g@nctu.edu.tw>

Time and Place

The course is scheduled for 4 hours per week:

  • Monday, 15:40--17:30, Engineering Building IV, Room 111
  • Wednesday, 15:40--17:30, Engineering Building IV, Room 111

Course Objectives

This course is an advanced course in information theory. Based on the theory we have learned in the course Information Theory we will continue to explore the most important results concerning data compression and reliable communication over a communication channel: mainly we will concentrate on multiple-user communication and lossy compression schemes. The course will cover approximately the following topics:

  • Maximum entropy
  • Methods of types
  • Rate distortion theory (lossy compression)
  • Multiple-users channels:
    • Multiple-access channel
    • Broadcast channel
    • Relay channel
    • Interference channel
  • Gel'fand-Pinsker problem: channels with random parameters known at the transmitter
  • Correlated source encoding (Slepian-Wolf)
  • Information theory and the stock market

We hope that a student who finishes the course will be able to better understand the principles underlying all state-of-the-art communication systems and the difficulties encountered when designing and trying to improve them.

Prerequisites

  • Probability
  • Information Theory

Textbook

The Course will be mainly be based on

  • Thomas M. Cover and Joy A. Thomas: "Elements of Information Theory," Wiley, 1991.
You find here a link to an electronic version of the book.

Further references and recommended readings:

  • Robert G. Gallager: "Information Theory and Reliable Communication," Wiley, 1968.
  • Raymond W. Yeung: "A First Course in Information Theory," Kluwer Academic Publishers, 2005.
  • Imre Csiszár, János Körner: "Information Theory: Coding Theorems for Discrete Memoryless Systems", 3rd edition, Akademiai Kiado, Budapest.

Grading

The exercises are an essential part of this lecture and we will spend a considerable amount of time in discussing and solving them during class. There will be one exercise every week consisting of a about four to six problems. The time in class will not be sufficient to solve all problems, i.e., the students are asked to finish the problems at home. For the understanding of the course and also as a preparation for the mid-term and final exam we highly recommend to solve the exercises! Since the material of this course is rather demanding by itself, we have decided not to further challenge the students with additional tasks like, e.g., a presentation of a paper. We hope that the saved time will be used instead for solving the exercises, going over the notes, and reading the textbook!

Your grade will be an average of

  • your homework (15%)
  • the midterm exam (35%)
  • the final exam (50%)

The grade of your homework will not be based on the correctness of your answers, but rather the effort you show in trying to solve them. To pass the course there is the additional condition that at least 10 exercises have to be handed in.

This course is worth 3 credits.

Time Table

Date Topic Handouts Exercise (due on) Solutions Comments
11 Sept. Maximum entropy Syllabus Exercise 1 (18 Sept.)    
13 Sept. Example macro/microstates, diff. entropy rate        
18 Sept. Spectrum estimation, Burg's theorem, method of types   Exercise 2 (25 Sept.)    
20 Sept. Method of types     Solutions 1  
25 Sept. Method of types, large deviation theory (Sanov's theorem)   Exercise 3 (2 Oct.)    
27 Sept. Conditional limit theorem     Solutions 2  
2 Oct. Conditional limit theorem, strongly typical sets   Exercise 4 (11 Oct.)    
4 Oct. Strongly typical sets     Solutions 3  
9 Oct. No lecture   -----    
11 Oct. Jointly strongly typical sets   Exercise 5 (16 Oct.) Solutions 4  
16 Oct. Rate distortion theory   Exercise 6 (23 Oct.)    
18 Oct. Rate distortion theory     Solutions 5  
23 Oct. Rate distortion theory   Exercise 7 (30 Oct.)    
25 Oct. Rate distortion theory: Gaussian sources     Solutions 6  
30 Oct. Characterization of rate distortion function, error exponent for rate distortion function   Exercise 8 (13 Nov.)    
1 Nov. Error exponent for rate distortion function: type covering lemma     Solutions 7  
6 Nov. Midterm Exam   -----    
8 Nov. Error exponent for rate distortion function        
13 Nov. Multiple descriptions   Exercise 9 (20 Nov.)    
15 Nov. Multiple descriptions     Solutions 8  
20 Nov. Multiple descriptions, Wyner-Ziv problem: rate distortion with side-information   Exercise 10 (27 Nov.)    
22 Nov. Wyner-Ziv problem     Solutions 9  
27 Nov. Wyner-Ziv problem, Slepian-Wolf problem   Exercise 11 (4 Dec.)    
29 Nov. Slepian-Wolf problem     Solutions 10  
4 Dec. Slepian-Wolf problem, MAC   Exercise 12 (11 Dec.)    
6 Dec. MAC: achievability     Solutions 11  
11 Dec. MAC: converse, Gaussian MAC   Exercise 13 (18 Dec.)    
13 Dec. Gaussian MAC, transmission of correlated sources over a MAC     Solutions 12  
18 Dec. Transmission of correlated sources over a MAC, Gel'fand-Pinsker problem: channels with non-causal side-information   Exercise 14 (25 Dec.)    
20 Dec. Gel'fand-Pinsker problem     Solutions 13  
25 Dec. Converse of Gel'fand-Pinsker problem, broadcast channel   Exercise 15 (3 Jan.)    
27 Dec. Degraded broadcast channel     Solutions 14  
1 Jan. No lecture   -----    
3 Jan. Marton region of general broadcast channel   Solutions 15 Today class evaluation, please come to class!
8 Jan. Final Exam   -----    
10 Jan. Discussion final exam        

Special Remarks

The lecture will be held in English.


-||-   _|_ _|_     /    __|__   Stefan M. Moser
[-]     --__|__   /__\    /__   Senior Researcher & Lecturer, ETH Zurich, Switzerland
_|_     -- --|-    _     /  /   Adj. Professor, National Chiao Tung University (NCTU), Taiwan
/ \     []  \|    |_|   / \/    Web: http://moser-isi.ethz.ch/


Last modified: Wed May 13 06:19:18 UTC+8 2009