
Back to overview over all lectures...
Advanced Topics in Information Theory Autumn 2006/2007
News
 Class Evaluation: The class evaluation takes place on January 3 during the normal lecture hours. I would very much appreciate your feedback, so please come to class at this date! Thanks!
 Final Exam: The final exam will take place on
 Monday, January 8, 15:4018:30 (Note that this is one hour longer than usual!)
Regulations:
 open book: any book is allowed
 not allowed are: any telecommunication devices like mobile phones, any laptop with wireless capabilities, any "friend", or any other help from outside...
 covered material: everything covered in class
 MidTerm Exam: The midterm exam will take place on
 Monday, November 6, 15:4018:30 (Note that this is one hour longer than usual!)
Regulations:
 open book: any book is allowed
 not allowed are: any telecommunication devices like mobile phones, any laptop with wireless capabilities, any "friend", or any other help from outside...
 covered material: everything covered in class until October 30 (not including error exponents)
 Note: On October 9 there will be no lecture!
Instructor
Stefan M. Moser
Engineering Building IV, Room 727
phone: 03571 21 21 ext. 54548
email:
Teaching Assistant
Zheng YanXiu
Engineering Building IV, Room 811
phone: 03571 21 21 ext. 54571
email: <non2000.cm88g@nctu.edu.tw>
Time and Place
The course is scheduled for 4 hours per week:
 Monday, 15:4017:30, Engineering Building IV, Room 111
 Wednesday, 15:4017:30, Engineering Building IV, Room 111
Course Objectives
This course is an advanced course in information
theory. Based on the theory we have learned in the course Information Theory we will continue
to explore the most important results concerning data
compression and reliable communication over a communication
channel: mainly we will concentrate on multipleuser
communication and lossy compression schemes. The course will
cover approximately the following topics:
 Maximum entropy
 Methods of types
 Rate distortion theory (lossy compression)
 Multipleusers channels:
 Multipleaccess channel
 Broadcast channel
 Relay channel
 Interference channel
 Gel'fandPinsker problem: channels with random parameters known
at the transmitter
 Correlated source encoding (SlepianWolf)
 Information theory and the stock market
We hope that a student who finishes the course will be able
to better understand the principles underlying all stateoftheart
communication systems and the difficulties encountered when designing
and trying to improve them.
Prerequisites
 Probability
 Information Theory
Textbook
The Course will be mainly be based on
 Thomas M. Cover and Joy A. Thomas: "Elements of Information Theory," Wiley, 1991.
You find here a link to an electronic version of the book.
Further references and recommended readings:
 Robert G. Gallager: "Information Theory and Reliable Communication," Wiley, 1968.
 Raymond W. Yeung: "A First Course in Information Theory," Kluwer Academic Publishers, 2005.
 Imre Csiszár, János Körner: "Information Theory: Coding Theorems for Discrete Memoryless Systems", 3rd edition, Akademiai Kiado, Budapest.
Grading
The exercises are an essential part of this lecture and we will spend
a considerable amount of time in discussing and solving them during
class. There will be one exercise every week consisting of a about
four to six problems. The time in class will not be sufficient to
solve all problems, i.e., the students are asked to finish the
problems at home. For the understanding of the course and also as a
preparation for the midterm and final exam we highly recommend to
solve the exercises! Since the material of this course is rather
demanding by itself, we have decided not to further challenge the
students with additional tasks like, e.g., a presentation of a
paper. We hope that the saved time will be used instead for solving
the exercises, going over the notes, and reading the textbook!
Your grade will be an average of
 your homework (15%)
 the midterm exam (35%)
 the final exam (50%)
The grade of your homework will not be based on the correctness of
your answers, but rather the effort you show in trying to solve
them. To pass the course there is the additional condition that
at least 10 exercises have to be handed
in.
This course is worth 3 credits.
Time Table
Date 
Topic 
Handouts 
Exercise (due on) 
Solutions 
Comments 
11 Sept. 
Maximum entropy 
Syllabus 
Exercise 1 (18 Sept.) 


13 Sept. 
Example macro/microstates, diff. entropy rate 




18 Sept. 
Spectrum estimation, Burg's theorem, method of types 

Exercise 2 (25 Sept.) 


20 Sept. 
Method of types 


Solutions 1 

25 Sept. 
Method of types, large deviation theory (Sanov's theorem) 

Exercise 3 (2 Oct.) 


27 Sept. 
Conditional limit theorem 


Solutions 2 

2 Oct. 
Conditional limit theorem, strongly typical sets 

Exercise 4 (11 Oct.) 


4 Oct. 
Strongly typical sets 


Solutions 3 

9 Oct. 
No lecture 

 


11 Oct. 
Jointly strongly typical sets 

Exercise 5 (16 Oct.) 
Solutions 4 

16 Oct. 
Rate distortion theory 

Exercise 6 (23 Oct.) 


18 Oct. 
Rate distortion theory 


Solutions 5 

23 Oct. 
Rate distortion theory 

Exercise 7 (30 Oct.) 


25 Oct. 
Rate distortion theory: Gaussian sources 


Solutions 6 

30 Oct. 
Characterization of rate distortion function, error exponent for rate distortion function 

Exercise 8 (13 Nov.) 


1 Nov. 
Error exponent for rate distortion function: type covering lemma 


Solutions 7 

6 Nov. 
Midterm Exam 

 


8 Nov. 
Error exponent for rate distortion function 




13 Nov. 
Multiple descriptions 

Exercise 9 (20 Nov.) 


15 Nov. 
Multiple descriptions 


Solutions 8 

20 Nov. 
Multiple descriptions, WynerZiv problem: rate distortion with sideinformation 

Exercise 10 (27 Nov.) 


22 Nov. 
WynerZiv problem 


Solutions 9 

27 Nov. 
WynerZiv problem, SlepianWolf problem 

Exercise 11 (4 Dec.) 


29 Nov. 
SlepianWolf problem 


Solutions 10 

4 Dec. 
SlepianWolf problem, MAC 

Exercise 12 (11 Dec.) 


6 Dec. 
MAC: achievability 


Solutions 11 

11 Dec. 
MAC: converse, Gaussian MAC 

Exercise 13 (18 Dec.) 


13 Dec. 
Gaussian MAC, transmission of correlated sources over a MAC 


Solutions 12 

18 Dec. 
Transmission of correlated sources over a MAC, Gel'fandPinsker problem: channels with noncausal sideinformation 

Exercise 14 (25 Dec.) 


20 Dec. 
Gel'fandPinsker problem 


Solutions 13 

25 Dec. 
Converse of Gel'fandPinsker problem, broadcast channel 

Exercise 15 (3 Jan.) 


27 Dec. 
Degraded broadcast channel 


Solutions 14 

1 Jan. 
No lecture 

 


3 Jan. 
Marton region of general broadcast channel 


Solutions 15 
Today class evaluation, please come to class! 
8 Jan. 
Final Exam 

 


10 Jan. 
Discussion final exam 




Special Remarks
The lecture will be held in English.
 __ __ / ____ Stefan M. Moser
[] ____ /__\ /__ Senior Researcher & Lecturer, ETH Zurich, Switzerland
__   _ / / Adj. Professor, National Chiao Tung University (NCTU), Taiwan
/ \ [] \ _ / \/ Web: http://moserisi.ethz.ch/
Last modified: Wed May 13 06:19:18 UTC+8 2009
