You are here

An Introduction to Markov Processes

Daniel W. Stroock
Springer Verlag
Publication Date: 
Number of Pages: 
Graduate Texts in Mathematics 230
[Reviewed by
Ita Cironic Donev
, on

An Introduction to Markov Processes came to life from a series of lecture notes written by D. W. Stroock during his teaching of stochastic processes at MIT. The book is technical but very suitable for graduate students in the sciences.

To prepare the reader better Stroock gives an overview of measure theory and probability theory in chapter 6. This “appendix” titled “Mild Measure Theory” gives the main theorems and results, from the concept of a σ-algebra to conditional probabilities and expectations.

Usually, books on stochastic processes start with some preliminary concepts of probability theory and trivial examples so that the reader can develop intuitive appreciation for the subject. Stroock starts with a chapter titled “Random Walks — A Good Place to Begin”, and indeed it is. The first chapter uses random walks to develop (or remind the reader) some basic concepts such as recurrence and transience. The thrill of the first chapter, at least it should be for students, is the presentation of some basic calculations of random walks on Zd. This enables a careful reader to really connect and remind oneself of the basic structure of Markov processes.

The heart of the book is the theory of ergodic properties of Markov processes, which is present in several chapters. The first part of Chapter 2 discusses basic results of discrete Markov chains where the second part gives results of the Doeblin theorem and discusses elementary ergodic theory of Markov chains. Stroock continues to discuss ergodicity in chapter 3. Chapter 4 goes on to discuss Markov processes in continuous time, starting with well known Poisson processes. Reversible Markov processes are explained in chapter 5.

This is not the cookbook of Markov processes that students usually desire, but rather a book that presents the theory of Markov processes in a very clear and informative way. It is designed perfectly for students as it starts very lightly and also gives a whole chapter on measure theory. It gives enough time to adjust to the subject. Proofs given in the book are detailed and easy to follow. The author does indeed make an effort to explain the material effectively and slowly; the information is not packed but rather it flows through the pages. At the beginning of each chapter/sub-chapter he develops an intuition of why he is explaining this material by discussing the importance of certain definitions and gives leads as to where we are going with certain definitions and theorems. Furthermore, at the end of each chapter there are numerous exercises, which should enable students to really get a grasp on the subject. For students who already have had some prior experience in stochastic processes in their undergraduate studies this book will bring new light of understanding.

Ita Cirovic Donev is a PhD candidate at the University of Zagreb. She hold a Masters degree in statistics from Rice University. Her main research areas are in mathematical finance more precisely statistical mehods of credit and market risk. Apart from the academic work she works closely in consulting for financial institutions.


Chapter 1 Random Walks A Good Place to Begin
1.1. Nearest Neighbor Random Walks on Z
1.1.1. Distribution at Time n
1.1.2. Passage Times via the Reflection Principle
1.1.3. Some Related Computations
1.1.4. Time of First Return
1.1.5. Passage Times via Functional Equations
1.2. Recurrence Properties of Random Walks
1.2.1. Random Walks on Zd
1.2.2. An Elementary Recurrence Criterion
1.2.3. Recurrence of Symmetric Random Walk in Z2
1.2.4. Transience in Z3
1.3. Exercises

Chapter 2 Doeblin's Theory for Markov Chains
2.1. Some Generalities
2.1.1. Existence of Markov Chains
2.1.2. Transition Probabilities & Probability Vectors
2.1.3. Transition Probabilities and Functions
2.1.4. The Markov Property
2.2. Doeblin's Theory
2.2.1. Doeblin's Basic Theorem
2.2.2. A Couple of Extensions
2.3. Elements of Ergodic Theory
2.3.1. The Mean Ergodic Theorem
2.3.2. Return Times
2.3.3. Identification of π
2.4. Exercises

Chapter 3 More about the Ergodic Theory of Markov Chains
3.1. Classification of States
3.1.1. Classification, Recurrence, and Transience
3.1.2. Criteria for Recurrence and Transience
3.1.3. Periodicity
3.2. Ergodic Theory without Doeblin
3.2.1. Convergence of Matrices
3.2.2. Abel Convergence
3.2.3. Structure of Stationary Distributions
3.2.4. A Small Improvement
3.2.5. The Mean Ergodic Theorem Again
3.2.6. A Refinement in The Aperiodic Case
3.2.7. Periodic Structure
3.3. Exercises

Chapter 4 Markov Processes in Continuous Time
4.1. Poisson Processes
4.1.1. The Simple Poisson Process
4.1.2. Compound Poisson Processes on Zd
4.2. Markov Processes with Bounded Rates
4.2.1. Basic Construction
4.2.2. The Markov Property
4.2.3. The Q-Matrix and Kolmogorov's Backward Equation
4.2.4. Kolmogorov's Forward Equation
4.2.5. Solving Kolmogorov's Equation
4.2.6. A Markov Process from its Infinitesimal Characteristics
4.3. Unbounded Rates
4.3.1. Explosion
4.3.2. Criteria for Non-explosion or Explosion
4.3.3. What to Do When Explosion Occurs
4.4. Ergodic Properties
4.4.1. Classification of States
4.4.2. Stationary Measures and Limit Theorems
4.4.3. Interpreting ^πii
4.5. Exercises

Chapter 5 Reversible Markov Processes
5.1. Reversible Markov Chains
5.1.1. Reversibility from Invariance
5.1.2. Measurements in Quadratic Mean
5.1.3. The Spectral Gap
5.1.4. Reversibility and Periodicity
5.1.5. Relation to Convergence in Variation
5.2. Dirichlet Forms and Estimation of
5.2.1. The Dirichlet Form and Poincaré's Inequality
5.2.2. Estimating β+
5.2.3. Estimating β-
5.3. Reversible Markov Processes in Continuous Time
5.3.1. Criterion for Reversibility
5.3.2. Convergence in L2(π^) for Bounded Rates
5.3.3. L2(π^)-Convergence Rate in General
5.3.4. Estimating λ
5.4. Gibbs States and Glauber Dynamics
5.4.1. Formulation
5.4.2. The Dirichlet Form
5.5. Simulated Annealing
5.5.1. The Algorithm
5.5.2. Construction of the Transition Probabilities
5.5.3. Description of the Markov Process
5.5.4. Choosing a Cooling Schedule
5.5.5. Small Improvements
5.6. Exercises

Chapter 6 Some Mild Measure Theory
6.1. A Description of Lebesgue's Measure Theory
6.1.1. Measure Spaces
6.1.2. Some Consequences of Countable Additivity
6.1.3. Generating σ-Algebras
6.1.4. Measurable Functions
6.1.5. Lebesgue Integration
6.1.6. Stability Properties of Lebesgue Integration
6.1.7. Lebesgue Integration in Countable Spaces
6.1.8. Fubini's Theorem
6.2. Modeling Probability
6.2.1. Modeling Infinitely Many Tosses of a Fair Coin
6.3. Independent Random Variables
6.3.1. Existence of Lots of Independent Random Variables
6.4. Conditional Probabilities and Expectations
6.4.1. Conditioning with Respect to Random Variables