Probability
Markov Property
Transition probabilities, steady-state convergence and the Markov property in an example discrete-time finite Markov chain
State diagram
step
speed
Steady state $\pi$
Time in state
Adjust transition probabilities
* Setting a transition probability to 0 removes that edge from the state model
Key ideas
- Markov property: the next state depends only on the current state, not the path taken to get here.
- Steady state $\pi_j$: as the walk runs, the fraction of time in each state converges to $\pi_j$, regardless of the starting state.
- Balance equations: $\pi_j = \sum_k \pi_k \cdot p_{kj}$. The flow into each state equals the flow out.
- Convergence requires an irreducible (single recurrent class) and aperiodic chain.