Markov Chain

A natural way to deal with uncertainty is to introduce probabilistic rules. In the simplest case, we can imagine an FSM-like device having no commands but clock ticks associated with probabilities (see Figure 7.10).

 

This device starts its operation in state 1. The next clock tick switches it either to state 2 (80% chance) or to state 3 (20% chance). There is just one option to go from state 2, but state 3 has two equally probable paths.

This kind of device is known as a Markov process or Markov chain. Its primary function is to represent certain processes rather than to control something: there is no command sequence in a Markov chain model, so we can only watch it switching from state to state.

This device is suitable for modeling processes that satisfy Markov property: future evolution of a process should not depend on its history.