Continuous-Time Markov Chain

1. Definitions

Definition 1. We say the process $\{X(t),t\ge0\}$ is a continuous-time Markov chain if for all $s,t\ge0$ and nonnegative integers $i,j,x(u),0\le u<s$

\begin{aligned} (1) & P {X (t + s) & = j | X (s) = i, X (u) = x (u), 0 \leq u < s} \\ (2) & = P {X (t + s) = j | X (s) = i} \end{aligned}

$\begin{align} P\{X(t+s)&=j\ |\ X(s)=i,X(u)=x(u),0\le u<s \}\\ &= P\{X(t+s)=j|X(s)=i \} \end{align}$

If, in addition,

P {X (t + s) = j | X (s) = i}

$P\{X(t+s)=j|X(s)=i \}$

is independent of $s$ , the process is said to have statinonary or homogeneous transition probabilities.

The amount of time the process spends in a state $i$ , from the time it enters state $i$ to the time it transite into a different state, is exponentially distributed with parameter $v_i$ .

2. Birth and Death Process

Definition 2. A birth and death process is a continuous-time Markov chain with states $\{0,1,...\}$ for which transitions from state $n$ may go only to either state $n-1$ or state $n+1$ .

Let's say the transition probabilities are

P_{i, i + 1} = \frac{λ_{i}}{λ_{i} + μ_{i}} P_{i, i - 1} = \frac{μ_{i}}{λ_{i} + μ_{i}}

$P_{i,i+1}=\frac{\lambda_i}{\lambda_i +\mu_i}\\ P_{i,i-1}=\frac{\mu_i}{\lambda_i+\mu_i}$

The next state will be $i+1$ if a birth occurs before a death, and the probability that an exponential random variable with rate $\lambda_i$ wil occur earlier than an independent exponential random variable with rate $\mu_i$ is $\lambda_i/(\lambda_i+\mu_i)$ .

Example 1. (A Linear Growth Model with Immigration) A model in which

\begin{aligned} (3) & μ_{n} = n μ, & n \geq 1 \\ (4) & λ_{n} = n λ + θ, & n \geq 0 \end{aligned}

$\begin{align} &\mu_n =n\mu, &&n\ge1\\ &\lambda_n = n\lambda+\theta,&&n\ge0 \end{align}$

is called a linear growth model with immigration, let $X(t)$ denote the population size at time $t$ .

Suppose $X(0)=i$ , let $M(t)=E[X(t)]$ , we can determine $M(t)$ by deriving and solving a differential equation. Given $X(t)$ ,

\begin{aligned} (5) & M (t + h) & = E [X (t + h)] \\ (6) & = E [E [X (t + h) | X (t)]] \end{aligned}

$\begin{align} M(t+h) &= E[X(t+h)]\\ &= E[E[X(t+h)|X(t)]] \end{align}$

Consider $h$ is a sufficiently small period of time, by the properties of continuous Markov chain, we have

\begin{aligned} (7) & P {X (t + h) = X (t) + 1 | X (t)} = (X (t) λ + θ) h + o (h) \\ (8) & P {X (t + h) = X (t) - 1 | X (t)} = X (t) μ h + o (h) \\ (9) & P {X (t + h) = X (t) | X (t)} = 1 - (X (t) (λ + μ) + θ) h + o (h) \end{aligned}

$\begin{align} &P\{X(t+h) = X(t)+1 |X(t) \} = (X(t)\lambda+\theta)h+o(h)\\ &P\{X(t+h) = X(t)-1 |X(t) \} = X(t)\mu h+o(h)\\ &P\{X(t+h) = X(t) |X(t) \} = 1-(X(t)(\lambda+\mu)+\theta)h+o(h)\\ \end{align}$

Therefore,

E [X (t + h) | X (t)] = X (t) + (θ + X (t) λ - X (t) μ) h + o (h) M (t + h) = E [E [X (t + h) | X (t)]] = M (t) + (λ - μ) M (t) h + θ h + o (h)

$E[X(t+h)|X(t)] = X(t) + (\theta+X(t)\lambda - X(t)\mu)h+o(h)\\ M(t+h)=E[E[X(t+h)|X(t)]] = M(t) + (\lambda-\mu)M(t)h+\theta h+o(h)$

or, equivalently,

\frac{M (t + h) - M (t)}{h} = (λ - μ) M (t) + θ + \frac{o (h)}{h}

$\frac{M(t+h)-M(t)}h = (\lambda-\mu)M(t) + \theta +\frac{o(h)}h$

so we get the derivation of $M(t)$ :

M^{'} (t) = (λ - μ) M (t) + θ

$M'(t) = (\lambda-\mu)M(t) +\theta$

By solving the differential equation we have

M (t) = \frac{θ}{λ - μ} [e^{(λ - μ) t} - 1] + i e^{(λ - μ) t}

$M(t) = \frac\theta{\lambda-\mu}\left[e^{(\lambda-\mu)t}-1 \right]+ie^{(\lambda-\mu)t}$

Note that we have implicitly assumed that $\lambda\ne\mu$ .

Example 2. (A M/M/s Queueing Model) Suppose a service station with $s$ servers, the times between successive arrivals of customers are independent exponential random variables having mean $1/\lambda$ . Upon arrival, the customer joins the queue if there's no available server. The service time of each customer is assumed to be independent exponential random variables having mean $1/\mu$ .

This is actually a birth and death process with parameters

μ_{n} = {\begin{aligned} (10) & n μ, & 1 \leq n \leq s \\ (11) & s μ, & n > s \end{aligned} λ_{n} = λ, n \geq 0

$\mu_n =\left\{\begin{align}&n\mu,&&1\le n\le s\\&s\mu,&&n>s \end{align} \right.\\ \lambda_n = \lambda, n\ge0$

Let $T_i$ denote the time, starting from state $i$ , it takes for the process to enter state $i+1,i\ge0$ . Obviously, $E[T_0] = 1/\lambda$ . Let

I_{i} = {\begin{aligned} (12) & 1, & i f t h e f i r s t t r a n s i t i o n f r o m i i s t o i + 1 \\ (13) & 0,, & i f t h e f i r s t t r a n s i t i o n f r o m i i s t o i - 1 \end{aligned}

$I_i =\left\{\begin{align}&1,&&\mathrm{if\ the\ first\ transition\ from\ }i\mathrm{\ is\ to}\ i+1\\ &0,,&&\mathrm{if\ the\ first\ transition\ from\ }i\mathrm{\ is\ to}\ i-1 \end{align} \right.$

and note that

\begin{aligned} (14) & E [T_{i} | I_{i} = 1] = \frac{1}{λ_{i} + μ_{i}} \\ (15) & E [T_{i} | I_{i} = 0] = \frac{1}{λ_{i} + μ_{i}} + E [T_{i - 1}] + E [T_{i}] \end{aligned}

$\begin{align} &E[T_i|I_i=1] = \frac1{\lambda_i+\mu_i}\\ &E[T_i|I_i=0] = \frac1{\lambda_i+\mu_i} +E[T_{i-1}] + E[T_i] \end{align}$

That is, if the first transition from $i$ is to $i+1$ , then no additional time is needed, the time it occurs is exponential with rate $\lambda_i+\mu_i$ . If the first transition from $i$ is to $i-1$ , the time also has expectation of $1/(\lambda_i+\mu_i)$ , but it takes additional time to go back to $i$ , that is $E[T_{i-1}]$ , and additional time to go to $i+1$ , which is $E[X_{i+1}]$ .

Hence, since the probability that the first transition is a birth is $\lambda_i/(\lambda_i+\mu_i)$ , we see that

E [T_{i}] = \frac{1}{λ_{i} + μ_{i}} + \frac{μ_{i}}{λ_{i} + μ_{i}} (E [T_{i - 1}] + E [T_{i}])

$E[T_i] =\frac1{\lambda_i+\mu_i}+\frac{\mu_i}{\lambda_i+\mu_i}(E[T_{i-1}]+E[T_i])$

or, equivalently,

E [T_{i}] = \frac{1}{λ_{i}} + \frac{μ_{i}}{λ_{i}} E [T_{i - 1}], i \geq 1

$E[T_i] = \frac1{\lambda_i} +\frac{\mu_i}{\lambda_i}E[T_{i-1}],i\ge1$

Starting with $E[T_0] =1/\lambda$ , we can recursively calculate all $E[T_i]$ .

Example 3. For the birth and death process with parameters $\lambda_i\equiv\lambda,\mu_i\equiv\mu$ .

Using the conclusion from Example 2, we have

E [T_{0}] = \frac{1}{λ} E [T_{i}] = \frac{1}{λ} + \frac{μ}{λ} E [T_{i - 1}]

$E[T_0] = \frac1\lambda\\ E[T_i] = \frac1\lambda+\frac\mu\lambda E[T_{i-1}]$

\begin{aligned} (16) & E [T_{i}] = \frac{1 - (μ / λ)^{i + 1}}{λ - μ} \end{aligned}

$\begin{align} E[T_i] = \frac{1-(\mu/\lambda)^{i+1}}{\lambda-\mu} \end{align}$

3. The Transition Probability Function $P_{ij}(t)$

Let

P_{i j} (t) = P {X (t + s) = j | X (s) = i}

$P_{ij}(t) = P\{X(t+s)=j|X(s)=i \}$

denote the probability that a process presently in state $i$ will be in state $j$ in a time $t$ later.

Let

q_{i j} = v_{i} P_{i j}

$q_{ij} = v_iP_{ij}$

where $v_i$ is rate at which the process makes a transition when in state $i$ . We have

\begin{aligned} (17) & lim_{h \to 0} \frac{1 - P_{i i} (h)}{h} = v_{i} \\ (18) & lim_{h \to 0} \frac{P_{i j} (h)}{h} = q_{i j}, w h e n i \neq j \end{aligned}

$\begin{align} &\lim_{h\rightarrow0}\frac{1-P_{ii}(h)}h=v_i\\ &\lim_{h\rightarrow0}\frac{P_{ij}(h)}h=q_{ij},\quad \mathrm{when}\ i\ne j \end{align}$

Chapman-Kolmogorov Equation For all $s\ge0,t\ge0$ ,

P_{i j} (t + s) = \sum_{k = 0}^{\infty} P_{i k} (t) P_{k j} (s)

$P_{ij}(t+s) = \sum_{k=0}^\infin P_{ik}(t)P_{kj}(s)$

From the equation, we obtain

P_{i j} (h + t) - P_{i j} (t) = \sum_{k \neq i} P_{i k} (h) P_{k j} (t) - P_{i j} (t) (1 - P_{i i} (h))

$P_{ij}(h+t)-P_{ij}(t) = \sum_{k\ne i}P_{ik}(h)P_{kj}(t) -P_{ij}(t)(1-P_{ii}(h))$

and thus

lim_{h \to 0} \frac{P_{i j} (h + t) - P_{i j} (t)}{h} = \sum_{k \neq i} q_{i k} P_{k j} (t) - v_{i} P_{i j} (t)

$\lim_{h\rarr 0}\frac{P_{ij}(h+t)-P_{ij}(t)}h=\sum_{k\ne i}q_{ik}P_{kj}(t) - v_iP_{ij}(t)$

Hence, we have the following theorem.

Kolmogorv's Backward Equations For all states $i,j$ , and times $t\ge0$

P_{i j}^{'} (t) = \sum_{k \neq i} q_{i k} P_{k j} (t) - v_{i} P_{i j} (t)

$P_{ij}'(t)=\sum_{k\ne i}q_{ik}P_{kj}(t) - v_iP_{ij}(t)$

Example 4. (A Continuous-Time Markov Chain Consisting of Two States) Consider a machine can work an exponential amount of time having mean $1/\lambda$ before breaking down; and suppose it takes an exponential amount of time having mean $1/\mu$ to repair it. If the machine is working at time 0, what is the possibility that it will be working at time $t$ ?

Using the Kolmogorov's backward equations, we have

\begin{matrix} (3.1) & \begin{aligned} (19) & P_{00}^{'} (t) = λ P_{10} (t) - λ P_{00} (t) \\ (20) & P_{10}^{'} (t) = μ P_{00} (t) - μ P_{10} (t) \end{aligned} \end{matrix}

$\begin{align} P_{00}'(t) = \lambda P_{10}(t) - \lambda P_{00}(t)\\ P_{10}'(t) = \mu P_{00}(t) - \mu P_{10}(t) \end{align}\tag{3.1}$

Obviously, the above two equations are equivalent to

μ P_{00}^{'} (t) + λ P_{10}^{'} (t) = 0

$\mu P'_{00}(t) + \lambda P'_{10}(t) = 0$

Integrating on both sides, we have

μ P_{00} (t) + λ P_{10} (t) = c

$\mu P_{00}(t) +\lambda P_{10}(t) = c$

As $P_{00}(0)=1,P_{10}(0)=0$ , then $c = \mu$ . By replacing $P_{10}(t)$ with the first equation in (3.1), we obtain a differential eqaution

P_{00}^{'} (t) + (λ + μ) P_{00} (t) = μ

$P'_{00}(t) + (\lambda+\mu)P_{00}(t) = \mu$

By solving the differential equation with condition $P_{00}(0)=1$ , we have

P_{00} (t) = \frac{λ}{λ + μ} e^{- (λ + μ) t} + \frac{μ}{λ + μ}

$P_{00}(t) = \frac\lambda{\lambda+\mu}e^{-(\lambda+\mu)t}+\frac\mu{\lambda+\mu}$

Using the Chapman-Kolmogorov equations, we have

\begin{aligned} (21) & P_{i j} (t + h) - P_{i j} (t) & = \sum_{k = 0}^{\infty} P_{i k} (t) P_{k j} (h) - P_{i j} (t) \\ (22) & = \sum_{k \neq j} P_{i k} (t) P_{k j} (h) - [1 - P_{j j} (h)] P_{i j} (t) \end{aligned}

$\begin{align} P_{ij}(t+h) -P_{ij}(t) &= \sum_{k = 0}^\infin P_{ik}(t)P_{kj}(h)-P_{ij}(t)\\ &= \sum_{k\ne j}P_{ik}(t)P_{kj}(h) - [1-P_{jj}(h)]P_{ij}(t) \end{align}$

and thus we obtain the

Kolmogorov's Forward Equations For all states $i,j$ , and times $t\ge0$

P_{i j}^{'} (t) = lim_{h \to 0} \frac{P_{i j} (t + h) - P_{i j} (t)}{h} = \sum_{k \neq j} q_{k j} P_{i k} (t) - v_{j} P_{i j} (t)

$P'_{ij}(t) = \lim_{h\rarr 0}\frac{P_{ij}(t+h)-P_{ij}(t)}h = \sum_{k\ne j}q_{kj}P_{ik}(t) - v_jP_{ij}(t)$

Proposition 1. For a pure birth process

\begin{aligned} (23) & P_{i i} (t) = e^{- λ_{i} t}, & i \geq 0 \\ (24) & P_{i j} (t) = λ_{j - 1} e^{- λ_{j} t} \int_{0}^{t} e^{λ_{j} s} P_{i, j - 1} (s) d s, & j \geq i + 1 \end{aligned}

$\begin{align} &P_{ii}(t) = e^{-\lambda_i t},&&i\ge0\\ &P_{ij}(t) = \lambda_{j-1}e^{-\lambda_jt}\int_0^te^{\lambda_js}P_{i,j-1}(s)\ \mathrm ds,&&j\ge i+1 \end{align}$

Proof For a pure birth process, using the Kolmogorov's forward equation we have

\begin{aligned} (25) & P_{i i}^{'} (t) = - λ_{i} P_{i i} (t), & i \geq 0 \\ (26) & P_{i j}^{'} (t) = λ_{j - 1} P_{i, j - 1} (t) - λ_{i} P_{i j} (t), & j \geq i + 1 \end{aligned}

$\begin{align} &P'_{ii}(t) = -\lambda_iP_{ii}(t),&&i\ge0 \\ &P'_{ij}(t)=\lambda_{j-1}P_{i,j-1}(t)-\lambda_iP_{ij}(t),&&j\ge i+1 \end{align}$

The first equation is quite obvious, for the second equation we have

P_{i j}^{'} (t) + λ_{j} P_{i j} (t) = λ_{j - 1} P_{i, j - 1} (t)

$P'_{ij}(t) + \lambda_jP_{ij}(t) = \lambda_{j-1}P_{i,j-1}(t)$

or, equivalently,

\frac{d}{d t} [e^{λ_{j} t} P_{i j} (t)] = λ_{j - 1} e^{λ_{j} t} P_{i, j - 1} (t)

$\frac{\mathrm{d}}{\mathrm{d}t}\left[e^{\lambda_j t}P_{ij}(t) \right] = \lambda_{j-1}e^{\lambda_jt}P_{i,j-1}(t)$

Hence, since $P_{ij}(0)=0$ , we obtain the desired results.

4. Limiting Probabilities

The probability that a continuous-time Markov chain will be in state $j$ at time $t$ often converges to a limiting value that is independent of the initial state. That is, if we call this value $P_j$ , then

P_{j} \equiv lim_{t \to \infty} P_{i j} (t)

$P_j\equiv\lim_{t\rarr\infin}P_{ij}(t)$

where we are assuming that the limit exists and is independent of the initial state $i$ .

Consider the forward equaitons

P_{i j}^{'} (t) = \sum_{k \neq j} q_{k j} P_{i k} (t) - v_{j} P_{i j} (t)

$P'_{ij}(t) = \sum_{k\ne j}q_{kj}P_{ik}(t)-v_jP_{ij}(t)$

Now, if we let $t\rarr\infin$ , then assuming that we can interchange limit and summation, we obtain

\begin{aligned} (27) & lim_{t \to \infty} P_{i j}^{'} (t) & = lim_{t \to \infty} [\sum_{k \neq j} q_{k j} P_{i k} (t) - v_{j} P_{i j} (t)] \\ (28) & = \sum_{k \neq j} q_{k j} P_{k} - v_{j} P_{j} \end{aligned}

$\begin{align} \lim_{t\rarr\infin}P'_{ij}(t) &= \lim_{t\rarr\infin}\left[ \sum_{k\ne j}q_{kj}P_{ik}(t)-v_jP_{ij}(t) \right]\\ &= \sum_{k\ne j}q_{kj}P_k-v_jP_j \end{align}$

As $P_{ij}(t)$ is a bounded function, if $P'_{ij}(t)$ converges, then it must converges to $0$ , hence we have

\begin{matrix} (4.1) & \begin{aligned} (29) & v_{j} P_{j} = \sum_{k \neq j} q_{k j} P_{k}, f o r a l l s t a t e s j \\ (30) & \sum_{j} P_{j} = 1 \end{aligned} \end{matrix}

$\begin{align} &v_jP_j =\sum_{k\ne j}q_{kj}P_k,\quad \mathrm{for\ all\ states\ }j\\ &\sum_jP_j=1 \end{align}\tag{4.1}$

Thus we can solve the limiting probabilities.

Sufficient Conditions for the Existence of Limiting Probabilities:

all states of the Markov chain communicate in the sense that starting in state $i$ there is a positive probability of ever being in state $j$ , for all $i,j$ and
the Markov chain is positive recurrent in the sense that, starting in any state, the mean time to return to the state is finite.

If the above two conditions hold, then the limiting probabilities will exist and satisfy Equaitons (4.1). In addition, $P_j$ also will have the interpretation of being the long-run proportion of time that the process is in state $j$ .

5. Time Reversibility

Suppose a continuous-time Markov chain with limiting probabilities existing, if we consider the sequence of states visited in the continuous-time Markov chain process, ignoring the amount of time spent in each state during a visit, then the sequence consitutes a discrete-time Markov chain with transition probabilities $P_{ij}$ . We call such a discrete-time Markov chain as the embedded chain. Its limiting probabilities exists and we have talked about them before

\begin{aligned} (31) & π_{j} = \sum_{j} π_{i} P_{i j} \\ (32) & \sum_{i} π_{i} = 1 \end{aligned}

$\begin{align} &\pi_j = \sum_j\pi_iP_{ij}\\ &\sum_i\pi_i=1 \end{align}$

Since $\pi_i$ represents the proportion of transitions that take the process into state $i$ , and because $1/v_i$ is the mean time spent in state $i$ during a visit, it seems intuitive that $P_i$ , the proportion of time in state $i$ , should be a weighted average of the $\pi_i$ where $\pi_i$ is weighted proportionately to $1/v_i$ . That is

P_{i} = \frac{π_{i} / v_{i}}{\sum_{j} π_{j} / v_{j}}

$P_i =\frac{\pi_i/v_i}{\sum_j\pi_j/v_j}$

Suppose now a continuous-time Markov chain has been in operation of a long time, and suppose starting at some large time $T$ , we trace back the process. Suppose the process is in state $i$ in some large time $t$ , the probability that the process has been in this state for an amount of time greater that $s$ is $e^{-v_is}$ . That is,

P = \frac{P {X (t - s) = i} e^{- v_{i} s}}{P {X (t) = i}} = e^{- v_{i} s}

$P = \frac{P\{X(t-s)=i\}e^{-v_is}}{P\{X(t)=i\}}=e^{-v_is}$

Thus, the sequence of states visited by the reversed process consititues a discrete-time Markov chain with transition probabilities $Q_{ij}$ given by

Q_{i j} = \frac{π_{j} P_{i j}}{π_{i}}

$Q_{ij}=\frac{\pi_jP_{ij}}{\pi_i}$

Therefore, a continuous-time Markov chain will be time reversible, if the embedded chain is time reversible. That is, if

π_{i} P_{i j} = π_{j} P_{j i}

$\pi_iP_{ij}=\pi_jP_{ji}$

Using the fact that $P_i=\displaystyle{\frac{\pi_i/v_i}{\sum_j\pi_jv_j}}$ , we see that the preceding is equivalent to

P_{i} q_{i j} = P_{j} q_{j i}, f o r a l l i, j

$P_iq_{ij} = P_jq_{ji},\quad \mathrm{for\ all\ }i,j$

Since $P_i$ is the proportion of time in state $i$ and $q_{ij}$ is the rate when in state $i$ that goes to state $j$ , the condition of time reversibility is that the rate at which the process goes to directly from state $i$ to state $j$ is equal to the rate at which it goes directly from $j$ to $i$ .

Proposition 5.1 If for some set $\{P_i\}$

\sum_{i} P_{i} = 1

$\sum_iP_i=1$

and

P_{i} q_{i j} = P_{j} q_{j i}, f o r a l l i \neq j

$P_iq_{ij}=P_jq_{ji},\quad\mathrm{for\ all\ }i\ne j$

then the continuous-time Markov chain is time reversible and $P_i$ represents the limiting probability of being in state $i$ .

Proposition 5.2 A time reversible chain with limiting probabilities $P_j,j\in S$ that is truncated to the set $A\subset S$ and remains irreducible is also time reversible and has limiting probabilities $P_j^A$ given by

P_{j}^{A} = \frac{P_{j}}{\sum_{i \subset A} P_{i}}, j \in A

$P_j^A=\frac{P_j}{\sum_{i\subset A}P_i},\quad j\in A$

Proposition 5.3 If $\{X_i(t),t\ge0\}$ are independent time reversible continuous-time Markov chains, then the vector process $\{(X_1(t),...,X_n(t) ,t\ge0\}$ is also a time-reversible continuous-time Markov chain.

6. Uniformization

Consider a continuous-time Markov chain in which the mean time spent in a state is the same for all states. That is, suppose that $v_i=v$ , for all state $i$ . Let $N(t)$ denote the number of transitions by time $t$ , then $\{N(t),t\ge0\}$ will be a Poisson process with rate $v$ .

To compute the transition probabilities $P_{ij}(t)$ , we can condition on $N(t)$ :

\begin{aligned} (33) & P_{i j} (t) & = P {X (t) = j | X (0) = i} \\ (34) & = \sum_{n = 0}^{\infty} P {X (t) = j | X (0) = i, N (t) = n} P {N (t) = n | X (t) = j} \\ (35) & = \sum_{n = 0}^{\infty} P {X (t) = j | X (0) = i, N (t) = n} e^{- v t} \frac{(v t)^{n}}{n!} \end{aligned}

$\begin{align} P_{ij}(t) &= P\{X(t)=j|X(0)=i \}\\ &= \sum_{n=0}^\infin P\{X(t)=j|X(0)=i,N(t)=n \}P\{N(t)=n|X(t)=j \} \\ &= \sum_{n=0}^\infin P\{X(t)=j|X(0)=i,N(t)=n \}e^{-vt}\frac{(vt)^n}{n!} \end{align}$

Since the distribution of time spent in each state is the same, given that $P\{N(t)=n\}$ gives us no information about which states were visited. Hence,

P {X (t) = j | X (0) = i, N (t) = n} = P_{i j}^{n}

$P\{X(t)=j|X(0)=i,N(t)=n \}=P_{ij}^n$

where $P_{ij}^n$ is the $n$ -stage transition probabilities associted with the discrete-time Markov chain with transition probabilities $P_{ij}$ ; and so when $v_i\equiv v$

P_{i j} (t) = \sum_{n = 0}^{\infty} P_{i j}^{n} e^{- v t} \frac{(v t)^{n}}{n!}

$P_{ij}(t)=\sum_{n=0}^\infin P_{ij}^n e^{-vt}\frac{(vt)^n}{n!}$

The assumption $v_i\equiv v$ is quite limited in practice, but suppose we make a trick of allowing fictitious transitions from a state to itself, then most Markov chains can be put in that form. Let $v$ satisfying

v_{i} \leq v, f o r a l l i

$v_i\le v,\quad \mathrm{for\ all\ }i$

We can say any Markov chain satisfying the above condition can be thought of as being a process that spends an exponential amount of time with rate $v$ in state $i$ and then makes a transition to $j$ with transition probability $P^*_{ij}$ , where

P_{i j}^{*} = {\begin{aligned} (36) & \frac{v_{i}}{v} P_{i j}, & j \neq i \\ (37) & 1 - \frac{v_{i}}{v}, & j = i \end{aligned}

$P^*_{ij}=\left\{ \begin{align} &\frac{v_i}vP_{ij},&&j\ne i\\ &1-\frac{v_i}v,&&j=i \end{align} \right.$

Hence we can have the transition probabilities computed by

P_{i j} (t) = \sum_{n = 0}^{\infty} P_{i j}^{* n} e^{- v t} \frac{(v t)^{n}}{n!}

$P_{ij}(t)=\sum_{n=0}^\infin P_{ij}^{*n} e^{-vt}\frac{(vt)^n}{n!}$

7. Computing the Transiition Probabilities

For any pair of states $i$ and $j$ , let

r_{i j} = {\begin{aligned} (38) & q_{i j}, & i \neq j \\ (39) & - v_{i}, & i = j \end{aligned}

$r_{ij} = \left\{\begin{align}&q_{ij},&&i\ne j\\&-v_i,&&i=j \end{align} \right.$

So we can rewrite the Kolmogorov's forward and backward equations as follows:

\begin{aligned} (40) & P_{i j}^{'} (t) = \sum_{k} r_{i k} P_{k j} (t) & (b a c k w a r d) \\ (41) & P_{i j}^{'} (t) = \sum_{k} r_{k j} P_{i k} (t) & (f o r w a r d) \end{aligned}

$\begin{align} &P'_{ij}(t) = \sum_kr_{ik}P_{kj}(t)&&(\mathrm{backward})\\ &P'_{ij}(t) = \sum_kr_{kj}P_{ik}(t)&&(\mathrm{forward}) \end{align}$

This representation is especially revealing when we introduce matrix notation. Define the matrices $\bold{R}$ and $\bold{P}(t), \bold{P}(t)$ by letting the elements in row $i$ , column $j$ of these matrices be, respectively, $r_{ij}, P_{ij}(t)$ and $P'_{ij}(t)$ . Thus

\begin{aligned} (42) & P^{'} (t) = R P (t), & (b a c k w a r d) \\ (43) & P^{'} (t) = P (t) R, & (f o r w a r d) \end{aligned}

$\begin{align} &\bold P'(t) = \bold R\bold P(t),&&\mathrm{(backward)}\\ &\bold P'(t) = \bold P(t)\bold R,&&\mathrm{(forward)} \end{align}$

As the solution of the scalar differential equation

f (t) = f (0) e^{c t}

$f(t) = f(0)e^{ct}$

So the forward eqaution can be written as

P (t) = P (0) e^{R t}

$\bold P(t) = \bold P(0)e^{\bold Rt}$

Since $\bold P(0)=\bold I$ , this yields that

P (t) = e^{R t}

$P(t) = e^{\bold Rt}$

where the matrix $e^{\bold Rt}$ is defined by

e^{R t} = \sum_{n = 0}^{\infty} R^{n} \frac{t^{n}}{n!}

$e^{\bold Rt} = \sum_{n=0}^\infin\bold R^n\frac{t^n}{n!}$

posted @ 2023-12-10 11:45 kaleidopink 阅读(43) 评论(0) 编辑收藏举报

刷新页面返回顶部

登录后才能查看或发表评论，立即登录或者逛逛博客园首页

相关博文：

· Markov Chains

· Poisson Process

· 【智应数】Markov chains

· Markov Chain

· 时间可逆的马氏链(Time Reversible Markov Chain)

阅读排行：
· 全程不用写代码，我用AI程序员写了一个飞机大战
· DeepSeek 开源周回顾「GitHub 热点速览」
· 记一次.NET内存居高不下排查解决与启示
· MongoDB 8.0这个新功能碉堡了，比商业数据库还牛
· .NET10 - 预览版1新功能体验（一）

公告

联系: lunar_ubuntu@qq.com

昵称： kaleidopink
园龄： 5年1个月
粉丝： 21
关注： 8

+加关注

2025年3月

日

一

二

三

四

五

六

Kaleidopink

Continuous-Time Markov Chain

1. Definitions

2. Birth and Death Process

3. The Transition Probability Function $P_{ij}(t)$

4. Limiting Probabilities

5. Time Reversibility

6. Uniformization

7. Computing the Transiition Probabilities

公告

搜索

常用链接

最新随笔

我的标签

随笔分类 (133)

随笔档案 (136)

阅读排行榜

评论排行榜

推荐排行榜

最新评论

Kaleidopink

Continuous-Time Markov Chain

1. Definitions

2. Birth and Death Process

3. The Transition Probability Function Pij(t)Pij(t)P_{ij}(t)

4. Limiting Probabilities

5. Time Reversibility

6. Uniformization

7. Computing the Transiition Probabilities

公告

搜索

常用链接

最新随笔

我的标签

随笔分类 (133)

随笔档案 (136)

阅读排行榜

评论排行榜

推荐排行榜

最新评论

3. The Transition Probability Function $P_{ij}(t)$