【智应数】Markov chains

Markov Chain & Stationary Distribution#

Def(Finite Markov Chain). Let $\mathrm{\Omega }$ be a finite set of states, $\mathrm{\forall }x,y\in \mathrm{\Omega },P(x,y)\ge 0$ be a transition function, i.e., $\sum _{y\in \mathrm{\Omega }}P(x,y)=1.$ A finite Markov chain is a sequence of random variables

$$(X_0,X_1,X_2,...)$$

if $\mathrm{\forall }t$, for any $x_0,x_1,...,x_t\in \mathrm{\Omega }$ we have

$$\mathbb{P}(X_{t+1}=x_{t+1}\mid X_0=x_0,X_1=x_1,...,X_t=x_t)=\mathbb{P}(X_{t+1}=x_{t+1}\mid X_t=x_t)=P(x_t,x_{t+1}).$$

Given the initial distribution ${\mu }_0$ over $\mathrm{\Omega }$ and the transition matrix $P$, we can calculate the distribution after $t$ transitions

$${\mu }_t={\mu }_{t-1}P={\mu }_0{P}^t.$$

Def. A distribution $\pi$ is called stationary if $\pi P=\pi$.

Lem(Brouwer's fixed point theorem). Every continuous map $f$ from a convex, closed, bounded set to itself, has a fixed point $f(x)=x$.

Thm(Perron-Frobemins thm). Any Markov chain over finite states has a stationary distribution.

Pf. Let $|\mathrm{\Omega }|=n,{\mathrm{\Delta }}^{n-1}=\{\mu \in {\mathbb{R}}^n\mid {\mu }_i\ge 0,\sum {\mu }_i=1\}$. We can show that ${\mathrm{\Delta }}^{n-1}$ is convex, closed, bounded. Since $P$ is a linear map from ${\mathrm{\Delta }}^{n-1}$ to itself, by Brouwer's fixed point theorem there is a stationary distribution $\pi \in {\mathrm{\Delta }}^{n-1}$ s.t. $\pi P=\pi$.

Def. A finite Markov chain is irreducible (connected) if $\mathrm{\forall }x,y\in \mathrm{\Omega },\mathrm{\exists }t>0$ s.t. $P^t(x,y)>0$.

Thm(Fundamental Theorem of Markov Chains). Any connected Markov chain has a unique stationary distribution $\pi$.

Pf. Suppose $Ph=h$ for some $h\in {\mathbb{R}}^n$. Suppose $i_0=\arg \underset{i}{max}{h}_i$. Let $h(i_0)=M$. Then $\mathrm{\forall }j\in \mathrm{\Omega }$ s.t. $P(i_0,j)>0$, then $h(j)=M$. By irreducibility, $\mathrm{\forall }i,h(i)=M$. Thus $Ph=h$ implies $h=M\overrightarrow{1}$. It immediately gives $\text{rank}(P-I)=n-1$ and the theorem.

Def(Reversed Markov Chain). Given a Markov Chain $(P,\mathrm{\Omega })$. Denote $\pi$ as a stationary distribution. Let $$\hat{P}(x,y)=\frac{\pi (y)P(y,x)}{\pi (x)}.$$ Then $(\hat{P},\mathrm{\Omega })$ is the reversed Markov chain.

Def. If $\hat{P}=P$, then $(P,\mathrm{\Omega })$ is reversible.

Markov Chain Monte Carlo#

Given a distribution $\pi$ and a function $f$ over $\mathrm{\Omega }$ (usually $\mathrm{\Omega }=\{0,1...,n-1{\}}^d$), estimate

$$\underset{\pi }{\mathbb{E}}(f)=\sum _{x\in \mathrm{\Omega }}f(x)\pi (x).$$

MCMC method:

• Design $P$ s.t. $\pi P=\pi$.
• Generate $X_0,X_1,...,X_T$. $X_t$ is transferred from $X_{t-1}$ according to $P$.
• Calculate $\frac{1}{T}\sum _{t=1}^{T}f(X_T)$ as an estimate of $\mathbb{E}(f)$.

Let $\hat{\mu }=\frac{1}{T}\sum _{t=1}^T{\mu }_t$. If $\underset{t\to \mathrm{\infty }}{lim}{\mu }_t=\pi$, then $\underset{T\to \mathrm{\infty }}{lim}\underset{\hat{\mu }}{\mathbb{E}}(f)=\underset{\pi }{\mathbb{E}}(f)$, where. Since $X_t\sim {\mu }_t$, it works.

Consider a graph $G=(\mathrm{\Omega },E)$, where $E=\{(x,y)\mid P(x,y)>0\}.$

To ensure the time complexity, the degree of vertices in $G$ should be small.

To ensure the convergence rate, the diameter of $G$ should be small (or holds some other property).

For example, if $\mathrm{\Omega }=\{0,1{\}}^d$, a possible way is to let $G$ be the hypercube.

Metropolis-Hasting Algorithm

$$P(x,y)=\{\begin{array}{ll}\mathrm{\Phi }(x,y)(1\wedge \frac{\pi (y)}{\pi (x)})& x\ne y\\ 1-\sum _{z\ne x}\mathrm{\Phi }(x,z)(1\wedge \frac{\pi (z)}{\pi (x)})& x=y\end{array}$$

here $\mathrm{\Phi }$ is a symmetric transition matrix.

Lem(Detailed balance equation). For a given transition matrix $P$, if there exists a distribution $\pi$ s.t.

$$\pi (x)P(x,y)=P(y)\pi (y,x),\mathrm{\forall }x,y\in \mathrm{\Omega },$$

then $\pi$ is a stationary distribution and $P$ is reversible.

Ex. Given a graph $G=(V,E)$. Let $f:\{0,1{\}}^{|V|}\to {\mathbb{Z}}^{+}$ be the size of cut. Find $\underset{x}{max}f(x).$

Solution: Define ${\pi }_{\lambda }(x)={\lambda }^{f(x)}$. Applying MH algorithm with $\mathrm{\Phi }(x,y)=[\sum _i[x(i)\ne y(i)]=1]$. Increase $\lambda$ from $1$ to $\mathrm{\infty }$.(Simulated Annealing.)

Gibbs Sampling

$$P(x,y)=\{\begin{array}{ll}\frac{1}{d}\frac{\pi (y)}{\sum _a\pi (x_1,...,x_{i-1},a,x_{i+1},...,x_n)}& \{j\mid x_j\ne y_j\}=\{i\}\\ 0& otherwise\end{array}$$

Intuitively, choose a variables $x_i$ randomly and update it according to $\pi$. A alternative scheme is choose $x_i$ by sequentially scanning from $x_1$ to $x_d$.

Mixing time#

Def. The total variation distance between two distributions $\mu ,v$ is

$$d_{TV}(\mu ,v)=\underset{A\subseteq \mathrm{\Omega }}{max}|\mu (A)-v(A)|.$$

Prop. $d_{TV}(\mu ,v)=\frac{1}{2}\sum _{x\in \mathrm{\Omega }}|{\mu }_x-v_x|.$

Let $d(t)=\underset{x\in \mathrm{\Omega }}{max}\Vert P^t(x,\cdot )-\pi {\Vert }_{TV},\bar{d}(t)=\underset{x,y\in \mathrm{\Omega }}{max}\Vert P^t(x,\cdot )-P^t(y,\cdot ){\Vert }_{TV}.$

Lem 1. $d(t)\le \bar{d}(t)\le 2d(t)$.

Lem 2. $\bar{d}(s+t)\le \bar{d}(s)\bar{d}(t).$

Pf. $$P^{s+t}(x,w)=\sum _{z\in \mathrm{\Omega }}{P}^s(x,z)P^t(z,w)={\mathbb{E}}_{X_s}(P^t(X_s,w)),$$ where $X_s\sim P^s(x,\cdot )$.

$$\Vert P^{s+t}(x,\cdot )-P^{s+t}(y,\cdot ){\Vert }_{TV}$$

$$=\frac{1}{2}\sum _w|{\mathbb{E}}_{X_s}(P^t(X_s,w))-{\mathbb{E}}_{Y_s}(P^t(Y_s,w))|$$

$$\le \mathbb{P}(X_s\ne Y_s)\bar{d}(t)\le \bar{d}(s)\bar{d}(t).$$

Cor. For any positive integer $c$,

$$d(ct)\le \bar{d}(ct)\le \bar{d}(t{)}^c\le (2d(t){)}^c.$$

Def. The mixing time of a Markov chain is

$$t_{mix}(\epsilon )=min\{t:d(t)\le \epsilon \}.$$

$$t_{mix}=t_{mix}(\frac{1}{4}),t_{mix}(\epsilon )\le \ln \frac{1}{\epsilon }{t}_{mix}.$$

$$t_{ave}(\epsilon )=min\{t:max\Vert a_t-\pi {\Vert }_1\le \epsilon \}.$$

remark: $t_{ave}$ exists without the assumption that MC is aperiodic. For aperiodic chains, $t_{mix}(\epsilon )<t_{ave}(\epsilon )$

Random Walks on Undirected Graphs

Given an edge-weighted undirected graph, let $w_{x,y}$ denote the weight of the edge between nodes $x$ and $y$. Let $w_x=\sum _y{w}_{x,y}$.

Let $\mathrm{\Omega }=V,P_{x,y}=\frac{w_{x,y}}{w_x}$. One can check that $(\mathrm{\Omega },P)$ is reversible and $\pi (x)\propto w_x$ is the stationary distribution.

Thm. If Markov chain is reversible, finite, aperiodic, then for ${\pi }_{\ast }=\underset{x\in \mathrm{\Omega }}{min}\pi (x)>0$,

$$\frac{1-\delta }{\delta }\log \left(\frac{1}{2\epsilon }\right)\le t_{mix}(\epsilon )\le \frac{1}{2\delta }\log \left(\frac{1}{{\pi }_{\ast }\epsilon }\right),$$

where $\delta ={\lambda }_1-{\lambda }_2$ is the eigen gap of $P$.

Takeaway: $\frac{1}{\delta }$ reflects "conductance".

Def 4.2. For a subset $S$ of vertices, the normalized conductance of $S$ is

$$\mathrm{\Phi }(S)=\frac{\sum _{x\in S,y\in \bar{S}}{\pi }_x{p}_{x,y}}{min(\pi (S),\pi (\bar{S}))}.$$

Def 4.3. The normalized conductance of a Markov chain is

$$\mathrm{\Phi }=\underset{S\subset \mathrm{\Omega },S\ne \mathrm{\varnothing }}{min}\mathrm{\Phi }(S)$$

Thm(Cheeger's Inequality). Let $\delta$ be the eigen gap of $P$,

$$\frac{\delta }{2}\le \mathrm{\Phi }\le \sqrt{2\delta }.$$

Combined with the previous theorem,

$$\frac{c}{\mathrm{\Phi }}\log \frac{1}{\epsilon }\le t_{mix}(\epsilon )\le \frac{c}{{\mathrm{\Phi }}^2}\log \frac{1}{{\pi }_{\ast }\epsilon }$$

Thm 4.5.$$t_{ave}(\epsilon )=O\left(\frac{\ln (1/{\pi }_{\ast })}{{\varphi }^2{\epsilon }^3}\right).$$

Ex(1-D lattice). Let $G$ be the ring with $n$ vertices, i.e., $i$ is neighbor to $i-1,i+1$ and $P_{x,y}=\frac{1}{2}$ for all the edges.

Since it's periodic, we can make the process aperiodic by being "lazy": $P^{\prime }(x,y)=\{\begin{array}{ll}\frac{1}{2}P(x,y)& x\ne y\\ \frac{1}{2}& x=y.\end{array}$. For constant $\epsilon$, $t_{mix}$ of the lazy process $\ge$ $c{t}_{ave}$ of the original process.

Let $S_{\ast }$ be half of the ring, $\varphi =\varphi (S_{\ast })=\frac{2}{n}.$ Thus $t_{ave}=O(n^2)$.

Ex(2-D lattice). Let $G$ be the cyclic chessboard of size $n\times n$, $P_{x,y}=\frac{1}{4}$ for all the edges.

$\mathrm{\Phi }(S)=\frac{\mathrm{\#}(\text{across edges})/4n^2}{|S|/n^2}\ge \mathrm{\Omega }(\frac{\sqrt{S}/4n^2}{|S|/n^2})\ge \mathrm{\Omega }(\frac{1}{n}).$ Thus $t_{mix}=O(n^2)$.

Remark: 2-D lattice mixes faster than 1-D lattice and is better connected.

Ex(clique). $t_{mix}=\mathrm{\Omega }(1)$.

Coupling

Def. A coupling of Markov chain with transition $P$ are $\{X_1,...,X_T\},\{Y_1,...,Y_T\}$ s.t.

• $X_{t+1}\sim P(X_t,\cdot ),Y_{t+1}\sim P(Y_t,\cdot ).$
• If $X_s=Y_s$ then $X_t=Y_t$ for $\mathrm{\forall }t\ge s$.

Thm. Assume $X_0=x,Y_0=y$. Let ${\tau }_{couple}=min\{t:X_t=Y_t\}$,

$$\Vert P^t(x,\cdot )-P^t(y,\cdot ){\Vert }_{TV}\le \mathbb{P}(X_t\ne Y_t)={\mathbb{P}}_{(x,y)}({\tau }_{couple}>t).$$

Remark: $d(t)\le \underset{x,y\in \mathrm{\Omega }}{max}{\mathbb{P}}_{(x,y)}({\tau }_{couple}>t).$

Ex(1-D lattice). $P(x,y)=\{\begin{array}{ll}\frac{1}{2}& y=x\\ \frac{1}{4}& |y-x|=1\end{array}.$

The coupling constructed: $X_t$ moves and $Y_t$ stays w.p. $\frac{1}{2}$，$Y_t$ moves and $X_t$ stays w.p. $\frac{1}{2}$. Then $\mathbb{E}(\tau )=k(n-k)$, where $k=|x-y|$. Thus $d(t)\le \frac{n^2}{4t}$.

Ex(hypercube). $\mathrm{\Omega }=\{0,1{\}}^n,|\mathrm{\Omega }|=2^n,P(x,y)=\{\begin{array}{ll}\frac{1}{2}& y=x\\ \frac{1}{2n}& \Vert y-x\Vert =1\end{array}.$
The coupling constructed: Pick a random coordinate $i\in [n]$, generate $b_t\in \{0,1\}$ and let $X_{t+1}^{(i)}=Y_{t+1}^{(i)}=b_t$. Then $\mathbb{E}(\tau )=\mathbb{E}(\text{all coordinates are selected at least once})=n\ln n.$