期末复习--实用回归分析
\[y=\left[\begin{array}{c}
y_{1} \\
y_{2} \\
\vdots \\
y_{n}
\end{array}\right], X=\left[\begin{array}{cccc}
1 & x_{11} & \cdots & x_{1 p} \\
1 & x_{21} & \cdots & x_{2 p} \\
\vdots & \vdots & & \vdots \\
1 & x_{n 1} & \cdots & x_{n p}
\end{array}\right], \epsilon=\left[\begin{array}{c}
\epsilon_{1} \\
\epsilon_{2} \\
\vdots \\
\epsilon_{n}
\end{array}\right], \beta=\left[\begin{array}{c}
\beta_{0} \\
\beta_{1} \\
\vdots \\
\beta_{p}
\end{array}\right]
\]
\[\boldsymbol{y}=\boldsymbol{X} \boldsymbol{\beta}+\varepsilon
\]
\[\boldsymbol{X}=\left(\mathbf{1}, \boldsymbol{x}_{1}, \ldots, \boldsymbol{x}_{p}\right)--
n \times(p+1)
\]
\[\varepsilon=\left(\varepsilon_{1}, \ldots, \varepsilon_{n}\right)^{\prime}
\]
Gauss-Markov条件:
\[\left\{\begin{array}{l}
E\left(\varepsilon_{i}\right)=0, i=1, \ldots, n \\
\operatorname{Cov}\left(\varepsilon_{i}, \varepsilon_{j}\right)=0, i \neq j ; \quad \operatorname{Var}\left(\varepsilon_{i}\right)=\sigma^{2}
\end{array}\right.
\]
正态性假设:
\[\left\{\begin{array}{l}
\varepsilon_{i} \sim N\left(0, \sigma^{2}\right), i=1, \ldots, n \\
\varepsilon_{1}, \ldots, \varepsilon_{n} \quad \text { 相互独立 }
\end{array}\right.
\]
LSE
\[Q(\boldsymbol{\beta})=(\boldsymbol{y}-\boldsymbol{X} \boldsymbol{\beta})^{\prime}(\boldsymbol{y}-\boldsymbol{X} \boldsymbol{\beta})
\]
\[\frac{\partial Q(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}}=-\boldsymbol{X}^{\prime} 2(\boldsymbol{y}-\boldsymbol{X} \boldsymbol{\beta})=-2 \boldsymbol{X}^{\prime}(\boldsymbol{y}-\boldsymbol{X} \boldsymbol{\beta})=0
\]
\[\hat{\boldsymbol{\beta}}=\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime} \boldsymbol{y}
\]
\[\hat{\boldsymbol{y}}=\boldsymbol{X} \hat{\boldsymbol{\beta}}=\boldsymbol{X}\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime} \boldsymbol{y} \stackrel{\text {def}}{=} \boldsymbol{H} \boldsymbol{y}
\]
\(H=X(X'X)^{-1}X'\rightarrow H^2=X(X'X)^{-1}X'*X(X'X)^{-1}X'=X(X'X)^{-1}X'=H\)
\((I_n-H)^2=I_n^2-2HI_n+H^2=I_n-H\)
回归系数估计的最大似然法
\(y \sim N(X\beta,\sigma^2I_n)\)
如果正态分布假设满足，
\[\boldsymbol{y}=\boldsymbol{X} \boldsymbol{\beta}+\varepsilon, \quad \varepsilon \sim N\left(\mathbf{0}, \sigma^{2} \boldsymbol{I}_{n}\right)
\]
则 \(\boldsymbol{y}\) 的概率分布为 \(: \boldsymbol{y} \sim N\left(\boldsymbol{X} \boldsymbol{\beta}, \sigma^{2} \boldsymbol{I}_{n}\right),\) 这是似然函数为
\[L(\boldsymbol{\beta})=\left(2 \pi \sigma^{2}\right)^{-n / 2} \exp \left\{-\frac{1}{2 \sigma^{2}}(\boldsymbol{y}-\boldsymbol{X} \boldsymbol{\beta})^{\prime}(\boldsymbol{y}-\boldsymbol{X} \boldsymbol{\beta})\right\}
\]
对其做“-ln”变换，记为 \(\ell(\boldsymbol{\beta}),\) 可以证明 \(\boldsymbol{\beta}\) 的极大似然估计 \(\hat{\beta}_{M L E}\) 与最小二 乘估计等价。而误差项的方差 \(\sigma^{2}\) 的极大似然估计
\[\hat{\sigma}_{M L E}^{2}=\frac{1}{n} S S E=\frac{1}{n} \boldsymbol{e}^{\prime} \boldsymbol{e}
\]
EX:
对于多元线性回归模型: \(\mathrm{Y}=\mathrm{X} \beta+\varepsilon, \varepsilon \sim \mathrm{N}\left(0, \sigma^{2} \mathrm{I}_{n}\right)\)
(1).利用极大似然估计求出 \(\widehat{\sigma^{2}}\);
(2).求 \(\mathrm{E}\left(\widehat{\sigma^{2}}\right)\)
(1)如果正态分布假设满足，
\[\boldsymbol{y}=\boldsymbol{X} \boldsymbol{\beta}+\varepsilon, \quad \varepsilon \sim N\left(\mathbf{0}, \sigma^{2} \boldsymbol{I}_{n}\right)
\]
则 \(\boldsymbol{y}\) 的概率分布为 \(: \boldsymbol{y} \sim N\left(\boldsymbol{X} \boldsymbol{\beta}, \sigma^{2} \boldsymbol{I}_{n}\right),\) 这是似然函数为
\[L(\boldsymbol{\beta})=\left(2 \pi \sigma^{2}\right)^{-n / 2} \exp \left\{-\frac{1}{2 \sigma^{2}}(\boldsymbol{y}-\boldsymbol{X} \boldsymbol{\beta})^{\prime}(\boldsymbol{y}-\boldsymbol{X} \boldsymbol{\beta})\right\}
\]
对其做“-ln”变换，记为 \(\ell(\boldsymbol{\beta}),\) 可以证明 \(\boldsymbol{\beta}\) 的极大似然估计 \(\hat{\beta}_{M L E}\) 与最小二 乘估计等价。
\[\hat{\boldsymbol{\beta}}=\left(\boldsymbol{X}^{\prime} \boldsymbol{X}\right)^{-1} \boldsymbol{X}^{\prime} \boldsymbol{y}
\]
而误差项的方差 \(\sigma^{2}\) 的极大似然估计
\[\widehat{\sigma^{2}}=\frac{SSE}{n}
\]
(2)
\[E\left(\hat{\sigma}^{2}\right)=\left(\frac{n-p-1}{n}\right) \frac{SSE}{n-p-1}
\]
偏决定系数
\[\begin{array}{l}
\text { Model1: } y_{i}=\beta_{0}+\beta_{1} x_{1 i}+\beta_{2} x_{2 i}+\varepsilon_{i}, i=1, \ldots, n \\
\text { Model0 : } y_{i}=\beta_{0}+\beta_{2} x_{2 i}+\varepsilon_{i}, i=1, \ldots, n
\end{array}
\]
\[r_{y 1 ; 2}^{2}=\frac{\operatorname{SSE}\left(x_{2}\right)-\operatorname{SSE}\left(x_{1}, x_{2}\right)}{\operatorname{SSE}\left(x_{2}\right)} \stackrel{?}{=} \frac{\operatorname{SSR}\left(x_{1}, x_{2}\right)-\operatorname{SSR}\left(x_{2}\right)}{\operatorname{SSE}\left(x_{2}\right)}
\]
当模型中已有 \(x_{1}, \ldots, x_{j-1}, x_{j+1}, \ldots, x_{p}\) 时， \(y\) 与 \(x_{j}\) 的偏决定系数为:
\[\begin{aligned}
r_{y j ;-j}^{2} &=\frac{\operatorname{SSE}\left(x_{1}, \ldots, x_{j-1}, x_{j+1}, \ldots, x_{p}\right)-\operatorname{SSE}\left(x_{1}, \ldots, x_{p}\right)}{\operatorname{SSE}\left(x_{1}, \ldots, x_{j-1}, x_{j+1}, \ldots, x_{p}\right)} \\
&=\frac{S S R-\operatorname{SSR}(-j)}{\operatorname{SSE}(-j)}
\end{aligned}
\]
偏相关系数:
\[r_{j k}=\frac{S_{j k}}{\sqrt{S_{j j} S_{k k}}}
\]
\[r_{12 ; 3, \ldots, p}=\frac{-\Delta_{12}}{\sqrt{\Delta_{11} \Delta_{22}}}
\]
\[r_{12 ; 3}=\frac{r_{12}-r_{13} r_{23}}{\sqrt{\left(1-r_{13}^{2}\right)\left(1-r_{23}^{2}\right)}}
\]
