数理统计习题一
{\bf Question 1:}
Let $X\sim f_\theta(x)$ for some density function $f_\theta(x)$ and define
\[I(\theta) = E_\theta\left[ \left( \frac{\partial}{\partial\theta} \log f_\theta(X) \right)^2 \right].
\]
Prove that if $\partial^2/\partial\theta^2 \log f_\theta(x)$ exists for all $x$ and $\theta$
and
\[
\frac{\partial^2}{\partial\theta^2} \int f_\theta(x) \, dx =
\int\frac{\partial^2}{\partial\theta^2} f_\theta(x) \, dx,
\]
then
\begin{equation}\label{inf}
I(\theta) = -E_\theta \left(\frac{\partial^2}{\partial\theta^2} \log f_\theta(X) \right).
\end{equation}
{\bf Solution:}
Since
\[
\partial^2/\partial\theta^2 \log f_\theta(x) = \frac{ \frac{\partial^2}{\partial \theta^2} f_\theta(x)}
{f_\theta(x)} - \left( \frac{\frac{\partial}{\partial \theta} f_\theta(x)}{f_\theta(x)} \right)^2 =
\frac{ \frac{\partial^2}{\partial \theta^2} f_\theta(x)}
{f_\theta(x)} - \left( \frac{\partial}{\partial \theta} \log f_\theta(x) \right)^2,
\]
taking expectations gives
\[
-E_\theta \left(\frac{\partial^2}{\partial\theta^2} \log f_\theta(X) \right) =
E_\theta \left[ \left( \frac{\partial}{\partial \theta} \log f_\theta(x) \right)^2 \right]-
\frac{\partial^2}{\partial \theta^2} \int f_\theta(x)\,dx = I(\theta).
\]
{\bf Question 2: }
If $X_1, \ldots, X_n\ iid N(\mu,\sigma^2)$ and $S^2$ is the usual unbiased
sample variance, what value of $\alpha$ minimizes the MSE of $\alpha S^2$ as an estimator of
$\sigma^2$?
{\bf Solution:}
As a function of $\alpha$, we obtain
\[
E [( \alpha S^2 - \sigma^2)^2 ] = \alpha^2 E [(S^2)^2] - 2\alpha\sigma^2 E S^2 + \mbox{constant}.
\]
Since $(n-1)S^2/\sigma^2 \sim \chi^2_{n-1}$, we know $\Var S^2=2\sigma^4/(n-1)$ and so
\[
E[(S^2)^2] = \mbox{Var} S^2 + [E(S^2)]^2 = \sigma^4\left( \frac{2}{n-1} + 1 \right) =
\frac{\sigma^4(n+1)}{n-1}.
\]
Thus, if we factor out $\sigma^4$ and ignore the additive constant,
the MSE is
\[
\alpha^2 \left(\frac{n+1}{n-1}\right) - 2\alpha,
\]
and this quadratic is minimized at $\alpha=(n-1)/(n+1)$.
{\bf Question 3:\ }
Let $R(\theta, S)$ denote the risk in estimating $\theta$ by $S$. If $S$ and $T$ are two
estimators such that $R(\theta, T)\le R(\theta, S)$ for all $\theta$ with strict inequality
holding for at least one value of $\theta$, then we call $S$ {\em inadmissible}
as an estimator.
{\em Aside: Convince yourself that this is a sensible thing to call such an estimator.}
Prove that if $T$ is an unbiased estimator of $\theta$ and $b$ is a nonzero constant, then
$T+b$ is inadmissible. Assume a squared error loss function.
{\bf Solution:}
Using squared error loss, the risk is simply the MSE, which is variance plus bias squared.
Since $T$ and $T+b$ have the same variance, we conclude that
\[
R(\theta, T) = R(\theta, T+b) - b^2 \le R(\theta, T+b),
\]
with strict inequality whenever $b\ne0$. Thus, by definition, $T+b$ is inadmissible when
$b\ne0$.
{\bf Question 4:\ }
Suppose that $X_1, \ldots, X_n\iid \mbox{Bernoulli$(\theta)$}$.
Show that $\overline X$ achieves the Cram\'er-Rao lower bound and explain why this fact
may be used to argue that
$\overline X$ is a UMVU (uniformly minimum variance unbiased) estimator of $\theta$.
{\bf Solution:}
We will use the facts that $E X_i=\theta$ and $\ Var X_i=\theta(1-\theta)$.
Since $\log f_\theta(\mathbf x) = \sum_i x_i \log\theta + \sum_i(1-x_i) \log(1-\theta)$,
the information in the whole sample is
\[
- E_\theta \frac{\partial^2}{\partial \theta^2} \log f_\theta(\mathbf X) =
\frac{E_\theta \sum_i X_i}{\theta^2} + \frac{E_\theta \sum_i(1-X_i) }{(1-\theta)^2} =
\frac{n}{\theta} + \frac{n}{1-\theta} = \frac{n}{\theta(1-\theta)}.
\]
Since $\overline X$ is unbiased for $\theta$, the Cram\'er-Rao lower bound is given
by $1/I(\theta)$, which equals $\theta(1-\theta)/n$.
Finally, $\Var \overline X=\ Var X_i/n = \theta(1-\theta)/n$, so $\overline X$ achieves the
Cram\'er-Rao lower bound.
We conclude that $\overline X$ must be a UMVUE, since it achieves the smallest possible
variance for an unbiased estimator for all values of $\theta$. Stated in a different way:
If $W$ is another
unbiased estimate, then the information inequality guarantees that $\ Var_\theta(W)\ge
[I(\theta)]^{-1}=\ Var_\theta(\overline X)$ for all $\theta$, which means $\overline X$ is UMVU.
{\bf Question 5:\ }
For a positive unknown $\theta$,
let $X_1, \ldots, X_n \iid f_\theta(x) = (1/\theta) x^{(1-\theta)/\theta} I\{0<x<1\}$.
{\em NB: The assignment had a misprint: The $Y_i$ should have been $X_i$.}
\vspace{2ex}
{\bf(a)\ }
Derive the observed and expected (Fisher) information contained in this sample. Are they the same?
\vspace{2ex}
{\bf (b)\ }
Derive the MLE and MME (method of moments estimator) of $\theta$. Recall that the MME is
found by setting the first $k$ sample moments ($n^{-1}\sum_i X_i$, $n^{-1}\sum_iX_i^2$, \ldots)
equal to the first $k$ population moments ($E(X)$, $E(X^2)$, \ldots) and then solving for the
parameters, where $k$ is the number of parameters.
\vspace{2ex}
{\bf(c)\ } Which of the two estimators in (b), if either, is a UMVU estimator? Justify your answer.
{\bf Solution:}
First, to establish some facts that will be useful later, we observe that
letting $\eta=(1-\theta)/\theta$, the density function may be written as
\[
f_\eta(x) = \exp\{\eta \log x - [-\log(\eta+1)] \} I\{0<x<1\},
\]
which is a one-parameter canonical exponential family with sufficient statistic $\log X$.
Therefore, $E (\log X) = -1/(\eta+1)=-\theta$ and $\Var (\log X) = 1/(\eta+1)^2=\theta^2$.
\vspace{2ex}
{\bf(a)\ }
Taking two derivatives of the log-likelihood function gives
\[
-\frac{\partial^2}{\partial\theta^2} \log f_\theta(\mathbf X) =
-\frac{n}{\theta^2} - \frac{2\sum_i\log X_i}{\theta^3} = -\frac{1}{\theta^3} \sum_i (2\log X_i + \theta).
\]
The above expression, with $x_i$ in place of $X_i$, is the observed information.
Taking expectations, we conclude that the expected information is $n/\theta^2$.
\vspace{2ex}
{\bf(b)\ }
Setting the first derivative of the log-likelihood equal to zero gives
\[
\hat\theta = -\frac{\sum_{i=1}^n \log X_i}{n}
\]
as the MLE. (We know that this value maximizes the log-likelihood because
the first derivative is $-1/\theta^2$ times a linear function of $\theta$ with positive slope.)
For the MME, we obtain
\[
E X_i = \frac{1}{\theta} \int_0^1 x^{1/\theta} \,dx = \frac{1}{1+\theta}.
\]
Setting equal to $\overline X$, we find that the MME is $(1/\overline X) - 1$.
\vspace{2ex}
{\bf(c)\ }
Since $E (\log X_i)=-\theta$, the MLE is unbiased.
Also, since $\Var (\hat\theta) = \Var X_i/n = \theta^2/n$,
the MLE achieves the Cram\'er-Rao lower bound. We conclude that the MLE is
a UMVUE.
The UMVUE is unique, so the MME cannot be UMVUE for $\theta$. Another way to see
this is that the MME cannot be unbiased, since this would require
\[
E \left(\frac{1}{\overline X} \right) = \frac{1}{E (\overline X)}.
\]
