06-02.md

title	Class 6
date	February 06, 2023

Consistent Estimators

\begin{definition} Let $\hat{\Theta}_1, \hat{\Theta}_2,\ldots,\hat{\Theta}_n$ be a sequence of point estimators of $\theta$. We say that $\hat{\Theta}$ is a consistent estimator of $\theta$ if

$$ \lim_{n\rightarrow \infty} P(|\hat{\Theta} - \theta| \ge \epsilon) = 0 \quad \forall \epsilon > 0 $$ \end{definition}

An alternate definition:

\begin{definition} Let $\hat{\Theta}_1, \hat{\Theta}_2,\ldots,\hat{\Theta}_n$ be a sequence of point estimators of $\theta$. We say that $\hat{\Theta}_n$ is a consistent estimator of $\theta$ if

$$ \lim_{n\rightarrow \infty} MSE(\hat{\Theta}_n) = 0$$ \end{definition}

\begin{proof} \begin{align*} P(|\hat{\Theta} - \theta| \ge \epsilon) &= P(|\hat{\Theta} - \theta|^2 \ge \epsilon^2)\ &\le \frac{E\bs{|\hat{\Theta}_n - \theta|^2}}{\epsilon^2} \quad \text{By Markov inequality}\ &= MSE(\hat{\Theta}_n)/\epsilon^2 \end{align*}
\end{proof}

Maximum Likelihood Estimator

\begin{example} Bag contains 3 balls. Each ball is either red or blue. Let $\theta$ be the number of blue balls. Define random variables $X_1, X_2, X_3,X_4$ (balls are chosen with replacement that is why we can have more than three random variables). $$X_i = \begin{cases} 1 & \text{if } i^{th} \text{ chosen ball is blue} \ 0 & \text{otherwise} \end{cases} $$

Then $X_i \sim Bernoulli\bp{\frac{\theta}{3}}$. By independence, $P(x_1, x_2,x_3,x_4) = \prod_i P_{X_i} (x_i)$ where $P_{X_i} (x; \theta) = \binom{3}{x} \theta^x (1 - \theta)^{3 - x}$.

\textbf{Likelihood} function is then defined as: $L(x_1, x_2, x_3, x_4) = P(x_1, x_2,x_3,x_4;\theta)$.

For jointly continuous random variables, $L(x_1, x_2,x_3,x_4) = f(x_1, x_2,x_3,x_4;\theta)$. \end{example}

Bayesian Inference

Draw inference of an unknown random variable $X$ by observing random variable $Y$. The unknown variable is modelled with prior distribution $P_X(x)$.

After observing $Y$, we find posterior distribution of $X, \ P_{X|Y} (x|y)$. Usually found using Bayes' formula.

MAP Estimate is then shown by $\hat{x}_{MAP}$. We don't care about the denominator (in Bayes' rule) for MAP estimate because it is just a constant.

So we maximize $f_{Y|X}(y|x) f_X(x)$.