FS2 5.1 - Estimators, bias and standard error

FS2 5.1 - Estimators, bias and standard error

A sample gives one answer, but an estimator is a rule that would produce different answers from fresh samples. You will distinguish these ideas, calculate bias and standard error, estimate a population variance with the correct divisor, and compare the quality of estimators.

An estimate is one outcome of an estimator

Why divide the sum of squared sample deviations by n−1n-1, rather than by the number of observations nn? The sample mean has already moved towards the observations. Measuring deviations from that fitted centre systematically makes the uncorrected spread too small.

There is a surprising consequence: an unbiased variance estimator can give zero for some samples from a population with positive variance. “Unbiased” describes its average performance across repeated samples, not a promise about each result. You will see an exact four-sample example below.

You already know

For independent observations, combine means with signed coefficients and variances with squared coefficients. An independent sample mean from N(μ,σ2)\mathrm N(\mu,\sigma^2) has distribution N(μ,σ2/n)\mathrm N(\mu,\sigma^2/n).

A population parameter is a fixed feature of the model, such as μ\mu or σ2\sigma^2. A statistic is calculated from a sample; before the sample is observed, that statistic is a random variable.

Estimator and estimate

An estimator is a sample statistic used to estimate a population parameter. An estimate is its realised numerical value. For example, Xˉ=(X1+⋯+Xn)/n\bar X=(X_1+\cdots+X_n)/n is an estimator of μ\mu; an observed xˉ=12\bar x=12 is one estimate. The unknown parameter μ\mu is not the sample statistic.

To judge a rule, imagine applying it to a fresh random sample each time. Its sampling distribution describes the resulting collection of estimates.

Bias and standard error

For an estimator TT of a parameter θ\theta, its bias is E(T)−θ\mathrm E(T)-\theta. It is unbiased if E(T)=θ\mathrm E(T)=\theta for every allowed value of the parameter.

Its standard error is its sampling standard deviation, se⁡(T)=Var⁡(T)\operatorname{se}(T)=\sqrt{\operatorname{Var}(T)}. Bias describes systematic displacement; standard error describes variation between estimates. These defining expressions are not separately named in the formula booklet.

A small standard error does not guarantee a small bias. A narrow cluster can be centred in the wrong place.

Sample mean as an estimator

For independent observations from one population with finite mean μ\mu and variance σ2\sigma^2,

E(Xˉ)=μ,Var⁡(Xˉ)=σ2n\mathrm E(\bar X)=\mu,\qquad \operatorname{Var}(\bar X)=\frac{\sigma^2}{n}

Thus Xˉ\bar X is unbiased for μ\mu. These moment results do not require a normal population.

In the formula booklet.

Taking the positive square root gives se⁡(Xˉ)=σ/n\operatorname{se}(\bar X)=\sigma/\sqrt n. This follows from the printed variance; the same expression appears in the booklet's normal-mean standardisation. Normality is needed for that exact normal distribution, rather than for the two moment identities themselves.

Key idea

An unbiased rule gets the target right on average across samples. It need not hit the target in the sample you happen to observe.

Keep the rule, its output and its target distinct.

Compare the centre and spread of a proposed rule

A denominator can change both bias and standard error. Calculate the expectation before deciding what a statistic estimates.

Let X1,X2,X3X_1,X_2,X_3 be independent observations from one population with mean μ\mu and variance σ2>0\sigma^2>0. Normality is not needed for the following moment calculations. The linear expectation and independent-variance rules are recalled from the preceding lesson; the variance algebra is supplied in the booklet.

Worked example 1: an unbiased weighted average

Question. For T4=(X1+X2+2X3)/4T_4=(X_1+X_2+2X_3)/4, find its bias as an estimator of μ\mu, its variance and its standard error. Compare it with the ordinary mean of the three observations.

Plan. Add the signed expectation coefficients and the squared variance coefficients, then compare with σ2/3\sigma^2/3.

E(T4)=μ+μ+2μ4=μcoefficients sum to oneBias⁡(T4)=μ−μ=0unbiased for μVar⁡(T4)=12+12+2242σ2=38σ2independent squared coefficientsse⁡(T4)=38 σpositive square root38σ2>13σ2compare unbiased estimators\begin{aligned} \mathrm E(T_4)&=\frac{\mu+\mu+2\mu}{4}=\mu && \text{coefficients sum to one}\\ \operatorname{Bias}(T_4)&=\mu-\mu=0 && \text{unbiased for }\mu\\ \operatorname{Var}(T_4)&=\frac{1^2+1^2+2^2}{4^2}\sigma^2 =\frac38\sigma^2 && \text{independent squared coefficients}\\ \operatorname{se}(T_4)&=\sqrt{\frac38}\,\sigma && \text{positive square root}\\ \frac38\sigma^2&>\frac13\sigma^2 && \text{compare unbiased estimators} \end{aligned}

Why compare variances only after checking the target? Both T4T_4 and Xˉ\bar X are unbiased for the same μ\mu, so the smaller variance identifies the more concentrated unbiased rule.

Answer. T4T_4 is unbiased, has variance 3σ2/83\sigma^2/8 and standard error 3/8 σ\sqrt{3/8}\,\sigma. The ordinary sample mean has smaller variance.

Check. The variance difference is (38−13)σ2=σ2/24>0(\frac38-\frac13)\sigma^2=\sigma^2/24>0. Unequal weighting has increased the spread here.

Worked example 2 changes only the denominator, from 44 to 55. A smaller spread will come with a changed centre.

Worked example 2: shrinking towards zero

Question. For T5=(X1+X2+2X3)/5T_5=(X_1+X_2+2X_3)/5, find its bias as an estimator of μ\mu, its variance and its standard error.

Plan. Repeat both coefficient calculations with denominator 55, then subtract the target μ\mu.

E(T5)=4μ5new total expectation coefficientBias⁡(T5)=4μ5−μ=−μ5subtract the targetVar⁡(T5)=625σ2square the new denominatorse⁡(T5)=65 σtake the square root\begin{aligned} \mathrm E(T_5)&=\frac{4\mu}{5} && \text{new total expectation coefficient}\\ \operatorname{Bias}(T_5)&=\frac{4\mu}{5}-\mu=-\frac{\mu}{5} && \text{subtract the target}\\ \operatorname{Var}(T_5)&=\frac6{25}\sigma^2 && \text{square the new denominator}\\ \operatorname{se}(T_5)&=\frac{\sqrt6}{5}\,\sigma && \text{take the square root} \end{aligned}

Why is this not an unbiased estimator of μ\mu? Its expectation does not equal μ\mu for every possible μ\mu. Equality at the special value μ=0\mu=0 does not make the whole rule unbiased.

Answer. The bias is −μ/5-\mu/5, the variance is 6σ2/256\sigma^2/25, and the standard error is 6 σ/5\sqrt6\,\sigma/5.

Check. Since T5=45T4T_5=\frac45T_4, its variance is (45)2(3σ2/8)=6σ2/25(\frac45)^2(3\sigma^2/8)=6\sigma^2/25, agreeing with the direct calculation.

The variance has fallen, but the rule now systematically pulls estimates towards zero. Smaller variance alone cannot settle this comparison between a biased and an unbiased estimator.

Watch out

Wrong: a smaller standard error automatically makes an estimator better. Right: check the target and bias first, then compare spread. Among unbiased estimators of the same parameter, smaller variance is desirable.

Change only the denominator again.

Estimate variance after estimating the centre

The sample mean is chosen from the same observations whose spread you measure. Their deviations satisfy ∑(xi−xˉ)=0\sum(x_i-\bar x)=0, leaving only n−1n-1 freely varying deviations. The correction below removes the resulting downward bias in the variance estimate.

Unbiased sample variance

For an independent random sample from one population with finite variance, with n≥2n\ge2,

S2=1n−1∑i=1n(Xi−Xˉ)2S^2=\frac1{n-1}\sum_{i=1}^{n}(X_i-\bar X)^2

satisfies E(S2)=σ2\mathrm E(S^2)=\sigma^2. Lower-case s2s^2 is its value for the observed sample.

In the formula booklet.

Continue this lesson.

You’re previewing NovaLearn without an account. Create a free account to read the rest.

Create free account