SC1 - Type I errors

Errors in context

A daily fault test flags an increase from mean 6 whenever it records at least 11 faults. If the true mean is still 6, what kind of error is a flag, and how likely is it? A surprising random sample can trigger rejection even when the null hypothesis is true.

Suppose $\lambda$ is the mean fault count per day and we test $H_0:\lambda=6$ against $H_1:\lambda>6$. There are two possible decisions and two possible underlying situations. The decision alone does not reveal whether an error occurred.

Here a Type I error means concluding there is evidence that the mean fault count per day has increased, when it is still 6. A Type II error means finding insufficient evidence of an increase when the true daily mean is greater than 6. Include the mean, the direction and the reference value: “the machine is faulty” does not identify the population claim being tested.

For a test of whether a coin's head probability has decreased from 0.5, the context changes but the logic does not. A Type I error reports evidence of a decrease when $p=0.5$. A Type II error misses a decrease when $p<0.5$. When an alternative is two-sided, a Type II error can miss a change in either direction.

If a test rejects, a Type I error is possible, but a Type II error is not possible on that occasion. Rejection might equally be a correct decision if the alternative is true. You cannot decide the actual truth simply by naming the test's decision.

Apply both definitions to a new context.

Calculating actual significance

To calculate a Type I error probability, imagine repeatedly applying the same decision rule while $H_0$ is true. Count every outcome that triggers rejection, then evaluate its total probability using the null distribution.

For binomial counts, recall that $X\sim\operatorname{B}(n,p)$ requires a fixed number of independent trials with the same success probability. Its possible counts are $0,1,\ldots,n$. Its mass function, mean and variance are supplied on booklet page 11. For a normal statistic $T\sim\operatorname{N}(\mu,\sigma^2)$, standardise with $Z=(T-\mu)/\sigma$; the second parameter is the variance. This standardisation is assumed knowledge, not separately printed as an individual-observation formula in the booklet. For a mean of independent normal observations, the booklet gives standard error $\sigma/\sqrt n$ on page 11. Here all statistic distributions are given explicitly; no approximation is required.

Change the count model to binomial while keeping the same principle: the complete rejection tail under $H_0$.

A normal critical region is an area under a continuous curve. The same event logic applies, but there is no point mass at the boundary.