6.5 - Type I and Type II errors

Type I and Type II errors

A Type I error occurs when a true null hypothesis is rejected. A Type II error occurs when a false null hypothesis is not rejected. The labels describe the truth of the null together with the test's decision, rather than whether the sample calculation was performed correctly.

Worked example 1: a claimed improvement that is not real

Question. A factory tests $H_0:p=0.4$ against $H_1:p<0.4$, where $p$ is its fault probability. It rejects $H_0$, but in fact $p=0.4$. Identify the error in context.

Plan. Decide whether the null is true, then combine that truth with the rejection decision.

The actual value $p=0.4$ makes $H_0$ true. The test nevertheless rejects it.

Answer. A Type I error: the test finds sufficient evidence of a reduced fault probability when the fault probability is still $0.4$.

Check. “Reject a true null” is exactly the Type I combination in the decision table.

Worked example 2: an improvement that is missed

Question. For the same hypotheses, a test does not reject $H_0$, but the actual fault probability is $0.2$. Identify the error in context.

Plan. Compare the true value with the null, then interpret non-rejection.

Since $0.2\ne0.4$, $H_0$ is false; the stated alternative of a decrease is true. The test fails to detect enough evidence for that decrease.

Answer. A Type II error: the test fails to find sufficient evidence of a reduction although the actual fault probability has decreased to $0.2$.

Check. The conclusion is “insufficient evidence”, rather than a proof that $p=0.4$. An error can occur without anyone asserting the null is certainly true.

Watch out

The significance level is calculated assuming $H_0$ is true. It is not the probability that a rejected null is true, or the probability that the current decision is wrong after observing the data.

Error probabilities for a fixed rule

Let $R$ be the rejection region and $A$ its complement. Then

while, for a specified alternative parameter $\theta_1$,

The subscripts identify which population model supplies the probabilities. Not in the formula booklet: learn the meaning of the two events. Binomial and Poisson probability formulas are supplied.

Worked example 3: errors in a binomial test

Question. A test of $H_0:p=0.4$ against $H_1:p<0.4$ uses $X$, the fault count among $10$ independent items. Its fixed critical region is $X\le1$. Find the Type I error probability and the Type II error probability if the actual value is $p=0.2$.

Plan. Use the rejection region under $p=0.4$, then its complement under $p=0.2$.

Under the null, $X\sim\mathrm B(10,0.4)$, so

The acceptance region is $X\ge2$. Under the specified alternative, $X\sim\mathrm B(10,0.2)$:

Answer. Type I probability $0.0464$; Type II probability $0.624$ when $p=0.2$.

Check. Under $p=0.2$, the test detects the improvement with probability $0.3758096$; this and $0.6241904$ sum to $1$. They do not sum with the Type I probability, which uses a different population.

Worked example 4: errors in a Poisson test

Question. A test investigates an increase in mean calls per hour using $H_0:\lambda=4$, $H_1:\lambda>4$. The one-hour count has fixed critical region $X\ge9$. Find the Type I probability and the Type II probability when the actual mean is $6$ calls per hour.

Plan. Evaluate the upper rejection tail under $\operatorname{Po}(4)$, and the lower acceptance region under $\operatorname{Po}(6)$.

Answer. Type I probability $0.0214$; Type II probability $0.847$ at the stated alternative.

Check. The test misses the increase whenever the observed count is $0$ through $8$. Using $X\le9$ would incorrectly include a rejecting result in the Type II event.