SC2 - Type II errors and power

SC2 - Type II errors and power

A test designed to avoid false alarms can also miss a real change. You will fix a decision rule using the null model, calculate its chance of missing a specified alternative, and find its power. The same event-based method works for Poisson, binomial and normal tests.

Fixing the rejection region

A 5% test looks for an increase from 6 faults per day. If the true mean becomes 10, how often will the test detect the increase? Before calculating that probability, fix what the test counts as a detection.

A Type II error occurs when the test does not reject H0H_0 although a value in the alternative is true. Its probability depends on the particular true alternative value. “The mean has increased” covers many possible means; a mean of 6.1 is usually harder to detect than a mean of 20. You cannot attach one Type II probability to every value in a composite alternative.

There are two separate stages. First design a critical region using the distribution under H0H_0 and the chosen significance level. Then keep that region fixed and change the distribution to the specified alternative when measuring its performance. A supplied rejection rule is already fixed: use it as given.

Worked example 1: designing the 5% rule

Question. For a Poisson count XX, test H0:λ=6H_0:\lambda=6 against H1:λ>6H_1:\lambda>6 at the 5% level. Find the upper-tail critical region and the non-rejection outcomes.

Plan. Use mean 6 to compare neighbouring upper tails.

P⁡6(X≥10)=0.0839240…>0.05,P⁡6(X≥11)=0.0426209…<0.05\operatorname{P}_{6}(X\ge10)=0.0839240\ldots>0.05, \qquad \operatorname{P}_{6}(X\ge11)=0.0426209\ldots<0.05

Answer. Reject for X≥11X\ge11; do not reject for X=0,1,…,10X=0,1,\ldots,10.

Check. Those two sets partition all non-negative integers, including the boundary 10 on the non-rejection side. The actual significance is about 4.26%.

Change only the nominal level. This produces a different designed test, rather than changing the rule merely because a different true mean is being considered.

Worked example 2: designing a more permissive rule

Question. Keep H0:λ=6H_0:\lambda=6 and H1:λ>6H_1:\lambda>6, but use the 10% level. Find the critical region and the non-rejection outcomes.

Plan. Find the first upper tail at most 0.10 under the same null model.

P⁡6(X≥9)=0.1527625…>0.10,P⁡6(X≥10)=0.0839240…<0.10\operatorname{P}_{6}(X\ge9)=0.1527625\ldots>0.10, \qquad \operatorname{P}_{6}(X\ge10)=0.0839240\ldots<0.10

Answer. Reject for X≥10X\ge10; do not reject for X=0,1,…,9X=0,1,\ldots,9.

Check. The new rule includes every old rejection outcome and also 10. It can detect more increases, but under the null it also makes more false rejections.

A decision rule balances these risks. With the sample design and models fixed, enlarging a critical region increases or preserves both its Type I probability under H0H_0 and its detection probability under any given alternative. This follows directly because the old rejection set is contained in the new one.

Watch out

Find a critical boundary under H0H_0. Do not recompute that boundary using the alternative mean when calculating Type II error: that would evaluate a different test.

Identify a new fixed rule before measuring its performance.

Type II probability under an alternative

Return to the 5% fault test, whose fixed rejection region is X≥11X\ge11. If the true mean is 10, a Type II error means observing one of 0,1,…,100,1,\ldots,10 under Po⁡(10)\operatorname{Po}(10). The set comes from the null-designed rule; its probability comes from the alternative model.

Type II error probability

For a fixed critical region RR and a specified true parameter θ1\theta_1 in the alternative,

β(θ1)=P⁡θ1(T∉R)\beta(\theta_1)=\operatorname{P}_{\theta_1}(T\notin R)

Sum or integrate over all non-rejection outcomes under that specified alternative. This definition is not separately printed in the formula booklet.

For an integer count, rejection at X≥cX\ge c means non-rejection at X≤c−1X\le c-1. Rejection at X≤dX\le d means non-rejection at X≥d+1X\ge d+1. Rejection at X≤aX\le a or X≥bX\ge b, where a<ba<b, means non-rejection at a+1≤X≤b−1a+1\le X\le b-1. Do not discard zero when it is an allowed non-rejection outcome.

Worked example 3: missing a rise to ten

Question. A test of H0:λ=6H_0:\lambda=6 against an increase rejects when X≥11X\ge11. Find its Type II error probability when the true mean is 10.

Plan. Keep non-rejection X≤10X\le10, but evaluate it using mean 10.

β(10)=P⁡10(X≤10)=∑x=010e−1010xx!=0.5830398…\beta(10)=\operatorname{P}_{10}(X\le10) =\sum_{x=0}^{10}\frac{\mathrm{e}^{-10}10^x}{x!} =0.5830398\ldots

Answer. About 0.58300.5830, so this test misses that particular increase about 58.3% of the time.

Check. The complementary upper tail under the same alternative is P⁡10(X≥11)=0.4169602…\operatorname{P}_{10}(X\ge11)=0.4169602\ldots. The two probabilities sum to 1. A false-negative probability can be much larger than the test's 5% significance level.

Keep the rejection region fixed and change only the specified true mean. This isolates how the size of the increase affects detection.

Worked example 4: missing a larger rise

Question. For the same rule X≥11X\ge11, find the Type II error probability when the true mean is 12.

Plan. Evaluate the same non-rejection set under Po⁡(12)\operatorname{Po}(12).

β(12)=P⁡12(X≤10)=0.3472294…\beta(12)=\operatorname{P}_{12}(X\le10)=0.3472294\ldots

Answer. About 0.34720.3472, a smaller miss probability than at mean 10.

Check. Its complement is P⁡12(X≥11)=0.6527706…\operatorname{P}_{12}(X\ge11)=0.6527706\ldots. A larger mean places more probability beyond this fixed upper boundary.

The same logic works for a decrease. If a test rejects for X≤3X\le3, then under a specified true Poisson mean 2 its Type II probability is P⁡2(X≥4)=1−P⁡2(X≤3)=0.1428765…\operatorname{P}_2(X\ge4)=1-\operatorname{P}_2(X\le3)=0.1428765\ldots, not the lower tail. For a two-sided rule rejecting at X≤1X\le1 or X≥9X\ge9, non-rejection is 2≤X≤82\le X\le8; use F(8)−F(1)F(8)-F(1) under the specified alternative. Both endpoints follow from the original rule.

Watch out

Type II probability uses the alternative parameter and the non-rejection set. The null parameter gives Type I risk instead; the rejection set under the alternative gives detection rather than a miss.

Change both the context and the numerical values in your own calculation.

Power for Poisson, binomial and normal tests

A test's power at a specified alternative is its probability of detecting that alternative by rejecting H0H_0. Rejection and non-rejection are complementary, so the two performance measures fit together exactly.

Power

For a fixed critical region RR, at a specified true alternative θ1\theta_1,

power⁡(θ1)=P⁡θ1(T∈R)=1−β(θ1)\operatorname{power}(\theta_1)=\operatorname{P}_{\theta_1}(T\in R)=1-\beta(\theta_1)

Power lies between 0 and 1 and depends on the alternative, observation design and rule. This definition is not separately printed in the formula booklet.

Continue this lesson.

You’re previewing NovaLearn without an account. Create a free account to read the rest.

Create free account