FS1 8.1 - Size and power of a test | Edexcel A-Level Further Mathematics

An effective test must control false alarms while detecting changes that matter. You will calculate size and power, derive power functions, choose a feasible rejection rule and compare tests using detection, error rates and sampling cost.

A rejection probability viewed across possible populations

Can one test be better at detecting a small increase, but worse at detecting a larger one?

Yes. A single power value describes only one alternative. To compare tests properly, look at how their rejection probabilities change across the possible true parameter values, as well as their false-alarm probabilities and sampling costs.

You already know

Type I error is rejection under a true null; Type II error is non-rejection under a specified true alternative. Keep the rejection rule fixed when evaluating it at different true parameters. Normal calculations use standard deviation, not variance.

The same mathematical probability takes different names according to which true population parameter is being considered.

Size, power and power function

For a fixed test of a specified null value, its size is its Type I error probability.

Its power at a specified true alternative is the probability of rejecting $H_0$ at that alternative.

The power function gives the rejection probability as a function of the true parameter, with the test rule held fixed.

Power and Type II error concern opposite decisions under the same alternative, so they are complements.

Power function for a fixed test

For critical region $C$ , define

\pi(\theta)=\mathrm P_\theta(X\in C)

For a simple null $H_0:\theta=\theta_0$ ,

\text{size}=\pi(\theta_0)

At a specified true alternative $\theta_1$ ,

\text{power}=\pi(\theta_1)=1-\beta(\theta_1)

The function is defined across the parameter domain; at the null point its value is size, while at an alternative it measures detection. Not in the formula booklet: learn it.

A stated $5\%$ level is an upper limit for the false-alarm probability of the chosen rule. Its actual size can be smaller, especially with discrete outcomes. Increasing power by rejecting almost everything would also increase size, so power alone does not establish quality.

Worked example 1: size and power for a normal measurement

Question. A measurement $X$ has distribution $\mathrm N(\mu,4)$ . Test $H_0:\mu=10$ against $H_1:\mu>10$ , rejecting when $X>13.3$ . Find the size and the power at true mean $12$ , to four decimal places.

Plan. Evaluate the same upper tail first under the null mean, then under the specified alternative.

\begin{aligned} \text{size}&=\mathrm P_{\mu=10}(X>13.3) =\mathrm P\!\left(Z>\frac{13.3-10}{2}\right)\\ &=\mathrm P(Z>1.65)=0.0494714\ldots\\ \pi(12)&=\mathrm P_{\mu=12}(X>13.3) =\mathrm P\!\left(Z>\frac{13.3-12}{2}\right)\\ &=\mathrm P(Z>0.65)=0.2578461\ldots \end{aligned}

Why this step? The true mean changes between the two probability calculations; the variance $4$ and cutoff $13.3$ do not.

Answer. Size $=0.0495$ , power at $\mu=12$ $=0.2578$ .

Check. The size is just below $0.05$ . The alternative mean is closer to the critical region, so its rejection probability is larger.

Worked example 2 changes only the true alternative mean from $12$ to $14$ .

Worked example 2: detecting a larger normal mean

Question. Find the power of the same normal test when the true mean is $14$ .

Plan. Evaluate its unchanged critical tail under $\mathrm N(14,4)$ .

\begin{aligned} \pi(14)&=\mathrm P_{\mu=14}(X>13.3)\\ &=\mathrm P\!\left(Z>\frac{13.3-14}{2}\right) =\mathrm P(Z>-0.35)\\ &=0.6368306\ldots \end{aligned}

Why this step? Power varies with the actual mean. Replacing the cutoff by a new null-based boundary would change the test itself.

Answer. Power $=0.6368$ , to four decimal places.

Check. The cutoff lies below the true mean, so the probability exceeds $0.5$ . The test's size remains $0.0494714\ldots$ , because the null model has not changed.

For this test the whole power function is

\pi(\mu)=1-\Phi\!\left(\frac{13.3-\mu}{2}\right)

where $\Phi$ is the standard normal cumulative distribution function. This derived function is not printed in the booklet; it follows from the recalled normal tail calculation.

Watch out

Wrong: report a single power without stating the true alternative, or calculate it using the null parameter. Right: size uses the null; power uses the specified alternative with the same rejection rule.

Use an intermediate alternative.

Building a power function from the rejection event

An event description is the safest starting point for a power function. For example, a geometric rule rejecting when $T\ge b$ has rejection probability $(1-p)^{b-1}$ ; a rule rejecting when $T\le n$ has probability $1-(1-p)^n$ . These are derived geometric tail consequences, not separate booklet results.

Sometimes the geometric success probability comes from another model. Suppose each independent run of a program has a Poisson number of glitches with mean $\lambda>0$ . A fault-free run has probability

p=\mathrm P(\operatorname{Po}(\lambda)=0)=\mathrm e^{-\lambda}

The Poisson probability formula is supplied in the booklet. Assuming independent runs with the same mean, the trial $T$ of the first fault-free run is geometric with this success probability.

Power function for a fault-free-run rule

Test $H_0:\lambda=k$ against $H_1:\lambda<k$ , rejecting if a fault-free run occurs within $n$ independent runs. Then

\pi_n(\lambda)=\mathrm P_\lambda(T\le n) =1-(1-\mathrm e^{-\lambda})^n

The size is $\pi_n(k)$ ; the Type II error probability at a true alternative is $(1-\mathrm e^{-\lambda})^n$ . Here $n$ is fixed when the power function is evaluated. Not in the formula booklet: learn it.

The complement event is “every one of the first $n$ runs contains at least one glitch”. Each has probability $1-\mathrm e^{-\lambda}$ ; independence allows multiplication. This derives the result directly from the event, without working backwards from a printed answer.

Worked example 3: ten opportunities for a fault-free run

Question. Let $k=6$ and reject if the first fault-free run occurs within $10$ runs. Find the size and the power at true mean $\lambda=2$ , to four decimal places.

Plan. Substitute the null and alternative glitch means into the same function with $n=10$ .

\begin{aligned} \text{size}&=\pi_{10}(6) =1-(1-\mathrm e^{-6})^{10} =0.0245128\ldots\\ \pi_{10}(2)&=1-(1-\mathrm e^{-2})^{10} =0.7663975\ldots \end{aligned}

Why this step? A lower glitch mean increases the probability of a fault-free run, making this short-wait rejection event more likely.

Answer. Size $=0.0245$ , power at $\lambda=2$ $=0.7664$ .

Check. Both are probabilities between $0$ and $1$ , and the rejection probability is much larger under the substantial improvement.

Worked example 4 changes only the trial limit from $10$ to $20$ . This compares two different tests; within either test the limit stays fixed while $\lambda$ varies.

Worked example 4: twice as many opportunities

Question. Repeat the size and power calculation for rejection within $20$ runs.

Plan. Replace $n=10$ by $n=20$ , retaining the two population means.

\begin{aligned} \text{size}&=1-(1-\mathrm e^{-6})^{20} =0.0484248\ldots\\ \pi_{20}(2)&=1-(1-\mathrm e^{-2})^{20} =0.9454298\ldots \end{aligned}

Why this step? More chances to see a fault-free run increase detection, but also increase false rejections under the null.

Answer. Size $=0.0484$ , power at $\lambda=2$ $=0.9454$ .

Check. Both size and power exceed their values for $10$ runs. The new size is still below $0.05$ , using its unrounded value.

A probability of a zero count comes from one Poisson run; the probability of the first such run comes from a geometric model across independent runs. Keeping those levels separate prevents a common modelling error.

Watch out

Wrong: $\pi_n(\lambda)=\mathrm e^{-\lambda}$ for rejection within $n$ runs. This is the chance of one fault-free run, not the chance of seeing one within the whole limit. Right: complement failure in every independent run to obtain $1-(1-\mathrm e^{-\lambda})^n$ , holding $n$ fixed as $\lambda$ varies.

Deriving the event across runs makes clear which quantity is being varied.

Problem-solving

Describe rejection, find its probability under a general true parameter, then substitute the null for size or an alternative for power. A power function belongs to a fixed rule.

Choose an intermediate trial limit.

Meeting a detection target without too many false alarms

A design can require both a small Type II error at an important alternative and an upper limit on size. Satisfying only one condition is not enough.

Retain the program model, null mean $6$ , and a true alternative mean $2$ . Write

q_1=1-\mathrm e^{-2},\qquad q_0=1-\mathrm e^{-6}

where each $q$ is the probability that one run is not fault-free. Both lie between $0$ and $1$ .

Integer limits from Type II error and size

For rejection on a fault-free run within $n$ runs, requiring $\beta<\delta$ at the chosen alternative gives

q_1^n<\delta \quad\Longrightarrow\quad n>\frac{\ln\delta}{\ln q_1}

Requiring size at most $\alpha$ gives

1-q_0^n\le\alpha \quad\Longrightarrow\quad n\le\frac{\ln(1-\alpha)}{\ln q_0}

Here $0<q_0,q_1,\alpha,\delta<1$ , and $n$ is a positive integer. Both logarithm denominators are negative, so the inequality directions matter. Not in the formula booklet: learn it.

Continue this lesson.

You’re previewing NovaLearn without an account. Create a free account to read the rest.

Create free account

Last Accessed

Core Pure Mathematics

Further Pure Mathematics 1

Further Pure Mathematics 2

Further Statistics 1

Further Statistics 2

Further Mechanics 1

Further Mechanics 2

Decision Mathematics 1

Decision Mathematics 2

FS1 8.1 - Size and power of a test

A rejection probability viewed across possible populations

Building a power function from the rejection event

Meeting a detection target without too many false alarms

Continue this lesson.

Learn faster with Nova