1.3 - Binomial Sign test

Use the direction of paired changes

The Binomial Sign test tests a paired difference when each informative pair can be classified into one of two directions.

The same people may make a choice in two conditions, such as accepting or declining help before and after a message. Alternatively, numerical paired scores can be reduced to increased or decreased. The aim is a difference, the design is related, and the analysed outcomes are two nominal categories, plus and minus. Keep each person's two observations paired: counting all choices as independent frequencies would lose the relationship between them.

The pairs must be independent of one another. Under the usual null, an informative pair has probability 0.5 of each direction, with the same probability across pairs. This is the assumption behind the ordinary sign-test table. It does not mean every categorical problem has equal probabilities: a different null probability would require a different binomial analysis.

A sign test does not require normally distributed or symmetric difference sizes, because sizes are discarded. It can therefore analyse directional changes when only their order is meaningful. This flexibility costs information: increases of 1 and 20 both count as a single plus. Where ranked difference sizes and symmetry are defensible, Wilcoxon can use more information. Neither procedure compensates for practice effects, biased sampling or poor measurement.

Count signs and remove unchanged pairs

Define what plus and minus mean before counting, and exclude every pair with no change.

For numerical scores, use a consistent subtraction such as after minus before. For accept/decline choices, define plus as a change from decline to accept, and minus as accept to decline. Accept/accept and decline/decline are both unchanged. Neither receives a sign. A zero numerical change is also excluded.

An original recall example produces these paired changes:

There are ten pluses, one minus and one unchanged pair. The effective $N=10+1=11$, rather than the original 12 people or 24 individual measurements. The observed S is the less frequent sign count, so $S=1$. This procedure counts signs, rather than adding the original differences or ranking their sizes.

Check that pluses, minuses and excluded pairs add to the original number of pairs. If every pair is unchanged, N=0: the usual sign test cannot provide the intended significance decision. Exclusion is a calculation rule, not permission to hide unchanged outcomes; report the original sample and number excluded too.

Why an imbalance can be unlikely

Binomial probabilities describe how often a sign imbalance would arise under the equal-direction null model.

Imagine four independent non-zero changes. Under the null, the chance that all four are plus is $0.5^4=1/16=0.0625$. There are 16 equally likely sign sequences. If the alternative predicted increases beforehand, all-plus is the extreme result in that predicted direction. For a non-directional alternative, all-minus is equally extreme, so the combined probability of all four signs agreeing in either direction is $2/16=0.125$.

That example explains why tail choice matters. A one-tailed analysis considers the predicted direction; a two-tailed analysis considers sufficiently extreme outcomes in either direction. Choose the hypothesis direction before viewing the result. A large decrease does not support an alternative predicting an increase, even if the smaller sign count is low.

For less extreme results, the significance probability includes outcomes at least as extreme, not just the exact observed sequence. A table has already combined the relevant binomial probabilities to identify the largest S that meets its threshold. You need not reconstruct its probabilities when the task supplies the critical value.

The p value conditions on the null assumptions. For example, $p=0.03$ means outcomes at least this extreme have probability 3% under that model. It does not mean a 3% probability that the null is true, or that only 3% of the observations are accidental.

Match N, direction and significance level

Sign-test significance uses a lower-tail decision: S must be at or below the appropriate critical value.

This supplied OCR workbook extract is for a one-tailed test:

Suppose the recall example's alternative predicted improvement before data collection. $N=11$ and $S=1$. At the 5% level the critical S is 2; $1\leq2$, and the majority of changes is positive as predicted. Reject the null in favour of evidence of increased recall. It also meets the 1% threshold of 1, including equality. S has no score unit: it is a count of signs.

A two-tailed table has different cutoffs. For $N=11$ under the usual exact equal-direction model, the two-tailed 5% cutoff is 1. To see the difference, the probability of zero or one plus is $(1+11)/2^{11}$; including the symmetric other tail gives $24/2048=0.01171875$. Adding outcomes with two pluses increases that two-tail probability to $134/2048=0.0654296875$, which exceeds 0.05. Thus $S=2$ meets the supplied one-tailed 5% threshold but does not meet the two-tailed one. Labels must accompany every table.

If S exceeds the correct critical value, fail to reject the null. This does not prove that there is no change. The sign test may miss a genuine effect because its sample is small, many pairs were unchanged or size information was discarded. Conversely, significance is not proof of a causal or useful improvement; assess the study's design and the size of the original changes.