4.2.3.3 - Probability and significance

Why probability matters

Inferential testing helps psychologists decide whether a pattern in a sample is convincing enough to treat as evidence about the wider population. A researcher might find that one therapy group improves more than another, or that two variables are correlated. The key question is not just "is there a difference or relationship in these data?" but "how likely would a result this extreme be if the null hypothesis were true?"

For a simple finite event, a probability of 0 means impossible and 1 means certain; $0.05=5/100=1/20=5\%$, whereas $0.01=1/100=1\%$. Multiply a probability by 100 to express it as a percentage. For equally likely outcomes, probability is the number of favourable outcomes divided by the total number of possible outcomes. Under a sign-test null model, a non-tied change has probability $1/2$ of being positive. If two pairs are independent, the probability that both signs are positive is $1/2\times1/2=1/4$. Sign tables consider the relevant pattern and more extreme patterns; the p value is not simply $S/N$.

The significance level, $\alpha$, is chosen before examining the results. The p value is calculated from the data under the null hypothesis; it is not the probability that the null hypothesis is true. The conventional threshold is $\alpha=0.05$, so a result is significant when $p\leq0.05$. This means that, if the null hypothesis were true, the probability of obtaining this result, or a more extreme result, is 5% or less. It does not mean there is a 95% chance that the alternative hypothesis is true, and it does not prove the researcher's claim. It means the result has passed a pre-set threshold for rejecting the null hypothesis.

A stricter threshold of $\alpha=0.01$ requires $p\leq0.01$: a 1% or lower probability of obtaining the result or a more extreme result, assuming the null hypothesis is true. This may be chosen when the cost of a false positive would be serious, such as claiming that a clinical treatment works when it does not. A more lenient level, such as $p\leq0.10$, makes it easier to find significance but increases the risk of treating chance findings as meaningful.

Counting outcomes under a simple null model

Imagine five independent non-tied changes, each equally likely to be + or − under the null hypothesis. There are $2^5=32$ equally likely sign sequences. Only one is all plus, so its probability is $1/32=0.03125$. A prediction made in advance of a positive change could treat all plus as significant at 0.05. A two-tailed test must also count the equally extreme all-minus sequence: $2/32=0.0625$. This fails the 0.05 threshold. The data have not changed; the hypothesis asked a different question. For less extreme results the calculation must include all qualifying outcomes, which is why a table is useful. This example also explains why a dash in the sign table is not a critical value of zero.

Using statistical tables

In AQA Psychology, students may be given a statistical table or enough information from one. The table does not calculate the result for you. It gives a critical value, which is the boundary used to decide whether the observed or calculated value from the study is statistically significant.

To use a table accurately, work through the lookup information before you compare numbers:

A common mistake is to grab the first critical value that looks close. Slow down. One-tailed versus two-tailed, N versus df, and the direction of the comparison rule can all change the conclusion.

For the sign test, $N$ is the number of pairs after excluding ties. Use this effective sample size, not the original number of participants.

Reading a sign-test table

The following critical values were calculated from the sign test's binomial model. Significance requires $S$ to be at most the critical value. For a one-tailed test, the predominant change must also match the direction predicted before data collection.

A dash means no possible value of $S$ reaches that significance level for that sample size. It does not mean zero.

Suppose 14 learners complete a memory task before and after practice. Two scores are tied; ten improve and two decline. Then $N=12$ and $S=2$. A prediction of a difference in either direction uses the two-tailed $0.05$ column. At row 12 the critical value is 2: equality counts, so the result is significant. At $0.01$ the critical value is 1, so this same result is not significant at that stricter level. Do not report an exact p value from a table that only supplies thresholds.

For a prediction that practice improves memory, the one-tailed column is justified only if that direction was specified in advance. A strong change in the opposite direction cannot support that prediction, even if the smaller sign count meets the table boundary.

Decision rules

The safest wording is: follow the rule printed with the statistical table. AQA questions often expect students to know the broad pattern:

Example 1: in a two-tailed test, a researcher reports a calculated Spearman's rho of +0.62. The correct table entry for N, p level and tail gives a critical value of 0.59, and the table says rho must be equal to or greater than the critical value. Since $0.62\geq0.59$, the result is significant. The researcher rejects the null hypothesis and concludes that there is a significant positive correlation.

For a two-tailed test with a positive critical value, compare the absolute magnitude: $|-0.62|=0.62$. The result would be a significant negative correlation. For a one-tailed prediction of a positive correlation, a negative correlation does not support the prediction. Never discard the sign before checking the tail and predicted direction. The same signed-direction caution applies to t statistics.

Example 2: a sign-test result gives S = 6. The correct critical value is 4, and the table says S must be equal to or less than the critical value. Since $6>4$, the result is not significant. The researcher retains the null hypothesis because the observed result is not unlikely enough under the null hypothesis. Retaining (or failing to reject) the null is not proof that it is true.

Example 3: a Mann-Whitney U result gives U = 17. The correct critical value is 22, and the table says U must be equal to or less than the critical value. Since $17\leq22$, the result is significant. The researcher rejects the null hypothesis and concludes that there is a significant difference between the two groups.

A strong conclusion names the comparison, the significance level and the hypothesis decision:

"The magnitude of rho is 0.62, above the supplied critical value of 0.59 at the 0.05 level for this two-tailed test. The result is significant, so the null hypothesis is rejected."