Statistics & critical appraisal

Section oneChoosing the right test

You never really choose a test. You describe your data accurately, and the data chooses the test for you.

Every test answers the same question: could this difference have arisen by chance alone? The p-value is the answer. The tests differ only in the shape of data they are built to handle, in the way a screwdriver and a spanner differ by the fastener in front of them.

Step one: what kind of data is the outcome?

Categorical (nominal) data sorts patients into buckets — infection yes or no, blood group A, B, AB or O. All you can do is count how many sit in each bucket. There is no average blood group.

Ordinal data can be ranked, but the gaps between ranks are not equal. ASA grade, Clavien-Dindo, a pain score, a Likert scale. ASA 4 is worse than ASA 2 but not twice as bad, so an average ASA of 2.7 means nothing.

Continuous data is measured on a real scale with equal intervals — haemoglobin, BMI, operating time — so it can be meaningfully averaged.

Discrete data is a count: lymph nodes retrieved, previous operations, polyps found. Whole numbers only, but the intervals are equal and real, so it behaves like continuous data for the purpose of choosing a test.

Displaying data

The same data type that picks your statistical test also picks your chart. Match the picture to what you actually want the reader to see.

• Histogram — the distribution of one continuous variable: its shape, spread and skew.
• Box-and-whisker — the same continuous data summarised (median, IQR, range and outliers), and the best choice for comparing several groups side by side.
• Scatter plot — the relationship between two continuous variables (then correlation or regression).
• Line chart — a value changing over ordered time.
• Bar chart — counts or proportions across categories. Pie charts only for a simple parts-of-a-whole, and best avoided in papers.

Step two: normal or skewed?

A normal distribution is the symmetrical bell curve. Length of stay is not: most patients go home in three days and a few stay sixty, dragging a long tail to the right. The mean is bullied by those outliers; the median is not.

Parametric tests use the actual values and the mean, and assume normality. Non-parametric tests discard the raw values and use ranks — first, second, third — which is why they are immune to skew and outliers and why they also work for ordinal data. They are the safer option, at the cost of less statistical power.

Step three: paired or independent?

Independent means two separate sets of people. Paired means the same patient measured twice, or deliberately matched pairs. Pairing removes all the between-person noise, but only if you tell the test you have done it.

The paired equivalents

Relationships rather than differences

When there are no groups, only two variables measured in the same patients, the question is correlation. Pearson's r for two normally distributed continuous variables in a straight-line relationship; Spearman's rank when either variable is skewed or ordinal. Both run from −1 to +1. Neither says anything about causation, and neither describes the steepness of the line.

If the stem is about prediction rather than association, you are in regression: linear for a continuous outcome, logistic for a binary one, Cox for time to an event, Poisson for counts.

Decoding the stem

Two habits resolve most stems: count the groups first, then look for the word median.

Section twoInterpreting results

More marks live here than anywhere else in the topic, and most of them turn on definitions rather than arithmetic.

Describing data

The mean is the average, the median the middle value, the mode the commonest. In a symmetrical distribution all three coincide. In a right-skewed distribution they separate: mode, then median, then mean.

The standard deviation describes how widely individual patients scatter. In a normal distribution, 68% lie within one SD of the mean, 95% within two, and 99.7% within three. So a mean of 45 with an SD of 10 puts about 95% of patients between 25 and 65.

The null hypothesis and the p-value

Every study begins by assuming the boring answer: there is no difference. Think of a criminal trial. The defendant is presumed innocent, the data are the evidence, and the p-value answers a single question — if the defendant really were innocent, how likely is evidence this incriminating?

So p = 0.03 means that if there were genuinely no difference, a result at least this extreme would arise 3% of the time. It does not mean there is a 3% chance the null hypothesis is true, and it says nothing about whether the difference matters. In 50,000 patients, a 1 mmHg difference will be highly significant and entirely useless.

The two ways to be wrong

A type I error convicts the innocent, and its probability is alpha, conventionally 0.05. Run twenty comparisons at that threshold and one will be significant by luck alone, which is why ANOVA exists rather than many t-tests, and why the Bonferroni correction divides alpha by the number of comparisons.

A type II error acquits the guilty, and is almost always caused by too few patients. Power is 1 minus beta, conventionally set at 80%. It rises with a larger sample, a larger true effect, less variability and a higher accepted alpha. A sample size calculation needs four ingredients: alpha, power, the minimum clinically important difference, and the expected variability.

Confidence intervals

A p-value gives a verdict; a confidence interval gives the range of effects the data are compatible with, which is far more useful. The move in an exam is to identify the value of no effect and see whether the interval crosses it. For ratios — odds ratio, relative risk, hazard ratio — that value is 1. For differences it is 0. Crossing it means the result is not significant, and p will exceed 0.05. The two always agree.

Risk and benefit

Take mortality of 12% with standard care and 8% with a new treatment.

• Absolute risk reduction = 12 − 8 = 4%
• Relative risk = 8 ÷ 12 = 0.67
• Relative risk reduction = 4 ÷ 12 = 33%
• Number needed to treat = 1 ÷ 0.04 = 25

Relative figures always sound better than absolute ones because they conceal the baseline. Cutting risk from 3% to 2% is also a 33% relative reduction, but the ARR is 1% and the NNT is 100. A percentage reduction quoted without a baseline is the trick being played.

Odds ratio or relative risk? Risk is events over everyone; odds is events over non-events. Relative risk needs the true underlying risk, so it is available only from cohort studies and trials. A case-control study fixes the number of cases in advance, so the odds ratio is the only option. When the outcome is rare the two are close; when it is common the odds ratio exaggerates.

Diagnostic accuracy

Draw the two-by-two table and everything falls out of it, depending on which direction you read.

Reading down the columns starts with the truth. Sensitivity is TP ÷ (TP + FN): of everyone with the disease, the proportion caught. Specificity is TN ÷ (TN + FP). These belong to the test and do not change with the population.

Reading across the rows starts with the result, which is what happens in clinic. PPV is TP ÷ (TP + FP), NPV is TN ÷ (TN + FN). These depend heavily on prevalence: in a low-prevalence population there are far more well people available to generate false positives, so PPV collapses while NPV rises. That is the central difficulty of screening.

SnNout: a Sensitive test, Negative, rules out. SpPin: a Specific test, Positive, rules in. Plotting sensitivity against 1 − specificity across thresholds gives the ROC curve; the area underneath summarises performance, with 0.5 a coin toss and above 0.8 considered good. Likelihood ratios express how far a result shifts pre-test probability, and a positive LR above 10 or negative LR below 0.1 is convincing.

Bias and confounding

Bias is systematic error built into the design, and no analysis fixes it afterwards. Selection bias: the people studied are unrepresentative. Recall bias: patients with disease remember exposures more thoroughly. Detection bias: the assessor's expectations colour the measurement, which blinding prevents. Attrition bias: dropouts differ from those who remain. Publication bias: negative trials never appear. Lead-time bias: screening moves the date of diagnosis earlier without changing the date of death. Length-time bias: screening preferentially catches indolent tumours because they linger in a detectable state.

Confounding is different. A third variable is independently associated with both exposure and outcome, manufacturing an apparent relationship. Unlike bias it can be handled — by randomisation at design, or by multivariable regression, stratification or matching at analysis.

Intention-to-treat analyses every patient in the arm they were randomised to, whatever they received. It preserves randomisation and gives a conservative, pragmatic estimate. Per-protocol analysis breaks randomisation and flatters the intervention.

Section threeStudy design and levels of evidence

Two questions define every design: did the investigator assign the exposure, or only watch? And which way did the enquiry travel in time?

The designs

Case report and case series. One patient or a handful, with no comparison group, so effectiveness can never be claimed. Hypothesis-generating only.

Cross-sectional. A snapshot at one moment. Exposure and outcome measured together, so temporal sequence cannot be established. Measures prevalence, not incidence. Surveys and diagnostic accuracy studies are cross-sectional.

Case-control. Start with people who have the outcome, find comparable controls, look backwards for exposure. Fast, cheap, and the design of choice for rare diseases or long latency. Vulnerable to recall bias, and choosing controls is genuinely difficult. Because the number of cases was chosen by the investigator, only an odds ratio can be calculated.

Cohort. Start with exposed and unexposed groups and follow forwards. Establishes sequence, measures incidence and therefore relative risk, allows several outcomes from one exposure, and suits rare exposures. Expensive, slow, and vulnerable to attrition and confounding.

Randomised controlled trial. The investigator assigns the exposure. Its unique strength, worth stating precisely, is that randomisation balances known and unknown confounders. Statistical adjustment can only handle confounders somebody thought to measure. Nothing else does this.

Two things that get confused: allocation concealment happens before randomisation and stops the recruiter foreseeing the next allocation, achieved with sealed opaque envelopes or a central service. Blinding happens afterwards and stops knowledge of the assigned treatment influencing care or assessment. A surgical trial may be impossible to blind yet have impeccable allocation concealment.

Weaknesses: expense, time, ethical limits, and often poor external validity because strict entry criteria produce a trial population unlike your clinic. Surgery is especially hard to randomise because of learning curves, unblindable operators, evolving technique and strong preferences.

Systematic review and meta-analysis sit at the top, but are hostage to the quality of what they contain.

Two extras

Ecological studies compare populations rather than individuals — national fat consumption against national cancer rates. The pitfall is the ecological fallacy: assuming a population-level association holds for the individuals within it.

Nested case-control studies draw cases and controls from within an existing cohort, keeping the efficiency of case-control while avoiding recall bias, because exposure was recorded before anyone developed the outcome.

Levels of evidence

Grades of recommendation follow: A from level 1, B from levels 2 to 3, C from level 4, D from level 5.

GRADE rates certainty as high, moderate, low or very low. Trials start high and observational studies start low, but both move. Downgrade for risk of bias, inconsistency, indirectness, imprecision or publication bias. Upgrade observational evidence for a very large effect, a dose-response gradient, or where plausible confounding would only have reduced the observed effect.