Choose a statistical method without starting from the test

A concrete reason to pause

Thirty records, one vague question, several incompatible answers#

AURORA-30 is a wholly synthetic teaching dataset: 30 fictional records constructed for methods training. It contains no real patients, institutions, interventions, or clinical results, and nothing calculated from it is clinical evidence. In the scenario, adults are followed for 30 days after a standardized discharge program or usual care. A colleague sends baseline and day-30 symptom scores and asks, “Did the program help?” It is tempting to open a test menu and compare whichever two numeric columns are visible.

But the rows do not decide the question. A between-group contrast at day 30 compares program with usual care; a baseline-to-day-30 contrast asks whether the same people changed over time; a between-group comparison of change asks whether those trajectories differed. All three can produce correct arithmetic and still support different sentences in the report. Without an explicit trail, the analyst can begin with one question and publish the answer to another without any software error announcing the switch.

Why evidence comes before technique

A result must carry the path that makes it interpretable#

In health research, the practical risk is not limited to choosing the wrong test from a menu. The unit of analysis, time point, missing-data handling, target effect, or primary claim can shift while the work is underway. A precise p-value cannot repair that drift. It may only make the wrong scientific answer look more authoritative.

An evidence trail keeps the question, data decisions, estimate, assumptions, diagnostics, and limits connected. It lets a collaborator or reviewer reconstruct the route, challenge a mismatch, and distinguish a prespecified conclusion from an exploratory finding. Technical depth matters here because every number inherits meaning from those decisions—not because complexity is valuable by itself.

Start before the software

Write the question and target effect#

A question such as “does the intervention work?” is not yet an analysis contract. Name the population, intervention or exposure, comparison, outcome, time point, and the effect you intend to estimate. A mean difference at day 30, an odds ratio over follow-up, and a survival contrast are different targets even when they come from the same dataset.

The estimand connects the scientific question to what the analysis should estimate. ICH E9(R1) formalizes this connection for clinical trials; outside that scope, the same discipline remains useful, but it does not turn an observational comparison into a causal effect.

The design changes the method

Count units, not rows#

1. 01

   Independent groups

   Different people or independent units contribute to each group.

2. 02

   Paired observations

   The same person or a matched pair contributes two linked measurements.

3. 03

   Repeated measures

   Several times or conditions sit inside the same unit and share correlation.

4. 04

   Clusters

   People may also be nested in clinics, households, wards, or sites.

The decision record

Turn the hidden choices into a one-page analysis contract#

Now the contract has a reason to exist: it is a compact, verifiable memory of the decisions that prevents the original request from changing silently while the analysis is built. Record the outcome scale, number of groups or times, dependence, target effect, sample-size justification, missing-data plan, family of claims that may require multiplicity control, diagnostics, and reporting package. Separate primary, prespecified claims from exploratory comparisons. Empty fields are useful because they expose decisions that have not been made.

The analysis contract is not a legal document, does not choose a method automatically, and is not paperwork to complete after the result. It narrows the defensible options before execution and gives a reviewer enough context to challenge a mismatch—for example, an independent-samples test applied to repeated measurements from the same participant.

{
  "study_id": "AURORA-30",
  "question": {
    "population_id": "eligible_participants",
    "outcome_id": "symptom_score_day_30",
    "comparison_ids": ["discharge_program", "usual_care"],
    "estimand_id": "mean_difference_day_30_program_minus_usual_care"
  },
  "structure": {
    "unit": "participant",
    "between_groups": "independent",
    "baseline_followup": "paired",
    "missingness_reviewed": true
  },
  "planning": {
    "sample_size_justification_id": "precision_for_target_effect",
    "primary_claim_ids": ["day_30_mean_difference"],
    "exploratory_comparisons_labeled": true
  },
  "report": {
    "required": ["estimate", "confidence_interval", "diagnostics", "limitation"],
    "causal_claim_allowed": false
  }
}

import copy
import json
import sys

REQUIRED_REPORT = {"estimate", "confidence_interval", "diagnostics", "limitation"}

def validate(contract):
    problems = []
    question = contract.get("question", {})
    structure = contract.get("structure", {})
    planning = contract.get("planning", {})
    report = contract.get("report", {})

    for key in ("population_id", "outcome_id", "comparison_ids", "estimand_id"):
        if not question.get(key):
            problems.append(f"question.{key} is required")
    if structure.get("unit") != "participant":
        problems.append("structure.unit must preserve the participant")
    if structure.get("between_groups") != "independent":
        problems.append("the day-30 group contrast must record independent groups")
    if structure.get("baseline_followup") != "paired":
        problems.append("baseline and follow-up must remain paired within participant")
    if structure.get("missingness_reviewed") is not True:
        problems.append("missingness review is required")
    if not planning.get("sample_size_justification_id"):
        problems.append("planning.sample_size_justification_id is required")
    if not planning.get("primary_claim_ids"):
        problems.append("planning.primary_claim_ids must name at least one primary claim")
    if planning.get("exploratory_comparisons_labeled") is not True:
        problems.append("exploratory comparisons must be labeled")
    missing_report = REQUIRED_REPORT - set(report.get("required", []))
    if missing_report:
        problems.append("report.required lacks: " + ", ".join(sorted(missing_report)))
    if report.get("causal_claim_allowed") is not False:
        problems.append("report.causal_claim_allowed must remain false for this teaching case")
    return problems

def show(label, contract):
    problems = validate(contract)
    status = "HOLD" if problems else "READY_FOR_METHOD_REVIEW"
    print(f"{label}: {status}")
    for problem in problems:
        print(f"  - {problem}")

with open(sys.argv[1], encoding="utf-8") as source:
    supplied = json.load(source)

print(f"{supplied.get('study_id', 'UNNAMED')} | structural analysis-contract check")
show("supplied_contract", supplied)

pairing_probe = copy.deepcopy(supplied)
pairing_probe["structure"]["baseline_followup"] = "independent"
show("pairing_regression_probe", pairing_probe)

planning_probe = copy.deepcopy(supplied)
planning_probe.pop("planning", None)
show("missing_planning_probe", planning_probe)

causal_probe = copy.deepcopy(supplied)
causal_probe["report"]["causal_claim_allowed"] = True
show("causal_claim_regression_probe", causal_probe)
print("scope: structural planning aid; no method selection or clinical authorization")

AURORA-30 | structural analysis-contract check
supplied_contract: READY_FOR_METHOD_REVIEW
pairing_regression_probe: HOLD
  - baseline and follow-up must remain paired within participant
missing_planning_probe: HOLD
  - planning.sample_size_justification_id is required
  - planning.primary_claim_ids must name at least one primary claim
  - exploratory comparisons must be labeled
causal_claim_regression_probe: HOLD
  - report.causal_claim_allowed must remain false for this teaching case
scope: structural planning aid; no method selection or clinical authorization

Assumptions are design claims

Diagnose the model, not normality by ritual#

A large p-value from a normality test does not prove normality, and a small p-value does not automatically require a rank test. Consider the design, residual structure, sample size, outliers, skew, variance pattern, measurement scale, and what parameter the method actually estimates. When relevant to the selected model, use visual checks such as Q–Q and residual-versus-fitted plots as diagnostics—not as mechanical proof that assumptions hold.

Rank-based methods are not generic “tests of medians.” Depending on the method and assumptions, they may address distributional ordering, probability of superiority, or a location shift; a median contrast requires additional conditions. When assumptions are doubtful, consider a method that targets the effect more directly, a robust analysis, or a prespecified sensitivity analysis.

Interpretation package

Do not let the p-value travel alone#

1. 01

   Estimate

   State the effect on a scale the question can defend.

2. 02

   Interval

   Show the uncertainty around that estimate.

3. 03

   Diagnostics

   Record the checks that matter for the selected method.

4. 04

   p-value

   Use it only when a prespecified hypothesis test is relevant.

5. 05

   Limitation

   Name what the design, missingness, bias, or data cannot establish.

Reusable checklist

Twelve questions before naming the method#

• ✓What is the question and estimand?
• ✓What population and unit of analysis are represented?
• ✓What is the outcome type and scale?
• ✓How many groups, times, or conditions exist?
• ✓Are observations independent, paired, repeated, or clustered?
• ✓How was the sample obtained, how was its size justified, and which biases are plausible?
• ✓Are missingness, outliers, units, dates, and coding reviewed?
• ✓Which assumptions are reasonable from design and data?
• ✓Was the analysis prespecified, which claims are primary or exploratory, and how many comparisons are planned?
• ✓Which method estimates the target effect directly?
• ✓Which effect, interval, and diagnostics accompany the result?
• ✓Which limitations prevent causal or broader claims?

Conclusion

Preserve the question, not only the result#

The central lesson is not that every health analysis needs a more sophisticated method. It is that the same data can support several mathematically correct analyses while only one matches the question the study promised to answer. A defensible result therefore carries its route: question, unit, target effect, assumptions, decisions about the data, estimate, uncertainty, diagnostics, and limits.

Before naming a method, write one sentence stating who is compared, on which outcome, at what time, and through which effect. If another qualified person cannot reconstruct the same analysis from that sentence and its decision record, the work is not ready for a test menu. The contract earns its value by making that gap visible early—when the question can still be repaired without rewriting the result.