Validation Suite

Every statistical method PsyStat Lab exposes is compared against a canonical reference on every commit. This page reads live from the CI-generated report.

Why this page exists

A statistical tool’s existential requirement is demonstrated numerical correctness, not asserted correctness. It is not enough to say “matches R.” A citable tool has to show which methods match, against which reference, to how many decimal places, verified when.

Every row below is a golden-number test that runs on every commit. If we ever change a formula and the output shifts, CI breaks and this page goes red until it’s fixed. When you cite a PsyStat Lab analysis in a paper, it’s bound to a specific commit and a specific validation snapshot — and both are recoverable years later.

A method’s absence from this page means “not yet validated,” not “wrong”: unvalidated methods run and produce output, but they carry no independent numerical guarantee yet. The roadmap section at the bottom describes how coverage grows.

Reference kinds

Each check declares which of four categories its expected value came from, in order of preference:

NIST A NIST Statistical Reference Datasets certified value, published by the U.S. National Institute of Standards and Technology to 15 significant figures. The strongest kind.
R The value R’s canonical function (stats::t.test, stats::lm, stats::wilcox.test, etc.) produces on the same input. The R code that reproduces the number is documented in references.py.
SCIPY-REF The value scipy 1.14 produces on the same input, frozen as a regression reference. The equivalent R command is documented so an independent reviewer can spot-check.
ANALYTICAL A closed-form value (e.g., mean of [1..5] = 3.0, Cramer’s V for a 2x2 = sqrt(chi²/n)). The strongest tolerance category — typically rtol=1e-10 or absolute equality up to floating-point.

Validated methods

Each row = one endpoint, its reference source, the dataset it was checked against, and how many independent statistics were verified. Click a row’s test ID to see the exact assertion in the source.

Method & endpoint	Reference kind	Dataset	Checks

How to verify independently

You do not have to trust this page. Every reference has a citation, and every reference’s R equivalent is documented in the source. A reasonable independent-verification workflow:

Read the reference for the method you care about in references.py. Copy the R code from its comment.
Run that R code against the same dataset in datasets.py. Compare R’s output to the frozen constant.
If they diverge, that’s a live discrepancy worth reporting — open an issue on GitHub and it will be triaged as a validation bug, not a cosmetic one.

NIST references are certified — if a NIST-referenced check diverges from PsyStat’s output, PsyStat is wrong, full stop.

Roadmap and honest coverage

PsyStat Lab currently exposes 278 endpoints across 77 routers. This page reflects 10 of those. The next tranches queued for validation are, roughly:

Effect sizes: Cohen’s d (both formulations), Glass’s delta, partial η², ω², rank-biserial r, Cliff’s delta.
ANOVA family: factorial ANOVA, repeated-measures ANOVA, ANCOVA, MANOVA (Wilks’ Λ and Pillai’s trace).
Regression family: multiple regression coefficient inference (NIST Longley & Filip datasets), logistic regression, Poisson regression, negative-binomial.
Psychometrics: Cronbach’s α, McDonald’s ω, CFA fit indices (CFI/TLI/RMSEA/SRMR), 2PL/3PL IRT item parameters.
Non-parametrics: Kruskal-Wallis H, Friedman’s χ², Kendall’s W, Fisher’s exact.
Bayesian: single-population µ posteriors, Bayes factor for one-sample and two-sample means (vs. R BayesFactor).

“Not on the roadmap” does not mean “not validated later”; it means we haven’t written the check yet. The pattern for adding one is documented in the validation README — contributions welcome.

Report last generated unknown. CI runs: GitHub Actions.