Sigma Statistics Validation

Baseline Characteristics / Descriptive Statistics

Validation objective

The descriptive statistics module was validated against an independent R reference implementation. The comparison covered missing-value preprocessing and point-decimal numeric input, automatic and overridden variable types, overall and grouped cohorts, missing and invalid numeric values, metric summaries, type-7 percentiles, categorical frequencies and modes, and data-quality warnings. The rendered table only rounds for display; the validation used the underlying full-precision values produced by the Sigma calculation core.

Validation design

A deterministic seed generator created 27 wide datasets at sample sizes n = 1, 2, 3, 4, 5, 10, 30, 100, 1000, with three independent replicates per sample size. These seeds were evaluated in 102 configurations covering no-group and grouped analyses, automatic statistic selection, and Show all statistics for all parameters.

Across the validation suite, the datasets covered normal, skewed, heavy-tailed, tied, constant, missing, and extreme-outlier numeric variables, a mixed numeric/text column, numeric and text binary variables, multi-level categorical variables, a numeric-coded category, a category with missing values, a unique subject identifier, and an all-missing column. Some grouped configurations included missing group assignments, which were retained in Overall and excluded from group-specific results. The smallest datasets necessarily did not realize every feature; for example, the extreme outlier was introduced from n ≥ 2, categorical missingness from n ≥ 4, and invalid mixed numeric values and missing group assignments from n ≥ 5.

The independent R comparison used point-decimal input throughout. Point, comma, and automatic decimal- separator branches were additionally exercised by JavaScript core unit tests; comma and automatic separator handling were therefore outside the independent R comparison.

Independent R reference

Reference results were calculated in R 4.5.2 using base-R formulas independent of the Sigma JavaScript implementation. The reference used mean, var, sd, qt, and quantile(type = 7), together with explicit formulas for SEM, confidence intervals, adjusted Fisher–Pearson skewness, adjusted excess kurtosis, missing percentages, frequencies, and modes. Both implementations emitted the same long-format result schema.

Comparison criteria

Counts, cohort labels, category levels, modes, type decisions, warning messages, and configuration metadata were required to match exactly. Numeric values were accepted when abs(Sigma − R) ≤ 1e−10 + 1e−10 × abs(R). This combined absolute and relative threshold protects near-zero results while remaining appropriate for the deliberately included very large outlier-driven moments and variances.

Validation results

R and Sigma each produced 256,822 keyed long-format result rows. All 256,822/256,822 rows passed, with 0 missing keys, 0 text mismatches, and 0 numeric failures. Agreement by result section was:

analysis metadata: 408/408,
cohort definitions and sizes: 396/396,
type detection and data quality: 10,710/10,710,
numeric statistics: 47,817/47,817,
percentiles: 20,493/20,493,
categorical summaries: 13,068/13,068, and
frequency rows: 163,930/163,930.

The largest absolute numeric difference was 4.8828125 × 10⁻⁴ for the sample variance of a deliberately extreme outlier dataset with variance approximately 5.0 × 10¹¹. Its scaled relative difference was only approximately 9.77 × 10⁻¹⁶. Across all results, the maximum scaled relative difference was approximately 1.19 × 10⁻¹³, well inside the predefined threshold.

Reproducibility

The validation bundle is stored under validation tests/descriptive_statistics. It contains the R seed generator, the independent R reference, the Sigma batch runner, the row-level comparison script, generated inputs and outputs, and a one-command validation runner. Re-running run_validation.sh regenerates all seeds and comparison artifacts from master seed 20260801.

Conclusion

The validation demonstrates exact structural agreement and numerical equivalence between Sigma and R across small-sample edge cases, large samples, missing and invalid values, grouping, automatic and manual type handling, distribution-shape statistics, custom percentiles, and categorical frequency calculations. The descriptive-statistics calculation core is therefore validated for the R-comparison scope and definitions described above. Decimal-comma and automatic separator handling have unit-test, but not independent R-comparison, coverage.

Normality Testing

Validation objective

The production JavaScript calculation core was compared with independent, established implementations of all four procedures in the tool: Shapiro–Wilk, the Lilliefors-corrected Kolmogorov–Smirnov test, Anderson–Darling, and D’Agostino–Pearson omnibus K². The comparison covered test statistics, p-values, estimability status, and the reject / do-not-reject decision at α = 0.05.

This supersedes the earlier tool-like comparison, in which several reference formulas had inadvertently reproduced Sigma’s own formulas. No cloned Sigma implementation was accepted as a reference in the present validation.

Validation design

The fixed validation corpus contains 350 datasets: seven scenarios (normal, skewed, heavy_tail, bimodal, outliers, ties, and degenerate), five sample sizes (8, 20, 100, 1000, and 5000), and ten deterministic replicates per combination. Of these, 340 datasets produced paired finite results for each test. The ten constant n = 8 datasets were non-estimable in both Sigma and the reference software.

Additional automated tests cover strict decimal-point and decimal-comma parsing, rejection of partially numeric and non-finite input, test-specific sample-size limits, constant samples, affine invariance, and the extreme Anderson–Darling branch A²* ≥ 10.

Independent reference methods

References were calculated with R 4.5.2 using stats::shapiro.test, and with nortest 1.0-4 using nortest::lillie.test and nortest::ad.test. D’Agostino–Pearson was independently compared with scipy.stats.normaltest from SciPy 1.17.0. The reference programs read the seed CSV files directly; Sigma results were emitted separately by the same calculation core loaded by the production page.

IEEE-754 arithmetic and elementary math functions can differ slightly across JavaScript, R, and Python. Bitwise identity was therefore not required; differences and inferential decisions were compared at full available precision without first rounding displayed values.

Validation metrics

For each procedure, the comparator measured the number of finite pairs, mean and maximum absolute differences in the statistic and p-value, disagreement in finite / non-finite status, and decision disagreement at α = 0.05.

Numerical results

Shapiro–Wilk (340 pairs): mean / maximum absolute difference 9.62 × 10⁻¹¹ / 3.95 × 10⁻¹⁰ for W, and 6.34 × 10⁻¹⁰ / 1.01 × 10⁻⁸ for the p-value.
Lilliefors (340 pairs): mean / maximum absolute difference 2.78 × 10⁻⁹ / 8.50 × 10⁻⁸ for D, and 8.72 × 10⁻¹⁴ / 3.09 × 10⁻¹² for the p-value.
Anderson–Darling (340 pairs): mean / maximum absolute difference 5.41 × 10⁻⁷ / 1.52 × 10⁻⁵ for adjusted A²*, and 2.89 × 10⁻¹³ / 3.29 × 10⁻¹¹ for the p-value.
D’Agostino–Pearson (340 pairs): mean / maximum absolute difference 0.0240 / 0.4149 for K², and 1.02 × 10⁻¹⁴ / 6.65 × 10⁻¹³ for the p-value. The largest statistic differences occurred only in deliberately near-degenerate samples with K² ≈ 17,821; both implementations returned p-values numerically equal to zero. Among all non-degenerate scenarios, the maximum absolute K² difference was 2.42 × 10⁻⁹.

Decision agreement at α = 0.05

There were zero decision mismatches and zero estimability-pattern mismatches for every procedure:

Shapiro–Wilk: 0 mismatches
Lilliefors-corrected KS: 0 mismatches
Anderson–Darling: 0 mismatches
D’Agostino–Pearson: 0 mismatches

Implementation-specific notes

The test explicitly labelled Lilliefors is the fitted-normal variant of the KS statistic, not the classical test with externally fixed normal parameters. Anderson–Darling displays the adjusted statistic A²*. Shapiro–Wilk is limited to 3 ≤ n ≤ 5000; Lilliefors requires n ≥ 5; Anderson–Darling reports its statistic from n ≥ 3 and its p-value from n ≥ 8; D’Agostino–Pearson requires n ≥ 8.

Reproducibility

The seed corpus and independent comparison scripts are stored under validation tests/normality. The validation runner loads the production normality_core.js, invokes the unmodified R package functions and SciPy reference, and calculates all differences from full-precision outputs.

Conclusion

Within the validated definitions and sample-size ranges, Sigma showed close numerical agreement with the independent R and SciPy references and identical significance decisions across all 340 estimable datasets. The validation supports the corrected calculation core; it does not replace appropriate judgment about independence, discreteness, sample size, or whether a normality test is scientifically informative for a particular analysis.

Chi-square (χ²) Test (2×2) and Fisher’s Exact Test

Validation objective

The Sigma 2×2 contingency table module was validated against R with respect to the statistical procedures implemented in the tool: Pearson χ² test with Yates’ continuity correction, two-sided Fisher’s exact test, and the associated contingency-table quantities required for these analyses. Sigma computes Yates-corrected χ² p-values, two-sided Fisher p-values, expected cell counts, a low-expected-count flag, and effect estimates for binary outcomes in 2×2 tables.

Reference methods

Validation was performed against standard R implementations for 2×2 table analysis. The χ² comparison used Pearson’s chi-square test with Yates’ continuity correction, and the exact-test comparison used two-sided Fisher’s exact test. In addition, the underlying 2×2 cell counts and expected counts were compared directly between Sigma and R.

Validation dataset

The validation dataset comprised 900 paired analysis cases. All 900 cases could be matched and processed between Sigma and R. The corrected production core and R both classified 771 tables as estimable for the Yates-corrected χ² test and the same 129 degenerate tables as non-estimable. Fisher’s exact test remained estimable in all 900 cases.

Validation metrics

Agreement between Sigma and R was assessed at four levels:

agreement in the number of usable observations,
agreement in the observed 2×2 cell counts,
agreement in expected counts and the low-expected-count flag, and
agreement in χ² and Fisher p-values, including the final significance decision at α = 0.05.

Results of the validation

Agreement between Sigma and R was exact for the basic table structure. The number of usable observations matched in 900/900 cases. The observed 2×2 cell counts matched exactly in 900/900 cases. The low-expected-count flag also matched in 900/900 cases. Expected counts were numerically identical up to floating-point precision, with a maximum absolute difference of approximately 5.0 × 10⁻¹².

Chi-square test validation

For the Yates-corrected χ² test, R returned a comparable finite result in 771 cases. In the remaining 129 cases, both R and Sigma reported no χ² result because a row or outcome margin was zero. Across the 771 comparable cases, the χ² statistic showed a maximum absolute difference of approximately 4.8 × 10⁻¹² between Sigma and R. Direct evaluation of the one-degree-of-freedom χ² survival function reduced the maximum absolute p-value difference to approximately 2.4 × 10⁻¹³.

A separate stress grid comprised 11,562 structured, zero-cell, random, and extreme-count 2×2 tables. Sigma and R agreed on all 291 non-estimable χ² constellations. Across the 11,271 estimable tables, the maximum absolute χ² p-value difference was approximately 3.8 × 10⁻¹⁴, with no significance-decision mismatch.

Fisher’s exact test validation

For Fisher’s exact test, all 900 cases were comparable. The maximum absolute difference in the two-sided Fisher p-value between Sigma and R was approximately 1.8 × 10⁻¹¹. In the additional 2×2 stress grid, the maximum absolute Fisher p-value difference was approximately 4.5 × 10⁻¹³. Fisher probabilities are evaluated on the logarithmic scale and summed with a log-sum-exp formulation, providing stable results even for sparse and large-count tables.

Effect estimates and edge cases

Odds ratios, Wald confidence intervals, risk ratios, and risk differences were compared with the corresponding R formulae throughout the stress grid. For finite comparable estimates, the maximum relative difference was below 5.9 × 10⁻¹⁵. The Haldane–Anscombe correction is applied to the odds ratio and its confidence interval when an observed cell is zero but both group and outcome margins remain positive. Effects are explicitly reported as non-estimable when a group or outcome margin is zero. The interface additionally requires an explicit event-category mapping for every selected outcome; reversing that mapping correctly reverses the event counts and effect direction while leaving χ² and Fisher p-values unchanged.

Decision agreement

At the conventional significance level of α = 0.05, Sigma and R yielded identical inferential decisions for all comparable analyses. There were 0 decision mismatches for the Yates-corrected χ² test and 0 decision mismatches for Fisher’s exact test in both validation suites.

Interpretation

The validation demonstrates that Sigma reproduces the same 2×2 contingency tables as R and yields numerically equivalent results for both Yates-corrected χ² testing and two-sided Fisher’s exact testing. Degenerate margins, zero cells, and explicit event coding are handled transparently rather than being silently converted into apparently valid estimates. The remaining numerical differences are negligible floating-point effects.

Conclusion

The corrected Sigma Chi-square / Fisher module showed exact agreement with R for usable sample sizes, contingency-table cell counts, estimability classification, and low-expected-count classification, together with essentially machine-precision p-value agreement and complete agreement in significance decisions. These results support the validity of Sigma’s implementation of 2×2 Pearson χ² testing with Yates’ correction, two-sided Fisher’s exact test, and the reported binary effect measures for routine analytical use.

Unpaired (Independent Samples) t-Test

Validation objective

The Sigma unpaired t-test module was validated against R with respect to the statistical procedures implemented in the tool: the Welch t-test for unequal variances and the Student t-test with pooled variance under the equal-variance assumption. In addition to p-values, the validation covered group-specific descriptive statistics, mean differences, confidence intervals, and the handling of invalid cases. Sigma reports the Welch two-sided p-value as the primary inferential quantity and additionally computes the pooled-variance t-test, confidence intervals, and standardized effect sizes.

Reference methods

Validation was performed against standard R implementations for independent-samples t-tests. Welch’s test was compared to the unequal-variance formulation in R, and the pooled-variance test was compared to the equal-variance formulation. Group means, standard deviations, sample sizes, mean differences, confidence intervals, and p-values were compared directly between Sigma and R.

Validation dataset

The validation dataset comprised 900 paired analysis cases. Of these, 896 cases were valid in both Sigma and R, while 4 cases were invalid because fewer than two usable observations remained in at least one group after filtering. Sigma classified these invalid cases consistently with the reference results.

Validation metrics

Agreement between Sigma and R was assessed at the following levels:

agreement in the number of usable observations per analysis,
agreement in group-specific descriptive statistics,
agreement in Welch test statistics, degrees of freedom, confidence intervals, and p-values,
agreement in the pooled-variance t-test results, and
agreement in the final significance decision at α = 0.05.

Results of the validation

Agreement between Sigma and R was exact for case validity and usable sample sizes. Among the 896 valid analyses, the number of usable observations matched in all cases. The remaining 4 analyses were consistently classified as invalid because at least one group contained fewer than two usable observations after exclusion of missing or non-numeric values.

Group-specific descriptive statistics were numerically identical up to floating-point precision. The maximum absolute mean difference was approximately 9.6 × 10⁻¹⁵, and the maximum absolute standard-deviation difference was approximately 4.8 × 10⁻¹⁴. These differences are negligible and reflect only standard floating-point behavior.

Welch t-test validation

Welch’s test showed excellent agreement between Sigma and R across all valid cases. The two-sided Welch p-value differed only minimally, with a maximum absolute difference of approximately 7.4 × 10⁻¹². After alignment to the same subtraction direction, the maximum absolute difference was approximately 2.6 × 10⁻¹³ for the test statistic and 7.6 × 10⁻¹³ for a confidence-interval limit.

The sign convention for the mean difference was consistent with the Sigma implementation, which defines the effect as Treatment − Control. When results were transformed to the same direction of comparison, Sigma and R agreed throughout.

Pooled-variance t-test validation

The pooled-variance t-test also showed agreement with the corresponding R implementation. The pooled mean difference, standard error, confidence interval, and p-value matched within negligible numerical tolerance. The maximum absolute pooled-test p-value difference was approximately 7.4 × 10⁻¹¹; the maximum aligned test-statistic and confidence-limit differences were approximately 2.6 × 10⁻¹³ and 6.1 × 10⁻¹⁴, respectively. No systematic deviations were observed.

Distribution, effect-size, and parser stress tests

An additional grid of 99 combinations of t statistics and degrees of freedom, including extreme tails and small degrees of freedom, was compared with R. The maximum absolute difference was approximately 2.2 × 10⁻¹² for two-sided p-values and 1.1 × 10⁻¹² for 95% critical values. Cohen’s d uses the pooled sample standard deviation, and the exact gamma-function correction for Hedges’ g was verified over the same degrees-of-freedom grid; the maximum absolute correction-factor difference was approximately 7.3 × 10⁻¹².

Strict numeric parsing was tested in 11/11 targeted cases covering decimal points, decimal commas, valid grouping separators, malformed partial numbers, non-finite tokens, and quoted CSV fields. Malformed values are treated as non-numeric rather than being partially converted.

Decision agreement

At the conventional significance level of α = 0.05, Sigma and R yielded identical inferential decisions for all valid analyses. There were 0 decision mismatches for the Welch and pooled-variance two-sided tests, including the additional distribution grid.

Interpretation

The validation demonstrates that Sigma reproduces the same independent-samples t-test results as R for both unequal-variance and equal-variance formulations. The exact small-sample correction used for Hedges’ g, strict input parsing, and explicit handling of non-estimable standard errors were also verified. Remaining numerical differences were negligible and are fully compatible with standard floating-point precision and numerical evaluation of the t distribution.

Conclusion

The Sigma unpaired t-test module showed exact agreement with R for valid versus invalid case classification and usable sample sizes, together with near-identical descriptive statistics, mean differences, confidence intervals, and p-values. These results support the validity of Sigma’s implementation of the Welch t-test and the pooled-variance independent-samples t-test for routine analytical use.

Paired t-Test

Validation objective

The Sigma paired t-test module was validated against R with respect to the statistical procedure implemented in the tool: the classical two-sided paired-samples t-test based on within-subject differences. The validation covered the number of complete pairs, descriptive statistics at Time 1 and Time 2, mean differences, test statistics, degrees of freedom, p-values, and confidence intervals. The post-correction validation harness calls the same shared statistical core used by the production page; p-values and confidence limits are therefore validated as direct production-core outputs rather than reconstructed from exported t-statistics. Sigma defines the paired difference as Time 2 − Time 1 and computes the paired t-statistic, confidence interval, and standardized effect sizes on this basis.

Reference methods

Validation was performed against the paired-samples formulation of R’s stats::t.test with paired = TRUE, a two-sided alternative, mu = 0, and a 95% confidence level. Agreement was assessed for the complete-case paired sample used in each analysis and for the corresponding inferential results based on the vector of within-subject differences.

Validation dataset

The validation dataset comprised 900 paired analysis cases. All 900 cases could be matched between Sigma and R. The number of usable complete pairs matched exactly in all 900 cases, and all 900 reference cases were estimable in both implementations.

Validation metrics

Agreement between Sigma and R was assessed at the following levels:

agreement in the number of complete paired observations,
agreement in descriptive statistics for Time 1, Time 2, and the paired differences,
agreement in the paired t-statistic and degrees of freedom,
agreement in p-values and 95% confidence intervals, and
agreement in the final significance decision at α = 0.05.

Results of the validation

Agreement between Sigma and R was exact for the complete paired sample. The number of usable complete pairs matched in 900/900 cases. Descriptive statistics also showed excellent agreement. The maximum absolute differences were approximately 4.89 × 10⁻¹⁵ for each timepoint mean, 8.22 × 10⁻¹⁵ for the mean paired difference, and 3.73 × 10⁻¹⁴ for the standard deviation of the differences.

The paired t-statistic matched R up to machine precision after alignment of the sign convention. The maximum absolute difference in the aligned t-statistic was approximately 3.27 × 10⁻¹³. The corresponding p-values were numerically equivalent, with a maximum absolute difference of approximately 2.80 × 10⁻¹².

Confidence interval validation

The 95% confidence interval for the mean paired difference also showed excellent agreement. After alignment to the same direction of subtraction, the maximum absolute difference in the lower and upper confidence limits was approximately 4.62 × 10⁻¹⁴. These differences are numerically negligible and are fully compatible with ordinary floating-point precision.

Targeted numerical and input-edge tests

The 900-seed campaign was supplemented with deterministic tests of behaviors that are not guaranteed to occur in the generated reference data. These tests cover strict locale-aware numeric parsing, quoted CSV fields, escaped quotes, trailing empty fields, pairwise exclusion of invalid values, subtraction overflow, fewer than two complete pairs, zero and numerically almost-zero variance of differences, and the explicit Time 2 − Time 1 direction.

An extreme-tail R reference with t = 10.878964153196 and df = 99 produced a non-zero two-sided p-value of approximately 1.33613 × 10⁻¹⁸ in the corrected production core. This verifies that the direct incomplete-beta tail calculation avoids the former loss of very small p-values through subtraction from a CDF rounded to one. Cohen’s d_z and the exact gamma-function correction for Hedges’ g_z were checked against deterministic analytic reference values. At df = 1, where the exact correction is undefined, Sigma reports the t-test but leaves g_z blank and emits a warning.

Direction of the paired difference

Interpretation of the paired t-statistic depends on the direction used to define the within-subject difference. Sigma defines the paired difference as Time 2 − Time 1. Accordingly, the sign of the mean difference, t-statistic, and confidence interval is tied to this direction. When Sigma and R results are expressed using the same subtraction order, the paired t-test results agree throughout.

Decision agreement

At the conventional significance level of α = 0.05, Sigma and R yielded identical inferential decisions in 900/900 analyses; the number of decision discrepancies was 0.

Interpretation

The validation demonstrates that Sigma reproduces the same paired-samples t-test results as R for complete paired observations. Remaining differences are negligible at the reported precision and arise from floating-point evaluation of descriptive statistics and Student-t functions. The edge-case suite additionally verifies explicit rejection or reporting of non-estimable inputs instead of silently presenting invalid inference.

Conclusion

The Sigma paired t-test module showed exact agreement with R for the number of complete pairs and near-identical agreement for paired means, mean differences, t-statistics, p-values, and confidence intervals, with no significance-decision discrepancies across 900 reference cases. Together with the targeted numerical and input-edge tests, these results support the validity of Sigma’s implementation of the classical paired-samples t-test, subject to the stated assumptions and complete-case limitations.

One-Way ANOVA — Welch, Classical, Tukey, and Games–Howell

Validation objective

The production calculation core of the One-Way ANOVA module was validated against an independently executed R workflow. The comparison covered data inclusion, group descriptives, classical and Welch omnibus tests, Brown–Forsythe variance testing, eta squared, omega squared, Tukey/Tukey–Kramer comparisons, and Games–Howell comparisons. All comparisons used unrounded full-precision values rather than the values formatted for the user interface.

Reference environment and procedures

The final reference run used R 4.5.2. Classical one-way ANOVA and its sums-of-squares table were obtained with stats::aov; Welch ANOVA used stats::oneway.test(..., var.equal = FALSE). Brown–Forsythe testing applied aov to absolute deviations from the group medians. Tukey estimates, adjusted p-values, and simultaneous confidence intervals used stats::TukeyHSD. Games–Howell results were calculated independently in R from group means, sample variances, Welch–Satterthwaite pairwise degrees of freedom, and stats::ptukey/stats::qtukey.

A common constant was subtracted before the inferential R calculations. Translation leaves every ANOVA, contrast, test statistic, and confidence interval mathematically unchanged and prevents loss of precision in the dedicated 10^9 + x offset stress scenario. Reported means were translated back to their original scale.

Deterministic validation design

Master seed 20260809 generated 90 independent CSV datasets. Sample sizes were n = 10, 30, 60, 120, 300, 800, 1500, 3000, 5000, with 10 replicates at every size. Dataset-specific seeds and configuration metadata are retained in the index. The design produced 770 complete analysis configurations.

Nine scenario types were represented: two-group null data, a two-group mean shift, a three-group equal-variance shift, an unbalanced four-group heterogeneous-variance design, skewed outcomes, rounded ties, missing groups and outcomes, a large numerical offset of (10^9), and five-group heavy-tailed data with an injected outlier. The suite therefore crossed two to five groups with null and alternative effects, balanced and unbalanced sizes, variance equality and inequality, non-normal shapes, missingness, ties, outliers, and numerical stress.

At n = 10, configurations used two or three groups so that every group contained at least three observations and the R studentized-range reference retained pairwise degrees of freedom of at least two. Four- and five-group configurations were included from n = 30 onward. This restriction concerns R-comparable inferential reference values; separate deterministic unit tests cover controlled insufficient- sample and zero-variance states.

Compared outputs

R and Sigma emitted a common long-format result schema. Rows were paired by dataset, analysis configuration, result section, group, pairwise comparison, and statistic. The comparison included:

file rows, usable observations, analyzed groups, exclusions, and result-status labels,
group n, mean, variance, SD, SE, t-based mean interval, quartiles, median, and IQR,
classical sums of squares, degrees of freedom, mean squares, F, p, eta squared, and omega squared,
Welch weighted mean, F, numerator and denominator degrees of freedom, and p-value,
Brown–Forsythe F, degrees of freedom, and p-value,
Tukey contrast, SE, q, degrees of freedom, adjusted p-value, critical value, and simultaneous interval, and
Games–Howell contrast, SE, q, pairwise degrees of freedom, adjusted p-value, critical value, and simultaneous interval.

Numeric acceptance criteria

Structural presence and text/status fields had to match exactly. Statistics not involving the studentized-range approximation used abs(Sigma − R) ≤ 5e−7 + 5e−9 × abs(R). Separate, still sub-reporting-scale criteria were retained for values in which R and Sigma intentionally use independent studentized-range algorithms:

Tukey adjusted p-values: 5e−7 + 1e−8 × abs(R);
Tukey critical values and simultaneous interval limits: 2e−6 + 1e−8 × abs(R);
Games–Howell adjusted p-values: 1e−5 + 1e−8 × abs(R);
Games–Howell critical values: 1.2e−3 + 1e−8 × abs(R); and
Games–Howell simultaneous interval limits: 2.5e−3 + 1e−8 × abs(R).

These separate limits account for the independently implemented studentized-range algorithms. R documents 16-point Legendre integration for ptukey and secant inversion for qtukey, whose quantiles are stated to be accurate to the fourth decimal place; Sigma uses 64-point quadrature and bracketed inversion. Every raw value and difference remains available in the comparison file. In addition to numeric tolerances, the validation required exact agreement of all p-value decisions at α = 0.05 and of the sign of every simultaneous confidence-interval boundary.

Validation results

R and Sigma each produced 104,770 result rows. All 104,770/104,770 keyed comparisons passed, with 0 structural, status, text, or numeric failures. Agreement by section was:

inclusion and preprocessing: 3,850/3,850,
group descriptives: 27,830/27,830,
classical ANOVA and effect sizes: 11,550/11,550,
Welch ANOVA: 6,160/6,160,
Brown–Forsythe: 3,850/3,850,
Tukey/Tukey–Kramer: 24,920/24,920, and
Games–Howell: 26,610/26,610.

All 8,770/8,770 comparisons of p-value decisions at α = 0.05 matched. The signs of all 17,980/17,980 pairwise confidence-interval boundaries also matched, so no comparison differed in whether its interval excluded zero.

Magnitude and source of observed differences

Omnibus p-values agreed particularly closely. The maximum absolute p-value differences were approximately 1.15 × 10⁻¹² for classical ANOVA, 1.30 × 10⁻¹² for Welch ANOVA, and 9.94 × 10⁻¹² for Brown–Forsythe. The maximum Tukey adjusted-p difference was approximately 4.30 × 10⁻⁸; for Games–Howell it was approximately 4.59 × 10⁻⁶.

The largest absolute difference anywhere in the suite was 0.00222164 at a Games–Howell confidence-interval boundary. It arose in an intentionally difficult four-group heterogeneous case with pairwise df = 2.003613. R returned a critical value of approximately 9.77916297, whereas Sigma's direct numerical integration returned approximately 9.77821353. For this pair the mean difference, SE, q statistic, and degrees of freedom agreed to floating-point precision, and the adjusted p-value differed by only approximately 1.83 × 10⁻⁹. The larger CI-scale difference is therefore attributable to the independently implemented low-df quantile calculations, not to a different Games–Howell formula or data cohort.

Absolute mean and quartile differences up to approximately 5.13 × 10⁻⁶ occurred only on the 10^9 outcome scale and correspond to relative differences near machine precision. Numerically centered sums of squares and inferential results remained stable.

Supplementary implementation and edge-case checks

Deterministic JavaScript tests separately verify analytical reference examples, the exact two-group studentized-range/t identity, locale-aware row parsing, large-offset stability, missing-value accounting, singleton groups, constant outcomes, zero group variance, zero pooled within-group variance, non-estimable post-hoc states, and invalid distribution arguments. The browser workflow was exercised with both a complete reference CSV and a file containing missing, non-numeric, and insufficient-group observations; the displayed estimates and controlled warnings matched the calculation core.

Reproducibility

The validation bundle is stored under validation tests/anova/full_validation. It contains the deterministic R seed generator, all CSV seeds, configuration index, independent R reference implementation, Sigma production-core runner, row-level comparator, complete result files, environment record, and text/JSON summaries. Running npm run validate:anova-full regenerates the seeds and both result sets before comparing them. A repeated generation followed by SHA-256 comparison reproduced all 90 seed files and the configuration index byte for byte.

Scope and interpretation

This internally conducted study establishes structural and numerical agreement with the stated R procedures across the tested independent one-way designs. It does not establish that ANOVA is appropriate for a specific scientific dataset, that observations are independent, that a selected outcome has a meaningful mean, or that multiplicity across several outcomes has been addressed. It is not an external software audit or regulatory qualification, and it does not validate methods outside the module, such as repeated-measures, factorial, multilevel, robust-rank, permutation, or covariate-adjusted analyses.

Conclusion

Within the documented scope, Sigma reproduced R's data cohorts, descriptive summaries, classical ANOVA, Welch ANOVA, Brown–Forsythe testing, classical effect sizes, Tukey/Tukey–Kramer inference, and Games–Howell inference across sample sizes from 10 to 5000. All 104,770 numeric/structural comparisons and every inferential decision passed the recorded criteria. The production One-Way ANOVA core is therefore validated for the configurations, assumptions, and limitations described in this report.

Univariable Logistic Regression

Validation objective

The Sigma univariable logistic regression module was validated against R with respect to the statistical procedure implemented in the tool: binary logistic regression with one predictor at a time, estimated by maximum likelihood. Validation covered the classification of analyzable versus non-analyzable datasets, regression coefficients, standard errors, Wald statistics, p-values, odds ratios, and 95% confidence intervals.

Reference methods

Validation was performed against standard univariable logistic regression in R using the binomial logit model. Agreement was assessed for the complete-case sample used for each predictor and for the corresponding model-based estimates derived from the fitted slope parameter.

Validation dataset

The validation dataset comprised 900 analysis cases. Of these, 139 cases were classified as invalid on the R side and were consistently represented as non-comparable error cases in Sigma. Among the remaining comparable analyses, 594 cases were classified as stable and 167 as separation or extreme cases.

Following the numerical-hardening update, the production core was rerun on all 900 files. The original 594-case stable subset was retained unchanged for direct comparison, and all 594 models converged in the updated core. Across the complete campaign, the updated safety logic classified 617 fits as converged without a numerical warning, 1 as converged with an instability flag, and 282 as non-estimable; the latter are reported without inferential quantities.

Validation metrics

Agreement between Sigma and R was assessed at the following levels:

agreement in valid versus invalid case classification,
agreement in the estimated regression coefficient \(\beta\),
agreement in standard errors, Wald z-statistics, and p-values,
agreement in odds ratios and Wald confidence intervals, and
agreement in stable versus separation-prone data constellations.

Overall interpretation of the validation

The comparison showed a clear separation between two types of scenarios. In stable datasets, Sigma and R agreed very closely for all core model quantities. In contrast, the largest discrepancies were concentrated in datasets with complete separation, quasi-separation, or otherwise extreme coefficient constellations. This pattern is methodologically expected for Wald-based logistic regression and is consistent with the warning logic implemented in Sigma.

Stable-case validation

In the 594 stable cases, agreement between Sigma and R was excellent. The maximum absolute difference in the regression coefficient was approximately 7.82 × 10⁻⁸. The maximum absolute difference in the standard error was approximately 3.01 × 10⁻⁴, the maximum absolute difference in the Wald z-statistic was approximately 3.32 × 10⁻³, and the maximum absolute difference in the Wald p-value was approximately 1.315 × 10⁻⁴.

Agreement for effect measures was likewise very good. In stable cases, the maximum absolute difference in the odds ratio was approximately 2.70 × 10⁻⁵. The maximum absolute difference in the lower 95% confidence limit of the odds ratio was approximately 1.05 × 10⁻², and the maximum absolute difference in the upper confidence limit was approximately 3.98 × 10⁻¹. These confidence-interval differences were still small relative to the underlying effect scale and are consistent with minor differences in convergence and floating-point behavior.

Stable-case validation across sample sizes

The excellent agreement in stable cases was preserved across all tested sample-size strata. For example, at n = 5, the maximum absolute coefficient difference in stable cases was approximately 8.76 × 10⁻⁹ and the maximum absolute p-value difference was approximately 3.6 × 10⁻⁵. At n = 100, the corresponding maxima remained very small, and at larger sample sizes such as n = 1000 and n = 5000, coefficient and p-value differences continued to remain in the range expected from negligible numerical approximation effects.

Thus, for data constellations without separation or extreme instability, Sigma reproduced R very closely over the full tested range of sample sizes.

Separation and extreme cases

The remaining 167 comparable cases were classified as separation-prone or extreme. In these analyses, discrepancies between Sigma and R were substantially larger than in the stable-case subset. This behavior is expected, because logistic regression estimates become unstable when event counts are very low, when predictor values nearly or perfectly separate events from non-events, or when the likelihood surface is numerically ill-conditioned.

In these situations, coefficient estimates, odds ratios, and Wald confidence intervals can become very large even when the underlying inferential conclusion is qualitatively similar. Sigma’s warning framework is therefore an important part of the implementation and should be viewed as a deliberate safeguard rather than as a discrepancy from the reference method.

Correction-focused regression tests

Additional tests confirmed invariance when the predictor was multiplied by 10⁻⁹ or 10⁹: slope and standard error changed only by the reciprocal unit factor, while the z-statistic and p-value remained unchanged. Continuous complete and quasi-complete separation produced blank inferential fields. Strict numeric parsing rejected malformed partial numbers, and the two-sided normal p-value was evaluated directly through the complementary error function, including extreme tails.

Invalid cases

A total of 139 cases were invalid in the reference comparison and were consistently treated as non-comparable error cases. These cases reflect data constellations in which a standard univariable logistic regression model is not meaningfully estimable, for example because one outcome category is absent, the predictor has no usable variability, or the data are otherwise structurally unsuitable for model fitting.

Interpretation

The validation demonstrates that Sigma reproduces R very closely for ordinary, well-behaved univariable logistic regression problems. The largest deviations occur in separation-prone or extreme datasets, where numerical instability is expected and where Sigma explicitly reports warnings to prevent overinterpretation. This is consistent with the intended design of the module and with the known behavior of maximum-likelihood logistic regression in difficult small-sample or separation scenarios.

Conclusion

The Sigma univariable logistic regression module showed excellent agreement with R in stable datasets, with negligible differences in regression coefficients, standard errors, Wald statistics, p-values, and odds ratios across the tested sample-size range. Discrepancies were concentrated in separation-prone and extreme cases, where instability is inherent to the method and appropriately flagged by Sigma. Overall, these findings support the validity of Sigma’s implementation of univariable logistic regression for routine analytical use, while confirming the need for caution in pathological data constellations.

Multivariable Logistic Regression

Validation objective

The Sigma multivariable logistic regression module was validated against R with respect to the statistical procedure implemented in the tool: binary logistic regression with one intercept and multiple predictors fitted simultaneously by maximum likelihood. Validation covered the classification of analyzable versus non-analyzable datasets, regression coefficients, standard errors, Wald statistics, p-values, odds ratios, and 95% confidence intervals for the fitted predictors.

Reference methods

Validation was performed against standard multivariable logistic regression in R using the binomial logit model. Agreement was assessed for the complete-case analysis sample used in each model and for the corresponding coefficient-level estimates derived from the fitted multivariable model.

Validation design

To assess performance under different model dimensions, validation was performed in two separate settings:

a model with 3 predictors fitted simultaneously, and
a model with 5 predictors fitted simultaneously.

Each setting comprised 900 analysis cases. This two-step design was used to assess whether agreement with R remained stable when the number of covariates increased and the likelihood of singularity, sparse-data problems, and separation became higher.

The hardened production core was subsequently rerun on all 1,800 files. Every model in the pre-specified stable comparison subsets remained converged: 493 models with 3 predictors and 448 models with 5 predictors. This preserved 1,432 and 2,201 finite coefficient-level comparisons, respectively.

Validation metrics

Agreement between Sigma and R was assessed at the following levels:

agreement in the number of usable observations, events, and non-events,
agreement in the classification of stable, invalid, and separation / singular cases,
agreement in regression coefficients, standard errors, Wald z-statistics, and p-values,
agreement in odds ratios and Wald confidence intervals, and
agreement across different model dimensions (p = 3 and p = 5).

Results for models with 3 predictors

In the validation setting with 3 predictors, all 900 analysis files could be matched between Sigma and R. Agreement in the basic model counts was exact: the number of usable observations, events, and non-events matched in all cases. Among these analyses, 493 were classified as stable, 323 as separation or singular cases, 80 as Sigma-invalid, and 4 as R-invalid.

In the stable subset, agreement was excellent. Across all comparable predictor coefficients (n = 1432 coefficient-level comparisons), the maximum absolute difference in the regression coefficient was approximately 2.02 × 10⁻⁷. The maximum absolute difference in the standard error was approximately 2.34 × 10⁻⁴, the maximum absolute difference in the Wald z-statistic was approximately 2.08 × 10⁻³, and the maximum absolute difference in the Wald p-value was approximately 7.80 × 10⁻⁵. The maximum absolute difference in the odds ratio was approximately 3.53 × 10⁻⁶.

Results for models with 5 predictors

In the validation setting with 5 predictors, all 900 analysis files could again be matched between Sigma and R, and agreement in the number of usable observations, events, and non-events was exact. In this more complex setting, 448 analyses were classified as stable, 261 as separation or singular cases, 87 as Sigma-invalid, and 104 as R-invalid.

In the stable subset, agreement remained excellent despite the higher model dimension. Across all comparable predictor coefficients (n = 2201 coefficient-level comparisons), the maximum absolute difference in the regression coefficient was approximately 1.71 × 10⁻⁷. The maximum absolute difference in the standard error was approximately 4.91 × 10⁻⁴, the maximum absolute difference in the Wald z-statistic was approximately 3.20 × 10⁻³, and the maximum absolute difference in the Wald p-value was approximately 1.040 × 10⁻⁴. The maximum absolute difference in the odds ratio was approximately 3.92 × 10⁻⁶.

Confidence intervals

Agreement in Wald confidence intervals was also very good in the stable subsets. For the p = 3 setting, the maximum absolute difference in the lower odds-ratio confidence limit was approximately 2.51 × 10⁻³, and the maximum absolute difference in the upper confidence limit was approximately 2.14 × 10⁻¹. For the p = 5 setting, the maximum absolute difference in the lower confidence limit was approximately 1.28 × 10⁻³, and the maximum absolute difference in the upper confidence limit was approximately 4.00 × 10⁻¹. These differences remained small relative to the scale of the corresponding odds ratios.

Interpretation of unstable and invalid cases

As expected, increasing the number of predictors increased the frequency of unstable model constellations. Compared with the p = 3 setting, the p = 5 setting showed a higher number of invalid and singular or separation-prone cases. This pattern is methodologically plausible, because multivariable logistic regression becomes more sensitive to sparse data, multicollinearity, quasi-separation, and near-singular information matrices as model complexity increases.

Importantly, however, these difficult cases were not hidden by the validation design. They were explicitly separated from the stable subset and are consistent with the warning logic implemented in Sigma, including warnings for low events-per-variable ratios, non-convergence, and separation or multicollinearity. In ordinary well-behaved datasets, agreement with R remained excellent in both the p = 3 and p = 5 settings.

Correction-focused regression tests

Targeted comparisons with R additionally covered predictor rescaling by 10⁻⁹ and 10⁹, exact linear dependence, and a complete-case dataset in which a predictor becomes constant only after listwise deletion. Scaling left z-statistics and p-values unchanged. For exact dependence, Sigma retained the identifiable coefficients and left the aliased coefficient blank, matching R’s estimability pattern. In the complete-case test, the constant predictor was excluded and the remaining slope agreed with R.

Interpretation

The validation demonstrates that Sigma reproduces R very closely for multivariable logistic regression in stable data constellations and retains this agreement when the number of covariates increases from three to five. The main effect of the larger model dimension was not a deterioration of numerical agreement in stable cases, but rather an expected increase in the number of unstable or non-estimable datasets.

Conclusion

The Sigma multivariable logistic regression module showed exact agreement with R for usable sample sizes, event counts, and non-event counts, together with excellent coefficient-level agreement in the stable subsets for both 3-predictor and 5-predictor models. Discrepancies were concentrated in separation-prone, singular, or otherwise pathological datasets, where instability is inherent to the method and appropriately reflected by Sigma’s warning framework. Overall, these findings support the validity of Sigma’s implementation of multivariable logistic regression for routine analytical use in well-behaved datasets.

Propensity Score Matching (PSM)

Validation objective

The production JavaScript core of the Sigma Propensity Score Matching module was compared directly with an independently executed R reference workflow. The comparison covered strict preprocessing, treatment and binary-variable coding, logistic propensity-score estimation, deterministic ATT-oriented greedy nearest-neighbor matching, calipers, exact pair identity, all displayed balance values, and overlap diagnostics. Display rounding was not used; comparisons used the underlying full-precision values.

The validation also targeted the numerical defect corrected in this release: fitted propensity scores and matches must remain invariant when numeric covariates are expressed in very small or very large units.

Reference environment and methods

The final reference run used R 4.5.2 and MatchIt 4.7.2. Logistic models were fit independently with R glm.fit(..., family = binomial()). Matching used MatchIt::matchit(method = "nearest", estimand = "ATT", ratio = 1, replace = FALSE, m.order = "largest") with the R-fitted propensity scores supplied as the distance. Calipers and standardization settings were passed identically. When a valid caliper yielded no pairs, MatchIt's “No units were matched” condition was normalized to the same valid empty matched set reported by Sigma.

Every displayed balance statistic was recomputed independently in R from the matched indices using the same stated definitions. Exact pairs were compared by treated and control identifiers rather than by the presentation order of MatchIt's match matrix.

Deterministic validation design

Master seed 20260808 generated 180 seed datasets. Sample sizes were n = 10, 30, 60, 120, 300, 800, 1500, 3000, 5000, with 10 repetitions at each size in each of two cohort structures: approximately balanced treatment groups and an approximately 5:1 control-to-treated distribution. Dataset-specific seeds are retained in the archive. Good, moderate, and deliberately limited-overlap scenarios were represented.

The generated data included metric covariates, text-coded binary variables (F/M), binary numeric labels other than 0/1 (1/2), multi-level categorical covariates, deterministic missing values, and invalid numeric tokens. Each dataset was evaluated with three model scales:

the original numeric covariates,
all numeric covariates multiplied by 10⁻⁹, and
all numeric covariates multiplied by 10⁹.

This produced 540 logistic model comparisons. Six matching configurations were evaluated per dataset, for 1,080 matching comparisons:

standardized PS calipers 0.05, 0.20, and 0.50 on the base model,
an absolute PS caliper of 0.20 on the base model, and
a standardized PS caliper of 0.20 for each rescaled model.

Validation metrics

The final comparator evaluated 1,816,729 full-precision result rows, including:

included and excluded rows, exclusion reasons, binary mappings, and design-matrix columns,
convergence, estimable rank, rank deficiency, deviance, propensity scores, and linear predictors,
matching status and counts, effective calipers, exact treated–control identities, and pair distances,
every pre- and post-match mean/proportion, ATT-standardized SMD, and descriptive z-difference, and
pre- and post-match overlap ranges, outside-support proportions, and warnings.

Text, mappings, counts, rank, and pair identities required exact equality. Numerical tolerances were 2×10⁻⁸ for propensity scores, pair distances, overlap, and effective calipers; 2×10⁻⁶ for linear predictors; 2×10⁻⁹ relative to magnitude for balance values; and 2×10⁻⁷ for deviance. Iteration counts were retained as algorithmic diagnostics but were not required to be identical. Scale invariance required a maximum propensity-score difference of 5×10⁻¹³ and exact pair identity.

Results

The complete validation passed without an unmatched or missing reference row and without a recorded mismatch. The pass counts were:

540/540 logistic model comparisons,
1,080/1,080 matching comparisons,
1,080/1,080 matching runs with exact treated–control pair identity, and
360/360 scale-invariance comparisons.

These totals include runs in which a strict caliper correctly yielded zero pairs. Such runs were compared as valid empty matched sets, not discarded as failed analyses.

Numerical agreement

Maximum absolute differences were 3.22×10⁻¹⁵ for propensity scores, 3.89×10⁻¹⁵ for pair distances, and 2.94×10⁻¹⁵ for overlap quantities. The maximum propensity-score difference between an original model and either rescaled Sigma model was 1.50×10⁻¹⁵.

The largest absolute balance difference, 1.53×10⁻⁵, occurred for a displayed post-match mean of approximately 6.48×10¹⁰ in a deliberately 10⁹-rescaled dataset; its relative difference was only 2.35×10⁻¹⁶. Across balance values, the largest relative difference was 9.05×10⁻¹³. The apparent absolute maximum is therefore a unit-scale artifact at machine precision, not a scientific discrepancy.

Balanced and 5:1 cohorts

Both the approximately balanced cohorts and the 5:1 control-dominant cohorts were included in every sample-size tier and all three overlap scenarios. Agreement was complete for preprocessing, model output, matched counts, exact pairs, balance, and overlap in both cohort structures. This directly replaces the earlier validation that compared only aggregated max-|SMD| summaries and a smaller number of available reference exports.

Scale invariance

Internal standardization removed the earlier dependence on raw predictor units. For all 360/360 original-versus-rescaled comparisons, the fitted Sigma propensity scores remained within the prespecified 5×10⁻¹³ tolerance and the treated–control pairs were identical. The observed maximum was 1.50×10⁻¹⁵.

Interpretation and scope

The results support numerical equivalence of the implemented complete-case, 1:1 nearest-neighbor PSM workflow to the specified R/MatchIt reference under the tested conditions. They do not establish that a chosen propensity-score model is causally sufficient, that overlap is adequate for a particular research question, or that unmeasured confounding is absent. Those remain study-design judgments. Likewise, outcome inference after matching must account for the paired matched structure and is outside this module.

Conclusion

Across 180 deterministic datasets, 540 model comparisons, 1,080 matching comparisons, and 1,816,729 full-precision result rows, Sigma showed no validation mismatch against R 4.5.2 and MatchIt 4.7.2. Exact pair identity was obtained in every matching run, and predictor rescaling by factors of 10⁻⁹ and 10⁹ did not change the matched design.

These findings support the corrected implementation for scientific use as a transparent design-stage tool for complete-case, ATT-oriented, 1:1 nearest-neighbor propensity-score matching without replacement, provided that model specification, overlap, measured-confounder selection, and the subsequent paired outcome analysis are assessed by the researcher.

Survival Analysis — Kaplan–Meier, Log-Rank, RMST, and Cox Regression

Validation objective

The Survival Analysis module was validated by comparing the production JavaScript calculation core directly with an independently executed R reference workflow. Validation covered preprocessing and exclusions, Kaplan–Meier estimation, reverse Kaplan–Meier median follow-up, fixed-time survival estimates, the omnibus log-rank test, restricted mean survival time (RMST), Cox proportional hazards regression, robust inference, stratified baseline hazards, concordance, and proportional-hazards diagnostics. Display rounding was not used in the comparison; all tests used the underlying full-precision values.

Reference environment and methods

The final reference run used R 4.5.2 with survival 3.8.3. Reference calculations used survfit with complementary log-log confidence limits, reverse-event survfit for median follow-up, survdiff(rho = 0) for the log-rank test, the restricted-mean output from summary.survfit, coxph for Cox models, concordance for Harrell's C, and cox.zph(transform = "km", terms = TRUE, global = TRUE) for term-level and global PH diagnostics. Efron and Breslow tie handling, model-based and sandwich covariance, clustering, and Cox strata were passed to the corresponding R procedures according to each configuration.

Deterministic validation design

Master seed 20260803 generated 90 independent seed datasets. The tested sample sizes were n = 10, 30, 60, 120, 300, 800, 1500, 3000, 5000, with 10 replicates at every size. Dataset-specific seeds are retained in the configuration index. The design produced 1,800 analysis configurations: 360 nonparametric and 1,440 Cox configurations.

The generated data included continuous follow-up, deliberately tied event times, heavy censoring, two- and three-level groups, missing group assignments, missing predictors, metric and categorical predictors, user-selected categorical reference levels, cluster identifiers, Cox strata, and deliberately non-proportional effects. Small datasets used lower-dimensional models where necessary to avoid making singularity inevitable, while the complete advanced-option factorial remained represented at every sample size.

Nonparametric coverage

Four nonparametric configurations were evaluated for each seed. They covered Overall, two-group, and three-group Kaplan–Meier estimates; risk sets, events, censoring, survival, Greenwood standard errors and confidence limits at every observed step; survival at requested time points; median survival and its interval; reverse-KM median follow-up; log-rank chi-square, degrees of freedom, p-value, and group-specific observed and expected events; and RMST/RMTL estimates, standard errors, intervals, differences, p-values, and ratios at a common restriction time.

Cox coverage

The principal Cox factorial crossed Efron versus Breslow ties, model-based versus subject-robust versus cluster-robust variance, and unstratified versus stratified baseline hazards. Additional configurations exercised purely metric models, common complete-case analysis with missing predictors, custom categorical references, and deliberately non-proportional categorical effects. Every Cox configuration included the multivariable model and common-cohort univariable models.

Compared outputs included coefficients, standard errors, z statistics, p-values, hazard ratios and confidence intervals; model-based and robust covariance matrices; null and fitted log partial likelihoods; AIC; likelihood-ratio, Wald, and score tests; overall factor tests; convergence metadata; Harrell's C, its standard error and comparable-pair count; and term-level and global PH tests.

Structural matching and numeric criteria

R and Sigma emitted the same long-format schema. Every row was matched in order by dataset, configuration, result section, model, cohort, term, statistic, and evaluation point. Presence and text values had to match exactly. Closed-form and nonparametric results, covariance matrices, concordance, and PH diagnostics used the strict criterion abs(Sigma − R) ≤ 5e−7 + 5e−7 × abs(R).

Cox coefficient, model, and term-test sections depend on separate iterative solvers and floating-point accumulation order. These sections therefore used abs(Sigma − R) ≤ 1e−5 + 1e−5 × abs(R). This tolerance is applied to full-precision values and is substantially smaller than ordinary reporting precision. The raw R value, Sigma value, absolute difference, scaled relative difference, and pass decision are retained for every comparison; the tolerance does not alter or round either program's result.

Validation results

R and Sigma each produced 3,833,940 result rows. All 3,833,940/3,833,940 keyed comparisons passed, with 0 structural, text, presence, or numeric failures. Agreement by result section was:

analysis metadata: 11,880/11,880,
Kaplan–Meier step rows: 3,457,700/3,457,700,
Kaplan–Meier summary and fixed-time rows: 48,510/48,510,
log-rank and group-detail rows: 3,330/3,330,
RMST estimates and comparisons: 10,800/10,800,
Cox coefficients: 76,720/76,720,
Cox model and term statistics: 154,880/154,880,
model-based and robust covariance rows: 52,600/52,600, and
PH diagnostics: 17,520/17,520.

The maximum absolute numeric difference was approximately 3.6261 × 10⁻⁴. It occurred for a Cox Wald chi-square statistic of approximately 117.3 in a dataset with n = 5000; its scaled relative difference was approximately 3.09 × 10⁻⁶. Across all values, the maximum scaled relative difference was approximately 5.34 × 10⁻⁶. Thus even the largest observed solver-related difference remained below the iterative Cox criterion and far below meaningful displayed precision.

Supplementary implementation checks

The Survival Analysis core additionally passed 22/22 deterministic JavaScript unit tests covering input parsing, distribution functions, near-tied time correction, Kaplan–Meier edge cases, RMST, log-rank calculations, Cox risk sets and ties, strata, robust covariance, PH diagnostics, and controlled error states. The optimized concordance implementation was compared with direct quadratic pair enumeration across 250 deterministic randomized small datasets containing time ties, prediction ties, strata, and clusters. Comparable-pair counts, concordance estimates, and standard errors agreed in every check.

Reproducibility

The complete validation bundle is stored under validation tests/survival_analysis/full_validation. It contains the deterministic seed generator, configuration index, R reference implementation, Sigma production-core runner, streaming row-level comparator, generated results, environment record, and concise summary. Running run_validation.sh rebuilds all seeds and both result sets before performing the comparison. The final one-command reproduction completed successfully with the counts reported above.

Scope and interpretation

This study establishes numerical equivalence to the stated R reference procedures across the tested right-censored, single-event scope. It does not validate methods that the module does not implement, including delayed entry, start-stop records, time-varying covariates, interactions, splines, frailty, recurrent events, interval censoring, or weighted and survey analyses. Simulation-based cross-validation cannot establish that a chosen model is scientifically appropriate for a particular dataset, and this internally conducted validation is not a substitute for an external software audit or regulatory qualification.

Conclusion

Within the documented scope, the validation demonstrates exact structural agreement and high-precision numerical equivalence between Sigma and R for Kaplan–Meier estimation, log-rank testing, RMST, Cox regression, robust and cluster-robust inference, stratified baseline hazards, concordance, and PH diagnostics over sample sizes from 10 to 5000. The production calculation core is therefore validated for the configurations, assumptions, and acceptance criteria described in this report.

Competing Risk Analysis — CIFs, Gray Test, and Fine & Gray Regression

Validation objective

The production calculation core of the Sigma competing-risk module was validated against an independent R reference for all analytical components implemented in the tool: non-parametric cumulative incidence functions (CIFs), their Aalen variances and pointwise confidence intervals, Gray’s two-group test (ρ = 0), and binary Fine & Gray proportional subdistribution-hazards regression. The comparison also covered preprocessing, pooled and group-specific censoring weights, both possible reference groups, model convergence, and controlled non-estimability.

Reference implementation

The reference calculations used R 4.5.2 and cmprsk 2.2-12. CIFs and Gray tests were obtained with cmprsk::cuminc; CIF estimates and variances were evaluated at every observed event time with cmprsk::timepoints. Fine & Gray models were fitted with cmprsk::crr using failcode = 1 or 2, cencode = 0, gtol = 10⁻⁶, at most ten Newton updates, and robust variance estimation. Comparisons used full-precision values rather than rounded display output.

Validation design

The deterministic suite contained 900 datasets: six nominal sample sizes (n = 5, 10, 20, 100, 1000, 5000), 15 scenarios per sample size, and ten independent replicates per scenario. The scenarios covered no group effect, effects on either cause, high and low censoring, rare causes, 80:20 group imbalance, differing censoring patterns between groups, rounded and integer ties, missing time, missing status, a piecewise non-proportional cause-1 effect, and a combined high-censoring / rare-cause / ties stress setting.

For every dataset, Sigma and R evaluated overall and group-specific CIFs for every detected cause at every distinct event time. Gray’s test was requested for causes 1 and 2. Fine & Gray regression was evaluated for both causes, with pooled and group-specific censoring distributions, and with Control and Treatment each used as the reference group. This produced:

2,089,198 paired CIF time points across 4,905 curves,
1,800 paired Gray-result rows, and
7,200 paired Fine & Gray result rows.

Rows with missing time or status were removed under the same rules on both sides. Numeric comparisons were performed only when both implementations classified a result as estimable and converged; all non-numeric status classifications were compared separately and required to match exactly.

Predefined acceptance criteria

Before the final comparison, the following absolute-difference limits were specified:

CIF time grid: 1 × 10⁻¹² and CIF estimate, variance, SE, and confidence limits: 1 × 10⁻¹⁰;
Gray statistic: 1 × 10⁻⁸ and Gray p-value: 1 × 10⁻¹²;
Fine & Gray coefficient, robust SE, z statistic, and p-value: 1 × 10⁻⁸;
reference-group symmetry checks: 1 × 10⁻¹⁰; and
exact agreement for dataset structure, detected causes, and all result-status classifications.

Cumulative incidence functions

All 2,089,198 CIF points were paired on identical file, scope, group, cause, and time keys; there were 0 key or time-grid mismatches. Across the 4,905 curves, the maximum curve-wise absolute differences were 5.55 × 10⁻¹⁶ for the CIF, 1.40 × 10⁻¹⁵ for the Aalen variance, 9.91 × 10⁻¹⁴ for the SE, 2.18 × 10⁻¹³ for the lower confidence limit, and 1.72 × 10⁻¹³ for the upper confidence limit. All results were well inside the predefined CIF tolerance.

Gray’s test

Status classifications matched in all 1,800/1,800 rows: 1,677 paired results were estimable, 44 were classified as not estimable, and 79 causes were not detected. Among the 1,677 estimable pairs, the maximum absolute difference was 3.85 × 10⁻¹¹ for the test statistic and 2.51 × 10⁻¹³ for the p-value; degrees of freedom matched exactly.

Fine & Gray regression

Status classifications matched in all 7,200/7,200 rows. The paired counts were 5,606 converged and estimable models, 1,032 non-estimable models, 246 non-converged models, and 316 rows in which the requested cause was not detected. Observation counts, total target-cause events, and events in both reference and comparison groups matched exactly.

Across the 5,606 estimable pairs, the maximum absolute differences were 1.41 × 10⁻¹³ for the coefficient, 9.19 × 10⁻¹⁵ for the robust SE, 2.64 × 10⁻¹² for the z statistic, and 3.72 × 10⁻¹⁴ for the p-value. The derived SHR, lower confidence limit, and upper confidence limit had maximum absolute differences of 1.13 × 10⁻¹³, 1.04 × 10⁻¹³, and 1.71 × 10⁻¹², respectively.

Reversing the binary reference group produced 2,803 estimable model pairs. The largest absolute coefficient sum was 1.63 × 10⁻¹³, the largest SE difference was 5.70 × 10⁻¹⁵, the largest p-value difference was 8.32 × 10⁻¹³, and the largest deviation of the reciprocal SHR product from one was 1.63 × 10⁻¹³.

Reproducibility

The reproducibility bundle is stored under validation tests/competingrisk/revalidation_v2. It contains the Sigma runner, the independent R runner, the comparison script, per-shard environment records, full row-level outputs, curve-level CIF comparisons, status comparisons, sample-size summaries, and the machine-readable final summary. The seed index in validation tests/competingrisk/Data/cases_index.csv records the scenario, replicate, and deterministic seed for every dataset.

Interpretation and limitations

The suite demonstrates exact structural and status agreement and numerical equivalence to the stated R reference over the validated designs, including very small samples, sparse causes, ties, missing time or status, unequal groups, high censoring, and sample sizes up to 5000. Agreement of a numerical result does not make a model scientifically appropriate: very small event counts can still make large-sample inference unreliable, and the Fine & Gray proportional subdistribution-hazards assumption must be judged for the study at hand. The non-proportional scenario validates numerical agreement under that input pattern; it does not validate proportional-hazards interpretation when the assumption is violated. Likewise, scenarios with differing censoring patterns test computation and do not remove the requirement for independent censoring within the selected censoring model.

This is an internally conducted numerical cross-validation against one specified reference implementation. It is not an external software audit, regulatory qualification, or proof that every possible dataset, operating environment, or scientific use case has been validated.

Conclusion

All predefined acceptance criteria passed. Within the documented scope, Sigma reproduces cmprsk results to high numerical precision for CIF estimation and uncertainty, Gray’s test, and binary Fine & Gray regression with pooled or group-specific censoring and either reference group. The revised production core is therefore validated for the configurations and definitions described in this report.