Phi 3.0.1: current assessment
The expanded assessment covers all eleven tools, ordinary and boundary datasets, import and worksheet selection, project files, adjustment presets, rendered results and raster export. Independent references used exact or high-precision arithmetic, with additional cross-checks in R. Inputs, expected values and numerical acceptance rules were frozen before execution.
The assessment identified and led to corrections for DeLong inference with fewer than two cases
per class, special category names such as __proto__, and cancellation in descriptive
sums and means. All 16 affected numerical cases now pass; every previously passing numerical case
still passes. No numerical tolerance or expected result was changed to obtain these outcomes.
| Assessment | Recorded outcome |
|---|---|
| 1,581 numerical cases | 1,578 PASS; 3 FAIL retained |
| 172 workflow contracts | 165 PASS; 6 FAIL; 1 REVIEW |
| Additional targeted Electron checks | 15 PASS, including corrected calculations and project restoration |
| Rendered table and plot reference checks | 22 PASS |
| Native integration checks | 11 save cancellations, 4 worksheet cases, 22 data-copy cases and 18 single-analysis exports passed |
| Raster export | 88 exports with correct pixel dimensions; DPI metadata limitations remain |
| Simulation | 12,000 datasets; no reference discrepancies, with interval coverage limitations below |
Open findings and interpretation
- Three very small logarithmic/exponential coefficient discrepancies remain formal failures under the frozen acceptance rules. The tested predictions pass; these flags have not been reclassified as passed or removed.
- Five adjustment-preset workflows do not restore the export resolution. Project files retain that setting in the tested workflows for all eleven tools. Check the resolution after loading an adjustment preset.
- One Excel header is displayed as
0.0rather than the expected0. All tested columns and cell values are retained. - ROC treats other non-empty outcome values as negative; Kaplan–Meier treats other non-empty status values as censored. Verify the coding before analysis. Stronger outcome validation and an independently stored bootstrap seed remain future work.
- Extremely small values can round to zero in some Kaplan–Meier and forest-plot displays. Pixel dimensions are verified for PNG, JPEG, TIFF and BMP; resolution metadata is correct in TIFF, absent in the tested PNG/JPEG files and zero in BMP.
The overall suite is therefore not fully passing. This assessment used the macOS source with Electron 25.9.8 and Node 24.12.0 for the numerical harness. Native dialog responses were supplied by test harnesses. One native-export readiness condition was corrected to accept the intentionally disabled Rename button for an unsaved analysis; data and export assertions were retained. Final packaged-app acceptance and Windows execution are separate from this assessment.
The calculation source is identified by the correction-run hashes. The subsequent version-metadata and documentation update to 3.0.1 leaves that calculation code unchanged. Earlier reports and their source hashes are retained; the full calculation suite was not rerun for this documentation update.
Historical report: Phi 3.0.0 · 17 September 2026
1. Validation conclusion
The results support the tested implementation of the documented descriptive, survival, ROC/AUC and curve-fitting methods. This is software verification and numerical validation. It does not establish clinical validity or determine whether a method is appropriate for a particular research question. Final packaged-app and native file-dialog acceptance is a separate assessment.
2. Scope and terminology
Verification
Checks that the production implementation follows the documented formulas, input rules and display behavior in the tested cases.
Numerical validation
Compares Phi's calculated outputs with independently evaluated references in R using the same definitions or explicitly justified tolerances.
The assessment covered:
- numeric and delimited-text parsing, invalid values, and tool-specific row inclusion;
- bar charts, min-to-max boxplots, categorical grouped and stacked bars, and paired dot plots;
- Kaplan–Meier estimates, risk sets, Greenwood variance, log–log intervals and the complementary incidence view;
- ROC coordinates, AUC, observed-value Youden cutoffs, DeLong statistics and paired comparisons;
- stratified bootstrap AUC intervals and pointwise ROC confidence bands;
- linear, logarithmic and exponential regression, model domains and degenerate inputs;
- forest-plot preservation and rejection rules for supplied estimates and intervals;
- actual result tables, plot data, and preservation of inputs, column mappings and settings through project restoration.
3. Validation design
The numerical harness called production functions. Existing validation hooks were supplemented by test-only access to private functions without changing their calculation bodies. R processed the same retained input datasets independently. The protocol, source hashes and expected case catalogues were fixed before the new calculations. Missing cases, duplicates and failed mandatory checks prevent a passing result.
| Area | Retained and additional coverage |
|---|---|
| Descriptive tools | 104 datasets with signed values, ties, constants, singletons and multiple groups/categories |
| Regression | 52 datasets per model; additional domain, insufficient-data and repeated-x checks |
| Kaplan–Meier | 56 baseline datasets; five additional complete-curve cases; five independently recalculated historical rounding cases |
| ROC/AUC | 55 baseline paired-marker datasets; eight complete-curve cases covering both directions, ties, separation and constant scores |
| Bootstrap bands | Eight shared-resampling cases, 100 iterations each, with both confidence limits at all 101 false-positive-rate grid points |
| Inputs and displays | 10 original checks, 53 additional rule/formatter checks and seven real table/plot checks |
The common resampling plans isolate the calculation of bootstrap percentiles from random sampling differences. Reproducibility, marker-name seed behavior and disabled bootstrap were also checked with Phi's unmodified random-number generator. A separate independent-stream bootstrap comparison remains supportive evidence rather than proof of identical random samples.
All eleven tool integration suites were rerun: 248 checks passed. A further 18 single-analysis export scenarios and 11 independent reopenings after process restart passed, giving 277 integration checks. All 52 archive, session and related core checks passed. The comparison system itself passed 20 control and corruption tests plus six command-line checks, including nonzero exit codes on deliberate failures. Native file-dialog responses in these integration tests were simulated.
4. Independent R reference
- R: 4.5.2
- survival: 3.8-3
- pROC: 1.19.0.1
- D3: 6.7.0; regression precision: 10 decimal places
- Application runtime used: Electron 25.9.8 / Node 18.15.0, macOS arm64
Base R supplied descriptive summaries, type-7 quantiles and independent regression calculations.
survival::survfit with conf.type = "log-log" supplied the baseline
Kaplan–Meier reference; additional independent product-limit and Greenwood calculations used Phi's
documented multiplier of 1.96. pROC supplied AUC and DeLong references. Complete ROC
coordinates and common-plan bootstrap percentiles were evaluated independently in base R.
5. Prespecified acceptance criteria
| Output class | Acceptance criterion |
|---|---|
| Counts, sample sizes and risk sets | Exact equality |
| Deterministic numerical results | Absolute difference ≤ 1 × 10−10 |
| Regression library outputs | Absolute difference ≤ 1 × 10−9 |
| KM limits versus R's precise normal quantile | Absolute difference ≤ 1 × 10−4 |
| Normal-approximation p-values | Absolute difference ≤ 1 × 10−6 |
| Shared-plan bootstrap percentiles and matched-definition KM limits | Absolute difference ≤ 1 × 10−10 |
| Independent-stream bootstrap limits | Absolute difference ≤ 0.04; supportive Monte Carlo comparison |
| Specified formatter, table and matched-definition display cases | Exact expected strings or table values |
| Case coverage and mandatory rule checks | No missing or duplicate cases; all mandatory checks pass |
Protocol 2.0 explicitly separates numerical agreement, correct formatting of Phi's own values, and stochastic reference differences. It supersedes the earlier blanket requirement for identical rounded values across every reference comparison. Historical results are retained and are not retrospectively described as satisfying that earlier requirement. Numerical tolerances were not loosened for this run.
6. Results
| Comparison family | Comparisons | Passed | Maximum absolute difference |
|---|---|---|---|
| Descriptive and categorical tools | 6,755 | 6,755 | 1.28 × 10−13 |
| Regression tools | 624 | 624 | 0 |
| Baseline Kaplan–Meier and risk sets | 7,698 | 7,698 | 1.32 × 10−5 |
| Additional KM boundaries, complements and matched-display values | 109 | 109 | 5.56 × 10−16 |
| Baseline ROC/AUC and paired DeLong statistics | 605 | 605 | 1.39 × 10−7 |
| Additional complete ROC coordinates | 184 | 184 | 5.00 × 10−16 |
| Shared-plan bootstrap limits and bands | 1,632 | 1,632 | 4.45 × 10−16 |
| Independent-stream bootstrap limits | 20 | 20 | 0.0167 |
| Total | 17,627 | 17,627 | All criteria passed |
Documented reference and rounding differences
The 23 historical differences in the diagnostic three-decimal comparison remain recorded: five Kaplan–Meier limits and 18 independently sampled bootstrap limits. The five KM cases agree in their specified rounded comparison when R uses the same multiplier of 1.96. Independent bootstrap limits all remain inside the 0.04 tolerance.
The additional shared-plan band checks contain 16 diagnostic rounding differences at the same boundary: 0.9125000000000001 versus 0.9125. Their absolute difference is about 1.11 × 10−16. These are confidence-band coordinates, not printed result labels. They satisfy the prespecified numerical criterion; no requirement for identical three-decimal projections was assigned to them. All 39 diagnostic differences are retained with their classification. No mandatory failure remains unresolved.
7. Interpretation
The new evidence adds 1,925 numerical comparisons to the original suite and confirms the tested data-preservation and display paths. A successful finite test suite is evidence for these cases and methods, not a guarantee for every possible dataset, platform or future modification.
8. Limitations and intended use
- The assessment does not establish clinical performance, diagnostic accuracy or regulatory fitness.
- Independent bootstrap runs can differ. Shared-plan checks assess the percentile calculations, not universal statistical coverage of the intervals.
- Actual native dialogs and final macOS/MAS packages require separate acceptance after the final builds; this report does not certify that pending package assessment.
- The recorded execution platform was macOS arm64; other platform executions are not implied.
- Forest plots display user-supplied estimates and intervals; those source estimates are not independently validated by Phi.
- Users remain responsible for data quality, study design, model choice, interpretation and independent scientific review.