Published now
- • Model version v1.3 and source assumptions
- • Input weighting and range rules
- • Outcome inclusion and privacy policy
- • Machine-readable validation status
Evidence status · September 6, 2026
NBMEcalc Validation Status
The current model is an independent planning heuristic. A reproducible outcome cohort and holdout report have not yet been published, so NBMEcalc does not claim a verified error rate, calibrated 95% interval, or clinical decision accuracy.
Not yet published
- • Auditable de-identified outcome cohort
- • Locked holdout evaluation
- • Verified range-coverage percentage
- • Subgroup performance estimates
Current published validation cohort: 0. Submitted outcomes are not promoted into a public metric until the dataset is large enough to report with clear inclusion rules and without exposing an individual student.
Outcome collection protocol
Free and paid users can explicitly opt in after calculating. The reminder is sent around the estimated score-release date.
- 1
Consent
A user supplies an email and exam date specifically for one outcome reminder.
- 2
Outcome
The user may report an exact score or a pass/fail band after release.
- 3
Evaluation
Eligible records are deduplicated, versioned, aggregated, and separated from model development before publication.
Metrics required before accuracy claims
A future report must state the model version, sample dates, cohort size, missing-data handling, exclusions, and uncertainty around every metric—not just the best headline number.
| Required metric | Why it matters |
|---|---|
| Mean and median absolute error | Shows the typical distance between a prediction and the real score. |
| Planning-range coverage | Shows how often the real score falls inside the displayed range. |
| Pass-risk calibration | Compares experimental pass estimates with observed pass/fail outcomes. |
| Source and recency subgroups | Tests whether performance changes by input mix or assessment age. |
Reproducibility and privacy
A public release will contain only de-identified, aggregated records that meet the stated inclusion criteria. Email, IP, free-text notes, and score-report links will not be included. Model changes will be evaluated against a locked holdout before replacing the current production version.