Removing two post-diagnostic fields cut every tested cardiovascular model by 0.049 to 0.051 AUROC
A leaderboard can make ten models look different even when the answer is hiding in the input. A new preprint on survey-based cardiovascular screening makes that problem unusually easy to see. The authors compared linear models, boosted trees, a neural network, an explainable boosting machine, TabPFN v2, and TabICL. Removing two post-diagnostic fields lowered every model's AUROC by almost exactly the same amount: 0.049 to 0.051.
After the removal, all ten models fit inside a 0.0045-wide band, from 0.8395 to 0.8440. Deciding which columns belonged in the screening task mattered far more than choosing among the tested models.
The study is a preprint, not a clinical product or medical recommendation. Its outcome is self-reported, ever-diagnosed myocardial infarction, so it measures a cross-sectional screening-triage problem among surviving, diagnosis-aware respondents. Within those limits, it gives developers a useful audit pattern for prediction systems built from records collected after the event they claim to predict.
The benchmark starts with 872,822 survey records
The researchers used 442,067 respondents from the CDC's 2022 Behavioral Risk Factor Surveillance System for development. They then froze models and thresholds before applying them to 430,755 respondents from the 2023 survey.
The outcome was whether a respondent reported ever being diagnosed with a myocardial infarction. The full feature tier had 39 fields. Two of them were a prior angina or coronary heart disease diagnosis and a chest CT scan. Those facts may belong in a health record. They are questionable inputs for a task framed as screening for a prior infarction because both may arrive after the diagnostic process has begun.
The authors removed those fields to create a 37-feature screening tier. They also tested stricter tiers: one removed prior stroke, another kept only fields common to both survey years, and a 15-feature tier retained only self-reportable items without clinician-mediated content. Calling one feature list "leakage free" would hide a judgment call. The tiered design shows how results move as the boundary becomes more conservative.
Model sophistication did not recover the leaked signal
On all 39 fields, every model landed between 0.8905 and 0.8933 AUROC. With the two post-diagnostic markers gone, scores fell to between 0.8395 and 0.8440. Removing prior stroke cost another 0.0075 to 0.0086. The self-report-only tier lost roughly another 0.021.
The decline was uniform across logistic regression, random forest, three boosted-tree implementations, an explainable boosting machine, a multilayer perceptron, TabPFN v2, and TabICL. No tested model recovered the signal lost with the suspect fields.
At the 37-feature tier, CatBoost scored 0.8439, TabICL 0.8440, and the glass-box explainable boosting machine 0.8430. The authors pre-specified a 0.005 non-inferiority margin, and the glass-box model was non-inferior to every comparator at every tier. That finding applies to this task, not to transparent and foundation models in general. On this data set, the separation was too small to support a claim that greater model complexity improved discrimination.
The near-tie came with a large compute difference. On the same 88,413-person scoring workload, the explainable boosting machine took 0.1 seconds. TabICL took 3.5 seconds to ingest context and 1,153.8 seconds for GPU inference. The foundation-model runs used default settings without inference optimization, so these figures are not lower bounds. Time and hardware still belong beside score differences this small.
Calibration and thresholds changed the practical story
AUROC did not expose every problem. Class-weighted classical models had expected calibration error between 0.234 and 0.291, while the two foundation models, which were not class weighted, had error of 0.010. Refitting the explainable boosting machine and CatBoost without class weights lowered their errors to 0.0031 and 0.0019. Isotonic recalibration brought every model to 0.005 or lower.
The imbalance treatment explained this difference better than the model family did. A ranking score can remain strong while its raw probabilities are poor risk estimates.
At a validation-selected screening threshold, the explainable model detected 75.4% of women's reported infarctions and 89.0% of men's. Per-group thresholds, reweighting, and an editable glass-box repair each reduced the true-positive-rate gap to about 0.01, with similar specificity. But closing one gap widened the positive-predictive-value gap. Those results attach fairness to a threshold and its tradeoffs. A high AUROC did not confer it automatically.
Uncertainty had the same problem. Marginal conformal prediction reached its overall guarantee while coverage was about 0.86 for men and 0.82 for adults over 60. Group-conditional Mondrian calibration brought each audited stratum to about 0.90, but it increased deferral among older adults and produced empty prediction sets for 9.5% of the youngest group. The repaired guarantee exposed its costs instead of averaging them away.
A practical leakage audit for tabular AI
The paper's open repository publishes the pipeline, tier definitions, model adapters, statistical tests, and edit logs. Its basic procedure can travel beyond health data:
- Write the prediction time explicitly. A field is suspect if it would not exist at that moment.
- Split features into nested tiers, from permissive to deployable, before fitting models.
- Run a simple baseline and the proposed complex model on every tier with the same splits.
- Freeze preprocessing, thresholds, and calibration before testing on a later period.
- Report calibration, subgroup behavior, latency, and abstention beside ranking metrics.
- Treat a uniform score collapse across models as evidence about the feature policy, not proof that every learner is equally capable.
This is the same discipline that makes benchmark reporting useful elsewhere. MarketBench found models mispricing their own coding work because the evaluation asked for a measurable operational decision. SciLitBench tested what models could recover from papers rather than accepting polished summaries. In each case, the task definition decides what the number means.
Teams exposing predictive or analytical models through api.ish.chat should version feature policies as carefully as model IDs. In ish.chat, a transparent audit trail can show which fields existed at prediction time, which tier produced a score, and which calibration and threshold were active. That record is more useful than a leaderboard rank when a result is challenged later.
The authors followed the TRIPOD+AI reporting framework, but the study still has important limits. The outcome and predictors are self-reported. Fatal and undiagnosed events are invisible. Both years come from one US surveillance program, and transport to clinical registries or other health systems remains untested. The fairness audit covers measured attributes at one operating point, and one year of stability does not guarantee future drift.
Two fields moved every model by roughly 0.05 AUROC, while ten models differed by less than 0.005 after their removal. On this benchmark, the feature policy carried far more weight than the choice of learner.
