OCD Clinical AI Reached AUROC 0.653 but Failed Site Generalization

A 2026 ENIGMA-OCD preprint tested a transformer model on resting-state fMRI from 1,706 participants and reached AUROC = 0.653 ± 0.039 for OCD classification.¹ The more important result was the failure mode: held-out-site performance ranged from AUROC 0.427 to 0.819, which is exactly the gap clinical AI has to close before brain-scan diagnosis is credible.

Resting-state fMRI measures synchronized blood-oxygen fluctuations while a person is not performing a task. The model in this study used those functional-connectivity patterns to classify obsessive-compulsive disorder (OCD), a psychiatric condition defined clinically by intrusive obsessions, compulsions, or both.

AUROC is the area under the receiver operating characteristic curve, a classification metric where 0.50 is chance and 1.00 is perfect separation. A value of 0.653 is above chance, but it is not close to a reliable diagnostic test.

AUROC 0.653 Is Not a Clinical Diagnostic Tool

Pak et al. used the ENIGMA-OCD consortium, which included 869 OCD patients and 837 healthy controls across 23 international sites. The dataset is unusually valuable because it forces a model to confront the mess that small single-site studies can hide: different scanners, protocols, populations, motion patterns, and clinical compositions.¹

The main model, Multi-Band Brain Net, achieved AUROC = 0.653 ± 0.039 with a 95% corrected confidence interval of 0.601 to 0.704. That was competitive with other tested models, including support vector machine and brain network transformer baselines. Competitive is not the same as clinically ready.

For diagnosis, the relevant question is not whether the model beats chance in a pooled research dataset. The relevant question is whether it can make a reliable individual-level decision in a clinic it has never seen. The answer here was no.

Leave-One-Site-Out Testing Exposed the Generalization Gap

Leave-one-site-out validation trains the model on all but 1 site, then tests it on the held-out site. This is stricter than randomly mixing participants across sites because it asks whether a model works when the scanner, protocol, and recruitment context shift.

Across 23 held-out sites, AUROC varied from 0.427 to 0.819. That range is the central finding. The best site makes the system look promising; the worst site makes it worse than useful. A clinical model cannot be trusted if its apparent performance depends heavily on which hospital, scanner, or acquisition protocol produced the image.

Prior ENIGMA-OCD work has already shown that OCD has detectable group-level brain patterns, including structural differences in large pooled samples.² Group-level biological signal is not enough for individual diagnosis. Psychiatric imaging effects are usually small, distributed, and sensitive to sampling choices.

Pretraining Improved Calibration, Not Case-Control Separation

Large-scale pretraining is often sold as the way to make AI more robust. In this paper, pretraining on 40,783 UK Biobank participants did not boost accuracy. The pretrained model reached AUROC = 0.639 ± 0.039, slightly below the from-scratch model.¹

Pretraining did improve calibration. Calibration means the model’s confidence matches its actual correctness. A well-calibrated model is less likely to say “I am 95% sure” when its predictions are only correct far less often. Expected calibration error improved with p < 0.001, and the model produced fewer overconfident 90% to 100% predictions without increasing case-control discrimination.

That is a real gain. In clinical AI, knowing when the model is uncertain is valuable. But the decision-referral analysis did not show that uncertainty could be used to create a ready triage tool, and pretraining did not close the held-out-site performance gap.

Scanner Effects Were Technical, Not Psychiatric Insight

The study separated clinical and technical sources of model uncertainty. Scanner manufacturer, head motion, data quality, site sample size, and acquisition parameters shaped reliability. For example, larger site sample sizes were associated with lower uncertainty, beta = −0.120, p < 0.001, while higher head motion increased uncertainty, beta = 0.033, p < 0.001.¹

That pattern is a warning. A model can look like it is learning OCD biology while partly learning scanner and protocol differences. Pretraining helped remove scanner-manufacturer bias, with manufacturer effects significant in the from-scratch model, p = 0.038, but not in the pretrained model, p = 0.325. That is progress on hardware invariance, not proof of diagnosis.

Evidence-strength note: this was a medRxiv preprint using a large multi-site research dataset. It is useful for stress-testing clinical-AI claims, but it should not guide diagnosis or treatment. The paper itself states that calibration remains insufficient for clinical integration.

OCD Biomarkers Need Reliability Before Interpretability

Interpretability is attractive because it offers a brain-map explanation for the classifier. The model highlighted functional-connectivity patterns that may fit known OCD circuits. That is scientifically useful, especially when it aligns with orbitofrontal, striatal, and broader network models from prior OCD research.³

Interpretability cannot rescue poor generalization. If the prediction changes drastically across sites, a plausible brain explanation may be partly post hoc decoration. For clinical use, reliability comes first: the same kind of patient scanned at a different site should not flip the model from strong to useless.

• Useful today: benchmarking multi-site psychiatric AI under realistic scanner and site variation.
• Not useful today: diagnosing OCD from an individual resting-state fMRI scan.
• Next technical step: domain-generalization methods trained to resist site shifts rather than merely harmonizing them away.

Clinical Deployment Would Need a Different Validation Target

A deployable OCD imaging model would need a target more specific than case-control separation. Clinicians do not usually need a brain scan to decide whether a patient has obvious obsessions and compulsions. The harder clinical questions involve severity, treatment selection, comorbidity, relapse risk, and response to exposure and response prevention, SSRIs, or augmentation.

That means the next generation of models should be tested against clinical decisions. A useful model might identify which patients are likely to need intensive exposure therapy, which patients are likely to relapse after discontinuing treatment, or which symptom dimensions map onto circuit-level targets. A modest case-control AUROC cannot answer those questions.

Pak et al.’s preprint is valuable because it refuses the easy single-site victory. The model had to face 23-site heterogeneity, and the result was uncomfortable. That discomfort is productive: it shifts the field from impressive-looking accuracy toward reliability, calibration, and site robustness.

For patients, the practical read is conservative. A brain scan that performs well in a research consortium still has to survive ordinary clinic variation: different MRI vendors, older machines, shorter scan time, medication differences, pediatric vs. adult cases, tic disorders, depression, anxiety, and motion from compulsive discomfort. Those are not edge cases in OCD care; they are the population.

A useful validation study would therefore test deployment-like conditions alongside pooled accuracy. It would report performance by site, scanner vendor, age group, medication status, comorbidity, and motion level, then show how often the model abstains when uncertainty is high. Without those strata, a single AUROC can hide the exact failure pattern clinicians need to know.

That reporting standard matters most for negative use cases: ruling out OCD, deciding not to refer, or reassuring a patient because a scan looks “low risk.” A weak model can do harm through false reassurance as easily as through false alarms.

Clinical target problem: OCD diagnosis may be the wrong first deployment target for imaging AI. The diagnosis is usually made from symptoms, impairment, duration, and clinical interview, not from a hidden brain state. A scan-based tool would have to beat ordinary assessment in cases where diagnosis is uncertain, comorbidity is heavy, or treatment planning needs information that symptoms alone cannot provide.

That shifts the benchmark. Instead of asking whether a model can separate OCD cases from controls, researchers could ask whether it predicts exposure and response prevention outcome, SSRI response, deep brain stimulation target selection, relapse risk, or symptom dimensions such as contamination, checking, or symmetry. Those tasks are harder, but they are closer to clinical value.

Questions About OCD Clinical AI

Did the model diagnose OCD accurately?

No. It classified above chance in a research dataset, but AUROC = 0.653 is too modest for diagnosis, and held-out-site performance varied widely.

Did UK Biobank pretraining help?

It helped calibration and scanner-manufacturer bias. It did not improve classification accuracy or close the site-generalization gap.

Does this mean OCD has no brain signal?

No. ENIGMA-OCD and other studies support group-level brain differences. The failure is individual clinical prediction across sites, not the absence of any neurobiological signal.

Related Posts:

• Extra-Virgin Olive Oil Raised Brain Connectivity in a 9-Person Pilot
• SloCAD Sleep EEG-fMRI Maps Brain Network Deviations Before Diagnosis
• Alzheimer’s Blood Biomarker Models Lose Rule-Out Power Across Cohorts
• Subcortical Connectivity Differs in Schizophrenia, Bipolar, Depression
• Autism vs. ADHD Connectome Study: Frontal-DMN Signal Tracks Autism Severity