Author profiles — Ahmed Taha: ORCID · Google Scholar · ResearchGate · GitHub · Hugging Face
Radiology vision–language models may change clinically meaningful report content when the same pathology is presented under different patient demographics, but observational subgroup audits cannot isolate this effect. We present SpineFairBench, a paired counterfactual benchmark for auditing demographic sensitivity in spinal radiology report generation. SpineFairBench varies apparent age and sex in source/counterfactual spinal radiographs under a target-pathology-preservation criterion. It assesses a frozen nine-model VLM panel under a locked report-generation prompt with two pre-registered primary endpoints: recommendation change rate and diagnostic-label consistency. Retained outputs demonstrate measurable recommendation drift in all nine models. In most retained models, management recommendations are less stable than diagnostic-label overlap under the same demographic edit. A pre-registered findings-first mitigation analysis with a binding interpretation rule produced a negative result on the eligible subset, supporting a predominantly perceptual rather than interpretive locus for the two models on which it could be evaluated. In blinded validation by three board-certified radiologists, clinical plausibility and target-pathology preservation were supported for 443/450 pairs (98.44%) under a 2-of-3 majority rule. Reviewers selected "Cannot tell" for 96.8% of edit-detectability responses.
Visual abstract. SpineFairBench is a paired counterfactual benchmark. A source spinal radiograph is edited to change its apparent age and sex — for example to an elderly-female presentation with lower apparent bone density — producing a near-identical counterfactual image in which the target pathology is held constant, verified by automated quality control. Both the source and the counterfactual image are then sent to a frozen nine-model vision–language panel under a locked prompt, and their generated reports are compared. Across the panel the diagnostic label stays stable while the management recommendation changes — for example, from routine follow-up to specialist referral.
@misc{taha2026spinefairbench,
title = {SpineFairBench: A Counterfactual Benchmark for Auditing
Demographic Sensitivity in Spinal Radiology VLM Reports},
author = {Taha, Ahmed and Taeha, Abdelrahman and Ahmadzada, Muzzammil},
year = {2026},
note = {Preprint},
doi = {10.13140/RG.2.2.12278.36160},
url = {https://www.researchgate.net/publication/404620116}
}