Geographic Prompts Shift LLM Kidney Screening Results
Summary:A study on zero-shot chronic kidney disease (CKD) screening kept patient clinical inputs unchanged while replacing the country attribute in the prompt with 25 geographic contexts. It evaluated Qwen3 8B and Mistral 7B on a community-based Bangladesh dataset. Non-CKD predictions were largely stable, while CKD-positive predictions were notably context-sensitive. For Qwen3, the number of CKD predictions fell from a baseline of 65 to as low as 50 across country contexts. The groupings did not follow
Geographic context should not function as a clinical variable
This study frames zero-shot chronic kidney disease (CKD) screening as a controlled intervention. It holds all patient clinical inputs constant while changing only the country attribute in the prompt and observes whether the model's output changes. The dataset contains 284 community participants from Bangladesh, including 112 with CKD and 172 without it. The experiment evaluates Qwen3 8B and Mistral 7B across 25 country contexts.
The main analysis uses a common-correct subset of 156 participants whom both models classified correctly at baseline. This isolates cases whose decisions can be perturbed by geographic context, rather than estimating overall diagnostic performance on the complete cohort.
The results show class-level asymmetry. Non-CKD predictions remain stable in nearly every country context, whereas CKD-positive predictions are substantially more sensitive. Qwen3's number of CKD predictions falls from a baseline of 65 to as low as 50. Mistral exhibits smaller but non-negligible variation. For the same patient data, changing only the country name can therefore push a model across the disease-decision boundary.
The perturbation may reflect token representations rather than real-world differences
The country contexts did not form clusters that matched known epidemiological similarities or healthcare systems. Their groupings instead appear to reflect how country tokens are represented internally by each model. A country name may therefore be acting as a hidden predictive feature rather than meaningful population-level evidence.
Mechanistically, country names enter the context as ordinary discrete tokens. Their embeddings influence attention and subsequent hidden states. If a country's token co-occurs with particular disease descriptions, clinical questions, or answer templates during training, the model can learn an implicit correlation. The country attribute then stops being merely contextual metadata and becomes a shortcut for predicting the label.
This behavior is particularly concerning for clinical decision support. A model can preserve an apparently plausible explanation while silently relying on an unstable geographic proxy. Audits based only on aggregate accuracy may miss such failures because the same country prompt can be used for every test case, hiding the fact that the model is fragile under context changes.
The evidence remains insufficient for clinical deployment
The study uses a limited, single-community cohort. Restricting the analysis to participants both models initially classified correctly removes difficult cases and may understate perturbation rates in broader clinical populations. The experiment also covers only two models in the 7B–8B class. It does not establish whether larger models, medically fine-tuned systems, quantized inference, or alternative prompt templates exhibit the same behavior.
The mechanism also requires further isolation. Country-token position, ordering, and surrounding instructions can alter attention allocation. It remains unclear whether the shifts arise primarily from learned disease priors, positional effects, sampling noise, or a fallback strategy when clinical evidence is incomplete.
The paper should consequently be treated as a robustness warning rather than a complete estimate of clinical risk. It identifies a plausible failure mode that deserves systematic testing.
Add geographic invariance tests before deployment
Medical AI evaluation should not rely only on model comparisons and aggregate AUC. Engineering teams can add geographic counterfactual tests: duplicate each patient record, change only the country, region, or hospital-context field, and flag outputs whose class, probability, or referral recommendation changes materially.
Useful metrics include:
- Prediction flip rate: the proportion of cases whose positive/negative label changes after replacing the country token.
- Probability shift: the difference between the maximum and minimum positive probability across country contexts for the same patient.
- Clinical threshold crossing: the proportion of outputs that cross a predefined referral threshold.
- Context-specific calibration: changes in calibration error across country contexts.
- Worst-context risk: sensitivity and false-negative rates under the least favorable geographic prompt.
Training data should include comparable cases from multiple countries so that geographic names do not become shortcuts correlated with disease labels. Paired examples with similar patient characteristics but different geographic backgrounds would provide a more direct test of proxy-variable reliance.
Use abstention when outputs are context-sensitive
In an operational screening pipeline, an LLM output should not directly trigger clinical action. A safer design can evaluate the same patient under multiple semantically equivalent geographic prompts. If the predicted class or referral threshold changes across contexts, the system should flag the result as contextually unstable and route it to a calibrated risk model, rules engine, or human review.
This approach increases inference cost, but it provides a more auditable failure signal than a single opaque completion. Prompt templates may also normalize or remove the country field before inference. At the model level, researchers can probe country-token associations with disease labels and measure attention concentration or hidden-state separability.
Practical significance
The study does not establish that the models acquired a valid understanding of geographic epidemiology. Instead, it shows that country names can become uncalibrated decision shortcuts when clinical information is incomplete or noisy. Its engineering value is a new admission criterion for clinical AI: when clinical inputs remain unchanged, non-clinical context should not systematically alter high-stakes decisions.
— END —Source: ArXiv Machine Learning (cs.LG) (2026-11-09)
Tags: #LLM Reliability #Medical AI #Geographic Bias #Prompt Sensitivity #CKD Screening
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments