Does Country Matter? Evaluating Geographic Sensitivity of Large Language Models in Zero-Shot Chronic Kidney Disease Screening
Paper · https://arxiv.org/abs/2611.08507
arXiv:2611.08507v1 Announce Type: new Abstract: Large language models (LLMs) show strong zero-shot reasoning capabilities, yet their sensitivity to prompt context raises questions about reliability in deployment. This study investigates whether geographic context embedded in prompts influences LLM-based chronic kidney disease (CKD) screening, independent of any change in patient features. Using a community-based CKD dataset from Bangladesh (284 participants with 112 CKD and 172 non-CKD), we systematically varied the country attribute across 25 global contexts while holding all clinical inputs constant, and evaluated two open-weight LLMs (Qwen3 8B and Mistral 7B). Experiments were conducted on a common-correct subset of 156 participants for whom both models produced correct baseline predictions. Results show that non-CKD predictions were highly stable across geographic contexts, with both models maintaining consistent non-CKD predictions in nearly all settings. In contrast, CKD-positive predictions were considerably more sensitive. For Qwen3, CKD prediction dropped from a baseline of 65 to as low as 50 across country contexts, while Mistral showed smaller but non-negligible variation. Country contexts did not group according to known epidemiological or healthcare similarities; instead, groupings appear to reflect how country tokens are internally represented within each model. These findings demonstrate that geographic tokens can systematically shift clinical screening outcomes despite identical patient inputs, with implications for the deployment of LLM-based decision support in diverse healthcare settings.
First seen by Oriel on .
AI summary
The study finds that changing only the country named in prompts can systematically alter LLM-based zero-shot chronic kidney disease screening outcomes.
- Researchers tested Qwen3 8B and Mistral 7B using a Bangladesh dataset of 284 participants: 112 with CKD and 172 without CKD.
- They varied the country attribute across 25 contexts while keeping clinical inputs unchanged, focusing on 156 participants both models classified correctly at baseline.
- Non-CKD predictions remained stable across nearly all geographic contexts, while CKD-positive predictions showed substantially greater sensitivity.
- Qwen3's CKD-positive count fell from 65 at baseline to as low as 50; Mistral showed smaller but still meaningful variation.
- Country groupings did not follow epidemiological or healthcare similarities, instead appearing related to each model's internal representation of country tokens.
#large language models #healthcare ai #chronic kidney disease #geographic bias
Generated by gpt-5.6-sol. Check the original for details.
Aliases
| Source | External ID | URL |
|---|---|---|
| arXiv AI Papers | oai:arXiv.org:2611.08507v1 | https://arxiv.org/abs/2611.08507 Read discussion |