This study evaluated three open-weight large language models—Gemma3-12B, Bielik-11B-v3, and Qwen3-4B—using World Values Survey data for 63 demographic personas across three countries. The models were analyzed for distributional misalignment with normalized Wasserstein distance.
Contrary to expectations, no model favored its home country; notably, the Chinese-built Qwen3-4B performed worst on its own Chinese population. Fine-tuning with LoRA on the five worst-case personas required fewer than 1,200 training pairs and less than 15 minutes on a single GPU. This process reduced bias by 16.8% for Bielik-11B, with all five personas showing improvement.
However, country-level analysis revealed that fine-tuning redistributed bias rather than eliminated it. For example, Bielik's worst-case personas shifted from American to Chinese elderly, with no overlap between pre- and post-correction sets. This indicates that bias mitigation can involve bias redistribution among demographic groups.
This is the first known study to target worst-case demographic personas with LoRA fine-tuning for cross-cultural bias mitigation in large language models.
Source: https://arxiv.org/abs/2609.04485