SinoGlyphBench is a benchmark designed to assess the impact of glyph-level obfuscation on language models. The benchmark consists of paired evaluations of 12 LLMs and MLLMs, comparing original and glyph-obfuscated Chinese text and image inputs. The benchmark identifies label-critical semantic anchors and perturbs them, along with background context, to differentiate between genuine corruption and surface variation.
Across 176,916 paired evaluations, the benchmark observed an increase of 6.1 percentage points in harmful false-negative rates and 4.7 percentage points in false-positive rates. Four-way accuracy decreased by 5.0 points. Models retained 75.7% of the decisions made on the original inputs.
Analysis indicated that full-scope perturbations caused the greatest degradation, while anchor-only perturbations were more damaging than background-only perturbations. Cross-script substitution proved particularly challenging within the text modality. The benchmark identified observable mismatches in visible-form reading, intended-message recovery, and final safety judgment.
The results demonstrate that current models remain brittle when processing Chinese content utilizing non-canonical glyphs. This highlights a critical area for improvement in language model moderation systems.
Source: https://arxiv.org/abs/2609.05843