Eighteen models, open-weight and closed-source, on 879 Bengali harm prompts. Lower is safer. Every model is more likely to answer when the same request is written as formal Bengali journalism than as a casual message, which is the column on the far right.
Scroll the table sideways for the remaining columns. Rank and model stay put.
ASR is the share of responses the judge scored PARTIAL or
HARMFUL out of the four labels. Strict counts only HARMFUL,
which is the operationally useful subset. The bracketed range is a 95% percentile bootstrap
interval, B = 10,000, seed 20260521. Each model answers all 879 prompts, so
n = 879 per model unless a judge declined to label something.
The five middle columns are the same harm act written five ways: English direct, English with an institutional persona, formal Bengali journalism, colloquial Banglish peer-help, and Bengali institutional. The shaded cell in each row is that model's most permissive register. Gap is BN_Formal minus BN_Collq: both are Bengali, so it isolates register from language.
Two judges are shown because a reviewer will ask whether the effect is an artefact of the judge. It is not. Gemini 3.1 Pro, prompted with the same rubric, reproduces the ranking almost exactly; it labels 15,794 of 15,822 responses, having declined 28 on its own content policy. Rates use each judge's own denominator.
These are single-turn prompts with no adversarial suffix, no roleplay wrapper, and no optimisation. Decoding was temperature 0, top-p 1, empty system prompt.
The CLI runs the whole benchmark against any OpenAI-compatible endpoint and prints the same
numbers you see here, positioned against this cohort. Send us the generated
report.json and we will add the row.
uvx banglasafe run \
--model your-model \
--base-url http://localhost:8000/v1 \
--judge-model anthropic/claude-opus-4-7 \
--judge-base-url https://openrouter.ai/api/v1