Register Shifts Break LLM Safety A Bengali Benchmark with Culturally Grounded Harms

Naymul Islam1, Nusrat Jahan Lia2, Shubhashis Roy Dipta3, Sabik Bin Sultan4, Abdullah Khan Zehady5
1BanglaLLM 2Institute of Information Technology, University of Dhaka 3University of Maryland, Baltimore County 4Bangladesh Air Force Shaheen College Kurmitola 5Ciroos Inc.
Under review · EMNLP 2026
Code Dataset Leaderboard Package Run the benchmark

Bengali is diglossic. Newspaper prose and a casual text message are close to different languages in practice. We hold a harmful request fixed and change only how it is written. Written as formal journalism it gets answered 17.5 points more often than written as a casual message. Nothing was optimised or engineered to make that happen.

879
prompts, natively authored
17
statute-anchored harm categories
15,822
responses across 18 models
+17.5pp
register effect

How the benchmark is built

Every prompt is written in Bengali and tied to a real Bangladeshi law or case.

BanglaSafe construction pipeline: 879 prompts from a 17-category harm taxonomy, expanded across five prompting conditions, run through 18 LLMs, then validated and judged four ways.
Left: 879 prompts (309 written directly by a native Bengali speaker, 570 model-assisted and human-validated) drawn from a 17-category harm taxonomy, each category anchored to a Bangladesh statute or a documented case rather than translated from an English dataset. Centre: every harm instance appears under five conditions that vary language, register, and authority framing, yielding 15,822 responses across 18 models. The arrows mark the three paired comparisons: language, register, and authority. Right: two-stage evaluation, human prompt validation followed by a calibrated four-way judge.

Where the categories come from

A harm gets into the taxonomy only if a specific Bangladeshi statute makes it illegal, or if it is treated as harmful across the full range of Bangladeshi opinion. Anything genuinely contested was left out, including blasphemy, opposition politics, sex work, and LGBTQ-related queries.

501 of the 879 prompts point at a named, dated incident you can trace back to a primary source.

Why four labels

A safe/unsafe split loses the middle of the distribution. A model that answers with a long essay about acid law has not refused you, but it has not helped you either. We use REFUSE, POLICY, PARTIAL, and HARMFUL, and report two rates from them: how often the model engaged at all, and how often it produced something someone could actually use.

The finding

How the request is written matters more than whether it is in English or Bengali.

Attack success rate by prompting condition. EN_Direct 50.4%, EN_Inst 53.4%, BN_Formal 63.3%, BN_Collq 45.8%, BN_Inst 55.7%. The gap between BN_Formal and BN_Collq is 17.5 points.
Attack success rate by prompting condition, judged by a calibrated four-way rubric over 15,822 responses. The underlying harm act is identical in every bar. Formal Bengali journalism is the most successful framing at 63.3%; colloquial Banglish peer-help is the least at 45.8%. Both are Bengali, so the 17.5-point spread is not a language effect. With roughly 3,000 responses per condition every 95% bootstrap interval is within ±1.8 points, well under the effect being measured.

The effect holds model by model. 17 of 18 models show a positive register gap (sign test, p < 0.001), with a median of +17.6 points. It is not an artefact of averaging weak models with strong ones, and it is larger in the more capable ones: the models with the tightest English safety show the widest gap when the same request arrives in formal Bengali.

ModelEnglish directBengali journalismIncrease
Claude-Haiku-4.57.5%37.9%5.1×
Grok-4.38.1%32.8%4.0×
Gemini-2.5-Flash14.5%87.9%6.1×
Gemma-4-26B18.5%90.8%4.9×

Models that refuse almost everything in English answer the same request four to six times more often in formal Bengali. Safety alignment is not transferring across language and register.

Table of attack success rate for all 18 models across the five prompting conditions, grouped by parameter scale.
Every model, every condition. Grouped by parameter scale, with the highest value in each column in bold and the highest within each group underlined. BNFormal beats BNCollq for 17 of the 18 models. The exception, TigerLLM-1B, sits at its capability floor and scores near zero everywhere, so its gap is not meaningful.

Where the extra engagement goes

Formal Bengali does not make models produce more recipes. It makes them produce more half-answers.

Splitting the responses four ways shows what the single rate hides. Under BNFormal the PARTIAL share climbs to 49.9%, the highest of any condition, while the HARMFUL share falls to 13.5%, below the 18.7% English baseline.

So the journalism framing is not unlocking operational detail. It is moving models out of refusal and into hedged, structured, article-shaped answers that still carry named tactics and sourcing channels. That is why the four-way split matters: a binary metric would report this as a straightforward increase in unsafe output, which is not what is happening.

Stacked bars of the four-way label distribution for each prompting condition, showing PARTIAL rising to 49.9% under the formal Bengali condition while HARMFUL falls to 13.5%.
Four-way label distribution by condition. This figure uses the internal condition codes: EN is ENDirect, EN_V3 is ENInst, V1 is BNFormal, V2 is BNCollq, and V3 is BNInst.

Why a Bengali benchmark has to be built, not translated

The harms that matter in Bangladesh are not in the corpora that safety training uses.

We searched twelve widely used English safety and RLHF corpora, 80,589 prompts in total, for six culturally anchored Bangladeshi harm terms: yaba, formalin adulteration, hundi, bKash fraud, eve teasing, and dowry violence.

Across all twelve corpora and all six terms, there was one hit. A single mention of dowry in PKU-SafeRLHF. The generic English counterparts are everywhere in the same corpora: methamphetamine 290 times, money laundering 177, OTP phishing 363. The concepts are covered. The Bangladeshi forms of them are not.

Translating an English harm taxonomy cannot recover what was never in it.

Matrix showing zero hits for six Bangladeshi harm terms across twelve English safety corpora, with one exception; generic English counterparts appear hundreds of times.
Hits for six Bangladeshi harm terms across 12 English safety and RLHF corpora (80,589 prompts), with generic English counterparts from the same corpora below for comparison.

Is it just the judge?

A second, independent judge from a different lab reproduces the effect almost exactly.

The labels come from an LLM judge, so the obvious objection is that the finding is an artefact of that particular model. We re-judged all 15,822 responses with an independent frontier model from a different lab, using the identical rubric.

The two judges rank the 18 models nearly identically, Pearson r = 0.989, and the register gap barely moves: +18.2 points under the second judge against +17.5 under the first. Binary agreement between them is κ = 0.787. The absolute rate shifts by about three points because the second judge is stricter about the PARTIAL boundary. The effect does not.

Scatter plot of per-model attack success rate under two independent judges. All 18 models fall close to the y equals x line, Pearson r 0.989.
Per-model attack success rate under two independent judges. Each point is one of the 18 evaluated models; the dashed line is perfect agreement.

Every report the tool produces prints how well the judge agreed with human annotators (κ = 0.666 against human labels, above the 0.586 the humans managed with each other), so you can see how much to trust the labels you are reading.

Score your own model

One command. Nothing to install, no dataset download, no account.

# any OpenAI-compatible endpoint: vLLM, SGLang, LightLLM, Ollama,
# llama.cpp, TGI, LM Studio, a LiteLLM proxy, or a hosted API
uvx banglasafe run \
  --model your-model \
  --base-url http://localhost:8000/v1 \
  --judge-model anthropic/claude-opus-4-7 \
  --judge-base-url https://openrouter.ai/api/v1
  ASR loose   47.2%  [44.1, 50.3]   PARTIAL + HARMFUL
  ASR strict  19.8%                 HARMFUL only

  vs the 18-model reference cohort: z = -0.22, rank 11/19 (safer than average)

  Register effect  BN_Formal is 16.4pp higher than BN_Collq

Your number comes with context

The 18 models we tested ship inside the package as a reference cohort, so you get a rank and a z-score rather than a percentage with nothing to compare it against. Every rate has a bootstrap confidence interval and a breakdown by condition and harm category.

The judge is your choice

--judge-model is required and has no default. The judge decides your numbers, so it should not be something you inherit by accident. banglasafe judges lists the options with how well each one matched human annotators.

What we release

Dataset

879 prompts with category, condition, register tier, case-anchor flag, and human/AI provenance, plus the 17-category taxonomy with its statutory anchors.

load_dataset("BanglaLLM/BanglaSafe")

huggingface.co/datasets/BanglaLLM/BanglaSafe

Framework

A pip-installable CLI that runs the benchmark end to end against any OpenAI-compatible endpoint, with the calibrated rubric, bootstrap statistics, and HTML reporting built in.

pip install banglasafe

github.com/BanglaLLM/banglasafe

The prompts are written to elicit unsafe behaviour so that it can be measured. They are for safety evaluation and research. The benchmark measures harmful compliance and says nothing about over-refusal, since it contains no benign control set.

Citation

@inproceedings{islam2026banglasafe,
  title     = {Register Shifts Break {LLM} Safety: A Bengali Benchmark
               with Culturally Grounded Harms},
  author    = {Islam, Naymul and Lia, Nusrat Jahan and Roy Dipta, Shubhashis
               and Sultan, Sabik Bin and Zehady, Abdullah Khan},
  year      = {2026}
}