LoSoNA: A Benchmark for Local Social Norm Adaptation in Group Conversations
Online group chats are social spaces with local conversational norms that are rarely stated explicitly. The ability a...
Source Evidence
What Changed
Online group chats are social spaces with local conversational norms that are rarely stated explicitly. The ability a...
Why It Matters
**Why it matters** A robust ability to infer and obey hidden conversational norms is a prerequisite for safe, persuasive, and context‑aware LLMs in team‑oriented applications—any shortfall could cause misunderstandings in collaborative tools, lead to reputational damage for brands, or expose biases when agents fail to respect local group culture. LoSoNA demonstrates that current flagship models still struggle, revealing a critical gap in real‑world deployment and a benchmark that will shape future training and alignment focus.
Confirmed Facts
Online group chats are social spaces with local conversational norms that are rarely stated explicitly. The ability and willingness of LLM-based agents to recognize and adapt to these norms remains mostly unexplored. We introduce LoSoNA, a benchmark for local social norm adaptation in multi-party chat. Each scenario gives a subject model a curated group-chat transcript in which non-subject participants demonstrate a hidden local norm, followed by a final elicitor turn that forces a response revealing whether the subject has inferred that norm. We evaluate eight frontier and open-weight models under four prompting conditions that vary how explicitly the model is told to treat the prior conversation as evidence for how it should answer. Naive prompting remains limited for most models; explicit norm-aware prompting helps unevenly, with Gemini 3.1 Pro reaching 84.2% and Claude Fable 5 reaching 81.6%, while several other models show small gains or regressions. LoSoNA contributes to recent calls for evaluating LLM social capabilities by testing whether models can infer local conversational norms from precedent and use them in a one-turn group-chat response.
Who Is Affected
- Anthropic
- Google DeepMind
- Fable
- Claude Fable
- AI product teams
What To Watch Next
- Watch for benchmark validation, API availability, pricing, limits, and early customer adoption.
- Watch whether additional sources confirm the same claim.
You will be redirected to huggingface.co.