How Group-Chat Agent Eval Is Done Today

A literature survey for deciding whether group-chat agent evaluation is worth teaching.

1. TL;DR

• The field is real but young and fragmented. The classic NLP foundations (addressee + response selection, conversation disentanglement) are mature and well-formalized, but the agent-era group-chat eval — an LLM acting as one participant in a live multi-party channel — is mostly 2024-2026 work with no shared leaderboard.
• No dominant single mechanism — but a recurring spine exists: classification-accuracy on addressee/turn decisions, plus LLM-as-judge rubrics for social quality, plus simulated-participant rollouts graded by state-diff (consensus change, leakage, milestones).
• The most distinctive group-chat skills are demonstrably unsolved. Frontier LLMs (GPT-4o) score near chance on addressee recognition (Inoue et al. 2025) and near-random on when-to-speak timing (Umair et al. 2024); best multi-party memory systems score 46% (GroupMemBench).
• "When to stay silent" is the biggest hole. Text group-chat benchmarks assume the agent always replies and only ask which reply; silence-as-correct-action is only studied in spoken/multimodal work.
• Simulator fidelity is the dominant validity threat. LLM-simulated participants are over-cooperative ("easy mode") and inflate agent scores (RealUserSim, Sim2Real-Gap) — so any group-chat eval must validate its fake humans before trusting the grade.
• Verdict for teaching: there is a small, honest, demoable toy here — addressee accuracy + a "should I even speak?" gate — that cleanly extends our existing judge (ch05), trajectory (ch06), and human-scoring (ch03) chapters. Not a full new module, but a strong half-chapter.

2. Landscape by sub-topic

2.1 Benchmarks & datasets (agent-in-group-chat)

Synthesis. Real benchmarks exist, but they split into two non-overlapping camps: agent-to-agent settings (MultiAgentBench, MAGPIE) and static-dialogue comprehension (Molweni, Ubuntu IRC). The genuinely new agent-era work (HSII, DICE-Bench, GroupMemBench) converges on the same hard core — track who said what across many speakers before acting — and reports low scores, confirming the difficulty is real. What's missing everywhere is an LLM embedded as one participant in a live human channel with interruptions and side-threads.

2.2 Turn-taking & when-to-speak

Synthesis. When-to-speak is a separately gradable capability from what-to-say, and current LLMs are bad at it — the cleanest result in the whole survey. But almost all rigorous timing work is acoustic/spoken (TRP, VAP); for text group chat the "react vs reply vs ignore" decision is real but essentially ungraded. MUCA's "excessive chime-in %" is the closest thing to a reusable text-restraint metric, and it's a bespoke human study.

2.3 Addressing & mention/speaker resolution

Synthesis. This is the most mature corner — addressee accuracy and disentanglement clustering metrics (VI / one-to-one / Shen-F) are well-established and automatable. The catch: they grade classification on human-authored transcripts, not whether a generative agent routes its own reply to the right person. The modern LLM result (Inoue et al.: GPT-4o barely beats an 80% majority baseline, below chance on next-speaker) shows the capability is unsolved precisely where it now matters most — inside a generating agent.

2.4 Social appropriateness, role/persona & multi-speaker context

Synthesis. Social/persona quality is graded almost entirely by LLM-as-judge rubrics (SOTOPIA's 7 dimensions, PersonaScore's 5 axes), and SOTOPIA's GPT-4-judge-vs-human validation is the template the field copies. The frontier shift is from grading single dialogues to grading emergent group dynamics — DEBATE and Social Laboratory measure conformity, partner influence, and public/private belief drift, which dyadic evals structurally cannot see. NormBank's role-conditioning (a line that's fine from one speaker but not another) is the key idea for context-dependent appropriateness, but per-turn speaker-conditioned scoring in a live group is still underdeveloped.

2.5 Simulators & user-simulation harnesses

Synthesis. The reusable machinery is simulated participants + state-diff grading: spin up LLM "humans," run a rollout, score the change they produce (consensus delta, leakage rate, milestone attribution) rather than a single turn. The honest warning that runs through this entire sub-topic — and is the single most important methodological takeaway — is that the simulated humans are too nice: RealUserSim and Sim2Real-Gap both show LLM simulators inflate agent success and that binary task reward is orthogonal to human-perceived quality. tau2-bench's pass^k is the standard reliability metric practitioners already reach for. True N-simulated-humans-plus-one-agent harnesses with grading remain rare (ProMediate, MUCA, GroupMemBench).

3. How group-chat eval actually works (the methods core)

Across all five sub-topics, the same handful of mechanisms recur. A "group-chat agent eval" is some composition of:

1. Addressee accuracy (the signature metric). Pick the correct interlocutor from prior speakers; scored as classification accuracy, sometimes 4-way (A/B/C/none) against a majority-class baseline (Inoue et al.), sometimes joint with response selection (Ouchi & Tsuboi's ADR + recall@k). It appears verbatim as HSII's r2 stage. This is the cleanest, most automatable group-chat probe and the historical root of the field.

2. Response-decision / when-to-speak F1. Binary "is this a valid moment to speak?" graded by precision/recall/F1 against human-labeled transition points (TRP work, TurnGPT, VAP family). The crucial variant — silence as a first-class correct label (respond vs backchannel vs stay-silent) — exists only in spoken/multimodal work (MM-When2Speak); MUCA's "excessive chime-in %" is the text proxy.

3. Conversation disentanglement / structure metrics. Before an agent can act it must untangle interleaved threads: reply-link P/R/F1 on edges, then clustering scored by VI, one-to-one overlap, and Shen-F (irc-disentanglement), plus discourse-dependency F1 (Molweni). This is the prerequisite-skill layer.

4. Simulated-user rollout + state-diff. Stand up LLM participants, run a multi-turn rollout, and grade the delta they produce: consensus change and mediator-effectiveness slope (ProMediate), engagement/evenness/consensus (MUCA), tool-action/DB end-state with pass^k reliability (tau2-bench), milestone attribution (MultiAgentBench). Always paired with a fidelity check — USI (Sim2Real-Gap) or PT3 (RealUserSim) — because over-cooperative sims inflate the grade.

5. LLM-as-judge social rubrics. Multi-dimension 1-5 / Likert rubrics for the qualities that have no exact-match answer: SOTOPIA's 7 dimensions, PersonaScore's 5 axes, ProMediate's Mediator Intelligence. SOTOPIA's GPT-4-judge-validated-against-humans is the calibration template — with the documented caveat that judges are weaker on subtle dimensions (Social Rules, Secret), and judge reliability across many speakers in long transcripts is largely unvalidated.

6. Group-specific safety / leakage. A uniquely multi-party axis: what does the agent reveal to other participants? Measured as contextual-integrity leakage rate (MAGPIE), and as speaker-grounded belief/audience-adaptation accuracy (GroupMemBench).

The recurring shape: a cheap, automatable classification gate (addressee, when-to-speak, disentanglement) + an expensive judge/rollout layer (social rubric, state-diff) + a fidelity check on the simulated humans.

4. Open gaps (the genuine frontier)

• No live, human-in-the-loop, embedded-agent benchmark. Everything is agent-vs-agent or static-transcript classification. An LLM as one participant in a sustained real channel with interruptions and side-threads is unmeasured.
• Silence-as-correct in text is unbenchmarked. Text MPC benchmarks assume the agent always replies; there is no adopted metric for over-eager interjection / false-positive responding in text group chat.
• Everything is dyadic-or-triadic. Genuine N>3 dynamics (overlapping threads, shifting floor) are barely covered, and triadic VAP shows dyadic models degrade as parties grow — existing metrics may not transfer.
• No unified scorecard. Addressee accuracy, evenness, leakage rate, disentanglement VI, and milestone KPIs are siloed per paper. There is no joint metric scoring whether + when + to whom + what in one pass, and no "group-chat competence" leaderboard.
• Speaker-grounded memory / belief tracking is barely off the ground. GroupMemBench (2026) is the first standardized probe; best systems sit at 46%.
• Simulator fidelity is unsolved and under-reported. Sims are "easy mode"; almost no group-chat eval reports a USI/PT3-style fidelity score alongside agent grades, and persona/decoding configs that move scores are rarely pinned or released.
• Judge validity at multi-party scale is unvalidated. LLM-judge bias is documented dyadically (SOTOPIA); no calibrated, speaker-attributing judge or inter-rater protocol exists for long multi-party transcripts.
• Multi-party safety beyond privacy is nearly absent — colluding participants, cross-user prompt injection, moderation of harmful cross-user dynamics.
• Real-platform grounding is thin. Most agent-era data is synthetic/GPT-generated; few use real Slack/Discord/Teams traces, so production ecological validity is unverified.

5. Relevance to our demo

Honest take: yes, there's a demoable toy — a narrow, sharp one — but it's a half-chapter extension, not a new module. The strongest single asset is that the most distinctive group-chat skill (addressee resolution) is cleanly automatable and demonstrably broken in frontier models — which is exactly the kind of "watch it fail with your own input" insight the curriculum is built around.

Concrete demoable toy: an "Is this for me, and should I speak?" gate.

• Hand the agent a short 3-4-person transcript (synthetic, hardcoded in data/ like our other toy sets) and ask it to (a) pick the addressee of the last message and (b) decide respond / react / stay-silent.
• Grade addressee against a gold label as accuracy vs a majority-class baseline (the Inoue et al. framing) — this viscerally shows "the model barely beats always-guessing." That's a great teaching beat.
• This is a near-exact transplant of our existing ch02 (code grader) exact-match/classification pattern, just over a multi-party transcript instead of a math answer.

How it extends specific existing chapters:

• ch06 (trajectory grader) — the natural host. A trajectory through a group chat adds a per-step addressee and speak/stay-silent decision to grade alongside tool calls. DICE-Bench's "params scattered across speakers" and ProMediate's intervention-latency are the conceptual upgrades; we'd toy-ify them, not import them.
• ch05 (LLM as judge) — directly reusable. SOTOPIA's 7-dimension rubric and the GPT-4-judge-validated-against-human pattern are exactly ch05's mechanism; a "social appropriateness" rubric (was the chime-in warranted? was it addressed to the right person?) drops into our existing judge with a different rubric, reusing ch04's rubric-design widget.
• ch03 (human as judge) — the calibration story carries over: MUCA's "56% of users said it chimed in too much" is the human-scored restraint signal, and our ch07 (graders disagree) calibration view already compares human vs judge.

What to not oversell: the full research frontier (live human channels, N>3 floor management, simulator fidelity, speaker-grounded memory) is heavy and unsettled — not toy-able in under three minutes. The simulator-fidelity caution (Sim2Real-Gap: "your fake users are too nice") is a genuinely good concept slide but not a runnable demo. Recommend: a single chapter ("Who is this for, and should I answer?") built on addressee-accuracy-vs-baseline + a social-rubric judge, explicitly reusing ch02/ch05/ch03 machinery. That's real, honest, and finishes in three minutes; anything larger is a research project, not a mentorship demo.