SlackBench Benchmarks Collaborative Agents Across Code and Workplace Conversations
Summary:SlackBench introduces a reproducible environment for evaluating agents on collaborative research and engineering workflows. It links multi-party workplace conversations with evolving GitHub changes and Google Docs decisions, covering heterogeneous-source search, abstention, goal-directed summarization, and circumstance inference across long conversations. Across 33 frontier models and agent harnesses, the strongest system reaches roughly 80%, while most remain below 66%. The results show that r
SlackBench moves collaborative-agent evaluation beyond static corpora
SlackBench evaluates agents in a dynamic environment reconstructed from collaborative research and engineering projects. Slack-style conversations, GitHub repository states, code changes, and Google Docs decisions are linked so that an agent must determine where relevant evidence lives, reconstruct its chronology, and reconcile it with the current state of the code.
The benchmark contains 140 queries covering heterogeneous-source agentic search, abstention on unanswerable or false-premise questions, goal-directed summarization, and circumstance inference over long conversations. These tasks cannot be solved by locating a single passage: agents must select relevant channels, traverse threads, inspect repository artifacts, and synthesize evidence across communication and code.
What the benchmark actually measures
A representative workflow may begin with identifying the relevant project channel, followed by locating a decisive discussion inside a long thread. The agent then checks the corresponding repository branch, commits, or tests, determines whether later documents superseded an earlier decision, and produces an answer whose scope is tied to the relevant point in time.
SlackBench therefore evaluates several connected capabilities:
- Relevance detection: separating current constraints from idle discussion and obsolete decisions;
- Context reconstruction: rebuilding event sequences from channels, direct messages, and threaded replies;
- Repository verification: checking files and changes rather than trusting informal chat claims;
- Conflict handling: recognizing stale, contradictory, or false-premise evidence;
- Constrained generation: abstaining when evidence is insufficient and qualifying conclusions by time and scope.
This is closer to the behavior required from enterprise knowledge assistants and engineering copilots than conventional static RAG evaluation. The answer is often not supported by one isolated passage but by multiple artifacts with different timestamps, authority levels, and permission boundaries.
The dataset is useful but still narrower than autonomous work
The benchmark's strongest feature is its reproducible integration of communication, code changes, and decision documents. However, it remains an information-seeking and synthesis benchmark rather than a complete test of autonomous execution.
A production engineering agent may need to edit code, run tests, interpret failures, revise its plan, and coordinate multiple roles. SlackBench evaluates important precursor capabilities, but cannot independently establish end-to-end planning, tool reliability, or error recovery.
The 140-query set may also be modest for fine-grained error analysis. Abstention and circumstance-inference tasks can be sensitive to annotation granularity, acceptable-answer ranges, and prompt design. Comparisons are also meaningful only when systems use comparable retrieval budgets, tool permissions, context windows, and answer post-processing.
Approximately 80% best performance reveals a structural gap
Across 33 frontier models and agent harnesses, the best system scores around 80%, while most score below 66%. The distribution points to persistent failures in several areas:
- recognizing whether a message still represents the project's current state;
- mapping decisions in discussions to concrete code changes;
- avoiding premature conclusions when evidence is stale or contradictory;
- abstaining reliably on unanswerable or false-premise questions;
- distinguishing facts, proposals, hypotheses, and finalized decisions in long threads.
Aggregate scores can conceal these differences. One system may perform well on conventional retrieval while exhibiting excessive compliance with false premises; another may summarize less fluently but remain better calibrated under uncertainty. Production selection should therefore inspect task-level results and failure categories rather than rely only on a single leaderboard metric.
Engineering implications: build evidence chains, not larger prompt templates
For teams building agents over Slack, Teams, GitHub, or enterprise documents, SlackBench suggests a concrete design direction.
Preserve source and temporal metadata. Every message, commit, and document section should carry authorship, timestamp, project, status, and reference information. Without these fields, a model may retrieve the right text but still mistake an obsolete statement for a current fact.
Separate retrieval from state verification. A retrieval stage can recall candidate threads, files, and commits, while a verification stage checks current branches, later commits, and decision documents. Concatenating everything directly into a vector-backed context conflates historical claims with present reality.
Maintain an evidence ledger. Each important claim should link to its originating message, code location, and validity window. Conflicting evidence should remain explicit rather than being flattened by the generator.
Treat abstention as a first-class requirement. Separate metrics should cover missed evidence, false refusals, stale-evidence use, and undisclosed conflicts in unanswerable or false-premise cases.
Control tool budgets. Agents should begin with targeted thread discovery and expand to repositories and documents only when needed. indiscriminately ingesting the entire workspace increases token cost and may amplify irrelevant context.
Benchmark value and limitations
SlackBench matters because it expands agent evaluation from isolated answers to traceable work states. It is useful for enterprise assistants, developer agents, and long-horizon agent teams seeking to measure cross-modal evidence discovery, temporal context reconstruction, and calibrated answers under uncertainty.
It should not, however, be treated as a direct proxy for production SLAs. Privacy-preserving reenactments may simplify real organizational permissions, noise, temporary work, and interpersonal dependencies. Model and harness results are also affected by retrievers, tool interfaces, token budgets, and prompt templates. Extending the query set, publishing fine-grained failure labels, and adding code modification, test execution, and recovery tasks would move the benchmark closer to end-to-end engineering-agent evaluation.
Conclusion
SlackBench shows that the main obstacle to genuine digital coworkers is not answer generation but reconstructing trustworthy context from dynamic, multi-source, and sometimes contradictory work environments. A best score near 80% demonstrates meaningful baseline capability, while most systems below 66% show that retrieval augmentation, long context, and tool use must be assembled into a workflow with temporal semantics, evidence tracking, and explicit abstention.
— END —Source: ArXiv AI (cs.AI) (2026-11-05)
Tags: #SlackBench #Collaborative Agents #Agent Evaluation #Heterogeneous-Source Retrieval #Long-Horizon Reasoning
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments