ZICQ
中 Log in / Sign up
Newsroom Research & Papers #PoolBech #Pooling Policy #Concept expression #LLMs & Foundation Models #Benchmark testing

PoolBench: A Benchmark for Pooling Strategies in Concept Representation Evaluation for Decoder-Only LLMs

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published: · 11 views

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:A new arXiv study introduces PoolBench, a benchmark designed to systematically evaluate pooling strategies in concept representation for decoder-only LLMs. Under a fixed evaluation protocol, it covers 17 concepts, 19 pooling strategies, and 3 open-weight models (Llama-3.1-8B, Gemma-2-9B, Mistral-7B), evaluated on 37,693 real-text passages. The key finding is that W4_hierarchical achieves a cross-model mean AUROC of 0.7799, significantly outperforming the widely used P1_last_token baseline (0.764


Overview

In decoder-only large language model (LLM) concept representation research, pooling is a critical yet often overlooked design choice. Researchers must collapse token-level hidden states into passage-level vectors, but the lack of a standardized comparison protocol makes results across studies difficult to compare. To address this, a new arXiv study introduces PoolBench, a benchmark specifically designed to evaluate pooling strategies, aiming to isolate the pooling variable and provide a standardized evaluation framework.

Core Content

Benchmark Design

PoolBench covers, under a fixed evaluation protocol:

  • 17 concepts: spanning various semantic categories.
  • 19 pooling strategies: including common ones like P1_last_token and W4_hierarchical.
  • 3 open-weight models: Llama-3.1-8B, Gemma-2-9B, and Mistral-7B.
  • Evaluation corpus: 37,693 audited real-text passages.

Evaluation is based on three axes:

  • D1 (Linear Separability): measures concept detection via AUROC.
  • D2 (Steered Concept Prevalence): assesses the impact of pooling on concept steering.
  • D3 (Output-level Disentanglement): evaluates separation at the output level.

Key Findings

  1. W4_hierarchical significantly outperforms baseline: It achieves a cross-model mean AUROC of 0.7799, compared to 0.7640 for the widely used P1_last_token, with statistical significance (Friedman+Nemenyi test, p=2.0e-36).
  2. Stable rankings across layers: Rankings are highly consistent across layers (rho=0.961--0.990), indicating that pooling strategy choice is layer-insensitive.
  3. Negative result: Strong detection does not imply strong steering; D2 and D3 are generally weaker than D1, suggesting a fundamental representational limit rather than a pooling failure.
  4. Construction method has larger impact: On mid-difficulty concepts, W4_hierarchical outperforms P1_last_token by 0.042--0.113 AUROC, but the construction method (e.g., DiffMean vs. REPE) has a larger effect (delta AUROC 0.15) than pooling itself (delta AUROC 0.016).

Technical Highlights

  • Isolation of experimental variables: PoolBench isolates pooling by fixing other factors, providing a clear basis for comparison.
  • Open resources: The team releases the corpus, pre-extracted activations, scorer models, steering vectors, and evaluation code for reproducibility and extension.

Industry Impact and Developer Advice

  • For researchers: PoolBench provides empirical evidence for pooling strategy selection, suggesting prioritizing W4_hierarchical and paying attention to construction methods.
  • For developers: When building LLM-based representation systems, consider the impact of pooling on downstream tasks and leverage open resources for rapid validation.

Conclusion

PoolBench fills the gap in pooling strategy evaluation, offering a standardized tool for concept representation research in decoder-only LLMs. Its findings not only reveal optimal pooling strategies but also highlight representational limitations, guiding future research directions.


Source: arXiv:2608.05162


Source: ArXiv NLP/LLM (cs.CL) (2026-08-07)

— END —

Tags: #PoolBech #Pooling Policy #Concept expression #LLMs & Foundation Models #Benchmark testing

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…