ZICQ
中 Log in / Sign up
Newsroom Research & Papers #Hallucination Detection #On-Device LLMs #NF4 Quantization #Semantic Entropy #Benchmark Audit

Audit Finds Hallucination Detector Rankings Reverse Across 4-Bit On-Device LLMs

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published:

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:This practitioner-facing audit evaluates four hallucination detectors under conditions that closely resemble edge deployments: Llama-3.2-3B-Instruct and Qwen2.5-3B-Instruct loaded in 4-bit NF4 on a single Colab T4 GPU. The compared methods include length-normalized token-margin calibration, a lexical n-gram baseline, semantic entropy, and SelfCheckGPT-NLI. Detector rankings reverse across the two models, with a Spearman ρ of -0.80 and Kendall τ-b of -0.667. The inexpensive lexical baseline is s


Core result

Under 4-bit NF4 quantization, a single T4 GPU, and closed-book TriviaQA, the ranking of hallucination detectors nearly reverses between Llama-3.2-3B-Instruct and Qwen2.5-3B-Instruct. There is no model-independent ordering of detectors: a method that performs best on one 3B model can fall near the bottom on another.

The reported Spearman ρ of -0.80 and Kendall τ-b of -0.667 indicate rank inversion rather than merely a small change in absolute performance. Results and thresholds established on 7B–70B models—or on one small model—should therefore not be transferred directly to another edge deployment.

How the four detector families work

  • Length-normalized token-margin calibration: The score is typically derived from the logit or log-probability gap between the selected token and competing tokens, then normalized by output length to reduce the bias caused by accumulating uncertain tokens in long answers. It usually adds little cost beyond the main generation path, but it is highly sensitive to changes in the model's probability distribution.
  • Lexical n-gram baseline: This method relies on surface overlap, repetition patterns, or answer-fragment matching without repeated sampling. It is inexpensive and easy to deploy, but can confuse lexical similarity with factual agreement, particularly for paraphrases and open-ended answers.
  • Semantic entropy: The model generates multiple candidate answers, semantically equivalent outputs are clustered, and entropy is computed over the resulting cluster distribution, typically as H(C)=-Σ p(c) log p(c). It can distinguish wording variation from genuine uncertainty, but repeated generation and semantic embedding create substantial overhead.
  • SelfCheckGPT-NLI: A natural-language inference model checks whether sampled answers support, contradict, or fail to entail one another. This is more semantically robust than lexical comparison, but introduces another model, additional latency, and another set of thresholds to calibrate.

Reported results

| Observation | Reported value | Engineering implication | |---|---:|---| | Cross-model Spearman rank correlation | -0.80 | Detector rankings materially reverse | | Cross-model Kendall τ-b | -0.667 | Rankings transfer poorly across models | | TruthfulQA-Generation AUROC | 0.46–0.59 | Near-chance performance consistent with labeler failure | | TriviaQA AUROC | 0.69–0.89 | The same pipeline performs substantially better on another dataset | | Lexical baseline versus sampling-based consistency | No significant difference under bootstrap 95% confidence intervals | Added sampling cost may not provide a demonstrable gain |

The lack of a statistically significant difference between the lexical baseline and the expensive consistency methods is operationally important for constrained edge hardware. A cheap baseline can establish the minimum viable detection level before a team pays for repeated generation, embedding, and NLI inference. However, failure to reject a difference is not proof of practical equivalence. A rigorous equivalence claim would require a predeclared margin and a larger evaluation.

The labeler may be the failing component

On TruthfulQA-Generation, all four detectors achieve only 0.46–0.59 AUROC across both models, which is close to random guessing. The paper attributes this to the token-overlap-based mapping from generated answers to multiple-choice correctness labels. The identical detector pipeline reaches 0.69–0.89 AUROC on TriviaQA, showing that the detectors are not uniformly ineffective within the same implementation.

Token overlap can misclassify semantically correct paraphrases, aliases, or answers with a different format as incorrect, while allowing answers that share surface tokens but contain a factual error to pass. Dataset difficulty may also affect AUROC, so the experiment does not prove that labeling is the sole cause. It does, however, establish an important evaluation requirement: answer-label correctness must be validated independently before detector performance is interpreted.

Engineering trade-offs and scope

  1. Cost does not scale linearly: Token-margin and lexical methods can usually reuse the primary generation result. Semantic entropy and SelfCheckGPT-NLI require additional samples and either embedding or NLI computation. Edge evaluations must report AUROC alongside end-to-end latency, peak memory, energy use, and throughput.
  2. 4-bit quantization can disturb calibration: NF4 changes numerical behavior in weights and intermediate activations. Quantized kernels, accumulation precision, and softmax implementation can shift logit margins. Detectors that depend on token probabilities therefore require fresh threshold calibration rather than inherited FP16 or BF16 cutoffs.
  3. The tested surface remains narrow: Although the abstract motivates the work with Phi-3-mini and Gemma-2-2B-Instruct, the experiment actually covers only Llama-3.2-3B-Instruct and Qwen2.5-3B-Instruct. It also tests one NF4 path, one T4 configuration, and a closed-book task; conclusions do not automatically extend to other quantizations, CPUs, mobile NPUs, or retrieval-augmented generation.
  4. Closed-book behavior is not RAG behavior: In a grounded system, hallucinations may arise from retrieval errors, conflicting evidence, or failure to use retrieved context. Sampling consistency measured without retrieval does not establish calibration for evidence-grounded answers.
  5. AUROC is not an operating point: When a detector gates deployment, false positives block correct answers while false negatives admit hallucinations. The paper's error-rate allocation framing is therefore a cost-sensitive decision between two error types, not demographic fairness.

Deployment guidance

  1. Build a model × quantization format × detector × task matrix and evaluate every target model separately instead of inheriting detector rankings.
  2. Start with single-pass token-margin and lexical baselines on 3B-class edge models. Treat sampling-based consistency methods as optional accuracy upgrades rather than default requirements.
  3. Report false-positive rate, false-negative rate, F1, PR-AUC, ECE, and Brier score at a fixed operating point in addition to AUROC, together with latency and memory limits.
  4. Insert human review or an independent semantic grader into the pipeline. Validate whether answers are correct before asking whether a detector can identify incorrect answers.
  5. Recalibrate thresholds after quantization and test changes in temperature, top-p, prompt templates, and domain distributions. Sampling methods also require checking for degeneration caused by an overly low temperature.
  6. If semantic entropy or SelfCheckGPT-NLI is necessary, reuse generation caches, parallelize samples, and test whether smaller embedding or NLI models preserve the detection gain.

Bottom line

The study's main contribution is not a stronger detector, but two warnings for edge deployments: detector rankings can change with the underlying model, and a flawed correctness labeler can make a functioning detector appear close to random. The practical path is to validate labels first, recalibrate a cheap detector on the actual quantized target model, and only then consider sampling-based consistency when its additional accuracy justifies the latency and memory cost.


Source: ArXiv AI (cs.AI) (2026-11-16)

— END —

Tags: #Hallucination Detection #On-Device LLMs #NF4 Quantization #Semantic Entropy #Benchmark Audit

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…