Decoder-Only LLM NLI Benchmark: Example Quality Matters More Than Count

Summary:A benchmark study evaluates decoder-only LLMs—including Llama, Mistral, Gemma, and Phi 4—on MNLI, QNLI, and SciTail under zero-shot and few-shot prompting. The results suggest that straightforward prompts can match or outperform more elaborate strategies such as ICS and MCQ-based prompting. The label attached to a single demonstration also has a substantial effect, with entailment-primed examples often performing best. Overall, selecting a few representative examples may be more effective than i

---

Core Findings

This study systematically compares prompting strategies for decoder-only large language models on natural language inference (NLI). It evaluates Llama, Mistral, Gemma, and Phi 4 across MNLI, QNLI, and SciTail. The central result is that simple, direct prompts can match or outperform more elaborate approaches such as ICS and MCQ-based prompting. Improvements in NLI therefore do not necessarily require complex sampling, chained instructions, or specially constructed answer choices. Clear instructions and representative demonstrations may matter more than prompt sophistication.

The study also shows that the number of demonstrations is not a monotonic source of gains. A small set of representative examples can outperform a larger collection of low-quality examples. The label attached to a one-shot demonstration significantly affects most models, with entailment-primed examples generally performing best. This suggests that the model may learn a decision prior from the demonstration label, not just an input format.

Prompting Mechanism

NLI requires a model to classify a premise–hypothesis pair as entailment, contradiction, or neutral. Few-shot demonstrations communicate the task format, label semantics, and decision boundaries. Their influence is not limited to semantic content: label balance, example ordering, and output conventions can all alter the model's behavior.

When a single demonstration carries an entailment label, the model may become more sensitive to logically supported conclusions while changing the balance of contradiction and neutral predictions. Demonstration selection should therefore include explicit checks for label balance, boundary coverage, and output ordering rather than relying only on semantic representativeness.

ICS and MCQ-style prompting represent two common attempts to increase complexity. ICS may add richer or retrieved context, while MCQ prompting reformulates inference as option selection. Both can impose retrieval, context-assembly, or answer-mapping overhead. If the gains are not consistent, that overhead translates directly into token cost, latency, and maintenance complexity.

Engineering Trade-offs

The basic cost of few-shot prompting can be approximated as:

[ C_{prompt}=C_{fixed}+n\cdot(C_{demo}+C_{output}) ]

Here, n is the number of demonstrations, C_demo includes each example and its label, and C_output represents generation cost. When the prompt prefix is cached using a KV Cache, the latency is closer to:

[ T(n)\approx T_{prefill}+T_{decode}(n) ]

Adding demonstrations can therefore increase KV Cache occupancy and time-to-first-token latency, not merely the number of visible input tokens. The results support using fewer but better examples, but they do not imply that a small fixed prompt will work reliably for every model and task.

Example effectiveness is also dataset- and model-dependent. Semantically typical examples may still omit difficult premises, long inputs, or boundary cases. A production workflow should validate combinations selected from a fixed candidate pool by class, length, and difficulty, then use offline evaluation to select the prompt rather than relying on intuition alone.

How to Interpret the Benchmark

Covering several models and three datasets reduces the risk that the result comes from a single model anomaly. However, the available abstract does not report exact accuracy values, confidence intervals, random seeds, decoding parameters, or statistical significance. If the gaps between prompting methods are small, rank changes under a few runs may not be meaningful.

MNLI, QNLI, and SciTail also differ in task construction and domain. MNLI focuses on general semantic inference, QNLI resembles question-answer entailment, and SciTail uses scientific text. The evidence should not be generalized to multi-hop reasoning, long-context tasks, structured label mapping, or safety-sensitive applications without additional evaluation.

Deployment Guidance

1. Establish a simple baseline first: Use explicit instructions, stable label names, and structured outputs for zero-shot and one-shot comparisons.
2. Prioritize example quality: Include examples from each class and avoid allowing one label to dominate the prompt.
3. Scale context selectively: Begin with a small representative set and add demonstrations only when hard slices show a consistent gap.
4. Control comparison variables: Keep model version, quantization, inference parameters, output parser, and prompt template fixed.
5. Build targeted regression sets: Cover label boundaries, long premises, negation, domain shift, and adversarial examples; report means and variance across runs.
6. Measure total cost: Track input and output tokens, time to first token, throughput, and KV Cache usage alongside accuracy.

Limitations

This is an empirical prompting benchmark rather than a new model architecture or training method. Its practical contribution is evidence that complex prompts are not always necessary and that demonstration quality, label balance, and evaluation configuration deserve priority. Because the available summary omits detailed metrics and ablations, the conclusion should be treated as a reproducible empirical tendency rather than a universal law.

---

Source: ArXiv NLP/LLM (cs.CL) (2026-11-10)

— END —

Tags: #Natural Language Inference Follow #In-Context Learning Follow #Prompt Engineering Follow #LLM Benchmarking Follow #Llama Follow