Prompts Engineering

NInfer vs llama.cpp vs vLLM: Quality and Speed Comparison

Prompt Edit here before copy/try — source content is unchanged

### Hardware Configuration
- RTX 5090 32GB (eGPU, OCuLink Gen4 x4)
- Ryzen 7 7840HS, 32 GB DDR5
- Ubuntu 26.04, nvidia driver 610.43.02 (open)

### Engine Configuration
| Engine | Quant | KV cache | Context | MTP | Concurrency | VRAM |
|------|-------|----------|---------|-----|-------------|------|
| llama.cpp | Q5_K_M GGUF | q8_0 | 196K | On (gate failed) | parallel=1 | 31.6 GB |
| vLLM | NVFP4 | FP8 | 262K | None | Continuous batch | 29.6 GB |
| NInfer | NVFP4 | FP8 | 240K | MTP3 (76% acceptance) | x2 lanes | 30.5 GB |

### Evaluation Method
- **Relevance Classification**: Determine if the content is industry-relevant (50 labeled items).
- **Needle Retrieval**: Implanted facts in real industry podcast/video transcripts at varying context lengths (64K/128K/192K/240K).
- **Multi-transcript QA**: Questions across 3-4 concatenated diarized transcripts, including unanswerable controls.
- **Reasoning with Thinking**: Numeric/logic problems with thinking mode on.
- **Structured Extraction**: Custom extraction prompt using json_mode (only tested on llama.cpp and vLLM).
- **Tool Replay**: Replayed recorded agent episodes (diagnostic only, all engines failed).

### Quality Results
| Metric | llama.cpp | vLLM | NInfer |
|------|----------|------|--------|
| Relevance | 86.0% | 84.0% | 86.0% |
| Needle (conditional) | 100% (29/29) | 100% (41/41) | 100% (41/41) |
| Transcript QA | 82.0% | 78.0% | 88.0% |
| Reasoning | 100% | 100% | 98.0% |
| Extraction | F1 0.300 | F1 0.350 | skipped |
| Tool Replay | 0% | all errors | 0% |

### Speed Results
| Metric | llama.cpp | NInfer | Speedup |
|------|----------|--------|---------|
| Decode 1K | 114 tok/s | 158 tok/s | 1.4x |
| Decode 32K | 109 tok/s | 213 tok/s | 2.0x |
| Decode 128K | 72 tok/s | 202 tok/s | 2.8x |
| Prefill 1K | 1,545 tok/s | 7,265 tok/s | 4.7x |
| Prefill 32K | 2,155 tok/s | 6,892 tok/s | 3.2x |
| Prefill 128K | 1,528 tok/s | 3,904 tok/s | 2.6x |
| TTFT 1K | 670 ms | 138 ms | 4.9x |
| TTFT 32K | 15.2 s | 4.8 s | 3.2x |
| TTFT 128K | 85.9 s | 33.6 s | 2.6x |

### Conclusion
NInfer NVFP4 replaces llama.cpp as my production engine, offering both quality and speed improvements.

Uses the text above; some sites may auto-send

Usage & requirements

Choose the appropriate engine based on hardware configuration and requirements, and refer to the comparison results for optimization

Tools: llama.cpp vLLM NInfer
Variables: hardware engine evaluation
Source: r/LocalLLaMA ↗

More Engineering prompts

Engineering

AI Agent Workflow Prompt Template

Fill in the target task in [GOAL] and follow the rules and checkpoints to design AI Agent workflow

Open & try

Engineering

Structured Prompt Template for AI Presentation Generator

Fill in the audience, main idea per slide, word limit, and content arc, then send to the AI tool for slide generation.

Open & try

Engineering

AI Community Promotion Rules Guide

Input the above points into the AI assistant to generate a detailed Reddit community promotion guide

Open & try

Engineering

Multi-stage Rocket Launch Simulation Coding Task

Paste code and review goal, then send

Open & try

Engineering

AI Assistant Dispatcher System Template

Paste this as a system message at the top of a new chat, then send any request. The AI assistant will route first, th…

Open & try

Engineering

Qwen 3.8 27B Model Performance Test Report

Use the test descriptions as input instructions to evaluate model performance

Open & try