News
· 2026-11-17
Amazon SageMaker AI has introduced `KV_AWARE` routing for real-time inference endpoints. Supported model containers publish KV-cache insertion and eviction events, while a managed index stores only truncated one-way hashes and instance pointers. For each request, SageMaker tokenizes the prompt, locates the instances holding the longest cached token prefix, and balances cache reuse against current instance load.
The feature targets workloads where fixed-prefix routing becomes unreliable, includi
· 2026-11-14
NVIDIA demonstrates serving GLM-5.3 with vLLM across two NVIDIA DGX Stations, reporting throughput of 607.8 tokens per second. The deployment illustrates how tensor parallelism, continuous batching, and efficient KV-cache management can scale large-model serving beyond a single workstation. However, the supplied excerpt does not disclose prompt and output lengths, concurrency, time-to-first-token, quantization, or the full hardware configuration, so the figure should be treated as a workload-spe
· 2026-11-14
The ggml-org project has released ggml v1.0.0, marking a stable release of its open-source C/C++ tensor and execution foundation for local large-language-model inference. The milestone provides a more dependable compatibility baseline for quantized model loading, CPU/GPU hybrid execution, hardware backend integration, and model conversion. This can reduce production integration and upgrade costs for developers maintaining local inference stacks. The collected release note does not expose the ful
News
· 2026-11-13
ggml-org has released llama.cpp v1.0.0, promoting the widely used open-source C/C++ inference framework to a stable 1.0 milestone. The release is significant primarily as an engineering baseline: it is intended to provide more predictable APIs, model-format handling, quantization workflows, and CPU/GPU backend compatibility for production deployments.
For the local AI ecosystem, reaching 1.0 can reduce upgrade friction and improve integration confidence, particularly for applications that combi
News
· 2026-11-11
Cloudflare’s quantization study separates weight-format effects from the serving stack itself. On an NVIDIA GB10, the authors replay the same 2,400 public decisions through Clef-flash (9B) and Clef (27B), measuring output flips caused by engine swaps, restarts, batch-size changes, and an automatically selected Gated DeltaNet kernel. Non-quantization perturbations move 0–0.92% of decisions; within one engine, Q4_K_M and NVFP4 move 3.50–5.79%, with the reported 4-bit intervals separating them from
· 2026-11-11
Yogurt is an MIT-licensed, cross-platform desktop meeting assistant developed by Eric at Frigade. It records meetings in the background, transcribes audio with whisper.cpp or Apple's SpeechAnalyzer, and uses llama.cpp to run an open-weight LLM selected according to the user's hardware for summaries and note-based questions. Recordings, transcripts, and summaries remain on the local machine. The app also includes calendar integration, scoped search, automatic recording termination, and a local MC
· 2026-11-11
The llama.cpp community has introduced support for an FP8 KV cache, targeting the memory cost of autoregressive inference, which grows with context length and concurrency. By storing key and value states in 8-bit floating point, the feature could reduce memory pressure for long-context and multi-user workloads. The practical benefit will depend on efficient hardware support for FP8 conversion and on how quantization affects output quality, latency, and cache management. The collected information
News
· 2026-11-10
The paper proposes precision-on-demand decoding for a small local model. Weights are stored at 8 bits, while the fast decoding path reads only the top 4 bits. A step is recomputed at full 8-bit precision when the low-precision output distribution appears uncertain, based on entropy or a small top-two probability margin.
On Qwen3-0.6B with simulated quantization and GSM8K, the entropy signal predicts argmax flips with an AUC of 0.93 across 36,724 replayed steps. At a 15% escalation budget, it ca
News
· 2026-11-10
A research effort retrofits million-token context onto Mistral-7B and Llama-3.1-8B without retraining or modifying their weights. It exploits the multi-frequency geometry of rotary position embeddings: the true-position near window is retained, while distant keys and values are rewritten into trained RoPE ranges using a word-based index. A rare token generated during inference triggers one exact re-encode to fetch the associated far-context units.
The llama.cpp implementation achieves 99–100% r
News
· 2026-11-10
LoRazor is a multi-LoRA serving framework designed for shared-backbone deployments. It allows the runtime to permanently release a request's LoRA adapter once the adapter is no longer necessary and continues decoding with the backbone alone. Active-row aware GPU kernels dynamically resize LoRA launches as adapters are released, allowing idle GPU blocks and associated compute to be trimmed.
Across two backbones and four tasks, the paper reports up to 3.02x higher request throughput and 2.53x hig
News
· 2026-11-10
A study of overloaded vLLM deployments finds that client- and proxy-side retries can keep an inference service unstable even after incoming load falls below capacity. Propagating cancellation did not restore service in any experiment, and only 12–38% of calls arriving after the overload were completed. Scheduler traces show that retried executions typically began prefill less than half a second before timing out and consumed 79–85% of the prompt tokens later wasted by cancellation; vLLM's prompt
News
· 2026-11-10
ChainFlow introduces a speculative-decoding drafter conditioned on the target model's hidden states. Rather than generating tokens autoregressively or denoising them from noise, it treats drafting as a short transport process in latent space and requires only one or two flow-matching steps. A variational autoencoder compresses target hidden states, lightweight feedback heads restore inter-token dependencies, and tree drafting improves acceptance rates. The authors report gains of 39% over EAGLE-
News
· 2026-11-09
A controlled study of vLLM under overload reveals that client-side retries can keep an LLM engine saturated even after incoming traffic falls below capacity. With retries enabled, cancellation propagation alone failed to restore service in every run, and only 12–38% of post-overload calls were delivered. Scheduler traces show that retried executions usually entered prefill less than half a second before timing out, while aborted executions had already consumed 79–85% of the prompt tokens compute
News
· 2026-11-09
ChainFlow introduces a speculative-decoding drafter conditioned on the target model's hidden states. Rather than generating drafts autoregressively or denoising tokens from noise, it treats drafting as a short transport process that can complete in one or two flow steps. A variational autoencoder compresses hidden states, lightweight feedback heads restore inter-token dependencies, and tree drafting improves acceptance. The authors report gains of 39% and 11% over EAGLE-3 and DFlash in vLLM on Q
News
· 2026-11-06
BlockLock addresses batch-dependent inference by fixing the numerical path observed by a protected request while allowing other requests to vary. It offers two mechanisms: BlockLock-Rows preserves batched decoding with fixed-width rows, while BlockLock-Pack uses terminal-anchored packing and isolates the target's KV state for target-only decoding. Across five dense models on NVIDIA A6000, A100, and H100 deployments, the reported experiments preserve target sequences and Top-20 logit bits. At equ
News
· 2026-11-03
arXiv introduces a novel post-training quantization (PTQ) method for large models, which decomposes the weights into a quantized matrix, a low-rank component, and a sparse matrix to compensate for quantization errors. By employing simulated annealing for decomposition optimization, the method demonstrates significant performance improvements across various large language models (e.g., Llama 3.2 and Qwen2.5 7B) using 4-bit NF4 and FP4 quantizers. This approach markedly outperforms direct quantiza
News
· 2026-10-30
Flide targets tree-structured queries in decision models and LLM agents. It decouples logical dependency trees from a flattened token stream and uses GPU-resident indices to implement ForkAttention across standard attention and recurrent Gated DeltaNet layers. In exact arithmetic, each branch is equivalent to an isolated forward pass, while checkpoints allow expanding agent histories to resume efficiently.
On public hybrid and decision models, the paper reports up to 9.5× faster query execution
· 2026-10-30
The vLLM project has introduced PLE (Persistent Lazy Execution) disk offload support in its latest nightly release, aiming to optimize large model inference performance. This feature offloads certain computational tasks to disk storage, reducing GPU memory usage and enabling the deployment of larger models with improved efficiency. The update is particularly valuable for AI applications in resource-constrained environments and highlights vLLM's ongoing innovation in model optimization.
News
· 2026-10-28
Hugging Face introduces WISD (Weakening-Induced On-Policy Self-Distillation), a novel self-distillation framework that constructs a weaker counterpart of the original model and leverages the contrast between them for training without requiring privileged information. Experiments demonstrate that WISD significantly improves reasoning performance, with quantization and pruning boosting average accuracy by 4.0 and 5.5 percentage points, respectively, across diverse benchmarks.
News
· 2026-10-28
ResCom is a novel low-rank correction method for large language model quantization that compensates for layer-output errors using activation rounding residuals while maintaining low-bit weight and activation quantization. Experiments on the Llama-3-8B model with 4-bit weights and activations show that ResCom significantly improves performance on benchmarks like WikiText-2 perplexity, MMLU, and GSM8K. The method also demonstrates scalability across models ranging from 1B to 70B parameters and eff
· 2026-10-26
Hob, an open-source desktop application by FrigadeHQ, offers a privacy-focused alternative to ChatGPT and Claude by running on users' own devices. It leverages open-weight models like Gemma 4 and gpt-oss, ensuring that all interactions and files remain local, with internet access limited to web searches and user-configured MCP connectors. Hob supports file operations, image recognition, code execution, and offline functionality, addressing the limitations of existing on-device AI applications in
News
· 2026-10-24
Ollama has announced support for speech models, starting with Phonon from Gradium. The model offers text-to-speech capabilities in five languages with dozens of voice options and can stream generated speech over 100 times faster than real-time on Apple Silicon devices. This technological advancement provides an efficient solution for multilingual voice interaction applications, demonstrating significant advantages in resource-constrained environments.
News
· 2026-10-19
Cloudflare's 27B-parameter multimodal decision model, Clef, is now available on Ollama's platform, allowing users to run it locally or leverage Ollama's cloud services for deployment. Clef supports processing multimodal data, including text and images, aiming to deliver efficient solutions for complex decision-making tasks. This release signifies the growing adoption of multimodal AI models in cloud environments, offering developers greater flexibility in application deployment.
· 2026-10-19
Cohere has announced full support for llama.cpp across its latest model architectures, including the vision encoder of North Micro Vision Instruct. This move enhances the compatibility and scalability of Cohere models for on-device deployment and within the open-source community. The support for llama.cpp not only improves the efficiency of model execution in resource-constrained environments but also provides developers with more flexible tool options. Additionally, Cohere Transcribe is now ava
News
· 2026-10-19
vLLM's open-source inference engine introduces bi-directional KV cache transfer, allowing prefill and decode nodes to reuse previously computed context across conversation turns, significantly reducing Time-To-First-Token (TTFT). Benchmarks on Amazon EC2 P5en instances show a 1.6x reduction in TTFT for Qwen3-32B and a 3.3x reduction for GLM-5-FP8 in multi-turn agentic workloads. This mechanism enhances accelerator utilization and reduces redundant computations, improving inference efficiency for
News
· 2026-10-19
vLLM, an open-source inference engine for large language models (LLMs), introduces a push-based KV transfer mechanism that significantly enhances inference performance. Benchmarks on Amazon EC2 P5en instances demonstrate up to a 3x improvement in time-to-first-token (TTFT) and a 30% reduction in time-per-output-token (TPOT) compared to the previous pull-based approach. This mechanism optimizes KV cache transfer by moving coordination off the critical path, reducing latency and increasing through
· 2026-10-13
AnythingLLM has announced its migration from Ollama to the Lemonade embedded SDK and llama.cpp. This shift aims to leverage the local deployment capabilities of the Lemonade SDK to enhance the efficiency and data privacy of AI agents, while utilizing the lightweight nature of llama.cpp to optimize resource usage. Post-migration, developers will be able to deploy AI agents more flexibly in local environments, reducing reliance on cloud servers.
News
· 2026-10-13
Ollama has partnered with Fly.io to integrate Ollama's cloud models into Fly.io's cloud computers, enabling AI agents to operate securely within a protected environment. This collaboration aims to enhance the safety and efficiency of AI agent execution, providing developers with a more reliable and efficient deployment solution for AI agents.
News
· 2026-10-13
AWS engineers have implemented kernel optimizations for fused Mixture-of-Experts (MoE) models on Amazon SageMaker AI, including Swap-AB matrix transposition for WGMMA utilization, Triton backend routing for FP8 quantization, and the elimination of Triton JIT recompilation. These optimizations result in up to a 10% increase in throughput and a 9% reduction in Time to First Token (TTFT) on H200 GPUs. This advancement highlights AWS's commitment to enhancing AI inference efficiency and provides cri
News
· 2026-10-12
AWS has released a technical guide on deploying decision models with llama.cpp on Amazon SageMaker AI. The solution supports OpenAI-compatible chat completions and can be extended with SemIf to return probabilities over declared options. It offers both CPU and GPU deployment options, with a standard text-generation path using the AWS Deep Learning Container (DLC) for llama.cpp and a structured-decision path utilizing the SemIf adapter. This provides developers with a flexible and efficient AI in