Amazon Bedrock Adds Explicit Prompt Caching for Open-Weight Models
Summary:AWS has extended explicit prompt caching to supported open-weight models on Amazon Bedrock, complementing the existing implicit mode. Through the Chat Completions API, teams can mark reusable prefixes, while Bedrock manages the cache state and exposes cache-read and cache-write token counts. The update targets workloads that repeatedly resend long system prompts, tool schemas, retrieved documents, or agent histories. The engineering payoff is lower prefill work and potentially lower input charg
Core update
AWS has extended explicit prompt caching to supported open-weight models through the Chat Completions API on Amazon Bedrock. The existing implicit mode remains available, and eligible models can support either approach or both. This is not a new model launch; it is an inference-infrastructure update for workloads that repeatedly transmit the same long prefixes, including multi-turn conversations, RAG systems, and coding agents.
Support is not universal. The applicable model, minimum prefix length, maximum number of breakpoints, TTL, API surface, and Regions must be checked in the model card. AWS also notes that cross-Region inference may produce additional cache writes during periods of high demand, so hit rates cannot be inferred from request structure alone.
Technical mechanism: the cache stores prefix state, not answers
Prompt caching is different from a conventional response cache. It does not retain a final answer for reuse. Instead, when a later request contains an identical prefix, the service reuses intermediate state already computed for that prefix. In mainstream autoregressive Transformers, such state commonly corresponds to per-layer Key and Value tensors, or KV Cache. AWS does not disclose Bedrock's exact cache format, hardware placement, or scheduling internals, so this should not be treated as a user-exportable, generic KV-cache file.
A hit requires the prefix to remain identical after tokenization. A single changed character, timestamp, or request ID near the beginning can invalidate reuse from that point onward. This is why static instructions, tool definitions, and reference documents should precede dynamic content.
Let P be the cached prefix length, M the new tail length, d the hidden dimension, and L the number of layers. An uncached dense-Transformer prefill is roughly O(L[(P+M)d² + (P+M)²d]). With a cache hit, QKV projections and MLP work for the prefix can be skipped, but queries for the new tokens must still attend to all P+M positions. The remaining attention cost is therefore approximately O(LM(P+M)d), plus cache-state access and tail computation. Prompt caching removes repeated prefix computation; it does not remove the model's need to attend to the full context.
KV state can itself be substantial. For a GQA model, the approximate bytes per sequence are 2 × L × H_kv × d_head × S × b, where H_kv is the number of KV heads, S is sequence length, and b is the byte width per element. With 32 layers, 8 KV heads, a head dimension of 128, and FP16, this is about 128 KiB per token; a 128K-token context would consume close to 16 GiB for KV state alone. Bedrock manages this allocation, but capacity, routing, and cross-Region replication can still affect write behavior and hit rates.
Choosing between implicit and explicit caching
| Dimension | Implicit caching | Explicit caching |
| --- | --- | --- |
| Request changes | No cache parameters required | Set prompt_cache_options.mode=explicit and add prompt_cache_breakpoint markers |
| Cached scope | Bedrock automatically selects eligible prefixes on a best-effort basis | Everything up to and including each breakpoint |
| Best fit | Conversations and agent loops with naturally stable prefixes | A shared document plus a changing question, or layered prefixes requiring exact control |
| Main risk | Changing tail content may be written even if it is never reused | The mode and breakpoint must both be configured correctly |
Implicit mode automatically identifies cacheable prefixes. In AWS's support-assistant example, the system prompt and product manual remain unchanged while the customer question changes. The first request writes to the cache; later requests can read the system prompt but may also write a small number of new question tokens. If an early conversation turn changes, the cache must be rebuilt from that point.
Explicit mode allows multiple cache layers. For example, global instructions and tool definitions can be cached first, followed by a customer-specific document set. Each request must include prompt_cache_options and place prompt_cache_breakpoint on the relevant text content part. Explicit mode without a breakpoint caches nothing; a breakpoint without explicit mode behaves like implicit caching. Supported optional ttl values are model-dependent, and every cache hit refreshes the TTL.
Engineering trade-offs: cache reads are cheap, but writes may not be
Cache reads are normally billed below the standard input rate, while writes can cost more than ordinary input for some models. Let S be the standard input price, w the write multiplier, and r the read multiplier. If one stable prefix incurs one write followed by N−1 hits, its simplified cost is PS[w + (N−1)r]; without caching, the cost is NPS. When r < 1 and writes carry a premium, the idealized break-even condition is N > (w-r)/(1-r). Real deployments must also account for TTL expiration, routing changes, and repeated misses.
This explains the practical value of explicit caching. If the content after a shared prefix changes on every request, a breakpoint prevents that tail from being written into an expensive cache. It also shows why enabling caching does not automatically reduce spend. Low reuse, short TTLs, or high write multipliers can make the feature more expensive than uncached input.
Latency gains are workload-dependent. Caching removes much of the repeated prefill work and can improve time to first token for long contexts. Cache lookup, state transfer, queueing, tail attention, and output decoding remain. For short prompts, low concurrency, or decode-bound workloads, end-to-end improvement may be small and can be offset by cache-write overhead.
Prompt caching also does not improve reasoning quality, extend the context window, or eliminate hallucinations. It reuses computation for the same prefix while the model still generates new tokens conditioned on the complete context. Production evaluations should therefore track cost reduction together with task success, factual accuracy, and structured-output validation.
Observed behavior and evaluation limits
AWS demonstrates a support assistant with an approximately 4,000-token water-filter manual. The first implicit call writes to the cache; subsequent calls read the system prompt and write a small number of new question tokens. A cold explicit run writes once, after which repeated requests read the entire prefix up to the breakpoint without writing. If the sample is rerun within the TTL, later implicit calls can also become read-only.
The OpenCode example shows a common coding-agent sequence. One agent turn may first send the system prompt, tool definitions, and user task. After the model returns a tool call, the agent sends an expanded request containing the previous request plus the tool call and execution result. Bedrock can read the earlier prefix and process only the new tail. This flow can use implicit caching without additional cache configuration.
These examples validate cache semantics, not performance. The official charts are explicitly illustrative and do not provide a fixed model, input length, concurrency level, hit rate, TTFT distribution, or cost reduction that can be generalized to other workloads. OpenCode 1.18.35 also does not record cache_write_tokens through its OpenAI-compatible provider. A cache.write=0 value in exported session data therefore reflects missing client telemetry, not proof that the backend performed no writes.
Developer deployment guidance
- Check the model card and Region first. Confirm whether the target open-weight model supports implicit caching, explicit caching, or both. Review the minimum prefix length, maximum breakpoints, TTL, read/write prices, Regional availability, and cross-Region inference behavior.
- Separate stable and dynamic prompt layers. Put system instructions, tool schemas, retrieved documents, and stable tenant data first. Put timestamps, request IDs, randomized questions, and user input last. A small change near the front of the prefix expands the invalidation boundary.
- Choose a mode based on the workflow. Implicit mode suits conversations and agent loops. Explicit breakpoints suit RAG, shared manuals, and layered tenant contexts. In explicit mode, configure both the request option and a content breakpoint.
- Read per-request usage. Use
usage.prompt_tokens_details.cached_tokensandcache_write_tokensto separate cached, written, and uncached tokens. Track aggregate hit rate and write amplification withCacheReadInputTokenCountandCacheWriteInputTokenCount. - Measure both performance and quality. Record cold and warm TTFT, end-to-end P50/P95 latency, cache-recovery time, cost per successful task, and output-quality regressions in addition to input cost. Add a unique run ID when a cold-cache test must avoid residual entries.
- Enforce credential and data boundaries. Use IAM permissions and short-lived Bedrock API keys valid for up to 12 hours rather than persisting long-term credentials. AWS states that prompts and outputs in Bedrock are not shared with model providers or used for model training, but each open-weight model's license, AWS service terms, and data-protection conditions still require review.
Conclusion
The significance of this update is not another model launch. It brings explicit prefix control into Amazon Bedrock's open-weight-model inference path, allowing agent workloads to avoid repeated prefill work while limiting cache writes more precisely. The actual return still depends on token-level prefix stability, cache lifetime, model pricing, and traffic patterns. Production deployment should treat prompt layout, cache mode, request accounting, and end-to-end regression testing as one inference-optimization system rather than a single feature toggle.
— END —Source: AWS Machine Learning Blog (2026-11-16)
Tags: #Amazon Bedrock #Prompt Caching #Open-Weight Models #Inference Cost Optimization #OpenCode
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments