Cutting Token Use by Up to 71% and Latency by 80% for Large JSON Workloads on Bedrock AgentCore
Summary:AWS documents a reusable context-engineering pattern for a Flightscape airline-operations agent running on Amazon Bedrock AgentCore with more than 40 tools and JSON payloads above 20 MB. The design separates live graph state from persisted state, stores heavy artifacts in Amazon S3 under deterministic SHA-256 content addresses, pages oversized lists through a result handle, and replaces large nested dictionaries with compact structural descriptors. The model receives a summary first and retrieve
Core Takeaway
AWS is not introducing a new model here; it is showing a reusable context-engineering pattern for agents. Flightscape, an airline-operations platform, built a Training Schedule Optimization agent with more than 40 tools and JSON inputs above 20 MB. In the original LangGraph design, request, response, and validation-report objects lived directly in agent state, and every tool call fed the full result back into the model context. The optimized turn follows hydrate_state → tool → compact result → optional get_result_page_tool → dehydrate_state: S3 holds the heavy artifacts while the model receives references and navigable summaries.
This turns context from a data plane into a control plane. For a typical dense-attention transformer, a naive prefill attention-score calculation scales as O(L²) for context length L, while KV-cache capacity grows roughly linearly with L. FlashAttention-style kernels reduce constants and avoid materializing the full score matrix, but they do not remove the cost of processing or retaining a much longer prompt. Shrinking the effective context therefore attacks token billing, prefill-stage latency, and session-capacity limits at the same time.
How the Three Strategies Work Together
1. Slim Persisted State: S3 Becomes the Data Plane
The system separates the live GraphState from the persisted PersistedState. The live state retains parsed domain objects required for tool execution. The persisted form stores only serialized messages and S3 references such as request_ref, responses_ref, and validation_report_ref. When a session resumes, lightweight state is restored from AgentCore Memory and heavy objects are fetched and parsed only when required. At the end of a turn, dehydration writes new artifacts back to S3.
References are not random keys. The implementation canonicalizes JSON with sorted keys and compact separators, computes a SHA-256 digest, and uses that digest as the object address. Identical payloads therefore map to the same key, making writes idempotent. A small in-process cache can also eliminate duplicate PUT or GET operations within one invocation. The benefit extends beyond storage savings: it prevents the same large JSON document from being serialized, transferred, and reprocessed across a multi-turn conversation.
The costs are equally concrete. Canonicalization and hashing consume CPU, while dehydration and hydration add S3 round trips and cold-start latency. AgentCore Memory pressure falls, but S3 request, storage, and cross-region transfer costs may become the new expense line. Content addressing also provides deduplication, not tenant isolation. Production deployments still need per-user or per-session authorization, encryption, lifecycle policies, and recovery rules for dangling references.
2. Page Large Lists: Give the Model a Directory First
When a tool result exceeds the sample threshold, the complete list is written to S3. The model receives only total, a small sample, and an _result_handle. If deeper inspection is necessary, it calls get_result_page_tool with the handle, optional field, offset, and limit. A fixed giant tool result is therefore replaced by an explicit retrieval protocol, preventing the first call from paying the context cost of unrelated rows.
Paging is not free. A model may fetch page after page and refill the context with large volumes of data. Offset pagination is also safe only for stable snapshots; changing collections require versioned snapshots or cursors. A better production interface performs filtering, projection, and aggregation at the service boundary before returning small pages, rather than forcing the model to traverse the raw collection.
3. Compact Nested Dictionaries: Keep Structural Cues, Drop the Body
Many tools return deeply nested dictionaries rather than lists. The original implementation serialized them in full. Once the result crossed a character limit, the optimized path generated a descriptor: list fields retained a count and sample, nested dictionaries retained the number of keys, selected scalars remained inline, and a result handle pointed to the full artifact. The model could still infer the shape of the data and decide whether to retrieve it, but a single tool response no longer carried the entire business payload.
Descriptors introduce information loss. They can hide correlations, outliers, and constraint details, while a model may mistake a field such as keys: 12 for complete content. A production descriptor should define explicit summary semantics, whitelist fields needed for common decisions, cap the amount retrieved in one turn, and validate any detail required for the final answer against business rules rather than assuming the model will notice a missing page.
Engineering Trade-offs
- Context versus storage and network traffic: Token use and AgentCore Memory pressure fall, but S3 GET/PUT operations, serialization, hashing, and connection pools can increase p95 latency. Keep the agent and bucket in the same region and monitor cross-region traffic.
- Summaries versus completeness: Paging and descriptors reduce input size but can increase retrieval calls. The optimization target should be cost per successful task, not the smallest possible tool response.
- Idempotency versus serialization discipline: A SHA-256 address requires every writer to follow the same canonicalization rules. Floating-point formatting, time zones, field ordering, and object versions can all prevent otherwise equivalent payloads from sharing a key.
- State separation versus consistency risk: Live state, AgentCore Memory, and S3 form three distinct state layers. Write ordering, failure compensation, and restore semantics must be defined so memory cannot advance while an S3 artifact remains uncommitted.
- Data minimization versus compliance: A handle is not an authorization mechanism. Sensitive values can still appear in samples, scalar descriptors, logs, or model requests, requiring redaction, least privilege, encryption, and auditing. Bedrock Guardrails can filter final responses, but it cannot replace data minimization at the tool-output boundary.
Reading the Reported Results
| Scenario | Token change | Average trace-latency change | | --- | --- | --- | | Five-question conversation | About 289,500 → 82,900, a 71% reduction | 15.4 s → 3.1 s, about an 80% reduction | | 100 sample questions | 23.6M → 8.5M, a 64% reduction | 7,930 ms → 2,461 ms, a 69% reduction | | Heaviest single query | Input about 32,000 → 8,000; output about 2,800 → 477 | About 165 s → 9.9 s, a 94% reduction |
The post compares the same questions and dataset before and after the redesign, making the direction of the improvement credible. Most of the gain comes from avoiding repeated full-payload injection rather than from changing the model. These remain vendor-reported figures for one representative workload, however. The article does not disclose the Bedrock model, prompt, sampling parameters, thresholds, S3 placement, cache-hit rate, task accuracy, dollar cost, or p95 and p99 latency. It also provides no quantitative quality comparison showing that descriptor-based retrieval is equivalent to reasoning over the complete payload. The 71% and 80% figures should therefore be read as peak observations from this workload, not general AgentCore performance guarantees.
Deployment Playbook
- Profile context contamination first: Use OpenTelemetry or AgentCore Observability to attribute output tokens, context length, model wait time, repeated payloads, S3 operations, and extra page fetches to individual tools. Fix the largest source first.
- Separate live and persisted state: Persist only stable references in AgentCore Memory. Give artifacts explicit versions, tenant identifiers, TTLs, and recovery rules instead of moving large objects into another inline field.
- Use token-aware thresholds: Character-to-token ratios vary substantially across JSON payloads. Estimate thresholds with the target model's tokenizer and configure separate sample limits for each response type.
- Push aggregation into the service: Expose count, group, filter, and projection operations, then page over the reduced result. Disable unbounded model-driven paging by default.
- Control retrieval behavior: Instruct the model to inspect the summary before fetching details, cap page count and total retrieved bytes, and return only answer-relevant fields to the conversational context.
- Do not rely on handles for security: Bind every handle to a user, tenant, session, and role. Encrypt S3 objects, apply lifecycle rules, re-authorize reads at the tool layer, and record access through CloudTrail.
- A/B test quality and tail latency together: Compare task accuracy, rule violations, hallucinations, unnecessary tool calls, cost per successful task, p50/p95/p99 latency, S3 cost, and recovery rate. A token reduction is useful only if quality remains stable and tail latency stays under control.
The transferable idea is not a particular AWS API. It is a change in the agent tool contract: return an addressable reference instead of assuming every result must be delivered in full. That pattern fits enterprise agents with many tools, large JSON payloads, and long conversations. If tool outputs are already small, adding S3, descriptors, and another state layer will only increase complexity.
— END —Source: AWS Machine Learning Blog (2026-11-17)
Tags: #Amazon Bedrock AgentCore #LangGraph #Amazon S3 #Context Engineering #Token Optimization
Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.
Community Comments