AWS Routes Bedrock Coding Agents by Stage with NeMo Switchyard

Summary:AWS has published a reproducible reference architecture for deploying LiteLLM with NVIDIA NeMo Switchyard on Amazon ECS Fargate and connecting it to models hosted by Amazon Bedrock. The Stage strategy examines recent tool activity, sends uncertain requests to a lower-cost model, and escalates to a more capable model when failure signals appear. A weighted Random strategy supports traffic splitting and A/B tests. The sample also measures cost per solved task rather than cost per token, reflectin

---

Optimize for Solved Tasks, Not Individual Tokens

AWS provides a reference architecture that combines LiteLLM with the NVIDIA NeMo Switchyard routing plugin on Amazon ECS Fargate and connects it to models served through Amazon Bedrock. Clients continue using an OpenAI-compatible API and send switchyard as the model name. LiteLLM receives the request, invokes the routing plugin, and then calls the selected model through the Bedrock Converse or ConverseStream API.

A coding-agent task may span many model calls. Its total cost depends on the number of turns, replayed conversation history, failed edits, retries, and attempts that never pass their tests. A useful routing metric is therefore:

[ CostPerSolvedTask=\frac{\sum C_{run}}{#{runs\mid tests\ pass}} ]

A cheaper model can reduce cost per call while still costing more per solved task if its failures trigger additional turns or force an expensive escalation.

How Stage Routing Uses Execution Signals

The Stage strategy in this integration expects exactly two candidates. Their declaration order defines their roles: the first is the capable model and the second is the efficient model. Switchyard examines tool activity and results in the request history:

• A new request has no tool results, so an uncertain signal sends it to the efficient model.
• Repeated failure signals in recent tool results trigger escalation to the capable model.
• confidence_threshold controls the confidence cutoff for execution signals.
• recent_window determines how many recent tool results are inspected.
• only_on_wrong_signal_escalation controls when an optional escalation note is added.

This design does not call a separate judge model. Its routing overhead primarily comes from inspecting history and evaluating execution signals. It does, however, require clients to send the complete supported request history. The upstream adapter supports text messages, function tool calls, and text tool results; unsupported message shapes fail before inference.

The Random strategy selects candidates according to configured weights. It is useful for traffic splitting and A/B testing, but it does not proactively escalate based on task state. Strategies requiring an intermediate classifier model, such as Task and Composite routing, are outside the scope of the current LiteLLM integration.

Reference Architecture and Request Flow

The sample uses AWS CDK to deploy the following path:

1. A coding agent sends an OpenAI-compatible request to an Application Load Balancer.
2. Route 53 and AWS Certificate Manager provide DNS and HTTPS.
3. AWS WAF applies IP reputation, known bad input, and per-IP rate-limit rules.
4. The load balancer forwards only /v1/chat/completions, /v1/models, and /health/liveliness.
5. An ECS Fargate task in private subnets runs LiteLLM and the Switchyard plugin.
6. The task role calls Amazon Bedrock through a Bedrock Runtime interface endpoint.
7. The x-litellm-model-name response header identifies the model that actually served the request, while the response body's model field remains switchyard.

The deployment keeps administrative and key-management paths behind the load balancer, restricts container ingress to the load balancer, stores the gateway secret in AWS Secrets Manager, and obtains AWS credentials from the ECS task role instead of placing long-lived credentials in model configuration.

The security posture still depends on deployment choices. The sample requires a restricted IPv4 /32 allowlist and supports a temporary plain-HTTP mode for testing. Plain HTTP exposes authorization headers, source code, prompts, tool output, and model responses, so it should be limited to short, non-sensitive tests. A Bedrock profile prefixed with global. may also process requests across supported commercial regions; workloads with strict data-residency requirements should use geographic or in-region profiles.

Measuring Cost and Quality Together

The repository provides bench/run.py and bench/multiturn.py. The first lets a coding agent list and read files, edit source files, and run unit tests, with task success determined by the final test result. The second runs multi-turn coding conversations and replays the full history on every turn.

Each turn is priced according to the model that served it and the returned token counts. If cached-input pricing is missing, a model has no entry in the price table, or the response lacks model or token metadata, the turn remains unpriced rather than being treated as free. This prevents missing metadata from systematically understating costs.

At minimum, teams should examine:

• SOLVED: the number of completed tasks;
• COST/TASK: average cost across all attempts;
• COST/SOLVED: cost normalized by successful runs;
• errors, actual model assignments, and task-level differences;
• truncation, output limits, and retry-related cost.

AWS explicitly warns that the sample contains only a few small tasks and cannot support broad cost or quality conclusions. Model outputs vary, so one or two additional solved tasks may be noise. Production decisions require more repetitions and, where appropriate, a statistical test such as Fisher's exact test for solve rates.

The Trade-off: Cheaper Calls Can Delay Failure

Stage routing introduces deliberate delay in escalation. Trying the cheap model first can reduce cost on ordinary tasks, but after failure the system must replay the complete earlier history when switching to the stronger model. Recovery may therefore cost more than starting with the capable model.

If a small set of difficult tasks dominates total cost, better options may include:

• assigning known difficult tasks to a dedicated model group;
• choosing an efficient candidate with a higher solve rate;
• applying deterministic project-level rules before model routing;
• reporting difficult tasks separately instead of relying only on aggregate metrics.

Output-token truncation can shorten the history replayed on later turns and underestimate cost. Prompt caching, batch inference, and pricing changes can also separate offline estimates from actual invoices. Teams should confirm results with AWS Cost and Usage Reports.

Deployment Guidance

1. Deploy the sample in an isolated non-production account, run the local Docker Compose smoke test, and then provision ECS, Fargate, the load balancer, WAF, and Bedrock permissions through AWS CDK.
2. Compare the routed group and single-model baselines on the same tasks with enough repetitions to distinguish real effects from sampling noise.
3. Match production agent frameworks, turn limits, tools, and prompts as closely as possible.
4. Record the serving model, token usage, cache hits, retry causes, and test results rather than summarizing token prices alone.
5. Sandbox model-written code and tool execution. The benchmark subprocess is not an operating-system sandbox and may read files available to the current user or access the network.
6. Re-run the evaluation whenever models, routing thresholds, prompts, or tool behavior changes.

The main contribution is not a new foundation model but a practical and auditable routing architecture. It reframes model selection around task outcomes and makes routing policy measurable through a common gateway and repeatable evaluation workflow. The Switchyard LiteLLM integration is still experimental, and switchyard-server 0.3.0 is documented as a demonstration and evaluation component. It is best treated as a starting point for building an enterprise coding-agent routing layer, not as a production-ready deployment in its current form.

---

Source: AWS Machine Learning Blog (2026-11-13)

— END —

Tags: #NeMo Switchyard Follow #Coding Agents Follow #Model Routing Follow #Amazon Bedrock Follow #Cost Optimization Follow