ZICQ
中 Log in / Sign up
Newsroom Agentic #Airbnb #Amazon Bedrock #Coding Agents #Cost Observability #LLM Evaluation

Airbnb Runs Coding Agents on Amazon Bedrock with Task-Level Cost Control

Avatar of Mr.Xu

By Mr.Xu Compiled & Reviewed by Editorial

Published:

中文阅读 (Chinese) English Version
Co-reading Select any sentence in the article to highlight, annotate, or copy a quote card. Join discussion ↓

Summary:Airbnb has published the operating model behind AirChat, its internal coding-agent platform, where requests pass through an LLM gateway to Amazon Bedrock. Before rollout, the company scores models on representative engineering tasks at each reasoning effort; afterward, it reconciles task-level evaluations with CloudWatch latency, throttling, token-mix, and spending telemetry. The relevant decision metric is not the price of one API call, but the list-price cost of every token consumed across all


Core facts and request path

Thousands of engineers and engineering-adjacent employees at Airbnb use AirChat, the company's internal coding-agent platform, every week. AirChat can run in an engineer's cloud-based AirDev workspace or through AirChat Remote, which executes agents in separate cloud workspaces. Both paths pass through Airbnb's LLM gateway before reaching models on Amazon Bedrock. AWS manages the model infrastructure, while Airbnb uses IAM, invocation logs, and CloudWatch for access control and operational governance. Code, prompts, and model responses are processed within AWS.

The model supply changes quickly: eight new models arrived in roughly five weeks, and by mid-October about two thirds of Airbnb's coding-agent requests were using those models. At that pace, defaults, reasoning-effort limits, and quota assumptions established at launch quickly become obsolete.

Airbnb's operating loop has two data paths:

  1. Pre-rollout evaluation: Representative engineering tasks are run against different models and reasoning efforts. Quality scores and task-level costs determine the default model, default effort, and enforced effort ceilings.
  2. Post-rollout observability: The LLM gateway records the model, client, team, and user for each request. CloudWatch supplies token, latency, error, and throttling metrics. Aggregated production data is then used to revisit evaluation-based choices and adjust capacity and routing.

Why task-level cost is the correct unit

A coding agent generally needs many model calls, and each call may reread the accumulated context. Comparing per-million-token prices or the cost of one request therefore understates retries, repeated context, tool loops, and low-quality attempts. Airbnb instead defines cost as the public list-price cost of all tokens consumed across every request required to complete one task.

For request j within a task, cost can be represented as:

C_j = p_u × U_j + p_r × R_j + p_w × W_j + p_o × O_j

C_task = Σ C_j

Here, U is uncached input, R is cache-read input, W is cache-write input, and O is output. Each category uses its corresponding model price. Airbnb records average cost by model, reasoning effort, and agent client, including failed or low-scoring attempts so that successful runs alone do not understate production economics.

The evaluation tasks come from real code changes, review comments, and engineering decisions that original tickets did not explicitly state. Airbnb maintains seven task suites, scored by test programs or model-based reviewers over frozen data. Long-running tasks simulate reviewer pushback across multiple rounds. The example rubric evaluates delivery correctness, verification quality, judgment, evidence and reporting, autonomy, readability, design and workload fit, and operational discipline. Runs requiring repeated human rescue are subject to a score ceiling. Task directories can use the open-source Harbor framework, including sandbox environments, resource limits, tests, and reference solutions.

Reasoning effort cannot be optimized independently of quality. On one model tested by Airbnb, moving from low to high effort increased the score by only 0.8 points while multiplying task cost by 2.5. The gateway therefore enforces a per-model effort ceiling rather than exposing every setting by default.

Why evaluation and production economics diverge

An anonymized model A scored more than three points above Airbnb's previous best model on the same tasks and grader, with roughly the same task-level cost. It was consequently selected as the default. Several weeks after rollout, however, each model A request cost about three times as much as the old model's request, and each user sent about three times as many requests. Average weekly spending consequently became several times higher.

The token mix explains the discrepancy. Output tokens accounted for only about one tenth of request cost; most spending came from agents repeatedly reading cached context. Once model A could operate autonomously for longer periods, users naturally shifted to longer self-directed sessions. The evaluation measured model efficiency on a fixed task, while production also measured how users changed their behavior after the model improved.

A later model achieved a higher score on the same tasks and grader at roughly one sixth of model A's task-level cost. After Airbnb moved model A's heavy users to the newer model, spending on their primary model fell to about one seventh during the first full week, with request volume remaining broadly unchanged. Static leaderboards are therefore insufficient for production routing; teams must continuously compare the cost-quality frontier on real tasks.

CloudWatch runtime signals and latency diagnosis

Airbnb uses several metrics in the AWS/Bedrock CloudWatch namespace:

  • TimeToFirstToken: queueing, prefill, and gateway delay before generation begins.
  • InvocationLatency: end-to-end invocation latency.
  • OutputTokenCount: response length, helping distinguish slower generation from longer answers.
  • InvocationThrottles: rejected or throttled requests.
  • InputTokenCount, CacheReadInputTokenCount, CacheWriteInputTokenCount, and OutputTokenCount: the four billable token categories.

Bedrock does not natively publish output tokens per second. Airbnb derives OTPS as follows:

OTPS = OutputTokenCount ÷ (InvocationLatency - TimeToFirstToken) × 1000

In one production incident, invocation latency rose by 40 percent while TTFT and OTPS remained flat. Investigation showed that a system-prompt change requested more thorough answers, increasing output tokens by 50 percent without reducing generation speed. Airbnb consequently treats prompt changes and model upgrades as deployment events and marks them on its dashboard. TimeToFirstToken is available only for streaming APIs, so workloads that need it must use ConverseStream or InvokeModelWithResponseStream.

TPM quota is not the bill

When a request reaches Bedrock, the service checks token-per-minute quota before admitting it. Its initial reservation is approximately:

reserved_tokens ≈ uncached_input + cache_write + max_tokens

Cache reads do not consume quota. As generation proceeds, Bedrock adjusts the reservation and returns unused max_tokens capacity after settlement. Output tokens settle at the model's burndown multiplier, which is 1:1 on most models but several times higher on some current models. A request can therefore be cheap in quota terms but expensive on the invoice, and a dashboard based only on settled tokens can understate admission pressure.

In Airbnb's example, the TPM quota was one million and traffic was 300 requests per minute. The dashboard showed roughly 30 percent quota consumption, but after applying a 5:1 output burndown rate, completed requests consumed about 77 percent. EstimatedTPMQuotaUsage includes cache-write tokens and output multipliers across the inference APIs, making it more suitable for proactive alarms, but thresholds must be calibrated to each model's settlement rules.

Lowering max_tokens can free admission capacity. In the example, a 4,096-token limit plus 600 uncached input tokens produced a reservation of 4,696 tokens. Reducing the limit to 1,024 lowered the reservation to 1,624 tokens, a reduction of roughly 65 percent while remaining above the approximately 800-token p99 response length. If actual output remains unchanged, final quota consumption does not increase. The trade-off is truncation: an overly small hard limit can damage long-task completion. Teams should monitor stopReason, use exponential backoff with jitter, and cap retries.

This creates a direct engineering trade-off. A smaller max_tokens improves concurrent admission but may lower long-horizon task completion, while a larger limit reduces truncation risk but inflates reserved tokens and burst throttling. The appropriate ceiling should be derived from task-length distributions, p99 output, truncation reasons, and quality evaluations rather than intuition alone.

Spend attribution, access control, and lifecycle governance

Bedrock runtime metrics can be split by ModelId, but that dimension identifies the serving model rather than the requesting team. Airbnb uses three attribution paths:

  1. IAM identity and session tags: Cost-allocation tags on caller roles or session tags appear in Cost Explorer and CUR 2.0. Behind a gateway, Bedrock sees only the gateway role, so the gateway assumes a user-specific role or attaches session tags. Credentials should be cached for the session to avoid exhausting AWS STS rate limits on every request.
  2. Application inference profiles: Because an application inference profile is model-specific, Airbnb creates a tagged profile for each application-model pair and invokes the corresponding ARN.
  3. Request metadata: requestMetadata supplied through Converse is written to model invocation logs but not directly to the bill. It must be priced using token counts from those logs. Because neither classic CUR nor CUR 2.0 provides a unified per-request ID, totals must be reconciled by model, usage type, and time interval.

Teams must also account for failures that occur before Bedrock resolves a model. Those requests appear only in series without the ModelId dimension, so filtering directly on that dimension can undercount errors.

Model invocation logging is disabled by default. Once enabled, logs may contain complete prompts, responses, and caller identities, requiring restricted read access and a defined retention period. Cost-allocation tags should also be activated before data collection because earlier spending is not backfilled. Bedrock Guardrails and contextual-grounding checks can filter harmful content and unsupported claims, but their latency and token costs must be included in task-level evaluations rather than assessed only by interception rate.

Engineering trade-offs and limits of the evidence

  • Quality and cost are not linear: A 0.8-point gain for 2.5 times the cost means higher reasoning effort is appropriate only for tasks with high value and a low cost of failure. Defaults should follow the Pareto frontier rather than always selecting the highest setting.
  • The task suite defines the validity boundary: Real code changes are closer to production than generic benchmarks, but seven internal suites may omit other repositories, languages, and task types. Grader quality, human-intervention cost, and model-grader bias all affect the results.
  • List prices aid reproducibility but do not equal the invoice: A dated public-price table permits consistent comparisons, but enterprise discounts, regional pricing, cache tiers, sandbox compute, gateway processing, and observability costs remain outside the task-cost calculation.
  • Production behavior is endogenous: Better models change session length, request frequency, and tool-use patterns. A one-time offline evaluation cannot permanently determine the default model.
  • Model churn introduces drift: Eight new models appeared in five weeks, making version pinning, prompt and parameter change tracking, and post-update reevaluation essential.
  • The numerical results require independent reproduction: The figures come from the Airbnb and AWS project teams. Model identities are anonymized, and sample sizes, confidence intervals, and full task composition are not disclosed, so the results should not be generalized uncritically.

Deployment recommendations

  1. Build task suites from real repository changes, code reviews, and implicit requirements, with reproducible environments, resource budgets, and trusted graders.
  2. Correlate all calls belonging to one task, record each of the four token categories separately, and calculate full task cost using a dated public-price table.
  3. Sweep every model and reasoning effort before launch, remove configurations that cost more without improving quality, and enforce defaults and effort ceilings at the gateway.
  4. Compare evaluation task cost with production cost per request and requests per user every week to distinguish token-mix changes from behavior changes.
  5. Put TTFT, generation-stage OTPS, output length, throttling, and all token categories on the same dashboard; tag prompt, model, and gateway configuration changes as deployments.
  6. Alarm on EstimatedTPMQuotaUsage, calibrate thresholds to each model's burndown multiplier, and monitor truncation caused by max_tokens limits.
  7. Attribute team spending with IAM roles, session tags, and application inference profiles, while alarming on missing invocation logs so attribution failures do not silently continue.
  8. Maintain at least one cross-family fallback with comparable evaluation performance for every major model, verify its capacity and quota before shifting traffic, and alarm on LegacyModelInvocations to avoid abrupt end-of-life failures.

Source: AWS Machine Learning Blog (2026-11-18)

— END —

Tags: #Airbnb #Amazon Bedrock #Coding Agents #Cost Observability #LLM Evaluation

Editorial & Fact-Checking Note: This article is compiled from primary research, official release documentation, and source papers by the ZICQ Newsroom pipeline with automated entity verification and human editorial review. If you notice any technical inaccuracy, please submit a correction via our corrections policy or email our editorial desk directly.

What's your take on this story?
🚀 Explore & Submit Works

Community Comments

Loading live comments and annotations…