AWS Connects Specialized Decision Models to AgentCore Evaluations

Summary:Amazon Bedrock AgentCore Evaluations can now invoke Jev or the open-source Strands Decider 2B through code-based AWS Lambda evaluators. The integration translates agent traces into fixed labels and calibrated probabilities for binary, multiclass, and ordered-score questions. Its main value is not to replace LLM-as-a-judge, but to create a fast screening tier: decision models inspect every session, while only low-confidence cases are escalated to more expensive and explainable LLM evaluators.

---

Core Update

Amazon Web Services has added a practical path for connecting specialized decision models to Amazon Bedrock AgentCore Evaluations. Beyond its 15 built-in evaluators, AgentCore can invoke custom logic through an AWS Lambda code-based evaluator. That function can call TypeSafe's hosted Jev service or the open-source Strands Decider 2B deployed under the user's control.

Decision models are not general-purpose chat models. They are optimized for a fixed set of outcomes and return labels together with probability distributions:

• noul produces a probability for True or False;
• choice selects from as many as 255 options and exposes the full distribution;
• score predicts one of 2 to 10 ordered levels and returns a probability-weighted value.

They therefore fit narrow evaluation questions such as whether a response is supported by tool results, whether an agent complied with a policy, or whether a task was completed.

How the Evaluation Path Works

AgentCore supplies the evaluator with OpenTelemetry spans. The Lambda adapter normalizes those spans and prepares the model request according to evaluation scope. A TRACE evaluation includes the target turn and a short amount of prior context. A SESSION evaluation includes the full conversation so that evidence is not lost during summarization.

The output adapter maps model probabilities into AgentCore's result schema. For noul, the True probability becomes the value and the most probable answer becomes the label. For score, the probability-weighted expected level is scaled to [0, 1]. For choice, the weighted average of predefined option values is returned.

For a four-level helpfulness rubric, the expected score can be expressed as:

E[score] = p0×0 + p1×(1/3) + p2×(2/3) + p3×1

This provides a continuous value for threshold control, but it does not remove ambiguity in the rubric itself. It is still a transformation of discrete model probabilities, not an independent measure of quality.

Why Decision Models Do Not Replace LLM Judges

An LLM-as-a-judge is valuable when evaluation requires connecting evidence, detecting missing information, or explaining the reasoning behind a judgment. A decision model is better suited to enforcing a fixed rubric repeatedly and returning a confidence signal that software can act on. Their output contracts differ: the former emphasizes a score plus explanation, while the latter emphasizes a label plus calibrated probability.

AWS recommends a cascade. Decision models screen every session, while failed or borderline sessions are escalated to the built-in LLM evaluators for deeper analysis. This reduces the number of expensive evaluator calls while retaining explanations for ambiguous cases.

A stronger production design has three layers:

1. Deterministic code for constraints that can be computed exactly;
2. Decision models for fixed-label and ordered-rubric checks;
3. LLM judges for semantic tasks such as faithfulness, helpfulness, and goal completion.

Calibration and Engineering Trade-offs

The main advantage of a decision model is not merely binary classification. Its confidence can drive operational thresholds. AWS recommends beginning with a labeled sample drawn from real traffic, including failures and edge cases. Three independent annotators should score each turn, with the majority vote used as the reference label.

Validation should cover three areas:

• Agreement: measure agreement with majority labels separately for unanimous and disputed cases;
• Calibration: group predictions by confidence and verify that actual accuracy rises with confidence;
• Disagreement analysis: inspect cases where the decision model and LLM judge differ to identify wording or rubric ambiguity.

A threshold of 0.5 should not be assumed. If predictions in the 0.7 confidence bucket are only 60% accurate, the threshold should be selected according to the business cost of false positives and false negatives. Re-run validation whenever the question wording or model version changes.

AWS reports that LLM judges typically take 2 to 10 seconds, whereas network-bound decision-model calls can complete in hundreds of milliseconds. This is not a controlled apples-to-apples benchmark: model size, context length, hosting location, and network path can materially change the result. Jev also introduces an external service dependency, data-transfer concerns, and API charges. Self-hosting Strands Decider 2B avoids those platform dependencies but adds GPU capacity, VPC networking, and operational overhead.

Developer Deployment Guidance

AWS also provides an HR Assistant sample built with Strands Agents. It includes deterministic response-length and regex-based fact checks, along with switchable Jev and Strands Decider backends.

A practical rollout sequence is:

1. Add deterministic checks for exactly computable constraints.
2. Select narrow questions with a limited answer space and high human-label agreement.
3. Run the decision model and existing LLM judge over the same labeled sessions.
4. Set thresholds according to the acceptable false-positive and false-negative costs rather than defaulting to 0.5.
5. Use the decision model as a universal screening layer and escalate only borderline cases to LLM evaluation.
6. Recalibrate after model, prompt, or rubric changes.

For high-risk systems, a decision-model probability is not equivalent to factual correctness. It represents the model's confidence under a predefined rubric and remains exposed to training-distribution shift, question wording, and annotation bias. Human validation and continuous drift monitoring remain necessary before release gating.

---

Source: AWS Machine Learning Blog (2026-11-04)

— END —

Tags: #AgentCore Follow #Decision Models Follow #Agent Evaluation Follow #LLM-as-a-Judge Follow #AWS Lambda Follow