Agent DLC on AWS: Evaluation Loops for Shipping LLM Agents

AWS's November 12, 2026 Agent DLC post argues enterprise agents need a lifecycle that looks more like regulated software plus statistical QA than a single green CI build. Agent Development Lifecycle (Agent DLC) wraps six activities around one gold standard business owners approve: criteria, thresholds, and exemplar queries.

The post is timely beside OpenAI chain-of-thought monitorability and China NDRC full-lifecycle safety standards — different regulators, same structural point: ship evidence, not vibes.

TL;DR

Three agent properties that break classic CI

AWS Table 1 maps:

That aligns with explainx.ai guidance in evaluating prompts and WorkflowArena orchestration benchmarks: aggregate benchmarks without per-intent gates hide regressions.

Reproducible benchmark stamps illustrating Agent DLC release gates and evaluation evidence

Evaluation harness bench echoing Agent DLC inner and outer feedback cycles

Measured HR assistant: precision versus coverage

AWS's fictional internal policy assistant (HR docs over Managed Knowledge Base) is a teaching artifact, not a customer benchmark.

First eval round highlights:

Lesson: chunk-level precision asks if passages are on-topic; fact-level precision asks how much of each chunk matters; retrieval coverage asks whether any chunk answers the sub-question. Sick-leave failed because corpus lacked a dedicated policy — prompt edits that demand strict grounding cannot invent missing files.

AWS then changed only the system prompt (ground in sources, ignore irrelevant retrieval, stay focused):

• Performance review improved on fact and response precision.
• Sick-leave worsened — confirming diagnosis before more prompt thrash.

A separate benefits pass at groundedness 0.90 still hallucinated a user profile contact preference by blending policy steps with noisy FAQ text — high groundedness did not catch a responsibility violation.

Five dimensions and three tiers

Dimensions: domain understanding, quality, responsibility, cost, performance — each with verifiable criteria ("every factual claim traceable to retrieved source" beats "be accurate").

Tiers:

• Hard constraint — boolean per run (e.g., cross-customer disclosure).
• Release gate — pass rate over k runs (e.g., 95% citation traceability).
• Monitoring only — tracked until evaluators calibrate.

Release matrix: any hard constraint fail → stop; gates missed → block intent or limited rollout with pre-approved safeguards; all pass → ship.

Inner versus outer loop confusion

AWS warns teams not to prompt-fix missing authorization criteria (outer problem) or widen standards mid-sprint (confounds causality). Assets — criteria, examples, evaluators, failure records — survive model swaps.

AWS documents Amazon Bedrock AgentCore as a platform with Evaluations, Observability, Gateway, Identity, and Memory services. Fotor's cited AgentCore runtime migration case shows acceptance criteria before architecture (latency, isolation, cost) — Agent DLC for platform, with quality eval labeled next step.

Getting started timeline

AWS proposes:

• Days 1–2: owner + engineers agree tiers; 20–30 golden examples.
• Days 3–5: implement evaluators; publish baseline with failures.
• Week 2: one controlled change; document release decision evidence.

Services named: AgentCore Evaluations (built-in + custom judges), Observability traces, CloudWatch ops metrics, runtime/Gateway/Identity/Memory for execution plumbing.

Connect to UAEF universal agent evaluation when you need cross-framework scoring; Agent DLC is the program management wrapper around those scores.

What builders should do Monday

1. Pick one intent (expense policy, access reviews, ticket routing).
2. Write hard constraints first — PII, cross-tenant reads, tool scopes.
3. Add fact-level and coverage metrics beside chunk precision for RAG agents.
4. Store eval bundles next to prompt versions in git or model registry.
5. Schedule weekly re-runs even when repo is quiet — dependencies still move.

Metric definitions (so teams do not argue in Slack)

AWS's sick-leave row is the teaching moment: three metrics must be read together or you ship a confident wrong answer.

Custom evaluators versus built-in

AgentCore Evaluations supports built-in judges plus custom code for hard constraints (regex on prohibited fields, authorization simulation). Map hard constraints to code where possible — LLM judges are for judgment, not law.

Sampling factor k

Non-determinism means k runs per golden example. AWS does not mandate a universal k; start with k equals 5 for high-risk intents, k equals 3 for low-risk FAQ — tune until pass-rate variance stabilizes below your release threshold.

Connecting Agent DLC to CI

Traditional CI still runs:

• Unit tests on tool wrappers
• IaC lint on AgentCore deployments
• Secret scanning on repos

Agent DLC adds eval bundles as merge gates — a failed hard constraint should block promote even when unit tests pass.

Outer loop examples beyond HR

Anti-patterns AWS warns against

• Widening thresholds after a failed release gate without owner sign-off
• Prompt-only fixes for missing retrieval coverage
• Mixing inner and outer loops in one sprint (confounds causality)

Economic framing

Eval runs cost tokens + engineer time. Agent DLC argues that cost is smaller than one production disclosure incident — parallel to Paris Call fail-safe language for systemic-risk systems.

Organizer roles

Without named owners, Agent DLC decks become slideware while production agents stay demo-grade.

Starting artifact list (week one deliverables)

1. Gold standard PDF — criteria, tiers, owners, thresholds.
2. Golden JSONL — 20–30 approved prompts + expected behavior notes.
3. Evaluator package version pinned in git tag.
4. Baseline report — failures with traces, not just scores.

Worked example: turning a failed sick-leave answer into action

AWS's narrative implies a playbook:

1. Retrieval coverage 0.00 → open ingestion backlog for missing HR PDF.
2. Fact precision 0.09 with chunk precision 1.00 → do not tune prompts; fix corpus.
3. After upload, re-run baseline before touching system instructions.
4. Only then consider prompt constraints for verbosity.

Teams that skip step 1 enter prompt endless loop, burning tokens while users still get wrong HR guidance.

Bridging Agent DLC to incident reviews

When production agents disclose data, post-incident reviews should ask:

• Did we violate a documented hard constraint?
• Was the criterion missing from gold standard (outer loop gap)?
• Did evaluators drift after model swap without re-baseline?

Tie answers to AgentCore Observability traces stored with eval bundle IDs.

Procurement language

RFPs can request "Agent DLC-aligned delivery" meaning vendor supplies gold standard, golden sets, and release gate reports — not merely "accuracy quoted in slide three."

Versioning evaluators with models

When Bedrock or OpenAI bumps a default model behind your agent, re-run the full gold set even if git is quiet. AWS's third property — dependencies move independently — is the reason scheduled eval belongs in SRE calendars, not only in release pipelines. Store model ARN and temperature beside each eval result row so regressions are attributable.

Cultural shift for engineering managers

Demo-first teams resist Agent DLC because it feels like slowing launches. Reframe: inner loop shortens time-to-diagnosis after a bad deploy. Outer loop prevents repeated incidents from missing criteria. Present mean time to understand failure metrics to leadership, not just pass rates.

Handoff from data science to production

Data scientists often own first demo eval notebooks. Agent DLC asks them to productize those notebooks into AgentCore Evaluations jobs with pinned dependencies. Define a ceremony when notebooks graduate: README, owner on-call, rollback script, and minimum golden set size enforced in CI.

SRE teams can attach Agent DLC eval failures to the same paging policy as synthetic monitors once baselines exist. That sounds heavy, but it is lighter than explaining a wrong HR benefits answer to regulators without a stored release gate record.

Related on explainx.ai

• Evaluating prompts
• OpenAI CoT monitorability
• WorkflowArena benchmark
• Bedrock KB ACL gateway pattern
• Raindrop tau2 bench memory
• AWS UAEF
• Longtail SWE benchmark

HR assistant tables and script names mirror the November 12, 2026 AWS blog; scores vary by run.