MediumAnswer, Verify, or Escalate: Building a Judge-Gated Healthcare Benefits Multi Agent System on AWS (Part 2) | by Ambuj | Nov, 2026 | Medium
Sitemap
Open in app

Sign in

Medium Logo
Write
Search

Sign in

Answer, Verify, or Escalate: Building a Judge-Gated Healthcare Benefits Multi Agent System on AWS (Part 2)

14 min readNov 5, 2026

--

Preserving every turn across the AI-to-human handoff, failing safely, and measuring whether the Judge deserves your trust

Disclaimer: Product names, store identifiers, and dialog examples in this article are synthetic.

We covered these topics in Part 1: why a benefit explanation is a composition across fragmented systems, how a Specialist Agent, deterministic evidence checks, a synchronous LLM-as-Judge, and a fail-closed decision gate run as a Strands Graph inside Amazon Bedrock AgentCore Runtime, and why the Judge’s scores are uncalibrated signals rather than probabilities. In Part 2, let’s continue with what happens when the gate says no, and how you’d know whether the gate is right.

7. Persistence: the record has to outlive the session

The human agent needs the conversation within seconds of a transfer, auditors may need it months later, and the evaluation pipeline needs to know exactly what the caller heard versus what was only proposed. Meanwhile, the AgentCore Runtime session holding the live conversation can end on an idle timeout or its maximum lifetime. So the record lives outside the runtime, in two services with one job each:

  • Amazon S3 is the system of record for conversation content: every turn event and the consolidated handoff snapshot.
  • Amazon DynamoDB is the index and control plane for handoffs: small records holding status, the Judge’s verdict, and a pointer into S3. It never stores transcript bodies; items are capped at 400 KB, and the lookup only needs a pointer.

Capture progressively, append-only

Each turn event is written as its own immutable object while the conversation progresses, rather than as one transcript rewritten after every message:

s3://customer-ai-conversations/
  conversations/conv-12345/
    events/
      000001-customer_message.json
      000002-ai_delivered.json
      000003-customer_message.json
      000004-specialist_proposed.json
      000005-judge_evaluation.json
      000006-escalation.json
    handoffs/
      handoff-001/context.json

Every event object is written with the If-None-Match: * header. S3 rejects a conditional write to an existing key with 412 Precondition Failed, and when conditional writes race for the same key, the first to finish wins.

Ordering comes from the orchestrator-assigned sequence number. Timestamps are recorded but never used for ordering, because clocks skew across hosts.

The event type encodes transcript integrity from the first write:

{
  "conversationId": "conv-12345",
  "seq": 4,
  "turnId": "turn-002",
  "eventType": "SPECIALIST_PROPOSED",
  "deliveredToCustomer": false,
  "content": "For SRV-1042 the member pays 20% coinsurance after the deductible.",
  "evidenceRefs": ["tooluse_7f3a", "tooluse_91bc"],
  "agentVersion": "specialist@2026-10-28.3",
  "modelConfig": "specialist-config-v14",
  "traceId": "1-6730c2f1-4b7d9e2a8c1f0e3d5b6a7c8d",
  "timestamp": "2026-11-05T16:01:02Z"
}

A rejected draft stays SPECIALIST_PROPOSED with deliveredToCustomer: false permanently. Only AI_DELIVERED events, written after the response is returned to the channel, represent what the customer received. Private model reasoning and system instructions are not stored; the handoff needs what happened, not how the model reasoned.

The customer-message write can run concurrently with the Specialist call, because it only needs to be committed before a snapshot is built. S3’s strong read-after-write consistency lets the snapshot builder read the event prefix as soon as those writes return, which also lets it rebuild the transcript after a session restart.

The handoff snapshot

On escalation, the orchestrator assembles one consolidated object for the human-side application:

{
  "conversationId": "conv-12345",
  "handoffId": "handoff-001",
  "channel": "VOICE",
  "caller": { "role": "PROVIDER", "providerRef": "prv-ref-5521", "memberRef": "mbr-ref-0917" },
  "conversationHistory": [
    { "turnId": "turn-001", "role": "CUSTOMER",
      "content": "I need to verify benefits for a member." },
    { "turnId": "turn-001", "role": "AI_SPECIALIST", "deliveredToCustomer": true,
      "content": "I've confirmed the member's coverage is active. Which service are you checking?" },
    { "turnId": "turn-002", "role": "CUSTOMER",
      "content": "Outpatient MRI, SRV-1042. What will the member owe, and is prior auth needed?" }
  ],
  "rejectedResponse": {
    "turnId": "turn-002",
    "deliveredToCustomer": false,
    "content": "For SRV-1042 the member pays 20% coinsurance after the deductible."
  },
  "judgeEvaluation": {
    "decision": "FAIL",
    "failureReasons": ["Group exception for freestanding imaging not applied",
                       "Prior authorization requirement not addressed"]
  },
  "handoffContext": {
    "customerIntent": "COST_SHARE",
    "completedLookups": ["eligibility", "plan", "serviceCode", "baseBenefit", "groupExceptions"],
    "unresolvedIssue": "Member cost share and prior authorization for SRV-1042",
    "escalationReason": "QUALITY_GATE_FAILURE"
  },
  "versions": { "specialist": "specialist@2026-10-28.3", "judge": "judge-rubric@v7" },
  "traceId": "1-6730c2f1-4b7d9e2a8c1f0e3d5b6a7c8d"
}

This is an illustrative data contract containing no real member information. The providerRef and memberRef values are reference tokens that the agent desktop resolves through its own authorized systems, rather than raw identifiers copied into another store. Whether the human re-verifies the caller after transfer is a compliance policy decision, not an AI decision.

The handoff record in DynamoDB

Three access patterns drive the schema: the agent desktop fetches a handoff by handoffId from the transfer metadata; operations finds every handoff for a conversation when a caller escalates more than once; and a reconciler finds handoffs stuck in PENDING.

Table: AgentHandoffContext
  PK    handoffId
  attrs conversationId, channel, handoffStatus, judgeDecision, judgeFailureReasons,
        escalationReason, s3Bucket, s3Key, s3VersionId, lastTurnId,
        contextSummary, traceId, createdAt, updatedAt

GSI1  PK conversationId   SK createdAt   → all handoffs for a conversation
GSI2  PK pendingShard     SK createdAt   → sparse: only items currently PENDING

GSI2 is sparse. The pendingShard attribute, a small random integer, is set when a record is created as PENDING and removed in the same update that moves it to READY or FAILED, so the reconciler queries only in-flight records. Sharding avoids what a plain handoffStatus index would create: a handful of partition key values absorbing every handoff write.

The lifecycle has four states. PENDING means the snapshot is being written and carries no S3 pointer. READY means the complete context is stored and readable. TRANSFERRED means the channel platform confirmed the handoff; the orchestrator never infers it from having requested one. FAILED means persistence didn't complete.

Every transition is an UpdateItem with a ConditionExpression on the current status, which makes transitions idempotent: a retried transition that already succeeded fails its condition and is confirmed with a read. DynamoDB TTL can expire records after the operational window, while S3 lifecycle rules, plus Object Lock if your retention policy requires immutability, govern the archive.

Writing to two stores without a distributed transaction

S3 and DynamoDB don’t share a transaction, so the protocol orders the writes to make every intermediate state safe.

Press enter or click to view image in full size
Figure 3 — The pointer is published only after the content is committed. The reconciler resolves anything left in between.
  1. Create the record as PENDING, conditioned on attribute_not_exists(handoffId).
  2. Write the snapshot to S3 with If-None-Match: * and capture the version ID (versioning is on).
  3. Update the record to READY with s3Key and s3VersionId, conditioned on handoffStatus = PENDING.
  4. Only then return the handoffId to the channel.

If the process dies after step 1, the reconciler finds a PENDING record with no snapshot, marks it FAILED, and alerts. If it dies after step 2, the reconciler finds the snapshot and completes the move to READY. A reader can never see READY pointing at a missing object, because READY is written only after S3 confirmed the write. The reconciler is a small scheduled function, such as an Amazon EventBridge Scheduler rule invoking an AWS Lambda function every minute against the sparse index.

Human handoff without repetition

Transfer metadata carries the handoffId and a few routing attributes, never the transcript. Contact-center attributes are often size-limited and visible in more places than an encrypted archive, so keeping PHI out of them reduces exposure, and the desktop fetches context under its own authorization.

The caller hears a fixed, pre-approved template, never text generated by the Specialist whose answer just failed:

"I want to make sure you get an exact answer, 
so I'm connecting you with a benefits specialist 
who can see our conversation so far."

The desktop resolves the handoffId to the DynamoDB record, then reads the snapshot using a role scoped to the handoff prefix and to kms:Decrypt on the archive key. It presents, in order: the unresolved issue and escalation reason, the lookups already completed, the delivered transcript, and the rejected draft labeled "Not sent to caller," with the Judge's reasons. Those reasons usually point at the step to check first; here, the group exception and the authorization requirement. The draft also carries a risk, because a busy agent might read it aloud, so label it unmistakably and consider collapsing it by default.

If the snapshot or record can’t be written, preserve the transfer anyway: a caller stuck in automation is worse off than a human working without context. Mark the record FAILED, alert, and let the desktop show "context unavailable" while it falls back to whatever turn events exist.

End to End Journey — One compelte Turn

  1. The channel transcribes the caller and passes the conversationId, verified provider and member identifiers, and the utterance; AgentCore Runtime routes the request
  2. The prepare node loads recent context (from the session, or from S3 events after a restart), assigns turn and sequence numbers, and starts the customer-event write in parallel.
  3. The Specialist follows its Navigator-derived procedure through read-only Gateway tools, with Policy evaluating each call; the evidence recorder captures every result, and the Specialist returns a SpecialistDraft.
  4. The orchestrator writes a SPECIALIST_PROPOSED event with deliveredToCustomer: false.
  5. Deterministic checks confirm required evidence, claim linkage, and numeric fidelity. A failure jumps to step 9.
  6. The Judge returns a JudgeVerdict from the draft and captured evidence, persisted as a JUDGE_EVALUATION event.
  7. The gate validates the verdict and applies calibrated thresholds.
  8. On PASS, the response returns to the channel and an AI_DELIVERED event is written. The next turn gets its own Judge evaluation.
  9. On ESCALATE, nothing is regenerated. The handoff protocol runs, the channel receives the handoffId, and the caller hears the fixed transfer message.
  10. The channel routes the contact and, on confirmation, the record moves to TRANSFERRED.
  11. The human desktop reads the record and snapshot and continues from where the AI stopped.
  12. Throughout, spans flow to AgentCore Observability; AgentCore Evaluations later scores sampled sessions, and the human’s disposition on the escalation is recorded for calibration.

Failure modes

Part 1 covered the Judge’s own failures, which the gate treats as ESCALATE, and unverified identity, which never reaches member-specific tools. The remaining cases:

  • A tool times out or errors. The evidence check fails and the turn escalates. Tools must return explicit errors, not empty data, because an empty Group Exceptions response has to be distinguishable from “this group has no exceptions.”
  • Amazon Bedrock throttles. Retry once with jittered backoff if the turn budget allows, then escalate. Retry at a single layer: the AWS SDK, the agent framework, and your code can each retry, and stacked retries multiply load exactly when the service is saturated.
  • Evidence is stale. Tool results carry as-of timestamps, and the evidence check rejects results outside the window your policy allows.
  • An event is written twice, or persistence fails. Conditional writes reject duplicates; a failed handoff write triggers the degraded transfer.
  • The Judge model has an extended outage. Every turn escalates, which is safe for callers and dangerous for staffing. Alarm on escalation rate and decide in advance when a sustained outage switches the entry point to human-first routing.

Two evaluation loops

The synchronous Judge answers one question: may this answer go out now? It can’t tell you whether the Judge itself is right. That takes a second loop, outside the response path.

Press enter or click to view image in full size
Figure 4 — The top loop protects each caller; the bottom loop measures the protector.

AgentCore Evaluations: managed and asynchronous

Online evaluation samples and scores live traces, and on-demand evaluation tests agents programmatically, including as a CI/CD regression gate. It provides 13 built-in evaluators plus custom evaluators, either LLM-based with your own prompt and model or code-based in Lambda. It reads OpenTelemetry traces from CloudWatch, which requires CloudWatch Transaction Search, and online evaluation can use only evaluators that don’t need ground truth.

Get Ambuj’s stories in your inbox

Join Medium for free to get updates from this writer.

Here it plays three roles. Online monitoring samples delivered sessions with built-in evaluators such as faithfulness and correctness, plus a custom benefits evaluator; if the Judge’s PASS rate holds steady while independent faithfulness scores fall, one of them is drifting. Regression testing runs a golden dataset of synthetic scenarios (carve-outs, authorization-required services, out-of-network providers, terminated coverage) on every change to a prompt, rubric, or model, and blocks deployment on regressions. Offline rechecks use a code-based evaluator to re-run numeric fidelity across historical sessions after a rule change. It stays out of the approval path by design.

Ground truth comes from people

  • Escalated turns. The human records whether the rejected draft was wrong, partially wrong, or correct. This measures the false-fail rate.
  • Delivered turns. Benefits QA audits a stratified sample of passed answers. This measures the false-pass rate, the error that matters most, and it’s the only way to see it, because a wrong answer that was delivered never triggers an escalation. Repeat contacts about the same service code within a short window are a useful proxy between audits.

Calibrating the gate

With labeled outcomes, treat each Judge dimension as a classifier score. Bucket the scores, plot observed correctness per bucket to see how far the self-reported numbers sit from calibrated, and choose thresholds by business cost: minimize false passes subject to an escalation rate the contact center can staff. Repeat whenever the Judge model, rubric, or Specialist changes, compare candidate Judge versions on the same labeled set before promoting one, and store thresholds as versioned configuration so every verdict traces back to the thresholds that judged it.

What to measure

For system quality: false-pass and false-fail rates, Judge invalid-output rate, evidence-check failures by category, escalation rate, p50, p95, and p99 latency per graph node, and tokens per turn by agent. For the business: self-service completion (no transfer and no repeat contact), transfer rate, repeat-information rate after handoff, handle time for escalated contacts against a pre-AI baseline, and caller corrections. None of these is an improvement until it’s measured against that baseline.

Alternatives and tradeoffs

Strands Graph instead of AWS Step Functions

Why it fits: a turn completes in seconds, in one session and one process, and needs node timeouts, conditional edges, and per-node spans; adding a domain specialist means adding nodes. A strictly linear pipeline could also be plain Python. The Graph earns its place through explicit structure, timeouts, tracing, and extensibility.

When I’d choose the alternative: Step Functions fits long waits such as mid-process human approval, coordination of many services that need durable state transitions and per-state retries, or a visual execution history for audit.

A synchronous Judge instead of after-the-fact evaluation only

Why it fits: a second model call adds latency and cost, but it’s the only option that prevents delivery, and cost-share and authorization answers justify that price.

When I’d choose the alternative: for low-stakes intents such as office hours or ID-card requests, deterministic checks plus asynchronous evaluation are enough. Routing by intent class lets one system use both.

Escalation instead of regenerate-on-fail

Why it fits: regeneration doubles latency and invites judge shopping, where drafts are regenerated until one slips past the Judge’s blind spots.

When I’d choose the alternative: on chat, one bounded retry can be reasonable when the failure reason is mechanical, such as a missing authorization statement whose evidence is already in the bundle, and the retry passes through the complete gate again.

Storage choices

DynamoDB fits the handoff metadata because the dominant access is a single-item read by key and the state machine relies on conditional writes; Amazon Aurora fits only if handoffs must join relationally with queues, agents, and cases in an existing operational database. S3, rather than AgentCore Memory, holds the record because Memory is designed to give agents context, while the record needs versioning, retention controls, optional Object Lock, scoped access for non-agent applications, and analytics through Amazon Athena.

Production considerations

Latency, with voice as the hard case

The synchronous path is serial: tool calls, Specialist generation, then Judge generation, and voice callers notice silence. Beyond the deterministic short-circuit from Part 1, the main levers:

  • A smaller, faster Judge model is worth testing, provided it holds accuracy on your labeled set.
  • Amazon Bedrock prompt caching reduces latency and input-token cost for static prefixes such as the Judge’s rubric and the Specialist’s instructions and tool definitions. Cached sections must stay identical between requests, and each model has a minimum token count per cache checkpoint.
  • Trim the evidence bundle to the fields the claims reference.
  • On voice, play a pre-approved holding phrase (“Let me check the plan details”) while the turn runs. Never stream a partial model answer, because nothing model-generated may reach the caller before the gate.
  • Warm structured-output schemas in the deployment pipeline. First-time grammar compilation for a new schema can take up to about 90 seconds, and compiled grammars are cached for 24 hours, so a schema change deployed straight into live traffic produces a latency spike.

Throughput and quotas

Two model invocations per turn consume roughly twice the request and token quota of a single-agent design, so size Bedrock quotas for peak volume including the Judge. Cross-Region inference can add throughput, but confirm data residency and BAA coverage for every Region involved. DynamoDB writes happen only on escalation, and S3 receives a few small writes per turn, spread by conversation ID.

Security and PHI

  • Least privilege per role. The Specialist gets read-only Gateway tools under default-deny policies, the Judge gets no tools, model permissions are scoped to specific model ARNs, and the desktop role reads only the handoff prefix.
  • Member scoping at the tool boundary. Where the Gateway’s inbound identity carries the verified member as a claim, write policies that reject lookups for any other member, so a confused or manipulated Specialist can’t query someone else’s benefits.
  • Minimum necessary. Tool Lambdas return only the fields an answer needs, and Guardrails sensitive information filters mask identifiers that should never appear in a response.
  • Indirect prompt injection. Exception notes and case history are free text. Delimit them as data, apply Guardrails prompt-attack filtering where appropriate, and keep Gateway policy as the backstop.
  • Encryption, network, and audit. Use SSE-KMS with a customer managed key and S3 Bucket Keys, a TLS-only bucket policy, Block Public Access, a customer managed key for DynamoDB, and VPC endpoints for S3, DynamoDB, and Bedrock. Enable CloudTrail data events on the archive bucket so you can answer who read which transcript.

Observability

AgentCore Observability emits OpenTelemetry spans to CloudWatch and surfaces them in the GenAI Observability dashboard after a one-time Transaction Search setup per account. Add custom metrics for gate outcome by reason, Judge invalid output, evidence failures by category, PENDING age, reconciler actions, and cost per turn, and write the traceId into every event and handoff record so any escalation traces back to its spans. Alarm on escalation rate against staffing capacity, Judge invalid-output rate, and PENDING age, and review the QA false-pass rate weekly.

Cost

Input tokens dominate. Tool results go into the Specialist and again into the Judge, so evidence is paid for twice, which makes trimming and prompt caching the biggest levers. The other drivers are the Judge model tier, AgentCore Runtime active session time, AgentCore Evaluations (the online sampling rate is a direct cost dial), Guardrails text units, and CloudWatch span ingestion. S3 and DynamoDB are small by comparison. And every false fail is a paid agent minute, so Judge strictness is a cost lever as much as a quality lever.

Final thoughts

Go back to the provider asking about an outpatient MRI. With this architecture, they hear one of two things: the $150 copay at a freestanding facility along with the authorization requirement, or a handoff to a specialist who can already see what was asked, what was checked, what the AI nearly said, and why it was stopped, and a durable record that lets a human start exactly where the AI stopped.

Official AWS references

Amazon Bedrock
Amazon Bedrock Agentcore
Agentcore Gateway
Aws S3
Dynamodb

--

--

Ambuj
Ambuj

Written by Ambuj

0 followers
1 following