Testing DEL + Laya on watsonx Orchestrate — From Mac Mini to the Cloud

This is the sixth post in the conversational triage series. The previous five are: “Configuration Driven Interactions”, “Agentic Triaging (or Dynamic Data Collection)”, “Adding memory when calling Agents from Assistants”, “From Rules to Epistemic Logic: Rethinking Conversational Triage” and “Replacing the LLM Intent Classifier with Laya”.

Following on from my previous experiments I wanted to prove the DEL + Laya approach within watsonx Orchestrate (wxO). After all, there is a moment in most experiments where you need to stop writing code and start asking whether the thing actually works end to end :-) This post is about that moment.

What I wanted to test is the combination of the three ideas that have developed across the series:

• The DEL epistemic planner — a deterministic model that decides which question to ask next by information gain, compiled from a plain-English journey document
• Laya — a non-autoregressive fast encoder that classifies intent in ~33 ms on GPU (MPS), replacing the per-turn LLM classification call
• Configuration driven conversational triage — watsonx Orchestrate as the agentic orchestration layer that wraps the whole thing in a proper agent and provides the conversational surface driven from a more declarative approach

The goal of this iteration was (with IBM Bob’s support) to put all three together, run them against a real wxO tenant, and validate that the integration holds up under the full test suite. Along the way there were two significant engineering problems to solve: choosing the right LLM for local fact extraction, and decoupling date extraction from date calculation so that calendar math is entirely deterministic and the LLM performance is optimised.

What this experiment is trying to show

The central claim running through this series is that conversational triage does not need to be probabilistic end to end. The LLM is genuinely necessary at the language boundary — somewhere you have to convert messy natural language into structured facts — but once a fact is extracted, everything downstream can be deterministic. The DEL planner decides what to ask next without an LLM. The epistemic state machine tracks what is known and what is not. The routing conditions are evaluated against known propositions, not inferred.

Laya’s role in this picture is to push the language boundary inward. Before Laya, the per-turn LLM classification call added 400–800 ms of non-deterministic latency to every single turn. Laya does the same job — classifying whether the customer is answering, repairing, revising, or going off-topic — in a single forward pass with no text generation, no sampling, and no hallucination risk. The LLM is then only invoked when Laya’s presence gate determines that a value actually exists in the utterance and needs extracting from natural-language prose.

The experiment aimed to demonstrate:

1. That the integration between wxO, ncf-service, Laya, and the AstraDB memory layer is solid enough to run a complete triage journey from greeting to routed outcome
2. That the Laya intent classifier correctly handles the full range of turn types — domain answers, repairs, revisions, out-of-scope utterances — without a live model checkpoint during tests
3. That the presence gate correctly suppresses LLM extractor calls when the utterance does not contain a value
4. That the convux wxO agent maintains session state correctly across turns, including through corrections and ambiguous answers
5. That date-based propositions are handled reliably by using the LLM strictly to locate date entities and deterministic code to compute policy windows
6. That all of this works reliably enough to produce 107 passing tests across four test layers

The deployment model — Mac mini backend with ngrok

The test environment I chose was a local deployment model, running the backend services on a combination of a Mac mini (for inferencing workloads) and a Macbook pro for the containerised backend services. All accessible from wxO running in IBM Cloud via ngrok, used expose the ncf-service over a stable public HTTPS URL.

The architecture looks like this:

Architecture Overview

Layer & Component Details

☁️ Cloud Layer (IBM Cloud / wxO SaaS)

• watsonx Orchestrate (wxO): Runs the conversational orchestrator agent (convux / del-triage).
• ncf_handler Tool: Custom Python ADK tool registered with wxO. Configured with a static connection string (https://<ngrok-domain>|<api-key>) which enables wxO to dispatch conversation turns across the internet to the local stack without requiring OpenShift Route reconfigurations.

🌐 Ingress & Tunnel (Ngrok)

• Static Domain: A permanent free static domain providing stable TLS ingress.
• Ngrok Web Inspector: Accessible locally at http://localhost:4040 to monitor, inspect, and replay live HTTP payloads exchanged between wxO and ncf-service.

💻 MacBook Pro (Podman Compose Environment)

• ncf-service (Port 8001:8000): Core Next Contact Flow orchestration service running inside a Podman container. Evaluates intent classifications, determines next steps, orchestrates memory storage, calls fact extraction, and applies business rules.
• memory-service (Port 8080:8080): Containerized ChromaDB service storing conversation turns, thread context, and multi-turn state.

🖥️ Mac Mini (Host Services)

• LM Studio (Port 1234): Serves LLM locally with Metal GPU acceleration. Exposes an OpenAI-compatible endpoint (/v1/chat/completions) for fact extraction and prose-to-entity parsing (e.g., converting informal amounts like "about four hundred pounds" into numerical facts).
• laya-service (Port 8000): Runs directly on the Mac Mini host with Apple Silicon MPS (Metal Performance Shaders) acceleration for low-latency rule evaluation and triage classification.

Network & Port Allocation Table

The key difference from a generic local dev setup is that laya-service runs directly on the Mac mini host (not inside a container), using Apple Silicon MPS acceleration. The Podman Compose file reflects this — ncf-service and memory-service run in containers, while LAYA_SERVICE_URL in .env.local points at the host IP (x.x.x.x:8000) of the Mac mini. Similarly, LM Studio runs on the Mac mini host and is reachable at x.x.x.x:1234 from inside the containers.

Why ngrok? Three alternatives were considered: a Python long-poll service, MQTT, and Cloudflare Tunnel. ngrok wins on simplicity. There is no broker, no extra service, no change to the wxO tool or agent. wxO calls the ngrok URL exactly as it would call an OpenShift Route. The free tier includes one permanent static domain, which means the wxO connection credential is set once and never changes. The built-in web inspector at http://localhost:4040 is genuinely useful — you can replay any wxO → ncf-service request directly from the browser, which saved a lot of debugging time during development.

What stays cloud-hosted: The wxO SaaS tenant, and DataStax AstraDB for session memory. These are the two components that genuinely benefit from being in the cloud — wxO because that is where the agent lives, and AstraDB because the sessions need to survive between test runs.

The entire stack starts with a single script:

./local-dev/start-local.sh

This builds the Podman Compose images, starts the two containerised services, waits for health checks on all ports, and then starts the ngrok tunnel. At the end it prints the wxO connection credential — the <base_url>|<api_key> string that goes into the wxO connection. Because the ngrok static domain never changes, this configuration step happens once.

The model problem — and what was done about it

The original plan used Granite 4.2 8B for local fact extraction via LM Studio on the Mac mini. Early testing revealed this was a not ideal for my compute set up.

Granite 4.2 8B is a chain-of-thought reasoning model. It always runs 200–900 thinking tokens silently before outputting, even when thinking: false is passed in the request body — that parameter is silently ignored. The measured throughput was ~25 tokens/second, with individual extraction calls taking 6–40 seconds. A single triage conversation passing through five LLM extraction calls could easily take 3–5 minutes. The full E2E live-route test suite was projected at around 45 minutes.

The fix was to switch to Granite 3.3 8B Instruct — a dense non-reasoning model. The measured throughput jumped to ~100 tokens/second, with calls consistently under 1 second. The full E2E live-route test suite now runs in 4m 22s.

The change was minimal: update WATSONX_AI_MODEL=granite-3.3-8b-instruct in .env.local, remove the thinking: false / extra_body parameters that Granite 3.3 doesn't accept, and reduce max_tokens from 1024 to 256 (prompts never exceed 200 tokens, and the model outputs just the structured JSON without any preamble). One minor Granite 3.3 quirk required a code fix: the model occasionally returns the string "null" as a JSON value rather than JSON null, which needed special-casing in _parse_response.

Before (Granite 4.2 8B reasoning model):
  Throughput: ~25 tok/s
  Per-call latency: 6–40s
  Full E2E suite: ~45 minutes

After (Granite 3.3 8B instruct model):
  Throughput: ~100 tok/s
  Per-call latency: <1s
  Full E2E suite: 4m 22s

The system prompt was also tightened: "extract ONLY if explicitly stated, do NOT infer". This matters for Granite 3.3, which is well-behaved but will sometimes infer active_policy: yes from the tone of an optimistic opening statement if not explicitly constrained. The guard is important for maintaining the epistemic contract — the planner should only be told a proposition is known when the customer actually stated it.

Moving date calculations out of the LLM

A key architectural refinement in this iteration was fixing how incident_recency was evaluated by establishing a strict separation of concerns: the LLM locates the date entity in conversational text, and deterministic code performs the date calculations and window classification.

incident_recency is an enum proposition with two policy outcomes: within_window (0–365 days ago) and out_of_window (more than 365 days ago). Previously, the system asked the LLM to perform both entity extraction and temporal reasoning in a single prompt:

1. Laya’s presence gate: Does this utterance contain a date? (Yes)
2. LLM extractor: Extract the date expression from the utterance → "about three weeks ago"
3. LLM classifier: Calculate the calendar delta relative to today and classify into within_window or out_of_window

Step 3 was the failure point. Asking an LLM to perform calendar arithmetic and evaluate whether “three weeks ago” falls within a rolling 365-day boundary is asking language models to do what they are worst at: reliable math. With reasoning models, it incurred unnecessary token overhead; with smaller instruction models like Granite 3.3, it led to arithmetic edge-case errors or hallucinations when evaluating non-date text.

The fix was to delegate all calendar math and window classification to Python code via two pure functions: parse_relative_date() and apply_date_to_enum().

Locating the date entity

The system uses a layered strategy to extract the raw date:

• Deterministic pattern fast-path (parse_relative_date): Resolves common natural-language date phrases directly without an LLM call. It covers exact keywords (today, yesterday, last year), quantified relatives (3 weeks ago, 18 months ago), word numbers (three days ago), named dates (3rd June 2025), and ISO formats. A second pass strips conversational sentence openers (e.g. "it happened about three weeks ago" resolves directly to "three weeks ago").
• LLM extraction fallback: When conversational phrasing is complex or irregular, the LLM is called strictly as an entity locator. Its prompt instructs it to return only the extracted date string (or ISO date), never to reason about policy windows or outcome options.

Deterministic window calculation (apply_date_to_enum)

Once a date is located (by regex fast-path or LLM extraction), Python code computes the exact elapsed days ((today - parsed_date).days) and evaluates it against the proposition's option_windows schema defined in the journey model:

"option_windows": [
  { "option": "within_window", "days_ago_min": 0, "days_ago_max": 365 },
  { "option": "out_of_window", "days_ago_min": 366 }
]

Windows are evaluated sequentially with deterministic bounds checking. If no date can be located in the utterance, the system fails cleanly and triggers a conversational repair turn rather than hallucinating an arbitrary window.

This separation reflects the core thesis of this architecture: use the LLM only at the boundary to extract raw language entities, and keep all reasoning, calculations, and state transitions strictly in deterministic code.

Configuring watsonx Orchestrate

The wxO assets for this project are minimal. There are two tools and one agent.

The ncf_handler tool

The tool that does the work is ncf_handler_tool.py. It takes three arguments: utterance (the customer's latest message), journey_name (the triage domain, insurance_claim in this case), and session_id (empty on the first call, then passed back on every subsequent call).

On the first call it hits POST /session to create a fresh epistemic state. On subsequent calls it hits POST /session/{session_id}/turn. In both cases it returns a NcfHandlerResult with three fields: session_id, next_step, and goal_satisfied.

The credential convention is worth noting. wxO's connection model requires a single credential value, but the tool needs both a base URL and an API key. These are packed as <base_url>|<api_key> and split at runtime:

parts = conn.api_key.split("|", 1)
ncf_base_url = parts[0].rstrip("/")
api_key = parts[1] if len(parts) > 1 else ""

This means the connection credential in the wxO UI is set to https://<ngrok-domain>|<NCF_API_KEY>, and the tool handles the split internally. It is a minor convention but it means zero changes are needed to the wxO connection when switching from local (ngrok URL) to production (ROKS Route URL).

The convux agent

The agent is straightforward. Here is the full YAML:

spec_version: v1
kind: native
name: convux
display_name: Conversational Triage Agent
llm: watsonx/meta-llama/llama-3-3-70b-instruct
style: react_intrinsic
tools:
  - ncf_handler
  - execute_triage_action

The instructions are the interesting part:

1. On the first user message, call ncf_handler with the user's utterance,
   journey_name="insurance_claim", and an empty session_id.
2. Read the next_step from the result. If next_step.type is "question",
   ask the user next_step.question_text exactly — do not paraphrase.
3. On each subsequent user message, call ncf_handler again, passing the
   utterance and the session_id returned by the previous call.
4. Repeat until goal_satisfied=true.
5. When goal_satisfied=true, call execute_triage_action with the action_name
   and collected_data from the final result.
6. Present the ActionResult summary to the user in plain language and close the conversation.

The critical instruction is number two: ask next_step.question_text exactly — do not paraphrase. The DEL planner has specific question wording for each proposition, including repair variants and clarifications. If the LLM paraphrases the question, that information is lost. The react_intrinsic style with temperature: 0 generally respects this, though it occasionally generates a framing sentence before the question text, which the test suite accepts as a pass since the question itself is still present.

Importing the assets

The import order matters. Connection first, then tools, then agent:

orchestrate connections import -f wxo/connections/ncf_service_conn.yaml
orchestrate tools import -k python -f wxo/tools/ncf_handler_tool.py \
  --app-id ncf_service
orchestrate tools import -k python -f wxo/tools/action_tool.py
orchestrate agents import -f wxo/agents/convux.yaml

The connection must exist before the tool is imported because the tool references it via app_id = "ncf_service". After import, the connection credential is set in the wxO UI (Connections → ncf_service → enter the <ngrok-url>|<api-key> string). This is the one manual step, and with a static ngrok domain it only happens once.

The DEL model and how it maps to Laya

The DEL model for this experiment is the insurance_claim journey. It has five propositions:

The planner evaluates propositions by information gain — which unknown proposition, if resolved, would eliminate the most possible outcome conditions? The model has terminal conditions including immediate exits for no active policy, early exits for travel / personal injury / liability claims (routing straight to the specialist team regardless of loss band), and the fasttrack path for low-value theft and property claims.

How Laya fits into this model:

For every customer turn, the planner needs to know two things before it can advance: what kind of turn was this (intent), and if it was a domain_fact intent, what value was provided?

For propositions with enum or boolean types — active_policy, claim_type, claim_reference_confirmed — the answer is usually one of a small set of known strings. A fast-path checks whether the utterance exactly matches one of the valid options or a synonym before calling Laya at all. "Travel", "theft", "yes", "yeah" all resolve at confidence 1.0 without a model call.

For incident_recency, the new deterministic date fast-path runs first. If parse_relative_date() succeeds, apply_date_to_enum() maps the date directly to within_window or out_of_window with no LLM involvement at all.

For the partition type (loss_band), no exact match is possible — the customer says "about 300 pounds", not "low". The fast-path explicitly returns None for partition props, so Laya's presence gate always runs. A single noul question asks: "does this utterance contain a numeric amount?" If yes (noul ≥ 0.65), the LLM extractor is called to parse the amount and map it to a band. If no, the planner treats the turn as a repair and re-asks.

The Laya intent classifier handles the broader case: the six intents it distinguishes are domain_fact, revision, repair_repeat, repair_clarify, repair_example, and out_of_scope. There is also an independent noul question for is_correction — this handles the ambiguous case where the customer's utterance is classified as domain_fact but also explicitly corrects something said earlier ("actually I meant travel"). When domain_fact is the top choice and the is_correction noul score exceeds 0.75, the result is upgraded to revision. The DEL planner then retracts the previously announced value and re-scores information gain with the new fact.

The per-turn pipeline now looks like this:

utterance
    │
    ▼
fast-path (~0ms): exact enum/boolean match?
    ├── hit  → domain_fact at confidence 1.0 (no Laya, no LLM)
    └── miss
         │
         ▼ for date-enum propositions (e.g. incident_recency)
         parse_relative_date() + apply_date_to_enum() (~0ms)
            ├── parsed → domain_fact at confidence 1.0 (no Laya, no LLM)
            └── no date found
                 │
                 ▼
                 Laya classify (~33ms MPS / ~200ms CPU)
                    │
                    ▼ if domain_fact
                    Laya presence gate (~33ms MPS / ~200ms CPU): value present?
                        ├── absent  → repair (no LLM call)
                        └── present → LLM extract (~300-800ms, Granite 3.3 8B)

Testing the approach

Working with Bob, I created a test suite that runs in four layers. With a full count of test totalling 107 tests.

Modelled conversation flow

The image below is generated from the domain model by running the real planner over every answer combination, so it shows what the service will actually ask.

Domain: insurance_claim

Blue boxes are questions the planner asks (data type below the name); purple boxes are terminal routing actions, listing the answers that select them. Arrow labels show the answer that takes a branch (no label = any answer). Grey dashed = part of the model but never selected by the planner on a fresh session (zero information gain), so only its exit routes are drawn.

Routing decisions

Test Run Summary

All 107 tests passed. 21 full wxO end-to-end conversations reached a routed terminal action. 24 live-route conversations drove the planner directly via the HTTP API with the full stack running.

Layer 1 — Unit tests (no conversation)

The 38 unit tests cover LayaIntentClassifier, LayaPresenceGate, and LayaPrior. All Laya model calls are mocked so no checkpoint download or GPU is needed.

A few of the more interesting test cases:

Get Tony Hickman’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The fast-path tests confirm that Laya’s Router.predict() is never called when the utterance exactly matches a valid option or a known boolean synonym. For claim_type = "travel" or active_policy = "yeah", the classifier returns domain_fact at confidence 1.0 without a model call. This is the most common path in real usage and it is important to verify the skip is unconditional.

def test_exact_enum_match_no_laya_call():
    classifier = LayaIntentClassifier()
    mock_router = MagicMock()
    classifier._router = mock_router

    result = classifier.classify(
        utterance="travel",
        current_proposition="claim_type",
        question_text="What type of claim would you like to raise?",
        valid_options=["property_damage", "theft", "travel"],
    )
    assert result.intent == "domain_fact"
    assert result.confidence == pytest.approx(1.0)
    mock_router.predict.assert_not_called()

The revision upgrade test confirms that domain_fact + is_correction.noul >= 0.75 → revision. This is the path that handles "actually I made a mistake earlier":

def test_laya_revision_upgrade():
    laya_result = _laya_answers("domain_fact", 0.85, 0.82)
    classifier, _ = _make_classifier_with_mock(laya_result)

    result = classifier.classify(
        utterance="actually I made a mistake earlier, it wasn't that",
        current_proposition="claim_type",
        ...
    )
    assert result.intent == "revision"
    assert result.confidence == pytest.approx(0.85)

The fail-safe tests confirm that if Laya’s router raises a RuntimeError at inference time, the classifier returns a domain_fact fallback at confidence 0.0 rather than propagating the exception. The presence gate has the same pattern: on error it returns present=True (fail open — let the extractor run rather than silently discarding a value that might be there).

The presence gate batch call test confirms that the extract_opening_utterance function makes a single batch predict call covering all non-enum, non-boolean propositions in one Laya round-trip rather than one call per proposition.

Layer 2 — Integration tests

The 24 integration tests in test_ncf_turn.py and test_memory.py cover the service boundaries using the FastAPI TestClient with mocked backends. No live stack is needed.

Key tests here include the repair-count persistence test — repair counters are held in the epistemic state, which is persisted to AstraDB after every turn. If the serialisation drops the repair_counts field, the escalation logic can never fire — the counter resets to zero on every load. The test simulates a save-load round trip through a mocked memory service and verifies the counter survives:

def test_repair_counts_survive_a_save_load_round_trip(monkeypatch):
    state = EpistemicState(session_id="s1")
    state.set_current_proposition("claim_type")
    state.increment_repair("claim_type")
    session_mod.save_session("u1", "insurance_claim", state)

    assert docs["u1"]["short_term_memory"]["insurance_claim"]["repair_counts"] == {"claim_type": 1}
    restored = session_mod.load_session("u1", "insurance_claim", INSURANCE_MODEL, session_id="s1")
    assert restored.get_repair_count("claim_type") == 1

The integration tests also cover the 503 propagation case: a memory service outage should be surfaced to the caller rather than silently swallowed.

Layer 3 — ncf-service live routes

The 24 live-route tests in test_live_routes.py are the structural backbone of the test suite. They drive real multi-turn conversations through the running ncf-service stack (real intent classifier, real LM Studio extractor, real memory service) using direct HTTP calls — the same calls the wxO ncf_handler tool makes, but without the wxO orchestrator LLM in the loop. This makes them deterministic enough to always reach a terminal action.

The ROUTES table covers every reachable terminal action in the insurance_claim model — 14 routes across claim types, loss bands, time windows, and specialist team exits:

Notice the last two rows: both involve out_of_window but terminate differently. When the customer does not state their policy status, the active_policy proposition defaults to the model's prior and no_active_policy fires. When the customer explicitly says "yes I have an active policy" in the opening utterance, the bonus scan captures that value and eliminates the no_active_policy condition, so claim_out_of_time fires instead. The date phrases ("two years ago", "18 months ago", "yesterday") are all handled by parse_relative_date() without any LLM involvement — the entire incident_recency proposition is classified deterministically.

The TestRepairs class confirms the planner's behaviour under adversarial inputs: unclear opening answers are re-asked, off-topic utterances are redirected, and the important constraint-violation test confirms that an out-of-range amount (£2,000,000) is rejected with a repair_clarify and a constraint_violation field in the response rather than being silently accepted or causing an error.

The TestEscalation class confirms max_repair_attempts=2: two consecutive repairs on the same proposition escalate to refer_to_agent, and a valid answer between repairs resets the counter so only consecutive repairs count.

There is also a coverage guard:

def test_every_reachable_terminal_action_is_covered(self):
    """Guard: ROUTES must cover every action the planner can reach from a fresh session."""
    graph = flowdiagram.get_graph(JOURNEY)
    reachable = {b[2:] for (a, b) in graph["edges"] if b.startswith("t:")}
    covered = {action for _, _, action, _ in ROUTES}
    assert reachable <= covered, f"routes without a test: {sorted(reachable - covered)}"

If a new terminal action is added to the journey model and no test covers it, this guard fails immediately. This is the kind of test that saves you from shipping a triage path that was never validated end to end.

Layer 4 — wxO end-to-end tests

The 21 wxO end-to-end tests in test_convux_wxo.py are the real measure. They run against the deployed convux agent using the wxO runs API, with the ncf-service exposed over ngrok. Every conversation is driven to a terminal action — the test suite never skips a conversation because it stalled; it fails.

The test client is a minimal synchronous wrapper around the wxO POST /v1/orchestrate/runs endpoint. It polls for completion, parses the step_history from the result to extract tool calls and tool responses, and surfaces goal_satisfied from the last ncf_handler response. Token from the active orchestrate environment is loaded automatically.

The tests are organised into five classes:

TestSessionStart — Seven tests that share a single conversation (the opening message "I need to make an insurance claim" is sent once). The class verifies: ncf_handler was called, a non-empty session_id was returned, the first next_step.type is question, the agent's final message contains a question, goal_satisfied=False, turn=1, and the journey completes. Sharing the conversation means the wxO run is not repeated seven times.

TestStraightPathTheft — Three independent theft journeys, each driven to a terminal action. Verifies the terminal action name contains “theft” or “fasttrack”, and that session_id is consistent across all turns (checked on every turn by _finish_journey).

TestRepairPath — Three tests covering ambiguous answers, off-topic utterances, and the explicit check that goal_satisfied is not True after an ambiguous turn. With LAYA_PRESENCE_GATE enabled, the planner re-asks the same proposition. With the gate off (which the local Compose default uses for simplicity), the planner may advance — either is acceptable as long as it asks a question rather than terminating.

TestCorrection — Two tests covering the revision path. The first drives to claim_type, provides "it was a theft", then sends "actually I made a mistake — it was property damage, not theft". The agent must still be engaged (not terminated or confused), and the session_id must remain the same — no new session is created when a correction is processed. The second test drives this same scenario and explicitly asserts session_id stability.

TestMultiTurnContext — Three tests covering thread-level continuity. thread_id must be stable across turns. Turn 2 must carry session_id from turn 1 and have turn=2. Two independent messages must produce different thread_id and session_id values.

TestLlmExtraction — Three tests that deliberately bypass all fast-paths to exercise the LM Studio path. Prose amounts (“about four hundred pounds”, “roughly fifteen hundred pounds”) are sent for loss_band; a prose date phrase ("it happened about three weeks ago") is sent for incident_recency. The assertion in each case is that the journey advances past the proposition — confirming that the LLM parsed the value and the planner moved on.

Here is a representative conversation trace from the wxO test run (the opening journey used by TestSessionStart):

Five turns, four facts collected, one terminal action. Notice turn 4: the answer "within_window" is the exact enum label, so it fast-paths through the keyword matcher at confidence 1.0 with no Laya call and no LLM call. The planner correctly skipped active_policy on this run because the journey model's condition evaluation determined it was not needed to reach a terminal condition given what was already known.

Test Report: watsonx Orchestrate (wxO) Flows

wxO Architecture & Interaction Model

TestSessionStart — Session Initialization & Turn Zero Setup

Example Flow: claim_type → loss_band → incident_recency → active_policy → TERMINAL: fasttrack_theft

Tests Passed: 7/7 (test_ncf_handler_is_called, test_session_id_returned, test_first_response_is_a_question, test_first_question_presented_to_user, test_goal_not_satisfied_on_first_turn, test_turn_number_is_1, test_conversation_completes)

Behavior Verified:

1. The agent intercepts the opening utterance ("I need to make an insurance claim"), instantiates a session via ncf_handler(session_id=""), and returns a unique UUID session_id.
2. Initial state reflects goal_satisfied: false, turn: 1, and asks the first missing proposition: "What type of claim would you like to raise?" (claim_type).
3. Driving subsequent turns leads cleanly to goal completion.

TestStraightPathTheft — Happy Path Low-Value Theft Flow

Example Flow: loss_band → incident_recency → claim_reference_confirmed → TERMINAL: fasttrack_theft

Tests Passed: 3/3 (test_reaches_terminal_action, test_terminal_action_matches_theft_path, test_session_id_consistent_across_turns)

Terminal Action Reached: fasttrack_theft (FastTrack automated theft claim — low value)

Turn Sequence:

1. Turn 1 (User): “I need to make a theft claim”
   Extraction: claim_type = theft, active_policy = yes
   Agent Question: "What is your best estimate of the total value of the loss or damage in pounds?"
2. Turn 2 (User): “about 300 pounds”
   Extraction: loss_band = low
   Agent Question: "When did the incident occur?"
3. Turn 3 (User): “within_window”
   Extraction: incident_recency = within_window
   Agent Question: "Before I transfer you, a claim reference number will be issued and sent to you by SMS and email. Are you happy to proceed?"
4. Turn 4 (User): “yes”
   Extraction: claim_reference_confirmed = yes
   Planner Goal: fasttrack_theft (Automated resolution).

TestRepairPath — Ambiguity & Out-of-Scope Handling

Example Flow: claim_type → claim_type → loss_band → incident_recency → active_policy → TERMINAL: fasttrack_theft

Tests Passed: 3/3 (test_ambiguous_answer_triggers_re_ask, test_out_of_scope_answer_triggers_re_ask, test_goal_not_satisfied_after_ambiguous_turn)

Behavior Verified:

• Ambiguous Utterance: When the user replies with non-committal answers (e.g. “hmm, I’m not really sure about that” or “erm, maybe, I think so”), no proposition is extracted. The planner keeps goal_satisfied = false and re-prompts the customer for the pending proposition (claim_type).
• Out-of-Scope Utterance: When the customer asks unrelated questions (e.g. “what’s the weather like today?”), the agent recognizes the deviation, refrains from advancing the NCF state or corrupting memory, politely refocuses the user back to the claim triage, and resumes normal processing on the subsequent turn.

TestCorrection — Mid-Journey User Corrections & Fact Updating

Example Flow: claim_type → loss_band → loss_band → incident_recency → claim_reference_confirmed → TERMINAL: fasttrack_theft

Tests Passed: 2/2 (test_revised_answer_is_accepted, test_correction_does_not_duplicate_session)

Terminal Action Reached: fasttrack_theft / fasttrack_property

Behavior Verified:

• If a user previously stated “it was a theft” and then issues a correction (“actually I made a mistake — it was property damage, not theft”), the system updates the conversational context without fragmenting the session ID.
• Handles repeated repairs appropriately and completes routing to either fasttrack_property or fasttrack_theft.

TestMultiTurnContext — Context Continuity & Thread Isolation

Example Flow: claim_type → loss_band → incident_recency → claim_reference_confirmed → TERMINAL: fasttrack_theft

Tests Passed: 3/3 (test_thread_id_stable_across_turns, test_second_turn_advances_journey, test_new_thread_starts_fresh)

Behavior Verified:

• thread_id and session_id remain stable throughout 5+ conversational turns in a single thread.
• test_new_thread_starts_fresh initiates two completely separate wxO threads back-to-back; validates that new threads start with empty state, new session UUIDs, and zero state leakage between conversations.

TestLlmExtraction — Natural Language Entity & Value Parsing

Example Flow: claim_type → loss_band → incident_recency → claim_reference_confirmed → TERMINAL: fasttrack_theft

Tests Passed: 3/3 (test_loss_band_extracted_from_prose_amount, test_loss_band_mid_value_extracted, test_incident_recency_extracted_from_date_phrase)

Extraction Mappings Verified:

• Prose Value (£400): “about four hundred pounds” →→ loss_band = low →→ Routes to fasttrack_theft.
• Prose Value (£1500): “roughly fifteen hundred pounds” →→ loss_band = mid →→ Routes to standard_theft_handler (divergent business path based on claim value).
• Relative Time Expression: “it happened about three weeks ago” →→ incident_recency = within_window.

wxO Turn-by-Turn Journey Summary Matrix

Performance & Operational Metrics

• wxO Run Turn Latency: Average ~8,500 ms — 11,500 ms per turn (inclusive of wxO orchestrator LLM pass, custom tool REST round-trip, entity extraction, and response generation).
• HTTP Status Code: 100% 200 OK across all wxO run invocations.
• Journey Completion Rate: 100% (22/22 end-to-end conversations reached a definitive terminal action with no hung or aborted turns).

Outcomes and what comes next

107 tests, 107 passed. 21 complete wxO conversations reached a routed terminal action. The integration between the DEL planner, Laya, and watsonx Orchestrate holds up under the full test suite including multi-turn context, repairs, corrections, LLM extraction from prose, and the memory service boundary.

A few things I noticed during testing:

Switching from Granite 4.2 to 3.3 was the single highest-leverage change. The test suite going from ~45 minutes to 4 minutes was not a small quality-of-life improvement — it is the difference between a test suite you run before every commit and one you run once and forget. Non-reasoning models are the right choice when you have a structured extraction task with a tight prompt. Save reasoning models for problems that actually require it.

Moving date classification out of the LLM was worth the effort. incident_recency was the proposition that produced the most false classifications with the LLM approach — not because the model was wrong about relative dates, but because it occasionally produced a classification when no date was present in the utterance at all. The deterministic path fails cleanly (returns None, triggers a repair request) rather than hallucinating a value. The planner is more trustworthy because it cannot advance incident_recency from thin air.

The session_id stability check was valuable. The _finish_journey helper asserts on every turn that the session_id returned by ncf_handler matches the first turn's value. This caught a bug early in development where the tool was creating a new session on turns where the previous session_id was not being passed correctly. That assertion costs nothing and prevents a class of silent failures where the journey appears to work but each turn starts a new session.

The “agent occasionally does not call ncf_handler” case is real. The react_intrinsic style with a ReAct-style LLM occasionally produces a direct reply without calling the tool — usually on the first turn of a new thread, before the LLM has fully engaged with the instructions. The test suite handles this with a two-retry pattern that re-sends the last utterance.

The ngrok inspector is underrated. Being able to replay any wxO → ncf-service request directly from http://localhost:4040 without re-running the test saved a lot of time when debugging extraction failures.

Next: The session-opening prior (laya_prior.py, controlled by LAYA_ENABLE_PRIOR=1). When a customer opens with "I want to raise a travel claim for about £800", Laya can scan all propositions in a single batch call and produce probability distributions: claim_type scores high on travel, loss_band scores high on mid. These priors are used to boost the planner's information-gain scores, causing it to skip propositions whose values are already strongly implied. The deterministic ordering guarantee is preserved when the prior is disabled or low-confidence — this is an additive optimisation, not a replacement for the epistemic model.

Conclusions

This experiment started with a concrete question: does the DEL + Laya architecture hold up when it is no longer running in a controlled local environment, but is instead being driven by a real watsonx Orchestrate tenant over the public internet? The answer from the test suite is an unambiguous yes — 107 tests, four layers, 21 complete multi-turn journeys, zero regressions from the local baseline.

A few things feel settled after this iteration:

The deterministic core holds at the integration boundary. Every concern about non-determinism in this architecture lives at the language boundary — in Laya’s intent classification and in the LLM fact extraction step. Everything beyond that boundary: the DEL planner’s question ordering, the window calculations, the routing conditions, the session state — behaves identically whether the request arrives from a unit test or from a live wxO agent. That boundary is narrower than it looks, and keeping it narrow was the right design decision.

Testing the integration is the only honest test. The unit and mock layers caught bugs during development, but the wxO end-to-end layer is the one that caught the session_id regression, the retry behaviour on the first turn, and the extraction failures that only surface under real model responses. Layer 4 is not a luxury — it is where the architecture is actually validated.

The tooling investment pays off. The ngrok inspector, the _finish_journey assertion helper, and the structured four-layer test organisation made debugging fast and failure modes obvious. Time spent on test infrastructure was not time away from the core problem; it was time that made the core problem tractable.

What comes next

Additional flow models. The theft journey is one of the simpler triage paths in the DEL domain — a single claim type with a small proposition set and a clean routing tree. The next step is to run the same test harness against more complex flow models: journeys with branching sub-paths, conditional proposition sets, and multiple competing routing outcomes. The goal is to stress-test the planner’s information-gain ordering under more ambiguous starting conditions and to confirm that the Laya classifier generalises cleanly across journey types it was not explicitly tuned on.

End-to-end observability with OpenTelemetry. Right now, the visibility into a live conversation is good locally (ngrok replay, structured logs, the triage report) but opaque once the request leaves ngrok and enters the wxO → ncf-service → AstraDB chain under production traffic. The next engineering investment is instrumenting the full request path with OpenTelemetry: spans from the wxO agent invocation, through the ncf_handler tool call, into the Laya classification and LLM extraction steps, and out to the AstraDB session read/write. The aim is a single trace per conversation turn that makes latency attribution, extraction failures, and planner decisions visible in one place — across both the Mac Mini development environment and any cloud deployment target. This matters especially for the flow model expansion work: when a new journey fails in a subtle way, a complete trace is the difference between a five-minute debug session and a two-hour log dive.

The series has now established a working baseline: a deterministic epistemic planner, a fast intent classifier, a structured extraction boundary, and a validated cloud integration. The next posts will push on the edges of that baseline — more complex journeys, richer observability, and the session-opening prior that allows the planner to skip questions when the customer’s opening utterance has already answered them.