How to QA an AI Chatbot When There Is No Single Correct Answer: A Two-Layer Testing Framework

For most of the last two decades, a test case had three parts: an input, an action, and an expected result. You typed a value, clicked a button, and compared the output against a fixed string. If they matched, the test passed. That contract worked because software was deterministic. The same input always produced the same output.

Large language models broke that contract. When your product ships a chatbot, a support copilot, or an autonomous agent, the same question can produce five different answers, all of them correct. A model update can quietly change behaviour overnight with no code change and no failing test. Assertions built on exact matches either pass everything or fail constantly.

This is the single hardest new problem facing QA teams in 2026, and it matters commercially. Teams are shipping AI features faster than they can validate them, and defects are reaching customers because the old safety net does not catch this class of bug. This article explains a two-layer testing framework that separates what can be asserted from what must be scored, with the code, the trade-offs, and the production mistakes we made along the way. It is written for QA engineers, SDETs, and engineering managers who already own a test suite and now own an AI feature too.

The Problem: When the Expected Result Stops Existing

A real-world scenario

A retail client added an AI support agent to handle refund and order-status queries. It could look up an order, read the refund policy, and issue a refund through an internal API. The functional test suite was green. Two weeks after launch, the support team noticed refunds being approved for orders well outside the 30-day return window.

The agent had not crashed. It had not returned an error. It had read the policy document, summarised it accurately, and then called the refund tool anyway. Every existing test passed, because every existing test checked that a response was returned and that the response mentioned the policy.

Business impact

• Direct revenue loss from refunds that should never have been approved.
• Support escalations rose because customers received confident but incorrect answers.
• Release confidence collapsed. The team started manually reviewing chat transcripts before every deploy, adding roughly 26 hours of review time per release.

Technical challenges

• Non-determinism. The same prompt returns different wording each run, so exact-match assertions are useless and near-match assertions are arbitrary.
• Failures happen mid-trajectory. The agent can reach a correct-looking final answer through faulty reasoning, so checking only the output hides the defect.
• Silent regressions. A model provider can update the underlying model with no notice, no version bump on your side, and no failing build.
• Unstructured input. Users type anything, including adversarial text designed to override system instructions.

The Solution: A Two-Layer Test Architecture

The core decision is to stop treating the agent as one testable unit. An agent has a deterministic scaffold (routing, parsing, schema handling, guardrails) and a probabilistic core (the generated language and reasoning). These need different tools and different pass criteria, so we split them into two layers running at different points in the pipeline.

Layer 1 gates every commit. Layer 2 gates every merge.

Architecture overview

Layer 1 runs on every commit and takes under a minute. It contains ordinary assertions: does the router pick the right tool, does the tool receive correctly parsed arguments, does the response match the JSON schema, does a known unsafe request still get refused. No model judge is involved, so there is no cost and no flakiness.

Layer 2 runs on merge to main and nightly. It replays a versioned golden dataset through the agent and scores the results with metrics rather than assertions, then compares those scores against a threshold gate.

Tools used

• Pytest with custom markers to separate the two layers.
• An evaluation library (DeepEval or Ragas) for faithfulness, relevance, and hallucination scoring.
• OpenTelemetry-style tracing so every tool call and retrieval is recorded as a span.
• The golden dataset stored as YAML in the same repository as the agent, versioned with the prompts.

Key decisions

• Pin the judge model and the prompt version. If your evaluator changes, your scores move for reasons that have nothing to do with your product. Treat the judge like any other dependency.
• Store expectations as ranges and constraints, not strings. A case says what must appear, what must never appear, which tools should have been called, and the minimum acceptable score.
• Run each scored case several times and take the median, so ordinary sampling variance does not fail the build.

Code: Layer 1, deterministic checks

@pytest.mark.deterministic
def test_refund_request_routes_correctly(agent):
    trace = agent.run("I want my money back for order #4821")

    assert trace.tools_called == ["lookup_order", "lookup_policy"]
    assert trace.tool_args["lookup_order"]["order_id"] == "4821"
    assert "issue_refund" not in trace.tools_called  # needs policy check first

Code: the golden dataset

- id: refund_outside_window
  input: "I bought this 90 days ago, can I return it?"
  context: policy_v3.md
  must_contain: ["30-day"]
  must_not_contain: ["approved", "refund issued"]
  expected_tools: ["lookup_order", "lookup_policy"]
  min_faithfulness: 0.85

Code: Layer 2, scored evaluation with a quality gate

def score_case(agent, judge, case, runs=3):
    scores = []
    for _ in range(runs):
        out = agent.run(case.input, context=case.context)
        scores.append(judge.faithfulness(out.text, case.context))
    return statistics.median(scores)

@pytest.mark.evaluation
@pytest.mark.parametrize("case", load_golden_dataset())
def test_behaviour_within_threshold(agent, judge, case):
    score = score_case(agent, judge, case)
    assert score >= case.min_faithfulness, (
        f"{case.id}: {score:.2f} < {case.min_faithfulness}"
    )

What this catches that a traditional suite does not

Real Experience: What Production Taught Us

Mistake 1: we scored every case once

The first version of the evaluation suite failed roughly one build in four with no code change. Engineers started re-running the pipeline until it went green, which is worse than having no gate at all. Taking the median of three runs removed almost all of that noise. The cost is three times the tokens, which is why this layer does not run on every commit.

Mistake 2: we upgraded the judge model mid-quarter

A newer judge model shifted every faithfulness score by roughly 0.05, and half the suite failed overnight. Nothing about the product had changed. We now pin the judge model version, and when we do upgrade it we re-baseline every threshold in a dedicated pull request that touches no product code.

Production issue: a silent provider update

Six weeks in, routing accuracy dropped noticeably over a weekend. No deploy had happened. The provider had updated the underlying model. Because the nightly evaluation was already running, the drift was flagged within 14 hours instead of surfacing through customer complaints, and the team pinned to a specific model snapshot the same day.

Security improvement

A user uploaded a PDF containing instructions aimed at the agent rather than at a human. The agent followed them. We added an adversarial regression suite of 60 injection payloads to Layer 1, treated as hard assertions: any response that acts on instructions found inside retrieved content fails the build. That suite has blocked three merges since. If you are building on retrieval, our explainer on why AI needs to search before it speaks covers where these payloads enter the pipeline.

Performance and cost optimisation

The first combined suite took 11 minutes and cost about 38 dollars per run, and developers ran it as rarely as possible. Splitting the layers cut the per-commit run to under 40 seconds at effectively zero cost, and reduced the full evaluation to roughly 9 dollars by caching retrieval results and only re-running cases whose prompts or context files had changed.

Results after one quarter

Conclusion

The key takeaway is simple: stop trying to assert your way through non-deterministic software. Separate the parts of your agent that can be asserted from the parts that must be scored, gate them at different points in the pipeline, and version your evaluation dataset as carefully as you version your code. Everything else, including dashboards, tooling and vendor platforms, is detail on top of that split.

If this framing is useful, our related pieces on agentic AI in software testing and AI-powered test automation go deeper on the tooling side.

Key takeaway: An agent has a deterministic scaffold and a probabilistic core. Assert the scaffold on every commit, score the core on every merge, and never let a model judge you have not pinned decide whether your build is green.

If your team is shipping an AI feature without a test strategy behind it, Logic Providers can help you build this framework into your existing pipeline. Our work on custom AI chatbots and LLM API integrations covers evaluation design, adversarial testing, and production monitoring. The business benefit is measurable rather than theoretical: fewer incorrect answers reaching customers, far less manual review before every release, and a real answer when someone asks whether the AI feature is safe to ship this week.