Traditional bugs crash as a symbol of failure. LLM bugs lie confidently. A null pointer
throws a stack trace you can grep for; a language model hands a customer a confident,
fluent, and completely fabricated refund policy and nothing in your CI pipeline blinks.
In 2024, Air Canada learned this the expensive way when a tribunal held the airline
liable for a policy its chatbot invented. The failure wasn't a crash. It was a plausible
sentence.
If you ship systems where AI is the product - chatbots, RAG pipelines, agents, your
assertion-based test suite is quietly lying to you. This article is about the framework I use to test these systems in regulated financial environments, where a wrong answer carries legal and financial consequences. I call it PROBE.
Why do your existing tests break
The instinct is to reach for the test you already know:
resp = chatbot.ask("What is the refund window?")
assert resp == "You have 30 days to request a refund."This passes Monday and fails Tuesday, same code, same model, same meaning, different wording. The test is now the problem. The failure isn't a bug in the model; it's a category error in the test. You've encoded a single point estimate as ground truth for a system whose output is a probability distribution over token sequences. Temperature, sampling, a silently updated model checkpoint, or a re-ranked retrieval context will all move that distribution, and your == will fail without anything being wrong.
Four properties make these failures fundamentally harder than traditional ones. The
signal is a plausible wrong answer instead of a loud crash, there's no exception to catch, so the failure is invisible to anything watching for thrown errors. Reproducibility is gone: the same input may not fail twice, which breaks the bisect-and-repro loop
debugging depends on. The failure space is unbounded natural language, you can't
enumerate the inputs, so coverage in the traditional sense is undefined. And the blast
radius is legal and reputational rather than functional, a single bad answer to the wrong
user is a headline, not a Jira ticket.
PROBE is five practices that replace exact-match testing with something that holds up
against non-determinism.
P Probabilistic Baseline
Stop testing for a single correct answer. Run the same prompt 20 to 100 times and
measure the distribution of outputs, then define a pass band instead of a pass value for
example, "95% of responses must stay on-policy."
Treat each run as a Bernoulli trial: on-policy or not. With n samples you're estimating the
true on-policy rate, and sample size sets your precision. At n=20 the 95% confidence
interval around an observed 95% is roughly ±9 points too loose to gate a release on; at
n=100 it narrows to about ±4. So the count isn't arbitrary: pick it from the precision you
need to distinguish "95% compliant" from "88% compliant," because in a regulated flow
that gap is the difference between pass and incident. Define the band against the lower
confidence bound, not the point estimate, so you fail safe.
The part teams forget is drift: today's 95% can quietly erode to 80% next month as
models, prompts, or knowledge bases change. Baseline the distribution, store it, and re-
run on every dependency change, a model version bump is a code change even when
your diff is empty. Monday-morning tool: promptfoo.
R Red Team It
Attack your own system before your users or an adversary do. Probe systematically for
prompt injection, jailbreaks, and policy bypass rather than relying on ad-hoc guesses.
The mechanics matter: direct injection overrides your system prompt in the user turn
("ignore previous instructions"), while indirect injection hides the payload in content the
model retrieves a poisoned document in your RAG corpus, a crafted web page an agent
browses so the attack rides in through data your pipeline trusts. The second class is the
one that bites RAG and agentic systems hardest, because the malicious instruction never appears in anything a human reviewed.
Anchor your attack catalog to established taxonomies - the OWASP LLM Top 10 and
MITRE ATLAS - so coverage is auditable rather than improvised, and so an auditor can
map your tests to a recognized framework. The discipline that pays off: treat every
successful attack as a regression test you keep forever. Your red-team corpus should only grow.
Monday-morning tools: DeepTeam; PromptArmor
O Observe in Production
You cannot pre-test an unbounded input space, so accept that production is your largest test set. Log real prompts, responses, latencies, and tool calls and for RAG, log the retrieved context alongside the answer, because a faithfulness failure is usually a
retrieval failure wearing a generation costume. You can't diagnose a hallucination if you
didn't capture what the model was given to work with.
Sample live traffic and score it continuously with the same evals you run in CI, so
production and pre-release signals are measured on one ruler. Critically, alert on
distribution shifts, not just hard errors, the dangerous failures here don't throw
exceptions, they drift. A faithfulness score sliding from 0.94 to 0.86 over two weeks is the real incident; no log line flags it unless you're watching the distribution.
B Behavioral Contracts
Write the rules in plain English, first, with your stakeholders in the room. Define explicit
MUST and MUST NOT statements: "MUST NOT quote a refund policy that isn't in the
knowledge base." These contracts become the specification your evals are written against — without them, "quality" is whatever the last person to look at the output decided it was.
The practical move is to make each contract measurable: a MUST NOT pairs with a rule-
based or model-graded check, a MUST pairs with a metric and a threshold. A contract
you can't express as an eval is a wish, not a spec. This is the cheapest practice on the list and the one that makes the other four meaningful.
Monday-morning tool: plain-English rules.
E Evals Not Assertions
Replace exact-match with graded quality. Score faithfulness, relevance, and safety rather than string equality, and combine rule-based and model-graded evals: rule based checks (regex, schema, banned-phrase lists) are deterministic and cheap but brittle, while model-graded checks catch semantic failures but carry their own variance so calibrate the grader against a human-labeled set before you trust it. Rules for the bright-line violations, model grading for the judgment calls.
Returning to the refund example, instead of asserting an exact string, check that the
answer is grounded in the knowledge base:
from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase
case = LLMTestCase(
input="What is the refund window?",
actual_output=resp,
retrieval_context=kb.search("30 days"),
)
metric = FaithfulnessMetric(threshold=0.9)
metric.measure(case)
assert metric.is_successful() # gate on the score, not the stringFaithfulness here decomposes the answer into atomic claims and checks each against the retrieval context, returning the proportion that are supported. The test no longer cares whether the bot said "30 days" or "you have a month" it cares whether every claim traces back to the source. Then gate releases on eval scores the same way you'd gate on a test pass rate.
Monday-morning tool: DeepEval
Knowing which problem you're testing
Before you run any of this, ask one question: did AI write the code, or is AI the system? If
AI wrote the code, traditional QA still applies SAST (SonarQube, CodeQL), package
verification (Snyk), and mutation testing (PIT, Stryker) though the risk profile has shifted,
since studies in 2025 found roughly 45% of AI-generated code carried security
vulnerabilities, with the rate exceeding 70% for AI-generated Java specifically. Tests still
apply; trust does not.
But if AI is the system, run the full PROBE loop. That's the category where assertions
break and where the framework earns its place.
Start tomorrow
Every tool named here is free and open-source. You don't need all five practices at once, pick one row and ship it this week. Write three behavioral contracts. Wrap one prompt in promptfoo and run it fifty times. Point a faithfulness eval at your highest-risk flow and watch the score for a fortnight.
The mindset shift is the whole game: stop asking "did it return the right string?" and start asking "did it stay within the band of acceptable behavior, and will I know when it drifts out?" Traditional bugs crash. LLM bugs lie. Test for the lie.
Tanvi Mittal is a Senior SDET and Test Automation Lead at U.S. Bank, where she leads
QA for AI/ML systems in regulated financial environments. She is the creator of the open-source LogMiner-QA and PromptArmor tooling, an IEEE Senior Member, and a speaker on testing non-deterministic systems. She writes on AI quality assurance at
medium.com/@tanvimittalQA.
