Probabilistic Services and Why Evaluation Is Engineering
Understand the unique nature of probabilistic services in AI, why evaluation is an engineering necessity, and how to identify and manage output variance sources. Learn to implement quality controls ensuring safe, auditable, and compliant production-ready AI systems through continuous monitoring and evidence-based assessment.
Our copilot behaves as a probabilistic system even when exposed through conventional APIs. The same question can produce different outputs because model generation is non-deterministic, and because retrieved context, tool results, or policy data may also change. The diagram below shows the resulting operating loop: the same input can produce a distribution of possible outputs across runs, monitoring tracks system performance, and evaluation and controls inform changes to the service.
Deterministic and probabilistic services
A deterministic service returns the same result when the input and system state are the same. Tests can usually compare the actual result against an expected result.
A probabilistic service can return different outputs for the same apparent input. A model may sample different wording, a retrieval index may change, a tool may return a partial response, or a policy cache may be out of date. Testing one successful answer does not describe the service’s behavior across real users and conditions.
Production readiness means showing that the copilot behaves acceptably across the range of inputs and operating conditions it will encounter. A successful demo cannot establish that.
Variance sources as architecture inputs
When inconsistent outputs appear, identify the most plausible variance source and collect evidence before assigning blame.
Model variance comes from stochastic decoding, conversation history, configuration changes, and model-version drift. Check whether repeated runs with the same context produce materially different answers, and record the model version and settings used.
Retrieval variance appears when the knowledge index, embeddings, chunking, ranking, or access filters change. Capture the documents retrieved, their ranking, and the entitlement decisions so a retrieval change can be told apart from a model change.
Tool variance appears when an external API returns an error, partial result, timeout, or changed schema. Capture tool-call traces and response validation, and verify that the copilot signals uncertainty or falls back safely instead of answering confidently from missing data.
Data-freshness variance appears when policies, records, or entitlements change while caches or replicas lag. Record the effective version or timestamp of the sources used: an answer can become incorrect even when the model and application code have not changed.
These sources shift the design responsibility from proving one correct answer to proving controlled behavior under change. Useful controls include baseline comparisons, abstention or escalation when evidence is weak, and defined degradation modes when retrieval or tools fail.
Evidence must capture the inputs that influenced an answer without creating a new privacy problem. Logging a document identifier and policy version is often safer than storing the entire restricted document.
Quality attributes that are measurable for GenAI
Classic nonfunctional requirements such as latency, reliability, and availability still apply, but they do not describe whether the content is justified, safe, private, or reconstructable.
The system also needs AI-specific quality attributes:
Grounding: The answer is supported by an approved source.
Safety: The answer does not recommend a prohibited action.
Privacy: The system uses and reveals only data the user may access.
Auditability: A reviewer can reconstruct the request, sources, configuration, and human decision.
Each attribute needs a measurement method. Grounding, for example, could measure the percentage of sampled answers whose key claims are supported by cited policy. Later chapters will turn these measures into a quality SLO, a contract that ties quality targets to release and operating decisions.
The table below organizes classic NFRs alongside grounding, safety, privacy, and auditability, with example evidence types under each.
Treat evaluation as part of engineering
Evaluation is not a one-time QA activity, because the service can change without any application code changing. A model provider may update a version, a policy index may get rebuilt, a prompt may grow, or an upstream tool may change its schema.
For every material change, plan a comparable before-and-after check:
Use representative questions, not only successful examples.
Include edge cases such as conflicting policies and missing records.
Pin the model version, retrieval snapshot, and relevant configuration.
Record the failure rate and the action required when the target is missed.
A demo, an anecdote, or a cherry-picked prompt can help discover a problem. None of them can establish readiness, because none show frequency, variance, or the effect of a controlled change.
Variance-to-evidence checklist
When inconsistent outputs appear, the first step is to identify the most plausible source of variance. Each source requires different evidence and may require different mitigation or containment measures, so the investigation should focus on attribution and appropriate controls before attributing the inconsistency to the model itself.
Model variance is suspected when the same retrieved context and tool outputs still produce materially different content across repeated runs, or when a new model version correlates with changed behavior. Minimum evidence is a reproducible replay that pins model version and configuration, plus a small sample of repeated runs to quantify spread rather than treating one surprising answer as definitive.
Retrieval variance is suspected when the assistant cites different documents, quotes different policy passages, or answers differently for two users with different entitlements even when the question is identical. Minimum evidence is the retrieved document set with ranking metadata and access-filtering outcomes, plus a comparison to prior index or ranking states if the inconsistency appears across days.
Tool variance is suspected when answers depend on operational data and failures correlate with timeouts, partial payloads, schema changes, or elevated error rates in upstream services. Minimum evidence is tool-call traces with response codes and payload validation outcomes, plus proof of the degradation behavior that occurred when the tool did not return a complete result.
Data freshness variance is suspected when policy-related answers change after known policy updates, or when different environments or caches serve different policy versions. Minimum evidence is the effective version or timestamp of the policy and records used for the response, plus confirmation of where caching or replication lag could have introduced stale inputs.
The table below maps each variance source to its likely observable symptoms and the minimum evidence needed to confirm or rule it out:
The preferred architecture decision changes once evidence shows whether inconsistency is dominated by model stochasticity, retrieval shifts, tool failure, or stale policy inputs. Each cause points to a different control surface and a different definition of acceptable variance under a future quality SLO. With an inconsistent output now distinguishable from a proven model defect, the next lesson turns the pilot’s symptoms and evidence gaps into a reusable portfolio shell and postmortem backlog.