Complete the Evaluation Specification

All required evaluation components are now defined: quality SLOs, the golden dataset, evaluator calibration, component and trajectory rubrics, and release gates. Assemble them into one document, the evaluation specification, so they function as a single enforceable evaluation contract. This artifact defines the controls needed to operate a probabilistic system: it defines acceptable outcomes in measurable terms, how those outcomes are evaluated offline and monitored in production, and which operational action follows when defined drift or failure thresholds are crossed. Write it for engineering leads, security and risk stakeholders, and the product stakeholder responsible for acceptance. It must support four defined actions: ship, hold, rollback, or reroute, with each action constrained by the agreed containment envelope and tool-permission rules.

Evaluate this artifact against two acceptance criteria: traceability and actionability. For traceability, check that every business KPI and NFR maps to an explicit quality SLO, that each SLO maps to a golden dataset stratum and a defined evaluator, and that each evaluator output maps to a release gate with a defined threshold and operational response. For actionability, check that every gate identifies the person or team responsible for it, the evidence artifacts that must be archived, a defined exception and approval path, and an operational response that can execute without requiring stakeholders to reinterpret the original decision during an incident or audit.

Identifiers, readers, and the gate sequence

Carry a small set of consistent identifiers through every layer of the specification. Give every requirement one or more quality SLOs with a stable slo_id. Give every dataset stratum a slice_id. Give every evaluator an evaluator_id and a calibration reference. Give every gate a gate_id with an explicit scope. With those IDs in place, engineering can implement the check and the logging, risk can verify that a high-impact slice was covered, and product can approve a trade-off explicitly instead of through an ad hoc launch.

The ship decision is not a single threshold. It’s a sequence of gates that reflects the system boundaries defined in a prior artifact. Component gates cover model response quality, retrieval relevance, and tool call correctness. End-to-end gates cover a user-visible task across the full integration boundary. Trajectory ...