Trajectory Evaluation for Multi-Step Agent Workflows

Component tests can verify retrieval and tool-call behavior against defined expectations in isolation. Neither evaluates whether the agent’s sequence of actions followed safety constraints or used resources efficiently. Multi-step agentic workflows require evaluation of the action sequence, not only the final output. A workflow can violate safety constraints or use unnecessary steps while still producing an apparently correct final output.

In the plan/execute agent variant, end-to-end success depends on behavior across multiple steps and systems of record. The workflow must retrieve policy, propose an action with citations that preserve source traceability, invoke read-only tools when additional evidence is required, and obtain the required approval before invoking any write tool. Write actions must be blocked unless the required authorization is present, and the workflow must escalate to human review when the defined escalation conditions are met.

Separate output quality from workflow quality in the success criteria. An apparently correct response can still fail the workflow criteria if the agent bypasses the approval gate, fabricates a tool result, cites a source without verifiable provenance, or repeatedly invokes a high-cost tool without reducing uncertainty or adding relevant evidence. An incomplete response can still be acceptable if it follows the defined safe-failure behavior by stopping when it reaches the defined action boundary and escalating with a decision-support packet that records what was checked, what remains unknown, and why human approval is required.

Trajectory evaluation dimensions for plan/execute agents

Trajectory evaluation scores the validity and quality of the path a workflow takes through planning, retrieval, analysis, tool use, approval gating, and finalization. The primary dimensions for this agent variant are step validity, tool-use justification, containment-envelope compliance, and termination quality. These ...