Artifact Spine Kickoff
Explore how to document evaluation gaps and produce comprehensive postmortems for generative AI pilots. Learn to organize artifacts in a portfolio, maintain evidence backlogs, and recommend phase gates. This lesson equips you with methods to track uncertainty, assign ownership, and prepare evidence packets that support safe and compliant AI system decisions.
The quality attributes and evaluation discipline from the last lesson are most useful when the copilot’s gaps are documented in a form that others can review. A useful GenAI postmortem should map the gaps in the available evidence: what was assumed, what was not measured, which controls were missing, and which responsibilities were unassigned.
Portfolio shell as the artifact spine
Use a portfolio folder tree as the consistent location for each decision and supporting artifact. Give each folder an index document that links to the current artifact and identifies the responsible person or team. Later chapters expand the portfolio with the intake brief, architecture options paper, ADRs, data and context specification, evaluation specification, threat model, observability plan, runbooks, cost model, governance mapping, and roadmap.
The shell gives evidence and decisions a stable home, so reviewers do not have to reconstruct history from chat messages, tickets, and disconnected diagrams. The diagram below shows the folder tree, captured as a baseline now and updated as later reviews add evidence.
Acceptance criteria for the postmortem-style critique
Separate three categories in the postmortem:
Observed fact: Something recorded in a ticket, log, audit event, or stakeholder report.
Hypothesis: A plausible explanation that still needs evidence.
Corrective decision: A proposed action, owner, and gate that changes what happens next.
For example:
Type | Example |
Observed fact | A user reported a restricted-document snippet in an answer. |
Hypothesis | Retrieval filtering may not have applied the user’s entitlement. |
Corrective decision | Security will review retrieval authorization tests before cohort expansion. |
This separation keeps the postmortem useful: it records uncertainty instead of turning a plausible explanation into a root-cause claim.
Fictional pilot inputs and how to read them
The fictional pilot description you will critique is intentionally thin and inconsistent. It says a regulated enterprise built an internal assistant to answer employee questions about policies and procedures, then expanded use after positive demos. It claims no sensitive data was used, but also claims the assistant cites internal documents without naming the documents or their approver.
The pilot reports three warning signs:
A response appeared to contain text from a restricted document.
The service timed out or returned errors during peak hours.
Users received inconsistent answers to the same question.
Note: Don’t fill missing details with imagined architecture. Record them as unknowns and identify the evidence required to resolve them.
Evidence backlog and reusable postmortem
An evidence debt is a missing measurement, control, approval, or owner that blocks a safe decision. Typical debts include retrieval access-control tests, peak-hour latency measurements, a grounding rubric, and version evidence for incident reconstruction.
Prioritize by risk and uncertainty. Privacy and authorization evidence normally comes before interface polish, because a polished workflow does not reduce the impact of an access-control failure.
Keep unknown as a valid status. If the pilot has no availability target, write “unknown” instead of inventing one: the absence of a target is itself evidence debt.
The completed postmortem outline below shows how to record each debt with the decision it unlocks, an owner, and a review gate, covering the system, scope, review date, audience, observed impact, confirmed timeline, requirements and quality attributes, existing evidence, hypotheses, corrective actions, residual risks, and the scale-up decision.
Practice: Produce the first review packet
Use the three pilot symptoms from earlier in this lesson. Produce:
A one-page postmortem summary.
An evidence-debt register with at least five entries.
A recommendation for the phase gate.
A defensible starting recommendation for this pilot: Don’t expand the user cohort until access-control evidence, peak-hour reliability evidence, and answer-quality evaluation exist and have named owners.
Paste your one-page postmortem summary, your evidence-debt register (at least five entries), and your phase-gate recommendation. A reviewer AI will read your packet the way a skeptical stakeholder would — checking whether you've separated observed fact from hypothesis, whether your evidence debts actually block a decision instead of restating the symptoms, and whether your recommendation is backed by the gaps you found rather than by how the demo felt. You can paste all three pieces at once or work through them one at a time.
I'll be reviewing your packet the way I'd review any pilot before signing off on cohort expansion — looking for what's actually known, what's still a guess, and whether your recommendation earns its conclusion.
Go ahead and paste your postmortem summary, your evidence-debt register, and your phase-gate recommendation for the policy and procedures copilot pilot. If you only have one piece drafted so far, that's fine — share what you have and I'll start there.
One thing before you begin: don't invent architecture details the pilot description doesn't give you. If something's genuinely unknown — which model, whether retrieval filtering applied, what caused the timeouts — write "unknown" and treat that as evidence debt in its own right. That's usually where I'll push hardest.
Your packet is ready when a skeptical reviewer can identify what is known, what is uncertain, who must act, what evidence is required, and what decision the evidence will unlock.
Chapter summary
This chapter established what an AI Solutions Architect is responsible for and what evidence supports those decisions: the decision chain from stakeholder goals to measurable requirements and supporting evidence, the three review gates that prevent a pilot from advancing without sufficient evidence, and the artifact structure used to organize decisions and evidence across later chapters. It also introduced the running example, the Policy & Procedures Copilot, and the discipline behind it: a probabilistic service requires potential sources of variance to be identified and supported with evidence, along with a postmortem-style analysis that distinguishes observed facts, hypotheses, and corrective actions. These artifacts are maintained in a portfolio structure that remains usable across multiple review cycles.
The next chapter builds on that foundation with the AI capability map: identifying the core capabilities available to a GenAI system, including prompting, retrieval, tool use, and agentic loops; mapping each capability to the risks and evidence requirements it introduces; and using that map to justify the simplest capability set that meets the copilot’s requirements and is supported by the available evidence.