Designing Retrieval and Evals
Design retrieval and evaluation for an internal research assistant at a wealth management firm, emphasizing compliance and accuracy when handling sensitive financial data. Apply a structured approach to retrieval-augmented generation (RAG), ensuring that only eligible documents are accessed under explicit source approval and advisor entitlement criteria. Check citation identity, claim support, and conflict handling while assessing cost and latency. Distinguish scripted fault-detection results from measured assistant performance. Preserve the internal-only boundary, document unresolved source decisions, and require validated controls, expert review, and named owners before an advisor pilot.
In an FDE deep dive, “we’d use RAG” is where the questions start, not where they end. The interviewer will ask which passages the model saw, who was allowed to see them, how a citation proves a claim, and what one answer costs. Aster’s June 10 inquiry is a good test: two eligible reports give different maximum fees, and a third, restricted report must never reach a ranking service or a model. This lesson designs the retrieval, the citations, the eval set, and the cost and latency budget for it.
What must the design respect?
The workflow is an advisor-initiated internal research aid. It gives no personalized client advice, sends no client messages, changes no portfolios, and places no transactions. OpenAI’s usage policies prohibit offering AI tools that provide tailored advice requiring a license without appropriate involvement by a licensed professional.
The facts from discovery carry over unchanged. R1 version 2 states a maximum of 4%, and R2 states a maximum of 5%. R1 version 1 was superseded on June 1, and R3 is outside the advisor’s entitlement. Nobody has decided which eligible report governs.
Aster still has to decide source precedence, whether the assistant may show a conflict, the real permissions, which written supervisory procedure covers the tool, which outputs are records under SEC Rule 17a-4, and who runs the system. The design below is a proposal until those owners sign off.
How should retrieval work over research documents?
Retrieval-augmented generation, or RAG, prepares documents, splits them into passages, retrieves the relevant ones, optionally reranks them, and puts the selected evidence in the model’s input. OpenAI’s built-in file search handles much of that pipeline for you; a custom pipeline gives the application more control.
For Aster, ingest approved document snapshots through a controlled process. Each searchable passage carries its source identity (document ID, authoritative version, approval state, effective and superseded dates, and a stable passage ID or offsets into the snapshot), its scope (product, intended use, and access metadata, linked to the policy record that governs the document), and its provenance (a permanent mapping back to its exact snapshot and version).
Keep a qualification such as “maximum” in the same chunk as the figure it qualifies. Overlap or structure-aware chunking helps, but inspect the boundaries: an exception in the next paragraph may need to travel with the claim. If required metadata is missing, send the document for review instead of treating it as approved.
R2 shows why this matters. Its applicability is known, but its authoritative version and exact effective date aren’t. A real pipeline must get the fields Aster marks as mandatory before indexing it, and a local placeholder ID isn’t a version. The diagram shows the June 10 request passing through the eligibility filter.
Why filter before ranking?
The application authenticates the advisor and gets entitlements from a trusted access ...