Designing Tool Schemas and Retrieval
Design tool schemas and retrieval processes that provide operators with accurate, authorized information they can verify. Restrict passage eligibility before ranking and generation to exclude unauthorized or superseded instructions. Define clear read-only tool contracts, use strict schemas, and keep trusted identity and scope in application code. Validate operator entitlements and filter documents using authoritative metadata before relevance ranking. Return exact passages with document versions and stable locators, and recheck access when citations are opened. Distinguish valid arguments, authorized reads, and supported claims, and route missing evidence or unsupported answers to qualified human review.
We'll cover the following...
- Why not filter the citations afterward?
- How do we design a tool the model uses correctly?
- Which documents are eligible?
- Hosted file search or your own retrieval?
- How do chunking, metadata, and filters fit together?
- Does a citation make the answer true?
- How do we run the boundary?
- How do we defend the boundary?
- What’s next?
In an offline review of 12 document-lookup queries, Northstar’s first ranking put a superseded instruction at the top for four queries, and an unauthorized customer-specific instruction at the top for two more. These are counts from the review set, not production rates, but either failure would hand an operator a fluent, cited answer they shouldn’t use.
A model sees your tools only through their names, descriptions, and JSON schemas, and your documents only through the passages retrieval hands it. Operators reviewing a triage draft need to know which service instruction applies and where to check it. That means two design jobs: tools the model calls correctly, and retrieval that returns only passages the operator may see, in a form they can verify.
Why not filter the citations afterward?
The tempting repair is to rank everything, let the model draft an answer, and strip forbidden citations afterward. That fails at the exposure boundary: the model has already read the private text and may have used it. It also leaves an obsolete instruction looking authoritative whenever its citation survives.
Before reading Northstar’s fix, decide what it should do first.
Raising the result limit only lets more ineligible text through. Restrict eligibility first, then test whether ranking finds the supporting passage.
How do we design a tool the model uses correctly?
A tool contract defines an operation, the arguments the model may supply, and what its results mean. OpenAI’s function-calling guidance comes down to a few habits. Use clear names, and describe each parameter’s purpose and format and what the result represents. Use enums and object structure so the model can’t express a contradictory call. If code already knows the operator’s identity or organization, supply it in code and leave it out of the schema. Keep fewer than 20 functions available at the start of a turn, and evaluate accuracy as you add more.
Turn on strict mode, too. With strict: true, every object sets additionalProperties: false and lists every field in required; an optional field becomes nullable by adding null to its type.
A raw database query tool would give the model control over tables and scope. Northstar instead exposes read_service_instruction(ticket_id, question): two required strings, no extra properties, and a read-only operation. The application validates the identifier’s format and the question’s length, but a well-formed ticket ID is still only a request and grants no access. Trusted application context supplies the operator’s identity and organization. The adapter then reads the authoritative ticket, verifies entitlement, and derives the equipment family and applicable date. The model chooses none of those, and it never picks a document version.
The tool returns one of three results: