Long-Context Reliability and Cost Trade-Offs
The analysis of long-context reliability and cost trade-offs highlights the importance of evidence use in generating reliable answers. Comparing full context and selective retrieval strategies reveals that while full context may retain critical dependencies, selective retrieval can omit essential exceptions, leading to incorrect conclusions. Bounded expansion emerges as the most effective strategy, achieving the highest success rate by ensuring comprehensive evidence checks. Additionally, the discussion emphasizes the significance of managing input budgets and caching mechanisms to optimize performance and cost, ultimately guiding decision-making in evidence retrieval and model utilization.
Compare two requests:
Full collection: 40,000 tokens of manuals plus 1,000 request-specific tokens.
Retrieval alternative: 4,000 evidence tokens plus the same 1,000 request-specific tokens.
Predict which produces more reliable answers and which costs less per successful answer. Neither conclusion follows from input length alone.
Evidence use determines reliability. Cache hits, additional calls, and downstream operations determine request economics. The useful decision pairs a tested evidence strategy with a complete operating ledger.
What does fitting establish?
At Brindle Equipment, a service-procedure assistant answers questions using authorized, current manuals. Its task is to preserve the applicable equipment revision, operating conditions, exceptions, and supporting citations.
A context window describes input capacity. It gives no guarantee that a model will identify, reconcile, and correctly apply every relevant passage within that capacity.
The opening’s 41,000 and 5,000 input tokens are supplied measurements, not estimates from character counts.
Broad inclusion retains dependencies that retrieval might miss. It also presents distractors and potentially conflicting revision information that the model must interpret correctly. Selective retrieval reduces input volume and irrelevant material, but selecting the general rule without its exception can produce a confidently wrong answer.
The previous lesson established admissibility, evidence sufficiency, and claim support as separate checks. Keep those checks fixed while changing how much evidence reaches the model.
Testing the R7 exception
Brindle's test uses two source passages that must be considered together:
General manual, restart procedure: After a pressure check, equipment may be restarted. Apply any revision-specific restart exceptions.
R7 revision supplement, seal replacement: For R7 equipment after a seal replacement, a technician inspection is required before restart. Completing a pressure check does not remove this requirement.
The request is: "For R7 equipment after a seal replacement, can I restart after the pressure check?"
This answer fails:
Yes. You may restart after the pressure check [General manual, Restart procedure].
The citation refers to a genuine source, but the answer omits the R7 inspection requirement that applies to this request. Selective retrieval may exclude the R7 supplement, leaving the model without the applicable exception. Even when both passages are available in the model's context, the answer may still be incorrect if the model overlooks the exception or applies the general rule instead. A correct answer must state that an R7 unit requires technician inspection after a seal replacement, even after a pressure check, and cite the R7 supplement.
Evaluate three approaches using the same 20 service questions, authorized document versions, and checks that each answer is supported by its cited evidence:
Full context: Passes 18 of 20 questions.
Selective retrieval: Passes 17 of 20 questions. Three answers omit an applicable exception or information required from another section.
Bounded expansion: Retrieves complete sections, checks whether the evidence is sufficient, and retrieves additional related sections within a defined limit when necessary. Passes all 20 questions.
These results come from a controlled test set. Bounded expansion performs best on these 20 questions, but the results do not establish that it will outperform other approaches in production.
Add irrelevant passages and vary whether the applicable exception appears at the beginning, middle, or end of the context. Check whether the approach still produces supported answers. Keep the authorized document scope unchanged across all test variations.
Determine whether the restart answer satisfies the source-evidence requirements, then select an approach to address any retrieval or evidence-use failures.
A larger prompt repairs missing evidence only if assembly actually includes the missing section. If the R7 supplement was already present, inspect evidence application and answer validation instead of treating capacity as the diagnosis.