Debug Retrieval Misses by Inspecting Chunks and Queries

A support user asks, Can I get a refund for a digital download after I started the download. The retrieval query sent to the vector store is refund policy digital download started, with a metadata filter {"doc_type":"policy","region":"US","version":"2024-01"} and k=3. The retrieval step returns three chunks about physical returns, shipping delays, and store credit, and none of the returned chunks contain the relevant sentence about non refundable digital goods after download starts. This lesson turns a miss like this into a routine: name the likely cause from what actually came back, fix it in an order that keeps the signal clean, and know when to stop trying and escalate instead.

Map miss types to first fixes

With a reproducible miss captured, the next move is to name the most likely cause in a way that immediately suggests the smallest safe fix. The goal is not to guess perfectly, it is to pick one change that should move the top results toward the missing policy line.

Common retrieval miss categories and what they look like in the returned chunks include:

• Query mismatch where the query uses different terms than the policy, so the top chunks cluster around a nearby topic. A query like refund started download may miss a policy that uses digital goods, delivered content, or chargeback language.

• Chunk boundary split where the rule and its exception land in different chunks, so the retrieved snippet contains half the story. A chunk might include refundable within 14 days but omit the next sentence that excludes digital downloads.

• Over tight filter where the filter blocks the only document that mentions the rule. A version value that does not exist in the index often yields plausible but wrong neighbors from the remaining subset.

• ...