Choosing the Right Capability from Evaluation Failures
Understand how to diagnose AI feature failures and select the correct capability—retrieval for missing information, tool calling for actions, or fine-tuning for model limitations. Explore common mistakes and how to combine capabilities effectively to maintain reliable AI features in production.
The previous lesson’s five questions pointed a failure at one of a few specific causes: missing information, a missing ability to act, or a genuine limit on what the model can do. This lesson takes those three outcomes and matches each one to the capability that actually addresses it, and spends real time on a mistake that’s easy to make even with the diagnosis already done correctly.
In this lesson, we will cover:
Why retrieval, tool calling, and fine-tuning aren’t a progression from simple to sophisticated
What each one actually fixes, and what it doesn’t
One feature needing two different fixes for two different failures at the same time
Where fine-tuning actually earns its place, and why that’s rare
How real products combine these capabilities rather than climbing through them
The three ways engineers get ...