A Non-Deterministic Function Inside a Deterministic System
Explore the challenges posed by integrating non-deterministic large language model calls into deterministic backend systems. Understand how this variability impacts caching, deduplication, retries, and testing, and learn strategies to handle AI outputs effectively for stable, reliable backend operations.
We'll cover the following...
Much of the application logic we write is expected to produce predictable results for the same inputs and relevant system state. We often rely on that predictability implicitly when designing downstream logic. An LLM call does not provide the same deterministic guarantee, so downstream code that assumes deterministic output can fail in ways it was not designed to handle.
Same input, different output
Rather than describe this behavior abstractly, let’s run a simple example. Below is a simplified simulation of an LLM call. Instead of ...