Search⌘ K
AI Features
Home
Courses
Applied AI Engineer: Build, Evaluate, and Operate Production LLM

Logging and Observability

Explore how to trace requests end-to-end in AI pipelines, capture structured logs without exposing sensitive data, and use tracing spans to pinpoint latency and failures. Understand how to set up in-app and proxy observability components for monitoring retrieval, model calls, and caching. Learn to detect silent failures related to context truncation and improve reliability through effective logging and operational dashboards.

We'll cover the following...

Users report a symptom that does not crash anything. The FastAPI endpoint returns 200, but the answer sometimes contains stale facts, and no one can reproduce it locally because local traffic, indexes, and caches differ from production.

This lesson covers evidence that lets us replay what happened for one request after the fact, without storing raw user text or secrets.

Trace one request as data flow

With a nondeterministic model call and a changing retrieval index, the only stable debugging unit is a single request moving through the full pipeline. Start at the HTTP boundary, then follow the data through auth and context attachment, retrieval or tool steps, the provider call, post-processing, and the response.

At each boundary, make sure some evidence is emitted that lets us correlate behavior later.

  • At ingress, emit request_id, route, and user_id_hash.

  • At prompt assembly, emit prompt_version plus a stable input_hash for the user text.

  • At retrieval, emit the retrieval_query plus top_k and retrieval_hit_ids.

  • At the provider call, emit provider, model, generation parameters, and provider_latency_ms.

  • At post-processing, emit output_schema_ok and any guardrail flags.

  • At egress, emit status_code, latency_ms, and optional tokens_in and tokens_out if available. ...