JSON and OpenAI Function-Calling Agents

In the previous lesson, the Structured Chat agent marked a clear improvement over basic ReAct parsing by instructing the LLM to emit JSON blobs for tool invocation. That approach handled multi-parameter tools far better than regex extraction from free text. But it carried a hidden fragility: the entire system depended on the LLM faithfully following prompt instructions to produce valid JSON every single time. In production, that assumption breaks down. Models drift, hallucinate extra fields, forget closing braces, or revert to plain text under ambiguous queries. For a customer-support agent that must reliably call order_status, initiate_refund, and check_inventory tools thousands of times per day, even a 1% JSON parsing failure rate translates to dozens of broken user interactions.

This lesson addresses that fragility head-on. It covers two progressively more robust solutions that move tool invocation from prompt engineering into structured API contracts. First, LangChain’s JSON agent, which adds validation and repair heuristics on top of prompt-instructed JSON. Second, OpenAI’s native function-calling mechanism and Amazon Bedrock’s structured outputs, which enforce output schemas at the API level, so the model cannot produce malformed responses. By the end, you will understand the mechanics of each approach, know when to choose one over the other, and have the monitoring vocabulary needed for production deployment.

The following diagram illustrates this reliability progression across three generations of tool invocation.

How JSON agents work in LangChain

LangChain provides create_json_agent and create_json_chat_agent utilities that formalize prompt-instructed JSON output into a repeatable execution cycle. The system prompt explicitly tells the LLM to emit every action as a JSON object containing two keys: action (the tool name) and action_input (the parameters). A JsonOutputParserA LangChain component that validates incoming text as JSON and extracts specified fields before passing them to downstream tool executors. sits between the LLM output and the tool dispatcher, catching malformed responses before they reach tool code.

The execution cycle

The agent follows a ...