Designing an Autonomous Reward Learning Agent

In this lesson, we analyze NVIDIA Eureka, an agentic system designed to automate one of the most challenging tasks in reinforcement learning: reward function design. Instead of relying on human engineers to manually craft reward signals, Eureka uses a coding-capable LLM to generate, evaluate, and iteratively refine reward programs. We will examine the architectural strategy it adopts, how it improves through reflection and search, and what its empirical results reveal about the design of autonomous systems.

The design challenge and goals

Reinforcement learning systems depend critically on reward functions. The reward defines what the agent should optimize, and therefore determines the behavior it ultimately learns. Designing a good reward function is notoriously difficult. A poorly shaped reward can lead to unintended behaviors, exploitation of reward loopholes, slow or unstable training, and failure to generalize. Even small changes in reward design can dramatically alter learned behavior.

In complex environments, such as robotics, dexterous manipulation, or locomotion, reward design becomes an iterative and time-consuming engineering process. Experts must repeatedly write reward code, train policies, observe behaviors, adjust reward terms, and repeat this cycle. This manual loop is expensive, slow, and highly specialized. The challenge Eureka addresses is this:

Can we automate reward design itself?

Instead of training an agent to act within an environment, Eureka trains an LLM-based system to write the reward function that shapes that learning. This reframes reward engineering as an agentic search problem. To automate reward design effectively, the system must:

• Generate reward programs without manual shaping

• Adapt reward structure based on observed performance

• Explore ...