Eureka: Automating Reward Design with Coding LLMs

Problem space: Automating reward design for reinforcement learning, using LLM-based agent systems.

Understanding the problem of reward design

In reinforcement learning (RL), agents learn to perform tasks by interacting with an environment, and receiving feedback in the form of rewards. The fundamental goal for an RL agent is to learn a policy that maximizes the total accumulated reward over time. However, this learning process is critically dependent on how well the “reward function” is defined. A reward function acts as a precise rulebook, signaling to the agent what constitutes desirable, or undesirable behavior in the environment. For instance, a positive reward might be given for successfully balancing a robot or picking up an object, while a penalty is applied for falling or dropping it. The agent then uses this feedback to iteratively adjust its actions and improve its performance.

While the concept seems straightforward, designing effective reward functions for real-world RL tasks is notoriously difficult. It’s far more complex than simply assigning a success point, or a failure penalty. Real-world behaviors are often nuanced and involve many subtle elements that are challenging to codify explicitly. For example, teaching a robotic hand to spin a pen isn’t just about the pen rotating. The reward function must also implicitly or explicitly encourage smooth motion, prevent the pen from being dropped, avoid unsafe angles, and possibly account for efficiency or style. If any of these subtle elements are missed, the agent might learn unintended or “exploitative” behaviors, such as flinging the pen too quickly accumulate a high reward. This intricate and often brittle process is known as “reward engineering,” or “reward design.”

This challenge has long been a significant bottleneck in applying RL to complex real-world problems. It demands a combination of deep technical skill, specific domain knowledge, and a substantial time investment in a trial-and-error process that is often inconsistent. Without a well-designed reward function, the agent is effectively learning without any direction, akin to a student trying to pass an exam without a clear syllabus or understanding of what’s being graded. The reward function is the crucial “syllabus” that provides the learning target. Automating this human-dependent and often unreliable process in an intelligent, and safe way is a powerful advancement in agentic system design.

Introducing Eureka: An agent that designs rewards

Recognizing the bottlenecks of manual reward design, ...