MuLan’s Planning and Progressive Generation
Explore MuLan’s approach to multimodal LLM agents by learning how it breaks down complex text-to-image prompts into ordered sub-tasks. Understand the role of an LLM as both a global and local planner to create a stepwise generation plan, guiding object placement and size. Discover MuLan’s use of attention guidance and overlap handling to progressively generate images with precise spatial relationships, all without custom model training.
In our last lesson, we introduced MuLan’s “divide and conquer” strategy. Instead of tackling a complex image generation task all at once, it breaks the problem down into smaller, more manageable pieces. In this lesson, we’ll explore the first pillar of this architecture in detail: how the agent creates its initial plan.
The LLM as a global planner
Let’s return to our analogy of a human painter. A painter doesn’t just start randomly dabbing paint on a canvas. They first create a mental plan or a light sketch, deciding which objects will form the background and which will be in the foreground, and the general order in which they will be painted.
MuLan’s first step is exactly this: LLM planning. At the very beginning of the process, before any image generation happens, an LLM is used to create a global plan. It takes the user’s single, complex prompt and decomposes it into an ordered sequence of objects to be generated.