MuLan’s Planning and Progressive Generation
Understand how MuLan uses large language models to create a global plan that breaks down complex image prompts into ordered sub-prompts. Learn how progressive generation with conditional diffusion and local LLM planning manages object placement and size. Discover attention guidance and candidate selection techniques to handle overlaps, allowing MuLan to generate detailed multi-object images step-by-step.
In our last lesson, we introduced MuLan’s “divide and conquer” strategy. Instead of tackling a complex image generation task all at once, it breaks the problem down into smaller, more manageable pieces. In this lesson, we’ll explore the first pillar of this architecture in detail: how the agent creates its initial plan.
The LLM as a global planner
Let’s return to our analogy of a human painter. A painter doesn’t just start randomly dabbing paint on a canvas. They first create a mental plan or a light sketch, deciding which objects will form the background and which will be in the foreground, and the general order in which they will be painted.
MuLan’s first step is exactly this: LLM planning. At the very beginning of the process, before any image generation happens, an LLM is used to create a global plan. It takes the user’s single, complex prompt and decomposes it into an ordered sequence of objects to be generated.