Generative diffusion models are renowned for their ability to scale during inference by re-allocating computation without retraining, typically by adjusting the number of denoising steps. However, recent work shows that performance gains tend to flatten after a certain point when simply increasing NFEs for denoising. This article explores inference-time scaling beyond denoising-step expansion by reframing the process as a search over better injected noises, guided by verifiers that provide feedback on sample quality and alignment with conditioning.
At the heart of this approach is a two-axis design space: verifiers that assess candidate samples and algorithms that search for superior noise candidates. Through a structured walk-through on ImageNet with class-conditioned generation and then extending to text-conditioned generation, the framework demonstrates that inference-time compute can yield substantial improvements even when denoising steps are fixed at efficient levels. This reframing mirrors trends in large language models (LLMs), where inference-time strategies-such as re-ranking and iterative refinement-boost performance without retraining, suggesting similar opportunities for diffusion models.
Intrinsic flexibility of diffusion models at inference
Diffusion models inherently allow varying compute at inference, since generation starts from pure noise and proceeds through a denoising process across multiple noise levels. Unlike GANs or VAEs, diffusion models have an explicit randomness source through the initial noise and optional injected noises during sampling, which means the final sample mapping is deterministic given the noise sequence. Yet, increasing denoising steps yields diminishing returns due to accumulated approximation and discretization errors, inviting alternative strategies beyond brute-force step augmentation.
Randomness from noise is not created equal: some noise configurations lead to notably better generations than others. This insight motivates searching for better noises in the sampling process rather than solely amplifying the number of denoising steps. The proposed framework uses verifiers to evaluate candidate samples and algorithms to discover superior noise candidates, thereby enabling inference-time scaling through search rather than step-count alone.
Framework: verifiers and algorithms
The proposed framework identifies two critical design axes:
- Verifiers provide feedback on candidate samples. They simulate the final evaluation or capture conditioning alignment and can represent different use cases, such as privileged evaluation information, conditioning-aware assessment, or no extra information.
- Algorithms search for better noise candidates given verifier feedback. These range from simple random searches to more structured zero-order or neighborhood-based searches that iteratively refine candidates with modest compute overhead.
In the simplest instantiation, a class-conditioned ImageNet task uses a SiT-XL model with a fixed denoising budget (e.g., 250 NFEs) while varying the search budget to explore initial noise candidates and selecting those that maximize a verifier score. Verifiers include:
- Oracle Verifier: privileged final evaluation information (theoretical upper bound on performance).
- Supervised Verifiers: models like CLIP and DINO that provide alignment or classification-based guidance, albeit with caveats about mode diversity and potential biases when optimizing solely for point-wise scores.
- Self-Supervised Verifiers: similarity measures derived from feature spaces (e.g., CLIP or DINO features) to avoid reliance on extra conditioning information.
Two broad categories of verifiers emerge: oracle-like, which may not be practical in real deployments, and realistic verifiers that rely on pre-trained models available in typical deployment scenarios. The framework explores how these verifiers guide the search for better noises and how their biases influence generation diversity and quality.
Search algorithms: from random to guided optimization
The progression of search strategies begins with Random Search, a Best-of-N approach over a pool of candidate noises evaluated via the verifier. While straightforward and effective in early experiments, this approach can overfit to verifier biases and reduce diversity when scaling up compute. To address this, the framework introduces more nuanced search strategies that refine candidates incrementally rather than exhaustively exploring the full Gaussian space at once.
Zero-Order Search, inspired by zero-order optimization, samples neighbors around a pivot noise and selects the best candidate according to the verifier, iterating to approach a local optimum. The number of iterations (how many passes through steps of generating neighbors and evaluating them) becomes the primary axis for scaling compute in search. A larger neighborhood or more iterations can yield a closer local optimum but increases compute; the stride parameter λ controls the step size of exploration, trading speed for precision. Practical implementations fix λ and study scaling with respect to the number of neighbors N per iteration.
Path-based and other structured searches further adjust the noise space by exploring trajectories through the sampling process, potentially improving robustness to verifier biases. First-order gradient-based search is theoretically possible by back-propagating through the sampling process, but it is often computationally prohibitive in practice due to the need to differentiate through the entire diffusion process. The framework emphasizes that many verifiers are differentiable, yet the cost of full back-propagation remains a barrier in typical settings.
Empirical insights: verifiers, biases, and sample quality
Experiments with oracle verifiers on ImageNet show that the straightforward random search can yield notable gains in FID and IS as the search budget increases. However, the practicality of oracle verifiers is limited because they rely on final evaluation details not always available in deployment. When using supervised verifiers like CLIP and DINO, improvements in sample quality are observed, but issues such as mode collapse and diversity loss can arise if the search overly focuses on point-wise scores without considering global sample statistics. Self-supervised verifiers demonstrate that alignment between early-stage feature similarities and final quality can serve as a viable alternative to extra conditioning data, enabling effective scaling even when conditioning information is scarce or unavailable.
Crucially, the work highlights that over-reliance on a single verifier can introduce biases, narrowing the generated distribution and reducing diversity. A verifier ensemble can help mitigate this risk by combining multiple evaluators that capture different aspects of quality, alignment, and aesthetics. The study also discusses the need to balance search costs with real-time constraints in interactive or production settings.
Extensions to text-conditioned generation and broader implications
The framework generalizes beyond ImageNet to text-conditioned diffusion models, confirming that inference-time scaling via search can benefit diverse generation tasks and model scales. In the text-conditioned setting, multiple verifiers provide complementary signals, and the corresponding search strategies adapt to the multimodal nature of the conditioning information. Overall, the results suggest that diffusion models can achieve higher generation quality by investing compute in a search over noise rather than simply increasing denoising steps.
These insights mirror a broader trend in generative AI: moving from static pipelines to adaptive generation loops where feedback guides generation in real time. The exploration of adaptive generation could lead to diffusion systems that dynamically adjust noise injection, leverage multi-step verifier feedback, and support interactive control, enabling more nuanced and controllable outputs while maintaining efficiency at inference time.
Key takeaways
- Inference-time scaling in diffusion models can go beyond increasing denoising steps by actively searching for better initial noises guided by verifiers.
- A two-axis framework of verifiers and algorithms enables tailored strategies for different tasks, conditions, and performance goals.
- Random, zero-order, and path-based search methods show varying trade-offs between compute, diversity, and sample quality, with verifier biases playing a central role.
- Self-supervised verifiers offer practical advantages when conditioning information is limited or unavailable, helping preserve diversity while improving quality.
- Extending these ideas to text-conditioned generation demonstrates broad applicability across modalities and model scales.
Table: summary of verifiers and findings
| Verifier type | Example signals | Observed effect |
|---|---|---|
| Oracle Verifier | Final evaluation metrics (FID, IS) | Highest theoretical gains; not widely practical |
| Supervised Verifier | CLIP, DINO logits | Quality improvements but risk of reduced diversity |
| Self-Supervised Verifier | Feature similarity at low noise | Effective gains without extra conditioning data |
