Beyond fixed canvases, a survey of dynamic-length diffusion LLMs

I just published a new survey preprint: Beyond Fixed Canvases: A Survey of Dynamic-Length, Fully Parallel, and Bidirectional Diffusion Language Models 📚

Read it on Zenodo: zenodo.org/records/23535396

It grew out of Predict-then-Diffuse, our IJCNN paper on choosing the answer length of a diffusion LLM before generation. While working on it, I kept running into papers that attack the same problem from very different angles, often without citing each other. This survey puts them on one map. Predict-then-Diffuse is one of the methods it covers, so read that part knowing that I am one of its authors.

Table of contents

• The fixed-canvas problem 🖼️
• Block diffusion solves length by giving something up 🧱
• Four places to decide the length 🗺️
  • Reading the length from the model 🔎
  • Growing and shrinking the canvas ↔️
  • Training for length 🎓
• Length is one efficiency axis among several ⚙️
• What we measure, and what we don’t 📏
• Open problems 🔭
• Limits of the survey ⚠️
• Conclusion 🎯
• References 📚

The fixed-canvas problem 🖼️

An autoregressive (AR) model writes one token at a time and stops when it emits an end-of-sequence token. Its length is decided along the way, for free.

A diffusion language model (dLLM) works differently. It starts from a canvas of masked positions and, over a number of denoising steps, fills all of them in parallel. Every position can look at every other position, to its left and to its right. That is the promise: full parallelism and bidirectional context, instead of strictly left-to-right generation.

The catch is that the canvas has to exist before generation starts, so its length L must be chosen in advance. The survey’s starting observation is that the cost of generation depends on the canvas that was allocated, not on the length of the answer that was actually needed. With standard attention, the work per layer and per step grows roughly with L², while memory grows roughly with L when memory-efficient attention is used.

The default behaviour is generate, then trim: allocate a generous canvas, denoise all of it, and throw away the padding at the end. Both ways of getting L wrong hurt:

• Over-provisioning wastes attention work superlinearly, puts pressure on device memory and can even lower quality. Rainbow Padding reports that with a larger allocated length, an instruction-tuned LLaDA can produce shorter answers and lower accuracy, an effect its authors call eos overflow.
• Under-provisioning truncates the answer. In reasoning or code, an incomplete answer can be close to useless, and fixing it means a retry with a larger canvas, at extra compute and with latency that depends on the query. Resizing a canvas on real hardware is not free either: dynamic shapes cost shape computation, memory planning and synchronization.

Try both failure modes here. Choose how long the answer really is, then how large a canvas is allocated:

Answer the user actually needs96 tokensCanvas allocated before generation384 tokens

answer tokenspadding (computed, thrown away)cut off1 cell = 8 tokens

The square's side is the canvas length, so its area follows L². The amber part is what the answer needed; the grey part is paid for anyway.

75% of the canvas is padding, but it costs 16.0× what the answer alone would have.

The AR fix does not transfer. Emitting an end token tells you where the answer stops, but the canvas, and its cost, have already been paid for.

Block diffusion solves length by giving something up 🧱

The best-known way to give dLLMs variable length is block diffusion. The model generates a sequence of fixed-size blocks: inside each block, positions are denoised in parallel and attend to each other in both directions, and each block is conditioned on the ones before it. The output grows by appending blocks until the model stops, much like AR decoding. In that narrow sense, the length problem is solved.

The survey’s argument is that this has a structural cost. Block k+1 conditions on block k, but a finalized block k is not ordinarily revisited in light of block k+1. Across blocks, generation is causal again: semi-autoregressive. The figure below is an interactive version of the survey’s Figure 1. Pick a regime and click a position to see which positions it can condition on.

AutoregressiveBlock diffusionSingle canvas

Block size4 tokens

5 earlier positions visible

2 later positions visible

7 positions that can see position 6

The three structures are a directed chain (AR), a chain of cliques (block diffusion) and a single clique (single-canvas diffusion), whose size is the thing that length-adaptive methods change.

Two qualifications keep this argument honest, and the survey states both:

1. Seeing context is not the same as using it. Li et al. (2026b) report that fast dLLMs often fall back to left-to-right, AR-like decoding in practice, mainly because of the sequential structure of training data such as long chain-of-thought. Whole-canvas attention is a precondition for using bidirectional context, not a guarantee.
2. The cost belongs to how block diffusion is usually trained, not to blocks as such. dLLM-Var decodes in a block-continuation pattern for native variable length while reportedly keeping full attention. Length flexibility and whole-sequence bidirectionality appear to be separable design goals.

Block diffusion also has a real advantage: because it never needs a global maximum canvas, it can scale to arbitrarily long outputs. The closest available evidence is consistent with a trade-off, not a winner. In Bertolani et al. (2026), full-sequence models did best on Sudoku, which requires global constraint satisfaction, while block-based decoding led on GSM8K and HumanEval. That comparison is not controlled for training data or model scale, so a matched comparison is still open.

The survey also keeps two things apart that are easy to conflate: trained block diffusion, where blocks are part of the model, and inference-time block-wise decoding of a fully parallel model, as in Fast-dLLM, PACE-dLLM or SemBlock.

	Autoregressive	Single canvas	Block diffusion
Parallelism	none	full	within a block
Dependency across units	causal	n/a	causal across blocks
Bidirectional attention	no	global	within a block only
Native length	yes (EOS)	no (fixed canvas)	effectively yes (block count)
Length mechanism	per token	add-on (the methods below)	block chaining

Four places to decide the length 🗺️

The core of the survey is a taxonomy of single-canvas, length-adaptive methods: approaches that keep one bidirectional canvas and change how long it is. It organizes them by intervention point, that is, where in the life of a request the length is decided:

1. Pre-hoc prediction: guess the length from the query, then generate. Predict-then-Diffuse trains a lightweight external predictor, adds a safety margin and keeps a fallback for when the canvas is still too short; PILL reads a single mask token’s hidden state from the frozen dLLM and predicts the length of an infilling span with a small probe.
2. Masked-canvas probing: ask the model itself before committing content. SmartCrop reads EOS probabilities from one pass over a masked canvas; Entropy-Valley scores a few candidate lengths and keeps the one the model is most confident filling.
3. In-process expansion: start short and grow (or shrink) during denoising. DAEDAL, CARVE, ρ-EOS and LR-DLLM.
4. Native training: teach the model to make the termination signal reliable. dLLM-Var, Rainbow Padding and VoidPadding.

The categories compose rather than exclude, and the map below shows it. Click a method to see how it compares in decision timing, correction and overhead:

All methodsTraining-freeCan shrink mid-run

1. Training the dLLM

2. Before denoising

3. Masked-canvas probe

4. During denoising

Predict-then-DiffusePILLSmartCropEntropy-ValleyDAEDALCARVEρ-EOSLR-DLLMdLLM-VarVoidPadding

training-free trains something

SmartCrop

Training

None

Decision timing

Masked-canvas probe, then crop

Correction during denoising

No: the length is decided once

Overhead

Probe pass plus denoising on the resized canvas

Notes

Reads EOS probabilities from the initial masked canvas; evaluated on LLaDA

Reading the length from the model 🔎

SmartCrop is the clearest example of the idea that length information may already be in the model, a claim that comes from the paper that introduced it, “Diffusion language models are natively length-aware” (Rossi et al., 2026). From the per-position EOS probabilities φ_j on the initial masked canvas, it builds a cumulative termination probability

Pr(L* ≤ ℓ) = 1 − ∏_{j = Lp+1}^ℓ (1 − φ_j)

the probability that the response has ended by position ℓ, given a prompt of length L_p. The canvas is cropped at the first position where this crosses a threshold, and ordinary denoising runs on the shorter canvas. This is not post-hoc truncation: the cost is one probe pass plus denoising on the resized canvas, not full denoising followed by a crop.

The product treats per-position termination events as independent, and the crop is only as good as the signal. The figure uses a made-up signal to show why calibration matters: switch to the blurry signal and watch small early probabilities accumulate until the crop lands before the answer ends.

The survey’s caveats for SmartCrop are concrete. It still needs an initial canvas large enough to read a useful signal from, the decision is made once, it was evaluated on LLaDA, and requests cropped to different lengths complicate synchronized batching unless they are bucketed or padded. LR-DLLM targets a related failure mode: it observes that average confidence is biased by the number of masked positions, and corrects it with a penalty proportional to the logarithm of the length. The two have not been evaluated together.

Growing and shrinking the canvas ↔️

Four methods correct the length during denoising rather than at a single decision point:

• DAEDAL starts from a deliberately short canvas, expands it until the model’s EOS confidence near the end is high enough, then inserts masks at low-confidence positions during denoising. Its own ablations show that local expansion cannot fully make up for a poor starting point.
• CARVE proposes inserting masks next to a high-entropy anchor and keeps the expansion only if predictions at the original unresolved positions stay stable, measured by a low mean Jensen–Shannon divergence. The authors report up to roughly half the FLOPs of fixed-length decoding in some settings.
• ρ-EOS tracks the density of implicitly predicted EOS tokens among the remaining masks, and removes or inserts trailing masks within one denoising stage.
• LR-DLLM probes exponentially spaced lengths, then adjusts the span by one position at a time toward the better-scoring neighbour.

Only ρ-EOS and LR-DLLM can shrink an oversized canvas as well as grow an undersized one. The price of continuous correction is a more dynamic serving policy: shapes change during generation, and the systems cost of those edits is often not reported.

Training for length 🎓

dLLM-Var fine-tunes a pretrained dLLM on 6.4 million instruction pairs so that it predicts EOS accurately. The model then appends a new block of masks only when the current one finishes without EOS. Its reported evaluations still cap generation at 1,024 tokens. Rainbow Padding and VoidPadding fix the double role of EOS as both terminator and padding: Rainbow Padding cycles through distinct padding tokens, and VoidPadding adds a dedicated [VOID] token so that [EOS] can mean “the answer ends here”.

These are not strictly better than the training-free methods. Those apply to frozen, already released checkpoints, while native variable length is a property of one fine-tuned model. Which trade-off is right is a deployment question.

Length is one efficiency axis among several ⚙️

Length adaptation shortens the canvas. Other lines of work make each step cheaper, or take fewer steps:

• Discrete flow matching: FS-DFM reports parity with a 1,024-step baseline using eight steps under its protocol.
• Sparse attention: SparseD and Focus-dLLM.
• KV-caching for bidirectional denoising, which is structurally harder than in AR models because no position’s keys and values are guaranteed to stay fixed: dKV-Cache, dLLM-Cache, Fast-dLLM’s DualCache, dInfer, Flash-dLLM, Streaming-dLLM, Sparse-dLLM and Mask Tokens as Prophet.
• Step-skipping and early exit: ES-dLLM, DyLLM, DMax and CreditDecoding.
• Speculative decoding, with the dLLM as its own drafter: Self Speculative Decoding reports lossless acceleration; PSD reports up to 5.5× tokens per forward pass, a per-forward-pass figure rather than wall-clock time.
• Hierarchical diffusion: HDLM, Zonkey and Diffusion in Diffusion.

It is tempting to multiply the speedups. The survey argues against it: changing the canvas can invalidate cache entries, alter sparsity patterns or change the confidence signals used for skipping and verification. No published system stacks all of these axes and measures the combined effect.

The cross-cutting comparison condenses to one pattern: the length-adaptive methods generally keep whole-canvas attention, while block diffusion explicitly introduces causal dependencies across blocks.

Family	Parallelism preserved	Bidirectionality preserved	Training needed
Pre-hoc prediction	yes	yes	predictor or probe only
Masked-canvas probing	yes	yes	none
In-process expansion	yes	yes	none
dLLM-Var	yes	yes (reported)	fine-tuning
Rainbow Padding / VoidPadding	yes	yes	fine-tuning
Block diffusion	within a block	within a block only	model training
Caching methods	yes	approximate (cached or delayed states)	mostly none

The reported gains in these rows are measured on different backbones, tasks and metrics, so the table is a structural map, not a leaderboard.

What we measure, and what we don’t 📏

The survey groups inference costs into four pillars that are related but not interchangeable:

1. Compute: FLOPs, or proxies such as the effective-token ratio, the fraction of a canvas that becomes real content.
2. Throughput: tokens per second, under a stated batch size and hardware.
3. Latency: wall-clock time per response, including its variance across queries.
4. Memory: peak device memory for activations, cached state and the canvas itself.

A method can improve throughput while making latency spikier through occasional expensive fallbacks, or cut FLOPs while raising peak memory by caching more. Papers rarely report all four for the same method, and latency variance, arguably the most user-facing, is the least consistently reported.

Open problems 🔭

The survey closes with concrete experiments rather than claims. The ones I find most interesting:

• Predictive initialization. Use a Predict-then-Diffuse estimate to seed DAEDAL’s or SmartCrop’s starting canvas, and test whether a better start reduces probes, expansions and retries without becoming brittle when the estimate is wrong.
• Calibration under shift. Evaluate length predictors across answer lengths, task families and domains, reporting under-allocation, padding and retry rates, not just average error.
• Purpose-built length objectives. Check whether an objective that supervises the termination position or the allocation directly beats EOS fine-tuning.
• Reasoning coherence under canvas edits. Check whether expansion, contraction or remasking changes the structure of an answer in mathematics, code and multi-step reasoning.
• Cache- and sparsity-aware resizing. Find out whether canvas savings survive the cost of maintaining caches and sparsity patterns on real hardware.
• A shared evaluation protocol with common backbones, tasks and length distributions, reporting quality, effective-token ratio, latency and its variance, throughput, peak memory, retries and resize events together.

Limits of the survey ⚠️

This is a fast-moving, preprint-heavy area, and the ranking of methods by reported gains will shift. The survey centres LLaDA and Dream 7B because they dominate the length-adaptation literature, and leaves training recipes, tokenizers, multimodal models and RL for dLLMs out of scope. Most importantly, no independent re-benchmarking was done: every comparison is bounded by what the original papers report.

Conclusion 🎯

Fully parallel diffusion language models offer a structural alternative to left-to-right generation, and the fixed canvas is the constraint that most limits it. Block diffusion buys length flexibility with causal dependencies across blocks. Single-canvas methods instead adapt the allocation while keeping whole-canvas attention, by predicting the length, probing the model, editing the canvas during denoising or training the model to know when to stop.

”Bidirectional” describes the attention pattern that is available, not a guarantee that committed tokens get revised or that the model uses context from the right. How much of the fully parallel, bidirectional promise survives in real serving systems is an empirical question, and the survey’s open problems are my attempt to make it a testable one.

References 📚

Cited in this post:

• Arriola, M., Gokaslan, A., Chiu, J. T., Yang, Z., Qi, Z., Han, J., Sahoo, S. S., and Kuleshov, V. (2025). Block diffusion: Interpolating between autoregressive and diffusion language models. International Conference on Learning Representations (ICLR).
• Bertolani, T., Bucciarelli, D., Zini, L., Cornia, M., and Baraldi, L. (2026). Diffusion language models: An experimental analysis. arXiv:2606.19475.
• Bouhedja, W., Mohamed, A., and Shang, G. (2026). CARVE: Verified expansion for variable-length generation in diffusion language models. Findings of EMNLP 2026.
• Chen, Z., Fang, G., Ma, X., Yu, R., and Wang, X. (2026). DMax: Aggressive parallel decoding for dLLMs. arXiv:2604.08302.
• Cheng, Z., Jia, R., Li, J., Yang, G.-W., Guo, M.-H., and Hu, S.-M. (2026). Improving variable-length generation in diffusion language models via length regularization. arXiv:2602.07546.
• Gao, Y., Ji, Z., Wang, Y., Qi, B., Xu, H., and Zhang, L. (2025). Self speculative decoding for diffusion large language models. arXiv:2510.04147.
• Huang, J., Zhang, Y., Yang, Y., Huang, B., Qi, B., Liu, D., and Zhang, L. (2025). Mask tokens as prophet: Fine-grained cache eviction for efficient dLLM inference. arXiv:2510.09309.
• Karimi Monsefi, A., Bhendawade, N., Ciosici, M. R., Culver, D., Zhang, Y., and Belousova, I. (2025). FS-DFM: Fast and accurate long text generation with few-step diffusion language models. arXiv:2509.20624.
• Kim, B., Jeon, D., Kim, D., Jeung, W., and No, A. (2025). Rainbow padding: Mitigating early termination in instruction-tuned diffusion LLMs. arXiv:2510.03680.
• Lee, Y., Dan, S., Lee, J., Park, J., and Ahn, J. H. (2026). DyLLM: Efficient diffusion LLM inference via saliency-based token selection and partial attention. International Conference on Machine Learning (ICML).
• Li, J., Dong, X., Zang, Y., Cao, Y., Wang, J., and Lin, D. (2026a). Beyond fixed: Training-free variable-length denoising for diffusion large language models. International Conference on Learning Representations (ICLR).
• Li, P., Muhtar, D., Chen, T., Yin, L., and Liu, S. (2026b). Why diffusion language models struggle with truly parallel (non-autoregressive) decoding? arXiv:2602.23225.
• Liu, C., Fan, Z., Yang, K., and Lamb, A. (2026). VoidPadding: Let [VOID] handle padding in masked diffusion language models so that [EOS] can focus on semantic termination. arXiv:2606.17999.
• Liu, Z., Yang, Y., Zhang, Y., Chen, J., Zou, C., Wei, Q., Wang, S., Zhu, Y., and Zhang, L. (2025). dLLM-Cache: Accelerating diffusion large language models with adaptive caching. arXiv:2506.06295.
• Long, L., Huang, Y., Bai, S., Gong, R., Zhang, J., Zhou, A., and Yang, J. (2026). Focus-dLLM: Accelerating long-context diffusion LLM inference via confidence-guided context focusing. arXiv:2602.02159.
• Lu, X., Guo, S., Ma, Z., Zhang, J., Liu, J., Guo, J., Che, H., and Guo, S. (2026). PACE-dLLM: Elastic block decoding via confidence cliff estimation for diffusion language models. arXiv:2609.26249.
• Ma, L., Cui, Y., Han, K., and Wang, Y. (2026). Diffusion in diffusion: Reclaiming global coherence in semi-autoregressive diffusion. arXiv:2601.13599.
• Ma, X., Yu, R., Fang, G., and Wang, X. (2025a). dKV-Cache: The cache for diffusion language models. Advances in Neural Information Processing Systems (NeurIPS).
• Ma, Y., Du, L., Wei, L., et al. (2025b). dInfer: An efficient inference framework for diffusion language models. arXiv:2510.08666.
• Nguyen-Tri, Q., Ranjan, M., and Shen, Z. (2026). Flash-dLLM: IO-aware KV caching and parallel decoding for fast, memory-efficient diffusion LLMs. arXiv:2609.26796.
• Nie, S., Zhu, F., You, Z., Zhang, X., Ou, J., Hu, J., Zhou, J., Lin, Y., Wen, J.-R., and Li, C. (2025). Large language diffusion models. Advances in Neural Information Processing Systems (NeurIPS).
• Rossi, V., Cirò, G., Beltrame, D., Gandolfi, L., Röttger, P., and Hovy, D. (2026). Diffusion language models are natively length-aware. arXiv:2603.06123.
• Rottoli, M., Roy, S., and Paraboschi, S. (2026). Predict-then-Diffuse: Adaptive response length for compute-budgeted inference in diffusion LLMs. International Joint Conference on Neural Networks (IJCNN). arXiv:2605.04215.
• Rozental, A. (2026). Zonkey: A hierarchical diffusion language model with differentiable tokenization and probabilistic attention. arXiv:2601.21768.
• Song, X., Wang, Z., Gao, M., and Tang, H. (2026). SemBlock: Semantic boundary dynamic blocks for diffusion LLMs. arXiv:2606.04964.
• Song, Y., Liu, X., Li, R., Liu, Z., Huang, Z., Guo, Q., He, Z., and Qiu, X. (2025). Sparse-dLLM: Accelerating diffusion LLMs with dynamic cache eviction. arXiv:2508.02558.
• Sun, S., Li, Y., Liu, R., et al. (2026). PSD: Pushing the Pareto frontier of diffusion LLMs via parallel speculative decoding. arXiv:2605.15609.
• Wang, K., Jiang, Z., Feng, H., Zhao, W., Liu, L., Li, J., Lan, Z., and Lin, W. (2025a). CreditDecoding: Accelerating parallel decoding in diffusion large language models with trace credit. arXiv:2510.06133.
• Wang, Z., Fang, G., Ma, X., Yang, X., and Wang, X. (2025b). SparseD: Sparse attention for diffusion language models. arXiv:2509.24014.
• Wu, C., Zhang, H., Xue, S., Liu, Z., Diao, S., Zhu, L., Luo, P., Han, S., and Xie, E. (2025). Fast-dLLM: Training-free acceleration of diffusion LLM by enabling KV cache and parallel decoding. arXiv:2505.22618.
• Xiao, Z., Hao, Z., Guo, J., Luo, Y., Liu, J., Xu, J., and Hu, H. (2026). Streaming-dLLM: Accelerating diffusion LLMs via suffix pruning and dynamic decoding. arXiv:2601.17917.
• Xu, H., Chen, S., Bei, Y., Chen, L., Yan, Y., Fu, D., He, J., and Tong, H. (2026). Predict, don’t iterate: Efficient adaptive-length infilling for diffusion language models. Conference on Empirical Methods in Natural Language Processing (EMNLP).
• Yang, J., Jiang, Y., and Shao, J. (2026). ρ-EOS: Training-free bidirectional variable-length control for masked diffusion LLMs. arXiv:2601.22527.
• Yang, Y., Wang, C., Wang, S., Wen, Z., Qi, B., Xu, H., and Zhang, L. (2025). Diffusion LLM with native variable generation lengths: Let [EOS] lead the way. arXiv:2510.24605.
• Ye, J., Xie, Z., Zheng, L., Gao, J., Wu, Z., Jiang, X., Li, Z., and Kong, L. (2025). Dream 7B: Diffusion large language models. arXiv:2508.15487.
• Zhan, Y., Hou, M., Zhang, W., and Gao, Z. (2026). Length-adaptive decoding for masked diffusion machine translation. Conference on Empirical Methods in Natural Language Processing (EMNLP).
• Zhou, C., Wang, C., Zhang, D., Tong, S., Wang, Y., Bates, S., and Jaakkola, T. (2025). Next semantic scale prediction via hierarchical diffusion language models. Advances in Neural Information Processing Systems (NeurIPS).
• Zhu, Z., Ren, F., Tan, Z., and Ma, K. (2026). ES-dLLM: Efficient inference for diffusion large language models by early-skipping. arXiv:2603.10088.

The survey cites further work, including other surveys of diffusion language models and of efficient dLLM inference.