Oriel — AI & technology worth noticing, every day.

Rethinking in Spikes: Mitigating Hallucinations in MDLMs with Step-Aware Decoding

Paper · https://arxiv.org/abs/2611.12901

arXiv:2611.12901v1 Announce Type: new Abstract: Recent advancements in multimodal diffusion language models (MDLMs) have exhibited strong global modeling capabilities through iterative refinement. We observe that mask uncertainty follows an overall downward trend during iterative denoising, while a few denoising steps exhibit localized entropy spikes closely associated with hallucinations. We argue that anomalous steps can be identified by measuring the entropy deviation of each denoising step from its neighboring temporal window. We hypothesize that discrete token commitment discards the distributional information at masked positions, limiting the continued interaction between candidate semantics and the context during entropy spikes. With this goal, we present StepRefine, a plug-and-play decoding strategy that leverages semantic context to achieve reliable denoising. The core of our method lies in a step-aware switch between discrete denoising and latent refinement. The model employs probability-weighted embeddings at entropy spike steps to preserve diverse reasoning hypotheses and performs multiple rounds of continuous refinement. Meanwhile, a refinement controller monitors distributional stability to reduce unnecessary loops and switch the model back to standard discrete denoising. Through extensive experiments, StepRefine demonstrates significant hallucination mitigation across different MDLMs on multimodal benchmarks.

First seen by Oriel on .

AI summary

The paper presents StepRefine, a plug-and-play, step-aware decoding strategy that mitigates hallucinations in multimodal diffusion language models by applying latent refinement when denoising entropy spikes occur.

  • The authors associate localized mask-entropy spikes during iterative denoising with hallucinations and detect anomalous steps through entropy deviation from a neighboring temporal window.
  • At spike steps, StepRefine replaces discrete token commitment with probability-weighted embeddings, preserving multiple semantic hypotheses for continued interaction with context.
  • The method performs multiple rounds of continuous refinement before returning to standard discrete denoising.
  • A refinement controller monitors distributional stability to avoid unnecessary refinement loops, while experiments report hallucination reductions across different MDLMs and multimodal benchmarks.

#multimodal models #diffusion language models #hallucination mitigation #decoding

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
arXiv AI Papersoai:arXiv.org:2611.12901v1https://arxiv.org/abs/2611.12901 Read discussion