Oriel — AI & technology worth noticing, every day.

SFA: A Statistical Physics-Inspired Diffusion Transformer for Efficient and High-Quality Image Generation

Paper · https://arxiv.org/abs/2611.00027

arXiv:2611.00027v1 Announce Type: new Abstract: Diffusion transformers have achieved remarkable success in image generation, but accelerating attention while preserving generation quality remains a major challenge at high resolutions. Inspired by statistical physics and the widespread occurrence of scale-free laws in nature, we introduce Scale-Free Attention (SFA), which preserves fine local interactions while summarizing distant regions at progressively coarser scales. For $N$ tokens, mean pooling, token-count correction, and a shared softmax over a scale-free spatial hierarchy give SFA $O(N\log N)$ complexity in both forward and backward passes. Our Triton implementation achieves $357.3\times$ forward and $265.6\times$ backward attention speedups over FlashAttention at 262,144 tokens. After 40 epochs of PixelFlow training from scratch on ImageNet at $256\times256$ resolution, SFA achieves the best generation quality among five attention mechanisms, including full attention. Fr\'echet inception distance evaluated on 50,000 generated samples (FID-50k) improves from 13.25 for full attention to 12.51, and the advantage over full attention persists through 80 epochs. SFA also achieves the best generation quality among the compared efficient-attention methods on Flickr-Faces-HQ (FFHQ). Beyond training from scratch, Stable Diffusion 3.5 Medium with attention replaced by SFA retains the generation quality of its full-attention counterpart under matched fine-tuning budgets. We obtain similar results when fine-tuning pretrained PixelFlow on ImageNet. SFA establishes scale-free structure as an effective design principle for efficient, high-quality diffusion transformers.

First seen by Oriel on .

AI summary

SFA is a scale-free attention method for diffusion transformers that reduces attention complexity to O(N log N) while matching or improving image-generation quality in the reported evaluations.

  • SFA preserves fine local interactions and summarizes distant image regions at progressively coarser scales using mean pooling, token-count correction, and a shared softmax.
  • A Triton implementation reports 357.3× forward and 265.6× backward attention speedups over FlashAttention at 262,144 tokens.
  • After 40 epochs of PixelFlow training on ImageNet at 256×256, SFA reaches FID-50k 12.51 versus 13.25 for full attention; its advantage persists through 80 epochs.
  • SFA also leads the compared efficient-attention methods on FFHQ, and Stable Diffusion 3.5 Medium retains full-attention generation quality after replacement and matched-budget fine-tuning.

#diffusion transformers #attention #image generation #triton

Generated by gpt-5.6-sol. Check the original for details.

Aliases

SourceExternal IDURL
arXiv AI Papersoai:arXiv.org:2611.00027v1https://arxiv.org/abs/2611.00027 Read discussion