Images Talk Back: Guiding MMDiTs by Suppressing Image-to-Text Feedback
Multimodal diffusion transformers (MMDiTs) allow the evolving image to update internal text representations through joint attention.
ProofPaper ↗
Key points
- These representations then guide subsequent image updates, raising a question: does this feedback help correct deviations from the prompt, or preserve them?
- We find that image-to-text (I2T) feedback can preserve the objects and attributes already represented in the evolving image, including errors.
- These findings motivate Semantic Feedback Guidance (SFG), which suppresses the influence of I2T feedback by guiding away from an auxiliary branch that strengthens this feedback and weakens text-to-image conditioning.
- Experiments across three image-generation backbones show gains in alignment and preference-model scores.
Sources (1)
- [1]Images Talk Back: Guiding MMDiTs by Suppressing Image-to-Text FeedbackarXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 9, 05:58 PM
Multimodal diffusion transformers (MMDiTs) allow the evolving image to update internal text representations through joint attention.
These representations then guide subsequent image updates, raising a question: does this feedback help correct deviations from the prompt, or preserve them?
Extractive summary: sentences quoted from the sources.
Before this
- Oct 9, 2026PhysForge: Material-Controllable 3D Reconstruction with Separable Components
- Oct 9, 2026Not All Dimensions Are Equal: Outlier-Aware Attention Linearization with Grouped Quadratic Kernels
- Oct 9, 2026Why DiT Models Underperform as Representation Learners without Long Skip Connections
- Oct 9, 2026Data-Adaptive Mahalanobis Metric Learning for Cross-Head Attention in Transformers
- Oct 9, 2026Learning Representations to Guide Pixel Diffusion Transformers
- Oct 8, 2026Deflating the Hessian: Rank-4 W4A4 Quantization for Multimodal Diffusion Transformers