ReGain: Restoring Subject Fidelity in Personalization on Synthetic Images

Key points

• Increasingly, these images come from a diffusion model rather than a camera.1
• We show that fine-tuning on such synthetic images degrades subject fidelity, producing oversaturated color and excess high-frequency detail.1
• We propose ReGain, a training-free correction applied at sampling time that measures how much each frequency band of the guidance is inflated relative to the base model and scales that band down accordingly.1
• On Stable Diffusion v1.5, ReGain closes 51-64% of the subject-fidelity gap to the model personalized on real photos, as measured by DINO, DINOv2 and CLIP-I.1

Sources (1)

• [1]ReGain: Restoring Subject Fidelity in Personalization on Synthetic Images
  Hugging Face Daily Papers · Sep 30, 12:00 AM

  Text-to-image diffusion models are personalized to a subject by DreamBooth fine-tuning on a handful of its images.

  Increasingly, these images come from a diffusion model rather than a camera.

Extractive summary: sentences quoted from the sources.