Diffusion Meta-Prompting and Steering for Generalizable Foundation Model Adaptation

Key points

• Prompt learning is a popular method for adapting foundation models, but learned prompts are typically task-specific and fail to generalize to new classes, domains, or compositions of tasks.1
• Given a repository of previously learned prompts, DMP is trained and sampled without access to the original task examples or task losses, and synthesizes new prompts conditioned on natural language task descriptions.1
• To improve the sampling stability, we introduce a test-time steering strategy for DMP, which uses the best training-selected prompt in the repository as a latent anchor during diffusion sampling, without retraining the DMP or accessing test classes.1
• DMP improves generalization across classification, retrieval and text-to-image generation tasks, supports concept composition and negative prompting without explicit training.1

Sources (1)

• [1]Diffusion Meta-Prompting and Steering for Generalizable Foundation Model Adaptation
  arXiv (AI, ML, NLP, CV, robotics, multi-agent) · Oct 8, 01:26 AM

  In this paper, we introduce a Diffusion Meta-Prompt (DMP) model , a framework that models the distribution of learned prompts using diffusion models.

  Prompt learning is a popular method for adapting foundation models, but learned prompts are typically task-specific and fail to generalize to new classes, domains, or compositions of tasks.

Extractive summary: sentences quoted from the sources.