Diffusion Meta-Prompting and Steering for Generalizable Foundation Model Adaptation

Prompt learning is a popular parameter-efficient method for adapting foundation models, but learned prompts are typically task-specific and fail to generalize to new classes, domains, or compositions of tasks. In this paper, we introduce a $\textbf{Diffusion Meta-Prompt (DMP)}$ model , a framework that models the distribution of learned prompts using diffusion models. DMP is trained only on a repository of previously learned prompts to synthesize new prompts conditioned on natural language task descriptions, without access to the task data. To improve the sampling stability, we introduce a test-time steering strategy for DMP, which uses the best training-selected prompt in the repository as a latent anchor during diffusion sampling, without retraining the DMP or accessing test classes. DMP improves generalization across classification, retrieval and text-to-image generation tasks, supports concept composition and negative prompting without explicit training. It reduces storage and inference costs by over 90% compared to prompt retrieval methods. For composite classification, DMP achieves upto $\textbf{2.0}$% average gain over prior meta-learning methods across 55 pairs of datasets with gains as high as $\textbf{8.5}$% on specific pairs such as Eurosat and Flowers. DMP also enhances cross-task generalization with $\sim$$\textbf{2-9}$% improvement for hierarchical classification task. We further provide a theoretical guarantee bounding the expected task loss of prompts sampled from a DMP.