Fourier-Enhanced Fine-Tuning Vision Language Models Using PEFT-VFPT

Learn about Visual Fourier Prompt Tuning (VFPT) in this 27-minute technical presentation that explores an innovative approach to fine-tuning large-scale Transformer-based vision models. Dive into how VFPT incorporates Fast Fourier Transform (FFT) into prompt embeddings to effectively handle both spatial and frequency-domain information. Understand the solution to performance limitations in parameter-efficient fine-tuning (PEFT) methods, particularly when dealing with significant differences between pretraining and fine-tuning datasets. Explore how Fourier-transformed prompts enhance model adaptability while maintaining minimal trainable parameters compared to traditional fine-tuning approaches. Discover the practical implementation that preserves the Transformer's original architecture by modifying only prompt embeddings through FFT, eliminating the need for additional adapters or layers. Examine empirical evidence demonstrating VFPT's superior performance over conventional fine-tuning and competing PEFT methods, especially in tasks with substantial data distribution variations.