wise-ft

Why we included this project

Fine-tuning a large vision-language model like CLIP usually improves accuracy on the training data but can hurt performance when the model meets shifted distributions. WiSE-FT, the method implemented here, sidesteps that trade-off by interpolating the weights of the zero-shot and fine-tuned checkpoints with a mixing coefficient. The result keeps most of the in-distribution gain while restoring much of the zero-shot model's out-of-distribution robustness, and it does so with no extra compute during fine-tuning or inference. This is the reference implementation from the paper, and it's a compact, readable starting point if you want to apply the same weight-space ensembling trick to your own CLIP-style fine-tuning pipeline, especially when you care about robustness to distribution shift.