punica

Why we included this project

Punica is aimed at teams that maintain many LoRA fine-tuned variants of a shared base model and want to serve them all without paying a separate memory and compute price for each one. It keeps a single copy of the pretrained weights in GPU memory and batches requests across all the adapters in one forward pass, using a custom CUDA kernel, SGMV, to apply the small LoRA matrices cheaply. The result is that a request for one fine-tuned variant barely slows the others, since the shared backbone still gets its usual batching benefit. Microbenchmarks in the paper show a substantial throughput gain over serving each adapter separately with common stacks like Hugging Face Transformers and vLLM, and the repo includes an interactive demo that runs several LoRA models side by side. Anyone operating many specialized low-rank adapters on one GPU should read the paper and test whether the kernel overhead and throughput trade-offs suit their workload.