Medusa

Why we included this project

Most token-generation speedups lean on a second, smaller draft model that has to be trained, tuned, and kept in sync. Medusa sidesteps that entirely: you train a few extra lightweight decoding heads on the model you already have, leaving the original weights untouched and your distributed setup about the same. That means less operational overhead and no extra model to babysit, and the authors' technical report shows 2.2-3.6x speedups on several common LLMs. It works for both greedy and sampling-based generation, so it's a practical option for teams serving fine-tuned models who want lower latency without changing their serving stack.