prompt-lookup-decoding

Why we included this project

Summarization, document QA, multi-turn chat, and code editing all share a quirk: the model spends much of its output copying phrases or code straight from the input. Prompt Lookup Decoding exploits that by replacing the usual draft model in speculative decoding with simple n-gram matching against the prompt. The matching continuation becomes the candidate tokens, verified in a single forward pass, so output quality stays the same while throughput typically improves 2x to 4x on these input-grounded tasks. The reference implementation is small enough to read in one sitting, and the technique has since been built into both Hugging Face transformers and vLLM, where you enable it with a flag instead of maintaining custom code. It works with any decoder model, greedy or sampling, and needs no external datastore.