streaming-llm

Why we included this project

Long-running conversations are where streaming LLMs usually break: the cache of past tokens balloons, and most models stop making sense once the text outgrows its training window. StreamingLLM sidesteps both problems with a compact trick, keeping the first few tokens as attention sinks so models like Llama-2 and MPT keep producing coherent replies from recent context without a KV-cache reset and without letting memory grow. The repo includes working code, evaluation scripts, and a streaming chatbot demo, so you can feel the behavior instead of taking the paper's word for it. The technique is already folded into TensorRT-LLM, HuggingFace Transformers, and Intel's extension for Transformers, which makes it a low-risk option for teams running long-lived or memory-constrained deployments.