#137 · Primary category： Inference & Local Deploy
qwen38-27b-rtx3090
Run Qwen3.8-27B on a single RTX 3090 with vLLM: ~1000 tok/s concurrent, 114-124 tok/s single-user, 150k+ context via MTP speculation, int8 GEMMs, and calibrated int4 quantization.
Project last updated：08/27/26
GitHub Stars
897
Forks
113
Contributors
8
License
Apache-2.0
Why we included this project
Serving a 27-billion parameter model from a single 24 GB consumer GPU usually means doing the quantization, patching, and memory tuning yourself. This project does that work up front: it publishes a prebuilt Docker image that downloads and requantizes the model, applies a set of vLLM patches, and exposes an OpenAI-compatible API with key auth, so one container command gets you a working server. Two launch profiles cover a handful of interactive chat users and a high-concurrency batch backend, and the README documents what each actually delivers on an RTX 3090: around 120 tokens per second for single-stream chat with speculative decoding, and roughly 1,000 tokens per second aggregate decode at 64 concurrent requests. A CI gate runs the verification script before each image is pushed, so the latest tag is a stack the maintainers have tested. If you own one high-end card and want to self-host a modern Qwen model rather than rent capacity, the benchmarks and requant scripts spell out exactly what each setting buys.
Articles for this project
No articles for this project yet.
To suggest a topic or contribute an article, contact us.
Related projects in this category
ollama
Get up and running with Kimi-K2.6, GLM-5.2, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
llama.cpp
LLM inference in C/C++
vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
gpt4all
GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
LocalAI
LocalAI is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required.