club-3090

Why we included this project

Anyone running one or two high-VRAM consumer cards, whether for a homelab or as a dev backend, can skip the usual fiddly part of serving modern open-weight LLMs: this repo collects working configs instead of leaving you to tune engines yourself. It bundles validated Docker Compose setups for vLLM, llama.cpp, and ik_llama, all exposing an OpenAI-compatible API on localhost, and it is honest about the trade-offs. The measured benchmarks tell you which configuration crashes under real tool-using agent workloads and which one fills a 200K context window cleanly, and the FAQ spells out per-card gotchas for the 3090, 4090, and 5090. Helper scripts and an interactive wizard walk you through downloading and verifying model weights, picking a variant for your GPU count, booting it, sanity-testing the endpoint, and running benchmarks. If you want a reproducible local inference baseline without tuning engines from scratch, the curated defaults and the hardware-aware picker genuinely save time.