optillm

Why we included this project

OptiLLM runs as a proxy in front of any OpenAI-compatible endpoint, intercepting requests and applying one of more than 20 inference-time techniques before the answer comes back. That gives teams who cannot fine-tune a model a way to push harder on math and coding tasks, with repo examples reporting gains like +30 points on AIME 2025 from extra compute rather than retraining. You can start using it by pointing an existing OpenAI client at a local port and prefixing the model name with a slug such as `moa-` or `mcts-`, which makes it a convenient harness for comparing best-of-N, chain-of-thought reflection, MCTS, and multi-agent strategies against each other on the same task set. It also routes through LiteLLM, so the same techniques can be tested against GPT, Claude, or open-weight models without touching your application code.