infinity

Why we included this project

If you're building a retrieval or RAG pipeline and want embeddings served on your own hardware, Infinity is a REST server that handles the inference plumbing for you. It serves text-embedding and reranking models, plus multimodal ones like CLIP and ColPali, behind an OpenAI-compatible API, so swapping it in for a hosted embedding endpoint is fairly painless. The engine runs on PyTorch, ONNX/TensorRT, and CTranslate2 backends, letting you match the backend to your hardware, from NVIDIA or AMD GPUs down to CPU-only machines. Dynamic batching and tokenization in dedicated worker threads keep throughput high under load, and one process can serve several models at once. For teams that need low-latency self-hosted embeddings, rerankers, or vision-language vectors, this is a practical drop-in option.