aici

Why we included this project

AICI is a runtime layer for controlling LLM output as it happens. You write small WebAssembly modules that run on the CPU while the GPU produces tokens, letting you enforce a strict grammar or coordinate several parallel generations against a shared plan without patching the inference engine. Controllers can be written in Rust, C, or even Python, and they are sandboxed, so you get speed and safety together. The project sits one level below higher-level libraries, which is why Guidance, LMQL, and the maintained llguidance can build on it; it also integrates with llama.cpp and HuggingFace Transformers. It is still labeled a prototype, so treat it as an experimental foundation for research or custom systems rather than a drop-in deployment tool.