higress

Why we included this project

Running an LLM-backed service usually means talking to several model providers, domestic and international, each with its own request format. Higress sits in front of them as a cloud-native API gateway built on Istio and Envoy, exposing one protocol to your services so they don't hard-code vendor-specific calls. On top of plain proxying it adds model-level load balancing with fallback, token-based flow control, observability tuned for AI workloads, and response caching, the kinds of practical concerns that only surface once models hit production. Behavior lives in Wasm plugins written in Go, Rust, or JavaScript, so you can customize streaming and security handling without recompiling the gateway. Because it doubles as a general Kubernetes ingress and microservice gateway, it stays useful even as your AI traffic patterns evolve.