Buoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM
Buoyant has released Kinference, an Apache 2.0 inference control plane for vLLM on Kubernetes. It adds declarative routing policies, KV-cache-aware load balancing and per-tenant metrics, and cut median TTFT 95% in Buoyant’s own benchmark. The post Buoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM appeared first on MarkTechPost .
阅读原文 · MarkTechPost本文由 SoccerDream AI 自动聚合索引，内容版权归原作者与 MarkTechPost 所有。
本站仅作技术动态聚合，不构成任何形式的建议。
本站仅作技术动态聚合，不构成任何形式的建议。