Sturnus: Open-Source Latency-Based LLM Load Balancer

A new open-source tool called Sturnus has been released to address latency-based load balancing for LLM API traffic without the complexity of enterprise AI gateways. Officially launched on October 26, 2026, the project offers developers a lightweight alternative for routing requests across multiple LLM providers based on real-time performance metrics.

What Sturnus Does

Sturnus distributes API traffic across LLM providers by monitoring latency and availability in real time. The tool uses a probe-based system where a portion of traffic continuously tests alternate providers, allowing the load balancer to automatically shift requests back to recovered or faster endpoints. This approach eliminates manual intervention when provider performance changes or outages resolve.

The project is built as a single static binary with no external dependencies, designed to minimize both deployment complexity and security attack surface. This architecture contrasts with larger AI gateway frameworks that bundle multiple features and often require enterprise licensing for production use.

Why Developers Are Building This

The creator built Sturnus to solve a specific infrastructure gap: wanting latency-aware routing without adopting a full-featured AI gateway platform. Many existing solutions combine load balancing with observability, prompt management, and rate limiting, creating what some developers view as unnecessary complexity for teams that only need traffic distribution.

• Single binary deployment with no runtime dependencies
• Automatic failover and recovery without configuration changes
• Reduced security exposure compared to multi-feature gateways
• No vendor lock-in or enterprise plan requirements

Technical Implementation

The load balancer continuously measures response times from configured LLM providers and adjusts traffic distribution accordingly. When a provider experiences degraded performance or becomes unavailable, Sturnus gradually shifts load to healthier endpoints while maintaining a probe percentage on the degraded provider. Once latency improves, the system automatically restores traffic flow without manual intervention.

This dynamic routing model aims to optimize both response times and reliability across heterogeneous LLM infrastructure. Teams running multiple provider accounts or testing different model endpoints can use Sturnus to ensure requests route to the fastest available service at any given moment.

Release Date and Availability

Sturnus officially released on October 26, 2026, and is available as open-source software on GitHub. The project is in active production use by its creator, though broader adoption metrics have not been disclosed. Developers can download the static binary and configure it for their LLM provider stack without requiring cloud services or third-party accounts.

What This Means

Sturnus represents a growing trend toward specialized, single-purpose infrastructure tools in the LLM operations space. As organizations deploy AI models across multiple providers for redundancy and cost optimization, lightweight load balancing solutions that avoid enterprise gateway overhead become increasingly relevant. The project's focus on security minimalism and automatic recovery aligns with developer preferences for composable tooling over monolithic platforms.