HomeNews

Apgar Catches Agent Regressions Without LLM Judge Costs

Saurabh Anand
Saurabh Anand
•
Oct 30, 2026 11:43 AM
•
0
 min read
Select Emergent as your Preferred news source
Apgar Catches Agent Regressions Without LLM Judge Costs

💡 TL;DR

  • Oimo AI released Apgar, an open-source tool that detects AI agent regressions in production without expensive LLM judges.
  • The system analyzes Langfuse trace data to catch failures that offline evaluations and sampling-based judges miss in real user sessions.
  • Financial AI agent builders can now monitor release health across full production traffic instead of small sampled slices.

Oimo AI has released Apgar, an open-source release health checker designed to detect AI agent regressions in production environments without relying on expensive LLM judges. The tool addresses a critical gap for teams building production agents: catching failures that slip past offline evaluations and sampling-based quality checks. Officially launched on October 29, 2026, Apgar monitors Langfuse trace data to identify when code changes degrade agent performance across real user sessions.

Why Traditional Agent Testing Falls Short

Production AI agents face a testing challenge that offline evaluations cannot solve. Oimo AI's engineering team found that their offline evals covered only a fraction of the question types and session patterns real users brought to their financial agents. Running an LLM judge across all production traffic proved prohibitively expensive, forcing the team to sample only a small slice of sessions. This sampling approach allowed regressions to reach users undetected, creating blind spots in release quality.

The problem intensifies as agent complexity grows. Harness changes, prompt updates, and model version switches can introduce subtle failures that only surface under specific user conditions. Without full-traffic monitoring, teams must choose between cost and coverage, a trade-off that leaves production quality to chance.

How Apgar Monitors Release Health

Apgar integrates directly with Langfuse, extracting release metadata from production traces and analyzing agent behavior across deployments. The tool compares performance metrics between releases, flagging statistically significant changes that indicate regressions. Key capabilities include:

  • Automatic release detection from trace metadata without manual tagging
  • Statistical comparison of success rates, latency, and error patterns across versions
  • Full-traffic analysis that eliminates sampling bias and cost constraints
  • Lightweight architecture that processes trace data without invoking additional LLM calls

The system focuses on observable metrics rather than subjective quality judgments, trading nuanced semantic evaluation for comprehensive coverage. This approach catches hard failures like increased error rates, timeout spikes, or tool-use breakage that LLM judges might miss or excuse.

Release Date and Availability

Apgar officially released on October 29, 2026, as an open-source project available on GitHub. Oimo AI built the tool to solve internal monitoring needs for their financial agent platform before making it public. The release includes integration documentation for Langfuse users and deployment guides for teams running production agents at scale.

Early adopters can implement Apgar alongside existing evaluation pipelines, using it as a safety net that complements rather than replaces LLM judges. The tool runs independently of agent inference, adding monitoring without affecting production latency or token costs.

Use Cases for Production Agent Teams

Financial services teams building AI agents represent the primary target audience, but the tool applies broadly to any production agent deployment. Common scenarios include detecting regressions after prompt updates, catching tool integration failures that only surface under load, and monitoring model version swaps for unexpected behavior changes. Teams operating call center agents or business automation systems face similar blind spots that Apgar addresses.

The system proves especially valuable for teams already invested in Langfuse observability infrastructure. Rather than building custom monitoring logic or paying for expanded LLM judge coverage, they can layer Apgar's statistical analysis on existing trace data. This approach extends monitoring reach without expanding evaluation budgets.

What This Means

Apgar represents a pragmatic middle path between expensive semantic evaluation and no production monitoring at all. By focusing on statistical health signals rather than subjective quality scores, it gives agent teams full-traffic visibility into release impacts. The open-source release lowers barriers for smaller teams building production agents, offering an alternative to enterprise observability platforms. As AI agent deployments scale beyond experimental phases, tools that catch regressions without per-request LLM costs become essential infrastructure for maintaining production reliability.

About the writer

Saurabh Anand Rai is the Head of Product at Emergent, where he leads product strategy and innovation for AI-powered tools that help people build software faster.

Start Building
on Emergent today
Try Emergent