TrnRT Open-Source Inference Engine for AWS Trainium

RunAnywhere, a YC W26 startup, officially released TrnRT (Trainium RunTime) today, an open-source inference engine purpose-built for AWS Trainium accelerators. The release arrives three weeks after the company began development and targets organizations seeking alternatives to GPU-based inference infrastructure.
Release date and availability
Officially launched on November 17, 2026, TrnRT is available now on GitHub under an open-source license. The project welcomes community contributions as RunAnywhere expands model support beyond its initial GLM-5.3 flash implementation. Sanchit, posting on behalf of the team, emphasized the early-stage nature of the release and the roadmap for additional model families.
Performance and architecture
TrnRT demonstrates substantial throughput on AWS trn2.48xlarge instances, processing 3.57 million tokens per minute on SemiAnalysis's AgentX benchmarks. The engine runs two concurrent instances of the GLM-5.3 flash model in FP8 precision, distributing workload across eight Trainium cores per box. A cache-aware router manages request distribution between the two instances, optimizing memory utilization and reducing latency.
The architecture reflects design choices specific to Trainium's memory hierarchy and interconnect topology. By maintaining two separate model replicas with independent KV caches, TrnRT avoids contention bottlenecks common in single-instance deployments serving high request volumes.
Model support and roadmap
At launch, TrnRT supports only GLM-5.3 flash, a compact model variant optimized for speed. RunAnywhere indicated plans to expand compatibility to additional model families but provided no specific timeline. The choice of GLM-5.3 flash as the initial target likely stems from its relatively small parameter count and existing FP8 quantization support, simplifying the bring-up process on new hardware.
Community contributors can examine the codebase to understand Trainium-specific optimizations, including custom kernels for attention operations and memory management strategies tailored to the chip's NeuronCore architecture. The open-source model allows enterprises to audit performance characteristics before committing to Trainium deployments.
Competitive context
TrnRT enters a crowded field of specialized inference engines, each targeting different hardware backends. While most inference optimization efforts focus on NVIDIA GPUs, AWS has invested heavily in Trainium as a cost-competitive alternative for training and inference workloads. An inference engine like TrnRT removes a friction point for teams evaluating Trainium by providing pre-built serving infrastructure rather than requiring custom integration work.
The 3.57 million tokens per minute benchmark on AgentX positions TrnRT competitively for agentic workloads requiring high throughput with moderate batch sizes. Organizations running multi-turn conversational agents or batch document processing pipelines may find the price-performance ratio attractive compared to equivalent GPU configurations.
What this means
TrnRT's release lowers the barrier to adopting AWS Trainium for LLM inference, particularly for teams already committed to AWS infrastructure. The open-source approach accelerates ecosystem development around Trainium by providing reference implementations and performance baselines. As RunAnywhere expands model support and the community contributes optimizations, TrnRT could emerge as a standard serving layer for Trainium deployments, challenging GPU dominance in certain inference segments where cost and integration simplicity outweigh raw performance.
on Emergent today





