HomeNews

TrnRT Open-Source Inference Engine for AWS Trainium

Naman
Naman
•
Nov 18, 2026 3:05 AM
•
0
 min read
Select Emergent as your Preferred news source
TrnRT Open-Source Inference Engine for AWS Trainium

💡 TL;DR

  • RunAnywhere open-sourced TrnRT, a new inference engine built specifically for AWS Trainium hardware acceleration chips.
  • The engine currently serves GLM-5.3 flash models in FP8 precision, processing 3.57 million tokens per minute on benchmark tests.
  • TrnRT runs two model instances per trn2.48xlarge box using cache-aware routing across eight Trainium cores.

RunAnywhere, a YC W26 startup, officially released TrnRT (Trainium RunTime) today, an open-source inference engine purpose-built for AWS Trainium accelerators. The release arrives three weeks after the company began development and targets organizations seeking alternatives to GPU-based inference infrastructure.

Release date and availability

Officially launched on November 17, 2026, TrnRT is available now on GitHub under an open-source license. The project welcomes community contributions as RunAnywhere expands model support beyond its initial GLM-5.3 flash implementation. Sanchit, posting on behalf of the team, emphasized the early-stage nature of the release and the roadmap for additional model families.

Performance and architecture

TrnRT demonstrates substantial throughput on AWS trn2.48xlarge instances, processing 3.57 million tokens per minute on SemiAnalysis's AgentX benchmarks. The engine runs two concurrent instances of the GLM-5.3 flash model in FP8 precision, distributing workload across eight Trainium cores per box. A cache-aware router manages request distribution between the two instances, optimizing memory utilization and reducing latency.

The architecture reflects design choices specific to Trainium's memory hierarchy and interconnect topology. By maintaining two separate model replicas with independent KV caches, TrnRT avoids contention bottlenecks common in single-instance deployments serving high request volumes.

Model support and roadmap

At launch, TrnRT supports only GLM-5.3 flash, a compact model variant optimized for speed. RunAnywhere indicated plans to expand compatibility to additional model families but provided no specific timeline. The choice of GLM-5.3 flash as the initial target likely stems from its relatively small parameter count and existing FP8 quantization support, simplifying the bring-up process on new hardware.

Community contributors can examine the codebase to understand Trainium-specific optimizations, including custom kernels for attention operations and memory management strategies tailored to the chip's NeuronCore architecture. The open-source model allows enterprises to audit performance characteristics before committing to Trainium deployments.

Competitive context

TrnRT enters a crowded field of specialized inference engines, each targeting different hardware backends. While most inference optimization efforts focus on NVIDIA GPUs, AWS has invested heavily in Trainium as a cost-competitive alternative for training and inference workloads. An inference engine like TrnRT removes a friction point for teams evaluating Trainium by providing pre-built serving infrastructure rather than requiring custom integration work.

The 3.57 million tokens per minute benchmark on AgentX positions TrnRT competitively for agentic workloads requiring high throughput with moderate batch sizes. Organizations running multi-turn conversational agents or batch document processing pipelines may find the price-performance ratio attractive compared to equivalent GPU configurations.

What this means

TrnRT's release lowers the barrier to adopting AWS Trainium for LLM inference, particularly for teams already committed to AWS infrastructure. The open-source approach accelerates ecosystem development around Trainium by providing reference implementations and performance baselines. As RunAnywhere expands model support and the community contributes optimizations, TrnRT could emerge as a standard serving layer for Trainium deployments, challenging GPU dominance in certain inference segments where cost and integration simplicity outweigh raw performance.

About the writer

A growth marketer with varied interests and the proven ability to acquire new skills fast. Currently engrossed in all things AI.

Start Building
on Emergent today
Try Emergent