Baidu Opens 280B ERNIE 5.5 Flash Base MoE With 14B Active

Baidu has opened access to ERNIE-5.5-Flash-Base, a 280-billion-parameter sparse Mixture-of-Experts language model designed specifically as a foundation for domain adaptation and research. With 14 billion parameters activated per token, the model was pre-trained on 33.5 trillion tokens and supports a 128,000-token context window out of the box, extending to one million tokens for fine-tuning. Officially released on November 5, 2026, the weights represent the checkpoint at the end of pre-training, before any instruction tuning or reinforcement learning.

Architecture and Efficiency Design

ERNIE 5.5 Flash Base features 43 Transformer layers with a hidden size of 4,096 and 64 attention heads. The architecture prioritizes efficiency at every level, from attention mechanisms to the output head. The model uses a hybrid attention system combining Multi-head Latent Attention (MLA) and Heavily Compressed Attention (HCA). Only one layer in seven maintains token-level access to the full context through MLA, while remaining layers compress every 128 tokens into a single cached entry. This design reduces KV cache requirements while preserving access to the entire sequence.

Layer-specific head dimensions allocate compute budget according to cost: HCA layers use wide 512-dimension heads since their compressed attention is cheap, while MLA layers use narrower 256-dimension heads. Full-context MLA layers implement sparse attention through a Dynamic Sparse Attention (DSA) mechanism that scores past positions and computes core attention only over top-scoring positions, keeping per-query cost constant as context grows.

Multi-Router MoE and Advanced Training Techniques

All layers except the first use a sparse MoE feed-forward block with a novel multi-router design. Instead of relying on a single router with hard top-k selection, ERNIE 5.5 Flash employs several independently initialized sigmoid routers that score experts jointly. This architecture addresses the weak correlation between single-router scores and actual expert utility, particularly in middle and deeper layers. Multiple routers develop distinct preferences, making routing less sensitive to noisy features and distribution shifts while maintaining the same number of activated experts.

The model replaces standard residual connections with mHC (multi-stream Highway Connection), where each layer reads from and writes to four parallel residual streams through a constrained mixing matrix. The output head uses MultiMax, a learnable piecewise temperature system that suppresses implausible tokens while preserving multiple plausible next tokens, significantly lowering pre-training loss with negligible overhead.

ECHO and Speculative Decoding

ERNIE 5.5 Flash implements ECHO (Efficient Contextual Head Optimization), which reuses the backbone's final layer to predict future tokens rather than attaching separate multi-token prediction modules. This design means the loss on future tokens directly trains the same parameters producing the next token, encouraging the model to anticipate sequence direction. The architecture delivers clear gains on reasoning and code tasks. The same layer serves as the draft model for speculative decoding and generalizes beyond its single-step training to draft multiple tokens with higher acceptance rates than auxiliary prediction modules.

Pre-Training Corpus and Infrastructure

The 33.5 trillion token pre-training corpus emphasizes capabilities difficult to add post-training, including repository-scale code, long-context understanding, and multilingual mathematical reasoning. The model uses a new BPE tokenizer with approximately 200,000 vocabulary entries, tokenizing numbers right-to-left in groups of up to three digits so digits with the same place value align across different-length numbers.

Baidu built two infrastructure components for this training run: a distributed Muon optimizer efficient across thousands of GPUs, and FlashMask-4, an attention kernel supporting head dimensions up to 576 for query/key and 512 for value. On SM100 GPUs, FlashMask-4 delivers up to 175 percent faster forward passes and 74 percent faster backward passes compared to baseline implementations. The training system extends context length gradually to 128,000 tokens during pre-training, with one million token support enabled for fine-tuning and inference.

Release Date and Availability

ERNIE-5.5-Flash-Base was officially released on November 5, 2026, through Baidu's research portal. The release provides the raw pre-training checkpoint without mid-training or post-training modifications, positioning it as a starting point for instruction tuning, reinforcement learning research, and domain-specific adaptation. Baidu frames this release as reducing the barrier to large-scale pre-training so developers can allocate resources toward capabilities and training methods specific to their applications.

What This Means

By releasing a 280-billion-parameter sparse MoE at the pre-training checkpoint stage, Baidu addresses the high cost of foundation model development while maintaining competitive efficiency through architectural innovations. The multi-router MoE design and hybrid attention system demonstrate that efficiency gains need not come at the expense of capability, particularly for long-context and reasoning tasks. For researchers and enterprises pursuing domain adaptation, ERNIE 5.5 Flash Base offers a starting point with 33.5 trillion tokens of pre-training already complete, allowing specialized training budgets to focus on task-specific behaviors rather than general language understanding. The one million token fine-tuning support positions the model for applications requiring extensive context, from repository-level code analysis to multi-document reasoning workflows.