Mid-training is a machine learning training stage that adapts a broadly pre-trained model toward targeted capabilities before final post-training. It uses carefully selected data and objectives to reduce the gap between general knowledge and tasks such as coding, reasoning, long-context use, or domain-specific assistance.
Mid-training
What Is Mid-Training?
Mid-training is an intermediate machine learning stage that adapts a broadly pre-trained model for targeted capabilities before final post-training. It sits between large-scale pre-training and methods such as supervised fine-tuning or reinforcement learning. Rather than continuing general pre-training, it changes the data mix and learning goals for capabilities such as coding, reasoning, long-context work, or tool use.
How Mid-Training Fits Between Pre-Training and Post-Training
Each stage has a different purpose, although their boundaries can overlap.
| Stage | Purpose | Typical data and objective | Expected outcome |
|---|---|---|---|
| Pre-training | Build broad language and knowledge capabilities | Large, diverse text and code collections; predict missing or next tokens | A capable general-purpose base model |
| Mid-training | Bridge general ability and a target capability | Carefully mixed general and specialist data; continued pre-training or targeted objectives | Stronger domain foundations with less loss of broad ability |
| Post-training | Shape task behavior, helpfulness, and safety | Examples, preference data, or reward signals through supervised fine-tuning, alignment training, or reinforcement learning | A model that follows instructions and meets product requirements |
How Mid-Training Works
Teams train and test for the new goal while checking capabilities they need to retain.
- Define a measurable target, such as reliable API tool calling or code assistance.
- Audit the starting model with general and target-task evaluations.
- Curate specialist data while retaining high-quality general data.
- Choose objectives for the target, such as long-context examples or tool traces.
- Run controlled experiments with tracked datasets and model versions.
- Evaluate target gains, general retention, safety, and failure cases before post-training.
Benefits of Mid-Training
Sound data and evaluations can make adaptation more deliberate.
- It can smooth the shift from broad data to specialized technical data.
- It can improve foundations for coding, domain language, reasoning, and tool use.
- It can support long-context behavior before instruction-level tuning.
- It can preserve broad capabilities better than training only on narrow data.
Practical Limits and Risks
Mid-training does not replace careful evaluation or safety work.
- Narrow or unbalanced data can cause catastrophic forgetting and over-specialization.
- Low-quality, duplicated, private, or improperly licensed data can create legal and reliability problems.
- Benchmark gains may not transfer to real user tasks or production integrations.
- Experiments require substantial compute, dataset governance, and repeatable evaluations.
- Final task testing, red-teaming, and post-training remain necessary.
A Practical Mid-Training Example for Software Engineering
For codebase-aware assistance, a team could mix clean source code, API documentation, issue-to-fix examples, and tool-call traces with general technical text. Holdout tests should check target behavior, such as selecting the correct repository tool, and retained behavior, such as clearly explaining an unfamiliar error. This helps prevent narrow coding gains from hiding losses in reasoning or instruction following.
Mid-Training and Emergent
Model providers usually manage training stages such as mid-training. Product teams can focus on behavior, integrations, and evaluation workflows. With an AI app builder, teams can create and test AI-powered software, while an AI agent builder can help turn reliable tool use into a product experience. Emergent helps teams ship full-stack AI applications while keeping testing and user outcomes central.
Start Building
on Emergent today
on Emergent today
Start Building
