Batch size is a machine learning hyperparameter that determines how many training samples a model processes in one forward and backward pass before updating its weights. It directly affects memory use, training speed, and how stable or noisy the learning updates are. The right setting depends on the model, data, hardware, and the goal of the training run.
Batch Size
What is batch size in machine learning?
Batch size is the number of training samples a machine learning model processes before it updates its weights. In one batch, the model makes predictions in a forward pass, measures error, sends that error backward to calculate gradients, then adjusts its weights. It applies to training, not the number of test samples.
How batch size works during training
Training repeats this cycle until the model has learned from the dataset several times.
- Shuffle or select training samples.
- Load one batch into memory.
- Run predictions and calculate the error.
- Backpropagate gradients that show how to reduce error.
- Update the model weights once.
- Repeat with the next batch.
For example, 10,000 samples with a batch size of 100 create 100 training steps per epoch. Training for 10 epochs produces 1,000 weight updates.
Batch size, epoch, iteration, and global batch size
These terms describe different parts of a training run.
| Term | Meaning |
|---|---|
| Batch size | Samples processed before one update. |
| Epoch | One pass through the training dataset. |
| Iteration or step | One batch and usually one weight update. |
| Global batch size | Per-device batch size multiplied by device count, then multiplied by gradient accumulation steps. |
Benefits of a larger batch size
Larger batches can make efficient use of accelerators.
- They may improve GPU or TPU throughput.
- They create fewer updates per epoch.
- They usually produce less variable gradient estimates.
- They can shorten elapsed training time, though not always time to desired model quality.
Limits of a larger batch size
A batch that is too big can create new problems.
- It can exhaust accelerator memory and cause out-of-memory errors.
- It reduces the number of learning updates in each epoch.
- It can change optimization behavior and validation quality.
- It often requires learning-rate retuning rather than a simple size increase.
What should my batch size be?
There is no universal best value. Powers of two are convenient for hardware, not a requirement.
- Start with a size that reliably fits memory.
- Record throughput and validation results as a baseline.
- Increase the size gradually.
- Compare training speed and validation quality.
- Retune the learning rate when results change.
- Keep the smallest batch that meets the training objective.
Training batch size versus inference and API batching
The same word describes different technical workflows.
| Context | Primary concern |
|---|---|
| Training | Memory, learning behavior, and convergence. |
| Inference | Latency for each user versus total throughput. |
| API batching | Lower request overhead while handling payload limits, retries, ordering, and partial failures. See Batch API. |
Using batch size in AI application development
Batch size also matters when an AI app processes imports, background jobs, or bulk model requests. Teams should test realistic batch limits and failure handling as part of app design and development. Emergent helps teams build and test full-stack AI applications where these workload choices can be validated before release.
Start Building
on Emergent today
on Emergent today
Start Building
