Hyperparameters are machine learning configuration settings that control how a model is trained and structured. Unlike model parameters, such as learned weights, they are chosen by developers or tuning tools rather than learned directly from training data. Their values affect training speed, accuracy, reliability, and the model's ability to work well on new data.
Hyperparameters
What Are Hyperparameters in Machine Learning?
Hyperparameters are machine learning configuration settings that control how a model is trained or structured. Developers set them, or use tuning software to select them, rather than the model learning them directly from training data. They are called hyperparameters because they govern the learning process that produces the model's ordinary parameters.
Choices such as learning rate and model size can affect accuracy, training time, and how well a model handles new data.
Hyperparameters vs. Model Parameters
Both influence a model's output, but they are created in different ways.
| Attribute | Hyperparameter | Model parameter |
|---|---|---|
| Set by | Developer or tuning process | Training algorithm |
| Changes during training | Usually remains fixed for a trial | Updates as data is processed |
| Example | Learning rate or tree depth | Neural network weight or bias |
Common Hyperparameter Examples
The relevant settings depend on the model type.
| Hyperparameter | What it controls |
|---|---|
| Learning rate | How large each training update is |
| Batch size and epochs | Data processed per update and training passes |
| Regularization strength | How strongly complexity is penalized |
| Tree depth and estimators | Complexity and number of trees |
| Layers, filters, kernel size | Neural network or CNN structure |
How Hyperparameter Tuning Works
Hyperparameter tuning compares candidate settings using data the model did not train on.
- Define a success metric, such as accuracy, error, or latency.
- Split data into training, validation, and final test sets.
- Choose practical ranges for the most important settings.
- Run trials with grid search, random search, or Bayesian optimization, which uses earlier results to select promising trials.
- Compare validation results and record code, data, and settings.
- Retrain the selected configuration, then evaluate it once on the untouched test set.
Benefits of Choosing Suitable Hyperparameters
Thoughtful settings improve more than a leaderboard score.
- They can improve predictive performance and training stability.
- They can reduce wasted compute and limit overfitting.
- Recorded settings make model builds easier to reproduce.
Practical Limits of Hyperparameter Tuning
Tuning is useful, but it is not a cure-all.
- Large search spaces can be expensive to explore.
- Repeated choices can overfit noisy validation results.
- It cannot fix poor data, data leakage, or an unsuitable model design.
How to Select Hyperparameters Responsibly
Start with documented defaults and tune the highest-impact settings first.
- Use sensible ranges, often logarithmic ranges for learning rates.
- Set a compute budget and track every experiment.
- Keep the test set untouched until the final evaluation.
- Do not assume 0.0001 is always a good learning rate. Its suitability depends on the model, optimizer, data, and batch size.
Hyperparameters in AI Application Development
In production software, hyperparameters should travel with versioned code, data, tests, and deployment records so model behavior can be rebuilt and checked. Teams building AI-enabled products can explore related concepts in the Emergent glossary and compare tools for creating applications in this guide to AI agent builders. Emergent helps teams build, test, and ship full-stack AI applications where repeatable model configuration supports reliable releases.
Start Building
on Emergent today
on Emergent today
Start Building
