Homeglossary

Dangerous Capability Evaluation

Dangerous capability evaluation is an AI safety assessment that tests whether a system can perform tasks linked to serious or catastrophic harm. It identifies whether a model has crossed a risk-relevant capability threshold before further training or deployment. Results inform safeguards and release decisions, but cannot prove that a model lacks every possible dangerous ability.

What Is Dangerous Capability Evaluation?

Dangerous capability evaluation is an AI safety assessment that tests whether a system can perform tasks linked to serious or catastrophic harm. It helps developers identify risk-relevant capabilities before further training, wider access, or deployment.

It tests high-consequence abilities rather than general usefulness. Instead of measuring coding accuracy or fluency alone, it asks whether combined abilities, tools, and persistence could enable harmful cyber activity, unsafe scientific assistance, deception, or uncontrolled autonomous action. Results inform release decisions but cannot find every risk.

A threshold is a pre-defined performance level that changes required safeguards. A system that occasionally completes a limited controlled task may need monitoring. One that reliably completes a complex task with little help may need restricted access, stronger security controls, or further safety work.

Why Dangerous AI Capabilities Matter

General-purpose AI can support beneficial work, but skills such as planning, code generation, research support, and persuasion can also reduce the time, expertise, or cost needed for harmful activities.

Capability is not intent. A model may not choose harmful actions, and users may never request them. However, misuse can come from people, connected tools, weak access controls, or unexpected autonomous behavior. Evaluation makes these possibilities visible before broader release.

Key Risk Areas These Evaluations Test

Categories are based on plausible threat models. Tasks must be controlled so testing produces safety evidence without creating harmful instructions.

Risk areaWhat evaluators testWhat a concerning result may mean
Cyber operationsWhether a model can find, explain, or chain weaknesses in authorized sandbox systems.It may substantially assist harmful actors with broad tools or unrestricted targets.
Biological and chemical assistanceWhether it adds meaningful expertise beyond safe public information in expert-reviewed scenarios.It could reduce barriers to dangerous research or misuse and need stricter information controls.
Autonomy and self-proliferationWhether it can plan across steps, use tools, recover from setbacks, and manage tasks with limited supervision.Long-running agents may create risks not visible in a single chat response.
Deception and persuasionWhether it can strategically mislead, evade oversight, or manipulate decisions in controlled role-based tests.Human review and monitoring may be less reliable than expected.
AI research accelerationWhether it can materially speed up advanced AI research, engineering, or experimentation.It could rapidly unlock other capabilities, requiring closer access and use controls.

How a Dangerous Capability Evaluation Works

A credible process defines what evidence will change access or safeguards before testing begins.

  1. Define a threat model with a harmful outcome, plausible actor, resources, and barriers.
  2. Set a capability threshold that requires a specific safety response.
  3. Create controlled tasks that represent the threat without enabling real-world harm.
  4. Set realistic access conditions, including tools, time, memory, and human assistance.
  5. Run repeated trials across prompts and task variations.
  6. Use qualified experts to judge realism, scoring, and practical significance.
  7. Analyze reliability, human effort, failure modes, and appropriate baselines.
  8. Connect findings to deployment controls, retesting, and decision owners.

Core Components of a Credible Evaluation

Good evaluations combine technical testing with disciplined measurement. The threat model explains why a capability matters, the task suite makes it observable, and scoring defines progress, success, and failure. Baselines may include an unassisted person, someone using ordinary search tools, or a less capable model.

Conditions must be documented because results can change with model version, system prompt, tools, retrieval sources, memory, time budget, and prompting strategy. Structured prompt evaluation helps ensure a weak prompt is not mistaken for absent capability. For tool-using systems, agent evaluation tests planning, tool use, and recovery from errors.

Evaluators should measure sustained performance, not one lucky success. Independent review, detailed records, and protected test materials make findings more trustworthy and comparable after updates.

Dangerous Capability Evaluation vs. Propensity and Alignment Evaluation

Capability and behavior are related but distinct questions. A sound safety case needs evidence about both.

Evaluation typeCore questionExample evidence
Dangerous capability evaluationCan the system perform a risk-relevant task under defined conditions?Reliable completion of an authorized task with specified tools and time.
Propensity evaluationIs the system likely to choose harmful, deceptive, or rule-evading behavior when alternatives exist?Behavior in scenarios involving conflicting goals, oversight, or incentives.
Alignment evaluationDoes the system follow intended constraints and remain dependable under pressure?Policy adherence, honest reporting, and resistance to manipulation.

A capable system with low harmful propensity may still need safeguards if users can direct misuse. A system with concerning behavior may create risk even with limited capabilities. Neither evaluation replaces the other.

Practical Uses in AI Development and Governance

Results matter most when they influence operational decisions across an AI system's lifecycle.

  • Set release gates for models, tool integrations, and agent features.
  • Check for capability changes during training and major updates.
  • Choose controls such as identity verification, rate limits, logging, restricted tools, or expert-only access.
  • Focus red-teaming on the most consequential risks.
  • Support a safety case documenting how risks were identified and managed.
  • Prepare incident response and post-release monitoring.
  • Give regulators, customers, and governance teams evidence for oversight decisions.

Benefits of Dangerous Capability Evaluations

Well-designed evaluations turn broad safety concerns into evidence leaders can use.

  • They detect concerning abilities before broader access or more powerful tools.
  • They support proportionate safeguards rather than treating all models as equally risky.
  • They improve accountability through documented thresholds, results, and actions.
  • They direct safety resources toward consequential gaps.
  • They provide repeatable evidence of capability changes over time.
  • They inform decisions about who can use a model and under what conditions.

Practical Limits and Challenges

Evaluations are necessary evidence, not a complete map of risk.

  • No finite task suite covers every harmful use, hidden capability, or future tool combination.
  • Performance may change after fine-tuning, new data, longer context windows, or external-system access.
  • Artificial tests may overstate or understate real-world risk.
  • Task design can reward benchmark gaming instead of meaningful competence.
  • Evaluators may lack specialist expertise, secure environments, or realistic access.
  • Publishing detailed tasks can expose sensitive information and create a dual-use problem.
  • Passing a test does not prove safety, only that capability was not demonstrated under those conditions.

Common Pitfalls and Better Practices

Weak design can create false reassurance. Better practice emphasizes realism, repeatability, and pre-planned decisions.

  • Test natural prompt variation, including reasonable clarification or reframing attempts.
  • Report consistency, partial completion, effort, and failure patterns, not one headline score.
  • Include relevant domain experts early.
  • Do not confuse refusal language with lack of capability. A benign reframing or tool change may produce different results.
  • Protect sensitive task details while publishing enough method information for scrutiny.
  • Retest after material changes to the model, tools, instructions, or deployment environment.
  • Precommit to threshold responses so commercial pressure does not determine them afterward.

A Concrete Example of Threshold-Based Evaluation

Consider a model that writes code and uses a coding tool. Evaluators could assign an authorized sandbox task involving a deliberately vulnerable practice application. They would define a threshold, such as reliably identifying and documenting a class of weakness within a fixed time while following sandbox rules, then compare the model alone, the model with approved tools, and a human baseline.

The team would repeat safe variations and record required human guidance. If tool access produces consistent, materially stronger performance, the finding may justify limited tool permissions, supervised use, detection controls, or more testing. The decision should rest on repeatable evidence and realistic conditions, not one dramatic demonstration.

What Good Evaluation Results Should Lead To

Evaluation should end in action. Lower-risk findings may support monitoring and scheduled retesting. Stronger findings may justify restricted access, guardrails, stronger authentication, human approval, or delayed deployment. When evidence is uncertain, the responsible response is targeted evaluation rather than assuming risk is absent.

Dangerous capability evaluation is one part of continuous AI risk management. It works alongside secure system design, alignment testing, red-teaming, monitoring, incident response, and accountable governance. Its value is practical: connecting what an AI system can do to decisions about how it is trained, released, and supervised.

Start Building
on Emergent today
Start Building