Homeglossary

SWE-bench

SWE-bench is a software engineering benchmark that tests whether an AI model or coding agent can fix real issues in existing software repositories. It gives a system an issue description and codebase, then checks whether its proposed patch passes the relevant tests. Unlike short coding puzzles, it evaluates repository-level debugging, code navigation, and validation under realistic constraints.

What Is SWE-bench?

SWE-bench is a software engineering benchmark that tests whether an AI model or coding agent can fix real issues in existing software repositories. It gives a system an issue description and codebase, then checks whether its patch passes relevant tests.

SWE means software engineering. Unlike a short programming puzzle, SWE-bench requires work in a real project with files, dependencies, conventions, and existing behavior. Based on historical GitHub issues and pull requests, it tests whether a system can understand a problem, find its cause, change the right files, and avoid regressions.

The original SWE-bench research paper introduced repository-level evaluation rather than isolated coding questions. It is widely used to compare AI coding agents, but scores need careful interpretation.

Why SWE-bench Was Created

Traditional code benchmarks often provide a complete prompt, a small function, and a known answer. They test programming fundamentals but omit much of everyday software work.

Real bug reports may be incomplete, logic can span multiple files, and projects use specific libraries and test tools. A correct fix must preserve existing behavior, not just satisfy a new example. SWE-bench puts these conditions into an evaluable format.

It also evaluates an agent's workflow, including file search, test inspection, command use, error analysis, patch revision, and verification. This matters when assessing an AI coding assistant for practical development work.

How the SWE-bench Test Works

A task recreates an open-source project before maintainers solved a reported issue. Evaluation checks whether an AI-generated patch produces the required behavior.

  1. A historical GitHub issue and linked fix are selected from an eligible repository.
  2. The repository is restored to the base commit, before the original fix.
  3. The evaluator prepares the language runtime, dependencies, and test commands.
  4. The model or agent receives the issue statement and repository access under the evaluation rules.
  5. The agent explores the code, writes a patch, and may run tools or tests if allowed.
  6. The evaluator applies the patch and runs task-specific and regression tests.
  7. The task is resolved only if the patch passes the defined tests.

This is execution-based evaluation. The benchmark runs code in a controlled environment, so a convincing explanation cannot compensate for a patch that fails at runtime.

Key Components of a SWE-bench Task

Each task includes an issue statement, base commit, and repository environment. The issue describes a bug or requested change, the base commit provides starting code, and the environment defines installation and test steps.

The historical gold patch is the maintainers' original solution. It helps validate the task, but an AI agent does not need to reproduce it. A different patch can pass if it meets the tests and preserves expected behavior. The candidate patch is the AI system's proposed change.

Test patches encode behavior that should change and, where possible, behavior that must remain intact. A sensible-looking patch can fail by missing an edge case or changing a public interface. Task materials and the harness are documented in the official SWE-bench repository.

SWE-bench, SWE-bench Verified, Lite, and Newer Variants

Benchmark labels are not interchangeable. Identify the dataset variant and evaluation protocol before comparing results.

VariantPurposeWhat to know
Original SWE-benchBroad repository-level evaluationThe original dataset contains 2,294 tasks across Python repositories. Its breadth is useful, but task clarity and evaluability vary.
SWE-bench VerifiedMore dependable comparisonsA human-reviewed subset of 500 tasks. Curation removes tasks with unclear requirements or unreliable evaluation.
SWE-bench LiteFaster, lower-cost experimentsA smaller subset of 300 tasks. It supports iteration but is not a full replacement for broader testing.
Pro and multimodal variantsNewer or expanded settingsThese may test harder tasks, different repositories, or inputs such as images. Rules and availability vary, so compare scores only within the same variant.

What SWE-bench Measures Well

SWE-bench is strongest for assessing whether an AI system can complete a bounded maintenance task in an unfamiliar codebase.

  • Connecting a natural-language issue to likely technical causes.
  • Finding relevant files, functions, and tests.
  • Tracing bugs across interfaces.
  • Producing targeted patches that fit the project structure.
  • Using test results and errors for iterative debugging.
  • Preserving established behavior while fixing a defect.

Benefits of SWE-bench for Evaluating AI Coding Agents

The benchmark offers a more practical signal than many single-function tests because tasks are repeatable and results are observable.

  • It uses real open-source maintenance work rather than invented exercises.
  • Executable tests reduce reliance on subjective code-quality judgments.
  • It evaluates repository navigation, tool use, editing, and validation.
  • It gives researchers a shared reference point for tracking progress.
  • It helps teams compare model-and-agent combinations before internal trials.

Practical Limits and Common Pitfalls

SWE-bench is useful evidence, not a complete measure of software engineering ability.

  • Benchmark contamination can occur if task details, patches, or related code appeared in training data.
  • Tests are incomplete specifications, so passing does not prove every edge case is handled.
  • Task difficulty varies, and a single percentage can hide failure patterns.
  • The original benchmark focuses on Python and open-source repositories, which may not match a team's stack.
  • Search tools, permissions, prompts, retries, and context limits can materially affect results.
  • Nondeterministic agents can produce different patches and scores across runs.
  • A passing patch is not proof that code is secure, maintainable, licensed appropriately, or production-ready.

How to Interpret SWE-bench Scores Fairly

The highest reported score is not automatically the best choice. A result describes one system under specific conditions.

Factor to checkWhy it changes meaningFair comparison approach
Benchmark variantVerified, Lite, and other variants have different task mixes.Compare results from the same dataset split.
Agent scaffoldTools and workflow rules can improve or constrain performance.Check prompts, tools, and permissions used.
Attempts and runtimeMore retries and longer execution can raise resolution rates.Compare equivalent attempt limits and time budgets.
Cost and latencyStronger results may require expensive or slow inference.Measure resolution rate alongside cost per task and elapsed time.
ReproductionVendor-reported results may not transfer to another setup.Prefer transparent methods and independent reruns when available.

Where SWE-bench Is Used

SWE-bench supports evaluation work across research and software organizations.

  • Model developers test whether changes improve code-repair performance.
  • Agent builders tune search, planning, testing, and patch-review workflows.
  • Engineering leaders use it when shortlisting tools for a pilot.
  • Teams run it after changing prompts, models, or tool permissions.
  • Researchers use public tasks and harnesses to support reproducible claims.

It is best considered alongside other measurements. For example, a model benchmark guide can distinguish coding-agent results from broader model evaluations.

A Concrete Example of a SWE-bench Evaluation

Imagine a Python web project where date filters fail for timezone-aware timestamps. An agent must inspect the request handler, date conversion utility, database query expectations, and nearby tests. It may add a normalization step and update a focused test.

The evaluator applies the patch to the original repository state. The timezone case must pass, along with tests for ordinary dates and invalid input. A patch that compiles but breaks another filter has not resolved the task.

Best Practices for Using SWE-bench Results

Use SWE-bench as a starting point, then validate results against your organization's work.

  • Compare like-for-like runs with the same variant, tools, retry policy, and budget.
  • Review generated patches for unnecessary edits, fragile logic, and unsafe patterns.
  • Test shortlisted systems on representative internal repositories with permission and security controls.
  • Measure security, maintainability, documentation quality, and review burden separately from task resolution.
  • Track cost, runtime, and human intervention, not only tasks resolved.
  • Use several benchmarks and real pilot tasks instead of one leaderboard for purchasing decisions.
Start Building
on Emergent today
Start Building