---
title: "OpenAI launches SWE-bench Verified: Existing frameworks underestimate the software engineering capabilities of models"
description: "OpenAI launches SWE-bench Verified, an improvement on the existing SWE-bench to more reliably assess AI models' ability to solve software problems. This initiative aims to evaluate their performance i"
type: "news"
locale: "en"
url: "https://longbridge.com/en/news/211496428.md"
published_at: "2024-08-13T23:47:30.000Z"
---

# OpenAI launches SWE-bench Verified: Existing frameworks underestimate the software engineering capabilities of models

> OpenAI launches SWE-bench Verified, an improvement on the existing SWE-bench to more reliably assess AI models' ability to solve software problems. This initiative aims to evaluate their performance in challenging tasks as systems approach AGI. This is business-related information and constitutes a significant event for the company

OpenAI just launched a more reliable code generation evaluation benchmark: SWE-bench Verified.

The most important sentence in the blog post is: "As our systems approach AGI more and more, we need to evaluate them in increasingly challenging tasks."

This benchmark is an improved version (subset) of the existing SWE-bench, aimed at more reliably evaluating the ability of AI models to solve real-world software problems.

SWE-bench is a popular software engineering evaluation suite used to assess the ability of large language models (LLMs) to solve real software problems extracted from GitHub. It evaluates by providing code repositories and problem descriptions to AI agents, and asking them to generate patches to fix the problems. While LLMs have made remarkable progress on SWE-bench, OpenAI's research found that the benchmark has some issues that may underestimate the models' autonomous software engineering capabilities.

Specifically, OpenAI identified three main issues with SWE-bench:

**1\. Unit tests are too strict**: Unit tests used to evaluate the correctness of solutions are often too specific, even irrelevant to the problem, which may result in rejecting correct solutions.

**2\. Unclear problem descriptions**: Many samples have vague problem descriptions, leading to ambiguity in understanding the problem and its solutions.

**3\. Difficult to set up development environment**: Sometimes it is challenging to reliably set up the SWE-bench development environment for agents, which may cause unit tests to fail regardless of the solution.

To address these issues, OpenAI collaborated with professional software developers to manually screen each sample in the SWE-bench test set to ensure the scope of unit tests is appropriate and problem descriptions are clear. In the end, they released SWE-bench Verified, which is a verified subset containing 500 samples, replacing the original SWE-bench and SWE-bench Lite test sets.

Additionally, OpenAI also worked with the authors of SWE-bench to develop a new evaluation tool that uses containerized Docker environments, making evaluations on SWE-bench easier and more reliable.

On SWE-bench Verified, GPT-4o solved 33.2% of the samples, while the best-performing open-source agent framework Agentless doubled its score to 16% OpenAI's research highlights the importance of a deep understanding and improvement of benchmark evaluation, especially as AI systems approach closer to Artificial General Intelligence (AGI). With the continuous improvement in AI model capabilities, we need to evaluate their performance more carefully to ensure that the evaluation results accurately reflect the true capabilities of the models.

OpenAI suggests:

**Deep Understanding of Benchmarks**: Even well-designed benchmarks may have issues that need ongoing improvement.

**Consider Progress in the Ecosystem**: Pay attention to the community's advancements in agent frameworks and consider potential external enhancements when assessing risks.

**Recognize Limitations**: Evaluations based on static datasets have inherent limitations and require supplementary evaluation methods.

For more information: https://openai.com/index/introducing-swe-bench-verified/

### Related Stocks

- [OpenAI.NA - OpenAI](https://longbridge.com/en/quote/OpenAI.NA.md)

## Related News & Research

| Title | Description | URL |
|-------|-------------|-----|
| Sam Altman And Dario Amodei Stir Controversy At India AI Summit Amid Photo-Op Gesture— OpenAI CEO Says 'I Just Wasn't Sure...' | At the India AI Impact Summit, OpenAI CEO Sam Altman and Anthropic CEO Dario Amodei stirred controversy by opting out of | [Link](https://longbridge.com/en/news/276431749.md) |
| Tata Group to deploy ChatGPT Enterprise across company - OpenAI | Tata Group to deploy ChatGPT Enterprise across company - OpenAI | [Link](https://longbridge.com/en/news/276296077.md) |
| Bartronics India Denies Collaboration With OpenAI | Bartronics India Ltd :COMPANY DENIES LAUNCHING 500 MW DATA CENTRECOMPANY DENIES COLLABORATION WITH OPENAI | [Link](https://longbridge.com/en/news/276327596.md) |
| India’s top telco tackles AI with $110 billion build plan and proven fast market dominance playbook | India’s top telco, Reliance Jio, plans to invest $110 billion in AI infrastructure over seven years to enhance its servi | [Link](https://longbridge.com/en/news/276406785.md) |
| After AMD, OpenAI Partners With Tata To Build Massive 1GW AI Data Center In India | OpenAI has partnered with Tata Group and Tata Consultancy Services to develop a large-scale AI data center in India, wit | [Link](https://longbridge.com/en/news/276304570.md) |

---

> **Disclaimer**: This article is for reference only and does not constitute any investment advice.
