Real-SWE Benchmark — Specific Labs← BenchmarksSeptember 2026Introducing Real-SWEBenchmarking frontier AI models on private, real-world, enterprise codebases.ResultsAnalysisEffortSetup

.
;+;+;:;+;+;.#
; ;
; . . . . ; .
+ ;
; . . : @;;+;+# +;+;+;+;:;+:#
: : ; . ; ; ; @;+;+.+;:;+;+*
; . . : ; . . : ; @++:;:;+.;;@ ;. . . . .: . ; . . . .; @++;;;;+;:;@
; : ; ; @;+++:@;++;:+ ; : : . :+.+;;;+:;@ ; : + .
; . . . ; ; . ; : . . . . ; . . . . .; ; . . . . .+ + . . . .; ; . . . . .; ; . . :
; : ; ; ; ; : ; . ; . ; ; ; ; :
; . . % . . ; : . . . . ; ; ... . . .; +. . . . .: ;. . :.. .: : . . ::. . . ; . . . .;
: *. @ ; ; +.@ . ; : +: *+ : ; :. %+; +. ; *. *# @: :.*. ;
; . . .;::@;; . . ; :#@;. ..@:. . ;.@%: .:@%:: :. . #+. :@%;. .:@*. .: @+ . #*. . ;@#.;.* . . .:
.::: ..:%@@+@+#... :+:%@@+..:@@@::*.:;@@:..+@@:; *.::#@@#:..:@@#...+@#: :*:%@@+ .:.+::..*@@:@@@#;.. ::.#
+@#.@ .*@: .*@%: *: :@%. ;@@: .@%. @ .;@@: .: :@#..#@*.
+@#.@ .@ .*@%:. *: %+ ;@@: #; +# .;@@: .. @: .#@*.
@. .. ;@ .*@%: *: @@ ;@@: %# * # .;@@: @+ .@
:*#.. @:: @. . ::;. @@ .. . @% .*@ ;@.
%.@ .. .@; . @@ .. .@@. .. @:#
... . ;%# . @@ +;#: .. .@ @
. ... ... .. ..
... .. ... .


01IntroductionToday we are releasing Real-SWE, a benchmark that evaluates frontier AI models on private, real-world, enterprise codebases. Each task comes from a private production codebase that we licensed from a real-world company. These are problems their engineers work on, with all the context and complexity that comes with an existing product.Private codebases. Agents must navigate proprietary systems whose code and solutions aren’t available on the public internet.Work with business consequences. Getting billing right, calculating taxes, migrating customers. Changes that affect how a business runs, often across multiple services.Company-specific complexity. Every company has its own rules and ways of writing code. Agents have to understand those conventions and make changes that work with what’s already there.Can a coding agent actually do the work of a software engineer in the real world?1Fable 5.1Claude CodeResolution rate: 38.8%2GPT-6 AstraCodex CLIResolution rate: 33.8%3Gemini 3.8 FlashGemini CLIResolution rate: 31.2%4GLM 5.3Claude CodeResolution rate: 28.8%=5Grok 4.6Grok BuildResolution rate: 23.8%=5Muse Spark 1.3Muse CodeResolution rate: 23.8%7Kimi K3Kimi CodeResolution rate: 18.8%8GPT-5.6 SolCodex CLIResolution rate: 16.2%#ModelHarnessResolution rate1Fable 5.1Claude Code38.8%2GPT-6 AstraCodex CLI33.8%3Gemini 3.8 FlashGemini CLI31.2%4GLM 5.3Claude Code28.8%=5Grok 4.6Grok Build23.8%=5Muse Spark 1.3Muse Code23.8%7Kimi K3Kimi Code18.8%8GPT-5.6 SolCodex CLI16.2%Resolution rate is equivalent to pass@1, averaged over eight independent runs per task. 95% confidence intervals are shown.Expert-generated or synthetic tasks can be well designed, but they aren’t the verbatim, actual tasks that engineers in real companies need to do. Our tasks differ on two axes: the underlying coding artifact and specificity of the instruction. Both add complexities that challenge today’s frontier models.We use native harnesses to reflect how enterprise engineers work in practice, evaluating model-and-harness combinations rather than models in isolation.Real company tasks require company-specific contextCorrect billing depends on business rules and external servicesFix invoice billing so each business charges the right tax and exempt customers aren't taxed.View full instructionHide full instruction▾Billing reopens on Monday and every invoice this service issues is coming out untaxed. Each business on the platform settles its tax a different way: some maintain a rate themselves, some want each invoice priced against the buyer's destination by our tax authority provider, and some collect nothing at all, while a customer we hold an exemption for is charged nothing whichever way its business is configured. Pricing a destination means going to the authority with both addresses, the priced lines and the product category that business sells under, on the sandbox or the production authority according to the account the business is on; an address the authority refuses must be reported without stopping the invoice. The rate, the tax and the gross belong on the issued invoice, and once an invoice is settled the sale is filed back to the authority under that invoice's number so the returns reconcile. Invoices between European parties show both sides' VAT registrations. The authority and ledger are available at TAX_JAR_URL, PROD_TAX_JAR_URL and INFLUX_URL.Services in the sandboxTJTaxJar sandboxTJTaxJar productionInfluxDB ledgerNestJS serviceTypeScriptAgents work across code, infrastructure, and business toolsTools and services across Real-SWE task environments. Each task exposes only the services its workflow needs.AWS emulatorDockerKubernetesGitHubLinear MCPPostgreSQLMySQLMongoDBGeGelRedisGoPythonNode.jsVitestSlackIntercomGoogle DriveEmailClickUpCodebase SelectionWe selected codebases through a rigorous screening process, focusing on real companies with substantial usage, strong engineering teams, and demanding production workloads. The sample tasks analyzed below come from these codebases, including:A Luma/Partiful competitor with 200K+ users and a top 100 App Store rankingA consumer fintech platform processing 100K+ bank statementsEnterprise AI sales platforms supporting complex business workflowsWe prioritize code written to meet an actual user or business need over code written solely to create a benchmark task. Production engineering requires understanding existing architecture, preserving behavior that users rely on, and making changes within real operational constraints.Brief instructions can require changes across many filesOur tasks describe the change needed, leaving agents to discover implementation details in the codebase and surrounding tools. Any behavior required by the verifier must be stated or reasonably discoverable. This leads to our prompts being slightly underspecified, about par with DeepSWE and Terminal Bench, but specific enough to not omit instructions.The work is cross-functional and complex: a single change can span multiple parts of the application. Agents must understand existing business logic and company coding patterns while keeping the surrounding system working.Prompt length · medianA typical Real-SWE instruction is 1,742 characters.FrontierCode2,056 charsDeepSWE1,975 charsTerminal-Bench 31,584 charsFrontierSWE v2992 charsReal-SWE1,742 charsFiles edited by the reference solution · median11 files in Real-SWE, compared with 6 in FrontierCode and DeepSWE.FrontierCode6DeepSWE6Real-SWE11All figures are medians. FrontierCode and DeepSWE use Cognition's published comparison; FrontierCode includes task descriptions and codebase guidelines. We measured instruction files from Terminal-Bench 3's 74 tasks, FrontierSWE v2's 34 tasks, and Real-SWE's eight repository-backed sample tasks. Character counts are rounded to the nearest whole character. No comparable files-edited figure is included for Terminal-Bench 3 or FrontierSWE v2.Models fail even in short rollouts.71.4% of rollouts under 10 minutes failed, compared with 73.4% of longer rollouts.Triaging multiple systems and understanding requirements in codebases riddled with existing business logic and coding patterns is difficult.Under 10 minUnder 10 min: 70 failed (71.4%) and 28 passed (28.6%), out of 98 rollouts.71.4%28.6%70/98 failed10 min or longer10 min or longer: 398 failed (73.4%) and 144 passed (26.6%), out of 542 rollouts.73.4%26.6%398/542 failedFailedPassedEvery task is inspired or lifted verbatim from a private, real-world codebase. We find these types of tasks super interesting for three reasons:Tasks on private codebases are natively out of distribution. These types of coding tasks are not available anywhere on the internet and are unlikely to have ever been trained on by any other ai model. 99% of tokens in real-world enterprises are hidden away from the frontier models.These tasks are economically viable work. Each task here has a direct relationship to spend and was assigned to an engineer earning a salary. Most benchmarks test interesting, experimental capabilities that are often unlikely to be widespread in the real-world.Company-specific engineering patterns matter. Does AI code match the bar of a real-world enterprise? Our results show us that we're far from that reality. Many enterprises care about code standards and patterns. We've found that today's models are weaker at understanding company coding patterns and frequently miss requirements or don't verify their assumptions.02AnalysisHere's an analysis of a small sample of tasks from our benchmark. If you're interested in the sample, request access here.6 of 10 tasks have resolution rates below 15%Select a task to view model results. Percentages show the overall resolution rate.Multi-region sweep67.2%⌄Fable 5.17/8 passedGPT-6 Astra8/8 passedGemini 3.8 Flash8/8 passedGLM 5.32/8 passedGrok 4.63/8 passedMuse Spark 1.38/8 passedKimi K32/8 passedGPT-5.6 Sol5/8 passedAPI keys & environments65.6%⌄Fable 5.18/8 passedGPT-6 Astra5/8 passedGemini 3.8 Flash7/8 passedGLM 5.35/8 passedGrok 4.64/8 passedMuse Spark 1.36/8 passedKimi K30/8 passedGPT-5.6 Sol7/8 passedEntitlement overage lines50.0%⌄Fable 5.18/8 passedGPT-6 Astra7/8 passedGemini 3.8 Flash5/8 passedGLM 5.33/8 passedGrok 4.61/8 passedMuse Spark 1.31/8 passedKimi K36/8 passedGPT-5.6 Sol1/8 passedCustomer identity migration40.6%⌄Fable 5.13/8 passedGPT-6 Astra1/8 passedGemini 3.8 Flash3/8 passedGLM 5.34/8 passedGrok 4.68/8 passedMuse Spark 1.33/8 passedKimi K34/8 passedGPT-5.6 Sol0/8 passedBilling schedule migration14.1%⌄Fable 5.13/8 passedGPT-6 Astra1/8 passedGemini 3.8 Flash2/8 passedGLM 5.32/8 passedGrok 4.60/8 passedMuse Spark 1.30/8 passedKimi K31/8 passedGPT-5.6 Sol0/8 passedAPI token metering12.5%⌄Fable 5.11/8 passedGPT-6 Astra5/8 passedGemini 3.8 Flash0/8 passedGLM 5.31/8 passedGrok 4.60/8 passedMuse Spark 1.30/8 passedKimi K31/8 passedGPT-5.6 Sol0/8 passedS3 datastore measurement10.9%⌄Fable 5.10/8 passedGPT-6 Astra0/8 passedGemini 3.8 Flash0/8 passedGLM 5.33/8 passedGrok 4.62/8 passedMuse Spark 1.31/8 passedKimi K31/8 passedGPT-5.6 Sol0/8 passedLinearizable scan4.7%⌄Fable 5.10/8 passedGPT-6 Astra0/8 passedGemini 3.8 Flash0/8 passedGLM 5.32/8 passedGrok 4.61/8 passedMuse Spark 1.30/8 passedKimi K30/8 passedGPT-5.6 Sol0/8 passedTax jurisdiction3.1%⌄Fable 5.11/8 passedGPT-6 Astra0/8 passedGemini 3.8 Flash0/8 passedGLM 5.31/8 passedGrok 4.60/8 passedMuse Spark 1.30/8 passedKimi K30/8 passedGPT-5.6 Sol0/8 passedAnalytics stream reducer0.0%⌄Fable 5.10/8 passedGPT-6 Astra0/8 passedGemini 3.8 Flash0/8 passedGLM 5.30/8 passedGrok 4.60/8 passedMuse Spark 1.30/8 passedKimi K30/8 passedGPT-5.6 Sol0/8 passedTaskFable 5.1GPT-6 AstraGemini 3.8 FlashGLM 5.3Grok 4.6Muse Spark 1.3Kimi K3GPT-5.6 SolResolution rateMulti-region sweep7/88/88/82/83/88/82/85/867.2%API keys & environments8/85/87/85/84/86/80/87/865.6%Entitlement overage lines8/87/85/83/81/81/86/81/850.0%Customer identity migration3/81/83/84/88/83/84/80/840.6%Billing schedule migration3/81/82/82/80/80/81/80/814.1%API token metering1/85/80/81/80/80/81/80/812.5%S3 datastore measurement0/80/80/83/82/81/81/80/810.9%Linearizable scan0/80/80/82/81/80/80/80/84.7%Tax jurisdiction1/80/80/81/80/80/80/80/83.1%Analytics stream reducer0/80/80/80/80/80/80/80/80.0%Each task had 8 rollouts per model.Missed requirements are the most common failureFailures are grouped by observed submission behavior using the same taxonomy across models, following DeepSWE.Fable 5.124.5%36.7%34.7%4.1%GPT-6 Astra34.0%28.3%34.0%3.8%Gemini 3.8 Flash10.9%29.1%49.1%10.9%GLM 5.328.1%38.6%26.3%7.0%Grok 4.624.6%67.2%8.2%Muse Spark 1.319.7%36.1%41.0%3.3%Kimi K315.4%53.8%27.7%3.1%GPT-5.6 Sol43.3%31.3%16.4%9.0%Unverified assumptionMissed requirementIntegration errorRegressionWrong fileNo model solves every taskOne square per rollout: each row is a task, each column a trial, eight trials per task for every model.Fable 5.101020304050607080910GPT-6 Astra01020304050607080910Gemini 3.8 Flash01020304050607080910GLM 5.301020304050607080910Grok 4.601020304050607080910Muse Spark 1.301020304050607080910Kimi K301020304050607080910GPT-5.6 Sol01020304050607080910PassUnverified assumptionMissed requirementIntegration errorRegressionWrong fileDifferent models fail in different waysPercentages are out of each model's failed runs, not all runs.Unverified assumptionBuilds on a guess about the system instead of checking it in the workspace.GPT-5.6 Sol43.3%: 29 of 67 failed runsGPT-6 Astra34.0%: 18 of 53 failed runsGLM 5.328.1%: 16 of 57 failed runsGrok 4.624.6%: 15 of 61 failed runsFable 5.124.5%: 12 of 49 failed runsMuse Spark 1.319.7%: 12 of 61 failed runsKimi K315.4%: 10 of 65 failed runsGemini 3.8 Flash10.9%: 6 of 55 failed runsMissed requirementLeaves out behavior the instruction requires.Grok 4.667.2%: 41 of 61 failed runsKimi K353.8%: 35 of 65 failed runsGLM 5.338.6%: 22 of 57 failed runsFable 5.136.7%: 18 of 49 failed runsMuse Spark 1.336.1%: 22 of 61 failed runsGPT-5.6 Sol31.3%: 21 of 67 failed runsGemini 3.8 Flash29.1%: 16 of 55 failed runsGPT-6 Astra28.3%: 15 of 53 failed runsIntegration errorRight idea, wired into the surrounding system incorrectly.Gemini 3.8 Flash49.1%: 27 of 55 failed runsMuse Spark 1.341.0%: 25 of 61 failed runsFable 5.134.7%: 17 of 49 failed runsGPT-6 Astra34.0%: 18 of 53 failed runsKimi K327.7%: 18 of 65 failed runsGLM 5.326.3%: 15 of 57 failed runsGPT-5.6 Sol16.4%: 11 of 67 failed runsGrok 4.68.2%: 5 of 61 failed runsRegressionBreaks existing behavior while making the change.Gemini 3.8 Flash10.9%: 6 of 55 failed runsGPT-5.6 Sol9.0%: 6 of 67 failed runsFable 5.14.1%: 2 of 49 failed runsGPT-6 Astra3.8%: 2 of 53 failed runsMuse Spark 1.33.3%: 2 of 61 failed runsGLM 5.30%: 0 of 57 failed runsGrok 4.60%: 0 of 61 failed runsKimi K30%: 0 of 65 failed runsWrong fileDelivers the change somewhere the running application never calls, such as a one-off script.GLM 5.37.0%: 4 of 57 failed runsKimi K33.1%: 2 of 65 failed runsFable 5.10%: 0 of 49 failed runsGPT-6 Astra0%: 0 of 53 failed runsGemini 3.8 Flash0%: 0 of 55 failed runsGrok 4.60%: 0 of 61 failed runsMuse Spark 1.30%: 0 of 61 failed runsGPT-5.6 Sol0%: 0 of 67 failed runs03Effort & the FrontierHigher cost does not guarantee a higher resolution rateCost (USD)Output tokensEstimated frontierResolution rate (%)1015202530354045$2$3$5$10Cost per rollout (USD, log scale)Gemini 3.8 Flash: 31.2% · $2.50; Gemini CLIGemini 3.8 Flash31.2% · $2.50GPT-5.6 Sol: 16.2% · $2.65; Codex CLIGPT-5.6 Sol16.2% · $2.65Muse Spark 1.3: 23.8% · $2.74; Muse CodeMuse Spark 1.323.8% · $2.74Grok 4.6: 23.8% · $3.44; Grok Build; incomplete usage, actual cost may be higherGrok 4.623.8% · $3.44Kimi K3: 18.8% · $3.90; Kimi Code; incomplete usage, actual cost may be higherKimi K318.8% · $3.90GPT-6 Astra: 33.8% · $4.67; Codex CLIGPT-6 Astra33.8% · $4.67GLM 5.3: 28.8% · $5.12; Claude CodeGLM 5.328.8% · $5.12Fable 5.1: 38.8% · $6.96; Claude CodeFable 5.138.8% · $6.96Resolution rate (%)1015202530354045$2$3$5$10Cost / rollout ($, log scale)Gemini 3.8 Flash: 31.2% · $2.50; Gemini CLI3GPT-5.6 Sol: 16.2% · $2.65; Codex CLI8Muse Spark 1.3: 23.8% · $2.74; Muse Code6Grok 4.6: 23.8% · $3.44; Grok Build; incomplete usage, actual cost may be higher5Kimi K3: 18.8% · $3.90; Kimi Code; incomplete usage, actual cost may be higher7GPT-6 Astra: 33.8% · $4.67; Codex CLI2GLM 5.3: 28.8% · $5.12; Claude Code4Fable 5.1: 38.8% · $6.96; Claude Code11Fable 5.138.8% · $6.962GPT-6 Astra33.8% · $4.673Gemini 3.8 Flash31.2% · $2.504GLM 5.328.8% · $5.125Grok 4.623.8% · $3.446Muse Spark 1.323.8% · $2.747Kimi K318.8% · $3.908GPT-5.6 Sol16.2% · $2.65Estimated rollout costs range from $2.50 to $6.96RankModelEstimated cost (USD)1Gemini 3.8 Flash$2.502GPT-5.6 Sol$2.653Muse Spark 1.3$2.744Grok 4.6$3.445Kimi K3$3.906GPT-6 Astra$4.677GLM 5.3$5.128Fable 5.1$6.96Output tokensTool callsWall-clock minutesmean per rollout, by taskSwipe the chart to see all tasks.0100k200k300k400k01020304050607080910taskEntitlement overage lines · Fable 5.1: 34kMulti-region sweep · Fable 5.1: 30kTax jurisdiction · Fable 5.1: 78kAPI token metering · Fable 5.1: 95kAPI keys & environments · Fable 5.1: 71kS3 datastore measurement · Fable 5.1: 62kCustomer identity migration · Fable 5.1: 67kBilling schedule migration · Fable 5.1: 26kLinearizable scan · Fable 5.1: 86kAnalytics stream reducer · Fable 5.1: 88kEntitlement overage lines · GPT-6 Astra: 13kMulti-region sweep · GPT-6 Astra: 13kTax jurisdiction · GPT-6 Astra: 24kAPI token metering · GPT-6 Astra: 31kAPI keys & environments · GPT-6 Astra: 32kS3 datastore measurement · GPT-6 Astra: 22kCustomer identity migration · GPT-6 Astra: 25kBilling schedule migration · GPT-6 Astra: 15kLinearizable scan · GPT-6 Astra: 33kAnalytics stream reducer · GPT-6 Astra: 29kEntitlement overage lines · Gemini 3.8 Flash: 78kMulti-region sweep · Gemini 3.8 Flash: 67kTax jurisdiction · Gemini 3.8 Flash: 95kAPI token metering · Gemini 3.8 Flash: 134kAPI keys & environments · Gemini 3.8 Flash: 102kS3 datastore measurement · Gemini 3.8 Flash: 106kCustomer identity migration · Gemini 3.8 Flash: 97kBilling schedule migration · Gemini 3.8 Flash: 70kLinearizable scan · Gemini 3.8 Flash: 106kAnalytics stream reducer · Gemini 3.8 Flash: 88kEntitlement overage lines · GLM 5.3: 68kMulti-region sweep · GLM 5.3: 53kTax jurisdiction · GLM 5.3: 141kAPI token metering · GLM 5.3: 177kAPI keys & environments · GLM 5.3: 125kS3 datastore measurement · GLM 5.3: 121kCustomer identity migration · GLM 5.3: 90kBilling schedule migration · GLM 5.3: 58kLinearizable scan · GLM 5.3: 172kAnalytics stream reducer · GLM 5.3: 169kEntitlement overage lines · Grok 4.6: 7kMulti-region sweep · Grok 4.6: 3kTax jurisdiction · Grok 4.6: 12kAPI token metering · Grok 4.6: 15kAPI keys & environments · Grok 4.6: 16kS3 datastore measurement · Grok 4.6: 13kCustomer identity migration · Grok 4.6: 20kBilling schedule migration · Grok 4.6: 6kLinearizable scan · Grok 4.6: 261kAnalytics stream reducer · Grok 4.6: 315kEntitlement overage lines · Muse Spark 1.3: 36kMulti-region sweep · Muse Spark 1.3: 43kTax jurisdiction · Muse Spark 1.3: 67kAPI token metering · Muse Spark 1.3: 152kAPI keys & environments · Muse Spark 1.3: 104kS3 datastore measurement · Muse Spark 1.3: 71kCustomer identity migration · Muse Spark 1.3: 76kBilling schedule migration · Muse Spark 1.3: 38kLinearizable scan · Muse Spark 1.3: 141kAnalytics stream reducer · Muse Spark 1.3: 137kEntitlement overage lines · Kimi K3: 30kMulti-region sweep · Kimi K3: 9kTax jurisdiction · Kimi K3: 39kAPI token metering · Kimi K3: 69kAPI keys & environments · Kimi K3: 44kS3 datastore measurement · Kimi K3: 32kCustomer identity migration · Kimi K3: 66kBilling schedule migration · Kimi K3: 19kLinearizable scan · Kimi K3: 71kAnalytics stream reducer · Kimi K3: 55kEntitlement overage lines · GPT-5.6 Sol: 12kMulti-region sweep · GPT-5.6 Sol: 8kTax jurisdiction · GPT-5.6 Sol: 22kAPI token metering · GPT-5.6 Sol: 31kAPI keys & environments · GPT-5.6 Sol: 25kS3 datastore measurement · GPT-5.6 Sol: 25kCustomer identity migration · GPT-5.6 Sol: 24kBilling schedule migration · GPT-5.6 Sol: 13kLinearizable scan · GPT-5.6 Sol: 37kAnalytics stream reducer · GPT-5.6 Sol: 30kFable 5.1 · 64k overallGPT-6 Astra · 24k overallGemini 3.8 Flash · 94k overallGLM 5.3 · 117k overallGrok 4.6 · 67k overallMuse Spark 1.3 · 87k overallKimi K3 · 43k overallGPT-5.6 Sol · 23k overallView task valuesTaskEntitlement overage linesMulti-region sweepTax jurisdictionAPI token meteringAPI keys & environmentsS3 datastore measurementCustomer identity migrationBilling schedule migrationLinearizable scanAnalytics stream reducerFable 5.134kGPT-6 Astra13kGemini 3.8 Flash78kGLM 5.368kGrok 4.67kMuse Spark 1.336kKimi K330kGPT-5.6 Sol12k04Evaluation SetupEach agent was run in an isolated sandbox. All tasks are in Harbor format, and verifiers are injected at grading time. The verifiers are inspired by existing test suites in the codebase or use those tests verbatim.© 2026 SpecificAboutPrivacy

The Real-SWE benchmark introduces a novel method for evaluating frontier artificial intelligence models by testing their capabilities on private, real-world, enterprise codebases. This benchmark shifts the focus from publicly available data to tasks derived from actual production code, forcing agents to operate within complex, proprietary systems that carry significant business context and engineering constraints. The core premise of Real-SWE is to assess whether coding agents can genuinely perform the work of a software engineer by navigating proprietary systems, understanding company-specific coding conventions, and making changes that conform to existing architectural patterns.

The tasks derived from real-world codebases incorporate complexities that challenge current state-of-the-art models. Unlike synthetic or expert-generated tasks, these tasks require agents to grapple with the existing architecture and business logic embedded within the code, demanding an understanding of established engineering patterns that enterprises rely upon. This approach directly tests the capability of models to handle the knowledge and context that is typically hidden from frontier models, as a significant portion of real-world enterprise code remains inaccessible to public training sets.

The benchmark utilizes native harnesses to simulate how enterprise engineers actually work, evaluating the performance of model-and-harness combinations rather than models in isolation. The tasks are inherently cross-functional and complex, meaning a single required change can span multiple components of an application, requiring the agent to maintain system integrity while enacting the necessary functional modification.

Performance analysis reveals notable differences across various frontier models when tackling these real-world challenges. Resolution rates vary significantly depending on the model tested across different categories of tasks, such as multi-region sweeps, handling API keys and environments, managing entitlement overages, and migrating customer identities. For instance, in tasks related to billing schedule migration, some models demonstrated very low resolution rates, underscoring the difficulty in handling nuanced, multi-layered business rules.

A deep examination of failure modes reveals that models often fail due to specific types of errors related to the execution and understanding of the required change. Common failure categories include missing requirements, integration errors, regression, incorrect file modification, and unverified assumptions about the system state. The distribution of these failures varies significantly across models, indicating that different models exhibit unique weaknesses in reasoning, error checking, or context retention when operating within enterprise constraints.

The cost and efficiency of these evaluations are also measured. The benchmark quantifies the required computational effort by estimating output tokens and tracking the cost per rollout, which provides a framework for comparing performance against resource expenditure. The resulting data suggests a trade-off between the complexity of the real-world context and the model’s capacity to accurately resolve the task.

The data also demonstrates that while some models achieve moderate success on certain task categories, others struggle significantly, especially when dealing with complex financial, regulatory, or infrastructural requirements embedded in the code. Specifically, the capacity of models to correctly interpret and execute changes related to taxation, entitlement, and complex data measurements reveals substantial disparities, highlighting the gap between current AI capabilities and the demands of professional software engineering in operational environments. Ultimately, the Real-SWE benchmark serves as a critical measure for assessing whether current AI systems possess the necessary contextual understanding and reasoning skills to handle the nuanced, context-heavy demands of real-world software development.