SWE-bench SWE-bench: Evaluate Language Models on Open Source Software Tasks Concept is “Can Language Models Resolve Real-World GitHub Issues?” AutoCodeRover is best SOTA I am aware of at 16% … anyone know better? another thread: Fully Autonomous AI Softwar…

One thing I like about swe-bench is that this it’s a pretty good way to compare claude versus GPT4 [image] This echos my experience as well. I don’t know why people are trash talking GPT4 in this forum.

I’m also a big fan of the SWE benchmark myself, although I think it’s often used to infer coding performance in an overly generalized way. Ideally I want to see multiple good benchmark scores on a variety of coding tasks, SWE bench only measures the model’s ability to solve GitHub issues. [imag…

All the current code evals are too easy to overfit, imho, though I’d love to hear about any that are not. I imagine the current set on swe-bench will suffer from overfitting, but fortunately it’s pretty easy to select a new set. So training on that eval would be a rather bad idea and could cause …

[image] qrdl: All the current code evals are too easy to overfit, imho, though I’d love to hear about any that are not. Totally agree, an interesting approach to this would be to create a new set of questions for an existing benchmark, and compare the performance between the two, I should per…

Swe-bench: Very exciting eval, looking for SOTA

Community

qrdl May 3, 2024, 11:22pm 2

One thing I like about swe-bench is that this it’s a pretty good way to compare claude versus GPT4

This echos my experience as well. I don’t know why people are trash talking GPT4 in this forum.

1 Like

Topic		Replies	Views
GPT-4-Turbo models perform better the older GPT-4 models in LMSys benchmark API gpt-4 , api	14	7012	May 13, 2024
OpenAI releases new coding benchmark SWE-Lancer showing 3.5 Sonnet beating o1 Community ai-coding	1	3789	February 19, 2025
Performance of GPT-4o on the Needle in a Haystack Benchmark API chatgpt , api , gpt-4o	13	6458	June 13, 2024
Situational-awareness.ai, a brief writeup by Leopold Aschenbrenner Community chatgpt	11	28101	December 30, 2024
List of fresh gpt-4o benchmarks, please add Community gpt-4o	1	3700	May 16, 2024

Swe-bench: Very exciting eval, looking for SOTA

Related topics