Performance of GPT-4o on the Needle in a Haystack Benchmark

[qrdl]

posted a few here

List of fresh gpt-4o benchmarks, please add

Thought I’d share some benchmarks I’ve ran into better perf https://twitter.com/LiamFedus/status/1790064966000848911 https://twitter.com/LouisKnightWebb/status/1790265899255017893 https://twitter.com/asankhaya/status/1790222284579582158 really great eval. note how opus was lobotomized here (likely on purpose for safety reasons) https://twitter.com/WenhuChen/status/1790128184505102442 same perf https://twitter.com/jon_mellon/status/1790151179269271653 hints of asymptotic behavior http…

For your query - https://twitter.com/LouisKnightWebb/status/1790265899255017893

but also note

https://twitter.com/JoshPurtell/status/1790102029773246861

I find the latter quite curious. It echo’s my openai-eval PR around multistep math problems, such that it has to provide the answer and no other text.