About whether AI benchmarks show real progress or are marketing.
Has this happened?
Yes 68 (58%)Not sure 18 (15%)No 31 (26%)
Votes cast 1–2 October 2026: 100,590 votes from 9,694 people.
Hacker News has set AI a lot of challenges over the years. Which ones has it met?
About whether AI benchmarks show real progress or are marketing.
Votes cast 1–2 October 2026: 100,590 votes from 9,694 people.
Whenever I try and use a "state of the art" LLM to generate code it takes longer to get a worse result than if I just wrote the code myself from the start. That's the experience of every good dev I know. So that's my benchmark. AI benchmarks are BS marketing gimmicks designed to give the appearance of progress - there are tremendous perverse financial incentives.
This will never change because you can only use an LLM to generate code (or any other type of output) you already know how to produce and are expert at - because you can never trust the output.