Goalposts

AI solving international math olympiad problems is not intelligence

But couldn't it be overfitting? LLMs are very good at deriving patterns, many of which humans simply can't tell apart from noise. With a few billion parameters and whatever black magic is going on inside CoT, it's not unreasonable to think even small amounts of fine-tuning combined with many epochs of training would be enough for it to conjure a compressed representation of that problem type.

Without an extensive audit, I'd be skeptical of OpenAI's claims, especially given how o1 is often wrong on much more trivial compositional questions.

What defines intelligence is generalization, the ability to learn new tasks from few examples, and while LLMs have made some significant progress here, they are still many orders below a child and arguably even many animals.