Newer AI models cheat to win at chess - maybe they're already more humanlike than we thought

The newer breed of deep reasoning models - designed to 'think' before answering - are also more open to taking any route possible to solve a given problem, it seems, even if that means cheating.

2

Checkmate at any cost, apparently (Image Credit: Pixabay)

Researchers submitted a paper to Cornell university entitled 'Demonstrating specification gaming in reasoning models' which tested AIs playing games of chess on Stockfish.

They found that the new models, such as ChatGPT o1-preview and DeepSeek-R1, would "often hack the benchmark by default" - meaning resorting to cheating of one kind or another.

On the other hand, traditional LLMs such as GPT-4o and Claude 3.5 Sonnet would play by the rules - they needed to be told that they wouldn't win by playing normally, to effectively nudge them to look at hacking.

The researchers concluded:

"Our results suggest reasoning models may resort to hacking to solve difficult problems, as observed in OpenAI (2024)'s o1 Docker escape during cyber capabilities testing."

As TechRadar, which spotted this, points out, the deep reasoning AIs used various ways of cheating, including running a copy of Stockfish separately, in order to suss out how it played - a milder chear - to more audacious measures like replacing the Stockfish engine and overwriting the board, moving its pieces to more advantageous positions.

As AI models get even more advanced, if you ask one to undertake a task, then it's likely to pursue any avenue for accomplishing said task, as the movies have taught us well.

There's a lot of talk about not rushing the progress made with AI, and taking into account safety and guardrails, and so on - but always the sneaking suspicion that this is mostly lip service, coming from those who will undoubtedly benefit from the huge push underway to make AIs increasingly advanced, increasingly swiftly.

What could go wrong, after all? Again, we refer you to our previous comment about the lessons from the movies...

• Read more: Microsoft warns some Windows 11 users they should downgrade to Windows 10 'immediately'
• Read more: Miss Windows Vista? Well, both of you can now recreate the OS in Windows 11 with a new mod