LLMs do fine on ARC-AGI-3 if they are allowed to search over game logs

I was reading the comments to this post and the opinion seemed to be that harness makes little/no difference for ARC-AGI-3. Turns out, it makes a huge difference: Hill-climbing ARC-AGI-3 TLDR: if you save game logs - taken actions, board states and scores - and let LLMs search over them with tools, LLMs are only moderately less efficient than humans in terms of the number of actions taken to beat ARC-AGI-3 games. https://preview.redd.it/kga97l39oqyg1.png?width=2048&format=png&auto=webp&s=49d27