Technology Updates : People are using Super Mario to benchmark AI now

Source:

https://techcrunch.com/2025/03/03/people-are-using-super-mario-to-benchmark-ai-now/

Apple Intelligence:

• Game Performance: Anthropic’s Claude 3.7 performed the best, followed by Claude 3.5. Google’s Gemini 1.5 Pro and OpenAI’s GPT-4o struggled.

• Game Mechanism: The game ran in an emulator and integrated with a framework, GamingAgent, to give the AIs control over Mario.

• Reasoning Model Performance: Reasoning models performed worse than “non-reasoning” models, despite being generally stronger on most benchmarks.

• AI Benchmarking Limitations: Games, while useful for benchmarking AI, have limitations as they are abstract, simple, and offer infinite data, unlike the real world.

• Evaluation Crisis in AI: Recent gaming benchmarks highlight a lack of clear metrics to assess the true capabilities of AI models.

• Uncertainty in AI Model Performance: There is a lack of clarity regarding the actual performance and capabilities of current AI models.