Why Current AI Cannot Truly See the World
Session Overview
Frontier AI models score 80β90% on standard benchmarks like MMMU, yet when tested on visual tasks any 6-year-old handles effortlessly β like counting objects in an image or navigating a maze β those same models fall to pieces. Most models hit high benchmark scores not through genuine visual understanding, but by coding: a workaround that scores well and reveals little. Strip that away and you're left with systems that struggle to solve a simple crossword puzzle, identify what's the same or different across two images, or navigate a basic 3D view β tasks that are essential to human-level reasoning, and that the current benchmark ecosystem was never built to evaluate.
Key Takeaways
- Why frontier models score 80β90% on benchmarks like MMMU yet fail visual tasks a 6-year-old handles with ease.
- How coding workarounds inflate benchmark scores while revealing almost nothing about real visual understanding.
- The structural and modeling reasons the current eval landscape systematically overstates capability.
- An insider's view from 12 years co-leading GLaM, PaLM 2, and Gemini at Google Brain and DeepMind on how we got here.
- What a more rigorous evaluation framework needs to look like β because fixing visual reasoning starts with fixing how we measure it.
Speaker
Andrew Dai spent 12 years as a Research Scientist at Google Brain and DeepMind. He wrote the 2015 paper that OpenAI later cited as the original recipe for ChatGPT, was a core Area Lead on Gemini, GLaM, and PaLM 2, and his published research has accumulated over 75,000 citations. Now, he leads Elorian AI, a company building AI systems that understand the visual medium and apply reasoning the way humans do. Elorian AI recently launched with $55M at a $300M valuation, backed by Menlo Ventures, Altimeter, Striker Venture Partners, NVIDIA, and Jeff Dean.