Evolution of Accuracy and Visual-Cognitive Errors in a Decade of Vision-Language AI Models

Problem Statement For roughly a decade, vision-language models have been declared to be approaching or matching human performance on scene description (captioning). The evidence for that claim has almost always come from the same family of benchmarks—most famously MS-COCO. Those images are typically clean, well-lit, and depict either no people or people performing simple, isolated actions (sitting, walking, holding an object). They rarely require the model to parse multi-agent social dynamics, subtle intentions, or the kind of relational reasoning humans perform effortlessly when watching a movie scene or a street interaction. Because the evaluation data are easy, the reported numbers look excellent. Automatic metrics such as BLEU-4, CIDEr, or even embedding-based scores like BERTScore further inflate the impression of progress: they reward surface lexical overlap more than genuine semantic fidelity. At the same time, almost no work has systematically catalogued which visual-cognitive failures models still commit, or how those failure modes have changed as architectures moved from CNN+LSTM captioners to today’s multimodal large language models (MLLMs). The result is a field that can claim “human-level performance” while remaining largely blind to whether the models actually understand the scenes that matter most in real applications—scenes full of people interacting. The authors therefore set out to answer two concrete questions that the existing literature left open: (1) How much of the apparent progress is an artifact of easy data? (2) Which specific error types have been eliminated and which stubbornly remain? Core Idea The core insight is that progress looks dramatically different once you force models to describe complex social behavior and once you measure not only overall accuracy but a taxonomy of visual-cognitive errors. By constructing a new 100-image Complex Social Behavior (CSB) dataset drawn from movie frames that require reasoning about multi-person in