Minimal videos: Trade-off between spatial and temporal information in human and machine vision.

TitleMinimal videos: Trade-off between spatial and temporal information in human and machine vision.
Publication TypeJournal Article
Year of Publication2020
AuthorsBen-Yosef, G, Kreiman, G, Ullman, S
JournalCognition
Date Published08/2020
KeywordsComparing deep neural networks and humans, Integration of spatial and temporal visual information, minimal images, Minimal videos, Visual dynamic recognition
Abstract

Objects and their parts can be visually recognized from purely spatial or purely temporal information but the mechanisms integrating space and time are poorly understood. Here we show that visual recognition of objects and actions can be achieved by efficiently combining spatial and motion cues in configurations where each source on its own is insufficient for recognition. This analysis is obtained by identifying minimal videos: these are short and tiny video clips in which objects, parts, and actions can be reliably recognized, but any reduction in either space or time makes them unrecognizable. Human recognition in minimal videos is invariably accompanied by full interpretation of the internal components of the video. State-of-the-art deep convolutional networks for dynamic recognition cannot replicate human behavior in these configurations. The gap between human and machine vision demonstrated here is due to critical mechanisms for full spatiotemporal interpretation that are lacking in current computational models.

URLhttps://www.sciencedirect.com/science/article/abs/pii/S0010027720300822
DOI10.1016/j.cognition.2020.104263

Associated Module: 

CBMM Relationship: 

  • CBMM Funded