Format

Send to

Choose Destination
J Exp Psychol Gen. 2016 Jan;145(1):82-94. doi: 10.1037/xge0000129.

Visual scenes are categorized by function.

Author information

1
Department of Computer Science, Stanford University.
2
Department of Electrical Engineering, Stanford University.
3
Department of Psychology, University of Illinois at Urbana-Champaign.

Abstract

How do we know that a kitchen is a kitchen by looking? Traditional models posit that scene categorization is achieved through recognizing necessary and sufficient features and objects, yet there is little consensus about what these may be. However, scene categories should reflect how we use visual information. Therefore, we test the hypothesis that scene categories reflect functions, or the possibilities for actions within a scene. Our approach is to compare human categorization patterns with predictions made by both functions and alternative models. We collected a large-scale scene category distance matrix (5 million trials) by asking observers to simply decide whether 2 images were from the same or different categories. Using the actions from the American Time Use Survey, we mapped actions onto each scene (1.4 million trials). We found a strong relationship between ranked category distance and functional distance (r = .50, or 66% of the maximum possible correlation). The function model outperformed alternative models of object-based distance (r = .33), visual features from a convolutional neural network (r = .39), lexical distance (r = .27), and models of visual features. Using hierarchical linear regression, we found that functions captured 85.5% of overall explained variance, with nearly half of the explained variance captured only by functions, implying that the predictive power of alternative models was because of their shared variance with the function-based model. These results challenge the dominant school of thought that visual features and objects are sufficient for scene categorization, suggesting instead that a scene's category may be determined by the scene's function.

PMID:
26709590
PMCID:
PMC4693295
DOI:
10.1037/xge0000129
[Indexed for MEDLINE]
Free PMC Article

Supplemental Content

Full text links

Icon for American Psychological Association Icon for PubMed Central
Loading ...
Support Center