We model how people make third party emotion attributions as integration of perceptual cues and conceptual event knowledge in an intuitive causal theory of mind. Novel stimuli generated from a televised gameshow provide authentic (not staged) dynamic displays of emotion in the context of a quantifiable and repeatable game (a one-shot prisoner's dilemma). The gameshow involves public acts of cooperation, commitment, and betrayal, with stakes spanning five orders of magnitude (max ≈ $200,000), and thus supports a wide range of inferred emotions. The raw footage is separated into expression cues and contextual descriptions such that each player's emotions can be inferred from the player's reactions to the outcome (i.e. facial expressions and body postures), or from the event context (i.e. stakes, actions, and outcomes), or from both information sources together.
Study participants attribute the experience of 20 nuanced emotions to the contestants based on (i) only dynamic visual emotion cues, (ii) only event descriptions, and (iii) combined dynamic visual cues and event descriptions. Principal component analysis and hierarchical clustering of emotion ratings are used to assess the dimensionality and structure of the attribution space supported by each unimodal signal as well as by the multimodal signal. The attributions are modeled using general linear regression with a priori features derived from behavioral economics and experimental psychology, including prospect theory, expected utility, and loss aversion. Finally, a Bayesian generative cue-combination model tests how effective joint conditioning on the unimodal signals is in explaining the emotion inferences participants make when given multimodal information.