Understanding social interactions (such as ‘hug’ or ‘fight’) is a basic and important capacity of the human visual system, but a challenging and still open problem for modeling. In this work we study visual recognition of social interactions, based on small but recognizable local regions. The approach is based on two novel key components: (i) A given social interaction can be recognized reliably from reduced images (called ‘minimal images’). (ii) The recognition of a social interaction depends on identifying components and relations within the minimal image (termed ‘interpretation’). We show psychophysics data for minimal images and modeling results for their interpretation. We discuss the integration of minimal configurations in recognizing social interactions in a detailed, high-resolution image.