Neuroscientific investigations interested in questions of person perception and impression formation have traditionally asked their participants to observe and evaluate isolated individuals. In recent years, however, there has been a surge of studies presenting third-party encounters between two (or more) individuals as stimuli. Owing to this subtle methodological change, the brain's capacity to understand other people's interactions and relationships from limited visual information--also known as people watching--has become a distinct topic of inquiry. Though initial evidence indicates that this capacity relies on several well-known networks of the social brain (including the person-perception network, the action-observation network, and the mentalizing network), a comprehensive framework of people watching must overcome three major challenges. First, it must develop a taxonomy of judgments that people habitually make when witnessing the encounters of others. Second, it must clarify which visual cues give rise to these encounter-based judgments. Third, it must elucidate how and why several brain networks work together to accomplish these judgments. To advance all three lines of research, we summarize what is currently known as well as what remains to be studied about the neuroscience of people watching.