Processing a famous face involves a cascade of steps including detecting the presence of a face, recognizing it as familiar, accessing semantic/biographical information about the person, and finally, if required, production of the proper name. Decades of neuropsychological and neuroimaging studies have identified a network of occipital and temporal brain regions ostensibly comprising the 'core' system for face processing. Recent research has also begun to elucidate upon an 'extended' network, including anterior temporal and frontal regions. However, there is disagreement about which brain areas are involved in each step, as many aspects of face processing occur automatically in healthy individuals and rarely dissociate in patients. Moreover, some common phenomena are not easily induced in an experimental setting, such as having a sense of familiarity without being able to recall who the person is. Patients with the semantic variant of Primary Progressive Aphasia (svPPA) often recognize a famous face as familiar, even when they cannot specifically recall the proper name or biographical details. In this study, we analyzed data from a large sample of 105 patients with neurodegenerative disorders, including 43 svPPA, to identify the neuroanatomical substrates of three different steps of famous face processing. Using voxel-based morphometry, we correlated whole-brain grey matter volumes with scores on three experimental tasks that targeted familiarity judgment, semantic/biographical information retrieval, and naming. Performance in naming and semantic association significantly correlates with grey matter volume in the left anterior temporal lobe, whereas familiarity judgment with integrity of the right anterior middle temporal gyrus. These findings shed light on the neuroanatomical substrates of key components of overt face processing, addressing issues of functional lateralization, and deepening our understanding of neural substrates of semantic knowledge.