The anterior temporal lobes (ATLs) play a key role in conceptual knowledge representation. The hub-and-spoke theory suggests that the contribution of the ATLs to semantic representation is (a) transmodal, i.e., integrating information from multiple sensorimotor and verbal modalities, and (b) pan-categorical, representing concepts from all categories. Another literature, however, suggests that this region’s responses are modality- and category-selective; prominent examples include category selectivity for socially-relevant concepts and face recognition. The predictions of each approach have never been directly compared. We used data from three studies to compare category-selective responses within the ATLs. Study 1 compared ATL responses to famous people vs. another conceptual category (landmarks) from visual vs. auditory inputs. Study 2 compared ATL responses to famous people from pictorial and written word inputs. Study 3 compared ATL responses to a different kind of socially-relevant stimuli, namely abstract non-person related words, in order to ascertain whether ATL subregions are engaged for social concepts more generally or only for person-related knowledge. Across all three studies a dominant bilateral ventral ATL cluster responded to all categories in all modalities. Anterior to this “pan-category” transmodal region, a second cluster responded more weakly overall yet selectively for people, but did so equally for spoken names and faces (Study 1). A third region in the anterior superior temporal gyrus responded selectively to abstract socially-relevant words (Study 3), but did not respond to socially-relevant concrete words (i.e., written names; Study 2). These findings can be accommodated by the graded hub-and-spoke model of concept representation. On this view, the ventral ATL is the centrepoint of a bilateral ATL hub which contributes to conceptual representation through transmodal distillation of information arising from multiple modality-specific association cortices. Partial specialisation occurs across the graded ATL hub as a consequence of gradedly-differential connectivity across the region.