Understanding and responding to other people's actions is fundamental for social interactions. Whereas many studies emphasize the importance of parietal and frontal regions for these abilities, several lines of recent research show that the human lateral occipitotemporal cortex (LOTC) represents varied aspects of action, ranging from perception of tools and bodies and the way they typically move, to understanding the meaning of actions, to performing overt actions. Here, we highlight common themes across these lines of work, which have informed theories related to high-level vision, concepts, social cognition, and apraxia. We propose that patterns of activity in LOTC form representational spaces, the dimensions of which capture perceptual, semantic, and motor knowledge of how actions change the state of the world.