We effortlessly categorise other people along socially relevant categories such as sex, age, and emotion. A core question in social vision relates to whether we perceive these categories independently or in relation to each other. Here, we investigated categorisation of sex and emotion from the body, finding that participants generally fail to fully ignore task-irrelevant variations of sex while judging body emotional expressions. In contrast, sex categorisation was unaffected by variations in emotional expression. This asymmetric interaction between sex and emotion may arise because of bottom-up visual processing, due to partially shared visual features used for both judgments, or because of top-down, categorical associations between sex and emotion categories. To disentangle these possibilities, we used cortico-cortical paired associative stimulation (ccPAS) to modulate the connectivity between primary visual cortex and the extrastriate body area. We posited that boosting forward connectivity between these regions would increase efficiency of feature-based processing, while boosting feedback connectivity would enhance the separability of semantic categories related to sex and emotion. We found that boosting forward connectivity eliminated the interference of sex on emotion judgments, while that interference remained unaffected with modulation of feedback connectivity. These findings suggest that interactions between sex and emotion in body perception emerge during the perceptual analysis of the stimuli, and add to our understanding of person perception and social categorization.