We efficiently infer others’ traits from their faces, and these inferences powerfully shape our social behaviour. Here, we investigated how sex is represented in facial appearance. Based on previous findings from sex-judgment tasks, we hypothesized that the perceptual encoding of sex is not balanced but rather polarized: for the processes that generate a sex percept, the default output is “male,” and the representation of female faces extends that of the male, engaging activity over unique detectors that are not activated by male faces. We tested this hypothesis with the logic of Treisman’s studies of visual search asymmetries, predicting that observers should more readily detect the presence of female faces amongst male distractors than vice versa. Across three experiments (N = 32 each), each using different face stimuli, we confirmed this prediction in response time and sensitivity measures. We apply GIST analyses to the face stimuli to exclude that the search asymmetry is explained by differences in image homogeneity. These findings demonstrate a property of the coding that links facial appearance with a significant social trait: the female face is coded as an extension of a male default. We offer a mechanistic description of perceptual detectors to account for our findings and posit that the origins of this polarized coding scheme are an outcome of biased early developmental experience.