People are able to accurately estimate personality traits, merely on the basis of “passport”-style neutral faces and, thus, cues must exist that allow for such estimation. However, up to date, there has been little progress in identifying the form and location of these cues. In this paper we address the problem of inferring true personality traits in highly constrained images using state of art machine learning techniques, in particular, deep networks and class activation maps analysis. The novelty of our work consists in that, differently from the vast majority of the current and past approaches (that refer to the problem of consensus personality rating prediction) we predict the genuine personality based on highly constrained images: the targets are self ratings on a validated personality inventory and we restrict to passport-like photos, in which so-called controllable cues are minimized. Our results show that self-reported personality traits can be accurately evaluated from facial features. A preliminary analysis on the features activation maps shows promising results for a deeper understanding on relevant facial cues for traits estimation.