Gaze direction perception triggers rapid visuospatial orienting to the location observed by others. When this is congruent with the location of a target, reaction times are faster than when incongruent. Functional magnetic resonance imaging studies suggest that the non-joint attention induced by incongruent cues are experienced as more emotionally negative and this could relate to less favorable trust judgments of the faces when gaze-cues are contingent with identity. Here, we provide further support for these findings using time-resolved event-related potentials. In addition to replicating the effects of identity-contingent gaze-cues on reaction times and trust judgments, we discovered that the emotion-related late positive potential increased across blocks to incongruent compared to congruent faces before, during and after the gaze-cue, suggesting both learning and retrieval of emotion states associated with the face. We also discovered that the face-recognition-related N250 component appeared to localize to sources in anterior temporal areas. Our findings provide unique electrophysiological evidence for the role of emotion in learning trust from gaze-cues, suggesting that the retrieval of face evaluations during interaction may take around 1000 ms and that the N250 originates from anterior temporal face patches.