Participants are slower to report a feature, such as color, when the target appears on the side opposite the instructed response, than when the target appears on the same side. This finding suggests that target location, even when task-irrelevant, interferes with response selection. This effect is magnified in older adults. Lengthening the inter-trial interval, however, suffices to normalize the congruency effect in older adults, by re-establishing young-like sequential effects (Aisenberg, Sapir, d’Avossa, Henik, 2014). We examined the neurological correlates of age related changes by comparing BOLD signals in young and old participants performing a visual version of the Simon task. Participants reported the color of a peripheral target, by a left or right-hand keypress. Generally, BOLD responses were greater following incongruent than congruent targets. Also, they were delayed and of smaller amplitude in old than young participants. BOLD responses in visual and motor regions were also affected by the congruency of the previous target, suggesting that sequential effects may reflect remapping of stimulus location onto the hand used to make a response. Crucially, young participants showed larger BOLD responses in right anterior cerebellum to incongruent targets, when the previous target was congruent, but smaller BOLD responses to incongruent targets when the previous target was incongruent. Old participants, however, showed larger BOLD responses to congruent than incongruent targets, irrespective of the previous target congruency. We conclude that aging may interfere with the trial by trial updating of the mapping between the task-irrelevant target location and response, which takes place during the inter-trial interval in the cerebellum and underlays sequential effects in a Simon task.