The default mode network (DMN) and semantic network (SN) are two of the most extensively studied systems, and both are increasingly used as clinical biomarkers in neurological studies. There are strong theoretical reasons to assume a relationship between the networks, as well as anatomical evidence that they might rely on overlapping cortical regions, such as the anterior temporal lobe (ATL) or angular gyrus (AG). Despite these strong motivations, the relationship between the two systems has received minimal attention. We directly compared the SN and DMN using a large (n = 69) distortion-corrected functional MRI (fMRI) dataset, spanning a range of semantic and nonsemantic tasks that varied input modality. The results showed that both networks fractionate depending on the semantic nature of the task, stimulus type, modality, and task difficulty. Furthermore, despite recent claims that both AG and ATL are semantic hubs, the two areas responded very differently, with results supporting the role of ATL, but not AG, in semantic representation. Specifically, the left ATL was positively activated for all semantic tasks, but deactivated during nonsemantic task performance. In contrast, the left AG was deactivated for all tasks, with the level of deactivation related to task difficulty. Thus, ATL and AG do not share a common interest in semantic tasks, but, rather, a common "disinterest" in nonsemantic tasks. The implications for the variability in the DMN, its cognitive coherence, and interpretation of resting-state fMRI data are discussed.