In recent years, multiple independent neuroscience investigations have implicated critical roles for the rostral temporal lobe in auditory and visual perception, language, and semantic memory. Although arising in the context of different cognitive functions, most of these suggest that there is a gradual convergence of sensory information in the temporal lobe that culminates in modality- and perceptually invariant representations at the most rostral aspect. Currently, however, too little is known regarding connectivity within the human temporal lobe to be sure of exactly how and where convergence occurs; existing hypotheses are primarily derived on the basis of cross-species generalizations from invasive nonhuman primate studies, the validity of which is unclear, especially where language function is concerned. In this study, we map the connectivity of the human rostral temporal lobe in vivo for the first time using diffusion-weighted imaging probabilistic tractography. The results indicate that convergence of sensory information in the temporal lobe is in fact a graded process that occurs along both its longitudinal and lateral axes and culminates in the most rostral limits. We highlight the consistency of our results with those of prior functional neuroimaging, computational modeling, and patient studies. By going beyond simple fasciculus reconstruction, we systematically explored the connectivity of specific temporal lobe areas to frontal and parietal language regions. In contrast to the graded within-temporal lobe connectivity, this intertemporal connectivity was found to dissociate across caudal, mid, and rostral subregions. Furthermore, we identified a basal rostral temporal region with very limited connectivity to areas outside the temporal lobe, which aligns with recent evidence that this subregion underpins the extraction of modality- and context-invariant semantic representations.