In primary progressive aphasia (PPA), speech and language difficulties are caused by neurodegeneration of specific brain networks. In the nonfluent/agrammatic variant (nfvPPA), motor speech and grammatical deficits are associated with atrophy in a left fronto-insular-striatal network previously implicated in speech production. In vivo dissection of the crossing white matter (WM) tracts within this "speech production network" is complex and has rarely been performed in health or in PPA. We hypothesized that damage to these tracts would be specific to nfvPPA and would correlate with differential aspects of the patients' fluency abilities. We prospectively studied 25 PPA and 21 healthy individuals who underwent extensive cognitive testing and 3 T MRI. Using residual bootstrap Q-ball probabilistic tractography on high angular resolution diffusion-weighted imaging (HARDI), we reconstructed pathways connecting posterior inferior frontal, inferior premotor, insula, supplementary motor area (SMA) complex, striatum, and standard ventral and dorsal language pathways. We extracted tract-specific diffusion tensor imaging (DTI) metrics to assess changes across PPA variants and perform brain-behavioral correlations. Significant WM changes in the left intrafrontal and frontostriatal pathways were found in nfvPPA, but not in the semantic or logopenic variants. Correlations between tract-specific DTI metrics with cognitive scores confirmed the specific involvement of this anterior-dorsal network in fluency and suggested a preferential role of a posterior premotor-SMA pathway in motor speech. This study shows that left WM pathways connecting the speech production network are selectively damaged in nfvPPA and suggests that different tracts within this system are involved in subcomponents of fluency. These findings emphasize the emerging role of diffusion imaging in the differential diagnosis of neurodegenerative diseases.