The goal of this study was to determine the relationship between age-related white matter changes, with a specific focus on previously identified language pathways, and language functioning in healthy aging. 228 healthy participants (126 female; 146 right-handed), ages 18 to 76, underwent 3.0 Tesla MR diffusion weighted imaging (DWI) and a battery of language assessments including the Boston Naming Test (BNT), the Peabody Picture Vocabulary Test (PPVT), the Controlled Oral Word Association Test (COWAT), Semantic Fluency Test (SFT), and a subset of the Boston Diagnostic Aphasia Examination (CI-BDAE). Using tract based spatial statistics (TBSS), we investigated measurements of fractional anisotropy (FA), axial diffusivity (AD), radial diffusivity (RD), and mean diffusivity (MD). TBSS was used to create a white matter skeleton that was then used to analyze white matter changes (indexed by FA, AD, RD, and MD) with age and language performance. Results focused primarily on significant relationships (p < 0.05, cluster-wise FDR corrected for multiple comparisons) in the canonical language white matter pathways. We found a diffuse linear decrease with age in global white matter FA and a significant focal increase in FA with age within the bilateral superior cerebellar peduncles (SCPs). We observed that increased BNT scores were associated with increased FA within the left SLF, and within the posterior and antero-lateral portions of the right inferior frontal-occipital fasciculus (IFOF). Increased SFT and PPVT scores were associated with increased FA within the posterior portion of the right IFOF and increased FA within the left body of the corpus callosum was associated with lower COWAT scores. We found no association between FA and BDAE. MD, RD, and AD, were found to be inversely proportional to FA within the IFOF, with AD showing a negative correlation with SFT, and RD and MD showing a negative correlation with BNT. There was no association between CI-BDAE and any of the white matter measures. Significant differences between sexes included more pronounced FA decrease with age within the right SLF in males vs. females; there were no differences in language performance scores between sexes. We also found that there was no decline in language testing scores with increasing age in our cohort. Taken together, our findings of varying relationships between DTI metrics and language function within multiple regions of the non-dominant IFOF suggest that more robust language networks with bilateral structural connectivity may contribute to better overall language functioning, regardless of age.