Despite the critical importance of close replications in strengthening and advancing scientific knowledge, there are inherent challenges to conducting replications of lesion-based studies. In the present study, we conducted a close conceptual replication of a study (i.e., Hope et al., 2016) that found that fluency and naming scores in post-stoke aphasia were more strongly associated with a binary measure of structural white matter integrity (tract disconnection) than a graded measure (lesion load). Using a different sample of stroke patients (N=128) and four language deficit measures (aphasia severity, picture naming, and composite scores for speech production and semantic cognition), we examined tract disconnection and lesion load in three white matter tracts that have been implicated in language processing: arcuate fasciculus, uncinate fasciculus, and inferior fronto-occipital fasciculus. We did not find any consistent evidence that binary tract disconnection was more strongly associated with language impairment over and above lesion load, though individual deficit measures differed with respect to whether lesion load or tract disconnection was the stronger predictor. Given the mixed findings, we suggest caution when using such indirect estimates of structural white matter integrity, and direct measurements (for example, diffusion weighted imaging) should be preferred when they are available. We end by highlighting the complex nature of replication in lesion-based studies and offer some potential solutions.