Fractal dimensionality (FD) measures the complexity within the folds and ridges of cortical and subcortical structures. We tested the degree that FD might provide a new perspective on the atrophy-compensation hypothesis: age or disease-related atrophy causes a compensatory neural response in the form of increased brain activity in the prefrontal cortex to maintain cognition. Brain structural and functional data were collected from 63 middle-aged and older adults and 18 young-adult controls. Two distinct patterns of FD were found that separated cortical from subcortical structures. Subcortical FD was more strongly negatively correlated with age than cortical FD, and cortical FD was negatively associated with brain activity during memory retrieval in medial and lateral parietal cortices uniquely in middle-aged and older adults. Multivariate analyses revealed that the lower FD/higher brain activity pattern was associated with poorer cognition—patterns not present in young adults, consistent with compensation. Bayesian analyses provide further evidence against the modal interpretation of the atrophy-compensation hypothesis in the prefrontal cortex—a key principle found in some neurocognitive theories of aging.