Purpose: In older women, breast cancer and its treatment can have profound impact on their physical, mental, and social health, especially in frail patients. This study evaluated the association between frailty and long-term health-related quality of life (HRQOL) in older women undergoing breast cancer treatment. Methods: Using the Carolina Senior Registry (CSR), participants with breast cancer were contacted to complete a follow-up HRQOL questionnaire (median 4 years). Baseline Geriatric Assessment (GA) variables were used to calculate the Carolina Frailty Index (CFI) and categorize participants as robust, pre-frail, or frail. Outcomes included HRQOL domains of physical function, social roles, fatigue, depression, anxiety, pain, and sleep disturbance assessed using PROMIS® instruments. Regression modeling compared outcomes between frailty groups using adjusted mean differences (AMD). Results: Of 190 eligible patients, 63 completed follow-up HRQOL survey. Mean age was 70 years (range 65–86) and 91% were white. Based on the CFI, 49 (78%) patients were robust, 11 (18%) pre-frail, and 3 (5%) frail. After controlling for age and cancer stage, patients identified as pre-frail/frail reported worse physical function (AMD − 9.2, p < 0.001) and social roles (AMD − 7.2, p = 0.002) and more fatigue (AMD 7.6, p = 0.008), depression (AMD 5.6, p = 0.004), and sleep disturbance (AMD 6.9, p = 0.008) compared to robust patients at follow-up. Conclusions: Frailty in older women with breast cancer was associated with worse long-term HRQOL outcomes. Further research is needed to develop interventions for frail patients at-risk for reduced HRQOL.