It is unclear how resting myocardial workload, as indexed by baseline measures of rate-pressure product (RPP) and physical activity (PA), is associated with the overall risk of cancer mortality. We performed prospective analyses among 28,810 men and women from the REasons for Geographic and Racial Differences in Stroke (REGARDS) cohort. We used a novel physical health (PH) composite index and categorized participants into one of four groups based on combinations from self-reported PA and RPP: 1) No PA and High RPP; 2) No PA and Low RPP; 3) Yes PA and High RPP; and 4) Yes PA and Low RPP. We examined the association between baseline PH composite and cancer mortality adjusted for potential confounders using Cox regression. A total of 1191 cancer deaths were observed over the 10-year observation period, with the majority being lung (26.87%) and gastrointestinal (21.49%) cancers. Even after controlling for sociodemographics, health behaviors, baseline comorbidity score, and medications, participants with No PA and High RPP had 71% greater risk of cancer mortality when compared to participants with PA and Low RPP (adjusted HR: 1.71, 95% CI: 1.42–2.06). These associations persisted after examining BMI, smoking, income, and gender as effect modifiers and all-cause mortality as a competing risk. Poorer physical health composite, including the novel RPP metric, was associated with a nearly 2-fold long-term risk of cancer mortality. The physical health composite has important public health implications as it provides a measure of risk beyond traditional measure of obesity and physical activity.