This research seeks to contribute new understanding of color disparities and gender in cognitive aging among older adults residing in Puerto Rico. We use the island-representative Puerto Rican Elderly Health Conditions (PREHCO) longitudinal study that measures cognitive health at baseline and cognitive decline between waves. In pooled models, we discern little or no color disparities in cognition at baseline. Sex-stratified models of baseline cognition indicate that Trigueño men slightly outperform white men. In contrast, color disparities in cognitive decline are apparent. In just four years between the two waves of PREHCO, on a 20-point cognitive test scale, Black men experienced 0.78 more points of cognitive decline, while Trigueño men experienced 0.44 more points of cognitive decline than white men in Puerto Rico. Mestiza women experience 0.80 less points of cognitive decline relative to white women. Nearly all of the color/race association with cognitive decline appears to be independent from health behaviors and conditions, individual human capital attainment, and family background. While lower-status color groups more frequently report discrimination, discrimination does not mediate the impact of color/skin tone and cognitive performance, suggesting the importance of further research on the role of broader dimensions of life course structural racism.