We investigated cross-sectional and longitudinal associations of neighborhood environment characteristics with accelerometer-measured sedentary time (SED), light-intensity physical activity (LPA), and moderate-to-vigorous intensity physical activity (MVPA). Participants were 2120 men and women in the year 20 (2005–2006) and year 30 CARDIA exams (2015–2016). Year 20 neighborhood characteristics included neighborhood cohesion, resources for physical activity, poverty, and racial residential segregation. Physical activity was measured by accelerometer at years 20 and 30. Multivariable linear regression models examined associations of standardized neighborhood measures at year 20 with SED, LPA, and MVPA assessed that year, and with 10-year changes in SED, LPA, and MVPA. Cross-sectionally, a one standard deviation (SD) increase in cohesion was associated with 4.06 less SED min/day (95% CI: −7.98, −0.15), and 4.46 more LPA min/day (95% CI: 0.88, 8.03). Each one SD increase in resources was associated with 1.19 more MVPA min/day (95% CI: 0.06, 2.31). A one SD increase in poverty was associated with 11.18 less SED min/day (95% CI: −21.16, −1.18) and 10.60 more LPA min/day (95% CI: 1.79, 19.41) among black men. No neighborhood characteristic was associated with 10-year changes in physical activity in the full sample; however, a one SD increase in cohesion was associated with a 10-year decrease of 25.44 SED min/day (95% CI: −46.73, −4.14) and an increase of 19.0 LPA min/day (95% CI, 1.89, 36.10) in black men. Characteristics of the neighborhood environment are associated with accelerometer-measured physical activity. Differences were observed by race and sex, with more robust findings observed in black men.