Introduction Research on perceptions of environmental factors in relation to transportation and recreation bicycling is limited in the United States. We explored the association between perceived social and built environment factors with total, transportation, and recreation bicycling in a sample of adult bicyclists in Austin, Texas, and Birmingham, Alabama. The objective of this study was to examine the relationship between perceived social and built environment factors and domain-specific bicycling in a sample of adult bicyclists. Methods Adults aged 18 to 65 who rode a bicycle at least once in the past year completed an internet-based survey that was developed for this study to specifically assess correlates of bicycling; the study was conducted from October 2016 through January 2017. Perceived environmental factors assessed were residential density, traffic safety, destination, connectivity, safety from crime, aesthetics, and bicycle infrastructure. Multivariable logistic regression models were used to estimate the association of each perceived environmental factor (tertile 1, lowest; tertile 3, highest) with recreation- only and transportation bicycling. Effect modification of the relation between environmental factors and bicycling outcomes by sex was also examined. Results The final analytic sample size was 801 participants. All environmental factors examined, including residential density, traffic safety, destinations, connectivity, aesthetics, bicycle infrastructure, and safety from crime showed significantly direct associations with transportation bicycling. Traffic safety, destinations, aesthetics, and bicycle infrastructure showed significant direct and inverse associations with recreation-only bicycling. Effect modification by sex was identified with residential density; a significant direct association with recreation-only bicycling was seen among women. Conclusion These findings illustrate that bicycling for transportation is associated with different perceived environmental factors than is recreation- only bicycling, with some significant modification by sex. Comprehensive tools that assess the perceived environment for bikeability in the United States are warranted.