Although ~ 25% of colorectal cancer or polyp (CRC/P) cases show familial aggregation, current germline genetic testing identifies a causal genotype in the 16 major genes associated with high penetrance CRC/P in only 20% of these cases. As there are likely other genes underlying heritable CRC/P, we evaluated the association of variation at novel loci with CRC/P. We evaluated 158 a priori selected candidate genes by comparing the number of rare potentially disruptive variants (PDVs) found in 84 CRC/P cases without an identified CRC/P risk-associated variant and 2440 controls. We repeated this analysis using an additional 73 CRC/P cases. We also compared the frequency of PDVs in select genes among CRC/P cases with two publicly available data sets. We found a significant enrichment of PDVs in cases vs. controls: 20% of cases vs. 11.5% of controls with ≥ 1 PDV (OR = 1.9, p = 0.01) in the original set of cases. Among the second cohort of CRC/P cases, 18% had a PDV, significantly different from 11.5% (p = 0.02). Logistic regression, adjusting for ancestry and multiple testing, indicated association between CRC/P and PDVs in NTHL1 (p = 0.0001), BRCA2 (p = 0.01) and BRIP1 (p = 0.04). However, there was no significant difference in the frequency of PDVs at each of these genes between all 157 CRC/P cases and two publicly available data sets. These results suggest an increased presence of PDVs in CRC/P cases and support further investigation of the association of NTHL1, BRCA2 and BRIP1 variation with CRC/P.