Background Resident and curriculum evaluation require tracking surgical resident operative performance, yet what and when to measure remains unclear. Methods From a multi-institutional database, we reviewed 611 resident/surgeon-paired assessments of ACGME Milestones and modified OPRS ratings for different cases and postgraduate years. Results Faculty Milestone ratings increased with each PGY (p=<0.001) and correlated with resident self-ratings (ICC = 0.83). Mean OPRS scores increased in small increments with substantial intra-year variability. Progression among individual OPRS subcategories was not apparent from more global analyses. Interestingly, male faculty offered lower ratings than female faculty. Conclusions Milestones and modified mean OPRS ratings suggest residents are learning, yet lack sufficient discrimination for promotion or curricular analysis. Differential progression through OPRS subcategories suggests a taxonomy of surgical learning that can be tailored to focus on different skills at each point in the training continuum. The effect of faculty gender on resident ratings awaits further study.