Background: Patients with limited health literacy (HL) have difficulty understanding written/verbal information. The quality of verbal communication is not well understood. Therefore, our aim was to characterize patient-surgeon conversations and identify opportunities for improvement. Methods: New colorectal patient-surgeon encounters were audio-recorded and transcribed. HL was measured. Primary outcomes were rates-of-speech, understandability of words, patient-reported understanding, and usage of medical jargon/statistics. Secondary outcomes included length-of-visit (LOV), conversation possession time, patient-surgeon exchanges, and speech interruptions. Results: Significant variations existed between surgeons in rates-of-speech and understandability of words (p < 0.05). Faster rates-of-speech were associated with significantly less understandable words (p < 0.05). Patient-reported understanding varied by HL and by surgeon. Conversation possession time and usage of medical jargon/statistics varied significantly by surgeon (p < 0.05) in addition to patient-surgeon exchanges and interruptions. Patients with limited HL had shorter LOV. Conclusions: Significant variations exist in how surgeons talk to patients. Opportunities to improve verbal communication include slowing speech and using more understandable words.