Randomized prospective trials have shown the benefits of adjuvant radiotherapy in decreasing local recurrence rates in stage II and stage III rectal cancer. However, some patients with stage II lesions have relatively low risks of local recurrence when treated with modern surgery alone. This article will discuss important prognostic, diagnostic, and therapeutic factors including depth of tumor invasion, tumor location, improvements in staging with endorectal ultrasound and magnetic resonance imaging, enhanced surgical technique with total mesorectal excision, circumferential tumor margin, and lymph node dissection that may help to better define a subset of stage II rectal cancer patients in which pelvic radiation may be safely omitted.