In this Letter, we present an analysis of the effects of polarization state on the pattern contrast in a structured illumination microscope. Using vectorial ray tracing methods, we show that the contrast varies nonmonotonically with both the numerical aperture of the microscope objective lens and the orientation of the electric field with respect to the meridional plane. By careful selection of these two parameters, high pattern contrast can be obtained without polarization rotation, reducing the cost and complexity of structured illumination imaging systems and increasing light throughput and imaging speed. We present experimental results that show good agreement with theoretical predictions and discuss the implications for super-resolution imaging.