When a cosmic microwave background (CMB) photon travels from the surface of last scatter through spacetime metric perturbations, the polarization vector may rotate about its direction of propagation. This gravitational rotation is distinct from, and occurs in addition to, the lensing deflection of the photon trajectory. This rotation can be sourced by linear vector or tensor metric perturbations and is fully coherent with the curl deflection field. Therefore, lensing corrections to the CMB polarization power spectra as well as the temperature-polarization cross correlations due to nonscalar perturbations are modified. The rotation does not affect lensing by linear scalar perturbations, but needs to be included when calculations go to higher orders. We present complete results for weak lensing of the full-sky CMB power spectra by general linear metric perturbations, taking into account both deflection of the photon trajectory and rotation of the polarization. For the case of lensing by gravitational waves, we show that the B modes induced by the rotation largely cancel those induced by the curl component of deflection.