We present fiber-based polarization-sensitive swept-source optical coherence tomography (SS-OCT) based on continuous source polarization modulation. The light source is a frequency swept laser centered at 1.31 microm with a scanning rate of 20 kHz. The incident polarization is modulated by a resonant electro-optic modulator at 33.3 MHz, which is one-third of the data acquisition frequency. The zeroth- and first-order harmonic components of the OCT signals with respect to the polarization modulation frequency have the polarimetric information of the sample. By algebraic and matrix calculations of the signals, this system can measure the depth-resolved Jones matrices of the sample with a single wavelength scan. The phase fluctuations of the starting trigger of wavelength scan and the polarization modulation are cancelled by monitoring the OCT phase of a calibration mirror inserted into the sample arm. We demonstrate the potential of the system by the measurement of chicken breast muscle and the volumetric measurement of an in vivo human anterior eye segment. The phase retardation image shows an additional contrast in the fibrous tissue such as the collagen fiber in the trabecular meshwork and sclera.