We present the results of polarimetric analysis of collagen on varying pathologies of breast tissues using second-harmonic patterned polarization-analyzed reflection confocal (SPPARC) microscopy. Experiments are conducted on a breast tissue microarray having benign tissues (BT), malignant invasive lobular carcinoma (ILC), and benign stroma adjacent to the malignant tissues (called the benign adjacent tissue, or BAT). Stroma in BAT and ILC exhibit the largest parameter differences. We observe that stromal collagen readings in ILC show lower depolarization, lower diattenuation and higher linear degree-of-polarization values than stromal collagen in BAT. This suggests that the optical properties of collagen change most in the vicinity of tumors. A similar trend is also exhibited in the non-collagenous extrafibrillar matrix plus cells (EFMC) region. The three highlighted parameters show greatest sensitivity to changes in the polarization response of collagen between pathologies.