Prostate adenocarcinoma undergoes neuroendocrine differentiation to acquire resistance toward antihormonal therapies. The underlying mechanisms have been investigated extensively, among which Sox2 has been shown to play a critical role. However, genetic evidence in mouse models for prostate cancer to support the crucial role of Sox2 is missing. The adult mouse prostate luminal cells contain both castration-resistant Sox2-expressing Sca-1+ cells and castration-responsive Sca-1- cells. We show that both types of the luminal cell are susceptible to oncogenic transformation induced by loss of function of the tumor suppressor Pten. The tumors derived from the Sca-1+ cells are castration resistant and are more inclined to develop castration-induced neuroendocrine differentiation. Genetic ablation of Sox2 suppresses neuroendocrine differentiation but does not impact the castration-resistant property. This study provides direct genetic evidence that Sox2 is necessary for androgen ablation-induced neuroendocrine differentiation of Pten null prostate adenocarcinoma, corroborates that the lineage status of the prostate cancer cells is a determinant for its propensity to exhibit lineage plasticity, and supports that the intrinsic features of cell-of-origin for prostate cancers can dictate their clinical behaviors.