We report a 1.5- mum band polarization independent single photon detector based on frequency up-conversion in periodically poled lithium niobate (PPLN) waveguides. To overcome the polarization dependence of the PPLN waveguides, we employed a polarization diversity configuration composed of two up-conversion detectors connected with a polarization beam splitter. We experimentally confirmed polarization independent single photon counting using our detector. We undertook a proof-of-principle differential phase shift quantum key distribution experiment using the detector, and confirmed that the sifted key rate and error rate remained stable when the polarization state was changed during single photon transmission.