X-ray phase contrast imaging (PCI) combined with phase retrieval has the potential to improve soft-material visibility and discrimination. This work examined the accuracy, image quality gains, and robustness of a spectral phase retrieval method proposed by our group. Spectroscopic PCI measurements of a physical phantom were obtained using state-of-the-art photon-counting detectors in combination with a polychromatic x-ray source. The phantom consisted of four poorly attenuating materials. Excellent accuracy was demonstrated in simultaneously retrieving the complete refractive properties (photoelectric absorption, attenuation, and phase) of these materials. Approximately 10 times higher SNR was achieved in retrieved images compared to the original PCI intensity image. These gains are also shown to be robust against increasing quantum noise, even for acquisition times as low as 1 s with a low-flux microfocus x-ray tube (average counts of 250 photons/pixels). We expect that this spectral phase retrieval method, adaptable to several PCI geometries, will allow significant dose reduction and improved material discrimination in clinical and industrial x-ray imaging applications.