Using scanning tunneling spectroscopy, we have studied the interface effect on quantum well states of Pb thin films grown on various metal-terminated (Pb, Ag, and Au) n-type Si(111) surfaces and on two different p-type Si(111) surfaces. The dispersion relation E(k) of the electrons of the Pb film and the phase shift at the substrate interface were determined by applying the quantization rule to the measured energy positions of the quantum well states. Characteristic features in the phase shift versus energy curves were identified and were correlated to the directional conduction band of the silicon substrate and to the Schottky barrier formed between the metal film and the semiconductor. A model involving the band structure of the substrate, the Schottky barrier, and the effective thickness of the interface was introduced to qualitatively but comprehensively explain all the observed features of the phase shift at the substrate interface. Our physical understanding of the phase shift is critically important for using interface modification to control the quantum well states.