Raman spectroscopy is a very sensitive tool for probing semiconductor nanocrystals. The underlying mechanism behind the size-dependent Raman shifts is still quite controversial. Here we offer a new theoretical method for the quantum confinement effects on the Raman spectra of semiconductor nanocrystals. We propose that the shift of Raman spectra in nanocrystals can result from two overlapping effects: the quantum effect shift and surface effect shift. The quantum effect shift is extracted from an extended Kubo formula, the surface effect shift is determined via the first principles calculations. Fairly good prediction of Raman shifts can be obtained without the use of any adjustable parameter. Closer analysis shows that the size-dependent Raman shifts in Si nanocrystals mainly result from the quantum effect shifts. For nanodiamond, the proportion of surface effect shift in Raman shift is up to about 40%. Such model can also provide a good baseline for using Raman spectroscopy as a tool to measure size.