A semiconductor quantum dot can generate highly coherent and indistinguishable single photons. However, intrinsic semiconductor dephasing mechanisms can reduce the visibility of two-photon interference. For an electron in a quantum dot, a fundamental dephasing process is the hyperfine interaction with the nuclear spin bath. Here, we directly probe the consequence of the fluctuating nuclear spins on the elastic and inelastic scattered photon spectra from a resident electron in a single dot. We find the in-plane component of the nuclear Overhauser field leads to detuned Raman scattered photons, broadened over experimental time scales by field fluctuations, which are distinguishable from both the elastic and incoherent components of the resonance fluorescence. This significantly reduces two-photon interference visibility. However, we demonstrate successful screening of the nuclear spin noise, which enables the generation of coherent single photons that exhibit high visibility two-photon interference.