Optoelectronic applications of colloidal quantum dots demand a high emission efficiency, stability in time, and narrow spectral bandwidth. Electronic trap states interfere with the above properties but understanding of their origin remains lacking, inhibiting the development of robust passivation techniques. Here we show that surface vacancies improve the fluorescence yield compared to vacancy-free surfaces, while dynamic vacancy aggregation can temporarily turn fluorescence off. We find that infilling with foreign cations can stabilize the vacancies, inhibiting intermittency and improving quantum yield, providing an explanation of recent experimental observations.