More than 90 capsular serotypes of Streptococcus pneumoniae coexist despite competing for nasopharyngeal carriage and a gradient in fitness. The underlying mechanisms for this are poorly understood and make assessment of the likely population impact of vaccination challenging. We use an individual-based simulation model to generalize widely used deterministic models for pneumococcal competition and show that in these models short-term serotype-specific and serotype non-specific immunity could constitute the mechanism governing between-host competition and coexistence. We find that non-specific immunity induces between-host competition and that serotype-specific immunity limits a type's competitive advantage and allows stable coexistence of multiple serotypes. Serotypes carried at low prevalence show high variance in carriage levels, which would result in apparent outbreaks if they were highly pathogenic. Vaccination against few serotypes can lead to elimination of the vaccine types and induces replacement by others. However, in simulations where the elimination of the targeted types is achieved only by a combination of vaccine effects and the competitive pressure of the non-vaccine types, a universal vaccine with similar-type-specific effectiveness can fail to eliminate pneumococcal carriage and offers limited herd immunity. Hence, if vaccine effects are insufficient to control the majority of serotypes at the same time, then exploiting the competitive pressure by selective vaccination can help control the most pathogenic serotypes.