People divide their time unequally among their social contacts due to time constraints and varying strength of relationships. It was found that high diversity of social communication, dividing time more evenly among social contacts, is correlated with economic well-being both at macro and micro levels. Besides economic well-being, it is not clear how the diversity of social communication is also associated with the two components of individuals' subjective well-being, positive and negative affect. Specifically, positive affect and negative affect are two independent dimensions representing the experience (feeling) of emotions. In this paper, we investigate the relationship between the daily diversity of social communication and dynamic affect states that people experience in their daily lives. We collected two high-resolution datasets that capture affect scores via daily experience sampling surveys and social interaction through wearable sensing technologies: sociometric badges for face-to-face interaction and smart phones for mobile phone calls. We found that communication diversity correlates with desirable affect states--e.g. an increase in the positive affect state or a decrease in the negative affect state--for some personality types, but correlates with undesirable affect states for others. For example, diversity in phone calls is experienced as good by introverts, but bad by extroverts; diversity in face-to-face interaction is experienced as good by people who tend to be positive by nature (trait) but bad for people who tend to be not positive by nature. More broadly, the moderating effect of personality type on the relationship between diversity and affect was detected without any knowledge of the type of social tie or the content of communication. This provides further support for the power of unobtrusive sensing in understanding social dynamics, and in measuring the effect of potential interventions designed to improve well-being.