Sociality should evolve when the fitness benefits of group living outweigh the costs. Theoretical models predict an optimal group size maximizing individual fitness. However, beyond the number of individuals present in a group, the characteristics of these individuals, like their sex, are likely to affect the fitness payoffs of group living. Using 20 years of individually based data on a social mammal, the Alpine marmot (Marmota marmota), we tested for the occurrence of an optimal group size and composition, and for sex-specific effects of group characteristics on fitness. Based on lifetime data of 52 males and 39 females, our findings support the existence of an optimal group size maximizing male fitness and an optimal group composition maximizing fitness of males and females. Additionally, although group characteristics (i.e., size, composition and instability) affecting male and female fitness differed, fitness depended strongly on the number of same-sex subordinates within the social group in the two sexes. By comparing multiple measures of social group characteristics and of fitness in both sexes, we highlighted the sex-specific determinants of fitness in the two sexes and revealed the crucial role of intrasexual competition in shaping social group composition.