The Questionnaire of Cognitive and Affective Empathy (QCAE) is a multiple dimensional measure of cognitive empathy [comprising primary factors for perspective taking (PT), online simulation (OS)], and affective empathy [comprising primary factors for emotion contagion (EC), proximal responsivity (PRO), and peripheral responsivity (PER)]. This study used independent clusters confirmatory factor analysis (ICM-CFA) and exploratory structural equation modeling (ESEM) to examine the scale's factor structure. A general community sample of 203 (men = 43, women = 160) between 17 and 63 years completed the QCAE. Although both the five-factor oblique and second order factor models showed good model fit, and clarity in the pattern of factor loadings, in the second-order factor model, none of the primary factors loaded significantly on their respective secondary factors, thereby favoring the five-factor oblique model. The factors in this model were supported in terms of external validity. Despite this, the factor for PRO in this model showed low reliability for meaning interpretation. A revised four-factor oblique model without the PRO factor showed good fit, clarity in the pattern of factor loadings, and reliability and validity for the factors in this model, thereby suggesting this to be the best model to represent ratings on the QCAE.