Cognitive flexibility, a core aspect of executive functioning, is required for the speeded shifting between different tasks and sets. Using an interindividual differences approach, we examined whether cognitive flexibility, as assessed by the Delis-Kaplan card-sorting test, is associated with gray matter volume (GMV) and functional connectivity (FC) of regions of a core network of multiple cognitive demands as well as with different facets of trait impulsivity. The core multiple-demand network was derived from three large-scale neuroimaging meta-analyses and only included regions that showed consistent associations with sustained attention, working memory as well as inhibitory control. We tested to what extent self-reported impulsivity as well as GMV and resting-state FC in this core network predicted cognitive flexibility independently and incrementally. Our analyses revealed that card-sorting performance correlated positively with GMV of the right anterior insula, FC between bilateral anterior insula and midcingulate cortex/supplementary motor area as well as the impulsivity dimension "Premeditation." Importantly, GMV, FC and impulsivity together accounted for more variance of card-sorting performance than every parameter alone. Our results therefore indicate that various factors contribute individually to cognitive flexibility, underlining the need to search across multiple modalities when aiming to unveil the mechanisms behind executive functioning.