Attention shifting in the working memory system plays an important role in goal-oriented behavior, such as reading, reasoning, and driving, because it involves several cognitive processes. This study identified brain activity leading to individual differences in attention shifting for dual-task performance by using the group comparison approach. A large-scale pilot study was initially conducted to select suitable good and poor performers. The fMRI experiment consisted of a dual-task condition and two single-task conditions. Under the dual-task condition, participants verified the status of letters while concurrently retaining arrow orientations. The behavioral results indicated that accuracy in arrow recognition was better in the good performers than in the poor performers under the dual-task condition but not under the single-task condition. Dual-task performance showed a positive correlation with mean signal change in the right anterior cingulate cortex (ACC) and right dorsolateral prefrontal cortex (DLPFC). Structural equation modeling indicated that effective connectivity between the right ACC and right DLPFC was present in the good performers but not in the poor performers, although activations of the task-dependent posterior regions were modulated by the right ACC and right DLPFC. We conclude that individual differences in attention shifting heavily depend on the functional efficiency of the cingulo-prefrontal network.