Attention-dependent modulation of neural activity in visual association cortex (VAC) is thought to depend on top-down modulatory control signals emanating from the prefrontal cortex (PFC). In a previous functional magnetic resonance imaging study utilizing a working memory task, we demonstrated that activity levels in scene-selective VAC (ssVAC) regions can be enhanced above or suppressed below a passive viewing baseline level depending on whether scene stimuli were attended or ignored (Gazzaley, Cooney, McEvoy, et al. 2005). Here, we use functional connectivity analysis to identify possible sources of these modulatory influences by examining how network interactions with VAC are influenced by attentional goals at the time of encoding. Our findings reveal a network of regions that exhibit strong positive correlations with a ssVAC seed during all task conditions, including foci in the left middle frontal gyrus (MFG). This PFC region is more correlated with the VAC seed when scenes were remembered and less correlated when scenes were ignored, relative to passive viewing. Moreover, the strength of MFG-VAC coupling correlates with the magnitude of attentional enhancement and suppression of VAC activity. Although our correlation analyses do not permit assessment of directionality, these findings suggest that PFC biases activity levels in VAC by adjusting the strength of functional coupling in accordance with stimulus relevance.