Participants categorized left- and right-pointing line drawings presented together with pictures, whose pointing direction (to the left or right) was ambiguous (spatially ambiguous pictures; Experiments 1 and 2) or that pointed neither to the left nor to the right (spatially neutral pictures; Experiment 3). Subsequently, the spatially ambiguous and neutral pictures were used in a Simon task, wherein participants made left and right keypresses on the basis of the color of the pictures, while ignoring the object that they depicted. In all three experiments, performance was facilitated when the response required by the color matched the pointing direction of the line drawings with which the picture had been previously paired. Performance was impeded when the response required by the color did not match the pointing direction of the line drawings with which the picture had been previously paired. Thus, responses indirectly associated with pictures by category membership were automatically triggered even when the responses were inappropriate.