In the absence of distractors, saccadic latencies are influenced by target characteristics such as contrast. The same characteristics were expected to influence the remote distractor effect (RDE) when varied in the distractor. We conducted three experiments in which we varied target and distractor contrast orthogonally. The results show that the RDE is not so much modulated by distractor contrast per se. Rather it strongly depended on the overall saccadic latencies afforded by the target: typically, shorter latencies resulted in stronger RDEs. We argue that average saccadic latencies to a target determine whether distractor-related activity temporally coincides with target-related activity. The temporal overlap is necessary for the respective neuronal signals to inhibit each other, thus evoking the RDE.