Action perception and action production are tightly linked and elicit bi-directional influences on each other when performed simultaneously. In this study, we investigated whether age-related differences in manual fine-motor competence and/or age affect the (interfering) influence of action production on simultaneous action perception. In a cross-sectional eye-tracking study, participants of a broad age range (N = 181, 20-80 years) observed a manual grasp-and-transport action while performing an additional motor or cognitive distractor task. Action perception was measured via participants' frequency of anticipatory gaze shifts towards the action goal. Manual fine-motor competence was assessed with the Motor Performance Series. The interference effect in action perception was greater in the motor than the cognitive distractor task. Furthermore, manual fine-motor competence and age in years were both associated with this interference. The better the participants' manual fine-motor competence and the younger they were, the smaller the interference effect. However, when both influencing factors (age and fine-motor competence) were taken into account, a model including only age-related differences in manual fine-motor competence best fit with our data. These results add to the existing literature that motor competence and its age-related differences influence the interference effects between action perception and production.