Humans are capable of making inferences about other individuals' intentions and goals by evaluating their actions in relation to the constraints imposed by the environment. This capacity enables humans to go beyond the surface appearance of behavior to draw inferences about an individual's mental states. Presently unclear is whether this capacity is uniquely human or is shared with other animals. We show that cotton-top tamarins, rhesus macaques, and chimpanzees all make spontaneous inferences about a human experimenter's goal by attending to the environmental constraints that guide rational action. These findings rule out simple associative accounts of action perception and show that our capacity to infer rational, goal-directed action likely arose at least as far back as the New World monkeys, some 40 million years ago.