Human social interactions crucially depend on the ability to represent other agents' beliefs even when these contradict our own beliefs, leading to the potentially complex problem of simultaneously holding two conflicting representations in mind. Here, we show that adults and 7-month-olds automatically encode others' beliefs, and that, surprisingly, others' beliefs have similar effects as the participants' own beliefs. In a visual object detection task, participants' beliefs and the beliefs of an agent (whose beliefs were irrelevant to performing the task) both modulated adults' reaction times and infants' looking times. Moreover, the agent's beliefs influenced participants' behavior even after the agent had left the scene, suggesting that participants computed the agent's beliefs online and sustained them, possibly for future predictions about the agent's behavior. Hence, the mere presence of an agent automatically triggers powerful processes of belief computation that may be part of a "social sense" crucial to human societies.