Participants memorized briefly presented sets of digits, a subset of which had to be accessed as input for arithmetic tasks (the active set), whereas another subset had to be remembered independently of the concurrent task (the passive set). Latencies for arithmetic operations were a function of the setsize of active but not passive sets. Object-switch costs were observed when successive operations were applied to different digits within an active set. Participants took 2 s to encode a passive set so that it did not affect processing latencies (Experiment 2). The results support a model distinguishing 3 states of representations in working memory: the activated part of long-term memory, a capacity limited region of direct access, and a focus of attention.