PMC

(A) In the first session, participants performed short-term recognition of faces, places, & objects inside the scanner. (B) At the beginning of the second session, outside the scanner, participants learned arbitrary cross-category pairs of stimuli. (C) Participants then returned to the scanner to perform short-term paired-associate recognition of the stimulus pairs they learned. Half of these trials included trial-irrelevant distraction during the delay period.

(A) In the first phase, participants performed short-term recognition of a pseudoword (phonological STM), a word (semantic STM) or two lines (visual STM). (B) In the second phase, during the same scanning session, participants performed short-term recognition with two stimuli (between-category combinations of pseudowords, words, and lines). On half of the trials, the same memory item was selected as behaviorally relevant by the first and second cues (repeat trials), and on the other half of trials the second cue selected the previously uncued item (switch trials).

Graph conventions are described in . The classifier training performance (A) and voxel importance maps (B) are shown for phonological (pho), semantic (sem), visual (vis), and resting state brain activity from the inter-trial interval (iti). Chance-level predication accuracy was 0.25 and is indicated by the dashed line on the top graph in (A). Unlike in , the voxels on the inflated brain hemispheres in (B) correspond to an overlap between one or more trial categories with the iti category. There were no important voxels that overlapped for pho, sem, and vis but not iti, and there were very few important voxels for the iti alone, and so all iti-related voxels were painted black.

(A) Classification results for the Phase 1 data is shown separately for all three categories on which it was trained: face (fac, yellow), place (pla, blue), and object (obj, red). Prediction accuracy is shown on the top graph (chance-level accuracy of 0.33 is indicated by the dashed line), and average classifier evidence is shown on the bottom graph. The evidence values reflect reliable category discrimination (e.g., for face trials, the classifier’s evidence for face was much higher than its evidence for either place or object). (B) Classifier-derived voxel importance maps show voxels whose activity exerted a strong influence on the classifier’s identification of a particular category. Group-averaged data are displayed on an inflated brain (left hemisphere in top tow, right hemisphere in bottom row; lateral view in left column; medial view in right column). Brain areas are colored according to the venn diagram in the center (e.g., black represents an overlap of all three categories).

Results are shown separately for repeat (left) and switch (right) trials. Classifier evidence values for phonological, semantic, and visual were relabeled and collapsed across all trials into three new categories: cued (red, the category of the memory item selected by the first cue), other (blue, the category of the other memory item), and irrel (grey, the trial-irrelevant category). The colored shapes along this horizontal axis indicate the onset of the targets (red and blue circles, 0 s), the first cue (red triangle, 10 s), the first recognition probe (red square, 18 s), the second cue (red or blue triangle, 22 s), and the final recognition probe (red or blue square, 30 s). Data for each category are shown as ribbons whose thickness indicate +/− 1 SEM across participants, interpolated across the 23 discrete data points in the trial-averaged data. Statistical comparisons of evidence values focused on within-subject differences. For every 2-sec interval throughout the trial, color-coded circles along the top of each graph indicate that the classifier’s evidence for the cued or other categories, respectively, was reliably stronger (p<0.002, based on repeated measures t-tests) than the evidence for the trial-irrelevant category (irrel).

Mean classifier evidence values are shown separately for the retrospective strategy group (left column: Retrospective) and the prospective strategy group (right column: Prospective), and separately for the distraction-absent trials (top row: Absent) and the distraction-present trials (bottom row: Present). Evidence values for the face, place, and object categories were relabeled and collapsed across all trials into three new categories: target (red, the category of the target stimulus on a given trial), assoc (blue, the category of the target’s associate stimulus), and irrel (grey, the trial-irrelevant category). Data for each category are shown as ribbons whose thickness indicate +/− 1 SEM across participants, interpolated across the ten discrete data points in the trial-averaged data. The colored bars along the horizontal axis indicate the onset of the target (red, 0 s), the distractors (grey, 6 s; distraction-present trials only), and the probe (blue, 12 s). Statistical comparisons of evidence values for the three categories focused on within-subject differences. For every 2-sec interval throughout the trial, color-coded circles at the top of each graph indicate the category whose evidence was greater (p<0.05, based on repeated measures t-tests) than the average evidence for the other two categories. Unlike the data from Phase 1 that was used to train the classifier, these data were not shifted in time, and therefore the peak response to a trial event appears approximately 4 to 6 s after the onset of the event.