We present a single molecule fluorescence study that allows one to probe the nanoscale segmental dynamics in amorphous polymer matrices. By recording single molecular lifetime trajectories of embedded fluorophores, peculiar excursions towards longer lifetimes are observed. The asymmetric response is shown to reflect variations in the photonic mode density as a result of the local density fluctuations of the surrounding polymer. We determine the number of polymer segments involved in a local segmental rearrangement volume around the probe. A common decrease of the number of segments with temperature is found for both investigated polymers, poly(styrene) and poly(isobutylmethacrylate). Our novel approach will prove powerful for the understanding of the nanoscale rearrangements in functional polymers.