Lateral diffusion of neurotransmitter receptors in and out of synapses has been postulated as a core mechanism for rapid changes in receptor number at synapses during plastic processes. In this study, we have used single particle tracking to investigate how changes in glycine receptor (GlyR) lateral diffusion properties might account for changes in receptor number at synapses after disruption of the cytoskeleton in dissociated spinal cord neurons. We found that pharmacological disruption of F-actin and microtubules decreased the amount of GlyR and gephyrin, the backbone of the inhibitory postsynaptic scaffold, at synapses. F-actin and microtubule disruption increased GlyR exchanges between the synaptic and extrasynaptic membranes and decreased receptor dwell time at synapses. GlyR lateral diffusion was predominantly controlled by microtubules in the extrasynaptic membrane and by actin at synapses. Both diffusion coefficients and confinement at synapses were affected after F-actin disruption. Our results indicate that receptor exchanges between the synaptic and extrasynaptic compartments depend on the properties of both the postsynaptic differentiation and the extrasynaptic membrane. Consequently, GlyR number at synapses may be rapidly modulated by the cytoskeleton through the regulation of lateral diffusion in the plasma membrane and of receptor stabilization at synapses.