We present a robust and straightforward method for fabricating remarkably responsive hydrogel scaffolds consisting of submicron-sized microgel particles. We demonstrate that the microgel particles assemble either through bridging or depletion interactions to yield a structure that swells or deswells at a macroscopic level in much shorter times as compared to a bulk polymer gel of similar characteristics. This approach offers a new way of fabricating functional hydrogel scaffolds with controllable responsiveness to applied stimuli and excellent loading capability for a wide variety of materials, irrespective of chemistry, size, and shape.