We propose a low-power all-optical switch based on the structure of a graphene-buried balanced Mach-Zehnder interferometer (MZI), where the signal light is switched between the two output ports of the MZI by the heat generated from graphene's absorption of the pump light. We use orthogonal polarizations for the pump and the signal light to maximize pump absorption and minimize graphene-induced signal loss. Our experimental device fabricated with polymer waveguides buried with 5-mm long graphene shows a pump absorption of 10.6 dB (at 980 nm) and a graphene-induced signal loss of 1.1 dB (at 1550 nm) and can switch the signal light with a pump power of 6.0 mW at an extinction ratio of 36 dB. The actual pump power absorbed by graphene for activating switching is estimated to be 2.2 mW. The rise and fall times of the switch are 1.0 and 2.7 ms, respectively. The switching characteristics are weakly sensitive to ambient temperature variations. Our device can be butt-coupled to single-mode fibers and could find applications in fiber-based and on-chip all-optical signal processing.