Laser-plasma particle accelerators could provide more compact sources of high-energy radiation than conventional accelerators. Moreover, because they deliver radiation in femtosecond pulses, they could improve the time resolution of X-ray absorption techniques. Here we show that we can measure and control the polarization of ultra-short, broad-band keV photon pulses emitted from a laser-plasma-based betatron source. The electron trajectories and hence the polarization of the emitted X-rays are experimentally controlled by the pulse-front tilt of the driving laser pulses. Particle-in-cell simulations show that an asymmetric plasma wave can be driven by a tilted pulse front and a non-symmetric intensity distribution of the focal spot. Both lead to a notable off-axis electron injection followed by collective electron-betatron oscillations. We expect that our method for an all-optical steering is not only useful for plasma-based X-ray sources but also has significance for future laser-based particle accelerators.