We show that a thermally isolated system driven across a quantum phase transition by a noisy control field exhibits anti-Kibble-Zurek behavior, whereby slower driving results in higher excitations. We characterize the density of excitations as a function of the ramping rate and the noise strength. The optimal driving time to minimize excitations is shown to scale as a universal power law of the noise strength. Our findings reveal the limitations of adiabatic protocols such as quantum annealing and demonstrate the universality of the optimal ramping rate.