We present an implementation of analytical quantum mechanical molecular gradients within the polarizable embedding (PE) model to allow for efficient geometry optimizations and vibrational analysis of molecules embedded in large, geometrically frozen environments. We consider a variational ansatz for the quantum region, covering (multiconfigurational) self-consistent-field and Kohn-Sham density functional theory. As the first application of the implementation, we consider the internal vibrational Stark effect of the C=O group of acetophenone in different solvents and derive its vibrational linear Stark tuning rate using harmonic frequencies calculated from analytical gradients and computed local electric fields. Comparisons to PE calculations employing an enlarged quantum region as well as to a non-polarizable embedding scheme show that the inclusion of mutual polarization between acetophenone and water is essential in order to capture the structural modifications and the associated frequency shifts observed in water. For more apolar solvents, a proper description of dispersion and exchange-repulsion becomes increasingly important, and the quality of the optimized structures relies to a larger extent on the quality of the Lennard-Jones parameters.