The retinal protonated Schiff-base (RPSB) in its all-trans form is found in bacterial rhodopsins, whereas visual rhodopsin proteins host 11-cis RPSB. In both cases, photoexcitation initiates fast isomerization of the retinal chromophore, leading to proton transport, storage of chemical energy or signaling. It is an unsolved problem, to which degree this is due to protein interactions or intrinsic RPSB quantum properties. Here, we report on time-resolved action-spectroscopy studies, which show, that upon photoexcitation, cis isomers of RPSB have an almost barrierless fast 400 fs decay, whereas all-trans isomers exhibit a barrier-controlled slow 3 ps decay. Moreover, formation of the 11-cis isomer is greatly favored for all-trans RPSB when isolated. The very fast photoresponse of visual photoreceptors is thus directly related to intrinsic retinal properties, whereas bacterial rhodopsins tune the excited state potential-energy surface to lower the barrier for particular double-bond isomerization, thus changing both the timescale and specificity of the photoisomerization.