Organic materials can tune the optical properties in layered (2D) hybrid perovskites, although their impact on photophysics is often overlooked. Here, we use transient absorption spectroscopy to probe the Dion-Jacobson (DJ) and Ruddlesden-Popper (RP) 2D perovskite phases. We show the formation of charge transfer excitons in DJ phases, resulting in a photoinduced Stark effect which is shown to be dependent on the spacer size. By using electroabsorption spectroscopy, we quantify the strength of the photoinduced electric field, while temperature-dependent measurements demonstrate new features in the transient spectra of RP phases at low temperatures resulting from the quantum-confined Stark effect. This study reveals the impact of spacer size and perovskite phase configuration on charge transfer excitons in 2D perovskites of interest to their advanced material design.