In solar energy harvesting devices based on molecular semiconductors, such as organic photovoltaics (OPVs) and artificial photosynthetic systems, Frenkel excitons must be dissociated via charge transfer at heterojunctions to yield free charges. What controls the rate and efficiency of charge transfer and charge separation is an important question, as it determines the overall power conversion efficiency (PCE) of these systems. In bulk heterojunctions between polymer donor and fullerene acceptors, which provide a model system to understand the fundamental dynamics of electron transfer in molecular systems, it has been established that the first step of photoinduced electron transfer can be fast, of order 100 fs. But here we report the first study which correlates differences in the electron transfer rate with electronic structure and morphology, achieved with sub-20 fs time resolution pump-probe spectroscopy. We vary both the fullerene substitution and donor/fullerene ratio which allow us to control both aggregate size and the energetic driving force for charge transfer. We observe a range of electron transfer times from polymer to fullerene, from 240 fs to as short as 37 fs. Using ultrafast electro-optical pump-push-photocurrent spectroscopy, we find the yield of free versus bound charges to be weakly dependent on the energetic driving force, but to be very strongly dependent on fullerene aggregate size and packing. Our results point toward the importance of state accessibility and charge delocalization and suggest that energetic offsets between donor and acceptor levels are not an important criterion for efficient charge generation. This provides design rules for next-generation materials to minimize losses related to driving energy and boost PCE.