In this contribution, the optical losses and gains attributed to periodic nanohole array electrodes in polymer solar cells are systematically studied. For this, thin gold nanomeshes with hexagonally ordered holes and periodicities (P) ranging from 202 nm to 2560 nm are prepared by colloidal lithography. In combination with two different active layer materials (P3HT:PC61BM and PTB7:PC71BM), the optical properties are correlated with the power conversion efficiency (PCE) of the solar cells. A cavity mode is identified at the absorption edge of the active layer material. The resonance wavelength of this cavity mode is hardly defined by the nanomesh periodicity but rather by the absorption of the photoactive layer. This constitutes a fundamental dilemma when using nanomeshes as ITO replacement. The highest plasmonic enhancement requires small periodicities. This is accompanied by an overall low transmittance and high parasitic absorption losses. Consequently, larger periodicities with a less efficient cavity mode, yet lower absorptive losses were found to yield the highest PCE. Nevertheless, ITO-free solar cells reaching ~77% PCE compared to ITO reference devices are fabricated. Concomitantly, the benefits and drawbacks of this transparent nanomesh electrode are identified, which is of high relevance for future ITO replacement strategies.