Intense current research is directed at the evaluation of nanomaterials as catalysts for the oxygen reduction reaction. This is commonly undertaken by means of voltammetric measurements supported on an electrode surface presumed inert other than for providing electrical contact. At their basis these factors usually involve measurement of a current or current density, measured at a fixed potential. However we now report that the current/current density at a fixed potential can vary with the surface coverage of the nanoparticles in the catalyst, without any change in fundamental kinetic or thermodynamic parameters, even though the voltammetric signal shows that the reduction is fully transport controlled. This finding leads us to the conclusion that caution should be expressed when comparing catalysts in this way. In particular the essential need is emphasised for characterising the coverage, porosity and particle size, when inferring inherent electrochemical activity and using a suitable physical model to extract catalytic parameters.