The call for accountability in health care quality has fueled the development of consumer-oriented Web sites that provide hospital ratings. Taking the consumer perspective, we compared five Web sites to assess the level of agreement in their rankings of local hospitals for four diagnoses. The sites assessed different measures of structure, process, and outcomes and did not use consistent patient definitions or reporting periods. Consequently, they failed to agree on hospital rankings within any diagnosis, even when using the same metric (such as mortality). In their current state, rating services appear likely to confuse, rather than inform, consumers.