The evaluation of claims that a new diagnostic test is better than the current gold standard test is hindered by the lack of a perfect reference judge. However, this problem may be sidestepped by focusing on the clinical consequences of the decision rather than on estimation of accuracy. Consequences can be assessed by use of a "fair umpire" test that is not perfect yet can discriminate between disease and nondisease cases and is not biased in favor of 1 test. This article discusses 3 principles to aid judgments about the value of new tests. First, the consequences are best examined in cases with disagreement between the current and new tests. Second, resolving these disagreements requires a fair, but not necessarily perfect, umpire test. Finally, umpire tests include consequences, such as prognosis and response to treatment, as well as causal exposures and other test results.