We evaluate the long-term test-retest reliability and procedural validity of phobia diagnoses in the UM-CIDI, the version of the Composite International Diagnostic Interview, used in the US National Co-morbidity Survey (NCS) and a number of other ongoing large-scale epidemiological surveys. Test-retest reliabilities of lifetime diagnoses of simple phobia, social phobia, and agoraphobia over a period between 16 and 34 months were K = 0.46, 0.47, and 0.63, respectively. Concordances with the Structured Clinical Interview for DSM-III-R (SCID) were K = 0.45, 0.62, and 0.63, respectively. Diagnostic discrepancies with the SCID were due to the UM-CIDI under-diagnosing. Post hoc analysis demonstrated that modification of UM-CIDI coding rules could dramatically improve cross-sectional procedural validity for both simple phobia (K = 0.57) and social phobia (K = 0.95). Based on these results, it seems likely that future modification of CIDI questions and coding rules could lead to substantial improvements in diagnostic validity.