There is an extensive literature documenting that people with schizophrenia have marked impairments in social role functioning and social skill. One of the most widely employed strategies for assessing social skill has been role-play tests: simulated social interactions that are videotaped for subsequent behavioral coding. There has been considerable discussion of the validity of the approach in the literature, but there has not been adequate consideration of other psychometric characteristics of role-play tests. This paper examines the psychometric characteristics of a representative role-play measure: the Maryland Assessment of Social Competence (MASC). Data from 5 large schizophrenia studies that included the MASC were examined: a study of victimization in women who abuse drugs, a study of health care among people with diabetes, a study of vocational outcomes, a study of social skill among drug abusers, and a clinical trial comparing two antipsychotic medications. Data were examined in terms of five questions: (1) Can role-play scenes be rated reliably? (2) How are role-play ratings distributed across populations? (3) How many and which behaviors should be rated? (4) How many role-play scenes are required? (5) Is role-play behavior temporally stable? Overall, the data suggest that the MASC, and by implication other similar role-play procedures, does have good psychometric properties. However, several things often taken for granted in the literature warrant careful consideration in the design of research using role-play. Implications of the results for design of research are discussed.