PMC

Data management flowchart. In stage 2, system validation was performed by sensitivity and specificity testing, receiver operating characteristic curve, or observer variation and can be identified at title and abstract level with MeSH search. We identified 27 articles from 346 full-text articles and 283 articles at the title and abstract level.

A stratified view of health information technology evaluation. Level 1 targets system specification to understand user-task interaction for system development. Level 2 examines task performance to assess system validation and human–computer interaction. Level 3 aims to incorporate environmental factors to identify work processes and system impacts in real settings. Task/expectation complexity, user variances, and organizational support are factors that influence the use of the system, but are not problems of the system itself, and need to be differentiated from system-related issues.