|
|
Stud Health Technol Inform. Author manuscript; available in PMC 2009 September 2. Published in final edited form as: | PMCID: PMC2736630 NIHMSID: NIHMS138085 |
Evaluating the Effectiveness of Modeling Principles for Data Models Miguel Humberto Torres-Urquidy,ab1 Amit Acharya,c Pedro Hernandez-Cott,d Jonathan Misner,e and Titus Schleyerab aDepartment of Biomedical Informatics, University of Pittsburgh bCenter for Dental Informatics, University of Pittsburgh cDepartment of Healthcare Informatics, University of Medicine and Dentistry of New Jersey dUniversity of Puerto Rico School of Dental Medicine eSchool of Dental Medicine, University of Pittsburgh We evaluated the effectiveness of modeling principles intended to harmonize the information representation between terminology-ontology models and information models. Our study utilized dental clinical statements and sample dental record questions. We asked experts to define the equivalency (mapping) of these elements and measured their agreement. We modified the data elements and asked the experts to conduct subsequent mappings. We measured the agreement and compared the levels of agreement before and after changes, expecting that agreement would increase. The level of agreement (Kappa) before modeling was 0.3 – 0.4 and after was 0.5 (p<0.05). The difference was small but statistically significant. Our results suggest that the modeling principles improve information representation since agreement increased. Keywords: Information modeling, evaluation methods, data models, terminology, ontology, dentistry, dental informatics Improvements in clinical documentation and communications can lead to the reduction of medical errors [ 1]. Presently, electronic documentation of patient records builds on data models designed to convey information. Rector [ 2] and Tu et al [ 3] characterize three types of data models currently supporting the representation and communication of information in healthcare: 1) The Terminology-Ontology model “represents the relations between the types of information (‘concepts’), e.g. SNOMED-RT’s compositional representation [ 4], GALEN’s Common Reference Model [ 5, 6]” or SNODENT [ 7]; 2) the Information-Data model which maintains “the structure of how instances of different types of information may be related to each other in a data repository or message,” e.g. Health Level 7 Reference Information Model [ 8] [ 9], ISO/CEN Electronic Health Record Architecture (ECHRA) [ 10];3) the Inference-Problem Solving model “describes what should be done in highly defined situations taking into account a range of information”, e.g. Medical Logical Modules (MLMs) [ 11] Traditionally, these models have been created by different groups, resulting in discrepancies that can lead to misrepresentation and misinterpretation of information. Additionally, current adoption of such models makes their modification and proper integration more difficult. Consequently, this creates potential for errors due to ambiguity or duplication of information [ 2] Several modeling principles have been proposed to reduce these discrepancies [ 2, 12, 13]. These principles provide guidance by defining how information should be represented and in which model. In other words, these principles assist in defining the models’ mutual constraints and obligations in order to keep communication unambiguous [ 2]. E.g. “each type of information should be represented uniquely in only one place in only one way.” Regrettably, very few studies exist that have measured the effectiveness of the proposed principles. A study by Qamar et al [ 14] showed that by modifying the information model, the data mapping accuracy (of open EHR data archetypes to terminology concepts) improved from 64.7% to 80.5%. The result from this study supports the idea that modeling improves the way information is represented. However, it was unfeasible for the authors to measure the changes in accuracy if changes had also been applied to the terminology model. This reason prompted our current study. We hypothesize that the application of Rector’s modeling principles to both, the information model and terminology model, improves the representation of information. Dentistry, as other branches of healthcare, is an information-intensive domain. However, when compared to medicine in general, the need for information exchange is different. This has resulted in limited levels of clinical computerization [ 15]. Consequently, terminology and information models have not been much in demand. Although this situation could be considered a disadvantage (lacking years of sophistication), it provides the opportunity to learn from previous experiences in medical informatics. The Center for Dental Informatics at the University of Pittsburgh is currently pursuing two projects to develop clinical data models for general dentistry. The first project is focused on the development of a terminology and ontology of dental diagnostic concepts. The second addresses the creation of an information model for the electronic dental patient record. These two projects are in their early development stage and require the utilization of raw data from clinical sources. This early status provides an excellent opportunity to evaluate the effectiveness of the modeling principles [ 2] when applied to both models. In order to reduce discrepancies, we believe that our models (terminology-ontology and information model) should be developed in parallel. But before choosing definitive development strategies, we decided to determine how appropriate the modeling principles [ 2] would be to our specific content area. In our study, we evaluated the effectiveness of the modeling principles proposed by Rector et al. [ 2] by using simple raw clinical statements that contained diagnostic information and sample dental record questions (items). We called both of these “primitives.” We asked experts to conduct mappings (or defining their semantic equivalency) using these primitives and measured their agreement. Then we modified the primitives using Rector’s modeling principles [ 2] and asked the experts to replicate the mapping process. After this, we measured the agreement and compared the levels of agreement before and after the modifications, expecting that agreement would increase. We proposed that the change in agreement provides evidence of the effectiveness of the modeling principles. Formally, our hypothesis is that the level of agreement between clinicians increases as a result of an improved model harmonization caused by the application of the modeling principles. 1. Modeling principles Alan Rector [ 2] established rules or principles that assist in the joint development of the terminology-ontology model and the information model. With these principles, Rector delineates how the information should be allocated and establishes the constraints and obligations for each model. 1.1. In this study we evaluated the following rules that we chose to apply only to the terminology ontology model:
a) Separation between kernel and status concepts. The terminology-ontology should explicitly separate kernel concepts (a type of “atomic concept”) vs. status concepts (modifiers or qualifiers). For our experiment the modeling included parsing the statements and stating whether the resulting terms included kernel or status concepts.b) Reduction of ambiguity by identifying semantic types. The semantic type guides the interpretation of a concept-term by providing a better context. For the modeling, we included the semantic type of the terms that were created in the parsing process (previous paragraph).
1.2. Concurrently, we evaluated the application of the modeling principles applicable to the information model. These focused on two aspects: Ambiguity and Structure. We chose and evaluated the following principles:
c) Separation of the multiple pieces of information embedded in a single information item. E.g. “Cheek biting/lip biting” was separated into “Cheek biting” and “Lip biting”.d) Elimination of the generic information items. E.g. “Are you aware of any problem?”, “Comments”.e) Canonically renaming ambiguously named information items to enhance the intended purpose of the item. E.g. “Chewing” was renamed to “Difficulty in chewing”.f) Canonically grouping the synonymous information items together to prevent duplication of information. E.g. “Do you get nervous before dental treatment?” and “Are you usually nervous during dental visit?”g) Organization of the information items into meaningful categories (and sub-categories). E.g. category “Previous dental care information “, subcategory “Surgical.”
By applying these principles we expect that experts will be able to better recognize when two data elements are equivalent. These include elements from terminology-ontology project and elements from the information model project. 2. Data elements We asked experts to evaluate the equivalence of elements from the following two sources: 2.1. Terminology-Ontology Model Project The terminology-ontology model project has the goal of representing the diagnoses and findings used in general dental care. The original data consists of 5300 raw statements each ranging from a single word to a full sentence of clinical information. A dentist and a medical librarian, both knowledgeable in medical terminologies, extracted the statements from 80 patient records obtained from two dental schools. The statements will serve eventually as a source for developing a reference ontology-terminology. All of the statements are of diagnostic interest. Examples of statements include “broken tooth” or “mesio-distal caries on tooth number 14.” From the 5300 statements, we pre-selected 752 that would contain information pertaining to a patient’s dental history. From these, we randomly selected 150 for the first part of the experiment (training set) and 4 sub-sets of 40 for the second part (test set). We called these primitive dental statements. 2.2. Information Model Project The information model project has the goal of developing a validated and refined list of data elements/information items that will support the documentation (capture/store) and retrieval process of patients’ health information in the Electronic Dental Record (EDR). Examples include “ are any teeth loose? yes/no”, “ dental anxiety: yes/no” or “ frequency of brushing:”. The list of information items was developed using a bottom-up approach by extracting information items from a sample of 10 dental paper record formats (four from practicing dentists, two from dental schools, and four from commercial vendors) [ipsilon 16] and 10 documented patient charts from School of Dental Medicine, University of Pittsburgh. The list consisted of 70 information items pertaining to the “dental history” part of the overall model. We called these primitive information model items. The raw data for both projects was collected independently. 3. Defining Equivalency (Mapping) before Modeling The first part of the experiment consisted of defining the equivalency of primitive dental statements with the primitive information model items by two experts. We employed a dentist (PHC) and a senior dental student (JM) to conduct the mappings. The experts received a training set of 150 primitive dental statements and 70 primitive information model items. The experts were instructed to read the first primitive dental statement and then decide whether any of the 70 primitive information model items would be equivalent (could accommodate the meaning). If any of the items was fully equivalent in meaning, we asked them to mark the item(s) as providing a “Complete” mapping. When the primitive information model item could accommodate only part of the meaning expressed by the primitive dental statement, we instructed the experts to mark the mapping as “Partial.” For example, the primitive dental statement “caries on tooth #8”, when mapped to the primitive information model item “presence of caries: Yes/No” would be considered partial since the information model item cannot capture the anatomical information (“#8”). For partial mappings, we asked reviewers to provide reasons for their rating. If the primitive dental statement could not be accommodated by any of the primitive information model items, we asked reviewers to indicate that there was no mapping (no equivalence). 4. Defining Equivalency (Mapping) after modeling For the second part of the experiment we asked our clinicians to repeat the previous step (define equivalency of data elements), but on this occasion, they received modified data elements using the principles described in Section 1. The modified data elements were presented in four sets of data. We presented them in sets in order to measure the effect that the different modeling principles would have depending the changes occurred in the terminology-ontology model, the information model or in both. Additionally, we presented a fourth set with items similar to the first part of the experiment to control for learning. Thus, the four sets were: A. Primitive dental statements – modified information model items set We asked the experts to map 40 primitive dental statements to 85 modified information model items. We expected that the expert’s agreement when mapping this set would be better than the training and the control set but less than the set that contains modifications to both the dental statements and information model items. B. Modified dental statements – primitive information items set We asked the experts to map 40 modified dental statements to 70 primitive information model items. We expected that the expert’s agreement when mapping this set would be better than the training and the control set but less than the set that contains modifications to both the dental statements and information model items. C. Modified dental statements – modified information items set This set integrated 40 modified dental statements and 85 modified information model items. We expected that this set would generate the highest level of agreement between experts since it includes modifications to the dental statements and information model items. Control set This set was identical to the training set but with fewer primitive dental statements (only 40 instead of the original 150). The number of primitive information model items was 70. We expected that the experts’ agreement when defining the equivalency (mapping) of this set would be similar to the agreement found in the training set. The number of information model items increased from 70 to 85 as result of applying the modeling principles. The categorization of the items changed from one all-inclusive Dental History category to six major categories and 18 subcategories. Data types which suggested the structure of information (binary, text, etc.) were dropped because the experts’ made clear that the “implementation level” of the information items clearly limited their ability to represent information. The modification of dental statements was done independently from information model items by having two of the investigators working separately (MHTU, AA respectively). Doing otherwise would generate the risk of creating a confounding factor in our evaluation. 5. Statistical analysis We used Cohen’s Kappa [ 17] statistic to measure inter-rater agreement because it corrects for chance. We determined the agreement of the experts’ mapping for the five different sets (1 set during the training part and 4 sets during the second). Our null hypothesis was
where K training is the level of agreement obtained by clinical experts when determining the equivalency (mapping) of the training set (no modifications); K control is the experts’ level of agreement when mapping the control set; K A is the experts’ level of agreement when mapping the primitive dental terms to the modified information model items; K B is the experts’ level of agreement when mapping the modified dental terms to the primitive information model items and; K C is the experts’ level of agreement when mapping the modified dental statements to the modified information model items. For testing the hypothesis we conducted two-sided pair-wise comparison using the confidence intervals obtained from the levels of agreement. We defined the level of statistical significance at p<0.05. The statistical analysis was conducted using Microsoft Excel (Redmond, WA) and SAS (Cary, NC). Table 1 shows the levels of agreement as well as the confidence intervals for the mapping process. The level of unweighted kappa for the training set was 0.3302 (.95 confidence interval: lower limit 0.3025, upper limit 0.3579). The level obtained while using the control set was 0.4388 (.95 confidence interval: lower limit 0.4084, upper limit 0.4691). The difference between these two kappa values showed that there was some external factor that caused agreement to increase. | Table 1The table includes the levels of agreement and confidence intervals. The level of agreement reached when using set C was the highest and its difference was statistically significant. |
The level of agreement for data set A (primitive dental terms and modeled information items) was 0.4358 (.95 confidence interval: lower limit 0.4089, upper limit 0.4626). The level of agreement for data set B (modeled dental terms mapped to primitive information items) was 0.4402 (.95 confidence interval: lower limit 0.4102, upper limit 0.4701). When comparing these two datasets to the training set, it was possible to perceive a statistically significant difference, showing an increment in agreement when modeling was used for at least one of the models. On the other hand, the agreement from data sets A (KA = 0.4358) and B (KB = 0.4402) when compared to the control set (Kcontrol = 0.4388) show that there was no statistically significant difference between them. As mentioned above, the control set was similar to the training set in the sense that no modeling was used (only included primitives). This indicates the possible presence of an external confounding factor that increased agreement artificially. The level of agreement for the final data set which included modeled dental terms and modeled information model items was 0.5041 (.95 confidence interval: lower limit 0.4776, upper limit 0.5306). When comparing to the agreement of the other four data sets, we can appreciate a small but statistically significant difference, showing that there was an agreement increment. The existence of different data models creates potential for erroneously expressing information. In order to reduce this potential for errors, it is possible to establish constraints and obligations between data models. Explicitly, these constraints and obligations can be seen as modeling principles whose objective is to determine the most proper way to allocate information. Our study evaluated the effect of certain modeling principles by determining the change in clinician’s agreement before and after their application. The results suggest that when applying modeling principles, agreement increases. In other words, the clinicians found more information equivalencies after applying the modeling principles. Specifically, the level of agreement for the set where the modeling principles were applied to both, dental statements (terminology – ontology model) and information items (information model), agreement increased. On the other hand, our study had several limitations. First, because of the study design we were able to detect a confounding factor since there was an increment in agreement between the training and control sets. This is possibly the consequence of a “Carryover Effect” [ 18]. Thus, our results should be interpreted cautiously. The second limitation could be that our study only used two experts. Other studies should have more experts participating in the mapping process. Additionally, this study addressed a limited conceptual area (dental diagnostic concepts relevant to dental history). It is possible that in other conceptual areas the modeling principles perform differently. Finally, although the change in kappas was significant, the reached level suggests only moderate agreement [ 19]. This could be caused by the difference in levels of experience of our participant clinicians, one being a clinician with more than 20 years of experience and the second a final year dental student. These levels of experience possibly create clinical interpretation differences that reduce agreement and hide the true effect of the modeling principles. Another element to consider is that possibly, the modeling changes helped one expert more than the other. Further analysis should explore this possibility. Conversely, the highest level of agreement was reached when applying the modeling principles to the dental statements and information items. This difference was statistically significant suggesting that the early (during the development stage) and parallel (to both models) application of the principles facilitates the creation of enhanced data models. In conclusion, we found that agreement between experts increased after applying modeling principles proposed in the literature. This provides favorable evidence suggesting that applying such principles alone improves the representation of information. The authors would like to thank Dr. James Bost for his support with the statistical analysis, Drs. W. Chapman, H. Harkema, H. Johnson and our anonymous reviewers for their comments. Study supported by the NIH grant R21DE01548-01 and the American Dental Association. 1. Bates DW, Gawande AA. Improving safety with information technology. N Engl J Med. 2003 Jun 19;348(25):2526–2534. [PubMed] 2. Rector AL. The interface between information, terminology, and inference models. Medinfo. 2001;10(Pt 1):246–250. 3. Tu SW, Musen MA. Modeling data and knowledge in the EON guideline architecture. Medinfo. 2001;10(Pt 1):280–284. 4. Spackman KA, Campbell KE, Côté RA. SNOMED RT: a reference terminology for health care. Proc AMIA Annu Fall Symp. 1997:640–644. [PubMed] 6. Rector AL. Thesauri and formal classifications: terminologies for people and machines. Methods Inf Med. 1998 Nov;37(4–5):501–509. [PubMed] 7. Goldberg LJ, Ceusters W, Eisner J, Smith B. The Significance of SNODENT. Stud Health Technol Inform. 2005;116:737–742. [PubMed] 9. ASTM Committee E31. E1384-02a Standard practice for content and structure of the electronic health record (EHR). West Conshohocken, PA: ASTM International; 2006. 10. CEN/WG1. ENV13606: Electronic Healthcare Record Architecture, CEN; 1999. 11. Pryor TA, Hripcsak G. The Arden syntax for medical logic modules. Int J Clin Monit Comput. 1993 Nov;10(4):215–224. [PubMed] 12. Rector AL, Johnson PD, Tu S, Wroe C, Rogers J. Artificial Intelligence in Medicine. Berlin: Springer; 2001. Interface of inference models with concept and medical record models; pp. 314–323. 13. Rector AL, Rogers J, Taweel A. Models and inference methods for clinical systems: a principled approach. Medinfo. 2004;11(Pt 1):79–83. 14. Qamar R, Kola J, Rector AL. Unambiguous data modeling to ensure higher accuracy term binding to clinical terminologies. AMIA Annu Symp Proc. 2007 Oct;11:608–613. [PubMed] 15. gamma Schleyer TK, Thyvalikakath TP, Spallek H, Torres-Urquidy MH, Hernandez P, Yuhaniak J. Clinical computing in general dentistry. J Am Med Inform Assoc. 2006 May;13(3):344–352. [PubMed] 16. Schleyer T, Spallek H, Hernández P. A qualitative investigation of the content of dental paper-based and computer-based patient record formats. J Am Med Inform Assoc. 2007 Jul–Aug;14(4):515–526. [PubMed] 17. Hripcsak G, Heitjan DF. Measuring agreement in medical informatics reliability studies. J Biomed Inform. 2002 Apr;35(2):99–110. Review. [PubMed] 18. Friedman CP, Wyatt JC. Evaluation methods in medical informatics. New York (NY): Springer; 1997. Design, Conduct and Analysis of Demonstration Studies; pp. 155–203. 19. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174. [PubMed]
|
The publisher's final edited version of this article is available at 