Competing Interpretations of Disorder Codes in SNOMED CT and ICD

Stefan Schulz, MD, Alan Rector, MD, PhD, [...], and Kent Spackman, MD PhD

Additional article information

Abstract

Under ontological scrutiny we have identified two competing interpretations of disorder concepts in SNOMED. Should codes be interpreted as representing pathological conditions themselves or the situations in which a patient has those conditions? This difference has significant implications for the proposed harmonization between SNOMED CT and the new ICD-11 disease classification and indeed for any systematic review of the correctness of the SNOMED CT hierarchies. Conditions themselves are distinct, whereas in any given situation a patient may have more than one condition. In such cases, SNOMED codes that represent combinations of conditions - which can be regarded as “additive” - are evidence for interpreting the codes as referring to situations. There are clearly some such codes. We conducted a survey to determine the extent of this phenomenon. Three criteria were used – analysis of the SNOMED CT fully specified name, the corresponding logical definition, and the children of the concept under scrutiny. All three showed that at least 11% of concepts met our criteria for representing situations rather than conditions, with a satisfactory inter-rater reliability for the first two. We, therefore, conclude that if a uniform interpretation of SNOMED disorder codes is desired, they should be interpreted as representing situations.

Background

Since 2007, the World Health Organization has been working on the revision of the International Classification of Diseases (ICD-11)1 Compared to past revisions the ICD 11 process is characterized by the following differences:

  1. The authoring is done electronically, supported by ontology-driven tools2.
  2. It distinguishes between a multihierarchical ICD foundation component (FC) and target specific monohierarchical linearization products, as excerpts from the FC.
  3. The FC is intended to share a common ontology with SNOMED CT3 as an international clinical terminology standard, following an institutional agreement between WHO and IHTSDO4, put into practice by a joint WHO - IHTSDO advisory group (JAG).

This paper focuses on the third item. The agreement on a common model of meaning requires clear consensus criteria with regard to the linking of representational units (classes, concepts). Both SNOMED CT and ICD 11 are subscribing to principles of Applied Ontology5, i.e. the meaning of domain terms (and the concepts or classes they refer to) are described by logics, rooted in an ontological framework. In other words, they require a clear explanation which entities in the clinical world can be truly denoted by a given representational unit.

Backbones of ontology artifacts are taxonomic orders of classes. Different from typical thesauri like MeSH or UMLS the justification of a hierarchical link between A and B is not whether B has a broader meaning in natural language than A. Instead, in ontologies the question is posed as follows: “are all members of the class A equally members of the class B ?”, analogously to the subset relation in set theory. Only if this is the case, A qualifies as a taxonomic descendant of B. In other words, a subclass relation holds between A and B. For any alignment of two hierarchically structured vocabularies it is therefore crucial that both of them subscribe to a shared understanding of what makes up a taxonomic link. To avoid contradictions, if, as a result of alignment, A’ is considered equivalent to A, and B’ is considered equivalent to B, then if A is a subclass of B, the subclass relation between A’ and B’ must be in line with the axioms in the target system.

Consequently the JAG has formulated as goals for the SNOMED CT – ICD11 project that (i) each class in the ICD-11 foundation component has to correspond to exactly one class in SNOMED CT, and (ii) the transitive closure of taxonomic relations in FC must be included in the transitive closure of taxonomic relations in SNOMED CT.

Situation vs. Condition

In preparation for this task, the JAG has intensively analyzed and discussed the fundamental principles and the ontological basis of both ICD and SNOMED, and has identified the following two diverging interpretations of disorder terms:

  1. Disorder terms denote patient-borne Conditions such as body processes, states, dispositions, or (patho-) anatomical structures, which are reportable in the context of medical records6, such as my myopia, John’s pulmonic stenosis, and Mary’s retinopathy. The contrasting view is that
  2. Disorder terms denote Clinical Situations, which are defined as phases of a patient’s life, during which he/she is bearer of (some combination of) pathological conditions. They are, in principle, independent of whether the patient participates in an episode of medical care or observation. Situations can be short (e.g. a seizure) or lifelong (e.g. a congenital malformation).

Let us take, as a simple example, the SNOMED CT concepts Fracture of radius and Fracture of ulna.

Fracture of radius and fracture of ulna are two distinct pathological conditions. In a hierarchy of conditions they would be represented as disjoint classes, as the class of radius fractures does not contain members that are ulna fractures and vice versa. Such conditions can occur separately or in conjunction. That is, we can distinguish three different clinical situations: First, a situation of a patient with a broken ulna alone; second, a situation of a patient with a broken radius alone; and third, a situation of a patient with both a broken ulna and radius.

Let us now investigate how these entities are represented in SNOMED CT, which is increasingly compliant with the Semantic Web language OWL7, which draws on Description Logics8. We here use OWL Manchester syntax9 and translate SNOMED CT relational expressions into description logics according to Spackman at al.10

  • ‘Fracture of radius AND ulna (disorder)’ equivalentTo        (1)
    • ‘Fracture of radius (disorder)’ and ‘Fracture of ulna (disorder)’ and
    • Group some (‘Associated morphology’ some ‘Fracture (morphologic abnormality)’ and
      • ‘Finding site’ some ‘Bone structure of radius (body structure)’) and
    • Group some (‘Associated morphology’ some ‘Fracture (morphologic abnormality)’ and
      • ‘Finding site’ some ‘Bone structure of ulna (body structure)’)
  • ‘Fracture of radius (disorder)’ equivalentTo        (2)
    • ‘Fracture of forearm (disorder)’ and ‘Injury of radius (disorder)’ and
    • Group some (‘Associated morphology’ some ‘Fracture (morphologic abnormality)’ and
      • ‘Finding site’ some ‘Bone structure of radius (body structure)’)
  • ‘Fracture of ulna (disorder)’ equivalentTo        (3)
    • ‘Fracture of forearm (disorder)’ and ‘Injury of ulna (disorder)’ and
      • Group some (‘Associated morphology’ some ‘Fracture (morphologic abnormality)’ and
        • ‘Finding site’ some ‘Bone structure of ulna (body structure)’)

We can derive from this that every member of the class of ‘Fracture of radius AND ulna (disorder)’ is both a member of the class ‘Fracture of radius (disorder)’ and of the class ‘Fracture of ulna (disorder)’

This excludes the interpretation that the latter two classes have only isolated fracture of radius or ulna as members, as it underlies, obviously the representation in ICD-10, assuming disjoint categories, viz. ICD10:S52.2 Fracture of shaft of ulna, ICD10:S52.3 Fracture of shaft of radius, and ICD10:S52.4 Fracture of shafts of both ulna and radius, which would correspond to the following disjointness axioms:

  • ‘Fracture of shaft of ulna’ and ‘Fracture of shaft of radius’ subClassOf Nothing      (4)
  • ‘Fracture of shaft of radius’ and ‘Fracture of shafts of both ulna and radius’ subClassOf Nothing  (5)
  • ‘Fracture of shafts of both ulna and radius’ and ‘Fracture of shaft of ulna’ subClassOf Nothing  (6)

If looking more closely at the SNOMED CT definitions (1 – 3) we notice that the pathological entity itself (namely the fracture morphology at some bone structure) appears enclosed in a Group. This group has the status of an OWL object property and can often be interpreted as a ‘has-processual-part’ relation.11

This means that from the perspective gained through ontological analysis, the SNOMED concept ‘Fracture of radius (disorder)’ is most likely to be interpreted as something that includes a condition which is a fracture of some bone structure of the radius. It might include other objects or processes, which then explains that the combined concept ‘Fracture of radius AND ulna (disorder)’ appears as subclass.

What is this something? It appears that it is most compatible with a clinical situation, according to the abovementioned condition / situation distinction if we interpret the concepts in (1 – 3) as referring to situations, because every situation with a combined fracture is also a situation with some ulna or radius fracture.

The conditions themselves are therefore represented by the two expressions

  • (‘Associated morphology’ some ‘Fracture (morphologic abnormality)’ and          (7)
  • ‘Finding site’ some ‘Bone structure of radius (body structure)’)

and

  • (‘Associated morphology’ some ‘Fracture (morphologic abnormality)’ and          (8)
  • ‘Finding site’ some ‘Bone structure of ulna (body structure)’)

For the further planning of the SNOMED CT – ICD11 harmonization project, we need to know whether the fracture example represents a general building principle on which we can rely in the SNOMED CT Disorder subhierarchy or whether it constitutes an exceptional case. To this end we analyzed a sample of SNOMED CT disorder concepts.

Methods

By randomization we obtained a first sample for a pilot study with 50 SNOMED CT concepts (S1) from the disorder hierarchy. The four authors, who are MDs and terminology experts, analyzed the concept names, definitions and their place in the hierarchy.

In a first run all four experts analyzed the whole sample S1 according to the following criteria:

  1. Does the fully specified name suggest a situation rather than a single pathological entity?
  2. Is there evidence for the situation interpretation by the combination of their taxonomic parents?
  3. Is there evidence for the situation interpretation by one or more taxonomic children?
  4. Does the formal definition suggest a situation interpretation?
  5. Number of direct parents
  6. Number of role groups
  7. Whether the concept is primitive or sufficiently (fully) defined

After this, diverging results were identified and reviewed in an online discussion. In this discussion we identified several reasons for considerable divergence between the experts. Different SNOMED CT versions and browsers were used, e.g. the web-based VETMED browser, the SNOMED CT workbench, CliniClue, SNOB, as well as a SNOMED OWL version in Protégé. The main difference between these tools was the way they exhibited role groups. For instance, the SNOMED CT workbench did not visualize single role groups. Only in Protégé / OWL were role groups belonging to the ancestors easily visualized. But it was not only the use of different tools that explained the divergences. For several concepts there was no consensus on their exact meaning. Additionally, the following observations were made:

  • The selection of an equal proportion of cases by hierarchy depth yielded too many uninteresting upper-level and leaf concepts.
  • The hierarchical level (minimal distance from root) should be blinded.
  • The analysis is time-consuming, and no rater should assess many more than about 100 concepts.

As a result, the selection process was modified, and the criteria for the second phase were altered as follows:

The following metrics were derived programmatically from the current SNOMED CT release:

  • Depth (minimal distance from root);
  • Number of role groups at the level of the concept to be examined;
  • Number of inherited role groups;
  • Number of children;

The stratification was done by depth according to Table 1, yielding 400 concepts (regardless of being primitive or fully defined):

Table 1.
Stratification of sample by distance from the root of the SNOMED CT hierarchy

Three criteria were subject to expert rating:

  • CRITERION 1: All the ancestors of a sample concept (SC) are inspected. In case there are ancestors or role groups at any hierarchical level, which are additive with regard to conditions (i.e. which constitute disjoint classes under a condition point of view), then the concept is rated “y”. Otherwise (no evidence for SITUATION / ADDITIVITY), the rating is “n”.

    EXAMPLES: Neoplasm of heart (disorder) has the parents Neoplasm of heart AND/OR pericardium (disorder) and Structural disorder of heart (disorder). Given that a neoplasm can perfectly be a Structural disorder of the heart, there is no evidence of a SITUATION interpretation. The inspection of the ancestors does not reveal any incompatibility either. Therefore, the rating here would be “n”.

    Combined gastric AND duodenal ulcer (disorder) has the two parents Duodenal ulcer disease (disorder) and Gastric ulcer (disorder). A duodenal ulcer cannot be equally a gastric ulcer. Therefore both parents are additive, which suggests a SITUATION interpretation. Therefore, the rating would be “y”, and no further ancestors need to be inspected. In some cases, parenthood relations may be debatable, e.g. Heart disease isA Disorder of Mediastinum. This is a different issue, not under evaluation here. We therefore should be tolerant here, e.g. treating “disease” and “disorder” as synonyms, and interpret “disorder of” as “disorder located at”.

  • CRITERION 2: All children (direct descendants) of a sample concept SC are reviewed. All occurrences in which a child concept of SC has another parent which does not compose but sums up to SC, are counted. Example: SC = Pulmonic valve stenosis. There is the child: Rheumatic pulmonary valve stenosis with insufficiency (disorder), which has also the parent Pulmonic valve regurgitation (disorder). This clearly sums up and does not compose (no stenosis is also a regurgitation). The same can be said about the child Combined valvular-subvalvular pulmonic stenosis (disorder). Finally we have the child concept Tetralogy of Fallot (disorder), as discussed in Schulz et.al.12 We have therefore three children that have a second parent that is additive to the sample concept Pulmonic valve stenosis, therefore the value assigned is “3”.
  • CRITERION 3: The wording of the Fully Specified Name is analyzed. If this suggests a SITUATION interpretation, then it is rated “y”, otherwise “n”.

    EXAMPLE: Fracture of radius AND ulna (disorder). This is clearly the addition of two disjoint conditions (the condition Fracture of radius is never a Fracture of ulna. Therefore a SITUATION interpretation is strongly suggested by the wording. The rating is therefore “y”.

All raters used the General Release Version 2.3.1 of the IHTSDO Workbench. This workbench is the official editing and developing environment used in developing SNOMED CT. A stand-alone version is available through the IHTSDO. The features relevant for this study are that it provides the fully specified, preferred names and synonyms, the normalized form of the logical definitions, and the hierarchies including all ancestors and children.

The stratified random sample was assigned in overlapping sets to four raters. Thus, each rater analyzed 125 concepts, 25 of which were evaluated by a second expert. The inter-rater agreement, based on the hundred double evaluations was computed as Cohen’s Kappa.

Results

Table 2 shows a global rough interpretation of the rating results.

Table 2.
Rating results

We can summarize them as follows: According to the criteria used, between 11% and 18% of the SNOMED CT concepts appear to require a situation interpretation. If we look at children only, the rate goes up to 40%, but here we have no representativity any more, and at least in our sample this result is strongly influenced by an outlier. When a concept requires a situation reading, all its descendants do too (because properties – here being a descendant of Situation – propagate through the taxonomy). We therefore added a binary criterion C2’ on whether there is a situation interpretation at some child concept or not.

The results showed rather low inter-rater reliability for the analysis of logical definitions (kappa = 0.32) and a much better one for the analysis of names (kappa = 0.61). The analysis of children showed less reliability. These analyses were therefore disregarded.

Tables 3 – 5 show other results of interest. There is a very discrete indication that concepts with role groups are more likely to engender a situation interpretation than concepts without (Table 3). However, there is no evidence that the situation interpretation grows with the number of role groups as hypothesized. Interestingly there is no dependency at all between the number of situation-like concepts and their depth in the hierarchy (Table 4). Finally, Table 5 shows that among terminal concepts the rate of situation-likeness is much higher than among non-terminal ones.

Table 3.
Influence of role groups
Table 4.
Influence of depth (stratified, see Table 1)
Table 5.
Influence of graph topology

Discussion

One lesson that was not entirely unknown to us but was surprising in its impact on our study was the influence of the use of different browsers to evaluate the selected concepts. We suspect many people may assume, as we did, that there is a relatively minor impact of display on the proper use of terminology. The variability caused by use of different browsers greatly influenced our initial evaluation and resulted in a methodological change; we believe this indicates a need for further investigation of the influence of browsing and display on the proper use of terminology in medical records.

According to our study, roughly one out of five SNOMED CT concepts from the disorder hierarchy exhibits evidence for a situation reading. For each of the possible cases we here list some typical examples:

Situation-like Fully Specified Name, but no indications from the logical axioms

  • Allopurinol poisoning of undetermined intent (disorder): The reference to the intent gives a clear indication that, apart from the intoxication process itself, the circumstances in which the poisoning took place are assessed.
  • Closed fracture of multiple cervical vertebrae without spinal cord injury (disorder): The negated clause requires looking beyond the fracture itself, because it is only interpretable within a given time frame, as the spinal cord injury may occur after the fracture. The definitional axioms provide no evidence for a situation reading because the concept is primitive and the negated clause is not expressed, perhaps due to restrictions of the underlying logic.

Evidence from logic, but a condition-like Fully Specified Name

  • Hematoma of kidney: This concept has two parents, renal mass, and hemorrhage. A renal mass is an object, and a hemorrhage is ordinarily interpreted as a process. Therefore, the two classes should not overlap. If Hemorrhage is interpreted as “Situation in which a hemorrhage occurs”, and Renal mass as “Situation in which a renal mass exists”, the two classes can have members in common, e.g. the situation depicted in Fig. 2. An alternative interpretation requires Hemorrhage to be interpreted as a structural condition, meaning blood that has left the vascular space. Under this interpretation, the renal mass and the extravasated blood are the same thing, and a condition interpretation is acceptable. This example illustrates the difficulty of reaching consensus between experts about whether a code necessarily represents a situation, or not.
    Figure 2.
    Example of a clinical situation which contains the two conditions bleeding and renal mass
  • Laugier-Hunziker syndrome (disorder): According to the inherited axioms this disorder has its location both in the nail and in the mouth. It is a common descendant of Disorder of nail and Disorder of mouth. As nothing can sit simultaneously in both body parts, and a time-sequential interpretation (first in the mouth, then in the nail) would contradict the notion of a syndrome (even if compatible with the DL interpretation, which is currently undefined for SNOMED CT with regard to time). So it is a clear case of additivity, and therefore requires a situation interpretation.

Evidence for additiveness in some child concept

  • Disseminated candidiasis (disorder): This concept is unsuspicious to a situation reading regarding its fully specified name and its formal definition. However, it has a fully defined child concept, Neonatal systemic candidosis, which has the additional parent Sepsis due to candida. Sepsis and candidosis, seen as condition classes, have no common members.

Conclusion

The distinction in interpretation between representing (i) a clinical condition itself and (ii) the clinical situation that includes it sounds abstract, even abstruse. However, it has a far-reaching impact on the interpretation of taxonomies.

The condition interpretation (i) suggests that a) sibling classes should be mutually exclusive, and that b) combinations of codes representing combination of conditions – e.g. Closed fractures with(out) spinal cord injury – should be complexes in which Closed Fracture and Spinal cord injury are classes of ontologically related entities, which, however, have no members in common, i.e. the complexes would be kinds of neither Closed fracture nor Spinal cord injury.

By contrast, under the interpretation of codes as situations, such combinations represent phases of a patient’s life in which all the conditions are present. Therefore the class of the complex is a subclass of each of the situation classes (each of which includes a single condition) which make it up.

The results of this study show that a considerable proportion, possibly 20% or more, of the codes in the SNOMED CT disorder hierarchy can only be sensibly interpreted as situations, whether or not the other ones are compatible with an interpretation as conditions. Being compatible with interpretation as conditions does not prevent them being interpreted as situations instead, because for each condition concept a related situation concept can easily be created, at least in the standard context of the use of both terminologies, viz. clinical coding.

The situation interpretation is also a rationale for the fact that nearly half of all SNOMED CT disorder concepts exhibit only one single role group in their formal definition. This is a phenomenon that looks strange at first sight; given that role groups were created for concepts that involve more than one finding site and more than one associated morphology, to make explicit which morphology is related to which site. Under the new interpretation as situations, there is a new rationale for role groups: to relate a situation (the disorder concepts) to a condition (conjunction of a morphology with a finding site role). Role groups could then sensibly reinterpreted as a general has-part relation.

The inter-observer agreement in making the distinction between codes that must be interpreted as situations and those that need not be was less than what we had expected. This clearly demonstrates that such a distinction is difficult, given the large number of borderline cases. Any attempt to segregate the disorder codes into two groups, one for conditions and one for situations, would probably be doomed to failure.

The alternative would be to declare all disorder codes to have a situation interpretation. This approach is simple and compatible with the existing usage in electronic health records and, we believe, appropriate to ICD - SNOMED CT mapping. This option would leave open the challenge of developing a SNOMED CT representation for conditions, and rules for determining when codes for conditions are pre-coordinated into the terminology.

​
Figure 1.
Broken forearm

Acknowledgments

This work was supported by the World Health Organization (WHO) and the International Health Terminology Standards Development Organization (IHTSDO) through their Joint Advisory Group (JAG). We also thank Mathias Brochhausen (UAMS) for useful comments.

Article information

AMIA Annu Symp Proc. 2012; 2012: 819–827.
Published online 2012 Nov 3.
PMCID: PMC3540515
PMID: 23304356
Stefan Schulz, MD,1,2 Alan Rector, MD, PhD,3 Jean-Marie Rodrigues, MD PhD,4,5 and Kent Spackman, MD PhD6
1Institute for Medical Informatics, Statistics and Documentation, Med. University Graz, Austria
2Institute of Medical Biometry and Med. Informatics, Univ. Medical Center Freiburg, Germany
3School of Computer Science, Univ. of Manchester, UK;
4INSERM UMR 872 Eq 20, Paris, France
5Department of Public Health and Medical Informatics, University of Saint Etienne, CHU, France
6International Health Terminology Standard Development Organisation, Copenhagen, Denmark
Copyright ©2012 AMIA - All rights reserved.
This is an Open Access article: verbatim copying and redistribution of this article are permitted in all media for any purpose
Articles from AMIA Annual Symposium Proceedings are provided here courtesy of American Medical Informatics Association

References

1. Tudorache T, Falconer S, Nyulas C, Storey MA, Ustün TB, Musen MA. Supporting the Collaborative Authoring of ICD-11 with WebProtégé. AMIA Annual Symposium Proceedings; Washington, DC. 2010. pp. 802–806. [PMC free article] [PubMed] [Google Scholar]
2. Rodrigues JM, Kumar A, Bousquet C, Trombert B. Using the CEN/ISO standard for categorial structure to harmonise the development of WHO international terminologies. Studies of Health Technologies and Informatics. 2009;150:255–259. [PubMed] [Google Scholar]
3. International Health Terminology Standards Development Organisation (IHTSDO) SNOMED CT: [ http://www.ihtsdo.org/snomed-ct/]. Last visited Mar 15, 2012.
4. Collaboration Arrangement between the International Health Terminology Standards Development Organization FMBA (IHTSDO) and the World Health Organization (WHO) Jul, 2010. [ apps.who.int/classifications/whoihtsdo/]. Last visited Mar 15, 2012.
5. Smith B. Applied Ontology: A New Discipline is Born. Philosophy Today. 1998;12(29):5–6. [Google Scholar]
6. Schulz S, Spackman K, James A, Cocos C, Boeker M. Scalable representations of diseases in biomedical ontologies. Journal of Biomedical Semantics. 2011;2(Suppl 2):S6. [PMC free article] [PubMed] [Google Scholar]
7. W3C OWL Working Group OWL 2 Web Ontology Language Document Overview. 2009.
8. Baader F, et al. The Description Logic Handbook: Theory, Implementation, and Applications. 2nd Edition. Cambridge University Press; 2007. [Google Scholar]
9. Horridge M, Patel-Schneider P. OWL 2 Web Ontology Language Manchester Syntax. 2009. [ http://www.w3.org/2007/OWL/wiki/ManchesterSyntax]. Last visited Mar15, 2012.
10. Spackman KA, Dionne R, Mays E, Weis J. Role grouping as an extension to the description logic of ONTYLOG, motivated by entity modeling in SNOMED. AMIA Annual Symposium Proceedings; Washington, DC; 2002. pp. 712–716. [PMC free article] [PubMed] [Google Scholar]
11. Schulz S, Hanser S, Hahn U, Rogers J. The semantics of procedures and diseases in SNOMED CT. Methods of Information in Medicine. 2006;45(4):354–358. [PubMed] [Google Scholar]
12. Schulz S, Rector A, Rodrigues JM, Chute C, Üstün B, Spackman K. Ontology-based convergence of medical terminologies. SNOMED CT and ICD-11. In: Schreier G, Hayn D, Hörbst A, Ammenwerth E, editors. Proceedings of eHealth2012; 2012 Mai 10–11; Vienna, Austria: OCG; 2012. 2012. [Google Scholar]
AMIA Annu Symp Proc. 2012; 2012: 819–827.
Published online 2012 Nov 3.
Copyright/LicenseRequest permission to reuse

Figure 2.

An external file that holds a picture, illustration, etc. Object name is amia_2012_symp_0819f2.jpg

Example of a clinical situation which contains the two conditions bleeding and renal mass

Table 1.

Stratification of sample by distance from the root of the SNOMED CT hierarchy

Distance from rootNumber of conceptsProportionNumber in SampleStratified distanceProportion in StratumConcepts in Stratum
010.0%0114,2%56
1780.1%01
221273.3%131
3709010.8%431
41565723.9%96223.9%96
51760226.9%108326.9%108
61345720.6%82420.6%82
763929.8%39514,4%58
823193.5%145
95770.9%45
10920.1%15
1140.01%05

Table 2.

Rating results

C1: positive additivity of parent conceptsC2: Child concepts with additivity (count)C2’:At least one child concept evident for SituationC3: Evidence of Situation by fully specified name (ratio of positive ratings)At least one positive rating at the concept level (C1 or C3), ratio of pos. ratingsAt least one positive rating at concept + Child level (C1, C2’,C3), ratio of pos. ratings
Sample size400559400400400400
Number in sample43223726488143
Ratio10.8%39.9%18.0%16.0%22.0%35.8%
Cohen’s Kappa for binary ratings0.32-0.180.610.560.26
Normalized to all disorder concepts (total = 65396)
With 95% confidence interval		7,030 [± 1,962]26,088 [± 2,001]11,771 [± 2,425]10,463 [± 2,341]14,387 [± 2,648]23,379 [± 3,061]

Table 3.

Influence of role groups

Number of role groups (inherited plus asserted)Number of concepts in the samplePositive ratings at concept level (C1 or C3)Percentage of positive ratings
019210.5%
11943920.1%
21242923.3%
3521528.8%
4 and more11327.3%

Table 4.

Influence of depth (stratified, see Table 1)

Hierarchical levelNumber of concepts in the samplePositive ratings at concept level (C1 or C3)Percentage of positive ratings
1561221.4%
2962121.8%
31082422.2%
4821923.1%
5581220.7%

Table 5.

Influence of graph topology

TopologyNumber of concepts in the samplePositive ratings at concept level (C1 or C3)Percentage of positive ratings
Terminal concepts (without children)2696925.6%
Non-terminal concepts (with children)1311914.5%