Logo of bmcbioiBioMed Centralsearchsubmit a manuscriptregisterthis articleBMC Bioinformatics
BMC Bioinformatics. 2006; 7: 250.
Published online May 8, 2006. doi:  10.1186/1471-2105-7-250
PMCID: PMC1533868

Statistical modeling of biomedical corpora: mining the Caenorhabditis Genetic Center Bibliography for genes related to life span

Abstract

Background

The statistical modeling of biomedical corpora could yield integrated, coarse-to-fine views of biological phenomena that complement discoveries made from analysis of molecular sequence and profiling data. Here, the potential of such modeling is demonstrated by examining the 5,225 free-text items in the Caenorhabditis Genetic Center (CGC) Bibliography using techniques from statistical information retrieval. Items in the CGC biomedical text corpus were modeled using the Latent Dirichlet Allocation (LDA) model. LDA is a hierarchical Bayesian model which represents a document as a random mixture over latent topics; each topic is characterized by a distribution over words.

Results

An LDA model estimated from CGC items had better predictive performance than two standard models (unigram and mixture of unigrams) trained using the same data. To illustrate the practical utility of LDA models of biomedical corpora, a trained CGC LDA model was used for a retrospective study of nematode genes known to be associated with life span modification. Corpus-, document-, and word-level LDA parameters were combined with terms from the Gene Ontology to enhance the explanatory value of the CGC LDA model, and to suggest additional candidates for age-related genes. A novel, pairwise document similarity measure based on the posterior distribution on the topic simplex was formulated and used to search the CGC database for "homologs" of a "query" document discussing the life span-modifying clk-2 gene. Inspection of these document homologs enabled and facilitated the production of hypotheses about the function and role of clk-2.

Conclusion

Like other graphical models for genetic, genomic and other types of biological data, LDA provides a method for extracting unanticipated insights and generating predictions amenable to subsequent experimental validation.

Background

In the design, analysis and interpretation of experiments, biomedical and clinical researchers encounter the problem of evaluating and summarizing prior knowledge on the subject under investigation. Traditional solutions include examining articles in scientific journals, primarily via PubMed access to MEDLINE, and interrogating WWW-based sources such as Entrez Gene [1] and Online Mendelian Inheritance in Man (OMIM) [2]. Consider a scenario in which a researcher seeks enhanced knowledge about a protein implicated in aging. Typically, the steps involved in addressing this problem include interrogating structured data resources: searching protein sequence databases to identify homologs in other species, querying warehouses of genomic information to determine key non-coding regions and polymorphisms, examining collections of high-throughput molecular profiling data sets to ascertain genes with similar patterns of expression, probing ontologies such as the Gene Ontology (GO) [3] to uncover other genes with similar patterns of annotation, and so on. The entire procedure is accompanied by examination of the literature to determine, for example, classes of proteins mentioned in the same article as the putative gerontogene.

Advanced techniques and sophisticated tools for interacting with structured data are well known, widely available and include BLAST [4] for sequence databases, Ensembl [5] and the UCSC Browser [6] for genomes, GEO [7] for transcript profiles, and tools available from the GO that allow navigation of terms in the ontology. This is less true for the scientific literature. Given the time-consuming yet critical importance of synthesizing information in text corpora such as MEDLINE, the problem of making data interpretation a more systematic and automated endeavor is emerging as an important topic of research (see, for example, [8-10]). The development of strategies capable of providing a user the ability to assimilate and act upon information present in resources of structured and unstructured data remains an important goal.

The primary aim of biomedical text mining is the systematic analysis of document collections such as MEDLINE abstracts and full-text journal articles with the goal of generating useful and unanticipated scientific discoveries (for recent reviews of current methods and illustrative applications, see [11-13]). Examples of tasks addressed by text mining methods include identifying literature relevant to specific molecules, finding associations between genes and diseases, determining putative functions for proteins, and predicting regulatory networks.

A common approach to text mining is to treat the problem as one of natural language processing (NLP) [14]. NLP methods concentrate on the linguistic structure of documents and make explicit use of syntactic, relational, and ontological knowledge. In biology, such approaches [15] have been employed for information extraction: the task of ascertaining facts, relations, and entities in unstructured written language such as protein-protein interactions, protein subcellular location, and gene names. Tools based on these ideas include Textpresso [16] and Telemakus [17]. Elsewhere, NLP has been used in conjunction with LocusLink and GO to compare OMIM to MEDLINE [18].

Recently, nouns extracted from MEDLINE abstracts tagged with parts of speech were combined with knowledge from other sources, Principal Component Analysis, and means linkage clustering to find associations between genes and phenotypes [19]. In general, NLP can be effective in circumscribed domains where linguistic knowledge is available and the terminology evolves slowly, is consistent, largely unambiguous, and relatively simple. However, the paucity and incomplete nature of such information for biomedical corpora suggests that the full potential of text mining in biology remains unrealized.

An alternative to NLP is to frame the problem from the perspective of information retrieval (IR) [20]. Statistical IR methods explore large quantities of information and often involve capabilities for clustering, classifying, categorizing, summarizing, and detecting novel, similar and relevant objects. The most successful testaments to the real-world utility of IR techniques are Internet search engines. Thus, statistical IR models of biological document collections could reveal rich, complex, and previously unappreciated relationships. Such results would complement insights derived from analysis of molecular profiling, protein-protein interaction, gene knock-out, and similar types of data. With systematic deployment of IR tools, the interrogation of biomedical corpora could become as routine and indispensable a part of research as analyzing genomic and genetic data is today. The analogy is more than superficial and extends to the direct use of IR techniques such as singular value decomposition in bioinformatics (see for example [21]). Thus, the common mathematical foundations for algorithms that underpin IR and genome analysis make it possible to envision integrated procedures that combine primary biological data with biological corpora.

This work describes an application of statistical IR methodology to the analysis of a biomedical text corpus, the Caenorhabditis Genetic Center (CGC) Bibliography (Figure (Figure1).1). The specific model at the heart of this study is the Latent Dirichlet Allocation (LDA) model [22], a hierarchical Bayesian model employed previously to analyze text corpora and to annotate images [23]. Recently, LDA has been used to extract and analyze the topics present in a document corpus consisting of articles published in the journal Proceedings of the National Academy of Sciences [24].

Figure 1
One of the 5,225 free-text items in the CGC Bibliography in its original form.

In general, IR methods assume that the order of words in a document can be neglected and view documents as "bags of words." The loss of information incurred by ignoring word order is offset by the ability to devise efficient computational algorithms that are viable for large corpora. Although there is no theoretical justification for casting a document in this manner, the practical benefits and utility of doing so are considerable. The LDA model considered here is a model for a corpus viewed as a collection of bags of words. It assumes that each word of each document is generated by one of several "topics"; each topic is associated with a different conditional distribution over a fixed vocabulary. The same set of topics is used to generate the entire set of documents in a collection but each document reflects these topics with different relative proportions. Thus, LDA is a mixture of mixtures model, i.e., the mixture components are shared across all documents but each document exhibits different mixture proportions. As a generative probabilistic model, the LDA can handle unseen or novel data, i.e., a document that was not one of the bag of words used to estimate the model.

The fundamental entities in LDA, random variables representing topics and words, are grouped together in such a way to form a corpus, i.e., a group of groups of words. The hierarchical nature of the model stems from the fact that documents are modeled as probability distributions across topics, and topics are modeled as probability distributions across words. A notable virtue of LDA is that a given topic can occur with high probability in multiple documents, and that a given word can occur with high probability in multiple topics. Topics are treated as latent variables, namely entities that are not present explicitly in the data (a set of sequences of words), but are presumed to be present implicitly and are to be inferred by statistical analysis.

To analyze the CGC Bibliography, each item in the corpus was recast as a bag of words and the resultant data set of documents was used to estimate the parameters of three different statistical IR models. The predictive performance of the LDA model was better than that of two simpler bag of words models, a unigram model and a mixture of unigrams model, trained on the same data set. The potential of LDA in assisting biological studies was illustrated by considering the phenomenon of nematode aging. In order to illuminate the hidden factors permeating a corpus and captured by the topics discovered by a trained CGC LDA model, LDA topics were labeled via an automated process that assigned words from the CGC vocabulary (corpus-based labels) and GO terms (ontology-based labels) to each topic. Examination of these labels indicated that the CGC topics captured meaningful and plausible facets of nematode biology. To investigate aging, topics whose corpus-based labels included many CGC words corresponding to the names of genes known to influence life span were identified. For the two topics with the greatest number of such CGC-based topic labels, novel candidates for age-related genes were equated with other CGC-based topic labels that corresponded to gene names (guilt-by-association). Finally, an LDA-based measure of pairwise document similarity was devised and used to address the problem of searching a database of documents to determine topic-space homologs of a query document. Inspection of the "document homologs" of the CGC item shown in Figure Figure11 resulted in enhanced understanding of the biology of the clk-2 gene.

This work highlights the potential and utility of LDA in organizing and exploiting one type of widely available information resource, a collection of documents in the form of free or unstructured text. However, researchers are faced with a plethora of resources including images and structured data such as molecular sequences, transcript profiles, disease information, and so on. Thus, there is a compelling need for techniques and systems able to condense, integrate and present large amounts of disparate data to a user. This paper concludes with a discussion of how the family of probabilistic graphical models, of which LDA is a specific example, provides a framework for integrating heterogeneous data and thus meets this challenge.

Results

LDA outperforms mixture of unigrams, unigram and random models

In order to compare different models of text, a data set of C. elegans related documents was created. In particular, each CGC Bibliography free-text item was transformed into a bag of words yielding a corpus of M = 5, 225 documents and a V = 28, 971 word vocabulary.

The generalization performance of three statistical models was assessed: an LDA model (Figure (Figure2),2), a mixture of unigrams model (right, Figure Figure3),3), and a "baseline" unigram model (left, Figure Figure3).3). A model was trained using 90% of the 5,225 documents in the CGC corpus and tested on the remaining 10%. LDA and mixture of unigrams models with K = 5, 10, 20, 50, and 100 latent topics were estimated; a single unigram model was estimated because such models harbor no notion of topic. The perplexity (inverse of the per-word likelihood) of the held-out test set of J = 525 documents (Equation 6) was computed for each trained model. Figure Figure44 shows the generalization performance of each model as a function of the number of latent topics. LDA has consistently smaller perplexity scores than the two extant models indicating better performance on unseen documents. Since the perplexity of 50- and 100-topic LDA's is low and similar, a latent space with 50 topics appears to provide a parsimonious description of the CGC corpus.

Figure 2
Graphical model representation of the LDA model (left) and the variational distribution used to approximate the posterior in LDA (right) [22]. LDA defines a distribution on a collection of documents in much the same manner that a profile hidden Markov ...
Figure 3
Graphical model representation of a mixture of unigrams model with K latent topics (right) and a unigram model (left). The corpus depicted contains M documents and each is a sequence of N words. Open circles represent latent variables (z)or parameters ...
Figure 4
The perplexity of LDA, mixture of unigrams, and unigram models estimated and evaluated on the CGC corpus. The score of test documents is shown against the number of latent topics (the perplexity of the unigram is constant because this statistical model ...

The ability of three specific models to retrieve a set of 842 aging-related documents in the collection of 5,225 CGC documents was assessed: a 50-topic LDA model estimated using all documents in the corpus, a 50-topic mixture of unigrams model estimated using all documents, and a model which ordered all documents randomly. For each model, an average precision/recall (PR) curve was constructed by computing the ranking of other documents given each aging-related document as a query, and the average F1 measure was computed. Figure Figure55 shows average PR curves for the three models. The average F1 measure (standard error) for the LDA model, the mixture of unigrams model and a random model is 0.30(2.86e – 06), 0.22(7.85e – 06), and 0.29(3.02e – 05) respectively. Although the F1 values for the LDA and random models are similar, the smaller standard error of the LDA model indicates its superiority to the random model. In addition, the average PR curves indicate that the LDA places more of the age-related documents higher up its rankings than the random model. Thus, of the three models investigated, LDA is best able to retrieve the set of related documents. Overfitting by the mixture of unigrams model results in a performance worse than the random model.

Figure 5
Precision/recall (PR) curves for three models of text (LDA, mixture of unigrams, random) and the task of retrieving a set of aging-related documents (842 CGC items that refer to one or more of the genes listed in Table 1). Precision is the fraction of ...

All subsequent discussion of an LDA model and/or a mixture of unigrams model pertain to a K = 50 topic model estimated using all M = 5, 225 CGC documents in the corpus.

LDA latent topics embody concepts associated with nematode biology

A systematic strategy for clarifying the nature of the hidden factors permeating a corpus was devised and applied to a CGC LDA. Topic annotation (topic labeling) is defined as an automated process that creates a verbose (compact) description of an LDA topic. The method designed to annotate and label topics exploited the corpus-level parameter β (Figure (Figure2).2). The K × V topic-word matrix β collates the multinomial distributions over the V words in the vocabulary that characterize the K topics. For a given LDA model of a particular corpus, the kth row specifies the topic-specific word distribution for topic k and an element, βkv, denotes the likelihood of the vth word given the kth topic. For each of the K = 50 topics in the CGC LDA model, the V = 28, 971 βkv values were ordered and used to generate a word rank versus topic-specific word probability plot. In every case, the 500 top-ranked words accounted for most of the probability mass. Thus, these 500 high probability CGC words were designated topic annotation words (the same word from the V-wordvocabulary could annotate multiple topics).

Two different approaches were used to create labels for each topic. Corpus-based topic labels are topic annotation words that are unique to a topic and represent descriptors applicable to only one topic. CGC-based topic labels were equated with topic annotation words that were not assigned to any of the other 49 topics, i.e., the 50 sets of CGC-based topic labels formed disjoint subsets of words from the CGC vocabulary. Ontology-based topic labels are the outcome of filtering topic annotation words using an external knowledge source and represent descriptors applicable to one or more topic. The ontology exploited here was the GO. The relationship between GO controlled vocabulary terms can be depicted as a directed acyclic graph (DAG). Each node corresponds to a term from one of three aspects, for example, "exodeoxyribonuclease," "mitochondrial derivative" and "ethylene mediated signaling pathway" are exemplars of GO terms from the "Molecular Function," "Cellular Component" and "Biological Function" aspects respectively. The structure of the DAG underpinning the GO vocabulary defines semantic relationships amongst terms so that, for example, the node for the GO term "intracellular" is a parent of the node for the more specific GO term "nucleus." Recall that the topic annotation words for topic k are the 500 words from the CGC vocabulary that best characterize the topic. These 500 words were mapped to nodes in the GO DAG. A node where a topic annotation word coincided with a GO term was designated an explicit node. GO-based topic labels were equated with the GO terms for both explicit nodes and the children and grandchildren of explicit nodes.

Examination of the automatically generated CGC- and GO-based labels suggests that LDA topics capture meaningful and coherent facets of the molecular, cellular, and behavioral biology of C. elegans. Figure Figure6 shows6 shows results for four selected CGC topics (the results for all 50 topics are available in Additional file 1). The hidden factors permeating the CGC corpus include one pertaining to sexual reproduction (Topic 6), chromosome structure and function (Topic 14), cell death (Topic 20) and locomotion (Topic 27).

Figure 6
Results for four illustrative latent topics specified by a 50-topic CGC LDA model estimated from a corpus with 5,225 documents and a vocabulary of 28,971 words. Each panel shows results for a particular topic. The y-axis of the graph is topic-specific ...

Ontology-based topic labels derived from structured knowledge for domains other than molecular and cellular biology are needed to clarify the nature of some CGC hidden factors. Figure Figure77 shows topics that have CGG- but no GO-based labels. Inspection of the CGC-based labels for Topics 3, 29, 38, and 41 suggest the presence of hidden factors that are concerned with scientific protocols and procedures that are independent of any biological question, and that allude to evolution.

Figure 7
CGC LDA topics that have no GO-based topic labels and capture hidden factors in the CGC corpus that pertain to the practical aspects of investigating biological mechanisms and processes. Topics are represented in the same manner as in Figure 6.

Interrogation of LDA topics provides insights into genes influencing life span

A guilt-by-association approach was devised to identify genes that may be involved in a phenomenon of interest and the procedure illustrated using genes implicated in modifying life span. A "gene word" is a word in the 28,971 word CGC vocabulary that corresponds to the name of a C. elegans gene. CGC-based topic labels are the ≤ 500 words in the vocabulary that best characterize a topic. If a number of the topic labels are gene words and most of the genes are known to be associated with a specific phenomenon, then the other gene words can be equated with genes likely to be involved in the same phenomenon. One factor influencing the biological insights that can be derived from this approach is the human curation component of the process used to create the CGC Bibliography, i.e., the individual who defined the set of genes in the Genes record believed to be discussed in the Abstract record. In addition to limitations in the data used to estimate a statistical model of text, the LDA remains a model based on the simple bag of words representation of a document. While this LDA-based approach is not an automated method for formulating sophisticated and detailed hypotheses, it does highlight how a model that ignores syntax and semantics can organize information in a manner that provides a user the ability to exploit their background knowledge and enhance understanding of the subject in hand.

Table Table11 lists the names of genes known to extend or shorten life span and designated aging-related gene words. Inspection of the two CGC LDA topics with the greatest number of CGC-based labels that are aging-related gene words suggests that akt-1, akt-2 and ges-1 may be associated with aging. Figure Figure88 shows these two topics, Topic 9 and Topic 12. In decreasing topic-specific word probability and with genes listed in Table Table11 in bold, the Topic 9 CGC-based labels that are gene words are age-1, clk-1, mev-1, daf-18, fer-15, clk-2, gro-1, daf-23, akt-1, akt-2, clk-3, pdk-1, rad-8, sod-3, old-1, ctl-1, tkr-1, daf-28, sod-2, sir-2, daf-9, cln-3, ins-1, age-2, and spe-10. The genes depicted in a normal font have properties similar to known life span modifying genes such as dauer (daf) phenotypes (Table (Table1).1). For example, the Wormbase [25] annotations for akt-1 and akt-2 include "protein serine/threonine kinase" and "inhibition of both akt-1 and akt-2 leads to dauer-constitutive phenotype."

Figure 8
The two CGC LDA topics with the greatest numbers of aging-related gene words (CGC-based topic labels corresponding to the names of genes implicated in modifying life span). Each topic is represented the same manner as Figure 6. In the graph, words in ...
Table 1
C. elegans genes known to extend or shorten life span. The list is taken from the Genes database of SAGEKE http://www.sageke.org

The mechanisms of action of putative gerontogenes suggested by different topics may not be identical. For Topic 12, the CGC-based labels that are gene words are ges-1, osm-3, osm-5, che-2, osm-1, lov-1, pkd-2, daf-19, che-11, unc-116, che-13, che-10, osm-10, che-1, che-14, pho-1, dyf-1, che-12, and pkd-1. ges-1 is a gut-specific carboxylesterase, a molecular function not ascribed by Wormbase to life span modifying genes. Since many topic labels are associated with the osm phenotype, osmoregulation may be a feature that differentiates Topic 12 aging-related genes from those of Topic 9.

Exhibition of multiple latent topics by an LDA document reflects the complexity of issues discussed in documents

By virtue of its superior generalization performance and retrieval ability, an LDA model of the CGC corpus is a better statistical model of text than a mixture of unigrams model. A distinct advantage of LDA is that although both models are generative, an LDA document is the manifestation of many topics whereas a mixture of unigrams document is the product of only one topic. The subject matter of (biological) documents is rarely limited to a single area so the benefit of a CGC LDA is that words in a single document could come from, for example, a combination of Topic 6 (sexual reproduction) and Topic 14 (chromosome structure and function). The mixing of LDA topics in a CGC item was investigated by examining the document-specific, word-level parameter [var phi] (Figure (Figure2).2). The variational posterior topic probability [var phi]n(zn= k) indicates the extent to which the nth word is associated with the kth topic. A value that is both large and significant is an indicator of the topic most likely to have generated the word.

The CGC item shown in Figure Figure11 is primarily a mixture of two topics. Figure Figure99 shows the topics most likely to have produced words in the document discussing the life span modifying clk-2 gene (Table (Table1).1). Of the assigned words, 34 have posterior probabilities peaked on the aging-related Topic 9 (Figure (Figure8)8) and 23 on the general purpose Topic 38 (Figure (Figure7).7). Three words are allocated to Topic 7, two to Topic 19, two to Topic 13, and one to Topic 34.

Figure 9
The LDA topics most associated with words in the CGC item shown in Figure 1. A word is identified with the topic k given in parenthesis when the document-specific, variational posterior topic probability exceeds a threshold, [var phi]n(zn= k) > ...

Utility of LDA in formulating hypotheses: insights into clk-2 function

Searching a document database to identify homologs of a query document yields insights that can complement those obtained from sequence-, structure- and function-based analysis of genes and proteins. Prior studies of clk-2 revealed that it encodes a sequence homolog of Tel2, a protein required for normal telomere length regulation in yeast (reviewed in [26]). To enhance knowledge of how clk-2 might influence life span, homologs of a document discussing clk-2 (Figure (Figure1)1) were identified by computing the topic-space pairwise similarity score between this query q and every CGC document t (Equations 2 and 3). Although a given gene may appear in the Genes record of many CGC items, the results described below are based on analysis of document homologs of the single CGC item shown in Figure Figure1.1. Figure Figure1010 shows the three most related items, CGC documents with the three largest An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i1.gif(q, t) values. Figure Figure1111 shows the three topics most associated with them. The third best homolog is relatively uninformative: the text indicates a general review of aging mutants and Topic 7 labels are general words pertaining to life span.

Figure 10
CGC homologs of the clk-2 item shown in Figure 1. When this clk-2 item is used as the query document, the three items shown have the largest topic-space pairwise similarity scores, An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i1.gif(q, t). The documents are depicted in the same format as Figure 9. As ...
Figure 11
The CGC LDA topics most associated with words in the document homologs of the clk-2 query shown in Figure 10.

LDA-based analysis leads to the hypothesis that clk-2 may have a role in coordinating signals between the outside and inside of cells. Since the top two topic-space document homologs discuss nuclear receptors, clk-2 may have a direct or indirect involvement in receptor biology. Topics 44 and 45 include the GO-derived labels "host cell plasma membrane," "regulation of fgf receptor signaling pathway" and "regulation of beta 2 integrin biosynthesis." Circumstantial evidence supports a possible role for clk-2 in signal transduction and tissue biology. FGF-2 regulates telomerase activity in human endothelial cells [27]. Integrins are cell surface receptors important in communication between the extracellular environment and the nucleus [28]. The suggestion that clk-2 may influence telomere length via a mechanism not involving direct physical association with telomeres is plausible since a recent genome-wide screen for Saccharomyces cerevisiae deletion mutants that affect telomere length identified genes with very diverse functions (overrepresented categories included DNA and RNA metabolism, chromatin modification, and vacuolar traffic) [29]. Recent experimental results support a connection between vacuolar protein-sorting genes and telomere length homeostasis [30].

Discussion

This study demonstrates how a specific statistical IR model, an LDA model, can be employed to infer the hidden factors permeating a biomedical text corpus and exploited to synthesize and organize information about complex biological phenomena. The results indicate that despite being estimated from a simple bag of words representation of items in the CGC Bibliography, the intra-document statistical structure captured by an LDA model is sufficient for the model to be used to enhance understanding of C. elegans biology. For example, analysis of the corpus-, document- and word-level parameters of a trained LDA model enabled the exploration and creation of hypotheses about known and putative nematode aging-related genes.

The CGC corpus studied here had M = 5, 225 documents and a V = 28, 971 word vocabulary. Estimating a 50-topic LDA model from such training data took 3 hours on a Macintosh Powerbook G4. It should be straightforward to estimate a model for larger corpora such as MEDLINE where the number of documents is many orders of magnitude greater (M ~ 107) and the vocabulary size is only one order of magnitude larger (V ~ 105). In estimating an LDA, the computational bottleneck is the variational E-step, i.e., computing the posterior topic Dirichlet distribution for each document. Fortunately, this procedure can be parallelized because given a model, the posterior for each document can be assessed independently. Thus, it is feasible for the techniques described in this study to be applied to other corpora and to address questions other than life span modification.

From an applications perspective, it is possible to envisage a scenario involving the creation of a library of LDA models where each constituent model was estimated from a user- and/or computationally-defined corpus of documents focused on a specific area such as "aging," "cancer," "yeast biology," "response to stress," "antibiotics," "HIV," "Parkinson's disease," "kinases," and so on. A query document would be compared to each subject-specific LDA in the collection as opposed to a single model as described here. This approach seeks to mirror a common strategy in sequence analysis whereby a query sequence is rated against a library of hidden Markov models (HMMs) [31] estimated for domains of interest, for example, the Pfam database of protein families [32]. Since seaching a database of probabilistic models of proteins in order to identify "remote" sequence homologs is known to be effective, an analogous approach could prove useful in retrieving distant document homologs.

The ideas discussed in this work can be extended and improved in a variety of ways. Currently, the number of latent topics in an LDA model K is a user-defined parameter but recent research has examined the task of choosing K [33]. In particular, if the LDA is augmented with a non-parametric Bayesian prior known as a hierarchical Dirichlet process, both topic probabilities and the number of topics can be estimated from data. Under the hierarchical Dirichlet process prior, the number of topics grows as data are added to a collection [44]. The significance and practical importance of this feature is that when modeling scientific documents, the nature and size of the corpus is evolving constantly meaning that discovering topics is an ongoing task. Further studies are required to devise rigorous methods for determining a set of words best able to characterize a topic: the current approach is somewhat arbitary in that corpus-based topic labels were defined as the 500 words in CGC vocabulary with the highest values for the likelihood of the word given the topic.

In the LDA model, topics represent entities that are presumed to permeate a corpus and these latent variables are to be inferred by statistical analysis. These implicit concepts are assumed to be equally related to one another, i.e., the topics are not organized in any way and form a "flat structure." It seems reasonble to believe that, for example, topics embodying the concepts of "DNA repair" and "chromosome structure and function" should be more related to each other than to a topic focused on "locomotion." Thus, a model in which topics were themselves arranged as a hierarchy could prove useful and inspection of the relationship(s) between topics might be informative. The current LDA model represents a model for a particular instantiation of a corpus. It might be interesting to formulate dynamic models of corpora which sought to capture how a collection such as the CGC or MEDLINE changes over time. In addition to theoretical studies of their properties and behavior, such models might be useful not only for biomedical researchers and clinicians, but also policy makers, historians, and sociologists interested in the evolution of biology and its disciplines.

Statistical IR models are applicable not only to "traditional" document collections, but also to genome-scale biological data. In one example of such an LDA model, "documents" could correspond to genes, "topics" to regulatory networks and "words" to motifs. Since LDA does not require mutually exclusive clustering, a given gene can participate in several networks and a given motif can appear in several genes. Merging this type of analysis with that described here would result in evidential support for the functional behavior and role of a gene being derived both from primary biological data and from corpus-based analysis. This capacity to fuse support from disparate sources has been illustrated using a variant of LDA known as "correspondence LDA" and in the context of automated image annotation [23].

Conclusion

LDA is a special case of the family of probabilistic graphical models. This family includes a wide variety of other models that have proved useful in biology such as HMMs, phylogenetic trees, and pedigrees [34]. The graphical model formalism allows such graphical components to be combined into heterogenous, large-scale statistical models that integrate evidence from multiple sources. Doing so would yield a system for facilitating the formulation of ideas that could be interrogated and verified experimentally.

Methods

CGC document corpus

The CGC Bibliography (October 2002 release) was downloaded [35] and included abstracts from the published literature, "worm meetings," and the "Worm Breeder's Gazette." Each CGC item is a series of defined records (Key, Medline, Authors, Title, Citation, Type, Genes, Abstract) and associated free-text (see Figure Figure11 for an exemplar). In an item, the Genes record associated with an Abstract record was added by a curator and so reflects the personnel interest(s) and background of the individual, i.e., the list of genes discussed in the abstract was not derived in a systematic, automated manner and according to a fixed scheme and/or philosophy. For each item, the Genes, Title, and Abstract records were concatenated. The resultant text was tokenized by partitioning on the basis of white spaces and punctuation. Variants of the same word were stemmed to produce a single word by removing the suffixes s, ss, ies and sses. A word was discarded if it was in a set of 619 generic stop words that included ii, iii, iv, iv, a, a., yourselves, z, and zero. The ensuing text was not tagged, i.e., words were not annotated in terms of syntax (parts of speech) or semantics (pre-defined class such as "gene name").

A CGC document is the bag of words (vector space representation) obtained after tokenizing, stemming, and removing suffixes from a CGC item. The CGC vocabulary is the non-redundant set of discrete objects produced after application of the preceeding processing steps to all items in the corpus. Words in this vocabulary are unigrams because phrases such as "DNA repair" are considered to be two objects rather than one. The final CGC corpus had M = 5, 225 documents and a vocabulary of V = 28, 971 words. The shortest and longest documents had N = 9 and N = 261 words respectively. It should be noted that words originating from the Genes record were not marked in any way so neither the vocabulary constructed for the corpus nor the bag of words representation of a particular document contained explicit knowledge of, or information about the curation effort.

Latent Dirichlet Allocation (LDA) model

A detailed description of the LDA model and attendant algorithms can be found elsewhere [22,36]. Figure Figure22 gives a graphical model representation of LDA. Consider a corpus of M documents, {w1, ..., wM}, based on a vocabulary of words indexed by [1, ..., V]. A document is a sequence of N words, w = [w1, ..., wN]. Each word in a document is represented by a V-dimensional unit-basis vector in which only one component is equal to one, i.e., a word that is the vth word in the vocabulary is described by a vector where wv = 1 and wu = 0 for all u ≠ v.

LDA generates a document according to the following process,

1. Choose a point θ from a Dirichlet distribution parameterized by α: θ ~ Dirichlet (α).

2. For each word wn in turn,

(a) Choose a topic zn from a multinomial distribution parameterized by θ: zn ~ Multinomial(θ).

(b) Choose a word wn from the multinomial distribution associated with the selected topic and parameterized by An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i2.gif: wn ~ Multinomial (An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i2.gif).

The parameters of a K-iopic LDA are the Dirichlet parameter, α, and the topic-word matrix, β. α is a K-dimensional Dirichlet parameter that determines the distribution over topic proportions, α = [α1, ..., αK] where αk> 0. β is a K × V matrix that determines the likelihood of the vth word in the vocabulary given the kth topic, βkv= p(wv = 1 | zk = 1). The topic-specific word distribution βkAn external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i2.gif is the kth row of β. The document-specific topic proportions θ lie in the (K - 1)-dimensional simplex, θ = [θ1, ..., θk] where θk = p(zk= 1 | θ), θk> 0, and An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i3.gif.

The likelihood of an LDA document is obtained by marginalizing over the latent variables,

The probability of a corpus is the product of the marginal probabilites of single documents.

Inference and parameter estimation

Estimating the parameters of an LDA from data and calculating the probability of a document both involve inference or computing the posterior distribution of latent variables given a document,

The denominator of this expression, the likelihood of a document, cannot be computed exactly because of the coupling between θ and z (left, Figure Figure2).2). Instead, approximate inference methods are required for the LDA model. Such methods include the convexity-based variational approach used here [22], Markov chain Monte Carlo sampling [37] and expectation propagation [38].

Variational inference approximates the posterior by finding a lower bound on the likelihood of a document [22]. The family of graphical models obtained by uncoupling θ and z (right, Figure Figure2)2) is characterized by the following variational distribution,

where the Dirichlet parameter γ and the multinomial parameters {[var phi]1, ..., [var phi]N} are the free variational parameters for document w. For the nth word, [var phi]n(zn = k) is the (variational) posterior topic probability for the kth topic. Optimal document-specific values can be found by minimizing the Kullback-Leibler (KL) divergence between the variational distribution and the true posterior,

The variational Dirichlet parameter γ* (w) provides a representation of the document in the topic simplex. The variational multinomial parameters [var phi]*(w) = {An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i8.gif, ..., An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i9.gif} approximate the true, but intractable distributions p(zn|w).

Estimating an LDA from data involves finding the Dirichlet parameter α* and topic-word matrix β* which maximize the log marginal likelihood of a corpus. A variational Expectation-Maximization procedure results in parameter estimates that are a (possibly local) maximum of a lower bound of the log marginal likelihood. This alternates between maximizing a lower bound with respect to the variational parameters for each document, and maximizing the lower bound with respect to the model parameters.

An LDA-based measure of pairwise document similarity

The transformation of bags of words into bags of topics by LDA provides a means to address the task of searching a corpus to retrieve similar and/or relevant items. Word-space representations of documents (high-dimensional, variable length, vectors of discrete-valued features) are converted into topic-space representations (low-dimensional, fixed length, vectors of real-valued features). In particular, the variational posterior Dirichlet parameter γ* (w) indicates the degree to which each of the K topics is referenced by document w.

A new measure of pairwise document similarity was formulated using the Dirichlet probability distribution specified by the document-level LDA parameter γ (Figure (Figure2).2). Recall that the variational posterior Dirichlet distribution for document d is Dirichlet(γd) where An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i10.gif and An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i11.gif denotes the extent to which document d refers to the kth topic. A random variable θ drawn from Dirichlet(γd) has the probability density

where An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i12.gif, An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i14.gif, An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i11.gif > 1, and Γ(·) is the Gamma function. Given two Dirichlet densities, the KL divergence is given by

The similarity between two documents i and j can be quantified by computing the KL divergence between their corresponding Dirichlet distributions as follows

where γi and γj are the parameters for the two documents. ψ (·) is the digamma function and arises when taking expectations of log θ.

Let An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i1.gif(q, t) be the topic-space pairwise similarity score between a query document q and a target document t. Here, this score is defined as the symmetrized KL divergence between the variational posterior Dirichlet distributions,

where the component KL terms are computed using Equation 2.

An alternative definition of the topic-space pairwise similarity score is the Jensen-Shannon divergence

where γx = (γq + γt)/2 so that γxkis the average degree to which documents q and t refer to topic k.

Given a database of D documents, t1, ..., tD, the task of retrieving documents related to a query q was addressed by computing the score between q and each document in the collection, An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i1.gif(q, t1), ..., An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i1.gif(q, tD) (cf. searching a protein or nucleic acid sequence database to find homologous sequences). For simplicity, a "homolog" of a document q is a CGC document td that has a high topic-space pairwise similarity score An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i1.gif(q, td).

Mixture of unigrams model and unigram model

Figure Figure33 shows graphical model representations of two existing models of text. Like the graphical model representation of the LDA model (left, Figure Figure2),2), the open circles represent parameters for the mixture of unigrams model (mixing weights θ; word distribution βk)and unigram model (word distribution β).

The unigram model contains no latent variables and each word in every document is assumed to have been drawn from the same multinomial distribution. Denoting this word distribution as β, the likelihood of a unigram document is

The mixture of unigrams model [39] assumes that each document is generated by first choosing one of K topics, and then drawing words independently, conditioned on that topic. As in LDA, the K multinomial distributions represent an underlying semantic structure in the corpus but a mixture of unigrams document is a manifestation of only one of these topics. Denoting the mixing weights as θ and the word distributions as β, the likelihood of a mixture of unigrams document is

The LDA model builds upon the mixture of unigrams in that documents are able to manifest multiple topics. Overall, the estimated LDA word distributions are better reflections of the underlying topics in a corpus, particularly in light of heterogenous documents.

Assessment of statistical models of text

Perplexity: generalization performance

The performance of a statistical model on unseen data was evaluated by computing the perplexity of a test set of J documents not used to estimate the model,

where Nj is the number of words in test document wj. The perplexity is equivalent to the inverse of the geometric mean per-word likelihood and a lower score indicates better generalization performance.

The three different bag of words models described above were assessed by determining their respective perplexity on the same set of test documents. The likelihood of a test document p(wj) was computed using Equation 1 for an LDA model (left, Figure Figure2),2), Equation 4 for a unigram model (left, Figure Figure3),3), and Equation 5 for a mixture of unigrams model (right, Figure Figure3).3). A model was trained using 90% of the CGC corpus (4,700 documents) and the remaining 10% (J = 525) used to compute perplexity. A single unigram model was estimated and evaluated whereas LDA and mixture of unigram models with varying numbers of latent topics were estimated and evaluated (K = 5,10, 20, 50,100).

Precision/recall (PR) curves and F1 measure: retrieval capability

The ability of a statistical model to retrieve a set of related documents was evaluated using a language modeling approach [20,40]. Let ND be the number of "relevant" documents in a collection of D documents, documents that are related according to some criterion. Let NP be the number of these relevant documents that are present in a list of P of these documents (P D). Precision, An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i22.gif, is the fraction of documents in the list that are relevant and is defined as NP/P. Recall, An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i23.gif, is the fraction of relevant documents in the list and is defined as NP/ND (An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i23.gif = 1 when the list of documents and collection of documents are identical, NP = ND). The F1 measure is the harmonic mean of these two statistics,

Given a ranking of the documents in the collection, D lists of documents are produced by selecting the top P-ranked documents where P = 1, ... ,D. For the dth list, the precision, An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i22.gifd, recall, An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i23.gifd, and F1 measure, F1d are computed. The average of the D Fl measures for the collection, F1, ..., FD, is calculated.

Three different models were assessed by determining their respective retrieval capability on the same set of relevant documents present in the collection of D = 5, 225 CGC documents (an LDA model, a mixture of unigrams model, and a model which ranks documents randomly). The common subject matter of the relevant documents was genes implicated in extending or shortening the life span of C. elegans. This set of ND = 842 aging-related documents was created by identifying CGC items in which the Genes record (Figure (Figure1)1) refers to one or more of the genes listed in Table Table11 that are known to modify life span.

A K = 50 topic LDA model was trained using all D CGC documents. Given this model, the symmetrized KL divergence between an aging-related document posterior (γq) and every CGC document posterior (γt) was computed using Equations 2 and 3. These D topic-space pairwise similarity scores, An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i1.gif(q, t1), ..., An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i1.gif(q, tD), were used to rank the CGC documents from most (highest score) to least similar. This process was repeated for each relevant document in turn, q1, ..., An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i25.gif, resulting in ND rankings of the CGC documents. Let Pi be the P top-ranked CGC documents for query qi. The number of known aging-related documents in this list was determined and used to calculate precision and recall. The average precision (average recall) is the mean of the precision (recall) value computed for each list P1,..., An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i26.gif. A series of such average precision and average recall values were computed by varying the number of top-ranked documents used to create lists, P = 1, ..., D. A PR curve was constructed from these D pairs of average precision and average recall values. The average F1 measure is the mean of the F1 measure computed for each of the ND relevant documents.

A K = 50 topic mixture of unigrams model was trained using all D CGC documents. Given this model, the symmetrized KL divergence between an aging-related document wq and a CGC document wt was computed using

In the mixture model, the divergence is between posteriors of the single latent topic rather than the vector of latent topic proportions as is the case with LDA. The component KL term between documents wi and wj is calculated using

where p(z|wi) and p(z|wj)are the posterior probabilities of class z given documents wi and wj respectively. The mixture model pairwise similarity scores, An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i29.gif(wq, An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i30.gif),..., An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i29.gif(wq, An external file that holds a picture, illustration, etc.
Object name is 1471-2105-7-250-i31.gif) were used to rank the CGC documents from most (highest score) to least similar and the procedure repeated for each of the ND relevant documents. The resultant ND rankings were used to construct a PR curve and calculate average F1 as described above for the LDA model.

A random model was created by randomly ordering the D CGC documents and repeating this process ND times. These ND "rankings" of the D documents in the collection were used to construct a PR curve and calculate averate F1 as described above for the LDA model.

Authors' contributions

DMB, MIJ, and ISM conceived and designed the study, analyzed the results, and wrote the manuscript. DMB formulated algorithms, wrote the software, and performed the experiments. KF wrote the software needed for the Gene Ontology-based analysis of LDA topics. All authors approved the manuscript.

Supplementary Material

Additional File 1:

Results for each of the LDA topics specified by a 50-topic model estimated from a corpus of 5,225 documents and a 28,971 word vocabulary.

Acknowledgements

This work was supported by the National Science Foundation, Office of Naval Research, National Institute on Aging, National Institute of Envirnomental Health Sciences, U.S. Department of Energy (OBER) and California Breast Cancer Research Program. DMB was additionally supported by a Fellowship from the Microsoft Corporation.

References

  • Entrez Gene. http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=gene
  • Online Mendelian Inheritance in Man (OMIM) http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=OMIM
  • Gene Ontology (GO) http://www.geneontology.org/
  • BLAST. http://www.ncbi.nlm.nih.gov/BLAST/
  • Ensembl. http://www.ensembl.org
  • UCSC Genome Browser. http://genome.ucsc.edu
  • Gene Expression Omnibus (GEO) http://www.ncbi.nlm.nih.gov/geo/
  • MacCallum R, Kelley R, Sternberg M. SAWTED: Structure Assignment With Text Description – Enhanced detection of remote homologues with automated SWISS-PROT annotation comparisons. Bioinformatics. 2000;7:125–129. [PubMed]
  • Jenssen T, Laegreid A, Komorowski J, Hovig E. A literature network of human genes for high-throughput analysis of gene expression. Nature Genetics. 2001;7:21–28. [PubMed]
  • Raychaudhuri S, Chang J, Imam F, Altman R. The computational analysis of scientific literature to define and recognize gene expression clusters. Nucleic Acids Research. 2003;7:4553–4560. [PMC free article] [PubMed]
  • Shatkay H, Feldman R. Mining the biomedical literature in the genomic era: an overview. J Comput Biol. 2003;7:821–855. [PubMed]
  • Hirschman L, Park J, Tsuji J, Wong L, Wu C. Accomplishments and challenges in literature data mining for biology. Bioinformatics. 2002;7:1553–1561. [PubMed]
  • Yandell M, Majoros W. Genomics and natural language processing. Nature Reviews Genetics. 2002;7:601–610. [PubMed]
  • Manning C, Schütze H. Foundations of Statistical Natural Language Processing. Cambridge, MA: MIT Press; 1999.
  • BioNLP. http://www.bionlp.org
  • Textpresso. http://www.textpresso.org/
  • Telemakus. http://www.telemakus.net/
  • Libbus B, Kilicoglu H, Rindflesch T, Mork J, Aronson A. BioLink 2004: Linking Biological Literature, Ontologies and Databases. Association for Computational Linguistics; 2004. Using Natural Language Processing, LocusLink and the Gene Ontology to Compare OMIM to MEDLINE; pp. 69–76.
  • Korbel J, Doerks T, Jensen L, Perez-Iratxeta C, Kaczanowski S, Hooper S, Andrade M, Bork P. Systematic association of genes to phenotypes by genome and literature mining. PLoS Biol. 2005;7(5):e134. [PMC free article] [PubMed]
  • Baeza-Yates R, Ribeiro-Neto B. Modern Information Retrieval. New York: ACM Press; 1999.
  • Alter O, Brown P, Botstein D. Singular value decomposition for genome-wide expression data processing and modeling. Proc Natl Acad Sci. 2000;7:10101–10106. [PMC free article] [PubMed]
  • Blei DM, Ng AY, Jordan MI. Latent Dirichlet Allocation. Journal of Machine Learning Research. 2003;7:993–1022.
  • Blei D, Jordan M. Proceedings of the 26th annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM Press; 2003. Modeling annotated data; pp. 127–134.
  • Griffiths T, Steyvers M. Finding scientific topics. Proc Natl Acad Sci. 2004;7:5228–5235. [PMC free article] [PubMed]
  • Wormbase. http://www.wormbase.org
  • Rothman J. Aging: from radiant youth to an abrupt end. Current Biology. 2002;7:R239–R241. [PubMed]
  • Kurz D, Hong Y, Trivier E, Huang H, Decary S, Hong Z, Luscher T, Erusalimsky J. Fibroblast Growth Factor-2, But Not Vascular Endothelial Growth Factor, Upregulates Telomerase Activity in Human Endothelial Cells. Arterioscler Thromb Vasc Biol. 2003;7:748–754. [PubMed]
  • Bissell M, Radisky D. Putting tumours in context. Nat Rev Cancer. 2001;7:46–54. [PMC free article] [PubMed]
  • Askree S, Yehuda T, Smolikov S, Gurevich R, Hawk J, Coker C, Krauskopf A, Kupiec M, McEachern M. A genome-wide screen for Saccharomyces cerevisiae deletion mutants that affect telomere length. Proc Natl Acad Sci. 2004;7:8658–8663. [PMC free article] [PubMed]
  • Rog O, Smolikov S, Krauskopf A, Kupiec M. The yeast VPS genes affect telomere length regulation. Current Genetics. 2005;7:18–28. [PubMed]
  • Durbin R, Eddy S, Krogh A, Mitchison G. Biological Sequence Analysis: Probabilistic Models of Proteins and Nucleic Acids. Cambridge: Cambridge University Press; 1998.
  • Bateman A, Birney E, Durbin R, Eddy S, Howe K, Sonnhammer E. The Pfam protein families database. Nucleic Acids Research. 2000;7:263–266. [PMC free article] [PubMed]
  • Blei D, Griffiths T, Jordan M, Tenenbaum J. Neural Information Processing Systems. Vol. 7. MIT Press, Cambridge MA; 2003. Hierarchical topic models and the nested Chinese restaurant process.
  • Jordan M. Graphical models. Statistical Science. 2004;7:140–155.
  • Caenorhabiditis Genetic Center Bibliography. http://elegans.swmed.edu/wli/cgcbib
  • C implementation of LDA. http://www.cs.princeton.edu/~blei/lda-c
  • Griffiths T, Steyvers M. A probabilistic approach to semantic representation. Proceedings of the 24th Annual Conference of the Cognitive Science Society. 2002.
  • Minka T, Lafferty J. Expectation-propagation for the generative aspect model. Uncertainty in Artificial Intelligence (UAI) 2002.
  • Nigam K, McCallum A, Thrun S, Mitchell T. Text classification from labeled and unlabeled documents using EM. Machine Learning. 2000;7:103–134.
  • Ponte J, Croft B. A Language Modeling Approach to Information Retrieval. ACM SIGIR 1998. 1998. pp. 275–281.
  • Moler E, Chow M, Mian I. Analysis of molecular profile data using generative and discriminative methods. Physiological Genomics. 2000;7:109–126. [PubMed]
  • Moler E, Radisky D, Mian I. Integrating naïve Bayes models and external knowledge to examine copper and iron homeostasis in Saccharomyces cerevisiae. Physiological Genomics. 2000;7:127–135. [PubMed]
  • Bhattacharjee A, Richards W, Staunton J, Li C, Monti S, Vasa P, Ladd C, Beheshti J, Bueno R, Gillette M, Loda M, Weber G, Mark E, Lander E, Wong W, Johnson B, Golub T, Sugarbaker D, Meyerson M. Classification of human lung carcinomas by mRNA expression profiling reveals distinct adenocarcinoma subclasses. Proc Natl Acad Sci. 2001;7:13790–13795. [PMC free article] [PubMed]
  • Teh YW, Jordan MI, Beal MJ, Blei DM. Hierarchical Dirichlet processes. JAMA (in press)

Articles from BMC Bioinformatics are provided here courtesy of BioMed Central
PubReader format: click here to try

Formats:

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...

Links

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...