Distribution of AUC and sensitivity for all SeqFEATURE models listed in Table 2. (a) Distribution of model AUC. Most models have AUC greater than 0.8, with 47% having AUC >0.95 and a few poor performers less than 0.5. (b) Distribution of model sensitivity. We plot the sensitivity of each model at the default score cutoff of 99% specificity based on training data. Most models have a sensitivity greater than 0.6-0.7 at this cutoff, and many have a sensitivity greater than 0.8.

Example ROC curves, precision-recall curves, and Z-score distributions for SeqFEATURE models. A sample of performance plots for ADH_ZINC.2.HIS.ND1 (top), ASP_PROTEASE.4.ASP.OD1 (middle), and ZINC_FINGER_C2H2_1.9.HIS.ND1 (bottom) are shown, representing a model with excellent performance, good performance, and somewhat satisfactory performance, respectively. The leftmost plot in each row gives the ROC curve in red and random performance in blue, the middle plot shows the precision versus recall (sensitivity) curve, and the rightmost plot shows the distribution of scores for positive sites (red) and negative sites (blue) from training. Because there are many more negative sites than positive sites, the score distributions on the right are normalized to Z-scores.

Performance on PROSITE true positives, false positives, and false negative test sites. We show the (a) true positive (TP), (b) false negative (FN), and (c) false positive (FP) prediction rates for SeqFEATURE (at 95%, 99%, and 100% specificity) and PROSITE on test sites derived from the corresponding PROSITE patterns. The PROSITE values represent the maximum possible for each category. Not all patterns had a false negative or false positive test set. Most of SeqFEATURE's incorrect predictions at 95% and 99% specificity cutoffs arise from poor performance on a small subset of the patterns.

Sensitivity trends of SeqFEATURE, Gene3D, Pfam, and HMMPanther at low sequence identities. We compared the sensitivity of SeqFEATURE at three specificity cutoffs against the sensitivity of Gene3D, Pfam, and HMMPanther on test sets filtered for low sequence identity. We evaluated each method on the subset of the original test set that had less than the specified sequence identity to the training sets. As sequence identity decreases, the sequence-based methods show a clear trend towards lower sensitivity. In contrast, SeqFEATURE at all three cutoffs shows no such downward trend, indicating robust detection of function even when sequence identity is very low.

Summary performance on PROSITE true positives (TP), false positives (FP), and false negative (FN) test sites. We summarize total numbers of predicted true positives, false negatives, and false positives for PROSITE and SeqFEATURE at 100%, 99%, and 95% specificity cutoffs. SeqFEATURE (at the default 99% specificity cutoff) misses about 18% of the PROSITE true positives on average, but it also predicts 60% fewer false positives and 78% fewer false negatives than PROSITE. The three different specificity cutoffs also show tradeoffs in the numbers of true positives and false predictions made by SeqFEATURE, demonstrating that one can adjust the cutoff to fit desired performance. For example, one can attain a very high positive predictive value by using SeqFEATURE's 100% specificity cutoffs - although sensitivity decreases to about 50%, almost no false positive predictions are made.

Overview of the SeqFEATURE pipeline. SeqFEATURE forms training sets by (a) extracting sequence (one-dimensional) motifs from PROSITE and (b) identifying the annotated functional amino acids. We extract examples of the one-dimensional motif with known three-dimensional structure in the PDB and center FEATURE training sites on each functional atom of each functional amino acid annotated in the PROSITE pattern. We choose negative sites matched for atom density randomly from the PDB that do not contain the function. (c) FEATURE then creates a model of the sites by summarizing the biochemical and biophysical features found in concentric shells around the functional atom center. (d) The resulting three-dimensional fingerprint specifies the properties that are in relative abundance or paucity in the site, representing the model. (e) A protein of interest is converted into features and scored with the model using a naïve Bayes scoring function, and predictions are made using score cutoffs, which can be based on desired performance statistics. The scores are calibrated into Z-scores using the training set used to derive each model.

Local environments of SeqFEATURE predictions for three TargetDB structures with unknown function. Three predicted functional sites from TargetDB structures are shown compared to known examples of the predicted function. The predicted and known sites are shown in yellow, positively charge atoms (nitrogens) are shown in blue, and negatively charged atoms (oxygens) are shown in red. Carbons and secondary structure are shown in grey. All atoms within 7.5 angstroms of the site are shown. (a) The active site of a known zinc protease (1KAP) is shown to the left of a zinc protease site in 2EJQ predicted by SeqFEATURE. Note the presence of two histidine residues (one can be seen clearly above each site) and a number of negative charges distributed throughout both local environments. Note also the similarity in secondary structure. (b) The local structure of 1FI6 (left), which contains a known EF hand calcium-binding motif, is compared to SeqFEATURE's predicted calcium-binding site in 2OGF. Note the similar distribution of negative charges and closely matching loop structures. The calcium is visible as a brown sphere in 1FI6, surrounded by oxygen atoms. (c) 1K8U is another known EF hand containing protein, shown to the left of the uncharacterized protein structure 2OX6, for which SeqFEATURE predicts calcium-binding. These figures were created using VMD [43].