Assessment of three-dimensional RNA structure prediction in CASP15

The prediction of RNA three-dimensional structures remains an unsolved problem. Here, we report assessments of RNA structure predictions in CASP15, the first CASP exercise that involved RNA structure modeling. Forty two predictor groups submitted models for at least one of twelve RNA-containing targets. These models were evaluated by the RNA-Puzzles organizers and, separately, by a CASP-recruited team using metrics (GDT, lDDT) and approaches (Z-score rankings) initially developed for assessment of proteins and generalized here for RNA assessment. The two assessments independently ranked the same predictor groups as first (AIchemy_RNA2), second (Chen), and third (RNAPolis and GeneSilico, tied); predictions from deep learning approaches were significantly worse than these top ranked groups, which did not use deep learning. Further analyses based on direct comparison of predicted models to cryogenic electron microscopy (cryo-EM) maps and X-ray diffraction data support these rankings. With the exception of two RNA-protein complexes, models submitted by CASP15 groups correctly predicted the global fold of the RNA targets. Comparisons of CASP15 submissions to designed RNA nanostructures as well as molecular replacement trials highlight the potential utility of current RNA modeling approaches for RNA nanotechnology and structural biology, respectively. Nevertheless, challenges remain in modeling fine details such as non-canonical pairs, in ranking among submitted models, and in prediction of multiple structures resolved by cryo-EM or crystallography.


This Supplemental Information file contains one Supplemental
Rank 1 st 2 nd 3 rd sum e 1 st 2 nd 3 rd sum e 1 st 2 nd 3 rd sum e 1 st 2 nd 3 rd sum e 1 st 2 nd 3 rd sum e 1 st 2 nd 3 rd sum e  e For each group, the number of times the submitted models were ranked 1st, 2nd, or 3rd were counted and the weighted sum indicated in SUM (with a weight of 3 for 1st, 2 for 2nd and 1 for 3rd ranks).Supplemental Figure 4. EM metrics for all targets.Scores for all models submitted for all targets are depicted.Models passing the RMSD filter are indicated with larger dots, colored if submitted by top performing groups and dark gray otherwise.The black line indicates the EM metric scores for the experimentally determined model.Supplemental Figure 6.Map-to-model analysis and RMSD filtering.(A) the Spearman correlation between metrics used in Z RNA , Z EM , as well as RMSD, computed for all models passing the RMSD filter.RMSD and clashscore were multiplied by -1 before calculating the correlation so that higher scores corresponded to better accuracy for all metrics.(B) The number of models which have RMSD to target less than the filter cutoff; these models were used in the final EM ranking.(C) An example of when EM metrics can be misleading.Reference structure in green, experimental map in grey, predicted models in blue.GDT-TS is reported as an example of model-to-model metric and SMOC as an example of map-to-model metrics.Green scores are seen as "good" while red are "poor" scores

Supplemental
reached for all the models submitted.b RMSD reached by the model ranked #1 by the group.c Same as a,b for Deformation Index (DI).d Same as a,b for Interaction Network Fidelity (INF).

Figure 2 .
How a good prediction can get high TM-score but low GDT_TS.For the R1138 model 4 submitted by AIchemy_RNA2 (R1138TS232_4) and the cryo-EM R1138 structure, the residue-residue distances between the C3' and C4' atoms were calculated using the superimposed coordinates determined by US-align and LGA, respectively; the traces are similar.The black dashed line represents the (soft) distance threshold used in US-align to compute TM-score ( = 13.59Å), which is set  0 based on the molecule length; for this 720-nucleotide target the value is large and most residues align  0 within the threshold, leading to a high TM-score for this target.In contrast, the gray lines indicate the threshold values used in GDT_TS (1 Å, 2 Å, 4 Å, and 8 Å).These threshold values do not change with molecule length and so do not take into account the increased flexibility expected for longer RNA molecules, leading to small GDT_TS values for this target.Supplemental Figure 3. Fits of CASP15 RNA models to EM maps.The best fitting (by Z EM ) predicted model (blue) fit into an experimental cryo-EM map for each target (gray).

Supplemental Figure 7 .
Comparison of groups' top model by GDT-TS and the model they selected as model #1.(A) For each group, the percent of targets they participated in where their best model by GDT-TS (out of up to five models submitted) was assigned as model #1.(B) For all targets and all groups, the percent difference in GDT-TS from model #1 to the top model for that group.The mean values over CASP groups in (A,B) are shown as black lines.For comparison, the gray bars in (A,B) mark the 95% confidence interval for values from random shuffling to select "model #1" (1,000 and 10,000 bootstraps respectively).Supplemental Figure 8.Molecular replacement analysis of all groups for R1117.LLG (top) and TFZ (bottom) are plotted in red with a horizontal line at 60/7 respectively representing the normal criterion for successful placement.Global Map CC is plotted in blue with a horizontal line at 0.2 representing agreement between the placed model and the solved crystallographic structure.
Table and six Supplemental Figures.