Logo of bioinfoLink to Publisher's site
Bioinformatics. Sep 1, 2008; 24(17): 1963–1965.
Published online Jun 25, 2008. doi:  10.1093/bioinformatics/btn319
PMCID: PMC2732216

Comparing simulation results of SBML capable simulators

Abstract

Motivation: Simulations are an essential tool when analyzing biochemical networks. Researchers and developers seeking to refine simulation tools or develop new ones would benefit greatly from being able to compare their simulation results.

Summary: We present an approach to compare simulation results between several SBML capable simulators and provide a website for the community to share simulation results.

Availability: The website with simulation results and additional material can be found under: http://sys-bio.org/sbwWiki/compare. The software used to generate the simulation results is available on the website for download.

Contact: fbergman@u.washington.edu

1 INTRODUCTION

The Systems Biology Markup Language (SBML) (Hucka et al., 2003) is a format for storing information about biochemical kinetic models and is currently supported by over eighty simulation tools (www.sbml.org). However, these simulation tools have different strengths and weaknesses that are not apparent to users or sometimes even the maintainers of these packages. As a result, we see many groups developing their own simulators supporting a subset of the SBML specification, rather than relying on existing ones. Having a set of simulation results for a well curated set of models, representing a high percentage of the features provided by the SBML specification, would provide developers of simulation packages the ability to compare and test their efforts. As no authoritative set of simulation results exists, we describe a strategy that allows us to compare simulation results from a set of simulators for a given model.

2 METHODS

The BioModels Database (Le Novére et al., 2006) is an annotated resource of quantitative biomedical models. The database includes both non-curated as well as curated models. The 9th release of the database includes one hundred and fifty curated models. These models have been carefully curated to ensure that they reproduce the figures from the publications from which the models are derived. The curated models in the BioModels Database cover a wide range of features of the SBML language and are therefore an optimal choice as a base set of models for simulator comparison.

We propose to compare the simulation runs of the curated models across multiple deterministic simulators to determine how well the results are correlated. A poor correlation can be used as indicator for developers of simulation packages to re-test their simulator. Towards this aim we wrapped a set of widely used deterministic simulation packages for use in a common interface. Simulators tested initially include: COPASI (Hoops et al., 2006), Jarnac (Sauro, 2000), JSim (Raymond et al., 2003), Oscill8 (Conrad, 2006), RoadRunner (Bergmann and Sauro, 2006) and SBML ODE Solver (Machné et al., 2006). This process was largely possible through the use of the Systems Biology Workbench (SBW) (Bergmann and Sauro, 2006), a resource sharing framework that allows applications to share functionality with each other. Jarnac and Roadrunner are already included with an SBW installation. Emery Conrad enabled Ocill8 for use in the SBW framework. For the comparison we wrapped COPASI and SBML ODE Solver to be available for use within SBW.

With this infrastructure in place, we simulated all 150 BioModels from time 0 to 10s with 1000 output steps, storing files with the simulation results along with error messages provided by the simulators. This time span was empirically chosen to ensure that all of the SBML features encoded could be compared while at the same time minimizing the risk of numerical instabilities. In some cases, some features may be missed due to the short time span used in the runs; however there will be other models that test the features successfully. The output timescale is divided into a thousand equidistant output steps to be able to capture results of fast changing models. The output steps do not represent the steps used internally by the simulators. All modern simulators use adaptive step size methods to deal with stiff equations (Gear, 1971) and as a result simulators are free to choose the internal number of steps they need to solve the problem.

In order to re-run the simulations for each simulator, we encapsulated this step in a command line tool. The simulation results were then compared between all pairs of simulators and absolute and relative errors calculated. As this preprocessing step has to be repeated each time the simulation results changed, we developed a second command line tool for this process. These datasets were then processed in a viewer application. For each model, the viewer tests which simulator returned results and what output columns the simulation results include. This information was then used to generate graphs detailing the simulation in the specified interval. The viewer can also compile all results into a static web site. The static web site provides a good overview over the obtained simulation results. For a more interactive analysis an online viewer application has been developed, which allows one to dynamically display the results based on a researcher's interest. For example, the number of simulators can be changed, the models filtered by SBML features of interest (i.e. discrete events, SBML rules) and by selected simulators, and finally the resulting graphs can be customized.

For the implementation of the command line tools, the standalone viewer and the online application we use the C# programming language. The Mono project (Novell Inc., 2005) also enabled us to run the wrappers, the comparison tools and the viewer on Linux, OS X and Windows operating systems.

3 RESULTS

Figure 1 shows the simulation results obtained for BioModel #24. Six out of 12 simulation packages returned results for this model. However, only two of them seem to agree on a specific behavior. This illustrates the problem we are trying to solve. As no authoritative result set exists, it is hard to devise a metric based on the simulation results, that would tell us whether a given simulation result is ‘correct’ or not. For this reason, we provide visual representations of the time courses and differences. We also provide an agreement metric that tells us how well simulators returning simulation results agree with each other. In this calculation, the model is seen as a dynamical system consisting of a mix of continuous and discrete time state equations, where state variables describe the current state of the model (Dorf and Bishop, 2007). For the SBML models, state variables include floating species (i.e. an SBML species element without a boundary condition) or any other variable that changes due to an SBML rule, or SBML event.

Fig. 1.
Simulation Results for BioModel #24 (Left) The viewer application, displaying the simulation results (lower panel) and differences between them (upper panel) (Right) An online viewer application is available to view all results customized to a researcher's ...

To determine the agreement metric we proceed as follows: For each model we calculate the agreement of all pairs of simulators returning results for the model. We begin by determining the sub set of state variables returned by both simulators. For each of the state variables, we then determine the relative error. As the relative error is not symmetric we calculate it for simulator1 versus simulator2 and vice versa. This computation results in two vectors of relative errors for all shared state variables. For each of those vectors we take the second highest value as the relative error for the state variable (i.e. high single point failures are not penalized). The overall difference between the simulation results of the two simulators is then given by the averaged maximal error over all state variables.

This metric is unbiased with respect to both simulation results (as compared to the relative error alone) and does not penalize large single point failures. Ignoring large single point failures, as they occur, for example, when a discrete event is assigned improves the metric. With respect to the example in Figure 1 the metric tells us that two out of six simulators agree with each other. In Figure 2 this value appears in the 1% slice 30–40%. (Please note that the metric draws no conclusions as to the correctness of the results).

Fig. 2.
Percent agreement between simulation results: only 10% of the simulation results have an agreement factor smaller than <70%. (100% agreement: 63%, 90–99% agreement: 15%, 80–90% agreement: 12{%).

Using this approach it is easy to integrate further simulation results into the comparison and we have requested other groups to also provide their simulation results. Bruce Shapiro, the author of MathSBML (Shapiro et al., 2004) provided the first third party results. Further, simulation results were contributed for the simulators BioUML, BioDyn, JWS Online, SBToolBox2 and VCell. In future additional results will be available, for example, from JigCell. More information about these simulation packages can be found online at the SBML Software Guide (SBML.org., 2008) or the community maintained SBML Feature Matrix (VCell Web, 2008).

These results have already proved to be a valuable asset in discovering flaws with several simulation packages that have since been resolved. Of the simulators that generated results, there was complete agreement on ~63% of the models. The overall result can be found in Figure 2, only 10% of the simulation results are below 70% agreement.

4 OUTLOOK

We will continue updating the comparison report with every new release of the BioModels Database and update the simulation results for a given simulator, either when results are submitted or when a new version of a wrapped simulator is available. We are also working on providing more statistical analysis of the simulation and difference sets on our website.

Currently we only generate the simulation results for 10 s, with a resolution of 1000 output time points. Furthermore, the comparison right now is estimating the state variables to compare based on either SBML floating species or SBML elements controlled by discrete events or rules. This causes many of the comparison graphs to become too complex for a human observer and thus should be simplified. In order to solve these problems, the specification and adoption of the MIASE Ontology (Köhn and Le Novére, 2007) will be of great help. This ontology is meant to describe the minimum information needed to repeat a simulation experiment. Similar efforts are also underway in the CellML community with the CellML simulation metadata specification (CellML.org., 2008).

Another possible extension involves the comparison of stochastic simulators. Currently only deterministic simulators participate in this comparison project. However, some of the BioModels are intended to be simulated stochastically. Thus an extension to stochastic simulators could be envisioned. This comparison effort would deal with comparing the distribution of simulation trajectories, as described by Wilkinson (Evans et al., 2007).

ACKNOWLEDGEMENTS

We wish to thank the submitters for providing simulation results, as they drive this project and help elucidate further inconsistencies.

Funding: This work was supported by generous grants from NIH (1R01GM081070-01) and the US DOE GTL Program.

Conflict of Interest: none declared.

References

  • Bergmann F, Sauro H. Proceedings of the 38th Conference on Winter Simulation. Winter Simulation Conference, Monterey. 2006. SBW –a modular framework for systems biology. pp. 1637–1645.
  • CellML.org. CellML Simulation Metadata Specification. [Last accessed date June 20, 2008];2008 Available from http://www.cellml.org/specifications/metadata/simulations/
  • Conrad E. [Last accessed date November 30, 2007];2006 Oscill8. Available from http://oscill8.sf.net/
  • Dorf R, Bishop R. Modern Control Systems. NJ, USA: Prentice-Hall, Inc., Upper Saddle River; 2007.
  • Evans T, et al. The SBML discrete stochastic models test suite. Bioinformatics. 2007;24:285–286. [PubMed]
  • Gear C. Numerical Initial Value Problems in Ordinary Differential Equations. NJ: Prentice Hall, Englewood Cliffs; 1971.
  • Hoops S, et al. COPASI—a COmplex PAthway SImulator. Bioinformatics. 2006;22:3067–3074. [PubMed]
  • Hucka M, et al. The systems biology markup language (SBML): a medium for representation and exchange of biochemical network models. Bioinformatics. 2003;19:524–531. [PubMed]
  • Köhn D, Le Novére N. MIASE – Minimum Information About a Simulation Experiment. [Last accessed date November 21, 2007];2007 Available from http://www.ebi.ac.uk/compneur-srv/miase/
  • Le Novère N, et al. BioModels Database: a free, centralized database of curated, published, quantitative kinetic models of biochemical and cellular systems. Nucleic Acids Res. 2006;34:689–691. [PMC free article] [PubMed]
  • Machné R, et al. SBML ODE Solver Library: a command-line tool and library for numerical analysis of reaction networks. Bioinformatics. 2006;22:1406–1407. [PubMed]
  • Novell Inc. The Mono Project. [Last accessed date August 31, 2007];2005 Available from http://www.mono-project.com/
  • Raymond G, et al. JSIM: free software package for teaching physiological modeling and research. Exper Biol. 2003;280:102.
  • Sauro H. Proceedings of the Animating the Cellular Map 9th International BioThermoKinetics Meeting. Stellenbosch, South Africa: Stellenbosch University Press; 2000. Jarnac: a system for interactive metabolic analysis. pp. 221–228.
  • SBML.org. SBML Software Guide. [Last accessed date May 30, 2008];2008 Available from http://sbml.org/SBML_Software_Guide.
  • Shapiro B, et al. MathSBML: a package for manipulating SBML-based biological models. Bioinformatics. 2004;20:2829–2831. [PMC free article] [PubMed]
  • VCell Web (2008) SBMLFeatureToolMatrix. [Last accessed date May 30, 2008]; Available from http://ntcnp.org/twiki/bin/view/VCell/SBMLFeatureToolMatrix.

Articles from Bioinformatics are provided here courtesy of Oxford University Press
PubReader format: click here to try

Formats:

Related citations in PubMed

See reviews...See all...

Cited by other articles in PMC

See all...

Links

  • PubMed
    PubMed
    PubMed citations for these articles

Recent Activity

Your browsing activity is empty.

Activity recording is turned off.

Turn recording back on

See more...