1.1 Scope

The Sequence Read Archive (SRA) at NCBI accepts primary sequencing data from so-called “next generation” sequencing platforms, including Roche 454^®, Illumina Illumina^®, Life Technologies SOLiD^®, Helicos Biosciences HeliScope^®, CompleteGenomics^®, Pacific Biosciences SMRT^®.

Sequencing data should be submitted to the SRA rather than the regular Trace Archive. The Trace Archive is intended as the repository of sequencing data from gel/capillary platforms (Applied Biosystems 370^® and 3730^® , Megabace, and Licor sequencers).

1.2 Related Documents

• NCBI Sequence Read Archive

• SRA XML Schema Documents

• Sequence Read Format

• Aspera Download Guide

• Submission Quick Start Guide

• SRA File Format Guide

1.3 Notices

Reference herein to any specific commercial products, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government, and shall not be used for advertising or product endorsement purposes.

2.1 Permanence

Accessions issued by the SRA are always maintained and never reused. If a desired record has been withdrawn, then a message to this effect will be displayed to anyone who tries to access it. If a record has been superseded by a successor record, this fact will be presented to anyone trying to access it. Only in rare cases where the record needs to be expunged from the archive will a user not be able to access it.

2.2 Authentication

Submissions are managed through secure channels. These channels include PDA, NIH level login through CIT, and FTP accounts secured by passwords. We will correspond with submitters via email about submission and curation issues, but we do not exchange data by email. At this time MyNCBI is used for authenticating to the SRA submission pages or accounts.

Please keep your PDA and file transfer accounts secure. Please do not reuse someone else’s accounts. Center accounts are provided for the convenience of automated pipelines and where multiple users need to manage submissions. The authentication information for such an account should be maintained securely by the Center. Accounts may be disabled or withdrawn after a long period of disuse in order to comply with NCBI security requirements.

2.3 Limitations

The Sequence Read Archive at NCBI is a public resource and the decision whether to submit data to this resource is the responsibility of the submitter. Prospective submitters should be aware of the following issues:

Never submit data without the permission of the principal investigator.

Most human data gathered from research subjects are under strict privacy controls and/or usage restrictions and must be handled with protections as determined by the research institution’s Institutional Review Board (IRB), the funding agencies, and the laws of the United States or the submitter’s home country. The dbGaP resource at NCBI may be a more appropriate broker for human sequencing data requiring controlled access due to these considerations. Data from whole genome, transcriptome, epigenome, and metagenome (which may include human contaminants) may fall into this category. Data gathered from human subjects, certain cell lines, and metagenomes may be covered as well.

Data submitted as part of a journal manuscript may have a publication embargo placed on it by the journal editors. The submitter can place a “hold until publish” restriction on the submission to the SRA as part of the submissions process.

Data that might relate to patents and intellectual property may be submitted to NCBI, but the submitter is responsible for ensuring that procedures and policies of his/her institution or company are observed.

Some environmental data gathered in the territory of certain countries, including territorial waters, may have sovereign legal restrictions on their use. NCBI cannot accept such data since NCBI is not able to enforce any usage restrictions..

Submitters must ensure that data obtained as part of a criminal investigation is free of any judicial restrictions on its use.

Submitters are responsible for obtaining all necessary permissions from the collecting institution for forensic and paleontological data.

The United States and many other countries have laws governing trade in endangered species. Please be aware that nucleotide material gathered from such samples may be subject to restrictions. Over-specific metadata accompanying submissions may also be inappropriate for when the samples are rare or endangered.

2.4 Modification

NCBI allows submitters to modify their records. Such requests must be formally entered using the SRA submission mechanisms. Informal requests by email will not be accepted. Only the center or individual that created the record can change it. Please write NCBI if you have changed affiliations and wish to update old records. This may require agreement from the original institution.

2.5 Curation

From time to time records deposited at the SRA must be updated with changes needed in order that the data continue to conform with the data model for the archive, to update data as it changes (for example finalizing publication information), to change data that are clearly wrong (for example correcting external references to other data or resources), and to add additional relevant metadata as they become available. NCBI will contact the submission owner on a best effort basis. The submission owner should maintain up to date contact information with NCBI to receive word of such changes.

Actual instrument data are not changed by NCBI. Only the submitter can make such modifications.

2.6 Availability

While NCBI tries to maintain maximum uptime of its servers on a 24x7 basis, no guarantee of availability is offered to users. Submissions that are interrupted by downtime may have to be restarted by the user.

Technical assistance is available on a limited basis during business hours USA Eastern Time. There is no guarantee for level of service regarding manual assistance.

4.1 Establish a NCBI Identity

Before interacting with NCBI, please obtain a personal identity account. This will allow you to make submissions, track results, change records now or later, and hold or release records. There are two kinds of NCBI identity sufficient to do business with SRA:

NCBI PDA - NCBI-created and managed account for primary data submitters. If you belong to a submitting Center and will play a role in monitoring and maintaining primary data submissions, please identify this fact through your account profile and also email SRA with this information.

NIH - For any NIH personnel who has credentials managed through CIT and can use their NIH identity to login

4.2 Establish a Center Name

Certain submitters might find it necessary to establish a more formal relationship for their sequencing center or lab. These are typically large groups that are planning to use an automated system to create submissions and transfer their files.

If your lab, center, or group has submitted in past, there might already be a center established. Please check here for your center or lab. ftp://ftp-trace.ncbi.nlm.nih.gov/sra/reports/Centers/centers.tab Please contact sra/at/ncbi.nlm.nih.gov to be added to an established Center’s user list or to provide information for creating a new Center.

To create a new Center, please provide the following information:

1.

suggested center abbreviation (16 char max)

2.

center name (full)

3.

center URL

4.

center mailing address (including country and postcode)

5.

phone number (main phone for center or lab)

6.

contact person (someone likely to remain at the location for an extended time)

7.

contact email (ideally a service account monitored by several people)

Please read section 5.3 Transmitting Data to NCBI to select a method of data transfer for your center.

5.1 Understanding Submission Modes

5.2 Packaging Data for Submission

Multiple data files may be packed into a tar archive, however each file within the tar archive must be individually specified in a run to be linked properly.

5.3 Transmitting Data to NCBI

You will have to transmit your run data files to NCBI. This cannot be done through the interactive submission tool. Run data files (SFF, SRF, etc) can be quite large. It is NOT necessary to compress files transmitted to NCBI but files compatible with either gzip or bzip2 can be processed.

ftp://sra:<password>@ftp-private.ncbi.nlm.nih.gov/

ftp: ftp-private.ncbi.nlm.nih.gov
login: mycentre_trc
passwd: !jXYZZ3@ce


 > cd short_read
 > put myfiles.tar.gz
 > quit

ascp -i <key file> -QT -l600m -k1 <file(s) to transfer> asp-sra@upload.ncbi.nlm.nih.gov:<directory>

ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIEAoQNz1WIxVOvdRL9fx
 … jVp9nc= rsa-key-20090113

../puttygen -O private -t rsa -b 1024 -o puttyprivate.ppk

../puttygen puttyprivate.ppk -O public-openssh -o publicssh.pub

ssh-keygen -f sra-rsakey

ascp -i <private key file> -QTr –k1 <file(s) to transfer> -l200M asp-<center>@upload.ncbi.nlm.nih.gov:test/

ascp -i <private key file> -QTr <file to transfer> -l 300M asp-<center>@upload.ncbi.nlm.nih.gov:incoming/

putty.exe -i <private key file> asp-<center>@upload.ncbi.nlm.nih.gov

 ls -l test #lists the content oftest subdir in long format
 ls -l incomimg #lists the content of incoming subdir in long format
 ls -l #lists the content of home directory in long format
 ls -lR #lists the content of all entries in home directory

rm incoming/badfile

ascp -i <private key file> -QTr <files to transfer> -l100M asp-<center>@upload.ncbi.nlm.nih.gov:test/

ascp: session open failed.
>> ascp: (remote) failed to initiate session, consult log.
>> Ssh error: SSH connection failure: 130.14.29.99:22 Server reported 
>> failure exit code 1

7.1 Preparing Run Data

The SRA is intended as a repository of data output by “primary analysis” phase of the sequencing platform: sequencing results in fasta form along with instrument data indicating probability of correctness for each basecall (qualities) and signal intensity measurements (intensities).

The Sequence Read Archive does NOT accept fasta only datasets due to the inability to evaluate the quality of such data.

All submitted data must be raw data received from the sequencing machine without any edits.

illumina2srf -R -P -N <run>:%l:%t: -n %x:%y 
-o <center_name>_<run>_<lane>.srf s_<lane>_*_seq.txt

illumina2srf -R -P -N <run>:%l:%t: -n %x:%y -2 <cycle>
-o <center_name>_<run>_<lane>.srf s_<lane>_*_seq.txt

8.1 BioProject Registration

Whole genome sequencing projects should be registered with the BioProject resource at NCBI before submitting to SRA. Please access the BioProject Submission Form to submit.

8.2 Taxonomy Registration

Most single organism genome and transcriptome sequencing projects need a Taxon Id to help specify the sample being sequenced. Please consult the Entrez Taxonomy resource to see whether your organism is represented, and request an entry to be created if not. The taxon id is needed for submission preparation.

8.3 Reference Fields and Namespaces

All the XML files can take either names (aliases) to identify dependencies. These names need only be unique throughout the submission. Eventually, the SRA will replace these names with actual accessions while preserving the referential integrity of the records.

8.4 Required Fields

Each XML file has certain required fields. The XML schema document these entries and most are self-explanatory. Decisions that the submitter should make include:

• Study type

• Whether a BioProject id exists for the study

• The center project name or id

• Whether a Taxonomy id exists for the sample

• Whether an anonymous id exists for the sample

• Sequencing platform used (for example, 454 GS 20 or 454 GS FLX)

• Library source, strategy, selection, layout if applicable, protocol

• Spot or cluster layout (use of adapters, linkers, bar codes, etc)

• Read processing selection

In addition, the submitter should think about the relationship between experiments and samples, runs and experiments, and whether to split any of these objects to represent distinct information

Finally, submitters may consider providing ancillary information, including links, Entrez links, and attribute tag-value pairs. These can be created for any of the five record types.

8.5 Preparing Submission Files

Aspects of the submission process pertaining to the submission itself have been broken out into their own XML descriptor. Contact information, transaction requests, exceptions, and file manifests can be listed here. Contacts should be provided for questions or problems pertaining to the particular submission.

A checksum should be computed for each run file delivered as part of the submission and entered into the submission.xml record. Please use the unix md5sum or equivalent utility. It is not necessary to provide checksums for the metadata xml files.

8.6 New Submission Protocol

Check your XML files for correctness with respect to the current published schema.

Check your XML files for completeness and referential integrity.

Verify checksums on run and analysis files.

Open FTP to the trace FTP site for your Center.

Change directory to ./short_read (FTP), or ./incoming (aspera)

Deposit the files using the ftp put or mput command (or ascp for aspera).

Confirm receipt of the submission in the SRA Tracking Page. Once processed, you will be able to download the submitted data through this page. Please let us know if you are under a tight publication deadline and we will try to accommodate your needs.

Please write to us at sra/at/ncbi.nlm.nih.gov with any questions about status or access.

9.1 Update Submissions

Submitters can update their records through the interactive submission tool. It is recommended that you revisit your submission in order to annotate it with publication links and additional sample information. Once loaded, certain aspects of a run record cannot be updated (please write to NCBI if you wish to do this for some reason).

Bulk updates can be performed on the Project, Sample, and Experiment objects by submitting replacement XML files for the affected objects. An update XML document must identify the target schema (study, experiment, sample, run, etc) using the MODIFY action that calls out a file of replacement XML. The alias of the submission xml document must correspond to an existing submission that covers the objects that are to change.

9.2 Hold Until Publish (HUP)

An essential feature of the SRA is the ability to hold a submission until a manuscript reporting on the research is accepted or released by a journal:

Hold until date - This is appropriate for scheduled release of a publication

Release - The dataset can be released immediately to the public.

The hold can expire its term, or the submitter may send a Release message to NCBI indicating that the submission can be released to the public. A Release message can apply to the entire SRA object, or individual objects within the SRA submission. Any dependent objects are implicitly released. For example, releasing a certain experiment has the effect of releasing all its runs as well.

9.3 Versioning

SRA submissions are not explicitly versioned. Rather, a complete change history is stored for metadata and any version of the metadata can be accessed. Content such as run and analysis data are never modified. If these must be changed then current ones are deprecated (Withdrawn) and replacements added.

9.4 Curation

From time to time NCBI needs to update metadata in order to correct mistakes, propagate changes in other resources (for example taxonomic changes), and edit information in order to comply with editing requirements, copyrights, and data release policies. These “curation” changes may occur without necessarily seeking the approval of the original submitter. Run and analysis data will never be changed in this way. Also, original titles, descriptions, and names will be preserved as much as possible.

9.5 Suppression

At submitter’s request, a certain record (submission, study, experiment, sample, run) can be suppressed.

Suppression simply marks the record as deprecated. Suppressed records are never actually deleted (except for technical reasons including for example a loading error). Suppressed records can still be accessed by accession, but the accession will be marked as having been suppressed. Suppressed records are not indexed, and copies of them are removed from download facilities.

Please write to NCBI if you wish to request suppression of a record or dataset.

10.1 Interactive Submission Tool

The Interactive Submission Tool can be used to monitor progress of submissions. There are two views:

Submission View – This view presents the SRA submissions and their component metadata objects (last updated submission is the most recent listed). Color coded boxes indicate how many components are at which state (green is loaded).

Tracking View – This view lists the files that have been received but not yet processed. Files may persist in this view if the file has not been identified as belonging to a run, or has failed to load, or whose checksum does not match that specified in the submission metadata (due to transmission or other error).

Please see the SRA Quick Start Guide for more details.

10.2 Accessions Report

The Accessions Report is a list of SRA metadata objects and their status. This report is a tab delimited file called SRA_Accessions. The fields are defined as follows:

Accession – Accession (SRX, SRR, etc) of the object at NCBI.

Submission – Submission accession (SRA) associated with the object.

Status – Status of the object in the archive:

Live – The object is indexed and available for retrieval

Suppressed – The object has been removed from indexing but can still be retrieved. This state usually reflects objects that have been superceded by successor objects.

Unpublished – The object has not been published.

Withdrawn – The object has been expunged from the Archive. This state reflects rare situations where data was inappropriately released and copies in the Archive must be completely removed.

Updated – ISO date of the last update of the object.

Published – ISO date of the initial publication (release) of the object, and when it appeared on the Archive public site.

Received – ISO date at which the Archive received the data from the submitter.

Type – The object’s document type, currently {SUBMISSION, STUDY, SAMPLE, EXPERIMENT, RUN, ANALYSIS}

Center – Short name for the submitting center.

Visibility – Whether the object has been archived at the open SRA no usage restrictions (public), or at the controlled access SRA (usage restrictions in place, the user must apply for access to the data). Note that visibility is orthogonal to the publication or embargo status of the data.

Alias – The submitter’s name for the object.

Md5sum – The MD5 checksum of the metadata object. This is computed in a canonical way, see below.

10.3 Metadata XML

The XML metadata for each submission in the dump are emitted in XML files annotated with the any accessions assigned during the submission process. For the daily increment, only those submissions that are new for that day are dumped. Each month, a comprehensive dump of all the live metadata in the Archive is produced. Users could seed their own local database of SRA metadata from this dump. The XML has certain portions of the public record removed from the dump, including submitter contact information and data load directives, which are not important to users of the Archive.

The location of these public archive records is:

http://trace.ncbi.nlm.nih.gov/Traces/sra/sra.cgi?cmd=show&f=faspftp_metadata&m=downloads&s=download_reports

In addition, each submitting center has the same data pertaining to their submissions (both public and released, and unreleased or suppressed) available from their aspera upload directory (aspera account holders only):

For open SRA

upload@ncbi.nlm.nih.gov/asp-mycenter/outgoing/Metadata

For protected SRA

gap-upload@ncbi.nlm.nih.gov/asp-mycenter/outgoing/Metadata

Use the ascp or ftp programs to retrieve the incremental or last comprehensive metadata package file.

The metadata dump can provide input to the “roundtrip” processing module for a submitters LIMS system. Newly submitted documents can be downloaded to obtain the accessions assigned by NCBI. Documents can be downloaded in order to compare with internal state to make sure that the version at NCBI is current. Documents can be inspected to see that certain modification operations succeeded.

10.4 How to compare metadata versions

SRA metadata are not versioned in an explicit, public way. Rather, metadata are tagged when they change in a substantive way. A checksum is used to record the current content. One can tell whether the content has changed by comparing the checksum to a previously computed value.

Accession Submission  Status    Md5sum
SRA000001    SRA000001    public        d703b0b98a686a84ff232b9967e3d55c
SRP000057    SRA000001    public        bdac682ff9dca87f158b3d327832ec66
SRR000289    SRA000001    public        2fc12909aff893cdc20c48c0aa875bdf
SRS000246    SRA000001    public        7bf577c49d282529bc5f35f63137e6c0
SRX000068    SRA000001    public        a474043e97911936fe49f06f7a301aa5
SRA000002    SRA000002    public        7810982f118198eaf207a351e1550aa9
SRP000058    SRA000002    public        4c5d2a1c8a7fca885a09d690e49a5d06
SRR000290    SRA000002    public        aed8942276489b55ab98e282725ee920
SRS000247    SRA000002    public        999486234ce2e7420e16f169c2a86578

The md5sum value is equivalent to putting an xmllint 'noblanks' version of the xml associated with an accession in a file by itself (without a line feed) and executing md5sum -b on that file. If there is no meta data difference, then no increment is generated for that center on that day.

On the 1st of every month, a complete meta data dump is created in addition to an incremental dump. The first meta data dump for a new center is both an incremental and complete dump.

Obtain a copy of the script used to get md5 values for each accession chunk in a meta data xml file:

wget ftp-trace.ncbi.nlm.nih.gov:/sra/utilities/getMetaMd5.pl

The usage is:

getMetaMd5.pl < meta xml file path (based on ending in .xml) >


       OR


getMetaMd5.pl < file containing list of meta xml file paths >


       OR


<list of meta xml file paths> | getMetaMd5.pl

So you can provide the path to a single .xml file, a file containing a list of paths to xml files, or pipe to it a list of paths to xml files.

The getMetaMd5.pl script runs on Linux and requires

• perl to be at /usr/local/bin/perl (version 5.8.3 or higher),

• xmllint in your executable path (libxml 20630 or higher),

• xsltproc in your executable path (version 10102 or higher),

• Digest::MD5, a perl library for calculating md5 sums, and,

• parseMeta.xsl, to be in the same directory as the getMetaMd5.pl executable.

10.5 How to view files you have uploaded to your aspera account

To access NCBI servers with limited shell access a submitter must use their secret key (usually used with ascp for file transfer).

For access from unix/linux/macos the secret key must be in openssh format. In this case ssh command is used and command line is as:

For open SRA account:

 ssh -i secretkey.openssh asp-YOURNAME@upload.ncbi.nlm.nih.gov

For protected SRA account:

 ssh -i secretkey.openssh asp-YOURNAME@gap-upload.ncbi.nlm.nih.gov

For similar access from windows the key must be in putty format. And the putty.exe command should be used. The command line is as:

For open SRA account:

 putty.exe -i secretkey.ppk asp-YOURNAME@upload.ncbi.nlm.nih.gov

For protected SRA account:

 putty.exe -i secretkey.ppk asp-YOURNAME@gap-upload.ncbi.nlm.nih.gov

This limited shell has aspsh> as a prompt and allows only few commands like ls,cp,mv,rm. The cd command is not allowed so you must use ls with directory name as an argument.

Examples:

aspsh> ls -l
total 240
drwxrwsr-x  2 5608 trace  4096 Dec  9  2008 analysis
drwxrwsr-x  2 5608 trace 65536 May  7 10:08 incoming
drwxrwsr-x  2 5608 trace  4096 Jul  1  2008 logs
drwxrwsr-x  3 5608 trace  4096 Apr 15  2009 outgoing
drwxrwsr-x  2 5608 trace  8192 Apr 27 20:04 test
drwxrwsr-x  2 5608 trace 12288 May  7 11:05 trash


aspsh> ls -l analysis
total 0


aspsh> ls -l incoming
total 15663023352
-rw-rw-r--  1 asp-bcm trace   16504539868 May  6 10:06 0083_20090930_2_SP_ANG_HSAP_NG_005sA_01003244491_4.srf
…