Getting the preformatted database files

Preformatted BLAST database files offer several advantages over the FASTA files:

• The preformatted databases are broken into smaller volumes and therefore can be downloaded more readily with fewer errors
• A convenient Perl script (update_blastdb.pl found in the bin directory of a locally installed blast+ package) is available to simplify the download of these preformatted databases
• Preformatted database files remove the makeblastdb formatting steps, and saves valuable processing time and diskspace
• Taxonomic information is encoded within the preformatted databases and can be used to limit the scope of a blast search, and sequence retrieval, and scientific name addition through the included taxdb files
• Sequences in FASTA format can be generated easily from the preformatted databases using the blastdbcmd utility when needed

Preformatted databases must be downloaded in binary mode, downloading through the update_blastdb.pl script is recommeded. An example command line for getting the preformatted refseq_rna nucleotide database and the session output are given below.

$ perl ../bin/blast+/update_blastdb.pl --passive --decompress refseq_rna
Connected to NCBI
Downloading refseq_rna (7 volumes) ...
Downloading refseq_rna.00.tar.gz... [OK]
Downloading refseq_rna.01.tar.gz... [OK]
Downloading refseq_rna.02.tar.gz... [OK]
Downloading refseq_rna.03.tar.gz... [OK]
Downloading refseq_rna.04.tar.gz... [OK]
Downloading refseq_rna.05.tar.gz... [OK]
Downloading refseq_rna.06.tar.gz... [OK]
Decompressing refseq_rna.00.tar.gz ... [OK]
Decompressing refseq_rna.01.tar.gz ... [OK]
Decompressing refseq_rna.02.tar.gz ... [OK]
Decompressing refseq_rna.03.tar.gz ... [OK]
Decompressing refseq_rna.04.tar.gz ... [OK]
Decompressing refseq_rna.05.tar.gz ... [OK]
Decompressing refseq_rna.06.tar.gz ... [OK]

The complete options of this script (obtained using specific option "--help") are shown below.

$ perl update_blastdb.pl --help
NAME
    update_blastdb.pl - Download pre-formatted BLAST databases

SYNOPSIS
    update_blastdb.pl [options] blastdb ...

OPTIONS
    --decompress
      Downloads, decompresses the archives in the current working directory,
      and deletes the downloaded archive to save disk space, while
      preserving the archive checksum files (default: false).

    --showall
      Show all available pre-formatted BLAST databases (default: false). The
      output of this option lists the database names which should be used
      when requesting downloads or updates using this script.

      It accepts the optional arguments: 'tsv' and 'pretty' to produce
      tab-separated values and a human-readable format respectively. These
      parameters elicit the display of additional metadata if this is
      available to the program. This metadata is displayed in columnar
      format; the columns represent:

      name, description, size in gigabytes, date of last update (YYYY-MM-DD
      format).

    --blastdb_version
      Specify which BLAST database version to download (default: 4).
      Supported values: 4, 5

    --passive
      Use passive FTP, useful when behind a firewall or working in the cloud
      (default: true). To disable passive FTP, configure this option as
      follows: --passive no

    --timeout
      Timeout on connection to NCBI (default: 120 seconds).

    --force
      Force download even if there is a archive already on local directory
      (default: false).

    --verbose
      Increment verbosity level (default: 1). Repeat this option multiple
      times to increase the verbosity level (maximum 2).

    --quiet
      Produce no output (default: false). Overrides the --verbose option.

    --version
      Prints this script's version. Overrides all other options.

    --num_cores
      Sets the number of cores to utilize to perform downloads in parallel
      when data comes from GCS. Defaults to all cores (Linux and macos
      only).

DESCRIPTION
    This script will download the pre-formatted BLAST databases requested in
    the command line from the NCBI ftp site.

EXIT CODES
    This script returns 0 on successful operations that result in no
    downloads, 1 on successful operations that downloaded files, and 2 on
    errors.

BUGS
    Please report them to <blast-help@ncbi.nlm.nih.gov>

COPYRIGHT
    See PUBLIC DOMAIN NOTICE included at the top of this script.

Using the preformatted BLAST database files

The --decompress option of updated_blastdb.pl automatically decompresses and extract the archives of the requested database files. When manually downloading preformatted databases, those compressed archives must be downloaded in binary format using the passive mode, then inflated with gunzip or other decompress utilities. The working database files can then be extracted out of the resulting tar archive using tar program in Unix/Linux or WinZip and StuffIt Expander on Windows and Macintosh platforms, respectively.

Large databases are formatted in multiple gigbytes-sized volumes, which are named using the "name.##.tar.gz" convention. To reconstitute a given multi-volume database, all volumes with the same base database name are required. A database alias file, with ".pal" extension for protein or ".nal" extension for nucleotide, is provided to tie the volumes together. The database can be called using the base database name. For example, binary programs from the blast+ package can call the nt database using the command line option of "-db nt" option argument.

For proper setup of standalone blast+, it is recommended that database files be stored in a centralized directory, with the path to this directory be encoded by the BLASTDB variable. Details are available in the setup document for Windows and Mac/Linux/Unix.

Sequence files under the "/db/FASTA/" subdirectory

This subdirectory contains sequence files in the FASTA format. With preformatted databases readily available, only a few commonly used databases are available in this format.

For local BLAST searches, the recommendation is to use the preformatted version given in the parent directory. For those without preformatted counterparts, the FASTA sequence file first need to be inflated using gunzip or other comparable utilities, the resulting file can then be formatted by makeblastdb from the blast+ package. Example command lines for formatting igSeqNt and igSeqProt are given below.

$ makeblastdb –in swissprot –dbtype prot –parse_seqids 
$ makeblastdb –in nt –dbtype nucl –parse_seqids 

For vector screening needs, get the FASTA sequences from this ftp directory and formatted them using makeblastdb:

https://ftp.ncbi.nlm.nih.gov/pub/UniVec/ 

Chromosome entries are available in the refseq_euk_rep_genomes and refseq_prok_rep_genomes databases. Organism-specific sequences are available in FASTA format under the genomes FTP directory:

https://ftp.ncbi.nlm.nih.gov/pub/genomes/all/ https://ftp.ncbi.nlm.nih.gov/pub/factsheets/Factsheet_Assembly.pdf   

The NCBI Datasets tool is another way to get the genomic data from NCBI. Please refer to this page for more information: https://www.ncbi.nlm.nih.gov/datasets

Database files under the "/db/v4/" subdirectory

This subdirectory contains the preformatted blast dtabase files in the older version 4 format. Those databases will not be updated. They are meant as a stop-gap measure to ease the transition to version 5 of the databases. Content description for this subdirectory will be skipped.

Database update

In general, BLAST databases are updated daily. There is no established incremental update scheme due to sequence removal and update. It is recommended that databases be downloaded at regular intervals to keep the content of local copy current. The update_blastdb.pl script can help streamline this download process. If the original database.##.tar.gz files are kept, this utility can automatically check the time stamps to determine if file refreshing is required or not.

Faster download through Aspera plug-in is also possible (downloadable from the Aspera Soft site under the download tab). A web interface for the NCBI FTP site is at: https://www.ncbi.nlm.nih.gov/public/. Aspera’s commandline client ascp can be used to access Aspera indexed NCBI ftp site.

Contents of the "/blast/executables/LATEST/" subdirectory

This directory is a symbolic link pointing to the LATEST release of BLAST+ programs built from the NCBI C++ toolkit. Packages for commom platforms available in different formats are summarized in Table 5 below.

All non-source code archives or packages are equivalent. They contain a standard collection of standalone command line programs and accessory utilities for different platforms. Installation of the package enables local BLAST searches, custom database preparation from FASTA sequences, as well as sequence retrieval from existing databases formatted with the "-parse_seqids" argument.

Details on individual programs from the package as well as installation procedures are available for Windows and Mac/Linux/Unix.

Note that blast+ does not provide a separate client-server tool. That function is built into individual blast programs, i.e. blastn, blastp, blastx, tblastn and tblastx, and can be invoked using the "-remote" option. The "-remote" option enables remote search against databases at NCBI using NCBI’s computation resources. In addition, blast+ does not provide package equivalent to the decommissioned wwwblast package.

Contents of the "/blast/executables/blast+/" subdirectory

This subdirectory archives all releases of the blast+ package. Each release is under its own directory named using its version number. Available builds start at version 2.2.18, with verision 2.2.20 skipped. Optional version 5 database support began in release 2.8.0alpha.

Contents of the "/blast/executables/legacy.NOTSUPPORTED/" subdirectory

This subdirectory archives the releases of legacy blast package based on the NCBI C Toolkit. They are meant for historical references, NCBI no longer supports them. Each release is under its own directory with the version number as its name. Available builds start at version 2.0.7.

Packages with version number 2.2.10 or newer are packaged with a built-in directory structure to better organize the distributed contents.

Contents of the "/blast/executables/igblast/" subdirectory

This subdirectory archives the different releases of standalone igblat package (under the release subdirectory. The separate data files and database files required by the binary programs in this package are available in subdirectories separate from each releases.

Contents of the "/blast/executables/magicblast/" subdirectory

A subdirectory for the releases of a next generation sequence read mapper from NCBI. The LATEST directory maps to the current release’s directory. Refer to the README file under the subdirectory for more details. Details technical description of the package is at: https://ncbi.github.io/magicblast/.

Contents of the "/blast/executables/rmblast/" subdirectory

This subdirectory archives the repeat masker BLAST. Only two releases, 2.2.27 and 2.2.28, are available. Refer to the readme for more details.