• 1. Introduction
• 2. Installation and setup
  • 2.1. Installation
  • 2.2. Firewall settings
• 3. Practical usage examples
  • 3.1 General nucleotide searches
  • 3.2 General protein searches
  • 3.3 Translated BLAST searches
    • 3.3.1 blastx
    • 3.3.2 tblastn
    • 3.3.3 tblastx
  • 3.4 Genome BLAST searches
• 4. Appendix: Parameters and their accepted values

1. Introduction

NCBI BLAST web server provides a convenient and user friendly way for individuals to search their queries against different public sequence databases. Even though this server can take multiple quereis and perform batch searches, true large scale batch searches may not go through if the input queries are long and the search settings are less stringent. In addition, the available databases for the web interface is somewaht limited. BLAST client provides a way to circumvent those limitations.

The BLAST client, or blastcl3, bypasses the web browser and interacts directly with the NCBI BLAST server that powers the NCBI web BLAST service (www.ncbi.nlm.nih.gov/BLAST/). It performs the batch search with multiple sequences by taking one query sequence at a time from the input file, formulating the search according to the settings of the command line parameters, and sending the search through the internet connection to NCBI BLAST server for processing. The program receives the search results from the BLAST server, in the format set in the search command line, and saves it to a local file specified. The program loops through all the queries in the input file until all are searched.

This program has no graphic user interface (GUI) and must be executed from command line under a terminal window. Users control the program through command line options. Detailed list of command line options are in Section 4. For usages and situation examples, see Section 3.

2. Installation and setup

NCBI provides the BLAST client as netblast-initialed archive, separate from that for standalone command line, blast-initialed, and standalone server blast, wwwblast-initialed. All of them can be found at:

ftp.ncbi.nlm.nih.gov/blast/executables/LATEST/

2.1 Installation

For Linux or Unix environment, installation is straight forward. One can place the archive in a desired directory and extract the archive using the following command line:

tar zxvf netblast-##-**.tar.gz

The resulted netblast-#.#.# directory contains bin, doc, and data subdirectories. The program, blastcl3, is under the bin subdirectory. The matrices BLAST needs for protein alignments are under the data subdirectory, while the doc subdirectory contains netblast.html and firewall.html with more information on configuration of blastcl3 behind firewalls.

The package for Windows can be extracted using WinZip. It does not have this directory structure.

2.2 Firewall settings

The setup for NCBI network clients has been greatly simplified. Users, not behind a firewall, can use the program after the extraction above. For those users behind a firewall, and already use Sequin or Entrez, or if your system administrator has already performed the setup, they should also be able to start performing searches after installation.

If the above are not the case, users will need to make sure that the following IP address/port combinations are open in the firewall configuration.

In addition to this, users will need to create an .ncbirc file to instruct blastcl3 how to make the connection to NCBI. This file should contain the information listed below and be placed in the home directory. For PC running Windows, the file is named ncbi.ini which should be placed under the windows directory.

[NCBI]
DATA=/home/johndoe/netblast-#.#.#/data

[CONN]
FIREWALL=TRUE

[NET_SERV]
SRV_CONN_MODE=SERVICE

We may encounter problems while using blastcl3. The most common cause for this problem is firewall configuration related. A representative error message generally would contain "[CONN_Open] Cannot open connection", "<<< Re-establishing NETBLAST Service >>>", or something in that order.

Adding the following two lines in the .ncbirc (or ncbi.ini) file will increase the timeout setting and generate more informative messages that are useful in debugging the problem:

TIMEOUT=300

DEBUG_PRINTOUT=DATA

Search related errors from NCBI BLAST server typically are accompanied by RID for the relevant searches. Those RIDs should be saved and sent to NCBI blast-help@ncbi.nlm.nih.gov for trouble shooting purposes.

As an alternative to blastcl3, NCBI BLAST web server also supports URLAPI, which uses URL encoded command to interact with Blast.cgi directly to "Put" search requests onto the BLAST server, or to "Get" search results from the the same server. For details on BLAST URLAPI, please refer to:

http://www.ncbi.nlm.nih.gov/staff/tao/URLAPI/new/

3. Practical usage examples

Before we get into the actual use, we need to discuss the format of the input query. The only query format blastcl3 accepts is FASTA. In this format, the query begins with a "greater than" sign (>) initialed definition line, or defline as it is commonly known. This defline contains a basic description of the sequence, such as its source, the gene it represents, or ways the sequence is identified. The defline terminates with a hard return. The actual sequence immediately follows the defline in one or more lines, each terminates with a hard return. Multiple query sequences should be concatenated together one after another. Sample query sequences are presented below for your reference.

>gi|4557757|ref|NP_000240.1| MutL protein homolog 1
MSFVAGVIRRLDETVVNRIAAGEVIQRPANAIKEMIENCLDAKSTSIQVIVKEGGLKLIQIQDNGTGIRK
EDLDIVCERFTTSKLQSFEDLASISTYGFRGEALASISHVAHVTITTKTADGKCAYRASYSDGKLKAPPK
PCAGNQGTQITVEDLFYNIATRRKALKNPSEEYGKILEVVGRYSVHNAGISFSVKKQGETVADVRTLPNA
STVDNIRSIFGNAVSRELIEIGCEDKTLAFKMNGYISNANYSVKKCIFLLFINHRLVESTSLRKAIETVY
AAYLPKNTHPFLYLSLEISPQNVDVNVHPTKHEVHFLHEESILERVQQHIESKLLGSNSSRMYFTQTLLP
GLAGPSGEMVKSTTSLTSSSTSGSSDKVYAHQMVRTDSREQKLDAFLQPLSKPLSSQPQAIVTEDKTDIS
SGRARQQDEEMLELPAPAEVAAKNQSLEGDTTKGTSEMSEKRGPTSSNPRKRHREDSDVEMVEDDSRKEM
TAACTPRRRIINLTSVLSLQEEINEQGHEVLREMLHNHSFVGCVNPQWALAQHQTKLYLLNTTKLSEELF
YQILIYDFANFGVLRLSEPAPLFDLAMLALDSPESGWTEEDGPKEGLAEYIVEFLKKKAEMLADYFSLEI
DEEGNLIGLPLLIDNYVPPLEGLPIFILRLATEVNWDEEKECFESLSKECAMFYSIRKQYISEESTLSGQ
QSEVPGSIPNSWKWTVEHIVYKALRSHILPPKHFTEDGNILQLANLPDLYKVFERC
>gi|68348711|ref|NP_001234.2| tumor necrosis factor receptor 8
MRVLLAALGLLFLGALRAFPQDRPFEDTCHGNPSHYYDKAVRRCCYRCPMGLFPTQQCPQRPTDCRKQCE
PDYYLDEADRCTACVTCSRDDLVEKTPCAWNSSRVCECRPGMFCSTSAVNSCARCFFHSVCPAGMIVKFP
GTAQKNTVCEPASPGVSPACASPENCKEPSSGTIPQAKPTPVSPATSSASTMPVRGGTRLAQEAASKLTR
APDSPSSVGRPSSDPGLSPTQPCPEGSGDCRKQCEPDYYLDEAGRCTACVSCSRDDLVEKTPCAWNSSRT
CECRPGMICATSATNSRARCVPYPICAAETVTKPQDMAEKDTTFEAPPLGTQPDCNPTPENGEAPASTSP
TQSLLVDSQASKTLPIPTSAPVALSSTGKPVLDAGPVLFWVILVLVVVVGSSAFLLCHRRACRKRIRQKL
HLCYPVQTSQPKLELVDSRPRRSSTQLRSGASVTEPVAEERGLMSQPLMETCHSVGAAYLESLPLQDASP
AGGPSSPRDLPEPRVSTEHTNNKIEKIYIMKADTVIVGTVKAELPEGRGLAGPAEPELEEELEADHTPHY
PEQETEPPLGSCSDVMLSVEEEGKEDPLPTAASGK

The program runs under a command or terminal window. On PC the command window can be launched using "Start ► Program ► Accessories ► Command Prompt". On Mac, the Terminal program icon usually is under the Utilities folder. Double click the grey icon will launch it.

In the terminal window, cd to the directory containing the blastcl3, then run the program from there. Typing "blastcl3 -" without quotes followed by a return should display the command line options on the screen. On Mac and Unix/Linux platform, type "./blastcl3 -" without quotes.

Since the list of available database has increased significantly, we have removed them from this document. Users can find the list from this file:

http://www.ncbi.nlm.nih.gov/staff/tao/URLAPI/remote_accessible_blastdblist.html

    3.1 General nucleotide searches

The primary use of nucleotide BLAST search is to identify the input query by finding if exact match(es) are present in the database, which can also be used to identify the genomic counterpart of an input mRNA sequence or vice versa. Another use is to search with primer pairs to identify the annealing target and possible secondary annealing sites.

For sequences from well studied non-prokaryote model organisms, a good approach is to search against the refseq_rna database with Entrez limit. Alternatively, search against nr with or without limit to the target organism can also offer good lead.

The following example command lines search the input query file new_seq.txt against either the refseq_rna or nr database and save the result in n_refm.out and n_nr.out, respectively.

blastcl3 -p blastn -i new_seq.txt -p blastn -d refseq_rna -o n_refm.out

blastcl3 -p blastn -i new_seq -p blastn -d nr -o n_nr.out

We can further adjust the setting to restrict the search to the mouse entries in those two databases by using entrez limit, invoke megablast algorithm, and use a lower expect value of 0.001. The last two settings increase the search stringency. The actual paramter settings in the command line are:

-u "mouse[organism]" -n T -e 0.001

For easy parsing of the BLAST search result, we can request the result be returned in either XML or "Hit Table" (tabular) format using "-m 7" or "-m 9" (without quotes) in the command line.

Seaching a genomic DNA against nucleotide database, we should invoke the repeat filter to mask the repeat region and prevent BLAST program from being inundated by spurious hits to those regions. The following two -F settings are for human and rodents.

-F "m L; R"

-F "R -d rodent.lib"

Other species-specific repeat filters are also available. The command line convention is:

-F "R -d repeat/taxid_repeat"

Refer to the end of Table 2 in the "Remotely Accessible BLAST Database List" for more information.

Combining these together, the following command line searches the n_seq input nucleotide query file against the human subset in the refseq_genomic database with low complexity and human repeat filter and megablast algorithm. The expect value cutoff is set to 2x10^-10 and the output is saved in refg.output. Note that command line should be in a single line, the wrapping is due to line length limit.

blastcl3 -i n_seq -p blastn -d refseq_genomic -u "human[orgn]" -n T -F "m L; R" 
-e 2e-10 -o refg.output

    3.2 General protein searches

A protein BLAST search can be used to identify the input query protein and its function through matching to other known proteins and their annotation. One such database is refseq_protein. The following command line searches protein sequences in my_query.txt against this database using blastp. The result is saved in my_output.txt.

blastcl3 -p blastp -i my_query.txt -d refseq_protein -o my_output.txt

For functional analysis of unknown proteins, a more effective way is to searching against cdd database since matches from cdd search will identify the conserved functional domain(s) present in the query. Defline and annotation from these matched domains will provide a better revelation of the query's function. The following command line does such a search against the cdd database, "-d cdd", using rpsblast, "-R T" and saves the result in my_output.txt:

blastcl3 -p blastp -R T -i my_query.txt -d cdd -o my_output.txt

Specific search against pdb database can be used to identify existing structures with matching sequences useful for structure modeling purposes. We do not support PSI-BLAST or PHI-BLAST searches through blastcl3.

    3.3 Translated BLAST searches

Translated searches can be very informative in revealing the possible function of a nucleotide query since the search and alignment is performed at the protein level, which is more sensitive, with higher level of conservation, and biologically relevant. Translated search for protein query is an effective way to find unidentified homologs/paralogs. We can divide the translated searches in to three types based on whether the translation is done for the input, the database or both.

         3.3.1. blastx: translation of the the nucleotide queries

This program searches a nucleotide query against a protein database. It translates the query in all six frames first and then searches the translated protein products against the target protein database. It is useful in identifying the protein product(s) the query may encode. It could provide information on the functions of the protein(s) should a good match to a well characterized protein is indeed found.

In the example command line below, we search the nucleotide sequences in my_query.txt against refseq_protein database. The results are saved in my_oputput file.

blatcl3 -p blastx -i my_query.txt -d refseq_protein -o my_output -m 9 -v 50 -b 50

The -m 9 instructs blast server to return the results in tabular format. The upper limit of matched database sequences per query is set to 50.

         3.3.2 tblastn: protein query against a translated nucleotide database

This program function searches an input query protein sequence against a target nucleotide database to find other protein similar products that might be encoded by those nucleotide sequences. It is a good way to find out yet unidentified homolog/paralog of a give protein query. During the search, the nucleotide database entries are first translated in all six frames. The query protein is then compared against those potential products to identify the matches.

Example given below searches the input protein query file my_query.txt against est_human database to identify human est entries that may encode proteins similar to the query. The result is saved to my_output.

blastcl3 -p tblastn -i my_query.txt -d est_human -o my_output

         3.3.3. tblastx: searches with translated nucleotide query and nucleotde database

This program function compares all six-frame translations of an input nucleotide query against those from a nucleotide database. Since this search is very computationally expensive, we recommend that our users use it with great caution by employing a higher search stringency (lower -e setting, with -F T and requires few hits than the default -v 500 -b 250) , and limiting the search to a smaller more specific subset of the database using entrez limit.

blastcl3 -p tblastx -i my_query.txt -d chromosome -u "bacteria[orgn]" 
-e 2e-5 -o my_output

The above command line searches the sequences in my_query.txt against the bacterial entries in the chromosome database. The search stringency is increased by lowering the -e setting to 2 * 10^-5. The The result is saved in my_output.

Due to the extremely high computational intensity of tblastx searches, we suggest that users set up local standalone blast to performing such searches if the search volume is large and/or the need is regular.

     3.4 Genome BLAST searches

Genome BLAST pages collect the genomic sequences and other sequences specific to an organism in a centralized place for easy access. In addition, the matches from searching these databases often contain links to the graphic display in the Mapviewer for that organims. Those organism specific sequence databases are also available for search using blastcl3 with one exception: hits will not be linked to the Map Viewer.

Databases for higher eukaryotes are grouped according to orgamisms, each with its own unique database prefix. The genome assemblies and the annotation products from them are build-specific. The dataset are updated only when new assemblies are made available. They are kepted under gpipe directory. Other organism-specific databases not related to assembly are updated regularly and placed in a separate directory with general gp prefix. For example, the human genome database and other human specific databases have the "gpipe/9606/" or "gp/9606.9558" prefix.

blastcl3 -p blastn -i my_query.txt -d gp/9606/ref_contig -F "m L; R" -o my_output

The above command line searches the query sequences (-i my_query.txt) against against reference contigs from NCBI (-d ref_contig), using low complexity and repeat filter (-F "m L; R"). The output is saved to my_output.

For microbes and low eukaryotes, the available dataset will vary dependent on the status of the sequencing project. Some are finished with accompying protein data, some are unfinished partial wgs with or without accompanying protein data. The database naming convention is "Microbial/Taxid".

blastcl3 -p blastx -i my_query.txt -d Microbial/83333 -o my_output

The example command line above uses blastx to search the nucleotide sequences in my_query.txt against the protein dataset for E.coli K-12 strain. The result is saved in my_output.

Please refer to http://www.ncbi.nlm.nih.gov/staff/tao/URLAPI/remote_accessible_blastdblist.html for more information on databases available for remote access by blastcl3. For some organism, not listed in the document, searching against nt, wgs, est_others, htgs, or nr with appropriate Entrez limit may serve as a productive alternative.

blastcl3 -p blastn -i my_query.txt -d wgs -n T -m 9 -o my_output
-u "Oryctolagus cuniculus[Organism] AND wgs[prop]" 

The above command line uses blastn to search sequneces in my_query.txt. It also invokes the megablast algorithm and requests the tabular result. The search is limited to rabbit wgs entries through Entrez queries fed to -u parameter.

4. Appendix: Parameters and their accepted values

As mentioned before in Section 1, blastcl3 has no GUI and works only under a command terminal. Users execute the the program by issuing command lines, and control the search through parameter/value pairs in the command line. The command line parameters and their accepted values for this program are listed individually here. The options commonly adjusted during actual searches are: -i, -d, -p, -o, -e, -F, -u, -b, -v, -m, and -n. The first four are mandatory.

Program      Query                   Database
blastn       nucleotide              nucleotide
blastp       protein                 protein
blastx       nucleotide, translated  Protein
tblastn      protein                 nucleotide, translated
tblastx      nucleotide, translated  nucleotide, translated

    0    Pairwise
    1    query-anchored showing identities
    2    query-anchored no identities
    3    flat query-anchored, show identities
    4    flat query-anchored, no identities
    5    query-anchored no identities and blunt ends
    6    flat query-anchored, no identities and blunt ends
    7    XML Blast output
    8    tabular (not post processing)
    9    tabular with comment lines (post-processed, sorted)
    10   ASN, text
    11   ASN, binary

Program       blastn       megablast       tblastx       others
Value           30            20              0            15

-I T: gi|223046|prf||0410468A ... 
-I F: prf||0410468A ...

Program   blastp   blastn   blastx   tblastn   tblastx  megablast
Value       11       0        12       13        13        0

Program   blastn   megablast  all others
Value       11        28          3

Meaning  Input sequence    Reverse complement    Both
Value          1                   2               3

Program   blastn   megablast   tblastx   all others
Value       50        50         25          0

Program   blastn   megablast   all others
Value        0         0          40