CCB » Software » PhymmBL

Note on support

Until further notice, PhymmBL is provided as-is. I will continue to develop the program (including incorporating useful suggestions and fixes sent in by users) as I find time, but my current job doesn't provide a whole lot of leeway for that, so it'll be sporadic and drawn-out.

Also until further notice, I'm suspending individual user support. I'm humbled and gratified that people continue to use the program, but I'm not doing anyone any favors by making them wait 3-4 months for an answer (roughly my current turnaround time). If anyone would like to start a support forum for the software -- it's fully open-source, so the more experienced programmers out there can probably help stuck users at least as well as I can -- I thoroughly endorse the idea. I ask only that you tell me about any such forums as they go live, so that I can point future support requests there (and so I can make sure there aren't redundant forums popping up in different places).

My apologies to any users who've emailed me between now and the last time I had some extra time to devote to development and support (around September 2012). Your requests are still in my queue, but I can't predict when I'll be able to attend to them.

-Arthur
2013.03.13

About PhymmBL

Metagenomics sequencing projects collect samples of DNA from uncharacterized environments that may contain hundreds or even thousands of species. One of the main challenges in analyzing a metagenome is phylogenetic classification of raw sequence reads into groups representing the same or similar species. Such classification is a useful prerequisite for genome assembly and for analysis of the biological diversity present in a sample. The newest sequencing technologies have simultaneously made metagenomics easier, by making the sequencing process faster, and more difficult, by producing shorter read lengths than previous technologies. Methods for classifying sequences as short as 100 base pairs (bp) have until now been relatively inaccurate, requiring metagenomics projects to use older, long-read technologies. Phymm, a new classification approach for metagenomics data which uses interpolated Markov models (IMMs) to taxonomically classify DNA sequences, can accurately classify reads as short as 100 bp. Its accuracy for short reads represents a significant leap forward over previous composition-based classification methods. PhymmBL (rhymes with "thimble"), the hybrid classifier included in this distribution which combines analysis from both Phymm and BLAST, produces even higher accuracy.
 
 

VERSION HISTORY

PhymmBL v4.0 is the current stable release.
 
> New in v4.0 [2012.08.31.1640]:
 
  • A fleet of across-the-board stability upgrades and minor code changes.
  • Enabled detailed logging of all levels of operation for easier progress tracking & troubleshooting.
  • Support has been enabled for the BLAST+ applications, making the BLAST portion of PhymmBL's processing pipeline significantly faster.
  • A file descriptor/redirection complaint specific to Ubuntu has been identified and fixed. Thanks to David Kelley for first pointing it out.
  • The setup process has been made substantially more robust against errors and inconsistencies in RefSeq's GenBank-encoded metadata and NCBI's taxonomic trees.
  • A rare bug in the BLAST database-rebuilding components of several scripts has been identified and eliminated.
 
> New in v3.2 [2011.02.23.1546]:
 
  • Custom genome data can now be added in batches instead of having to add one organism at a time. See the README for details and instructions on using the new batch mode.
 
> New in v3.1 [2010.10.18.1651]:
 
  • Reconfigured raw Phymm output format to deliver a huge reduction in file size.
  • Fixed a rare inifite loop potential in rebuildBlastDB.pl.
 
> New in v3.01 [2010.09.17.1300]:
 
  • Fixed a minor bug in addCustomGenome.pl that occasionally resulted in the loss of taxonomic metadata for new organisms.
 
> New in v3.0 [2010.06.25.1425]:
 
  • Confidence scores are now listed in the PhymmBL results files, translating raw scores into usable estimates of predictive accuracy. Please see the README for an important discussion of how to interpret and work with these scores.
  • Date stamps are now given in each phase of PhymmBL's terminal output to let users know how long each phase of analysis has taken.
  • ICM IDs are now listed in the raw Phymm output to allow for disambiguation between ICM scores assigned by different ICMs within the same species.
 
> New in v2.03 [2010.06.11.1327]:
 
  • Semicolons in species/strain names are now handled properly with respect to local database directory structure.
  • The database of known GenBank taxonomic-labeling inconsistencies has been updated.
  • A workaround has been added for kernels that complain when the 'cat' command is passed too many arguments, which can affect the construction of the local BLAST database. (If you didn't see an error during setup, you don't have to worry about this.)
  • A section has been added to the README with suggestions on incorporating mate-pair information into your classification run.
 
> New in v2.02 [2010.06.07.1246]:
 
  • A bug in addCustomGenome.pl preventing full assimilation of new genomes has been corrected. If you have any of the 2.x versions, and you attempted to add your own genomic data, check your PHYMM_DIR/.genomeData/.userAdded/ADDED_ORGANISM/ directory; if it doesn't contain any .icm files, please redownload the Phymm installer and add your custom genomes again. You will not need to regenerate the core RefSeq libraries or alter the main genomic database in any way.
  • A bug in the new-copy RefSeq download subroutine has been fixed. If you installed one of the 2.x versions for the first time and no genome data appeared, this version will correct the problem. Timeouts for RefSeq downloads have been extended, and the interface for addCustomeGenome.pl has been tweaked to make the taxonomic data entry a little clearer.
 
> New in v2.01 [2010.05.27.1634]:
 
  • The README has been substantially expanded to include instructions on parallelization, notes on interpreting PhymmBL's numeric scores, and several other minor changes. Program code has not been changed. Thanks to Liam Elbourne for helpful discussions.
 
> New in v2.0 [2010.05.25.1335]:
 
  • A script has been added allowing users to add their own custom genomic sequence data to the local database. The script takes new sequence data (as FASTA/multiFASTA files), adds them to the BLAST database, and creates IMMs to model them. The user is polled to provide taxonomic data for each new organism.
  • The setup script has been completely rewritten; users can now choose whether to download a completely new copy of the RefSeq microbial database, or to update the existing local database with only RefSeq sequences which have been added or have changed since the last install. (Genome content, taxonomic information and model files for user-added organisms are stored separately from the RefSeq data, so updates won't affect any custom content.)
  • A script has been added to manually regenerate the local BLAST database.
  • PhymmBL's combined scoring function was tweaked for the case of BLAST's E-value being reported as "0.0", resulting in slightly better overall accuracy in this case.
  • A collection of minor bugs and irritations has been fixed.
  • Mac OS is now formally supported, but please see the README for a note on obtaining wget, which isn't provided with the OS X suite of developer tools and is needed for setup to run properly.
 
 

Accuracy

Because one of the main challenges of metagenomic analysis is the fact that species are frequently encountered which have never before been sequenced, we examined the performance of this system using increasingly less data from organisms related to those from which query reads were sampled. The table below summarizes predictive accuracy results from PhymmBL, the hybrid method incorporating information from both Phymm and BLAST.
 
For instance, the information in the cell indexed by "Family excluded" and "Phylum" means that when, for each query read in our test set, all organisms belonging to the same family as the organism from which that read was sampled were excluded from consideration -- i.e., when the best possible prediction is one made at the order level -- PhymmBL was able to predict the correct phylum of query reads 57.5% of the time, with a standard deviation (measured over 10 runs) of ± 0.6%.
 
Note that accuracy, as reported in the table below, is measured as the percentage of all 100-bp query reads in the test data that received a correct label; no reads are left unlabeled.
 
Please see the paper for details on these and other experiments. All synthetic test data used for the experiments described in the paper (10 sets of 100-bp reads, plus one set each containing reads of 200, 400, 800 and 1000 bp) can be downloaded here.
 


 Species   Genus   Family   Order   Class   Phylum 
 All matches allowed   95.4 ± 0.2   99.1 ± 0.1   99.7 ± 0.1   99.8 ± 0.1   99.9 ± 0.1   99.9 ± 0.0 
 Species excluded   ---   58.5 ± 0.6   63.7 ± 0.6   66.3 ± 0.6   71.0 ± 0.5   76.8 ± 0.8 
 Genus excluded   ---   ---   26.9 ± 0.6   33.0 ± 0.6   44.6 ± 0.6   63.4 ± 0.6 
 Family excluded   ---   ---   ---   19.3 ± 0.5   33.4 ± 0.5   57.5 ± 0.6 
 Order excluded   ---   ---   ---   ---   23.8 ± 0.5   53.2 ± 0.6 
 Class excluded   ---   ---   ---   ---   ---   43.5 ± 0.7 

PhymmBL percent prediction accuracy and standard deviations for classification experiments with 100-bp reads and different clade levels excluded from comparison.
 

Obtaining the Software

This software is OSI Certified Open Source Software.   
 
Click to download the PhymmBL installation software as either a gzipped tarball or as a .zip file.

After downloading, move the downloaded file into a directory in which you intend to store the PhymmBL program files and downloaded genomic data, then uncompress it by typing
 
     tar zxvf phymmbl_installer.tar.gz
 
PhymmBL's subdirectory structure will be created in your target directory, as will the installer script and a README file with instructions on building and using the system.
 
The software was developed and tested on a multi-core Linux system; it is expected to work properly on any Unix-like system which meets its system requirements (see the README for details, including an extra step Mac OS users will need to take).
 
PLEASE NOTE: Setup is particularly computationally intensive: even on a relatively powerful server, you should expect ground-up installation to take at least 24 hours.

 

References

 

Funding

This work is supported in part by NIH grant R01-LM006845 to S.L. Salzberg.