Hunting Invasive Species with HTCondor: High Throughput Computing for Big Data and Next Generation Sequencing

Summary

Large amounts of data are being generated that require hours, days, or even weeks to analyze using traditional computing resources. Innovative solutions must be implemented to analyze the data in a reasonable timeframe. The program HTCondor (https://research.cs.wisc.edu/htcondor/) takes advantage of the processing capacity of individual desktop computers and dedicated computing resources as a single, unified pool. This unified pool of computing resources allows HTCondor to quickly process large amounts of data by breaking the data into smaller tasks distributed across many computers. This project team implemented HTCondor at the USGS Upper Midwest Environmental Sciences Center (UMESC) to leverage existing computing capabilities for [...]

Summary

Large amounts of data are being generated that require hours, days, or even weeks to analyze using traditional computing resources. Innovative solutions must be implemented to analyze the data in a reasonable timeframe. The program HTCondor (https://research.cs.wisc.edu/htcondor/) takes advantage of the processing capacity of individual desktop computers and dedicated computing resources as a single, unified pool. This unified pool of computing resources allows HTCondor to quickly process large amounts of data by breaking the data into smaller tasks distributed across many computers.

This project team implemented HTCondor at the USGS Upper Midwest Environmental Sciences Center (UMESC) to leverage existing computing capabilities for data processing and analysis. HTCondor can be used for a wide range of projects including processing DNA sequencing data (currently done as part of invasive species monitoring); validating new statistical models over a wide range of possible parameter combinations; and analyzing long-term vegetation and fish data from the Upper Mississippi River. The HTCondor pool is online and operational at UMESC. The USGS Wisconsin Water Science Center was able to connect to the pool at the USGS Oregon Water Science Center through “flocking” (fig. 16). This test identified cybersecurity and data transfer challenges that can be overcome in future work. Flocking with HTCondor requires communication among machines in various centers which, in turn, requires traffic to be allowed through the firewall of each center. When flocking takes place, many connections must be made, but all traffic is consolidated into a single port which is 9618. As a result, only port 9618 must be opened, and traffic can be limited to USGS computers. This minimizes the risk of allowing two-way traffic through firewalls from center to center. The technology underlying the test was shown to be successful and was an important step toward connecting and leveraging computing resources throughout the USGS.

Principal Investigator : Sunnie (Grace) McCalla, Michael N Fienen, Richard A Erickson, Randall J Hunt, Jon J Amberg
Cooperator/Partner : Jonathan M Knudson

Accomplishments

The accomplishments for this project are described below.

HTCondor was installed and configured at UMESC.
Configuration files and testing for flocking among USGS centers were created and performed.
Documentation examples for using HTCondor to scale scientific processing with cluster computing were uploaded to USGS BitBucket at https://my.usgs.gov/bitbucket/projects/CDI/repos/hunting_invasive_species_with_htcondor/browse (content no longer available).
Testing of Docker as a technology to exchange configuration of complex workflows and datasets throughout HTCondor pools was performed.

Note: This description is from the Community for Data Integration 2016 Annual Report.

Contacts

Contact :: Sunnie (Grace) McCalla
CMS Group :: Community for Data Integration - CDI

Attached Files

sorting

Metadata First

Recent First

Alphabetical

Click on title to download individual files attached to this item.

6-McCalla-CDI_project_20170308.pdf “CDI Monthly Meeting Presentation”		5.55 MB	application/pdf
Figure16_condor_flocking.png “Schematic of flocking with HTCondor between USGS centers”		164.09 KB	image/png

Related External Resources

Type: Source Code

Project Bitbucket Repository - Content no longer available	https://my.usgs.gov/bitbucket/projects/CDI/repos/hunting_invasive_species_with_htcondor/browse

Project Extension

projectStatus	Completed

Hunting Invasive Species with HTCondor: High Throughput Computing for Big Data and Next Generation Sequencing

Summary

Summary

Accomplishments

Contacts

Attached Files

sorting

Metadata First

Recent First

Alphabetical

Project Extension

Preview Image

Communities

Tags

Provenance

Item Actions

View Item as ...

Save Item as ...

View Item...

Summary

Summary

Accomplishments

Contacts

Attached Files sorting Metadata First Recent First Alphabetical

Related External Resources

Project Extension

Preview Image

Communities

Tags

Provenance

Item Actions

View Item as ...

Save Item as ...

View Item...

Attached Files

sorting

Metadata First

Recent First

Alphabetical