CloudMan is a tool that facilitates Galaxy deployments in cloud environments. Quoting the brief description from the official web site:
“CloudMan is a cloud manager that orchestrates the steps required to provision and manage compute clusters on a cloud infrastructure, all through a web browser. In minutes, you can use a vanilla SGE cluster or a complete bioinformatics analysis environment.”
As such it was an obvious choice for us to deploy our Galaxy based Image Processing and Analysis Toolkit on the NeCTAR cloud. Although spawning new clusters when the CloudMan is already setup on a cloud it’s quite easy, there is a fair amount of work required to get to this stage, which includes among others:
- creating images and volume snapshot with our customized Galaxy
- setting up the Cloudman/Galaxy configuration files in object storage
- … and contributing to the CloudMan project itself to fix some issues with OpenStack compatibility
The good new is that we have managed to deploy CloudMan and our customized Galaxy with image processing tools on an OpenStack cloud. We can now build on-demand Galaxy/SGE (Sun Grid Engine) clusters with up to 20 nodes.
Galaxy in this configuration uses DRMAA specification to submit computational task to SGE to be scheduled for execution on the cluster. That (given access to sufficient cloud resource allocation) let’s us scale the application with increasing number of users.
We have also managed to successfully enable MPI support in the cloud SGE, which we can now use to execute MPI based tools (e.g. some of the X-TRACT components) utilizing the cluster resources for speeding up parallelizable computation.
The CloudMan relies heavily on the use of volumes (cloud block
storage) and volume snapshots that are not currently publicly available in
the NeCTAR cloud. The experimental support
has been around for while and it seems that production version may be available
soon and then we can migrate out deployment to NeCTAR.
No comments:
Post a Comment