Next Generation Sequencing (NGS) is a fundamental practice in bioinformatics. Pipelines are comprised of complex, multi-step processes involving many different tools and intermediate data formats. With easy access to cloud infrastructure and containerized applications that are portable across clouds, users are increasingly extending pipelines to the cloud.
In part I of this article, we discussed Nextflow, a leading tool for managing bioinformatics workflows and showed how it can be used with Univa Grid Engine (UGE) and Navops Launch to facilitate transparent hybrid cloud bursting to multiple cloud providers.
In this second article, we’ll look at the mechanics of how cloud bursting is enabled in UGE and Navops Launch and explain how bioinformatics users can enable bursting to multiple clouds regardless of their chosen pipeline tools and management frameworks.
Configuring Cloud Bursting with Nextflow, Navops Launch and Univa Grid Engine
Under the covers, transparent cloud bursting is provided by Navops Launch and its integration with Univa Grid Engine. An easy way to get started with a working environment is to use the Univa Grid Engine offering on the AWS Marketplace. The marketplace deployment provides a ready-to-use cloud-resident UGE master host with Navops Launch and Univa Unisight (for reporting and management) pre-installed.
When you log in to master host, you’ll be provided with step-by-step instructions on how to add and remove cluster nodes using the Navops Launch add-nodes and delete-node commands.