We have a number of users of workflow managers such as nextflow, and in principle other similar tools thata want to have long-term continuous presence on the cluster to submit and manage workflows.
NERSC has a really nice page that covers workflow managers in general and gives various commentary and tips. They also have a page on nextflow linked there that mentions some settings users can use to avoid overwhelming the scheduler. They also offer access to what they call “workflow nodes” that appear to be older nodes in the cluster that they have dedicated to such tasks.
I am curious to know whether anyone has created OOD modules to support any workflow managers, and Pegasys and Nextflow in particular. Are there ways to set up such “workflow management” sessions for long-term use with OOD, perhaps using tools like screen to manage disconnections and reconnections? Any thoughts on this topic?