Is there a way to determine the frequency at which OnDemand polls Slurm nodes (login or ssh hosts) for job state? And is there a way to control this frequency?
The frequency should be set in the
slurm.conf file itself using the
JobAcctGatherFrequency. That setting has more documentation here:
Looking at the entry for
JobAcctGatherFrequency it looks like for
tasks the interval is 30 seconds and for any other datatype the default is 0 seconds, meaning it will wait until termination of the job.
OnDemand will just use the settings in that file to configure itself for polling which is why the path for the
slurm.conf in the
/etc/ood/config/clusters.d/my_cluster.yml is included.
That’s exactly what I was looking for, thank you!
Hey @milberg I was incorrect in what I posted earlier. You can actually set this polling for OnDemand.
The default on the last piece shows we have a default of 10 seconds actually.
You can set this in your
env file like so:
That’s very helpful, thanks!