We’re using Open OnDemand 1.6.22 with Slurm 19.05.4.
A common use case for our users is to use long-running Interactive Desktop sessions to launch shorter-running interactive jobs which need their own resources. This is done by using salloc to request a new allocation, and then using srun to launch the interactive jobs with this allocation.
I’m seeing an unexpected behavior, which is that by default it appears no environment variables are exported to interactive (srun) jobs when run inside of the OOD Interactive Desktop app. In order to avoid a job erroring out, I have to explicitly export all environment variables. This is not the behavior I see with interactive jobs, or nested interactive jobs, outside OOD.
nc060:~ % salloc -c4 salloc: Granted job allocation 16924909 nc060:~ % srun --pty bash slurmstepd: error: execve(): bash: No such file or directory
Here is the workaround:
nc060:~ % salloc -c4 salloc: Granted job allocation 16924910 nc060:~ % srun -c4 --export=ALL --pty bash nc265:~ %
Based on my read of the srun man page, the default Slurm behavior should always export all environment variables to an interactive job.
Can anyone shed light on why this behavior is not seen in Open OnDemand? Thank you.