Interactive jobs started inside Interactive Desktop don't export environment variables (Slurm)

We’re using Open OnDemand 1.6.22 with Slurm 19.05.4.

A common use case for our users is to use long-running Interactive Desktop sessions to launch shorter-running interactive jobs which need their own resources. This is done by using salloc to request a new allocation, and then using srun to launch the interactive jobs with this allocation.

I’m seeing an unexpected behavior, which is that by default it appears no environment variables are exported to interactive (srun) jobs when run inside of the OOD Interactive Desktop app. In order to avoid a job erroring out, I have to explicitly export all environment variables. This is not the behavior I see with interactive jobs, or nested interactive jobs, outside OOD.

Example:

nc060:~ % salloc -c4
salloc: Granted job allocation 16924909
nc060:~ % srun --pty bash
slurmstepd: error: execve(): bash: No such file or directory

Here is the workaround:

nc060:~ % salloc -c4
salloc: Granted job allocation 16924910
nc060:~ % srun -c4 --export=ALL --pty bash
nc265:~ %

Based on my read of the srun man page, the default Slurm behavior should always export all environment variables to an interactive job.

Can anyone shed light on why this behavior is not seen in Open OnDemand? Thank you.

Yes, in 1.6 this functionality doesn’t exist (we always pass the environment variable SBATCH_EXPORT=NONE.

In 1.7 it does not pass this by default (it passes nothing, which in turn uses SLURM default) and that actually breaks some desired behaviors, so it’s slightly broken and I hope to patch it in the coming days.

If you upgrade to 1.7 and you need to export additional entries, you’ll need to add these to the submit.yml.erb.

script:
  copy_environment: true
  job_environment:
    FOO: BAR

Which would translate to --export=ALL,FOO