I noticed that when in OnDemand interactive jobs, the subsequent “srun” commands start with empty environment.
That is, e.g. if I start Interactive Desktop, and open a terminal in this desktop, I have full environment, but, if I subsequently start an executable with a srun command (e.g. to run MPI), I get:
$ srun -n 1 hostname slurmstepd: error: Unable to create TMPDIR [/scratch/local//2666718]: Permission denied slurmstepd: error: Setting TMPDIR to /tmp slurmstepd: error: execve(): hostname: No such file or directory
which is equivalent to “srun --export=NONE”.
Digging into the OOD’s SLURM adapter code at ood_core-0.13.0/lib/ood_core/job/adapters/slurm.rb, there is a section:
# we default to export NONE, but SLURM defaults to ALL. # we do this bc SLURM setups a new environment, loading /etc/profile # and all giving 'module' function (among other things shells give), # where the PUN did not. # --export=ALL export the PUN's environment. def export_arg(env, copy_environment) if !env.empty? && !copy_environment env.keys.join(",") elsif !env.empty? && copy_environment "ALL," + env.keys.join(",") elsif env.empty? && copy_environment # only this option changes behaivor dramatically "ALL" else "NONE" end end
I am wondering if that is responsible for setting the “–export=NONE” for the srun commands inside of the OOD job, or, if this is just for “sbatch” that starts the interactive job.
Or, any other thoughts in this regard?