Selinux blocks ondemand from talking to remote slurm server

Hi,
I’m installing OOD on Centos 8, with the slurm server on a different machine, and with selinux enabled.
I have enabled the “ondemand_use_slurm” boolean.
I’m getting selinux failures, so as per 3. Modify System Security — Open OnDemand 1.8.12 documentation I’m posting it here :slight_smile:

I’m getting messages of the following form:
denied { name_connect } for pid=266555 comm=“sbatch” dest=6817 scontext=system_u:system_r:ood_pun_t:s0 tcontext=system_u:object_r:slurmctld_port_t:s0 tclass=tcp_socket

(I would note that the boolean doesn’t do what I’d expect - sesearch -A -b ondemand_use_slurm only shows the following rule:
allow ood_pun_t var_run_t:sock_file { getattr write }; [ ondemand_use_slurm ]:True
which doesn’t seem to be helpful).

I can work around this easily enough for now, but it should probably be fixed at some point.

Cheers

David

Hi and welcome! @tdockendorf do you have any insight into this?

@David.Monro Do you have some site specific SELinux policies deployed that define slurmctld_port_t because that does not look like something done by the OS. If that’s some site specific feature then there is not an update in OnDemand to fix that as we cannot create policies for non-standard contexts that are not part of the OS because then the policies would not compile when we build the RPMs.

Also to clarify the ondemand_use_slurm only allows connecting to the munge socket, there is nothing specific to SLURM with that boolean.

Doh! It was indeed a local policy. And what is worse - apparently I wrote it :man_facepalming:

However, trying it on a box without that policy, I still had a couple of issues:

  • Instead of slurmctld_port_t I got 'unreserved_port_t` - which was solved by enabling ondemand_use_torque (slightly unexpected when I’m using slurm, but it works. Not sure if that enables any other rules which would be undesirable.) Perhaps the ondemand_use_slurm boolean should also enable allow ood_pun_t unreserved_port_t:tcp_socket name_connect; ?

  • Attempting to create a developer app from a remote repository generated the following AVC:
    type=AVC msg=audit(1610594458.016:116334): avc: denied { map } for pid=281140 comm=“git” path="/home/XXXX/ondemand/dev/jupyter/.git/config" dev=“0:50” ino=1089193880 scontext=system_u:system_r:ood_pun_t:s0 tcontext=system_u:object_r:nfs_t:s0 tclass=file permissive=0
    which was solved by enabling the ‘domain_can_mmap_files’ boolean. Perhaps that could be documented somewhere?

Cheers

David

Would you mind giving this a try? ondemand/ondemand-selinux.te at selinux · OSC/ondemand · GitHub . Pull request: Fixes for SELinux policies by treydock · Pull Request #847 · OSC/ondemand · GitHub

That is the modified SELinux policy for OnDemand. Should only need to modify @VERSION@ at top to some number greater than what’s currently installed to be able to build and install:

cd /path/to/ondemand-selinux.te
make -f /usr/share/selinux/devel/Makefile
semodule -i ondemand-selinux.pp

I gave the same unreserved port allow from the torque boolean to SLURM. Also rather than enabling domain_can_mmap_files I opted to add what I hope is the correct allow to use_nfs_home_dirs boolean.