I am working on a proof of concept of OnDemand for us on our large clusters. I was able to setup a basic Install on one of our test clusters. I built a desktop environment that starts up a singularity container on a compute node, and that worked well. I also setup a jupyter notebook/lab system in singularity container.
In looking at how to reliably deploy the OnDemand Frontend services, we wanted to see if it would run in a singularity container, such that after a reinstall of the controlling nodes it could just be started up quickly by just executing the container. We are starting it as the root user, and ran into some issues early on with the container not having the whole user database available, but worked past that by binding the nss paths into place.(I think this is what left the strange socket around). Once we cleaned that up the system works just as it did when it was installed on the main system.