Unable to launch fully containerized Rstudio

We are trying to fully containerize Rstudio however we run the image we get the following error. Is there something we are missing here? Ref:

Jun 03 08:09:53 pdlhpc1cn001 rserver[13797]: ERROR Could not change permissions for specified ‘server-data-dir’ - the directory (/var/run/rstudio-server) must be writeable by all users and have the sticky bit set; LOGGED FROM: int main(int, char* const*) src/cpp/server/ServerMain.cpp:608

Jun 03 08:09:53 pdlhpc1cn001 rserver[13797]: ERROR system error 30 (Read-only file system) [path: /var/run/rstudio-server]; OCCURRED AT rstudio::core::Error rstudio::core::system::changeFileMode(const rstudio::core::FilePath&, mode_t) src/cpp/server/ServerMain.cpp:62; LOGGED FROM: int main(int, char* const*) src/cpp/server/ServerMain.cpp:609

Jun 03 08:14:57 pdlhpc1cn001 rserver[15512]: ERROR system error 13 (Permission denied) [path: /var/run/rstudio-server, target-dir: ]; OCCURRED AT rstudio::core::Error rstudio::core::FilePath::createDirectory(const string&) const src/cpp/shared_core/FilePath.cpp:634; LOGGED FROM: int main(int, char* const*) src/cpp/server/ServerMain.cpp:555

@charles8ronson

Can you share the script.sh.erb that launches RStudio?

We figured out that we needed to mount /tmp to /var/run in the container
but now it’s not adhering to the authorization portion of the script:


# Load the required environment
setup_env () {
  # Additional environment which could be moved into a module
  # Change these to suit
  export RSTUDIO_SERVER_IMAGE="/depot/tmp/singularity-images/de-gmorto/rstudio-server-centos7r4.sif"
#don't change paths in the container. everything should work since it's all contained inside the container above
  export SINGULARITY_BINDPATH="/tmp:/var/run,/active,/archive"
  export PATH="/R/4.0.0/bin:/usr/lib/rstudio-server/bin:$PATH"
  export SINGULARITYENV_PATH="$PATH"
  export MODULEPATH="/depot/Modules/modulefiles"
  # In Singularity 3.5.x it became necessary to explicitly pass LD_LIBRARY_PATH
  # to the singularity process
  # export SINGULARITYENV_LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/lib:/usr/lib64"
}
setup_env

#
# Start RStudio Server
#
module use /depot/Modules/modulefiles
module load python

# PAM auth helper used by RStudio
export RSTUDIO_AUTH="${PWD}/bin/auth"

# Generate an `rsession` wrapper script
export RSESSION_WRAPPER_FILE="${PWD}/rsession.sh"
(
umask 077
sed 's/^ \{2\}//' > "${RSESSION_WRAPPER_FILE}" << EOL
  #!/usr/bin/env bash

  # Log all output from this script
  export RSESSION_LOG_FILE="${PWD}/rsession.log"

  exec &>>"\${RSESSION_LOG_FILE}"

  # Launch the original command
  echo "Launching rsession..."
  set -x
  exec rsession --r-libs-user "${R_LIBS_USER}" "\${@}"
EOL
)
chmod 700 "${RSESSION_WRAPPER_FILE}"

# Set working directory to home directory
cd "${HOME}"

export TMPDIR="$(mktemp -d)"
export TMPDIR="$(mktemp -d)"

mkdir -p "$TMPDIR/rstudio-server"
python -c 'from uuid import uuid4; print(uuid4())' > "$TMPDIR/rstudio-server/secure-cookie-key"
chmod 0600 "$TMPDIR/rstudio-server/secure-cookie-key"

module unload python

set -x
# Launch the RStudio Server
echo "Starting up rserver..."

singularity run -B "$TMPDIR:/tmp" "$RSTUDIO_SERVER_IMAGE" \
 --www-port "${port}" \
 --auth-none 0 \
 --auth-pam-helper-path "${RSTUDIO_AUTH}" \
 --auth-encrypt-password 0 \
 --rsession-path "${RSESSION_WRAPPER_FILE}"

echo 'Singularity has exited...'```
1 Like

Have you looked at any of the other threads on here about Rstudio? A lot of people have been able to get it working, but there always seem to be certain containerization or authentication issues that crop up. Most of the threads have solutions posted. For example:

Hello, that was not the solution. It was part of the solution. Authorization is still not functioning. I get to the login and the username/password aren’t getting validated to allow me to log in.when I type them in manually, it also doesn’t work.
This is following the button Connect to RStudio Server:

This is where it fails after I launch it.
image

I can’t add new posts to this topic so I’m editing this post:

I mounted /home
I updated the template/bin/auth to include the changes for rstudio server 1.3 support
still failing at authorization phase.

I installed within the container:
R-4.0.0.tar.gz to /R/4.0.0
rstudio-server-rhel-1.3.959-x86_64.rpm to default location: /usr/lib/rstudio-server/bin

Still have auth issues. Any other ideas would be greatly appreciated.

I’d ensure that HOME is mounted too. You seem to be passing credentials OK, but can the container find this file `--auth-pam-helper-path "{RSTUDIO_AUTH}"` (It ends up being ~/ondemand/data/sys/dashboard/batch_connect/sys/rstudio/output//bin/auth)?

What version of RStudio Server are you using? We noticed that the new 1.3 version fails to authenticate with “Error: Incorrect or invalid username/password”. We just didn’t get around to posting the issue yet.

Hi,

The authentication issue in 1.3 is due to a change in how the auth-pam-helper script is invoked by rserver. Previously, it was called with just one argument, $USERNAME.
In 1.3 it is called with three arguments, “$USERNAME rstudio 1”. The test in line 4 of the bin/auth script:

#!/usr/bin/env bash

# Confirm username is supplied
if [[ $# -ne 1 ]]; then
  echo "Usage: auth USERNAME"
  exit 1
fi
...

fails and needs to be modified to handle the extra arguments. I just replaced “-ne” with “-lt” to get authentication to work.

1 Like

@dugan Thanks for this!

I opened an issue to get this merged into bc_osc_rstudio_server here https://github.com/OSC/bc_osc_rstudio_server/issues/34

Onbehalf of eisenfaust (can only do 3 posts atm):

I mounted /home
I updated the template/bin/auth to include the changes for rstudio server 1.3 support
still failing at authorization phase.

I installed within the container:
R-4.0.0.tar.gz to /R/4.0.0
rstudio-server-rhel-1.3.959-x86_64.rpm to default location: /usr/lib/rstudio-server/bin

Still have auth issues

Thanks for this. We’ve now fully remove Singularity from our RStudio Server app.

Try pointing RStudio Server to your /tmp.

singularity run -B "$TMPDIR:/tmp" "$RSTUDIO_SERVER_IMAGE" \
--www-port "${port}" \
--auth-none 0 \
--auth-pam-helper-path "${RSTUDIO_AUTH}" \
--auth-encrypt-password 0 \
--rsession-path "${RSESSION_WRAPPER_FILE}" \
--server-data-dir /tmp

Unfortunately that didn’t solve my issue for the authorization issue I’m currently having.
It does seem to be a potentially good way to not have to map /tmp:/var/run. If we end up doing this, I think we will need to also supply:
--server-pid-file arg (=/var/run/rstudio-server.pid)
probably something like:
--server-pid-file /tmp/rstudio-server.pid

The issue with the /var/run may be due to write permissions since it was built with fakeroot:
singularity build -f

Would love to work out this Authorization issue though.

Hi, are you certain that you need to have this bind?

We’re using the Singularity Hub RStudio image

with this configuration

In this configuration, I’m only binding /fast which is our GPFS file system mount which contains the user’s homes.

Did you try using the --writable-tmpfs option when launching singularity?

Without the bind, it fails to open the port.

The --writable-tmpfs is interesting to me but it fails to open a port with that added to the singularity run in the script.

version 1.1.463 works fine for me when I use R 3.6.3
But rstudio server version 1.3.959 fails to authenticate. I’m using R 4.0.0 in this image

I would say next steps are answering these two questions. Can the job find the $RSTUDIO_AUTH in the container (if you’re not mounting home as an example)? Then is the $RSTUDIO_AUTH working as it’s expected?

  1. can it find the $RSTUDIO_AUTH file. To debug this, I’d write a simple wrapper like this. This ensures that the file exists in the container and you’re correctly mounting HOME.
#!/bin/bash

# load any modules you may need
module purge
module restore
module load ondemand
module load app_rstudio_server/3.6.3 rstudio_launcher/centos7

TMPDIR=$(mktemp -d)

# replace with your file location
RSTUDIO_AUTH="/users/PZS0714/johrstrom/ondemand/data/sys/dashboard/batch_connect/sys/bc_osc_rstudio_server/output/72ca052a-6af0-47dd-a521-0b2ed966170e/bin/auth"

# notice I'm using exec here (so i can execute ls), but it's the same arguments we use 
SINGULARITYENV_LD_LIBRARY_PATH="$LD_LIBRARY_PATH" singularity exec --contain -B "$TMPDIR:/tmp" \
 "$RSTUDIO_SERVER_IMAGE" \
 ls -lrt $RSTUDIO_AUTH
  1. The right env variables are being populated checked in the RSTUDIO_AUTH file. I’d say modify the $RSTUDIO_AUTH file to redirect output to a log file. It looks like stdout and stderr from this script are being eaten, so let’s redirect it to a file to see where exactly it’s failing. The $RSTUDIO_PASSWORD is created in the before.sh and exported from there. And $USER, is well, the system $USER.

Agh, thanks @dugan, I spent way too much time on this silly thing. How did you figure the extra auth-pam-helper arguments out? strace?

Did anyone succeed with getting the RStudio 1.4 to work in user space? For me, up to 1.3 works fine, but 1.4 throws another annoying error:
[rserver] ERROR system error 13 (Permission denied) [path: /var/lib/rstudio-server, target-dir: ]

I am not successful in finding a runtime option that would steer this directory somewhere writeable.

@mcuma, I have not been able to get 1.4 working in a way that is suitable for users yet. The problem you identified happens because it wants to create a sqlite database in that directory.
There is a new commandline option “–database-config-file path_to_config_file”, and the directive
to put in that config file to change the location of the database is:

directory=“some_directory”

With these changes to script.sh.erb the server will start without error but when I try to connect I get the dreaded rstudio login screen. If I enter my username and the “passwd” from the connection.yml file I almost get in. This process added “AppUri” to the end of the url and I get a “page not found” error. If I remove that bit from the url in the address bar and type return I do get in.

The real problem here is that they added some extra “security” features in 1.4. In particular they
added a csrf-token to the login screen. We need to figure out what the value is and pass that in along with the username and passwd in order to get past the login screen. I don’t know how to do
this automatically. This is where I’m stuck. I added this line to view.html.erb:

I still get the login screen when I connect but I can just look at the page source and see what
csrf-token is set to and then manually add a line to connection.yml:

csrfToken: …

then when I connect I get in without hitting the login screen.

So that’s where I’m at.