Unable to launch fully containerized Rstudio

[charles8ronson]

We are trying to fully containerize Rstudio however we run the image we get the following error. Is there something we are missing here? Ref:

Jun 03 08:09:53 pdlhpc1cn001 rserver[13797]: ERROR Could not change permissions for specified ‘server-data-dir’ - the directory (/var/run/rstudio-server) must be writeable by all users and have the sticky bit set; LOGGED FROM: int main(int, char* const*) src/cpp/server/ServerMain.cpp:608

Jun 03 08:09:53 pdlhpc1cn001 rserver[13797]: ERROR system error 30 (Read-only file system) [path: /var/run/rstudio-server]; OCCURRED AT rstudio::core::Error rstudio::core::system::changeFileMode(const rstudio::core::FilePath&, mode_t) src/cpp/server/ServerMain.cpp:62; LOGGED FROM: int main(int, char* const*) src/cpp/server/ServerMain.cpp:609

Jun 03 08:14:57 pdlhpc1cn001 rserver[15512]: ERROR system error 13 (Permission denied) [path: /var/run/rstudio-server, target-dir: ]; OCCURRED AT rstudio::core::Error rstudio::core::FilePath::createDirectory(const string&) const src/cpp/shared_core/FilePath.cpp:634; LOGGED FROM: int main(int, char* const*) src/cpp/server/ServerMain.cpp:555

[mario]

@charles8ronson

Can you share the script.sh.erb that launches RStudio?

[eisenefaust]

We figured out that we needed to mount /tmp to /var/run in the container
but now it’s not adhering to the authorization portion of the script:

# Load the required environment
setup_env () {
  # Additional environment which could be moved into a module
  # Change these to suit
  export RSTUDIO_SERVER_IMAGE="/depot/tmp/singularity-images/de-gmorto/rstudio-server-centos7r4.sif"
#don't change paths in the container. everything should work since it's all contained inside the container above
  export SINGULARITY_BINDPATH="/tmp:/var/run,/active,/archive"
  export PATH="/R/4.0.0/bin:/usr/lib/rstudio-server/bin:$PATH"
  export SINGULARITYENV_PATH="$PATH"
  export MODULEPATH="/depot/Modules/modulefiles"
  # In Singularity 3.5.x it became necessary to explicitly pass LD_LIBRARY_PATH
  # to the singularity process
  # export SINGULARITYENV_LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/lib:/usr/lib64"
}
setup_env

#
# Start RStudio Server
#
module use /depot/Modules/modulefiles
module load python

# PAM auth helper used by RStudio
export RSTUDIO_AUTH="${PWD}/bin/auth"

# Generate an `rsession` wrapper script
export RSESSION_WRAPPER_FILE="${PWD}/rsession.sh"
(
umask 077
sed 's/^ \{2\}//' > "${RSESSION_WRAPPER_FILE}" << EOL
  #!/usr/bin/env bash

  # Log all output from this script
  export RSESSION_LOG_FILE="${PWD}/rsession.log"

  exec &>>"\${RSESSION_LOG_FILE}"

  # Launch the original command
  echo "Launching rsession..."
  set -x
  exec rsession --r-libs-user "${R_LIBS_USER}" "\${@}"
EOL
)
chmod 700 "${RSESSION_WRAPPER_FILE}"

# Set working directory to home directory
cd "${HOME}"

export TMPDIR="$(mktemp -d)"
export TMPDIR="$(mktemp -d)"

mkdir -p "$TMPDIR/rstudio-server"
python -c 'from uuid import uuid4; print(uuid4())' > "$TMPDIR/rstudio-server/secure-cookie-key"
chmod 0600 "$TMPDIR/rstudio-server/secure-cookie-key"

module unload python

set -x
# Launch the RStudio Server
echo "Starting up rserver..."

singularity run -B "$TMPDIR:/tmp" "$RSTUDIO_SERVER_IMAGE" \
 --www-port "${port}" \
 --auth-none 0 \
 --auth-pam-helper-path "${RSTUDIO_AUTH}" \
 --auth-encrypt-password 0 \
 --rsession-path "${RSESSION_WRAPPER_FILE}"

echo 'Singularity has exited...'```

[alanc]

Have you looked at any of the other threads on here about Rstudio? A lot of people have been able to get it working, but there always seem to be certain containerization or authentication issues that crop up. Most of the threads have solutions posted. For example:

Rserver in the Singularity container vs rstudio-server installed from rpm

Hi. We are trying to install the RStudio app following the successful model of Jupyter. After starting a session the output log has: FATAL: exec /.singularity.d/actions/run failed: no such file or directory The singularity container was created using “singularity pull”. It seems to call “rserver” but we installed RStudio on each node using the rpm which uses “rstudio-server” as the executable. Am I missing something obvious? Thanks Chris

Rstudio app works BUT each session begins with a single auth failure that must be ignored

Hiya everyone, We’re running centos 6 and slurm 19, and are one OOD release behind. I just setup the RStudio interactive app but am quite perplexed by this reproducible and ignorable authentication error that follows immediately after clicking “Connect to RStudio Server” for each session: “Error: Incorrect or invalid username/password” [2020-01-08 17_14_49-RStudio Sign In] One can simply go back to the OOD tab and click the button again (with or without closing this goofy error tab) and ev…

RStudio without Singularity?

I thought I’d read that OOD was now able to run RStudio without relying on Singularity but can’t find documentation to support that notion. Was I dreaming?? Thanks - Susan Litzinger PSC

[eisenefaust]

Hello, that was not the solution. It was part of the solution. Authorization is still not functioning. I get to the login and the username/password aren’t getting validated to allow me to log in.when I type them in manually, it also doesn’t work.
This is following the button Connect to RStudio Server:

[eisenefaust]

This is where it fails after I launch it.

I can’t add new posts to this topic so I’m editing this post:

I mounted /home
I updated the template/bin/auth to include the changes for rstudio server 1.3 support
still failing at authorization phase.

I installed within the container:
R-4.0.0.tar.gz to /R/4.0.0
rstudio-server-rhel-1.3.959-x86_64.rpm to default location: /usr/lib/rstudio-server/bin

Still have auth issues. Any other ideas would be greatly appreciated.

[jeff.ohrstrom]

I’d ensure that HOME is mounted too. You seem to be passing credentials OK, but can the container find this file `--auth-pam-helper-path "{RSTUDIO_AUTH}"` (It ends up being ~/ondemand/data/sys/dashboard/batch_connect/sys/rstudio/output//bin/auth)?

[maflister]

What version of RStudio Server are you using? We noticed that the new 1.3 version fails to authenticate with “Error: Incorrect or invalid username/password”. We just didn’t get around to posting the issue yet.

[dugan]

Hi,

The authentication issue in 1.3 is due to a change in how the auth-pam-helper script is invoked by rserver. Previously, it was called with just one argument, $USERNAME.
In 1.3 it is called with three arguments, “$USERNAME rstudio 1”. The test in line 4 of the bin/auth script:

#!/usr/bin/env bash

# Confirm username is supplied
if [[ $# -ne 1 ]]; then
  echo "Usage: auth USERNAME"
  exit 1
fi
...

fails and needs to be modified to handle the extra arguments. I just replaced “-ne” with “-lt” to get authentication to work.

[mario]

@dugan Thanks for this!

I opened an issue to get this merged into bc_osc_rstudio_server here https://github.com/OSC/bc_osc_rstudio_server/issues/34

[charles8ronson]

Onbehalf of eisenfaust (can only do 3 posts atm):

I mounted /home
I updated the template/bin/auth to include the changes for rstudio server 1.3 support
still failing at authorization phase.

I installed within the container:
R-4.0.0.tar.gz to /R/4.0.0
rstudio-server-rhel-1.3.959-x86_64.rpm to default location: /usr/lib/rstudio-server/bin

Still have auth issues

[maflister]

Thanks for this. We’ve now fully remove Singularity from our RStudio Server app.

[maflister]

Try pointing RStudio Server to your /tmp.

singularity run -B "$TMPDIR:/tmp" "$RSTUDIO_SERVER_IMAGE" \
--www-port "${port}" \
--auth-none 0 \
--auth-pam-helper-path "${RSTUDIO_AUTH}" \
--auth-encrypt-password 0 \
--rsession-path "${RSESSION_WRAPPER_FILE}" \
--server-data-dir /tmp

[eisenefaust]

Unfortunately that didn’t solve my issue for the authorization issue I’m currently having.
It does seem to be a potentially good way to not have to map /tmp:/var/run. If we end up doing this, I think we will need to also supply:
--server-pid-file arg (=/var/run/rstudio-server.pid)
probably something like:
--server-pid-file /tmp/rstudio-server.pid

The issue with the /var/run may be due to write permissions since it was built with fakeroot:
singularity build -f

Would love to work out this Authorization issue though.

[holtgrewe]

Hi, are you certain that you need to have this bind?

We’re using the Singularity Hub RStudio image

• https://singularity-hub.org/collections/463

with this configuration

• https://github.com/bihealth/ood-bih-rstudio-server

In this configuration, I’m only binding /fast which is our GPFS file system mount which contains the user’s homes.

[ssivy]

Did you try using the --writable-tmpfs option when launching singularity?

[eisenefaust]

Without the bind, it fails to open the port.

The --writable-tmpfs is interesting to me but it fails to open a port with that added to the singularity run in the script.

version 1.1.463 works fine for me when I use R 3.6.3
But rstudio server version 1.3.959 fails to authenticate. I’m using R 4.0.0 in this image

[jeff.ohrstrom]

I would say next steps are answering these two questions. Can the job find the $RSTUDIO_AUTH in the container (if you’re not mounting home as an example)? Then is the $RSTUDIO_AUTH working as it’s expected?

1. can it find the $RSTUDIO_AUTH file. To debug this, I’d write a simple wrapper like this. This ensures that the file exists in the container and you’re correctly mounting HOME.

#!/bin/bash

# load any modules you may need
module purge
module restore
module load ondemand
module load app_rstudio_server/3.6.3 rstudio_launcher/centos7

TMPDIR=$(mktemp -d)

# replace with your file location
RSTUDIO_AUTH="/users/PZS0714/johrstrom/ondemand/data/sys/dashboard/batch_connect/sys/bc_osc_rstudio_server/output/72ca052a-6af0-47dd-a521-0b2ed966170e/bin/auth"

# notice I'm using exec here (so i can execute ls), but it's the same arguments we use 
SINGULARITYENV_LD_LIBRARY_PATH="$LD_LIBRARY_PATH" singularity exec --contain -B "$TMPDIR:/tmp" \
 "$RSTUDIO_SERVER_IMAGE" \
 ls -lrt $RSTUDIO_AUTH

2. The right env variables are being populated checked in the RSTUDIO_AUTH file. I’d say modify the $RSTUDIO_AUTH file to redirect output to a log file. It looks like stdout and stderr from this script are being eaten, so let’s redirect it to a file to see where exactly it’s failing. The $RSTUDIO_PASSWORD is created in the before.sh and exported from there. And $USER, is well, the system $USER.