I’m trying to implement RStudio server 4.3.2 with a Rocker Singularity image and the OSC OOD RStudio app, reverting back to before bwrap is used and heavily modifying to remove all the options not applicable to our environment, with no success.
I’m wondering if anyone has implemented the Rocker RStudio 4.3.2 image, and/or if there is an OOD RStudio app that’s stripped down with just the essentials to run one RStudio singularity image.
Seems like our production application with singularity should work for you. What are the errors you’re seeing and maybe I can help debug them.
Here is my current output.log file.
Script starting...
Waiting for RStudio Server to open port 63571...
+ [[ x != \x ]]
+ set +x
Currently Loaded Modules:
1) singularity-rstudio/modulefiles/rstudio_singularity/4.3.2 2) r/4.3.2
c108.mgmt.hellbender
mkdir: cannot create directory ‘/tmp/run’: File exists
mkdir: cannot create directory ‘/tmp/lib’: File exists
++ date
+ echo 'Starting up rserver... at Fri Mar 22 14:21:19 CDT 2024'
Starting up rserver... at Fri Mar 22 14:21:19 CDT 2024
+ export APPTAINERENV_LD_LIBRARY_PATH=/cluster/software/common/R/4.3.2/dependencies/glib/lib:/cluster/software/common/R/4.3.2/dependencies/cairo/lib:/cluster/software/common/R/4.3.2/dependencies/nlopt/lib:/cluster/software/common/R/4.3.2/dependencies/libjpeg-turbo/lib:/cluster/software/common/R/4.3.2/dependencies/libxml2/lib:/cluster/software/common/R/4.3.2/lib
+ APPTAINERENV_LD_LIBRARY_PATH=/cluster/software/common/R/4.3.2/dependencies/glib/lib:/cluster/software/common/R/4.3.2/dependencies/cairo/lib:/cluster/software/common/R/4.3.2/dependencies/nlopt/lib:/cluster/software/common/R/4.3.2/dependencies/libjpeg-turbo/lib:/cluster/software/common/R/4.3.2/dependencies/libxml2/lib:/cluster/software/common/R/4.3.2/lib
+ export SINGULARITYENV_LD_LIBRARY_PATH=/cluster/software/common/R/4.3.2/dependencies/glib/lib:/cluster/software/common/R/4.3.2/dependencies/cairo/lib:/cluster/software/common/R/4.3.2/dependencies/nlopt/lib:/cluster/software/common/R/4.3.2/dependencies/libjpeg-turbo/lib:/cluster/software/common/R/4.3.2/dependencies/libxml2/lib:/cluster/software/common/R/4.3.2/lib
+ SINGULARITYENV_LD_LIBRARY_PATH=/cluster/software/common/R/4.3.2/dependencies/glib/lib:/cluster/software/common/R/4.3.2/dependencies/cairo/lib:/cluster/software/common/R/4.3.2/dependencies/nlopt/lib:/cluster/software/common/R/4.3.2/dependencies/libjpeg-turbo/lib:/cluster/software/common/R/4.3.2/dependencies/libxml2/lib:/cluster/software/common/R/4.3.2/lib
+ export APPTAINERENV_PATH=/cluster/software/common/R/4.3.2/dependencies/glib/bin:/cluster/software/common/R/4.3.2/dependencies/cairo/bin:/cluster/software/common/R/4.3.2/dependencies/libjpeg-turbo/bin:/cluster/software/common/R/4.3.2/dependencies/libxml2/bin:/cluster/software/common/R/4.3.2/bin:/cluster/software/modulefiles/common/singularity-rstudio/modulefiles/rstudio_singularity/bin:/home/redmonp/bin:/home/redmonp/.local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin:/cluster/software/user-scripts:/opt/dell/srvadmin/bin:/usr/lib/rstudio-server/bin
+ APPTAINERENV_PATH=/cluster/software/common/R/4.3.2/dependencies/glib/bin:/cluster/software/common/R/4.3.2/dependencies/cairo/bin:/cluster/software/common/R/4.3.2/dependencies/libjpeg-turbo/bin:/cluster/software/common/R/4.3.2/dependencies/libxml2/bin:/cluster/software/common/R/4.3.2/bin:/cluster/software/modulefiles/common/singularity-rstudio/modulefiles/rstudio_singularity/bin:/home/redmonp/bin:/home/redmonp/.local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin:/cluster/software/user-scripts:/opt/dell/srvadmin/bin:/usr/lib/rstudio-server/bin
+ export SINGULARITYENV_PATH=/cluster/software/common/R/4.3.2/dependencies/glib/bin:/cluster/software/common/R/4.3.2/dependencies/cairo/bin:/cluster/software/common/R/4.3.2/dependencies/libjpeg-turbo/bin:/cluster/software/common/R/4.3.2/dependencies/libxml2/bin:/cluster/software/common/R/4.3.2/bin:/cluster/software/modulefiles/common/singularity-rstudio/modulefiles/rstudio_singularity/bin:/home/redmonp/bin:/home/redmonp/.local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin:/cluster/software/user-scripts:/opt/dell/srvadmin/bin:/usr/lib/rstudio-server/bin
+ SINGULARITYENV_PATH=/cluster/software/common/R/4.3.2/dependencies/glib/bin:/cluster/software/common/R/4.3.2/dependencies/cairo/bin:/cluster/software/common/R/4.3.2/dependencies/libjpeg-turbo/bin:/cluster/software/common/R/4.3.2/dependencies/libxml2/bin:/cluster/software/common/R/4.3.2/bin:/cluster/software/modulefiles/common/singularity-rstudio/modulefiles/rstudio_singularity/bin:/home/redmonp/bin:/home/redmonp/.local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin:/cluster/software/user-scripts:/opt/dell/srvadmin/bin:/usr/lib/rstudio-server/bin
++ whoami
+ singularity run --contain -B /tmp:/tmp -B /home/redmonp/ondemand/data/sys/dashboard/batch_connect/dev/bc_osc_rstudio_server/output/3b52e53a-cd20-4a8d-9b1e-757ada5de43d -B /home/redmonp/ondemand/data/sys/dashboard/batch_connect/dev/bc_osc_rstudio_server/output/3b52e53a-cd20-4a8d-9b1e-757ada5de43d/etc:/etc/rstudio -B /etc/fonts -B /etc/profile -B /etc/profile.d -B /etc/alternatives -B false '' --www-port 63571 --auth-none 0 --auth-pam-helper-path /home/redmonp/ondemand/data/sys/dashboard/batch_connect/dev/bc_osc_rstudio_server/output/3b52e53a-cd20-4a8d-9b1e-757ada5de43d/bin/auth --auth-encrypt-password 0 --rsession-path /home/redmonp/ondemand/data/sys/dashboard/batch_connect/dev/bc_osc_rstudio_server/output/3b52e53a-cd20-4a8d-9b1e-757ada5de43d/rsession.sh --server-data-dir=/tmp/run --server-user=redmonp
INFO: Cleanup error: while unmounting /var/lib/apptainer/mnt/session/final directory: no such file or directory, while unmounting /var/lib/apptainer/mnt/session/rootfs directory: no such file or directory
FATAL: container creation failed: unable to add /home/redmonp/ondemand/data/sys/dashboard/batch_connect/dev/bc_osc_rstudio_server/output/3b52e53a-cd20-4a8d-9b1e-757ada5de43d/false to mount list: destination must be an absolute path
What is this last mount point supposed to be? It’s complaining about this not being real.
Looks like I had removed the conditional for module_dir. I’ve added it back in with a hardcoded value.
module_dir = '/usr'
...
-B <%= module_dir %> \
Now I’m getting the following.
+ singularity run --contain -B /tmp:/tmp -B /home/redmonp/ondemand/data/sys/dashboard/batch_connect/dev/bc_osc_rstudio_server/output/f42de6a4-1a0b-4e28-ad1d-6cb2533562c5 -B /home/redmonp/ondemand/data/sys/dashboard/batch_connect/dev/bc_osc_rstudio_server/output/f42de6a4-1a0b-4e28-ad1d-6cb2533562c5/etc:/etc/rstudio -B /etc/fonts -B /etc/profile -B /etc/profile.d -B /etc/alternatives -B /usr '' --www-port 12158 --auth-none 0 --auth-pam-helper-path /home/redmonp/ondemand/data/sys/dashboard/batch_connect/dev/bc_osc_rstudio_server/output/f42de6a4-1a0b-4e28-ad1d-6cb2533562c5/bin/auth --auth-encrypt-password 0 --rsession-path /home/redmonp/ondemand/data/sys/dashboard/batch_connect/dev/bc_osc_rstudio_server/output/f42de6a4-1a0b-4e28-ad1d-6cb2533562c5/rsession.sh --server-data-dir=/tmp/run --server-user=redmonp
WARNING: passwd file doesn't exist in container, not updating
WARNING: group file doesn't exist in container, not updating
FATAL: container creation failed: mount hook function failure: mount /var/lib/apptainer/mnt/session/dev->/dev error: while mounting /var/lib/apptainer/mnt/session/dev: destination /dev doesn't exist in container
It doesn’t like the fact that /dev isn’t inside the container - but looking at this very last entry in your output I don’t know what .sif file you’re trying to use to replicate. Did you obsfucate the .sif image or is that your actual command?
If that’s your actual command output - it doesn’t appear to be specifying the sif image correctly and I guess singularity has a default image it’s trying to run or something?
Good catch. The submit script was using the variable RSTUDIO_SERVER_IMAGE and the modulefile was setting the variable to be RSTUDIO_SINGULARITY_IMAGE. I’ve updated the submit script to match the modulefile variable.
Log below
Script starting...
Waiting for RStudio Server to open port 35407...
here's the image
+ [[ x != \x ]]
+ set +x
Currently Loaded Modules:
1) singularity-rstudio/4.3.2 2) r/4.3.2
c061.mgmt.hellbender
mkdir: cannot create directory ‘/tmp/run’: File exists
mkdir: cannot create directory ‘/tmp/lib’: File exists
++ date
+ echo 'Starting up rserver... at Fri Mar 22 15:26:17 CDT 2024'
Starting up rserver... at Fri Mar 22 15:26:17 CDT 2024
+ export APPTAINERENV_LD_LIBRARY_PATH=/cluster/software/common/R/4.3.2/dependencies/glib/lib:/cluster/software/common/R/4.3.2/dependencies/cairo/lib:/cluster/software/common/R/4.3.2/dependencies/nlopt/lib:/cluster/software/common/R/4.3.2/dependencies/libjpeg-turbo/lib:/cluster/software/common/R/4.3.2/dependencies/libxml2/lib:/cluster/software/common/R/4.3.2/lib
+ APPTAINERENV_LD_LIBRARY_PATH=/cluster/software/common/R/4.3.2/dependencies/glib/lib:/cluster/software/common/R/4.3.2/dependencies/cairo/lib:/cluster/software/common/R/4.3.2/dependencies/nlopt/lib:/cluster/software/common/R/4.3.2/dependencies/libjpeg-turbo/lib:/cluster/software/common/R/4.3.2/dependencies/libxml2/lib:/cluster/software/common/R/4.3.2/lib
+ export SINGULARITYENV_LD_LIBRARY_PATH=/cluster/software/common/R/4.3.2/dependencies/glib/lib:/cluster/software/common/R/4.3.2/dependencies/cairo/lib:/cluster/software/common/R/4.3.2/dependencies/nlopt/lib:/cluster/software/common/R/4.3.2/dependencies/libjpeg-turbo/lib:/cluster/software/common/R/4.3.2/dependencies/libxml2/lib:/cluster/software/common/R/4.3.2/lib
+ SINGULARITYENV_LD_LIBRARY_PATH=/cluster/software/common/R/4.3.2/dependencies/glib/lib:/cluster/software/common/R/4.3.2/dependencies/cairo/lib:/cluster/software/common/R/4.3.2/dependencies/nlopt/lib:/cluster/software/common/R/4.3.2/dependencies/libjpeg-turbo/lib:/cluster/software/common/R/4.3.2/dependencies/libxml2/lib:/cluster/software/common/R/4.3.2/lib
+ export APPTAINERENV_PATH=/cluster/software/common/R/4.3.2/dependencies/glib/bin:/cluster/software/common/R/4.3.2/dependencies/cairo/bin:/cluster/software/common/R/4.3.2/dependencies/libjpeg-turbo/bin:/cluster/software/common/R/4.3.2/dependencies/libxml2/bin:/cluster/software/common/R/4.3.2/bin:/cluster/software/modulefiles/common/singularity-rstudio/bin:/home/redmonp/bin:/home/redmonp/.local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin:/cluster/software/user-scripts:/opt/dell/srvadmin/bin:/usr/lib/rstudio-server/bin
+ APPTAINERENV_PATH=/cluster/software/common/R/4.3.2/dependencies/glib/bin:/cluster/software/common/R/4.3.2/dependencies/cairo/bin:/cluster/software/common/R/4.3.2/dependencies/libjpeg-turbo/bin:/cluster/software/common/R/4.3.2/dependencies/libxml2/bin:/cluster/software/common/R/4.3.2/bin:/cluster/software/modulefiles/common/singularity-rstudio/bin:/home/redmonp/bin:/home/redmonp/.local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin:/cluster/software/user-scripts:/opt/dell/srvadmin/bin:/usr/lib/rstudio-server/bin
+ export SINGULARITYENV_PATH=/cluster/software/common/R/4.3.2/dependencies/glib/bin:/cluster/software/common/R/4.3.2/dependencies/cairo/bin:/cluster/software/common/R/4.3.2/dependencies/libjpeg-turbo/bin:/cluster/software/common/R/4.3.2/dependencies/libxml2/bin:/cluster/software/common/R/4.3.2/bin:/cluster/software/modulefiles/common/singularity-rstudio/bin:/home/redmonp/bin:/home/redmonp/.local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin:/cluster/software/user-scripts:/opt/dell/srvadmin/bin:/usr/lib/rstudio-server/bin
+ SINGULARITYENV_PATH=/cluster/software/common/R/4.3.2/dependencies/glib/bin:/cluster/software/common/R/4.3.2/dependencies/cairo/bin:/cluster/software/common/R/4.3.2/dependencies/libjpeg-turbo/bin:/cluster/software/common/R/4.3.2/dependencies/libxml2/bin:/cluster/software/common/R/4.3.2/bin:/cluster/software/modulefiles/common/singularity-rstudio/bin:/home/redmonp/bin:/home/redmonp/.local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin:/cluster/software/user-scripts:/opt/dell/srvadmin/bin:/usr/lib/rstudio-server/bin
++ whoami
+ singularity run --contain -B /tmp:/tmp -B /home/redmonp/ondemand/data/sys/dashboard/batch_connect/dev/bc_osc_rstudio_server/output/1a173d69-4adf-4295-a975-7b04e5d65b09 -B /home/redmonp/ondemand/data/sys/dashboard/batch_connect/dev/bc_osc_rstudio_server/output/1a173d69-4adf-4295-a975-7b04e5d65b09/etc:/etc/rstudio -B /etc/fonts -B /etc/profile -B /etc/profile.d -B /etc/alternatives -B /usr /cluster/software/modulefiles/common/singularity-rstudio/4.3.2/singularity-rstudio.simg --www-port 35407 --auth-none 0 --auth-pam-helper-path /home/redmonp/ondemand/data/sys/dashboard/batch_connect/dev/bc_osc_rstudio_server/output/1a173d69-4adf-4295-a975-7b04e5d65b09/bin/auth --auth-encrypt-password 0 --rsession-path /home/redmonp/ondemand/data/sys/dashboard/batch_connect/dev/bc_osc_rstudio_server/output/1a173d69-4adf-4295-a975-7b04e5d65b09/rsession.sh --server-data-dir=/tmp/run --server-user=redmonp
set: invalid option: "--"
FATAL: "--www-port": executable file not found in $PATH
Yea so the first thing after the image is the executable you want to run. I think you need to issue the rserver command with --www-port as the first option like:
... singularity-rstudio.simg rserver --www-port 35407 ...
Right now my submit is a pared down version of the commit below.
singularity run --contain \
-B $TMPDIR:/tmp \
-B $WORKING_DIR \
-B $WORKING_DIR/etc:/etc/rstudio \
-B /etc/fonts \
-B /etc/profile \
-B /etc/profile.d \
-B /etc/alternatives \
-B <%= module_dir %> \
"$RSTUDIO_SINGULARITY_IMAGE" \
--www-port "${port}" \
--auth-none 0 \
--auth-pam-helper-path "${RSTUDIO_AUTH}" \
--auth-encrypt-password 0 \
--rsession-path "${RSESSION_WRAPPER_FILE}" \
--server-data-dir='/tmp/run' \
--server-user=$(whoami)
I tried putting an rserver in there and get
FATAL: "rserver": executable file not found in $PATH
I somehow got to a login screen. No idea why though. My submit looks like this right now.
singularity exec --cleanenv \
-B $WORKING_DIR/etc:/etc/rstudio \
"$RSTUDIO_SINGULARITY_IMAGE" \
/usr/lib/rstudio-server/bin/rserver \
--www-port "${port}" \
--auth-none 0 \
--auth-pam-helper-path "${RSTUDIO_AUTH}" \
--auth-encrypt-password 0 \
--rsession-path "${RSESSION_WRAPPER_FILE}" \
--server-data-dir='/tmp/run' \
--server-user=$(whoami)
That seems OK. What I think you need now is CSRF support for correctly authenticating to Rstudio. You can see this pull request on how we added the same for our production application, there are also likely a lot similar discourse topics for the same.
Narrowed it down to -B <%= module_dir %> \ being the culprit. I tried -B /usr \ but that didn’t work so I just removed the line.
I was missing the conn_params in the submit script, and readded it there. That gets me past the login page. I’m getting Can't connect to R session now. Below is the rserver.log.
2024-03-26T17:07:16.385281Z [rserver] INFO No rserver configuration found at /etc/rstudio/rserver.conf
2024-03-26T17:07:16.388558Z [rserver] DEBUG Adding user to cache for user: redmonp:116770 by name
2024-03-26T17:07:16.389078Z [rserver] INFO Running as server user 'redmonp' (without privilege)
2024-03-26T17:07:16.389972Z [rserver] DEBUG (PID 2537998): Initialized file locks (lock-type=linkbased, use-symlinks=false, timeout-interval=30s, refresh-rate=20s, log-file=)
2024-03-26T17:07:16.390472Z [rserver] INFO Running without privilege; generating secure key at /home/redmonp/.cache/rstudio/secure-cookie-key
2024-03-26T17:07:16.390975Z [rserver] DEBUG Creating secure key file: "/home/redmonp/.cache/rstudio/secure-cookie-key" key hash: (23413AE9)
2024-03-26T17:07:16.391635Z [rserver] INFO Reading database configuration from '/etc/rstudio/database.conf'
2024-03-26T17:07:16.392391Z [rserver] INFO Connecting to sqlite3 database at /tmp/lib/rstudio-server/rstudio-os.sqlite
2024-03-26T17:07:16.392853Z [rserver] INFO Creating database connection pool of size 6 (source: default maximum with 6 CPUs)
2024-03-26T17:07:16.400206Z [rserver] DEBUG Current Database Schema Version: 20220610180402900844286 : Spotted Wakerobin
2024-03-26T17:07:16.400702Z [rserver] DEBUG Highest Available Database Schema Version: 20220610180402900844286 : Spotted Wakerobin
2024-03-26T17:07:16.401133Z [rserver] INFO Database schema version is up to date.
2024-03-26T17:07:16.402403Z [rserver] INFO No environment variables found at /etc/rstudio/env-vars
2024-03-26T17:07:16.402893Z [rserver] INFO Running without privilege; generating secure key at /home/redmonp/.cache/rstudio/session-rpc-key
2024-03-26T17:07:16.403333Z [rserver] DEBUG Creating secure key file: "/home/redmonp/.cache/rstudio/session-rpc-key" key hash: (9663F975)
2024-03-26T17:07:16.404107Z [rserver] INFO Running without privilege; using secure key at /home/redmonp/.cache/rstudio/session-rpc-key
2024-03-26T17:07:16.404534Z [rserver] DEBUG Using secure key file: /home/redmonp/.cache/rstudio/session-rpc-key
2024-03-26T17:07:16.404991Z [rserver] DEBUG Secure key hash: (9663F975)
2024-03-26T17:07:16.446683Z [rserver] ERROR R did not return any output when queried for directory location information; LOGGED FROM: bool rstudio::core::r_util::{anonymous}::detectRLocationsUsingR(const string&, rstudio::core::FilePath*, rstudio::core::FilePath*, rstudio::core::config_utils::Variables*, std::string*, const string&, const string&, const rstudio::core::FilePath&) src/cpp/core/r_util/REnvironmentPosix.cpp:539
2024-03-26T17:07:16.470603Z [rserver] ERROR system error 71 (Protocol error) [description: Unable to parse version from R, version-info: , r-error: /usr/local/lib/R/bin/exec/R: error while loading shared libraries: libblas.so.3: cannot open shared object file: No such file or directory
]; OCCURRED AT rstudio::core::Error rstudio::core::r_util::rVersion(const rstudio::core::FilePath&, const rstudio::core::FilePath&, const string&, const string&, const string&, const rstudio::core::FilePath&, std::string*) src/cpp/core/r_util/REnvironmentPosix.cpp:947; LOGGED FROM: bool rstudio::core::r_util::detectREnvironment(const rstudio::core::FilePath&, const rstudio::core::FilePath&, const string&, std::string*, std::string*, rstudio::core::r_util::EnvironmentVars*, std::string*, const string&, const string&, const rstudio::core::FilePath&) src/cpp/core/r_util/REnvironmentPosix.cpp:790
2024-03-26T17:07:28.981154Z [rserver] DEBUG Caching auth userIdentifier: redmonp for system username: redmonp
2024-03-26T17:07:28.982689Z [rserver] DEBUG PAM login start - running: /home/redmonp/ondemand/data/sys/dashboard/batch_connect/dev/bc_osc_rstudio_server/output/385e8a4e-744b-4795-98bf-dac3b61594ee/bin/auth redmonp rstudio 1 <pw>
2024-03-26T17:07:28.988179Z [rserver] DEBUG PAM login result: for username: redmonp returns: authenticated
2024-03-26T17:07:28.988690Z [rserver] DEBUG User db query start: redmonp
2024-03-26T17:07:28.989368Z [rserver] DEBUG No record found for user in db: redmonp
2024-03-26T17:07:28.989841Z [rserver] DEBUG Adding user to database: redmonp
2024-03-26T17:07:28.990660Z [rserver] DEBUG Begin update last signIn time
2024-03-26T17:07:28.991147Z [rserver] DEBUG End update last signIn time
2024-03-26T17:07:28.992181Z [rserver] DEBUG Created new UserSession for: redmonp (total: 1)
2024-03-26T17:07:28.992654Z [rserver] DEBUG User db query start: redmonp
2024-03-26T17:07:28.993249Z [rserver] DEBUG Found user in db: redmonp
2024-03-26T17:07:28.993719Z [rserver] DEBUG Adding session cookie: redmonp|Tue%2C%2026%20Mar%202024%2018%3A07%3A28%20GMT|FhDDBXeNAWWHgX%2F%2FTpLTshALXLzRbqsEGqRu%2FBgV%2BgM%3D for user: redmonp
2024-03-26T17:07:28.994184Z [rserver] DEBUG Sign in for user: redmonp redirecting to: /
2024-03-26T17:07:31.846033Z [rserver] DEBUG - Start server proxy request POST /rpc/client_init (redmonp:18) for local stream: /tmp/run/rstudio-rsession/redmonp-d
2024-03-26T17:07:31.847009Z [rserver] DEBUG Launching session for user: redmonp id:
2024-03-26T17:07:31.847531Z [rserver] DEBUG Launching RStudio session for: redmonp id:
2024-03-26T17:07:31.848488Z [rserver] DEBUG Launched session process for user: : /home/redmonp/ondemand/data/sys/dashboard/batch_connect/dev/bc_osc_rstudio_server/output/385e8a4e-744b-4795-98bf-dac3b61594ee/rsession.sh pid: 2538123
2024-03-26T17:07:41.871175Z [rserver] DEBUG -- rpc error: connection unavailable for: /rpc/client_init (redmonp:18) in 10.25s error: system error 2 (No such file or directory)
It can’t find this libblas.so.3 library file.
I added the following to the rsession.sh file
export LD_LIBRARY_PATH="$LD_LIBRARY_PATH:/usr/lib64"
and am now getting in my rsession.log file
rsession: error while loading shared libraries: libR.so: cannot open shared object file: No such file or directory
Getting closer. Google is implying that I need to reinstall R with the --enable-R-shlib flag enabled as something to test.
Added path to R in rsession $PATH
export PATH="$PATH:/cluster/software/common/R/4.3.2/lib64/R"
and now I’m getting in my rsession.log
/home/redmonp/ondemand/data/sys/dashboard/batch_connect/dev/bc_osc_rstudio_server/output/8d460762-dc85-44a9-bfc5-b0d34941846f/rsession.sh: line 25: exec: rsession: not found
I managed to get no errors in the logs. I realized that I was trying to use an outside version of R instead of the R provided by the container. I added the lib paths to those and no errors in rsession, but I’m still unable to connect.
Do you still get prompted with the login screen? I’d still suspect there’s an issue with CSRF tokens.
I’m getting past the auth screen. It loads for a while and then I get a modal with:
rsession.log below
+ export R_LIBS_SITE=
+ R_LIBS_SITE=
+ export TZ=US/Eastern
+ TZ=US/Eastern
+ export HOME=
+ HOME=
+ export MODULEPATH_ROOT=/usr/share/modulefiles
+ MODULEPATH_ROOT=/usr/share/modulefiles
+ export MODULEPATH=/etc/modulefiles:/usr/share/modulefiles:/usr/share/lmod/lmod/modulefiles/Core:/cluster/software/modulefiles/common:/cluster/software/modulefiles/amd
+ MODULEPATH=/etc/modulefiles:/usr/share/modulefiles:/usr/share/lmod/lmod/modulefiles/Core:/cluster/software/modulefiles/common:/cluster/software/modulefiles/amd
+ export LMOD_PKG=/usr/share/lmod/lmod
+ LMOD_PKG=/usr/share/lmod/lmod
+ export R_LIBS_USER=/home/redmonp/R/rocker-rstudio/4.3.2
+ R_LIBS_USER=/home/redmonp/R/rocker-rstudio/4.3.2
+ export LD_LIBRARY_PATH=:/usr/lib/x86_64-linux-gnu:/usr/local/lib/R/lib:/usr/lib/x86_64-linux-gnu/openblas-pthread
+ LD_LIBRARY_PATH=:/usr/lib/x86_64-linux-gnu:/usr/local/lib/R/lib:/usr/lib/x86_64-linux-gnu/openblas-pthread
+ env
SHELL=/bin/bash
RSTUDIO_DEFAULT_R_VERSION_HOME=/usr/local/lib/R
R_HOME=/usr/local/lib/R
RS_SESSION_TMP_DIR=/tmp/run/rstudio-rsession
RS_SERVER_RPC_SOCKET_PATH=/tmp/run/rstudio-rserver/session-server-rpc.socket
RSTUDIO_R_MODULE=
RSTUDIO_LIMIT_RPC_CLIENT_UID=116770
PWD=/
LOGNAME=redmonp
RSTUDIO_R_VERSION_LABEL=
MANPATH=/usr/share/lmod/lmod/share/man::/opt/puppetlabs/puppet/share/man
R_LIBS_SITE=
TZ=US/Eastern
RSTUDIO_DEFAULT_R_VERSION=
RSTUDIO_R_REPO=
HOME=
LANG=en_US.UTF-8
R_INCLUDE_DIR=/usr/local/lib/R/include
R_LIBS_USER=/home/redmonp/R/rocker-rstudio/4.3.2
RSTUDIO_MINIMUM_USER_ID=1000
MODULEPATH_ROOT=/usr/share/modulefiles
LMOD_PKG=/usr/share/lmod/lmod
USER=redmonp
R_DOC_DIR=/usr/local/lib/R/doc
RS_MONITOR_SHARED_SECRET=484bae76-0056-4b08-b5a4-f4642b022b72
SHLVL=1
RSESSION_LOG_FILE=/home/redmonp/ondemand/data/sys/dashboard/batch_connect/dev/bc_osc_rstudio_server/output/51d1ff4e-2750-4dfe-a63d-b9133a616d59/rsession.log
RSTUDIO_VERSION=2023.12.0+369
LD_LIBRARY_PATH=:/usr/lib/x86_64-linux-gnu:/usr/local/lib/R/lib:/usr/lib/x86_64-linux-gnu/openblas-pthread
RS_SESSION_SERVER_RPC_SECRET=74d168590cb84a96b431694c76ae188c
RSTUDIO_R_PRELAUNCH_SCRIPT=
R_SHARE_DIR=/usr/local/lib/R/share
XDG_DATA_DIRS=/home/redmonp/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share
PATH=/cluster/software/modulefiles/common/singularity-rstudio/bin:/home/redmonp/bin:/home/redmonp/.local/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/puppetlabs/bin:/cluster/software/user-scripts:/opt/dell/srvadmin/bin:/usr/lib/rstudio-server/bin
MODULEPATH=/etc/modulefiles:/usr/share/modulefiles:/usr/share/lmod/lmod/modulefiles/Core:/cluster/software/modulefiles/common:/cluster/software/modulefiles/amd
RSTUDIO_REQUIRED_USER_GROUP=
_=/usr/bin/env
+ echo 'Launching rsession...'
Launching rsession...
+ exec /usr/lib/rstudio-server/bin/rsession --r-libs-user /mnt/home/redmonp/R/library-4.3 -u redmonp --session-use-secure-cookies 1 --session-root-path / --session-same-site 0 --session-use-file-storage 1 --launcher-token 0DFE9029 --r-restore-workspace 2 --r-run-rprofile 2
Hmmmmm… OK. Maybe you can increase the logging level to debug to see if there’s anything more it has to say.
When I see stuff like this it’s generally rstudio is bogged down trying to do something like look at your ~/.Rdata folder or similar (I pulled that directory from memory, but Rstudio will try to start the app based on the last time you ran the app and load all the data from the last time in memory).
Are you able to see the resource usage of that job? I’d bet it’s pegged at 100% cpu or maybe in an io wait status.
What logs/logging level are you suggesting I try increasing?
I’ve checked my home dir and I don’t see any hidden .R* directories. I’ll see if I can find where that data is.
There is no iowait and near 0% cpu on the node.