We have a instance of Open OnDemand 1.80.20. I have been working to get Jupyter running but have encountered errors that I cannot resolve. It seems that there are exceptions being thrown in the jupyter-notebook script but the job continues to attempt to run but fails with Permission error on /run/user/user_id followed by a timeout opening the port (see log and config below). I haven’t seen anything in the discussions that covers this series of errors.
form.yml
---
# **MUST** set cluster id here that matches cluster configuration file located
# under /etc/ood/config/clusters.d/*.yml
cluster: "summit"
# Define attribute values that aren't meant to be modified by the user within
# the Dashboard form
attributes:
# Set the corresponding modules that need to be loaded for Jupyter to run
#
# @note It is called within the batch job as `module load <modules>` if
# defined
# @example Do not load any modules
# modules: ""
# @example Using default python module
# modules: "python"
# @example Using specific python module
# modules: "python/3.5"
# @example Using combination of modules
# modules: "python/3.5 cuda/8.0.44"
modules: "python/3.5.1"
# Any extra command line arguments to feed to the `jupyter notebook ...`
# command that launches the Jupyter notebook within the batch job
extra_jupyter_args: ""
# All of the attributes that make up the Dashboard form (in respective order),
# and made available to the submit configuration file and the template ERB
# files
#
# @note You typically do not need to modify this unless you want to add a new
# configurable value
# @note If an attribute listed below is hard-coded above in the `attributes`
# option, then it will not appear in the form page that the user sees in the
# Dashboard
form:
- modules
- extra_jupyter_args
- bc_account
- bc_queue
- bc_num_hours
- bc_num_slots
- bc_email_on_started
output.log
Script starting...
Waiting for Jupyter Notebook server to open port 15141...
TIMING - Starting wait at: Thu Jun 10 11:19:53 MDT 2021
TIMING - Starting main script at: Thu Jun 10 11:19:53 MDT 2021
Currently Loaded Modules:
1) python/3.5.1
TIMING - Starting jupyter at: Thu Jun 10 11:19:54 MDT 2021
+ jupyter notebook --config=/home/jasw8470/ondemand/data/sys/dashboard/batch_connect/dev/my_jupyter_app/output/565fc17f-bc4f-4ddb-811a-9cd90d1262ea/config.py
[W 11:20:04.603 NotebookApp] Config option `disable_check_xsrf` not recognized by `NotebookApp`.
[W 11:20:04.608 NotebookApp] Config option `disable_check_xsrf` not recognized by `NotebookApp`.
Traceback (most recent call last):
File "/curc/sw/python/3.5.1/lib/python3.5/site-packages/traitlets/traitlets.py", line 501, in get
value = obj._trait_values[self.name]
KeyError: 'runtime_dir'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/curc/sw/python/3.5.1/bin/jupyter-notebook", line 11, in <module>
sys.exit(main())
File "/curc/sw/python/3.5.1/lib/python3.5/site-packages/jupyter_core/application.py", line 267, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/curc/sw/python/3.5.1/lib/python3.5/site-packages/traitlets/config/application.py", line 595, in launch_instance
app.initialize(argv)
File "<decorator-gen-7>", line 2, in initialize
File "/curc/sw/python/3.5.1/lib/python3.5/site-packages/traitlets/config/application.py", line 74, in catch_config_error
return method(app, *args, **kwargs)
File "/curc/sw/python/3.5.1/lib/python3.5/site-packages/notebook/notebookapp.py", line 1069, in initialize
self.init_configurables()
File "/curc/sw/python/3.5.1/lib/python3.5/site-packages/notebook/notebookapp.py", line 842, in init_configurables
connection_dir=self.runtime_dir,
File "/curc/sw/python/3.5.1/lib/python3.5/site-packages/traitlets/traitlets.py", line 529, in __get__
return self.get(obj, cls)
File "/curc/sw/python/3.5.1/lib/python3.5/site-packages/traitlets/traitlets.py", line 508, in get
value = self._validate(obj, dynamic_default())
File "/curc/sw/python/3.5.1/lib/python3.5/site-packages/jupyter_core/application.py", line 99, in _runtime_dir_default
ensure_dir_exists(rd, mode=0o700)
File "/curc/sw/python/3.5.1/lib/python3.5/site-packages/ipython_genutils/path.py", line 167, in ensure_dir_exists
os.makedirs(path, mode=mode)
File "/curc/sw/python/3.5.1/lib/python3.5/os.py", line 231, in makedirs
makedirs(head, mode, exist_ok)
File "/curc/sw/python/3.5.1/lib/python3.5/os.py", line 241, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/run/user/751315'
Timed out waiting for Jupyter Notebook server to open port 15141!
TIMING - Wait ended at: Thu Jun 10 11:21:45 MDT 2021
Cleaning up...