Using the docs I installed a Jupyter app but it’s very unreliable whether it will start. I can log in and create a new python3 notebook, but usually I get the brown ‘connecting to kernel’ button in the upper left and it fails to connect. In the web browser I see many messages such as
kernel.js:465 WebSocket connection to 'wss://carina-ondemand.usc.edu/node/carina-10-123-0-3/43741/api/kernels/78146f6b-6494-42c4-ade1-376d6e2617b1/channels?session_id=b1faef4b44834b2f81843e141e6faf96' failed: Error during WebSocket handshake: Unexpected response code: 503 Kernel.start_channels @ kernel.js:465 Kernel.reconnect @ kernel.js:359 i @ jquery.min.js:2 kernel.js:106 Kernel: kernel_disconnected (78146f6b-6494-42c4-ade1-376d6e2617b1) kernel.js:546 WebSocket connection failed: wss://carina-ondemand.usc.edu/node/carina-10-123-0-3/43741/api/kernels/78146f6b-6494-42c4-ade1-376d6e2617b1 true
Often, I can log out and then log back in and I can connect to a saved notebook fine. Other times, the app will connect to the kernel but when I log out and log back in it won’t connect.
Starting up I get messages similar to
Script starting... Waiting for Jupyter Notebook server to open port 43741... TIMING - Starting main script at: Sat Mar 6 19:30:40 UTC 2021 TIMING - Starting wait at: Sat Mar 6 19:30:40 UTC 2021 ModuleCmd_Load.c(213):ERROR:105: Unable to locate a modulefile for 'python' No Modulefiles Currently Loaded. TIMING - Starting jupyter at: Sat Mar 6 19:30:40 UTC 2021 + jupyter notebook --config=/home/christay/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/197fc270-5132-43ec-ab73-9491e1dddd26/config.py [W 19:30:41.859 NotebookApp] WARNING: The notebook server is listening on all IP addresses and not using encryption. This is not recommended. [I 19:30:41.865 NotebookApp] Serving notebooks from local directory: /home/christay [I 19:30:41.865 NotebookApp] Jupyter Notebook 6.2.0 is running at: [I 19:30:41.865 NotebookApp] http://carina-10-123-0-3:43741/node/carina-10-123-0-3/43741/ [I 19:30:41.865 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). Discovered Jupyter Notebook server listening on port 43741! TIMING - Wait ended at: Sat Mar 6 19:30:42 UTC 2021 Generating connection YAML file... [I 19:30:57.708 NotebookApp] 302 POST /node/carina-10-123-0-3/43741/login (10.123.0.7) 2.690000ms [I 19:30:57.963 NotebookApp] 302 GET /node/carina-10-123-0-3/43741/ (10.123.0.7) 0.930000ms [I 19:31:56.058 NotebookApp] Creating new notebook in [I 19:32:00.513 NotebookApp] Kernel started: f6da1793-920a-4201-ab33-53c1bbc2a75f, name: python3 [I 19:32:55.788 NotebookApp] Saving file at /jupytertest1.ipynb [I 19:33:02.499 NotebookApp] Starting buffering for f6da1793-920a-4201-ab33-53c1bbc2a75f:39a5363159f2436d8d993d99e7f720a4 [W 19:33:08.145 NotebookApp] Forbidden [W 19:33:08.146 NotebookApp] 403 GET /node/carina-10-123-0-3/43741/api/sessions?_=1615059062031 (10.123.0.7) 2.410000ms referer=https://carina-ondemand.usc.edu/node/carina-10-123-0-3/43741/tree? [W 19:33:08.146 NotebookApp] Forbidden
I think name resolution is working fine, my OnDemand server can resolve the (unqualified) hostname of the compute node running the notebook server. Even if I start a desktop and web browser on the actual node running the notebook server process and log in through OnDemand I get errors connecting to the kernel.
I don’t have any host firewalls running or selinux enabled.
I’m utterly at a loss where to try to start to troubleshoot- I can’t seem to find any pattern in when it works and when it doesn’t. Can someone please suggest how to troubleshoot? Thanks.