Clicking on New ->Python3 in Jupyter notedbook (interactive app) gives Proxy Error message

Get Help
1

I am trying to install Jupyter notebook interactive app and using anaconda-2021.05. I am able to connect to Jupyter notebook without any problem and can see the contents of my directories. However, when I click on New → Python3 to create a new Jupyter notebook, I just get an empty new tab trying to load the page and after some time I get the message ‘Proxy Error’ and the new tab disappeared. The interactive desktop app is working fine and I tried to open ipython, jupyter notebook, spyder etc. there so there is no issue with the installation of the package.

Does anybody has any idea or any suggestions which may help me to resolve my issue?

Any help is highly appreciated.
Thanks
Pankaj

2

Hi and welcome! @travert please advise.

3

Hi,

Would it be possible to see the relevant sections of your /etc/ood/config/ood_portal.yml file to see how the host_regex and node_uri are configured?

The following docs will hopefully help:
https://osc.github.io/ood-documentation/latest/reference/files/ood-portal-yml.html#configure-reverse-proxy

https://osc.github.io/ood-documentation/latest/app-development/interactive/setup/enable-reverse-proxy.html

Also, if you made a changes there already make sure you also ran the usual sequence:
sudo /opt/ood/ood-portal-generator/sbin/update_ood_portal
sudo systemctl try-restart httpd24-httpd.service httpd24-htcacheclean.service #CentOS 7

To make sure the changes take effect. If those aren’t helpful or steps you’ve already taken let me know.

4

Hi Travis,

We have followed all the steps you mentioned. I have also double checked them. Also, session gets generated fine, even the home page of Jupyter notebook works fine. The issues starts only when I click on New → Python3

5

Do all the selections under the New → have this behavior? There should be a terminal option for instance, does this also end up as a ‘Proxy Error’ if you select it?

6

It is only Python3, which exhibit this behavior. All the commands under the other tag are working fine.

7

Thanks for the info. Hmm, have you had a chance to check the output.log for the Jupyter app to see if there’s a stack trace or something that would clue us in to what Jupyter is doing?

8

I couldn’t find anything specific. The last line I got was creating new notebook. After that there is nothing written in the output.log.

9

These are few line of the bottom of output.log file:

Discovered Jupyter Notebook server listening on port -----!
TIMING - Wait ended at: Mon Aug 2 12:32:02 EDT 2021
Generating connection YAML file…
[I 12:32:10.639 NotebookApp] 302 POST /node/nodename/13972/login (ip) 2.140000ms
[I 12:32:10.708 NotebookApp] 302 GET /node/node/13972/ (ip) 0.790000ms
[I 12:32:16.839 NotebookApp] New terminal with automatic name: 1
TermSocket.open: 1
TermSocket.open: Opened 1
Websocket closed
[I 12:48:23.650 NotebookApp] Creating new notebook in

10

Forgot to mention:
In the footer of the web page: I saw a message "file failed to load mathjax/extentions/safe.js

11

Would it be possible to see the output of the Apache logs as well at the times for the requests to launch the notebook? It might be an issue of just a timeout from the extensions not loading and those might give some insight.

12

What kind of information should I look for? It is pretty long file. I could match error and Jupyter together

13

Those would be good to start for a quick look, but I think a good idea would be to try and create a notebook from within Jupyter and do something like a tail -f on the logs if possible at the same time so you can see what happens in real time.

Or, if using a log management system try to correlate the time of the launch of the notebook with the logs you search through, and then narrow the logs down by removing pieces through search.

I’m hesitant to just say search only on warnings and errors because it may be that everything is fine aside from a resource failing to be fetched for the extension or something like that. Is the apache instance handling more than just OOD?

14

This is what I get when I try to open the Jupyter notebook

[Tue Aug 03 08:45:02.100894 2021] [proxy:error] [pid 32701] [client IP_address:51786] AH00898: Error reading from remote server returned by /node/comute-node01/45411/api/sessions, referer: https://OOD_address.edu/node/comute-node01/45411/tree?
[Tue Aug 03 08:46:02.052163 2021] [proxy_http:error] [pid 28777] (70007)The timeout specified has expired: [client IP_address:51796] AH01102: error reading status line from remote server comute-node01:45411, referer: https://OOD_address.edu/node/comute-node01/45411/tree?
[Tue Aug 03 08:46:02.052212 2021] [proxy:error] [pid 28777] [client IP_address:51796] AH00898: Error reading from remote server returned by /node/comute-node01/45411/api/sessions, referer: https://OOD_address.edu/node/comute-node01/45411/tree?
[Tue Aug 03 08:46:02.121206 2021] [proxy_http:error] [pid 2153] (70007)The timeout specified has expired: [client IP_address:51797] AH01102: error reading status line from remote server comute-node01:45411, referer: https://OOD_address.edu/node/comute-node01/45411/tree?
[Tue Aug 03 08:46:02.121250 2021] [proxy:error] [pid 2153] [client IP_address:51797] AH00898: Error reading from remote server returned by /node/comute-node01/45411/api/terminals, referer: https://OOD_address.edu/node/comute-node01/45411/tree?
[Tue Aug 03 08:47:02.121055 2021] [proxy_http:error] [pid 14223] (70007)The timeout specified has expired: [client IP_address:51811] AH01102: error reading status line from remote server comute-node01:45411, referer: https://OOD_address.edu/node/comute-node01/45411/tree?
[Tue Aug 03 08:47:02.121134 2021] [proxy:error] [pid 14223] [client IP_address:51811] AH00898: Error reading from remote server returned by /node/comute-node01/45411/api/sessions, referer: https://OOD_address.edu/node/comute-node01/45411/tree?
[Tue Aug 03 08:47:02.124959 2021] [proxy_http:error] [pid 16250] (70007)The timeout specified has expired: [client IP_address:51815] AH01102: error reading status line from remote server comute-node01:45411, referer: https://OOD_address.edu/node/comute-node01/45411/tree?
[Tue Aug 03 08:47:02.125006 2021] [proxy:error] [pid 16250] [client IP_address:51815] AH00898: Error reading from remote server returned by /node/comute-node01/45411/api/terminals, referer: https://OOD_address.edu/node/comute-node01/45411/tree?
[Tue Aug 03 08:48:02.077174 2021] [proxy_http:error] [pid 28777] (70007)The timeout specified has expired: [client IP_address:51826] AH01102: error reading status line from remote server comute-node01:45411, referer: https://OOD_address.edu/node/comute-node01/45411/tree?
[Tue Aug 03 08:48:02.077208 2021] [proxy_http:error] [pid 22806] (70007)The timeout specified has expired: [client IP_address:51825] AH01102: error reading status line from remote server comute-node01:45411, referer: https://OOD_address.edu/node/comute-node01/45411/tree?
[Tue Aug 03 08:48:02.077225 2021] [proxy:error] [pid 28777] [client IP_address:51826] AH00898: Error reading from remote server returned by /node/comute-node01/45411/api/terminals, referer: https://OOD_address.edu/node/comute-node01/45411/tree?
[Tue Aug 03 08:48:02.077272 2021] [proxy:error] [pid 22806] [client IP_address:51825] AH00898: Error reading from remote server returned by /node/comute-node01/45411/api/sessions, referer: https://OOD_address.edu/node/comute-node01/45411/tree?
[Tue Aug 03 08:49:02.103880 2021] [proxy_http:error] [pid 32277] (70007)The timeout specified has expired: [client IP_address:51834] AH01102: error reading status line from remote server comute-node01:45411, referer: https://OOD_address.edu/node/comute-node01/45411/tree?
[Tue Aug 03 08:49:02.103933 2021] [proxy:error] [pid 32277] [client IP_address:51834] AH00898: Error reading from remote server returned by /node/comute-node01/45411/api/sessions, referer: https://OOD_address.edu/node/comute-node01/45411/tree?
[Tue Aug 03 08:49:02.125802 2021] [proxy_http:error] [pid 15151] (70007)The timeout specified has expired: [client IP_address:51835] AH01102: error reading status line from remote server comute-node01:45411, referer: https://OOD_address.edu/node/comute-node01/45411/tree?
[Tue Aug 03 08:49:02.125860 2021] [proxy:error] [pid 15151] [client IP_address:51835] AH00898: Error reading from remote server returned by /node/comute-node01/45411/api/terminals, referer: https://OOD_address.edu/node/comute-node01/45411/tree?
[Tue Aug 03 08:50:02.069175 2021] [proxy_http:error] [pid 32701] (70007)The timeout specified has expired: [client IP_address:51844] AH01102: error reading status line from remote server comute-node01:45411, referer: https://OOD_address.edu/node/comute-node01/45411/tree?
[Tue Aug 03 08:50:02.069223 2021] [proxy:error] [pid 32701] [client IP_address:51844] AH00898: Error reading from remote server returned by /node/comute-node01/45411/api/sessions, referer: https://OOD_address.edu/node/comute-node01/45411/tree?
[Tue Aug 03 08:50:02.130312 2021] [proxy_http:error] [pid 22806] (70007)The timeout specified has expired: [client IP_address:51845] AH01102: error reading status line from remote server comute-node01:45411, referer: https://OOD_address.edu/node/comute-node01/45411/tree?
[Tue Aug 03 08:50:02.130368 2021] [proxy:error] [pid 22806] [client IP_address:51845] AH00898: Error reading from remote server returned by /node/comute-node01/45411/api/terminals, referer: https://OOD_address.edu/node/comute-node01/45411/tree?
[Tue Aug 03 08:51:28.565771 2021] [proxy_http:error] [pid 15151] (70007)The timeout specified has expired: [client IP_address:51857] AH01102: error reading status line from remote server comute-node01:45411, referer: https://OOD_address.edu/node/comute-node01/45411/tree?
[Tue Aug 03 08:51:28.565828 2021] [proxy:error] [pid 15151] [client IP_address:51857] AH00898: Error reading from remote server returned by /node/comute-node01/45411/api/sessions, referer: https://OOD_address.edu/node/comute-node01/45411/tree?
[Tue Aug 03 08:51:28.566908 2021] [proxy_http:error] [pid 28777] (70007)The timeout specified has expired: [client IP_address:51858] AH01102: error reading status line from remote server comute-node01:45411, referer: https://OOD_address.edu/node/comute-node01/45411/tree?
[Tue Aug 03 08:51:28.566955 2021] [proxy:error] [pid 28777] [client IP_address:51858] AH00898: Error reading from remote server returned by /node/comute-node01/45411/api/terminals, referer: https://OOD_address.edu/node/comute-node01/45411/tree?
[Tue Aug 03 09:38:56.669087 2021] [proxy_http:error] [pid 7053] (70007)The timeout specified has expired: [client IP_address:52205] AH01102: error reading status line from remote server comute-node01:4197, referer: https://OOD_address.edu/node/comute-node01/4197/tree?
[Tue Aug 03 09:38:56.669170 2021] [proxy:error] [pid 7053] [client IP_address:52205] AH00898: Error reading from remote server returned by /node/comute-node01/4197/api/contents, referer: https://OOD_address.edu/node/comute-node01/4197/tree?
[Tue Aug 03 09:39:04.615147 2021] [proxy_http:error] [pid 2153] (70007)The timeout specified has expired: [client IP_address:52221] AH01102: error reading status line from remote server comute-node01:4197, referer: https://OOD_address.edu/node/comute-node01/4197/tree?
[Tue Aug 03 09:39:04.615197 2021] [proxy:error] [pid 2153] [client IP_address:52221] AH00898: Error reading from remote server returned by /node/comute-node01/4197/api/terminals, referer: https://OOD_address.edu/node/comute-node01/4197/tree?
[Tue Aug 03 09:39:04.616000 2021] [proxy_http:error] [pid 28777] (70007)The timeout specified has expired: [client IP_address:52216] AH01102: error reading status line from remote server comute-node01:4197, referer: https://OOD_address.edu/node/comute-node01/4197/tree?
[Tue Aug 03 09:39:04.616047 2021] [proxy:error] [pid 28777] [client IP_address:52216] AH00898: Error reading from remote server returned by /node/comute-node01/4197/api/sessions, referer: https://OOD_address.edu/node/comute-node01/4197/tree?
[Tue Aug 03 09:39:45.523803 2021] [proxy_http:error] [pid 14223] (70007)The timeout specified has expired: [client IP_address:52260] AH01102: error reading status line from remote server comute-node01:4197, referer: https://OOD_address.edu/node/comute-node01/4197/tree?
[Tue Aug 03 09:39:45.523860 2021] [proxy:error] [pid 14223] [client IP_address:52260] AH00898: Error reading from remote server returned by /node/comute-node01/4197/api/sessions, referer: https://OOD_address.edu/node/comute-node01/4197/tree?
[Tue Aug 03 09:39:45.537565 2021] [proxy_http:error] [pid 10401] (70007)The timeout specified has expired: [client IP_address:52261] AH01102: error reading status line from remote server comute-node01:4197, referer: https://OOD_address.edu/node/comute-node01/4197/tree?
[Tue Aug 03 09:39:45.537608 2021] [proxy:error] [pid 10401] [client IP_address:52261] AH00898: Error reading from remote server returned by /node/comute-node01/4197/api/terminals, referer: https://OOD_address.edu/node/comute-node01/4197/tree?
[Tue Aug 03 10:30:38.282134 2021] [proxy_http:error] [pid 2153] (70007)The timeout specified has expired: [client IP_address:53039] AH01102: error reading status line from remote server comute-node01:27511, referer: https://OOD_address.edu/node/comute-node01/27511/tree?
[Tue Aug 03 10:30:38.282186 2021] [proxy:error] [pid 2153] [client IP_address:53039] AH00898: Error reading from remote server returned by /node/comute-node01/27511/api/contents, referer: https://OOD_address.edu/node/comute-node01/27511/tree?

15

It looks like a timeout issue is happening based on repeated output like:
[proxy_http:error] [pid 2153] (70007)The timeout specified has expired:

It might be that the notebook needs to grab some extensions (such as mathjax as you mentioned) and while this is occurring the timeout hits the wall and kills the connection. Increasing the timeout might be the simple fix here, it just seems odd that you would have this issue and not others, so I’m not sure at the moment why that would happen.

What is the current TimeOut setting for Apache? Would it be possible to increase the timeouts and test again to see if that simple fix works?

I beleive the default for Apache is something like 60 seconds which seems plenty long enough for the reqeust to be satisfied, but maybe it has been hardcoded to something much smaller?

16

We are using the default timeout limit which is 60

17

Is there any other possible reason the new notebook is unable to open?

18

Ok, then at this point I don’t think it has anything to do with Apache.

The kernel for python3 is likely the issue since you said other kernels seem to launch fine. A fresh install of that kernel might do the trick if you are fine doing that, because it seems like when it tries to load that kernel nothing ever happens and eventually you get a timeout.

When you launch the terminal from within Jupyter and enter the command:
jupyter kernelspec list
What is the output for python 3? This also gives you the locations for the kernels if a reinstall is needed.

19

/directory/names/anaconda3/2021.05/share/jupyter/kernels/python3

20

IPython is working fine.