Websocket Connections Error

I have a new build of Open OnDemand and everything seems to be working as it should such as Files, Jobs, Job Monitor. The one thing that is not working is the ‘Clusters’ terminal access. We built this up in a lab environment and all was working fine, but after installing on the corporate network, the terminal access gives an error of “Failed to establish a websocket connection. Be sure you are using a browser that supports websocket connections.” I’m suspecting this might be a setting on the web browser, but have been unable to pin point. This happens with Chrome and Microsoft Edge. Any thoughts as what to look at? I did look at the /var/log/ondemand-nginx/$USER/error.log. nothing stuck out to me as being an error.

Thanks in advance, Kyle & Dom

There is a similar question here in which I identify troubleshooting steps. In fact, Firefox now ships within it’s dev tools the ability to see websocket messages (if there are any). Your suspicion about where the error is correct. Or at least it’s between you (the client browser) and the OOD server (apache).

You mention /var/log/ondemand-nginx, did you check the httpd24 logs as well? (the other user said there was nothing there, but maybe there is for you?).

Then just to be clear, and for documentation purposes, here’s what a successful connection looks like. Note the response code is 101 switching protocols.

Thanks for the reply. Kyle who is cc’d here has access to the server so he will check this and let us both know.

Thanks

Dom

Hi Jeff, I’ll attempt to get access to the server tomorrow and let you know what we see. Thanks, Kyle

Hi Jeff, I did get access today. In the httpd24 error logs I just see a mention of “req_is_websocket="false ”
not sure how to read this.

In the access log I didn’t see any mention of websocket.

I’ve attached a snippet of these two log files if you’d like to look. Thanks, Kyle

(Attachment error_log.txt is missing)

(Attachment access_log.txt is missing)

It says those txt files are missing. Also can you provide information on what your client browser sees (like I have in the devtools pane.)? I get a 101 status upgrade. What happens in your client? You may have to look into the console as well as the network tab.

I see the txt file was blocked. Here is a bit of the error log just so you can see. Kyle

[Tue Feb 04 07:47:20.659013 2020] [lua:info] [pid 4003] [client 10.48.17.42:55275] req_protocol=“HTTP/1.1” req_handler=“proxy-server” req_method=“GET” req_accept="/"
req_user_agent=“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36” res_content_length=“354781” req_content_type="" res_content_encoding="" req_status=“200” req_origin="" time_user_map=“366.972”
local_user=“nortech” req_referer=“http://10.48.195.9/pun/sys/file-editor/edit/home/nortech/fluent-13-error.log” res_content_language="" req_port=“80” log_time=“2020-02-04T13:47:20.658887Z” req_server_name=“10.48.195.9” log_hook=“ood” req_accept_charset=""
req_hostname=“10.48.195.9” res_content_location="" res_content_disp="" req_is_websocket=“false” remote_user=“nortech” res_location="" req_user_ip=“10.48.17.42” req_is_https=“false” req_filename=“proxy:http://localhost/pun/sys/file-editor/ace/1.2.6/ace.js
req_uri="/pun/sys/file-editor/ace/1.2.6/ace.js" time_proxy=“117.446” res_content_type=“application/javascript” req_accept_language=“en-us,en;q=0.9” req_cache_control="" req_accept_encoding=“gzip, deflate”, referer: http://10.48.195.9/pun/sys/file-editor/edit/home/nortech/fluent-13-error.log

[Tue Feb 04 07:47:20.849867 2020] [lua:info] [pid 4003] [client 10.48.17.42:55275] req_protocol=“HTTP/1.1” req_handler=“proxy-server” req_method=“GET” req_accept="/"
req_user_agent=“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36” res_content_length=“18028” req_content_type="" res_content_encoding="" req_status=“200” req_origin=“http://10.48.195.9” time_user_map=“95.167”
local_user=“nortech” req_referer=“http://10.48.195.9/pun/sys/file-editor/assets/application-ce8832b5e187a2e1c108b3ce5e808a12ee8c9a806c42eb27a160ca9724e7d974.css” res_content_language="" req_port=“80” log_time=“2020-02-04T13:47:20.849730Z” req_server_name=“10.48.195.9”
log_hook=“ood” req_accept_charset="" req_hostname=“10.48.195.9” res_content_location="" res_content_disp="" req_is_websocket=“false” remote_user=“nortech” res_location="" req_user_ip=“10.48.17.42” req_is_https=“false” req_filename=“proxy:http://localhost/pun/sys/file-editor/assets/bootstrap/glyphicons-halflings-regular-fe185d11a49676890d47bb783312a0cda5a44c4039214094e7957b4c040ef11c.woff2
req_uri="/pun/sys/file-editor/assets/bootstrap/glyphicons-halflings-regular-fe185d11a49676890d47bb783312a0cda5a44c4039214094e7957b4c040ef11c.woff2" time_proxy=“1.603” res_content_type=“application/octet-stream” req_accept_language=“en-us,en;q=0.9” req_cache_control=""
req_accept_encoding=“gzip, deflate”, referer: http://10.48.195.9/pun/sys/file-editor/assets/application-ce8832b5e187a2e1c108b3ce5e808a12ee8c9a806c42eb27a160ca9724e7d974.css

[Tue Feb 04 07:47:21.039163 2020] [lua:info] [pid 4003] [client 10.48.17.42:55275] req_protocol=“HTTP/1.1” req_handler=“proxy-server” req_method=“GET” req_accept="/"
req_user_agent=“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36” res_content_length=“2360” req_content_type="" res_content_encoding="" req_status=“200” req_origin="" time_user_map=“174.478”
local_user=“nortech” req_referer=“http://10.48.195.9/pun/sys/file-editor/edit/home/nortech/fluent-13-error.log” res_content_language="" req_port=“80” log_time=“2020-02-04T13:47:21.39045Z” req_server_name=“10.48.195.9” log_hook=“ood” req_accept_charset="" req_hostname=“10.48.195.9”
res_content_location="" res_content_disp="" req_is_websocket=“false” remote_user=“nortech” res_location="" req_user_ip=“10.48.17.42” req_is_https=“false” req_filename=“proxy:http://localhost/pun/sys/file-editor/ace/1.2.6/theme-solarized_light.js” req_uri="/pun/sys/file-editor/ace/1.2.6/theme-solarized_light.js"
time_proxy=“1.201” res_content_type=“application/javascript” req_accept_language=“en-us,en;q=0.9” req_cache_control="" req_accept_encoding=“gzip, deflate”, referer: http://10.48.195.9/pun/sys/file-editorit/home/nortech/fluent-13-error.log

[Tue Feb 04 07:47:21.050863 2020] [lua:info] [pid 1561] [client 10.48.17.42:55274] req_protocol=“HTTP/1.1” req_handler=“proxy-server” req_method=“GET” req_accept=“text/plain,
/; q=0.01” req_user_agent=“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36” res_content_length=“3419” req_content_type="" res_content_encoding="" req_status=“200” req_origin="" time_user_map=“175.297”
local_user=“nortech” req_referer=“http://10.48.195.9/pun/sys/file-editor/edit/home/nortech/fluent-13-error.log” res_content_language="" req_port=“80” log_time=“2020-02-04T13:47:21.50728Z” req_server_name=“10.48.195.9” log_hook=“ood” req_accept_charset="" req_hostname=“10.48.195.9”
res_content_location="" res_content_disp="" req_is_websocket=“false” remote_user=“nortech” res_location="" req_user_ip=“10.48.17.42” req_is_https=“false” req_filename=“proxy:http://localhost/pun/sys/files/api/v1/fs/home/nortech/fluent-13-error.log” req_uri="/pun/sys/files/api/v1/fs/home/nortech/fluent-13-error.log"
time_proxy=“7.403” res_content_type=“text/plain; charset=utf-8” req_accept_language=“en-us,en;q=0.9” req_cache_control="" req_accept_encoding=“gzip, deflate”, referer: http://10.48.195.9/pun/sys/file-editor/edit/home/nortech/fluent-13-error.log

[Tue Feb 04 07:47:40.738097 2020] [lua:info] [pid 4266] [client 10.48.17.42:55278] req_protocol=“HTTP/1.1” req_handler=“proxy-server” req_method=“GET” req_accept=“text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3”
req_user_agent=“Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36” res_content_length=“0” req_content_type="" res_content_encoding="" req_status=“304” req_origin="" time_user_map=“139.499” local_user=“nortech”
req_referer=“http://10.48.195.9/pun/sys/dashboard/” res_content_language="" req_port=“80” log_time=“2020-02-04T13:47:40.737941Z” req_server_name=“10.48.195.9” log_hook=“ood” req_accept_charset="" req_hostname=“10.48.195.9” res_content_location="" res_content_disp=""
req_is_websocket=“false” remote_user=“nortech” res_location="" req_user_ip=“10.48.17.42” req_is_https=“false” req_filename=“proxy:http://localhost/pun/sys/shell/ssh/grvhpc01.na.corp.owenscorning.com” req_uri="/pun/sys/shell/ssh/grvhpc01.na.corp.owenscorning.com"
time_proxy=“12.313” res_content_type="" req_accept_language=“en-us,en;q=0.9” req_cache_control="" req_accept_encoding=“gzip, deflate”, referer: http://10.48.195.9/pun/sys/dashboard/

OK… 304 Not Modified huh? Are you being cached somewhere in between yourself and the OOD server?

Looks like you’re hitting this function. The response is not cacheable, so the only other bit is the request is not fresh. For ‘freshness’ we look at etag and last-modified headers with this package. If you look at my request in the image above I use Cache-Control: no-cache which subverts this whole thing. I did not set that myself, it must be set somewhere/somehow.

    if (this.isCachable() && this.isFresh()) {
      this.notModified()
      return
    }

Do you have any updates? Are you being cached somewhere?

Hi Jeff, unfortunately we are finding that the client and the OOD server are on separate VLAN’s which are very tightly controlled. We are working on a
separate issue to open the firewall from client to where the OOD server resides. Once this is complete we will retest and see if we still see the issues with the terminal. Kyle

1 Like

@kgross were you able to resolve the issue?