I am trying to set up Jupyter notebook with ondemand and I have some issues with URL:
ondemand routes to
http:///node/compute-x-x/27006/login
which is not found
when checking the logs from the running job I see that the correct path should be:
http://compute-x-x:27006/node/compute-x-x/27006/login
which routes to the jupyter notebook but asks for a password?
How should I go about fixing the URL routing
and the password request?
You probably need to configure you’re reverse proxy. The password business will be taken care of by OOD.
In your /etc/ood/config/ood_portal.yml you’ll need these configs.
# this regex you should probably change to be more specific
host_regex: "[^/]+"
# these should be fine for you
node_uri: "/node"
rnode_uri: "/rnode"
Thanks for the quick response/
Any suggestions on how the host_regex should be?
our hosts are set with compute-x-x and gpu-x-x convention
we can add the <my_center.edu> part to the hostname if needed.
Right now it looks like this:
host_regex: ‘[\w.-]+.mycenter.edu’
node_uri: ‘/node’
rnode_uri: ‘/rnode’
Ok I was able to get that part fixed with adding the mycenter extension to the hostnames.
What’s remaining is the password prompt which says “invalid”
First it goes to login authentication page again and then “invalid” password for jupyter.
Any thoughts?
First are you running CAS authentication? If so it’s an issue of your CASScope. Even if you aren’t Here’s a very lengthy thread on how to debug this issue.
We don’t use CAS. authentication.
Will do, thanks…
It all seems like that the password from jupyter is not correctly passed into the instance through ondemand
Bummer, that would have been a quick fix. In any case, that topic has a lot of debugging steps that I would just replicate here. If you do look through it and still can’t find what you’re looking for update this topic with information and we’ll look into it more.
Things we’d want to know are
view.html.erb looks like (if it’s different from ours).connection.yml file (the directory for a given session is described in the other topic).Looking at that topic, one obvious difference in our case is that,
when we click the “connect to Jupyter” in the Sandbox environment, it prompts to the ondemand web authentications which users have already done when they logged into the website.
Then we pass that authentication, and then “jupyter password” webpage appears with a message “invalid authentication”
For the sake of testing, I tried to copy the password that is generated in
“connection.yml” file under <user_path>/jupyter-notebook/output/<jupyter_instance>
and when enter that, a blank page appears. It seems the same with entering any other password.
OK yea something’s very wrong with that. You should not have to authenticate twice. You’re going to need to use Chrome’s web dev tools to ‘open on new tab’ and see how you’re being redirected once you click ‘connect to jupyter’.
I’m not sure why/how you’re being redirected like that. What authentication method do you use. And are you being redirected to a different host? like sandbox.myorg -> production.myorg?
Here is what I see when clicking the “connect to Jupyter”:
This is the authentication pop-up that appears and I try to cancel –
The override.css looks like
<path_to_jupyter>/static/auth/css/override.css…
Is that the CAS that you were referring to??
CAS is Central Authentication Service. I don’t know much about it, but it’s a single sign on that some sites configure their apache for user login.
Can you take a second look at that thread I’ve linked. Specifically the 8th and 9th comments.
What does this screen shot look like for you? Specifically, the response headers (they’re a little cut off in this image) of the very first login request.
My response headers look like this. The big difference here is the ‘Location’. It should be relative (there’s no hostname in the location, it’s only the path) and from Jupyter itself (the ‘server’ is TornadoServer, that’s Jupyter).
How you’re being redirected seems very off to me, especially if you need to re-login.
Hi Jeff,
Thanks for bearing with us as we investigate the issue. We use LDAP for authentication not CAS.
I will revisit those steps and provide the web screenshot you requested. In the meantime, can you remind me what path should we expect as the output of each jupyter launch?
For some partitions, I get
"http://compute-x-x.<mycenter.edu>:16863/node/compute-x-x.<mycenter.edu>/16863/
which seems to be wrong?
Any hints on what to check to get that part fixed are appreciated
That’s the right path, but the wrong host. https://<OOD HOST>/node/compute-x-x.<mycenter.edu>/16863/ should be the host/path.
If you look at our view.html.erb (the box that has the ‘connect to jupyter’ button) the action is relative meaning you should still land on the open ondemand host and not be redirected directly to the compute node, we (OOD) should be proxying to the compute node for you.
Thanks for the information. Strangely, our nodes still directly use the compute node rather than the OOD HOST.
. Is there another config file that we should look for to route this correctly?
"
user@host$ cat view.html.erb <form action="/node/<%= host %>/<%= port %>/login" method="post" target="_blank"> <input type="hidden" name="password" value="<%= password %>"> <button class="btn btn-primary" type="submit"> <i class="fa fa-eye"></i> Connect to Jupyter </button> </form>
"
In your /opt/rh/httpd24/root/etc/httpd/conf.d/ood-portal.conf what does this LocationMatch look like?
<LocationMatch "^/node/(?<host>[\w.-]+\.osc\.edu)/(?<port>\d+)">
AuthType openid-connect
Require valid-user
# ProxyPassReverse implementation
Header edit Location "^[^/]+//[^/]+" ""
# ProxyPassReverseCookieDomain implemenation
Header edit* Set-Cookie ";\s*(?i)Domain[^;]*" ""
# ProxyPassReverseCookiePath implementation
Header edit* Set-Cookie ";\s*(?i)Path[^;]*" ""
Header edit Set-Cookie "^([^;]+)" "$1; Path=/node/%{MATCH_HOST}e/%{MATCH_PORT}e"
LuaHookFixups node_proxy.lua node_proxy_handler
</LocationMatch>
user@host$ cat /opt/rh/httpd24/root/etc/httpd/conf.d/ood-portal.conf
<LocationMatch "^/node/(?<host>[\w.-]+\.my.center\.edu)/(?<port>\d+)">
AuthType Basic
AuthName "private"
AuthBasicProvider ldap
AuthLDAPURL "ldaps://ldaps.my.center.edu:portnumber/OU=users,DC=dcname,DC=orgname?name"
AuthLDAPGroupAttribute cn
AuthLDAPGroupAttributeIsDN on
Require valid-user
AuthLDAPBindDN "cn=cnname,ou=users,dc=dcname,dc=orgname"
AuthLDAPBindPassword ***
# ProxyPassReverse implementation
Header edit Location "^[^/]+//[^/]+" ""
# ProxyPassReverseCookieDomain implemenation
Header edit* Set-Cookie ";\s*(?i)Domain[^;]*" ""
# ProxyPassReverseCookiePath implementation
Header edit* Set-Cookie ";\s*(?i)Path[^;]*" ""
Header edit Set-Cookie "^([^;]+)" "$1; Path=/node/%{MATCH_HOST}e/%{MATCH_PORT}e"
LuaHookFixups node_proxy.lua node_proxy_handler
</LocationMatch>Maybe it could be an issue of rewrites?
In your /etc/ood/config/ood_portal.yml do you have this set (or left it to the default)?
use_rewrites: true
Honestly I have no idea why you’re being redirected like that. You’re location - this bit here Header edit Location "^[^/]+//[^/]+" "" - doesn’t seem to be working.
That very first request you make to Jupyter should have a Location header, can you let me know what it is?
the use_rewrites is set to default.
See attached.
As mentioned, one difference with the other thread is that it routes to authentication again before getting to jupyter, i.e., after “connect to Jupyter” the authentication prompt window appears again. If I authenticate that, then I am prompted to the jupyter notebook with an “invalid credentials” message. Then, attached is the screenshot you asked for.
As a side note,
I just see something in the jupyter output file from one of the launches:
/home/user/ondemand/data/sys/dashboard/batch_connect/dev/jupyter_notebook/output/jupyter-id/before.sh: line 38: openssl: command not found which means traffic won’t be encrypted. I don’t think that’s related but just wanted to bring that.
Just for sanity checks: here is the websockify installation
I have verified that websockify is installed and working on the COMPUTE node:
[host]# websockify --help
Usage:
websockify [options] [source_addr:]source_port [target_addr:target_port]
websockify [options] [source_addr:]source_port – WRAP_COMMAND_LINE
cat /etc/ood/config/clusters.d/mycluster.yml | grep websockify
export WEBSOCKIFY_CMD="//usr/bin/websockify"
As a sanity check, be sure you’re passing the password in the form data. You’ll see it in under the ‘request headers’ portion of your chrome dev tools. I don’t think it’s a problem, but just to be sure.
Do you have a similar configuration as us?
I think specifically these items are important in this case.
c.NotebookApp.base_url = '/node/${host}/${port}/'
c.NotebookApp.allow_origin = '*'
c.NotebookApp.disable_check_xsrf = True
The prompt for re-authentication is strange and should not be happening. Is there anything else you’ve added to your apache ood-portal.conf?
This problem is so out of left field that at this point I’m willing to have a webex (or similar meeting technology) with you to help debug.
I have webex, so we could use that. Let me know if have time tomorrow or Friday.