Jupyter app ask for password first time, works second time

Hi,

I am working on adjusting the Jupyter app to our cluster, the app is currently in Sandbox.

When I click the button “Connect to Jupyter” it opens a new tab with Jupyter asking for password,
if I return and click again it opens Jupyter’s dashboard as it should.

I wonder what is the cause of not having Jupyter dashboard displayed from the first click

Thank you

Hi and welcome. The application isn’t going to be ready until the ~/ondemand/data/sys/dashboard/batch_connect/sys/bc_osc_jupyter/output/$UUID/connection.yml is created (this path may differ becuase we call our app bc_osc_juypter, you may call it something else). Before that it’s in a staring state. This file has your password in it. Or should anyhow.

I also wonder what’s going on.

Can you try this for me?

  • Create the Juypter instance, but don’t connect to it yet.
  • See what’s in the connection.yml file for that session. If there is no password, check this file again when you return and open the 2nd time.
  • Connect to juypter and record the password you sent. Was it the same as what’s in the yml? It would seem that it’s different (or non existent), and that’s why it’s prompting you for one.
    • You can do this in Chrome opening the dev tools -> settings -> checking ‘auto-open dev tools on popups’
    • It should be the only parameter you pass in the very first request to the /login api.

Hello Jeff Ohrstrom,

Thank you for answer and thank you for the wonderful tool Open OnDemand.
I followed your instructions, this is what I have:

{I have to cut a lot f output, you have restrictions to links on this forum, and It seems that is capturing a lot of my output as links}

I ask for a new instance of Jupyter Notebook and I got this job in our Torque/Moab system

Jupyter Notebook

And this Session ID from OOD

Session ID: 8ddcda60-91bf-4264-828e-3c6e3e30aff1

I did not click on “Connect to Jupyter”

Now, on the file system, I went to:

~/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/8ddcda60-91bf-4264-828e-3c6e3e30aff1

The Working Directory for the job and consistent with the address you mentioned. The file connection.yml is there and its contents are:

host: tcocs001.hpc.wvu.edu
port: 25339
password: PxMbNrGkj41wPO3h

And the config.py contains

c.NotebookApp.ip = ‘
c.NotebookApp.port = 25339
c.NotebookApp.port_retries = 0
c.NotebookApp.password = u’sha1:hNRTRQkWHbr8Cdvz:ec67bafc67b49a23472386325b537afcd2c22b7a’
c.NotebookApp.base_url = ‘/node/tcocs001.hpc.wvu.edu/25339/’
c.NotebookApp.open_browser = False
c.NotebookApp.allow_origin = '

c.NotebookApp.notebook_dir = ‘/users/gufranco’
c.NotebookApp.disable_check_xsrf = True

I have no way to check the password because I am loosing the SALT but I imagine the lines on before.sh are doing their job

Generate SHA1 encrypted password (requires OpenSSL installed)

SALT="(create_passwd 16)" password="(create_passwd 16)"
PASSWORD_SHA1="(echo -n "{password}${SALT}" | openssl dgst -sha1 | awk ‘{print $NF}’)"

Now I go back and click on “Connect to Jupyter” I got the page asking for password
I inspect the sources for that page close to Password and contains:

      <div class="center-nav">
        <form action="/node/tcocs001.hpc.wvu.edu/25339/logiXXX" method="post" class="navbar-form pull-left">
          <input type="hidden" name="_xsrf" value="2|ff030bad|024e132d8039e5ec72e01f690bc292fe|1571670441"/>
          
            <label for="password_input"><strong>Password:</strong></label>
          
          <input type="password" name="password" id="password_input" class="form-control">
          <button type="submit" class="btn btn-default" id="login_submit">Log in</button>
        </form>
      </div>

The file connection.yml has not changed after the first click

Now I click a second time and it opens the Jupyter Dashboard.

An inspection to the source code on (I have to replace tt with xx because I am not allowed to post links)

hxxps://ondemand-tf.hpc.wvu.edu/pun/sys/dashboard/batch_connect/sessions

Reveals the following code close to “Connect to Jupyter”

Connect to Jupyter

I see the password in that div section

And the value matches
connection.yml

host: tcocs001.hpc.wvu.edu
port: 25339
password: PxMbNrGkj41wPO3h

Still a mistery why I have to click twice to get the Jupyter Dashboard

Thank you very much, just to complete the context here are a few details of our system

RedHat 7.6
$ openssl version
OpenSSL 1.1.1a 20 Nov 2018

I am testing this from Firefox and Chrome, no difference from which browser you try to connect,

Thank you

On my previous message part of the HTML got caught by your forum HTML rendering, I am now correctly making it verbatim,

This is the content of the OOD sessions page:

<div id="8ddcda60-91bf-4264-828e-3c6e3e30aff1" class="panel panel-success session-panel" data hash="ce5f9645c16e9644cc9e8d2acfddc1e1"><div class="panel-heading"><h1 class="panel-title">
<a href="/pun/sys/dashboard/batch_connect/dev/jupyter/session_contexts/new"><strong>Jupyter  Notebook</strong>
</a> (120375.trcis002.hpc.wvu.edu)<div class="pull-right"><span class="badge">1 node</span> |  
<span class="badge">1 core</span> | Running</div><div class="clearfix"></div>
</h1></div><div class="panel-body">
<div><div><a class="btn btn-danger pull-right btn-delete" title="Delete Session" data-confirm="Are you sure?" data-toggle="tooltip" data-placement="bottom" rel="nofollow" data-method="delete" href="/pun/sys/dashboard/batch_connect/sessions/8ddcda60-91bf-4264-828e-3c6e3e30aff1">
<i class="fas fa-trash-alt fa-fw"></i> Delete</a></div>
<p><strong>Host:</strong> <a target="_blank" class="btn btn-primary btn-sm fas fa-terminal" href="/pun/sys/shell/ssh/tcocs001.hpc.wvu.edu">tcocs001.hpc.wvu.edu</a></p><p><strong>Created at:</strong> 2019-10-21 10:51:58 EDT</p><p>
<strong>Time Remaining:</strong> 18 minutes</p><p>
<strong>Session ID:</strong> <a target="_blank" href="/pun/sys/files/fs/users/gufranco/ondemand/data/sys/dashboard/batch_connect/dev/jupyter/output/8ddcda60-91bf-4264-828e-3c6e3e30aff1">8ddcda60-91bf-4264-828e-3c6e3e30aff1</a></p></div><div>
 <hr><div class="ood-appkit markdown"><form action="/node/tcocs001.hpc.wvu.edu/25339/login" method="post" target="_blank">
<input type="hidden" name="password" value="PxMbNrGkj41wPO3h">
<button class="btn btn-primary" type="submit">
<i class="fa fa-eye"></i> Connect to Jupyter
</button>
</form>

</div>
</div></div></div>

The second bit of html you’ve sent is good, however I’d imagine it was taken after you successfully connected.

I was able to replicate this issue by simply editing the html and sending the wrong password.

This was more what I was looking for with dev-tools. What password you’re sending that very first time. This API call I was able to trigger a 401 unauthorized. If you’re fast enough you can grab the same bit of html you’ve sent in your second post before it updates. Otherwise you’ll be able to see what you pass in the network tab as I’ve shown here.

In any case, that’s what we need to find out, what do you pass as the password the very first time (if you pass anything at all).

The UI re-reads that connection file every 30 seconds or so. I wonder if it’s just that first read that’s bad? I mean if you just waited a little bit after booting if it would work. Not necessarily the the second time you connect, but on the second read of connection.yml?

Unfortunately turning on debug logs doesn’t really provide anything. I’m able to see in the logs the 401 is returned, but it doesn’t say to whom or why.

I cannot believe it doesn’t read the connection.yml correctly the first time, but then again, that’s the only explanation I have at the moment.

I am doing this all over again, this is the HTML from the page before clicking on “Connect to Jupyter”

<hr><div class="ood-appkit markdown"><form action="/node/tcocs001.hpc.wvu.edu/58216/login" method="post" target="_blank">
<input type="hidden" name="password" value="73yAVrlG96uOIi26">
<button class="btn btn-primary" type="submit">
<i class="fa fa-eye"></i> Connect to Jupyter
</button>
</form>
</div>

I notice the line with the password -> value="73yAVrlG96uOIi26" already on that message.

I am taking time to write this, and I have not clicked yet on “Connect to Jupyter”
The file connection.yml is there::

 host: tcocs001.hpc.wvu.edu
 port: 58216
 password: 73yAVrlG96uOIi26

Now I will click on “Connect to Jupyter” (around 10 min after getting job start running)
and monitoring the network…
Sorry I am not fast enough and I cannot capture when Firefox opens the new tab but it is asking for password again!

I will ask around me here, my knowledge about web development is ground zero.
I will try to capture the message from the web server

Thank you

It’s no problem at all. You have to use Chrome because it has the option to auto load on new tab (at the very bottom of the settings), you’re not going to be able to capture it in FireFox by hand any other way.

Here’s an expansion of the image. The network tab at the top, the login api in the middle and the form data at the bottom (note the scroll bar on the right so it’s the very very bottom of that pane).

Thank you helping me with this.

I will be doing that, but just for now, let me point to an small difference in your case and my case.

In your case you have a message “Invalid Credentials” something that I dont have, I hast have a just the box asking for password.

Screenshot_20191021_151648

Another thing to explore is using a different Python and Jupyter version. I am using 3.7.4

And jupyter version:

$ jupyter --version
jupyter core : 4.5.0
jupyter-notebook : 6.0.1
qtconsole : 4.5.5
ipython : 7.8.0
ipykernel : 5.1.2
jupyter client : 5.3.2
jupyter lab : not installed
nbconvert : 5.6.0
ipywidgets : 7.5.1
nbformat : 4.4.0
traitlets : 4.3.2

We will be working from several fronts to figure out what is wrong here.

Thank you

@johrstrom I worked with Guillermo on this and was able to capture this in chrome. The password is available in the header. See below image as an example and thanks again for all the help!

Nate

OK forget the authentication bit. It looks like you’re correctly passing auth in the first request.

Let’s now assume it’s a problem with redirection and maybe setting and passing cookies. Sorry it took this long to get here.

Here is what a correct redirection flow is. Notice the status codes received from login and 55808 are 302 Found.

Seems that your 302 redirects are not working the first time?

Here’s what my very first 302 redirect response looks like. See the locaion is relative (yours has your SSO host in it) and it’s from a different Server even. You’re being redirected by your SSO or OOD (Apache 2.4.34 seems like it’s ours) whereas I’m being redirected by Juypter itself (TornadoServer is what Jupyter uses)
image

I don’t know much about CAS, but maybe you need to tweak your CASScope? I’ve seen in other threads folks need to set that to /.

Thanks @johrstrom for the help! The CASScope indeed fixed the issue. Sorry for all the questions lately but we are making good progress thanks to you and your team! I never expected the proxy to be the issue since it worked on the second attempt but I was definitely wrong on that one.

Awesome! I’m glad to get it fixed.

And feel free to ask any questions you like. You just helped the next person that’s may have this same issue.

1 Like