Dex LDAP not working

I used NVidia’s deepops Ansible playbooks to install Slurm / OpenOnDemand.

I also followed the instructions from https://osc.github.io/ood-documentation/latest/authentication/dex.html#configuring-ondemand-dex-for-ldap to install / configure Dex for LDAP authentication.

The openldap-dex service is enabled/running.

When trying to authenticate, I see the following error:
/var/log/httpd24/ondemand.jhuapl.edu_error_ssl.log:[Mon Nov 02 18:56:34.221146 2020] [auth_basic:error] [pid 37541] [client 10.100.10.124:46994] AH01618: user ticemp1 not found: /pun/sys/dashboard

The user exists on the server & in LDAP. Are there any other logs to check?

Hi and welcome!

The logs say auth_basic:error and it should be mod_auth_openidc. It seems you don’t have OIDC auth configured in your ood-portal.conf. What’s the auth section of /etc/ood/config/ood_portal.yml look like?

auth:

  • ‘AuthType Basic’
  • ‘AuthName “private”’
  • ‘AuthUserFile “/opt/rh/httpd24/root/etc/httpd/.htpasswd”’
  • ‘RequestHeader unset Authorization’
  • ‘Require valid-user’

Maybe I haven’t looked enough, but I didn’t see what to change this to to use Dex…

Yea so you need this, to configure apache to use OpenIDC.

# httpd_auth in your ansible configurations is the resulting auth in ood_portal.yml
httpd_auth:
  - 'AuthType openid-connect'
  - 'Require valid-user'

Then you’ll need all sorts of openidc related configurations to get apache to recognize your dex instance. Here’s the README for the ood-ansible role on how to configure open idc related items.

thanks, this is taking me down an unexpected rabbit hole, do I need to install /configure keycloak as well?

No, Keycloak is different identity provider, different from Dex that is. Dex is the one you’ve got installed and is easier to configure, so that’s it. But from the Apache side they’re essentially the same whether you use Dex, or Keycloak or even Google.

Also here’s the apache module’s documentation. I’d say focus on that and what those options are that you need to configure.

Thanks, so it turns out ondemand-dex wasn’t running because of the following:

Nov 03 13:10:43 aplcdhen01 ondemand-dex[23591]: failed to initialize server: server: failed to load web static: load dir: directory “web/themes/coreos” does not exist
Nov 03 13:10:43 aplcdhen01 systemd[1]: ondemand-dex.service: main process exited, code=exited, status=2/INVALIDARGUMENT

I’m not seeing a coreos directory in /usr/share/ondemand-dex/web/themes/, however there is a ondemand directory.

# ls -l /usr/share/ondemand-dex/web/themes/
total 0
drwxr-xr-x 2 root root 117 Nov 2 16:20 ondemand

I even went to clone the repo from https://github.com/OSC/ondemand-dex but it wasn’t there either. Is there a config somewhere that’s looking for that directory?

@miketice22 The OnDemand Dex theme can be found here: https://github.com/OSC/ondemand-dex/tree/master/web

Copy the entire contents of /web in the ondemand-dex repo to /usr/share/ondemand-dex/web. This is important because /web contains Bootstrap and other helper CSS styles that the ondemand theme uses.

Thanks, now different error:

[root@aplcdhen01 tmp]# git clone https://github.com/OSC/ondemand-dex
Cloning into ‘ondemand-dex’…
remote: Enumerating objects: 161, done.
remote: Counting objects: 100% (161/161), done.
remote: Compressing objects: 100% (107/107), done.
remote: Total 161 (delta 51), reused 133 (delta 33), pack-reused 0
Receiving objects: 100% (161/161), 278.29 KiB | 0 bytes/s, done.
Resolving deltas: 100% (51/51), done.
[root@aplcdhen01 tmp]# cd ondemand-dex/
[root@aplcdhen01 ondemand-dex]# cp -R web /usr/share/ondemand-dex/
[root@aplcdhen01 ondemand-dex]# systemctl restart ondemand-dex
[root@aplcdhen01 ondemand-dex]# systemctl status -l ondemand-dex
● ondemand-dex.service - OnDemand Dex - A federated OpenID Connect provider packaged for OnDemand
Loaded: loaded (/usr/lib/systemd/system/ondemand-dex.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2020-11-03 15:54:30 UTC; 8s ago
Process: 33230 ExecStart=/usr/sbin/ondemand-dex serve /etc/ood/dex/config.yaml (code=exited, status=2)
Main PID: 33230 (code=exited, status=2)
Nov 03 15:54:30 aplcdhen01 systemd[1]: Started OnDemand Dex - A federated OpenID Connect provider packaged for OnDemand.
Nov 03 15:54:30 aplcdhen01 ondemand-dex[33230]: time=“2020-11-03T15:54:30Z” level=info msg=“config issuer: http://127.0.0.1:5556/dex
Nov 03 15:54:30 aplcdhen01 ondemand-dex[33230]: time=“2020-11-03T15:54:30Z” level=info msg=“config storage: sqlite3”
Nov 03 15:54:30 aplcdhen01 ondemand-dex[33230]: time=“2020-11-03T15:54:30Z” level=info msg=“config static client: Example App”
Nov 03 15:54:30 aplcdhen01 ondemand-dex[33230]: time=“2020-11-03T15:54:30Z” level=info msg=“config connector: mock”
Nov 03 15:54:30 aplcdhen01 ondemand-dex[33230]: time=“2020-11-03T15:54:30Z” level=info msg=“config connector: local passwords enabled”
Nov 03 15:54:30 aplcdhen01 systemd[1]: ondemand-dex.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Nov 03 15:54:30 aplcdhen01 systemd[1]: Unit ondemand-dex.service entered failed state.
Nov 03 15:54:30 aplcdhen01 systemd[1]: ondemand-dex.service failed.

/etc/ood/dex/config.yaml is identical to https://github.com/dexidp/dex/blob/master/examples/config-dev.yaml

@miketice22 What does your /etc/ood/config/ood_portal.yml look like? Also make sure you run /opt/ood/ood-portal-generator/sbin/update_ood_portal after you update Dex configs or modify ood_portal.yml.

Your ood_portal.yml might be wrong, from your logs:

Nov 03 15:54:30 aplcdhen01 ondemand-dex[33230]: time=“2020-11-03T15:54:30Z” level=info msg=“config connector: mock”

You should have your LDAP connector configured in ood_portal.yml instead of a mock connector.

grep -v ‘^#’ /etc/ood/config/ood_portal.yml

servername: ondemand.jhuapl.edu

ssl:
- 'SSLCertificateFile /etc/pki/tls/certs/aplcdhen01.jhuapl.edu.pem'
- 'SSLCertificateKeyFile /etc/pki/tls/private/aplcdhen01.jhuapl.edu.key'
- 'SSLCertificateChainFile /etc/pki/tls/certs/aplcdhen01.jhuapl.edu.pem'

logroot: "/var/log/httpd24"

use_rewrites: False

lua_root: "/opt/ood/mod_ood_proxy/lib"

lua_log_level: "info"

user_map_cmd: "/opt/ood/ood_auth_map/bin/ood_auth_map.regex"

pun_stage_cmd: "sudo /opt/ood/nginx_stage/sbin/nginx_stage"

auth:
- 'AuthType openid-connect'
- 'Require valid-user'

root_uri: /pun/sys/dashboard

public_uri: "/public"

public_root: "/var/www/ood/public"

logout_uri: "/logout"

logout_redirect: '/oidc?logout=https%3A%2F%2Fopenondemand.jhuapl.edu'

host_regex: "[^/]+"

node_uri: /node

rnode_uri: /rnode

nginx_uri: /nginx

pun_uri: "/pun"

pun_socket_root: "/var/run/ondemand-nginx"

pun_max_retries: 5

oidc_uri: '/oidc'

dex:
  connectors:
    - type: ldap
      id: ldap
      name: LDAP
      config:
        host: aplid.jhuapl.edu:636
        insecureSkipVerify: false
        userSearch:
          baseDN: cn=users,dc=jhuapl,dc=edu
          filter: "(objectClass=person)"
          username: uid
          idAttr: uid
          emailAttr: mail
          nameAttr: displayName
          preferredUsernameAttr: uid
        groupSearch:
          baseDN: ou=Groups,dc=dom1-proxy,dc=apl-staging
          filter: "(objectClass=group)"
          userMatchers:
            - userAttr: DN
              groupAttr: member
          nameAttr: cn

@mario

I commented out those example/mock connectors in /etc/ood/dex/config.yaml and restarted, now returned the error I had before:

[root@aplcdhen01 ondemand-dex]# systemctl status -l ondemand-dex
● ondemand-dex.service - OnDemand Dex - A federated OpenID Connect provider packaged for OnDemand
Loaded: loaded (/usr/lib/systemd/system/ondemand-dex.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2020-11-03 18:29:58 UTC; 9s ago
Process: 1521 ExecStart=/usr/sbin/ondemand-dex serve /etc/ood/dex/config.yaml (code=exited, status=2)
Main PID: 1521 (code=exited, status=2)
Nov 03 18:29:58 aplcdhen01 systemd[1]: Started OnDemand Dex - A federated OpenID Connect provider packaged for OnDemand.
Nov 03 18:29:58 aplcdhen01 ondemand-dex[1521]: time=“2020-11-03T18:29:58Z” level=info msg=“config issuer: http://127.0.0.1:5556/dex
Nov 03 18:29:58 aplcdhen01 ondemand-dex[1521]: time=“2020-11-03T18:29:58Z” level=info msg=“config storage: sqlite3”
Nov 03 18:29:58 aplcdhen01 ondemand-dex[1521]: time=“2020-11-03T18:29:58Z” level=info msg=“config connector: local passwords enabled”
Nov 03 18:29:58 aplcdhen01 ondemand-dex[1521]: failed to initialize server: server: failed to load web static: load dir: directory “web/themes/coreos” does not exist
Nov 03 18:29:58 aplcdhen01 systemd[1]: ondemand-dex.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Nov 03 18:29:58 aplcdhen01 systemd[1]: Unit ondemand-dex.service entered failed state.
Nov 03 18:29:58 aplcdhen01 systemd[1]: ondemand-dex.service failed.

@miketice22 Ahh I see the error now, it’s here:

Nov 03 18:29:58 aplcdhen01 ondemand-dex[1521]: failed to initialize server: server: failed to load web static: load dir: directory “web/themes/coreos” does not exist

You need to override the theme.

Add this under the dex key in /etc/ood/config/ood_portal.yml:

  # This is the default, but illustrating how to change
  frontend:
    theme: ondemand

Once you do that make sure to run the command:

/opt/ood/ood-portal-generator/sbin/update_ood_portal

Your final portal config should look similar to:

dex:
  client_redirect_uris:
    - "https://localhost:4443/simplesaml/module.php/authoidcoauth2/linkback.php"
    - "https://localhost:2443/oidc/callback/"
  client_secret: 334389048b872a533002b34d73f8c29fd09efc50
  client_id: localhost
  connectors:
    - type: ldap
      id: ldap
      name: LDAP
      config:
        host: ldap:636
        insecureSkipVerify: true
        bindDN: cn=admin,dc=example,dc=org
        bindPW: admin
        userSearch:
          baseDN: ou=People,dc=example,dc=org
          filter: "(objectClass=posixAccount)"
          username: uid
          idAttr: uid
          emailAttr: mail
          nameAttr: gecos
          preferredUsernameAttr: uid
        groupSearch:
          baseDN: ou=Groups,dc=example,dc=org
          filter: "(objectClass=posixGroup)"
          userMatchers:
            - userAttr: DN
              groupAttr: member
          nameAttr: cn
  # This is the default, but illustrating how to change
  frontend:
    theme: ondemand

I made the change, however the generator said there was no change to the apache config, and it still shows that error even after restarting ondemand-dex.

i added

frontend:
theme: ondemand

to /etc/ood/dex/config.yaml and restarted ondemand-dex and that got rid of the error, but now I get a 500 error:

[Tue Nov 03 19:01:05.377797 2020] [auth_openidc:error] [pid 4207] [client 10.100.10.124:54234] oidc_check_userid_openidc: configuration error: the authentication type is set to “openid-connect” but OIDCRedirectURI has not been set.

[root@aplcdhen01 ondemand-dex]# grep -v '^#' /etc/ood/config/ood_portal.yml
---


servername: ondemand.jhuapl.edu


ssl:
- 'SSLCertificateFile /etc/pki/tls/certs/aplcdhen01.jhuapl.edu.pem'
- 'SSLCertificateKeyFile /etc/pki/tls/private/aplcdhen01.jhuapl.edu.key'
- 'SSLCertificateChainFile /etc/pki/tls/certs/aplcdhen01.jhuapl.edu.pem'

logroot: "/var/log/httpd24"

use_rewrites: False

lua_root: "/opt/ood/mod_ood_proxy/lib"

lua_log_level: "info"

user_map_cmd: "/opt/ood/ood_auth_map/bin/ood_auth_map.regex"



pun_stage_cmd: "sudo /opt/ood/nginx_stage/sbin/nginx_stage"

auth:
- 'AuthType openid-connect'
- 'Require valid-user'

default_auth_openidc:
  OIDCRedirectURI: 'https://localhost:4443/simplesaml/module.php/authoidcoauth2/linkback.php'
  OIDCCryptoPassphrase: ondemanddex

root_uri: /pun/sys/dashboard



public_uri: "/public"

public_root: "/var/www/ood/public"


logout_uri: "/logout"

logout_redirect: '/oidc?logout=https%3A%2F%2Fopenondemand.jhuapl.edu'


host_regex: "[^/]+"

node_uri: /node

rnode_uri: /rnode


nginx_uri: /nginx

pun_uri: "/pun"

pun_socket_root: "/var/run/ondemand-nginx"

pun_max_retries: 5


oidc_uri: '/oidc'






dex:
  client_redirect_uris:
    - "https://localhost:4433/simplesaml/module.php/authoidcoauth2/linkback.php"
    - "https://localhost:2443/oidc/callback/"
  client_secret: 334389048b872a533002b34d73f8c29fd09efc50
  client_id: localhost
  connectors:
    - type: ldap
      id: ldap
      name: LDAP
      config:
        host: aplid.jhuapl.edu:636
        insecureSkipVerify: false
        userSearch:
          baseDN: cn=users,dc=jhuapl,dc=edu
          filter: "(objectClass=person)"
          username: uid
          idAttr: uid
          emailAttr: mail
          nameAttr: displayName
          preferredUsernameAttr: uid
        groupSearch:
          baseDN: ou=Groups,dc=dom1-proxy,dc=apl-staging
          filter: "(objectClass=group)"
          userMatchers:
            - userAttr: DN
              groupAttr: member
          nameAttr: cn
  frontend:
    theme: ondemand

Very little configuration is needed to work with Dex. You do not need to set logout_redirect or logout_uri and default_auth_openidc isn’t something ood-portal-generator supports so just delete that, it’s not used by OnDemand.

Also if you omit auth and enable Dex the value you provided is the default set by OnDemand. Same goes for oidc_uri.

Also copying themes is not needed if you are using the RPM to install ondemand-dex. If you are not using the RPM then whatever method you used to install is flawed and not doing a proper install. It’s not clear to me from this thread exactly where the install method came from but the recommended install method is RPM and that will not require any modifications or copies of themes.

This is a minimal example of a working OnDemand instance I deploy at OSC to test Dex:

[root@webdev07 ~]# cat /etc/ood/config/ood_portal.yml
# File managed by Puppet - do not edit!
---
servername: webdev07.hpc.osc.edu
port: '443'
ssl:
- SSLCertificateFile /etc/pki/tls/certs/webdev07.hpc.osc.edu.crt
- SSLCertificateKeyFile /etc/pki/tls/private/webdev07.hpc.osc.edu.key
- SSLCertificateChainFile /etc/pki/tls/certs/webdev07.hpc.osc.edu-interm.crt
dex:
  connectors:
  - type: ldap
    id: ldap
    name: LDAP
    config:
      host: ldap1.infra.osc.edu:636
      insecureSkipVerify: false
      bindDN: cn=read,ou=Admin,dc=osc,dc=edu
      bindPW: <OMIT>
      userSearch:
        baseDN: ou=People,dc=osc,dc=edu
        filter: "(objectClass=posixAccount)"
        username: uid
        idAttr: uid
        emailAttr: mail
        nameAttr: gecos
        preferredUsernameAttr: uid
      groupSearch:
        baseDN: ou=Groups,dc=osc,dc=edu
        filter: "(objectClass=posixGroup)"
        userMatchers:
        - userAttr: DN
          groupAttr: member
        nameAttr: cn

I would try and start as simple as possible to isolate where things are failing. This is the produced contents for OIDC in my ood-portal.conf:

[root@webdev07 ~]# grep OIDC /opt/rh/httpd24/root/etc/httpd/conf.d/ood-portal.conf
  # OIDC configuration
  OIDCProviderMetadataURL https://webdev07.hpc.osc.edu:5554/.well-known/openid-configuration
  OIDCClientID webdev07.hpc.osc.edu
  OIDCClientSecret 8a3bac2f-bf9e-4585-9a2f-db97714af28c
  OIDCRedirectURI https://webdev07.hpc.osc.edu/oidc
  OIDCRemoteUserClaim preferred_username
  OIDCScope "openid profile email"
  OIDCCryptoPassphrase 339e00b1e2c9fca781f40f95e442b67e7b042917
  OIDCSessionInactivityTimeout 15
  OIDCSessionMaxDuration 60
  OIDCStateMaxNumberOfCookies 10 true
  OIDCCookieSameSite Off

After you’ve made changes to ood_portal.yml and run the update_ood_portal command please run the grep command from above and share the output, feel free to omit the secret.

[root@aplcdhen01 config]# cat /etc/ood/config/ood_portal.yml
# Ansible managed
---
servername: ondemand.jhuapl.edu
port: '443'

ssl:
- 'SSLCertificateFile /etc/pki/tls/certs/aplcdhen01.jhuapl.edu.pem'
- 'SSLCertificateKeyFile /etc/pki/tls/private/aplcdhen01.jhuapl.edu.key'
- 'SSLCertificateChainFile /etc/pki/tls/certs/aplcdhen01.jhuapl.edu.pem'

dex:
  client_secret: 334389048b872a533002b34d73f8c29fd09efc50
  client_id: localhost
  connectors:
    - type: ldap
      id: ldap
      name: LDAP
      config:
        host: aplid.jhuapl.edu:636
        insecureSkipVerify: false
        userSearch:
          baseDN: cn=users,dc=jhuapl,dc=edu
          filter: "(objectClass=person)"
          username: uid
          idAttr: uid
          emailAttr: mail
          nameAttr: displayName
          preferredUsernameAttr: uid
        groupSearch:
          baseDN: ou=Groups,dc=dom1-proxy,dc=apl-staging
          filter: "(objectClass=group)"
          userMatchers:
            - userAttr: DN
              groupAttr: member
          nameAttr: cn
  frontend:
    theme: ondemand

Then I ran the generator and restarted httpd24…

[root@aplcdhen01 config]# grep OIDC /opt/rh/httpd24/root/etc/httpd/conf.d/ood-portal.conf
[root@aplcdhen01 config]#

What version of OnDemand is installed? Can check with rpm -q ondemand and also is Dex installed via RPM? Can check with rpm -q ondemand-dex. The Dex logic for OnDemand was added with the 1.8 releases.

The code for OnDemand checks 3 things to see if Dex logic should be deployed and Dex defaults set:

  1. Dex configurations are specified - this you are doing so this is fine
  2. /etc/ood/dex directory exists , does that directory exist?
  3. /usr/sbin/ondemand-dex binary exists, does /usr/sbin/ondemand-dex exist?

If you installed via RPM #2 and #3 are handled automatically. If your using some other method for installing OnDemand then extra steps will need to be taken to ensure that Dex is seen as actually installed by OnDemand.