Unknown certificate authority when using openondemand-dex

I am trying open ondemand v1.8 with dex authentication but I see an “Internal Server Error” when trying to access and the log shows:

messages:Aug 26 10:44:52 ood ondemand-dex: http: TLS handshake error from 10.2.0.4:34412: remote error: tls: unknown certificate authority

I have obtained a certificate with letsencrypt (full script to do this is here: https://github.com/edwardsp/Azure-OnDemand/blob/master/azod-slurm/scripts/setup-https-certificate.sh)

The certificate works ok in the webpage and previously with basic auth.

Any ideas?

Thanks,
Paul

Dex uses the system’s CA trust store to validate certificates and will fail if it thinks the certificate is self-signed. The only way to get around this is to ensure the SSL certificates CA is in the system trust store. First I’d recommend seeing if an update to ca-certificates package solves the problem, that will update the system’s CA trust store.

If updating ca-certificates RPM does not solve the issue the next step is adding the Let’s Encrypt CA to system trust store. Based on your script do something like this:

cp /etc/ssl/${DOMAIN}/letsencryptauthorityx3.pem.txt /etc/pki/ca-trust/source/anchors/
update-ca-trust extract

Another thing that could be happening is if you used IP addresses and not hostnames present in cert then Dex will think the TLS cert is invalid. If you could provide the contents of /etc/ood/dex/config.yaml and remove anything sensitive.

Here is the config.yaml:

[hpcadmin@ood ~]$ sudo cat /etc/ood/dex/config.yaml
---
issuer: https://ood.gccuyzl1gunezber3txbsvqcja.ax.internal.cloudapp.net:5554
storage:
  type: sqlite3
  config:
    file: "/etc/ood/dex/dex.db"
web:
  http: 0.0.0.0:5556
  https: 0.0.0.0:5554
  tlsCert: "/etc/ood/dex/ood0097ca.westeurope.cloudapp.azure.com.crt"
  tlsKey: "/etc/ood/dex/ood0097ca.westeurope.cloudapp.azure.com.key"
telemetry:
  http: 0.0.0.0:5558
staticClients:
- id: ood.gccuyzl1gunezber3txbsvqcja.ax.internal.cloudapp.net
  redirectURIs:
  - https://ood.gccuyzl1gunezber3txbsvqcja.ax.internal.cloudapp.net/oidc
  name: OnDemand
  secret: 
oauth2:
  skipApprovalScreen: true
enablePasswordDB: true
frontend:
  dir: "/usr/share/ondemand-dex/web"
  theme: ondemand

My external address is ood0097ca.westeurope.cloudapp.azure.com and my internal address is ood.gccuyzl1gunezber3txbsvqcja.ax.internal.cloudapp.net (I have not edited any of these and it is what was created for me) - could these being different be the problem?

Thanks,
Paul

The internal vs external could be a problem if both addresses are not part of the certificate’s CN or Subject Alternative Name. Also the redirectURIs should probably point to external address.

The way the Dex authentication works is you come in from ood0097ca.westeurope.cloudapp.azure.com and then will get redirected likely to ood0097ca.westeurope.cloudapp.azure.com:5554 for Dex.

You can check your cert with something like openssl x509 -noout -text -in /etc/ood/dex/ood0097ca.westeurope.cloudapp.azure.com.crt. The Subject should have one of those internal or external addresses like CN=ood0097ca.westeurope.cloudapp.azure.com. You will then need to ensure there is a X509v3 Subject Alternative Name with a value like DNS:ood.gccuyzl1gunezber3txbsvqcja.ax.internal.cloudapp.net.

If possible you should probably only use the external address with Dex and not use the internal address since the external address is what Dex will see from user requests as well as what it needs to redirect back to. The default behavior of the ood-portal-generator is to use the address from servername config option and if that’s not defined uses the hosts FQDN. You could try adding this to ood_portal.yml:

servername: ood0097ca.westeurope.cloudapp.azure.com

The servername in ood_portal.yml made everything consistent in the dex config.yml. But adding the Let’s Encrypt CA to the system trust store fixed the problem (I didn’t know where to find an update ca-certificates rpm so I just ran those commands you pasted above).

Thanks for all your help,
Paul

Typically you’d just do yum update ca-certificates to see if there is an update available but that package isn’t updated that frequently so could be that trust store just missing Let’s Encrypt CA.