Enabling ssl/https failing

Get Help
1

I am installing a new production Open OnDemand on a RHEL 8.3 VM. Using Dex with ldap for authentication. I had everything working like I wanted until I tried to add the ssl certs and switch to https. As expected when you go to http://ondemand.rc.colorado it redirects to https but it fails with

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at root@localhost to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log.

The logs (/var/log/httpd/ondemand.rc.colorado.edu_error_ssl.log) reports the following errors:

[Tue Aug 03 11:09:06.185239 2021] [auth_openidc:error] [pid 24200:tid 140328719869696] [client 38.97.215.198:35442] oidc_util_http_call: curl_easy_perform() failed on: https://ondemand.rc.colorado.edu:5554/.well-known/openid-configuration (SSL certificate problem: unable to get local issuer certificate)
[Tue Aug 03 11:09:06.185355 2021] [auth_openidc:error] [pid 24200:tid 140328719869696] [client 38.97.215.198:35442] oidc_provider_static_config: could not retrieve metadata from url: https://ondemand.rc.colorado.edu:5554/.well-known/openid-configuration

Ports on firewall are open. I feel like I am missing something obvious.

Config file is all defaults except for the following:

---
servername: ondemand.rc.colorado.edu

ssl: 
  - 'SSLCertificateFile "/etc/pki/tls/certs/ondemand_rc_colorado_edu_cert.crt"'
  - 'SSLCertificateKeyFile "/etc/pki/tls/private/ondemand.rc.colorado.edu.key"'
  - 'SSLCertificateChainFile "/etc/pki/tls/certs/ondemand_rc_colorado_edu_interm.crt"'

use_maintenance: false

host_regex: '[^/]+'

node_uri: /node

rnode_uri: /rnode

dex:
  connectors:
    - type: ldap
      id: ldap
      name: LDAP
      config:
        host: ldap.rc.int.colorado.edu:636
        insecureSkipVerify: false
        userSearch:
          baseDN: ou=UCB,ou=People,dc=rc,dc=int,dc=colorado,dc=edu
          filter: "(objectClass=posixAccount)"
          username: uid
          idAttr: uid
          emailAttr: uid
          nameAttr: gecos
          preferredUsernameAttr: uid
        groupSearch:
          baseDN: ou=UCB,ou=Groups,dc=rc,dc=int,dc=colorado,dc=edu
          filter: "(objectClass=posixGroup)"
          userMatchers:
            - userAttr: DN
              groupAttr: member
          nameAttr: cn
2

Hi,

Looking at the error log it does seem like maybe a step was missed. You probably already did this but the first thing I wanted to check was after the changes, did you update restart the services:
sudo /opt/ood/ood-portal-generator/sbin/update_ood_portal
sudo systemctl restart ondemand-dex
sudo systemctl restart httpd

Assuming you did that, I would check the permissions on the directories and the files, along with correct ownership of the directories and files. Knowing those are all configured correctly would help with further troubleshooting.

3

I did rebuild the config and then restarted the services. I have even rebooted the VM. Which files and directories should I be concerned with?

4

The actual SSL directories and files, so:

/etc/pki/tls/certs/ondemand_rc_colorado_edu_cert.crt
/etc/pki/tls/private/ondemand.rc.colorado.edu.key
/etc/pki/tls/certs/ondemand_rc_colorado_edu_interm.crt

Ensure that root owns all the directories and files. Ensure the permissions on the directories are all 700 and permissions on the files themselves should be 600.

5

Checked file directory and permissions and corrected where necessary. Restarted httpd and ondemand-dex; no change; same error in logs.

Rebooted VM. No change.

Weird issue though. The instance is supposed to be operating as ondemand.rc.colorado.edu. When I try and go the actual VM name https://ondemand2.rc.int.colorado.edu, I get the apache test page and an insecure webpage warning,

Jack

6

Hm, it may be that some default apache configs are getting in the way.

I’m unfortunately not too familiar with RH 8’s httpd.conf file locations, but if you look somewhere like:
/opt/rh/httpd24/root/etc/httpd/conf/httpd.conf
Or possibly:
/etc/httpd/httpd.conf

See if any of the listen directives are still uncommented, and then change them like so:
#Listen 80

This may solve the problem you just mentioned, though not necessarily the larger issue, but let’s start there for now because it may be a port conflict that is throwing things off.

7

Also, I noticed there is a dc=int in the dex portion of the config, which is used in some of the URLs, but was not in the initial error message from the first message you had sent in.

Looking at the servername I also don’t see it set there either, which makes me wonder about whether the correct servername is even being used by OOD itself, and that might be why you are also seeing apache pages.

I’d see if adjusting the servername with the usual update_ood_portal and restarts to see if that has any effect.

8

Commented out the one line of Listen 80 in /etc/httpd/conf/httpd.conf. Restarted httpd and ondemand-dex; no change.

Found other other Listen directive in /etc/httpd/conf.d/ssl.conf. Commenting that one out prevents httpd from starting.

Changed servername to ondeman2.rc.int.colorado.edu (this is the internal hostname and what you get with hostname). Rebuilt config and restarted server; No change.

9

Hey sorry, I was speaking only about the httpd.conf files for those Listen directives.

I’m assuming you uncommented the ssl.conf entry after the failed httpd restart.

It seems like this has to be something simple, the permissions are fine, the ownership is fine, the locations look good, you’ve rebuilt and restarted the services, and when you are saying “no change” does that mean you are still seeing the initial “Internal Server Error” message? Trying to make sure I know what you are seeing correctly since the start.

Also, I’m not sure the regex in host_regex is capturing correctly. I think you need something like [^\/]+ for that to capture correctly. https://regex101.com is great to test on.

10

So I added the following two lines to /etc/httpd.conf.d/ood-portal.conf.

OIDCSSLValidateServer Off
OIDCOAuthSSLValidateServer Off

This took me one step closer to success. I am now seeing the DEX login screen. Login how ever fails with the following message on the screen.

image

There is no ca-certificates file anywhere on the system. Not really sure why it is looking in the /etc/ssl/certs directory in the first place. Here is a listing of that directory.

[root@ondemand2 conf.d]# ll /etc/ssl/certs/
total 24
lrwxrwxrwx. 1 root root   49 Jun 22  2020 ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
lrwxrwxrwx. 1 root root   55 Jun 22  2020 ca-bundle.trust.crt -> /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
-rw-------. 1 root root 4098 Jul 26 12:41 localhost.crt
-rw-------. 1 root root 2874 Aug  2 16:56 ondemand_rc_colorado_edu_cert.crt
-rw-------. 1 root root 5660 Aug  3 09:39 ondemand_rc_colorado_edu_interm.crt
-rw-------. 1 root root 2281 Jul 26 12:26 postfix.pem

Jack

11

Ok, this is looking more familiar. I think at this point the issue has to do with the Certificate Chain. The fix I’ve seen work previously is to cat your ondemand_rc_colorado_edu_interm.crt into the site cert:

cat ondemand_rc_colorado_edu_interm.crt >> ondemand_rc_colorado_edu_cert.crt

Then, remove the SSLCertificateChainFile entry in your ood_portal.yml and see if this now works.

So to be clear, your ood_portal.yml will only have two entries now under ssl, the SSLCertificateFile and the SSLCertificateKeyFile, with the SSLCertificateChainFile having been removed and the contents of the CAChain having been added to the CertificateFile itself.

Hopefully that will clear this latest error.

12

Yes that did it. Can now login and see that everything is working as expected. Thanks for the help.

Jack