Errors with rhel-server-rhscl-7-rpms repo in the past week?

We are having an issue that seems to have appeared since last week with building OOD on our centos platform and using the rhel-server-rhscl-7-rpms repo. In fact it appears to have emerged just over the past day. Yesterday I could work around it by forcing a specific baseurl for this repo. Today, even the base url I used yesterday is not working. :confused:

The yum install ondemand results in a bunch of 404 errors looking for packages:

rh-git29-2.3-5.el7.x86_64.rpm FAILED
http://www.gtlib.gatech.edu/pub/centos/7/sclo/x86_64/rh/rh-git29/rh-git29-2.3-5.el7.x86_64.rpm: [Errno 14] HTTP Error 404 - Not Found

Hmmm http://www.gtlib.gatech.edu/pub/centos/7/sclo/x86_64/ is accessible but http://www.gtlib.gatech.edu/pub/centos/7/sclo/x86_64/rh/ took a long time to access. Then the only git available from gtlib.gatech.edu is now Git 2.18 (https://www.softwarecollections.org/en/scls/rhscl/rh-git218/) and Git 2.9 (https://www.softwarecollections.org/en/scls/rhscl/rh-git29/) is no longer available from that source. Investigating…

Hey @efranz
We were debugging this issue last night, the techlinks mirror wasn’t working
Today it seems like the gitlab repo is down as well
http://www.gtlib.gatech.edu/pub/centos/7/sclo/x86_64/rh/rh-git29/rh-git29-2.3-5.el7.x86_64.rpm: [Errno 14] HTTP Error 404 - Not Found

Here’s an example of the error we saw on teklinks (this frequently resolves as our closest mirror via mirrorlist). It’s the same error as for gtlib just a different package. In the full output it’s all the sclo packages.

http://mirror.teklinks.com/centos/7.7.1908/sclo/x86_64/rh/rh-nodejs6/rh-nodejs6-nodejs-realize-package-specifier-3.0.3-2.el7.noarch.rpm: [Errno 14] HTTP Error 404 - Not Found

Given this is what I saw yesterday afternoon and was then able to use gtlib successfully, I suspect whatever the change was it occurred with yesterday’s mirror updates and teklinks just updated sooner than gtlib.

Because of the release of CentOS 7.7 they EOL SCL packages are now not included in the CentOS SCL repos as they didn’t build those RPMs as they are EOL.

As a temporary work around you will have to install the CentOS SCLo Vault.

cat > /etc/yum.repos.d/centos-scl-vault.repo <<EOF
[C7.5.1804-SCLo]
name=CentOS-7.5.1804 - Updates
baseurl=http://archive.kernel.org/centos-vault/7.5.1804/sclo/x86_64/sclo/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-SCLo
enabled=1

[C7.5.1804-SCLo-rh]
name=CentOS-7.5.1804 - Updates
baseurl=http://archive.kernel.org/centos-vault/7.5.1804/sclo/x86_64/rh/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-SCLo
enabled=1
EOF

I was unable to reproduce this with centos:7 docker container using this in /etc/yum.repos.d/CentOS-SCLo-scl-rh.repo

[centos-sclo-rh]
name=CentOS-7 - SCLo rh
baseurl=http://mirror.centos.org/centos/7/sclo/$basearch/rh/
#mirrorlist=http://mirrorlist.centos.org?arch=$basearch&release=7&repo=sclo-rh
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-SCLo

If/when the CentOS 7.6 vault gets populated, you could update the baseurl in the work around. As far as I could tell the CentOS 7.6 vault repos are still empty.

So if we update to CentOS7.7 does this issue magically go away? Is it just that we happen to be building on 7.6?

It depends on how your repo files are defined. If your repo files are pointed to SCL repos that are pinned to what was available when CentOS 7.6 was the latest OS , then you should be fine. If your repos are not pinned then this issue will manifest itself on a fresh install. If you installed OnDemand on CentOS 7.6 then do “yum update”, you’ll be fine but fresh installs won’t be able to access the EOL SCL packages if your just install “CentOS 7”.

If your doing a fresh install on CentOS 7.6 or 7.7 using stock CentOS repo files, the work around is necessary. If your upgrading existing CentOS 7.6 to 7.7 then this issue shouldn’t occur.

Our build environment was built using CentOS 7.6 a few months ago and all the packages were available then.

We install fresh all the time in dev. :slight_smile: We are using packer to build OOD images for our dev deployments and using the CentOS7 cloud images as the base. Current is v1905 which appears to be CentOS7.6. I’m gonna try out 1907 which might be 7.7.

So just to be clear, you are saying that regardless of 7.6 or 7.7 a fresh install will suffer this error unless we use the work around above?

I believe the workaround is needed regardless of 7.6 or 7.7 if you are doing a fresh install. Normally CentOS repos will just install CentOS 7 (latest) which now doesn’t include several SCL packages needed by OnDemand. Usually the only way to install non-latest is to use the Vault repos or use local copies that are points-in-time. Normally if you install CentOS 7.6 with stock repos then do “yum update” you’ll be at CentOS 7.7. Also installing packages with stock repos from CentOS 7.6 will pull in packages built for CentOS 7.7 as CentOS is a rolling upgrade type OS. If your using 7.6 specific repos this may not be the case but only ones I know of are the vault repos.

For my work around it looks like CentOS main mirror has the packages:

Use these base URLs:

http://mirror.centos.org/centos/7.6.1810/sclo/x86_64/rh/

http://mirror.centos.org/centos/7.6.1810/sclo/x86_64/sclo/

Upon further investigation the SCL repos from CentOS are minor version specific. The mirror URL generates 7.7 URLs so replacing the mirrorlist with a baseurl should be sufficient:

cat > /etc/yum.repos.d/centos-scl.repo <<EOF
[C7.6.1810-SCLo]
name=CentOS-7.5.1804 - SCLo
baseurl=http://mirror.centos.org/centos/7.6.1810/sclo/x86_64/sclo/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-SCLo
enabled=1

[C7.6.1810-SCLo-rh]
name=CentOS-7.6.1810 - SCLo rh
baseurl=http://mirror.centos.org/centos/7.6.1810/sclo/x86_64/rh/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-SCLo
enabled=1
EOF
1 Like

Fixing just the SCLo rh repo seems to fix the problem for us. Thank you!

@jpr I just happen to recall this. Sounds like you have a really advanced CD pipeline, do you have any documentation on that? Blogs or anything? We’d really like to hear what you’re up to and what benefits you’ve gotten out of such a workflow.