I started a new job 6 weeks ago and I'm still learning the environment. I got a new task yesterday and it was an issue which had been bounced around for a while. The issue seemed simple enough, enable brotli on a virtual host. This is the story of how simple may not be so simple.
Centos 6 is end of life as of November 2020. After a bit of discussion, the plan is to continue to investigate if we can do this as a stopgap until the entire solution can be reworked.
So, how would one investigate how to do this? It turns out, it isn't so trivial, and even just starting an investigate can be tricky. My first thought was, start a docker instance so I can poke around and try various RPMs.
docker pull centos:6.7
docker run -ti centos:6.7
Cool, I've got a prompt, now what?
After searching the web, I see that nginx.org has rpm packages. I'll try one and see if it has brotli support.
curl -LO https://nginx.org/packages/mainline/centos/6/x86_64/RPMS/nginx-1.19.5-1.el6.ngx.x86_64.rpm
curl: (35) SSL connect error
WAT? ok, maybe the certs on this 2yo docker image (I had checked docker hub) are out of date.
curl -k the same thing and get the same error. WTF? add the
-v to curl and see the libnss message
* warning: ignoring value of ssl.verifyhost Well ain't that something? At this point I mumble under my breath about how debian/ubuntu uses openssl linked curl by default and not libnss and I wonder if it would have behaved the same.
As I write this, I realize that maybe I should have used a more recent Centos 6 docker image, 6.10 perhaps. Unfortunately I'm not as experienced with Centos as I should be. Part of this fun is diving in and learning. I used 6.7 because that is what this server under question says it is.
Alright, so new problem...
yum update says It can't do its thing.
$ yum update Loaded plugins: fastestmirror Setting up Update Process Loading mirror speeds from cached hostfile YumRepo Error: All mirror URLs are not using ftp, http[s] or file. Eg. Invalid release/repo/arch combination/ removing mirrorlist with no valid mirrors: /var/cache/yum/x86_64/6/base/mirrorlist.txt Error: Cannot find a valid baseurl for repo: base
Alright, some searching tells me to use
baseurl instead of
mirrorlist and that it is now at
vault.centos.org. I try that and get a new error:
http://vault.centos.org/centos/6/os/x86_64/repodata/repomd.xml: [Errno 14] Peer cert cannot be verified or peer cert invalid Yes, that says http:, but port 80 just redirects to port 443 and then we get the cert error. We have a catch-22. We need to upgrade our package to get new certs but we need new certs to upgrade our package. We need to upgrade our package to get new certs but we need new certs to upgrade our package. We need to upgrade our package to get new certs but we need new certs to upgrade our package. We need to upgrade our package to get new certs but we need new certs to upgrade our package. cough ouch
(Update: I'm doing this often enough that editing that file takes too long and so there is this:
sed -i 's/#baseurl=http:\/\/mirror/baseurl=http:\/\/vault/;s/mirrorlist/#mirrorlist/' /etc/yum.repos.d/CentOS-Base.repo )
For a solution, I got lucky. I knew to try to get the ca-certificates rpm manually and install it without yum. Worse come to worse I could download it on another host and copy it with
docker cp or use netcat or python simple server or put it on a non-TLS webserver or any other method of moving data around. I was manually browsing the repo and got this URL
curl -LO https://archive.kernel.org/centos-vault/6.10/updates/x86_64/Packages/ca-certificates-2020.2.41-65.1.el6_10.noarch.rpm and it turns out that the cert serving archive.kernel.org was acceptable.
rpm -U ./ca-cert*.rpm
And now I can
yum update ; yum upgrade
Search search search, search the web and you will find no clear answers. There are posts about compiling it yourself. There are posts about using a subscription repo. (https://www.getpagespeed.com)
Incidentally, that subscription repo also had blog post which helped confirm my findings above: https://www.getpagespeed.com/server-setup/how-to-fix-yum-after-centos-6-went-eol along with https://forums.centos.org/viewtopic.php?f=13&t=78238 which verified what I had guessed about updating certificates.
Now my first thought was, nginx has this by default now, right? Use the newest package from nginx.org. Look at all of 'em here: https://nginx.org/packages/centos/6/x86_64/RPMS/
curl -LOed (curllo is a verb ya see) that https://nginx.org/packages/mainline/centos/6/x86_64/RPMS/nginx-1.19.5-1.el6.ngx.x86_64.rpm and installed it (
rpm -i ./nginx*rpm) and was disappointed that it had no brotli (
strings /usr/sbin/nginx | grep brot confirms). (That is after I
yum install initscripts as required by that package)
Keep searching and reading...
The next thing I tried were the packages from https://repo.aerisnetwork.com but, in short they didn't do what I wanted or had some drawback.
Next, I tried poking at the packages from getpagespeed.com, I even browsed in my browser, got this URL:
https://extras.getpagespeed.com/redhat/6/mainline/x86_64/RPMS/nginx-1.21.3-1.el6.ngx.x86_64.rpm and tried to download and install it, but it didn't actually download. Registration is required and apparently they whitelist registered users by IP. Jumping through those hoops would not be sustainable for me in my work environment so I discarded getpagespeed.com.
I have no idea why, but next I tried this
https://nginx.org/packages/centos/6/x86_64/RPMS/nginx-1.18.0-2.el6.ngx.x86_64.rpm Older than the pervious nginx.org package, but it isn't in the
mainline repo and I'm Centos n00b enough that I don't know the difference. Again,
strings /usr/sbin/nginx|grep brot showed nothing, so that got a quick
rpm -e nginx.
Next, I tried
curl -LO https://repo.aerisnetwork.com/archive/nginx-more-1.13.6-1.el6.x86_64.rpm and
https://repo.aerisnetwork.com/archive/nginx-more-1.14.2-4.el6.x86_64.rpm. (No idea why I grabbed 1.13 and 1.14.) There were a bunch of various nginx packages on
repo.aerisnetwork.com For this to work there were some requirements so I ran
yum install gd libxslt but there was still the case of
Next, I tried
repo.codeit.guru packages. I don't recall exactly where I found this repo. I think it was multiple sources. One of them was
https://nixcp.com/brotli-compression-nginx/. I'd probably also seen
https://codeit.guru/en_US/2020/04/nginx-1-18-0-stable-with-brotli-support-tls-1-3-final-rfc-8446-built-against-openssl-1-1-1g-for-red-hat-enterprise-linux-and-centos/. I was desperate enough to copy and page the root repo URL and hope for a Centos 6 directory. There was one!
curl -LO https://repo.codeit.guru/packages/mainline/centos/6/x86_64/nginx-1.19.5-1.el6.codeit.x86_64.rpm rpm -i ./nginx-1.19.5-1.el6.codeit.x86_64.rpm warning: ./nginx-1.19.5-1.el6.codeit.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 898b43f4: NOKEY error: Failed dependencies: libbrotli = 1:1.0.7 is needed by nginx-1:1.19.5-1.el6.codeit.x86_64 libmaxminddb.so.0()(64bit) is needed by nginx-1:1.19.5-1.el6.codeit.x86_64 curl -LO https://repo.codeit.guru/packages/mainline/centos/6/x86_64/libbrotli-1.0.7-1.codeit.el6.x86_64.rpm rpm -i ./libbrotli-1.0.7-1.codeit.el6.x86_64.rpm
There is obviously brotli support but also the same
Now I had to briefly forget nginx and focussing on where to find this dependency. Surely there is a package?
$ yum install libmaxmindddb Loaded plugins: fastestmirror Setting up Install Process Loading mirror speeds from cached hostfile No package libmaxmindddb available. Error: Nothing to do $ yum install libmaxmindddb-dev ... No package libmaxmindddb-dev available. $ yum install libmaxmindddb-devel ... No package libmaxmindddb-devel available.
It required much searching. I wondered if it was in the nginx-module-geoip rpm, but no.
Finally, I found some comments about epel. I didn't have epel repo enabled. It is not part of vault AFAICT, but I was able to web browse epel and find a package.
curl -LO https://archives.fedoraproject.org/pub/archive/epel/6/x86_64/Packages/l/libmaxminddb-1.1.1-5.el6.x86_64.rpm rpm -i ./libmaxminddb-1.1.1-5.el6.x86_64.rpm
And now nginx package from codeit.guru can be installed.
$ rpm -i ./nginx-1.19.5-1.el6.codeit.x86_64.rpm warning: ./nginx-1.19.5-1.el6.codeit.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID 898b43f4: NOKEY --------------------------------------------------------------------------- Thanks for using nginx! Please find the official documentation for nginx here: * http://nginx.org/en/docs/ Please subscribe to nginx-announce mailing list to get the most important news about nginx: * http://nginx.org/en/support.html Commercial subscriptions for nginx are available on: * http://nginx.com/products/ --------------------------------------------------------------------------- $ rpm -ql nginx /etc/logrotate.d/nginx /etc/nginx /etc/nginx/conf.d /etc/nginx/conf.d/default-ssl.conf.example /etc/nginx/conf.d/default.conf /etc/nginx/fastcgi_params /etc/nginx/koi-utf /etc/nginx/koi-win /etc/nginx/mime.types /etc/nginx/modules /etc/nginx/nginx.conf /etc/nginx/php.inc /etc/nginx/scgi_params /etc/nginx/uwsgi_params /etc/nginx/win-utf /etc/rc.d/init.d/nginx /etc/rc.d/init.d/nginx-debug /etc/sysconfig/nginx /etc/sysconfig/nginx-debug /usr/lib64/nginx /usr/lib64/nginx/modules /usr/sbin/nginx /usr/sbin/nginx-debug /usr/share/doc/nginx-1.19.5 /usr/share/doc/nginx-1.19.5/COPYRIGHT /usr/share/man/man8/nginx.8.gz /usr/share/nginx /usr/share/nginx/html /usr/share/nginx/html/50x.html /usr/share/nginx/html/index.html /var/cache/nginx /var/log/nginx $ strings /usr/sbin/nginx | grep brot ngx_http_brotli_static_module_ctx ngx_http_brotli_filter_module ngx_http_brotli_static_module ...
Look at those beautiful symbols!
Next task is to discuss with the team all of the reasons that we should NOT integrate this into our environment and instead migrate off of Centos 6 instead.