SOLVED Potential Firewall Issue

[krzykat]

I had a MAJOR issue this morning that affected a LOT of servers.
We couldn't ping anywhere - and thought we had a Vultr issue, only to realise that it wasn't only in one of their colo's.

Upon them saying it wasn't them it took a little while to find a resolution. We had tried disabling IPtables and that wasn't any help either.
Finally what worked was going into rc.local and commenting out 2 lines and restarting the server.

Those lines were for:
/etc/blockem.sh
and
/usr/local/sbin/iptables-restart

Me and my team have bene in frantic mode since like 8am figuring it out. Got it up a couple hours ago, but wanted to throw a quick post here in case anyone else had issues.

I've NOT tracked down the exact cause/effect yet, but plan to so that knowledge is shared.

 

[atsak]

Did your PBX IP end up on a blocklist or did someone screw up and put 0.0.0.0 on the list?

 

[dhoppy]

That seems to be related to the blacklists. I've also had problem with those and removed them from my servers.

 

[wardmundy]

The trick is to make sure your whitelisted entries that matter are above the blacklist rules. Then, if your IP address ends up on the blacklist, it won't blow your servers out of the water.

 

[krzykat]

OK - here is a detailed explanation, cause, effect, and resolution:

The issue is a server ipdeny.com SSL using letsencrypt coupled with openssl on centos.
If you issue the command

echo | openssl s_client -connect www.ipdeny.com:443 | head

You will find verify error: certificate has expired. This really messes things up.

The resolution is to modify the blockem.sh located in /etc and comment out the row (5 for me)
# wget -qO - http://www.ipdeny.com/ipblocks/data/countries/all-zones.tar.gz| tar zxvf -

and replace it with one that doesn't check for SSL certificate verification:

wget -qO - http://www.ipdeny.com/ipblocks/data/countries/all-zones.tar.gz --no-check-certificate | tar zxvf -

Then things will work properly. Please note, I have tested the openssl check on both Debian and Rocky, and both perform properly.

The openSSL that I have on Centos with the issue is

OpenSSL 1.0.2k-fips 26 Jan 2017

 

[krzykat]

Also - this is something new (started today for us on MANY servers - that are still on Centos 7) - in that we have our servers daily reissue the IPtables in order to get all the new badguys updates, and also if there is a FQDN of a whitelist that has changed, it gets updated too. I suspect that others with Centos will experience this problem when they reboot their servers or restart iptables.

 

[wardmundy]

OK - here is a detailed explanation, cause, effect, and resolution:

The issue is a server ipdeny.com SSL using letsencrypt coupled with openssl on centos.
If you issue the command

echo | openssl s_client -connect www.ipdeny.com:443 | head

You will find verify error: certificate has expired. This really messes things up.

The resolution is to modify the blockem.sh located in /etc and comment out the row (5 for me)
# wget -qO - http://www.ipdeny.com/ipblocks/data/countries/all-zones.tar.gz| tar zxvf -

and replace it with one that doesn't check for SSL certificate verification:

wget -qO - http://www.ipdeny.com/ipblocks/data/countries/all-zones.tar.gz --no-check-certificate | tar zxvf -

Then things will work properly. Please note, I have tested the openssl check on both Debian and Rocky, and both perform properly.

The openSSL that I have on Centos with the issue is

OpenSSL 1.0.2k-fips 26 Jan 2017

FYI: My records show this was fixed in the Dec. 1, 2021 release of http://incrediblepbx.com/blockem.tar.gz

 

[krzykat]

Could be, this was on older Centos 7 servers that hadn't been upgraded yet.