SOLVED iptables-restart disables access to to Incredible PBX

[panacea]

Hi all,

A week or so ago I used the GO PUBLIC script to make Incredible PBX 2020 (Asterisk 16.12.0 and Incredible GUI 15.0.12.28) publicly available and everything had been working great. However, as of a couple of days ago I started waking up to the extension on my phone being unregistered. I was unable to access the PBX or Webmin through a browser and I could no longer SSH into it either.

I contacted Crowncloud and they did something to fix the problem - they said they had "reconfigured the network" and things worked for the rest of the day. The following day the PBX was inaccessible again and I found that I could get access to it again by shutting down iptables (systemctl stop iptables). As I didn't want to leave the PBX just sitting there unprotected I started up iptables again and found that the PBX was accessible still (through the web and SSH and the extensions registered fine). I thought the issue was fixed so left it and the issue happened again today.

Today I have been able to reproduce the issue by running iptables-restart. When I run this command I lose access to the PBX. If I then shut down iptables (systemctl stop iptables) and then start it up again I can access the PBX again. Run iptables-restart and I lose access again.

Any pointers from anyone on how to troubleshoot this would be greatly appreciated.

Cheers

 

[krzykat]

I'd check your DNS server and make sure it has no issues. Not sure which setup you have, but I know in past, you could get jammed up if your firewall was trying to run a rule dependent upon a FQDN that it couldn't resolve.

 

[panacea]

Thanks kryskat.

In terms of setup I have a PBX hosted on CrownCloud using one the image they have available.

How/where can I check the DNS/FQDN side of things you mention. Does iptables-restart do something different to systemctl stop iptables followed by systemctl start iptables?

Cheers

 

[tbrummell]

Sounds like you're being banned by F2B, and when you restart iptables it clears the ban. Next time it happens run iptables -vnL and look for your IP in the list.
If it's there, you need to look at the Asterisk full logs to see why you're being banned, you can also check the fail2ban.log file to see when it happened, and that'll also tell you what jail is triggering it.
Right, and restart iptables with iptables-restart, not systemctl.

 

[krzykat]

Yes, I know that if you have an issue where you are banned because say for example you had an improper password on a device that was trying to register, that simply going in and doing iptables stop / start - will allow you back in and things will seem good. But then each day when iptables is reloaded, then Fail2Ban will reload that IP and ban it again. Do as @tbrummell says and check your fail2ban to see if your IP is in there, makes perfect sense that this is what is going on.

I'd go into /etc/fail2ban/jail.conf and make sure that you tell fail2ban to ignore your own public IP address. Should be around row 34

 

[tbrummell]

I'd go into /etc/fail2ban/jail.conf and make sure that you tell fail2ban to ignore your own public IP address. Should be around row 34

I was going to suggest that, but it doesn't fix the problem, that was my thought at least. Suggesting the solution makes them at least troubleshoot a little bit.
Teach a man to fish.....and all of that.

 

[panacea]

Thanks kryskat and tbrummell. I've looked through the fail2ban side of things as suggested and my IP does not appear to be blocked.

When this issue arises it is not just my connection that is blocked. All other extensions (12 in total) also get disconnected. The only access I have to the PBX is the VNC console from CorwnCloud. I've checked through the logs to retrieve the IPs where the extensions registered from and none of them are in the jail.conf or in the output of iptables -vnL.

I've been doing all the troubleshooting through the VNC console so hadn't noticed when the disconnection took place. Running through a putty session the disconnection takes place when setting specific permission after the base permissions have been set. Also, there's that error about not being able to create the symbolic link. A few seconds after the last line displays in the putty screen I get an error saying the connection has been lost.

WARNING: Always run Incredible PBX behind a secure firewall.
root@noreply:~ $ iptables-restart
Redirecting to /bin/systemctl restart iptables.service
ln: failed to create symbolic link ‘/usr/sbin/iptables’: File exists
Redirecting to /bin/systemctl restart ip6tables.service
Redirecting to /bin/systemctl restart fail2ban.service
Taking too long? Customize the chown command, See http://wiki.freepbx.org/display/FOP/FreePBX+Chown+Conf
Setting Permissions...
Setting base permissions...Done
Setting specific permissions...
2100 [>---------------------------]

Cheers

 

[tbrummell]

All other extensions (12 in total) also get disconnected.

Are they all at the same location, or multiple locations?

error about not being able to create the symbolic link.

That's normal.

the putty screen I get an error saying the connection has been lost.

That tells me the firewall rules are dumping your connection.

You need to check, when your disconnected, for the ban, not before. Also, F2B reloads at 12am, which also restarts iptables, that's probably when you're getting dumped.

Show us the complete output of iptables -vnL pls. Put it in Code blocks, or attach as a file.

 

[ostridge]

Does iptables-restart do something different

Yes it does on the RPi

root@incrediblepbx:~# which iptables-restart
/usr/local/sbin/iptables-restart
root@incrediblepbx:~# ls /usr/local/sbin/iptables-*
/usr/local/sbin/iptables-custom  /usr/local/sbin/iptables-facebook  /usr/local/sbin/iptables-knock  /usr/local/sbin/iptables-restart
root@incrediblepbx:~#
root@incrediblepbx:~# cat /usr/local/sbin/iptables-restart

This ONLY on RASPBIAN
##!/bin/bash
#systemctl restart netfilter-persistent
#/usr/local/sbin/iptables-custom
#if [ -d "/etc/fail2ban" ]; then
# service fail2ban restart
#fi
#/usr/sbin/fwconsole chown
#if [ -d "/var/www/html/avantfax" ]; then
# chmod -R 777 /var/www/html/avantfax
# chown -R asterisk:asterisk /var/www/html/avantfax
# chmod -R 0770 /var/www/html/avantfax/tmp /var/www/html/avantfax/faxes
# chown -R asterisk:uucp /var/www/html/avantfax/tmp /var/www/html/avantfax/faxes
#fi

So it runs a custom script 'iptables-custom' before restarting fail2ban.

EDIT:
/usr/local/sbin/iptables-custom includes the following code for you to edit for server access

# sample custom rule for full server access looks like this
# /usr/sbin/iptables -A INPUT -s yourFQDN.dyndns.org -j ACCEPT

Your system may differ.

 

[panacea]

Are they all at the same location, or multiple locations?

That's normal.

That tells me the firewall rules are dumping your connection.

You need to check, when your disconnected, for the ban, not before. Also, F2B reloads at 12am, which also restarts iptables, that's probably when you're getting dumped.

Show us the complete output of iptables -vnL pls. Put it in Code blocks, or attach as a file.

Success! My IP and a bunch of others are in the Chain f2b-ASTERISK and Chain f2b-asterisk-udp.

I ran fail2ban-client set asterisk unbanip xxx.xxx.xxx.xxx and after an iptables-restart I did not lose connectivity.

Thanks for the prompt responses. Really appreciate everyone's help.

 

[panacea]

Yes it does on the RPi

root@incrediblepbx:~# which iptables-restart
/usr/local/sbin/iptables-restart
root@incrediblepbx:~# ls /usr/local/sbin/iptables-*
/usr/local/sbin/iptables-custom  /usr/local/sbin/iptables-facebook  /usr/local/sbin/iptables-knock  /usr/local/sbin/iptables-restart
root@incrediblepbx:~#
root@incrediblepbx:~# cat /usr/local/sbin/iptables-restart

This ONLY on RASPBIAN
##!/bin/bash
#systemctl restart netfilter-persistent
#/usr/local/sbin/iptables-custom
#if [ -d "/etc/fail2ban" ]; then
# service fail2ban restart
#fi
#/usr/sbin/fwconsole chown
#if [ -d "/var/www/html/avantfax" ]; then
# chmod -R 777 /var/www/html/avantfax
# chown -R asterisk:asterisk /var/www/html/avantfax
# chmod -R 0770 /var/www/html/avantfax/tmp /var/www/html/avantfax/faxes
# chown -R asterisk:uucp /var/www/html/avantfax/tmp /var/www/html/avantfax/faxes
#fi

So it runs a custom script 'iptables-custom' before restarting fail2ban.

EDIT:
/usr/local/sbin/iptables-custom includes the following code for you to edit for server access

# sample custom rule for full server access looks like this
# /usr/sbin/iptables -A INPUT -s yourFQDN.dyndns.org -j ACCEPT

Your system may differ.

Thanks

 

[tbrummell]

Fixes it for now. If you have a misconfigured device it will just ban you again.

 

[panacea]

Ah...any way to tell why the IPs are being blocked?

 

[tbrummell]

/var/log/asterisk/full
Cat & grep that log for your IP...

 

[tbrummell]

Ie: cat /path/to/log | grep your.ip.address.

 

[panacea]

The log from 20200929 has three entries with an incorrect password when registering my extension from my blocked IP. Would that do it? Is three the magic number?

EDIT - I found all the IP addresses and at some point in the past four days all show up with incorrect password entries.

Thanks again for all the assistance.

 

[wardmundy]

That's the whole idea behind Fail2Ban.

 

[krzykat]

Yes, I know that if you have an issue where you are banned because say for example you had an improper password on a device that was trying to register

 

[tbrummell]

Is three the magic number?

Depends on your f2b settings, I'm a little more strict then that, but yes, 3 sounds like the default setting.

 

[krzykat]

Is three the magic number?

Depends on what part. Read the /etc/fail2ban/jail.conf file.

It can set SSH separate from SIP registry.