TIPS Narrowed down delay issue on Dashboard access but can use some help to fix it

[Halea]

I figured out a simple way to interface citadel mail with IncrediblePBX. Actually it would be more accurate to say with the basic Linux email transmission subsystem.

Basically, one needs to simply point the sendmail command to the citadel mail, so that Citadel-MTA sends out the email messages:

# Save original sendmail program, just in case:
mv /usr/sbin/sendmail /usr/sbin/sendmail.oldstuff

# Create a symbolic link between sendmail and citmail:
ln -s /usr/local/citadel/citmail /usr/sbin/sendmail

This also works with the mail command which is used by my sms to email gateway.

The last thing I need to do with Citadel Mail is to open up ports so that I can get sms return messages by email.

 

[Halea]

Ok Citadel mail is working fine on IncrediblePBX 2021 (only tested with Ubuntu 20.04 so far) for both outgoing and incoming* mail (including with SSL). Its web based user interface is fast and running on its own independent web server (webcit on a custom TCP port using SSL).

*But, to get it to work for the incoming and the web user interface I had to turn off IncrediblePBX's firewall temporarily, as I couldn't figure out a way to allow incoming access to the TCP ports 25,587,465,143,993,110,995.

@wardmundy : With the current firewall logic in place is there a way I can whitelist the above ports for any and all IP sources? I understand currently only whitelisted FQDNs can access and only a specific set of TCP/UDP ports.
For a PBX behind a pfsense firewall I wouldn't worry about turning off iptables permanently, but this test system is on a VPS and I like your firewalling approach very much.

 

[wardmundy]

Just add the ports into /etc/iptables/rules.v4 and iptables-restart.

For example, on non-PUBLIC server:

-A INPUT -p tcp -m tcp --dport 25 -j ACCEPT

On PUBLIC server:

-I INPUT -p tcp -m tcp --dport 25 -j ACCEPT

 

[Halea]

I finally got my Ubuntu 20.04 based IncrediblePBX testbench configured as I wanted. It works well. It's a bit different from the standard recipe. I got it built per the following steps:

1) Fire up Ubuntu 20.04 server on a $2/mo RackNerd VPS in Atlanta.
2) Install IncrediblePBX per Ward's instructions.
3) Install OpenVPN server.
4) Install Unbound DNS server.
5) Install Citadel Mail email server.
6) Install my own sms to email & email to sms gateway.
6) Install Xfce4.
7) Install Timeshift snapshot backup/restore utility.

The PBX works well.
The dashboard delay thing is history.
The only small problem is the UCP node not running message on the dashboard, but I understand that has to do with a commercial module which is no use to me. If that's indeed the case I should probably be able to uninstall it, right?

 

[Halea]

Just wanted to update this thread as it has been 3 months now that the Ubuntu based IncrediblePBX-2021 test bench mentioned in the OP has been in active use.

The configuration mentioned in my 3/9/2021 post held up well.

Several SIP trunks, some using chan_sip others chan_pjsip, have been setup and have worked well.
Among the termination providers that I typically use only SignalWire has been problematic when it was setup with the chan_pjsip driver. It would work, but for some obscure reasons it would almost always overwrite the outgoing CID with a random one from the verified phone numbers pool under my SignalWire account, rather than transmitting the verified phone number actually sent by the PBX. This odd behavior didn't occur when I converted the trunk to use chan_sip, go figure...

Several tens of physical phone extensions scattered in the USA, EU and Australia, all using chan_sip, have joined in for audio quality testing, codec handpicking and latency vs hardware/network optimization. Those calls also helped us figure out better ways to shape IP traffic through our pfsense routers depending on the connectivity providers in different countries. For the codecs, as one might expect, ulaw and alaw truly yield the best quality, especially while conferencing in speakerphone mode. For traffic shaping in pfsense Codel turned out to be the best most of the time even though its overhead turned out to be surprisingly high.

There is one issue that I have encountered repeatedly (on my Ubuntu instances only) for which I don't have a fix yet. That's the "Live Network Usage" applet on the dashboard which works for a while and suddenly can no longer update. My only recovery method for now is to reload the last snapshot made with Timeshift.

BTW, running Timeshift in the VM turned out to be a brilliant idea. It allowed me countless times to recover from mishaps which would have otherwise required a full re-install or dd-restore.

I can't recommend this enough: always install a lightweight desktop on your server instance (like xfce) and basic gui tools like bootrepair, grub-customizer, timeshift and gparted as a minimum.
You don't need to run the GUI all the time. To prevent it from autostarting simply edit /etc/X11/default-display-manager and replace the display manager line with "true". That will make the OS boot up to the main console and stop.
When you need the GUI you can run startx or startxfce4.

As for Timeshift, set it up so that its own scheduler is off. Make a cron job that runs "timeshift --create" a few times a day at times that you want/need. Timeshift is rsync based and it usually doesn't impact your PBX's audio performance unless your VM is truly overtaxed.

Two additional things that positively surprised me which are worth mentioning:
1) The RackNerd virtual machines work really well. Even the cheapest ones ($1 or $2 per month). I now have 3 test PBXs (2xDebian+1xUbuntu) and 3 VPNs (2xUbuntu+1xDebian) on RackNerd and experienced zero down time (in 4 months). (For production we use our own Debian blades colocated at a USA data center and connect directly to Hurricane Electric networks)
2) Hurricane Electric provided Internet connectivity (IP v4 and IP v6) works mindbogglingly well. Those guys are in a league on their own. If they are available for interconnect in your area and you can afford it - well don't think twice. They are the best.

Last bit: Although in the past I interchangeably used Debian and Ubuntu servers for anything and everything, after several months of more focused tests I came to realize that Debian is actually easier to troubleshoot than Ubuntu although Ubuntu is a tad easier to setup than Debian.

Edit: Best codec & limiter info added. Debian/Ubuntu differentiation added regarding the network applet refresh failure.

Edit2: Forgot to mention that I eventually elected to use low-latency kernels both for Debian 10 (Linux 5.10.0-0.bpo.5-rt-amd64) and Ubuntu 20.04 (Linux 5.8.0-55-lowlatency).