TIPS Suddenly could not log in to ipbx admin. Now navigating admin section is weird.

[wardmundy]

You can use a VirtualBox or VMware image if you prefer: https://sourceforge.net/projects/pbxinaflash/files/

Tutorials are also available on Nerd Vittles.

 

[JFrost]

This monster struck again.

About 3 months ago I had this exact same scenario repeat itself. This time on 2021. I do not know what triggers this issue. If I had to guess I'd say something in the way I'm doing updates to the system but that's just an outside wild guess.
I have taken your advice @wardmundy and set up a "matching" box. The versions aren't perfect but I think they might be close enough to compare. (see img



I restored a nightly config backup (the internal IPBX backup) to the new machine and the UI continues to work (there are a few module/config errors I don't understand but will deal with that separate)

I ran a file comparison on /var/www/html and while I see a few differences I don't see any that I think are significant to make the UI not work.

Since the restore mostly works to the new machine, my guess is whatever went awry on the old is elsewhere. Can anyone suggest where to look/how to diag the culprit here?
I like IPBX but I really don't feel like I should have to migrate machines every year or so due to UI "exploding" so I'd like to get to the bottom of the issue.

Ty.

 

[wardmundy]

Kinda depends upon what kind of "updates [you're] doing to the system."

 

[JFrost]

Kinda depends upon what kind of "updates [you're] doing to the system."

Really really radical and obscure ones. Stuff that should most asuredly and beyond a shadow of a doubt bork selective parts of the system when feisty noobs try to keep their internet exposed VPSs relatively up to date on patches.

Audacious things like apt-get update and a few other lines recommended by regulars around this forum. Really really obscure and daring commands like:

apt update && apt upgrade
fwconsole ma updateall
fwconsole reload
then do a server reboot

or truly crazy ones like this:

fwconsole ma upgradeall
rm -rf /tmp/*
fwconsole reload
/root/sig-fix
/root/sig-fix

I know, I know. Truly ballsy and avant-garde stuff here but I lead a daring life.

I was led to believe that these are among the right audacious ways to update a system but clearly I've been led astray. /s



Also, if you have any input on where to look and/or what to compare between the two systems while I have them, that too would be appreciated.

 

[wardmundy]

@JFrost apt upgrade can make major changes in the underlying operating system which can and probably will impact the operation and interaction of Asterisk, FreePBX, MySQL, and Apache among other components. Sorry, but this is not new news. It goes back at least 15 years.

Bottom Line: Upgrading FreePBX components is usually a good idea. Upgrading OS components with apt upgrade absent a security issue, probably not.

 

[JFrost]

@JFrost apt upgrade can make major changes in the underlying operating system which can and probably will impact the operation and interaction of Asterisk, FreePBX, MySQL, and Apache among other components. Sorry, but this is not new news. It goes back at least 15 years.

Bottom Line: Upgrading FreePBX components is usually a good idea. Upgrading OS components with apt upgrade absent a security issue, probably not.

Well, I've asked on multiple occasions for how process/procedure on how to properly update. This is the wisdom shared by the forum (I'd have to go back to be certain but I believe you may have contributed some of that.) When it was given it was not given with caveats.

Perhaps you can clarify once and for all how to PROPERLY update and maintain an entire IPBX box please.

And back to the current frame, I set up the two boxes at your suggestion so while I have them both, any input on what and where to look to find key differences?

J.

 

[wardmundy]

@JFrost: Since you still have the old (working) version, my recommendation would be to use that platform. Do NOT run apt upgrade on it unless there is a security issue discussed on this forum with tips on how to proceed.

As far as finding the culprit for your outage on the new server, that is next to impossible without knowing which components were changed (which you don't know). Did you try running: fwconsole chown

As for updating the old system in the future, my recommendation is to follow the update instructions accompanying all of the current Nerd Vittles tutorials:

Keeping FreePBX 16 Modules Current​

We strongly recommend that you periodically update all of your FreePBX modules to eliminate bugs and to reduce security vulnerabilities. From the Linux CLI, log into your server as root and issue the following commands:

rm -f /tmp/*
fwconsole ma upgradeall
fwconsole reload
/root/sig-fix
systemctl restart apache2
/root/sig-fix

 

[ostridge]

I'm trying to find this "config.php" file. Not sure where the IPBX web files are located (not overly fluent in Linux) but I'm trying to hunt them down.

# locate file : find [directory e.g. / ] -name [filename]
find /var/www/html/admin/ -name config.php

# check permissions
ls -l /var/www/html/admin/config.php

# edit the file
nano  /var/www/html/admin/config.php
#  to save edited file - do 'Ctrl x'   (without quotes)


# for help with 'nano'  
nano --help

 

[JFrost]

@JFrost: Since you still have the old (working) version, my recommendation would be to use that platform.

You've got it backwards, the older has a problem (the PBX works but no the UI). The newer system is freshly configured as a "model" to diff against and eventually probably I'll have to migrate to it (or another new install)

Do NOT run apt upgrade on it unless there is a security issue discussed on this forum with tips on how to proceed.

If this is really that dangerous I feel like this should be said more prominently in more places. There are those in this forum that have recommended it as part of general upgrade procedure.

Also, I have to question an internet facing/connected server that cannot have the OS updated regularly. This seems highly sus to me.

As far as finding the culprit for your outage on the new server, that is next to impossible without knowing which components were changed (which you don't know).

That *is* the purpose of the 'new' clean build. To have what to diff against.

As for updating the old system in the future, my recommendation is to follow the update instructions accompanying all of the current Nerd Vittles tutorials:

Keeping FreePBX 16 Modules Current​

We strongly recommend that you periodically update all of your FreePBX modules to eliminate bugs and to reduce security vulnerabilities. From the Linux CLI, log into your server as root and issue the following commands:

rm -f /tmp/*
fwconsole ma upgradeall
fwconsole reload
/root/sig-fix
systemctl restart apache2
/root/sig-fix

I will update my runbook

 

[wardmundy]

@JFrost: Did you try running: fwconsole chown

If you had run the command we recommended above, it would have told you what the problem was...
Just ran apt upgrade on one of our old Debian 10 servers and the executable bit for fwconsole got clobbered.

This fixed it:

chmod +x /var/lib/asterisk/bin/fwconsole
fwconsole restart
fwconsole chown

Probably worth noting a couple other things...

1. You're running a release that is now 3+ years old.
2. If you are running the software as intended, your server is NOT exposed to the Internet except whitelisted IP's.
3. MAKE BACKUPS and you won't be left with no options in the future.
4. In the future, if you want help, I'd suggest you work on your condescending tone. Doesn't encourage others to help.

 

[lowen]

So, I'm coming into this as a seasoned Linux systems administrator, and I'd like to suggest that a proactive approach be taken here.

Any system with any ports exposed to the Internet at large NEEDS to at least keep its kernel and libc updated, as well as general-purpose things that interact with packets from/to the internet, such as iptables/nftables itself. Debian and other dpkg-based systems have a very powerful pinning mechanism; I would suggest that an apt config with packages where there are known interactions be pinned/held back so that the best practice of periodic security updates remains in place. Perhaps only allowing security updates through. But leaving the kernel unupdated isn't a good idea if any ports are exposed to the Internet. EDIT: And if you're relying on the iptables rules to keep traffic allowed to only certain IP addresses, you're trusting the kernel and the iptables stack to perform that isolation. Security issues in the kernel/iptables stack CAN be triggered even from hosts not in the allowed IP list. That's partly why I have two routers with ACLs in front of the iPBX 2027D instance I'm running, but that's just one more layer of defense in depth, and is not guaranteed to be 100% secure (nothing is ever 100% secure, although the public-facing configurations of IPBX are pretty good).

Ultimately it is up to each individual administrator to decide the update/upgrade strategy for on-premise systems. The IncrediblePBX system (running 2027D here) should not be an exception, it should simply have the setup well documented and well-planned.

 

[wardmundy]

We're always eager to review contributions that help to maintain the security of Incredible PBX. As a systems administrator, your contribution would be invaluable. However, debugging upgrade methodologies to identify and eliminate breakage is a painful process that we simply lack the resources to perform. Where kernel or other updates are introduced to patch security flaws, we always try to push those out through the Automatic Update Utility in Incredible PBX, and we also alert users on this forum, on Twitter, and through the RSS Feeds displayed on the Incredible PBX Dashboard in the FreePBX GUI.

P.S. Our track record over the past 15 years has been pretty good... not a single reported compromised server.

 

[lowen]

We're always eager to review contributions that help to maintain the security of Incredible PBX. As a systems administrator, your contribution would be invaluable. However, debugging upgrade methodologies to identify and eliminate breakage is a painful process that we simply lack the resources to perform. Where kernel or other updates are introduced to patch security flaws, we always try to push those out through the Automatic Update Utility in Incredible PBX, and we also alert users on this forum, on Twitter, and through the RSS Feeds displayed on the Incredible PBX Dashboard in the FreePBX GUI.

Don't get me wrong, I understand and appreciate your efforts. And I'll spin up a testing VM to try to narrow down what caused the situation, and I plan to document what I find here on the forums. But, future installs that start on a Debian 11.9 base could potentially have the same breakage; if they don't that's actually good information, as it means refreshing the whole build stack (something I have to do with another piece of from-source software I use on a daily basis) might correct the problem. But if it's a versioning issue with a dependency, then it could hit fresh installs built on 11.9 just like installs that have been updated. But I always reserve the right to be wrong. And documenting what Debian 11 release is the officially supported starting point would be useful.

P.S. Our track record over the past 15 years has been pretty good... not a single reported compromised server.

If the track record hadn't been what it is we wouldn't have migrated from 3CX to 2027D in the first place. Finally got to a place where....well, I should probably stay quiet about the nonfinancial main reasons, but the bottom line was that with 3CX there was a forced change to subscription-only, and then there was a lack of funding to renew at $dayjob, which is a nonprofit, and there is or at least was at the time a complete absence of a mechanism to reduce the simultaneous call count to save some money. If it can't be afforded, it just can't be afforded. Would have been willing to consider a renewal of 3CX for a reduction in SC count.

 

[wardmundy]

Don't get me wrong, I understand and appreciate your efforts. And I'll spin up a testing VM to try to narrow down what caused the situation, and I plan to document what I find here on the forums. But, future installs that start on a Debian 11.9 base could potentially have the same breakage; if they don't that's actually good information, as it means refreshing the whole build stack (something I have to do with another piece of from-source software I use on a daily basis) might correct the problem.

We attempt to keep the Incredible PBX 2027 installers for Debian 11 and Ubuntu 22.04 compatible with current releases of their products. In the event there is a future hiccup, we'll work on it while offering ISOs of working versions of Debian 11 and Ubuntu 22.04 in the Incredible PBX Repository. And, of course, all of the Incredible PBX 2027 VM images are reliable and are also available in the repo for download. In the words of Carly Simon, "Nobody Does It Better."

 

[lowen]

Ok, so I started with a fresh Debian 11 install, using the debian 11.6 netinstall ISO provided by nerdvittles and ran through the installation.

Even the 11.6 netinstall ISO installs from the current Debian repos, and so I ended up with a 11.9 install, fully updated (note that I didn't manually run an apt update && apt full-upgrade, but right out of the installation the system is fully updated).

After waiting through the 2027D install script a while, the installation of UCP fails in the same way and at the same place as the system that has been updated with 'apt update && apt full-upgrade'

I was going to copy-paste the log, but by the time I started posting here the terminal session's scrollback had overflowed; I'll grab it later in the week. So as of today a fresh 2027D install is broken at building and installing UCP.

 

[wardmundy]

Ok, so I started with a fresh Debian 11 install, using the debian 11.6 netinstall ISO provided by nerdvittles and ran through the installation.

Even the 11.6 netinstall ISO installs from the current Debian repos, and so I ended up with a 11.9 install, fully updated (note that I didn't manually run an apt update && apt full-upgrade, but right out of the installation the system is fully updated).

After waiting through the 2027D install script a while, the installation of UCP fails in the same way and at the same place as the system that has been updated with 'apt update && apt full-upgrade'

I was going to copy-paste the log, but by the time I started posting here the terminal session's scrollback had overflowed; I'll grab it later in the week. So as of today a fresh 2027D install is broken at building and installing UCP.

Have you applied this UCP Bug Fix? Please post a sanitized install log which you will find in the /root folder.

 

[lowen]

Consolidating my UCP thread to https://www.voip-info.org/forum/thr...h-recently-updated-incrediblepbx-2027d.27577/