TIPS Hang and other error messages from update202044

[faixbp]

My RPi applied update202044 today, resulting in error messages other than the "rm: cannot remove", and now it's hung (see below). The ssh connection is still open, It will respond to pings, but I can't ssh in from another terminal and the webserver doesn't work.

I'm leaving it sit at this point pending advice from the forum.

Checking for update202044. NOT FOUND. Installing... update202044


Adding OutCNAM support to FreePBX...
--2022-11-03 07:02:08-- http://pbxossa.org/files/outcnam/outcnam-latest.tgz
Resolving pbxossa.org (pbxossa.org)... 208.113.173.86
Connecting to pbxossa.org (pbxossa.org)|208.113.173.86|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5344 (5.2K) [application/x-gtar-compressed]
Saving to: 'outcnam-latest.tgz'

outcnam-latest.tgz 100%[===================>] 5.22K --.-KB/s in 0s

2022-11-03 07:02:09 (102 MB/s) - 'outcnam-latest.tgz' saved [5344/5344]

outcnam/
outcnam/Outcnam.class.php
outcnam/README.md
outcnam/module.xml
outcnam/package-lock.json
outcnam/package.json
outcnam/page.outcnam.php
outcnam/views/
outcnam/views/main.php
rm: cannot remove '/tmp/systemd-private-224e7c77177b4ca0ae9e064ed2cab41e-apache2.service-0qIfJy': Is a directory
rm: cannot remove '/tmp/systemd-private-224e7c77177b4ca0ae9e064ed2cab41e-systemd-timesyncd.service-DoZqys': Is a directory
Updating tables outcnam_config...Done
Generating CSS...Done
Module outcnam version 0.0.6 successfully installed
Updating Hooks...Done
Chowning directories...
In PKCS.class.php line 511:

mkdir(): Input/output error


moduleadmin [-f|--force] [-d|--debug] [--edge] [--ignorecache] [--stable] [--color] [--skipchown] [-e|--autoenable] [--skipdisabled] [--snapshot SNAPSHOT] [--format FORMAT] [-R|--repo REPO] [-t|--tag TAG] [--skipbreakingcheck] [--sendemail] [--onlystdout] [--] [<args>]...


Whoops\Exception\ErrorException: mkdir(): Input/output error in file /var/www/html/admin/libraries/BMO/PKCS.class.php on line 511
Stack trace:
1. Whoops\Exception\ErrorException->() /var/www/html/admin/libraries/BMO/PKCS.class.php:511
2. mkdir() /var/www/html/admin/libraries/BMO/PKCS.class.php:511
3. FreePBX\PKCS->getKeysLocation() /var/www/html/admin/libraries/BMO/PKCS.class.php:583
4. FreePBX\PKCS->checkPermissions() /var/www/html/admin/libraries/BMO/PKCS.class.php:56
5. FreePBX\PKCS->__destruct() [internal]:0
Reload Started

In GPG.class.php line 537:

GPG took too long to run the command: [/usr/bin/gpg --homedir /home/asteris
k/.gnupg --no-permission-warning --keyserver-options auto-key-retrieve=true
,timeout=10 --no-tty --status-fd 3 --list-keys 0BDE0BFA09946D732091E26E1588
A7366BD35B34]


reload [--json] [--dry-run] [--skip-registry-checks] [--dont-reload-asterisk]

OutCNAM support added to FreePBX

Updates and notifications completed.
To continue, press the Enter key...

dig: couldn't get address for 'ns1.google.com': failure
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:04:59 --:--:-- 0

0 0 0 0 0 0 0 0 --:--:-- 0:35:47 --:--:-- 0
curl: (28) Resolving timed out after 300110 milliseconds

 

[wardmundy]

Your network connection failed:

dig: couldn't get address for 'ns1.google.com': failure

Ctrl-C. Reboot your server and try again.

 

[faixbp]

Your network connection failed:

dig: couldn't get address for 'ns1.google.com': failure

Ctrl-C. Reboot your server and try again.

Thanks Ward.

I rebooted, but apparently the MicroSD filesystem was is now damaged and the boot process can't repair it.

Is there an offline utility repair utility to try?

 

[wardmundy]

That's what backups are for, my friend. MicroSD cards wear out. Always keep a spare copy.

 

[faixbp]

That's what backups are for, my friend. MicroSD cards wear out. Always keep a spare copy.

Absolutely. I also think there may have been something wrong with the RPi or the MicroSD card, because this system was only running for a few weeks.

 

[ostridge]

something wrong with the RPi or the MicroSD card

Is there an offline utility repair utility to try?

Sometimes you can rescue the MSD card

Sometimes the Pi sets the 'Dirty' bit and lets you bugger about scratching your head wondering why you dont see the partition(s)
Its may be worth doing partition checks on another linux computer or from a Linux live CD.

Have you done a repair of the SD card to clear the 'Dirty bit'

# where mmcblk0p1 is your mSD card
 fsck.fat -a -w -v /dev/mmcblk0p1
## then
 e2fsck -f -y -v -C 0 /dev/mmcblk0p2

## Or to fix Dirty bit on usb device

##  -  where x is your usb mass storage device or SSD
fsck.fat -a -w -v /dev/sdx1
## AND
e2fsck -f -y -v -C 0 /dev/sdx2

Then your mSD card or usb-drive might be able to boot; or at least be ready to start again.

Once you get your mSD sorted, or maybe reinstall a new image, If you are able to set up your RPi4 with a USB hard drive - that is greatly preferable to relying on a microSD card.

 

[faixbp]

Sometimes you can rescue the MSD card

Sometimes the Pi sets the 'Dirty' bit and lets you bugger about scratching your head wondering why you dont see the partition(s)
Its may be worth doing partition checks on another linux computer or from a Linux live CD.

Have you done a repair of the SD card to clear the 'Dirty bit'

# where mmcblk0p1 is your mSD card
 fsck.fat -a -w -v /dev/mmcblk0p1
## then
 e2fsck -f -y -v -C 0 /dev/mmcblk0p2

## Or to fix Dirty bit on usb device

##  -  where x is your usb mass storage device or SSD
fsck.fat -a -w -v /dev/sdx1
## AND
e2fsck -f -y -v -C 0 /dev/sdx2

Then your mSD card or usb-drive might be able to boot; or at least be ready to start again.

Once you get your mSD sorted, or maybe reinstall a new image, If you are able to set up your RPi4 with a USB hard drive - that is greatly preferable to relying on a microSD card.

Thanks ostridge,

When I plug the MicroSD into a linux system, it doesn't seem to mount and gives the error messages in dmesg (see attached).

--faixbp

 

[ostridge]

Thanks ostridge,

When I plug the MicroSD into a linux system, it doesn't seem to mount and gives the error messages in dmesg (see attached).

--faixbp

well assuming that sdc: sdc1 sdc2 is the msd card in question - which you should double check ( after unmounting it ) by checking the dmesg log wirthout and with your card.

You may be able to find the partitions with

fdisk -l

and using the partition 1 address in the first check
fsck.fat -a -w -v /dev/sdc1
# and partition2 address in the
e2fsck -f -y -v -C 0 /dev/sdc2

After that maybe you could try mounting and see where you go from there.

 

[faixbp]

well assuming that sdc: sdc1 sdc2 is the msd card in question - which you should double check ( after unmounting it ) by checking the dmesg log wirthout and with your card.

You may be able to find the partitions with

fdisk -l

and using the partition 1 address in the first check
fsck.fat -a -w -v /dev/sdc1
# and partition2 address in the
e2fsck -f -y -v -C 0 /dev/sdc2

After that maybe you could try mounting and see where you go from there.

The MicroSD card is definitely sdc when plugged into my linux box with an adapter, but fdisk -l doesn't see it.

Whatever happened when/after update202044 ran seems to have dramatically impacted the card. Or perhaps the damage had already happened and the update just made is visible.

 

[dicko]

7 Ways to Recover Deleted Files on Linux - Make Tech Easier

File recovery tools are one of the most important parts of a Linux system. Learn how to recover files in Linux with these free tools.

www.maketecheasier.com

testdisk as last resort

 

[ostridge]

The MicroSD card is definitely sdc, but fdisk -l doesn't see it.

Whatever happened when/after update202044 ran seems to have dramatically impacted the card.

so try the 2 fixes one on each partition - to see if you can recover the card. If that works you can try to boot or even to re-use the card. Its unlucky if both partitions are down.

 

[dicko]

fdisk will fail on a damaged partition table which is located in the msdos boot-sector area (MBR) at the start of the disk way before linux is loaded, as that is the bit that needs repair, it's a catch-22 as you can't do it from the sd card itself cos it can't boot, use one or more of the methods described in my link, but next time just make a bloody image of the sd !

 

[faixbp]

fdisk will fail on a damaged partition table which is located in the msdos boot-sector area (MBR) at the start of the disk way before linux is loaded, as that is the bit that needs repair, it's a catch-22 as you can't do it from the sd card itself cos it can't boot, use one or more of the methods described in my link, but next time just make a bloody image of the sd !

Agreed. And given how quickly the SD card became corrupted, I wonder if the RPi or the SD card is defective, because there were definitely no power failures without halting the RPi.

 

[ostridge]

I wonder if the RPi or the SD card is defective, because there were definitely no power failures without halting the RPi.

Well nobody has an answer to that, except that more often than not it's the card. mSD cards are notable for shall we say trouble. The method above is simple and quick, and if that doesn't work it may be easier to use another card (they are cheap).
Once you have the Pi(?) booted and working again you could try hardware testing (on your own).

 

[wardmundy]

Agreed. And given how quickly the SD card became corrupted, I wonder if the RPi or the SD card is defective, because there were definitely no power failures without halting the RPi.

Looking at your original post, it looks like things started coming unglued when FreePBX attempted to chown all of the directories used by FreePBX. That is basically a waltz through large chunks of your microSD card resetting ownership on file system directories. It looks like it encountered a bunch of errors during that process. So, as you surmised, the card was probably already damaged and, when FreePBX attempted to reset ownership permissions on some of the directories, all hell broke loose. Definitely looks like a defective microSD card to me.

 

[faixbp]

Looking at your original post, it looks like things started coming unglued when FreePBX attempted to chown all of the directories used by FreePBX. That is basically a waltz through large chunks of your microSD card resetting ownership on file system directories. It looks like it encountered a bunch of errors during that process. So, as you surmised, the card was probably already damaged and, when FreePBX attempted to reset ownership permissions on some of the directories, all hell broke loose. Definitely looks like a defective microSD card to me.

Thanks Ward for the insight. I'll trash the MicroSD card and repurpose the RPi for something not mission-critical, in case the RPi itself was the cause of the MicroSD card damage.

 

[wardmundy]

Thanks Ward for the insight. I'll trash the MicroSD card and repurpose the RPi for something not mission-critical, in case the RPi itself was the cause of the MicroSD card damage.

I doubt it's your RasPi causing the problem. For better performance and reliability, switch to an external storage device. This Nerd Vittles article will show you how.

 

[faixbp]

I doubt it's your RasPi causing the problem. For better performance and reliability, switch to an external storage device. This Nerd Vittles article will show you how.

Yes, I'll do that; thanks. To protect the MicroSD, I was running this RPi with a LiFePO4wered/Pi+ UPS, so that the RPi would never lose power without an orderly shutdown and poweroff. But apparently that wasn't enough.

 

[dicko]

The extra buck sometimes is worth it

https://www.reddit.com/r/raspberry_pi/comments/ex7dvo

 

[faixbp]

Hi dicko, thanks. Would you consider using a WD Purple wear-leveling microSD card in an RPi to be in the same reliability ballpark as using an SSD a la Ward's posting? Or if not, at least good enough? I realize that this is an impossible question.