The System Just Dies

[esjones]

A couple of times in the past week (most recently, today) my PIAF system has just stopped functioning. CentOS continues to run, and there is a message on the console that Asterisk has exited with status 0. We've been away from the house both times this has happened, and I am at a loss to understand what is going on.

What logs should I be looking at? I'm a newbie, and would appreciate someone taking me under his/her wing and helping me learn.

Thanks.

Earl

 

[blanchae]

All of the logs are located in /var/log with a subdirectory /var/log/asterisk. You would want to look at all the logs just to see what is going on. The main asterisk log is /var/log/asterisk/full and it can be quite large. Also from the linux command prompt, type df to see how much room is available on your partitions.

 

[esjones]

I've been looking at /var/log/asterisk/full and do not see anything that is flagged as an error, or that jumps out as being out of the ordinary.

In last night's incident, the logging just ceased at 17:11:04 and didn't resume until I got home after 9PM and restarted Asterisk.

Is someone willing to take a look at my logs? I've attached an excerpt.

Many thanks!

 

[randy7376]

Looking at your log snippet, I don't think you're experiencing a crash. It looks like someone/something is connecting and shutting Asterisk down:

[2010-10-28 21:06:51] VERBOSE[6552] logger.c: -- Remote UNIX connection
[2010-10-28 21:06:52] VERBOSE[17131] logger.c: -- Remote UNIX connection disconnected
[2010-10-28 21:06:52] VERBOSE[6552] logger.c: -- Remote UNIX connection
[2010-10-28 21:06:52] VERBOSE[17134] logger.c: -- Remote UNIX connection disconnected
[2010-10-28 21:06:58] VERBOSE[6552] logger.c: -- Remote UNIX connection
[2010-10-28 21:06:58] VERBOSE[17167] logger.c: -- Remote UNIX connection disconnected
[2010-10-28 21:06:58] VERBOSE[6552] logger.c: -- Remote UNIX connection
[2010-10-28 21:06:58] VERBOSE[17170] logger.c: Waiting for inactivity to perform halt...

This is why it exits with a status of 0 (normal shutdown).

Your system isn't exposed directly to the Internet, by chance, is it?

 

[esjones]

Thanks, Randy!

Randy -

Thanks for taking a peek... No, my server is NOT exposed to the internet. I don't even have port 5060 open. Steve Gibson's "Shields Up!" external probe shows no ports open to my internal systems at all.

At the point in time you refer to, the server had long since died. That time was about when I got home and started poking around to see what had happened.

 

[MGD4me]

Try this...

Go into the Module Admin section of the FreePBX administration, and check to see if you have 'Java SSH' enabled. If it is, change it to 'disabled'. In my case, I found that with it enabled, the 'FreePBX System Status' page would eventually show the 'CPU' value getting higher, even with an idle system. Perhaps it was because I have a fairly minimal platform with only 512 MB of memory as a test bed. But once the CPU reached 100% (almost daily), the PBX would be brought to it's knees. Restarting Asterisk or re-booting, would restore the PBX for a time, but by the next day it failed again. Disabling 'Java SSH' worked for me. Perhaps more memory would do the trick also.

 

[esjones]

Thank you for the tip. I did have 'Java SSH' enabled, so I disabled it as you suggested. (What, exactly, does that do, anyway?)

I've got 2GB of RAM, so it shouldn't be a memory problem, but we'll see anyway.

 

[wardmundy]

Just an FYI that we run PBX in a Flash with Incredible PBX on virtual machines with 512MB of RAM. Zero problems! If your machine is blowing up with 2GB of RAM, then you've got some other problem... perhaps bad RAM or a faulty motherboard.

 

[randy7376]

I'd sure want to know what that "Remote UNIX connection" is doing before the shut down.

By all accounts, the exit status of 0 is a normal shut down. The time it shuts down is rather precise, too, with respect to the remote connection.

I'd run some kind of packet capture to see what's being sent "remotely" prior to shut down.

 

[blanchae]

The "remote unix connection", I believe refers to FreePBX polling or updating its GUI every 2 or 3 seconds. I've seen that message quite frequently.

 

[jroper]

"[2010-10-28 21:06:58] VERBOSE[17170] logger.c: Waiting for inactivity to perform halt..."

This message indicates that Asterisk has been asked to stop gracefully, that is to say when there are no calls on the box, stop the asterisk server.

Where the message is coming from to do this is anyone's guess. It is the default behaviour of amportal to stop gracefully.

Have you made any modifications to restart asterisk/amportal at anytime?

Joe

 

[esjones]

Ward - Thanks for the info, and for all your contributions to this technology and web sites. It may be worth noting that CentOS did not die... I would think that a memory or motherboard problem would not just take out the application.

Also, is there an intro guide to troublshooting PIAF published anywhere?

 

[blanchae]

Have you looked at ALL the logs in /var/log? That is the place to start.

 

[esjones]

The timestamp on this message tells me that it is me doing a restart of Asterisk after I got home and realized that the system had died.