No audio on inbound -- reboot always fixes

[randomng]

About every two or three months, my audio will stop working for inbound calls. Phones ring, etc., but the caller doesn't get a ring tone and upon answering neither party can hear each other. I can restart asterisk, etc. without effect. If I restart the OS, everything works again for a couple of months. I've switched ethernet interfaces on the machine. I've changed out a switch and a router. I've gone from Trixbox to PIAF. I've switched from cable to fiber (and changed ISPs), but this thing is still nagging at me. I'd like to get it fixed once and for all. Outbound always seems to work fine. My provider has helped troubleshoot a bit and reports that packets get dropped on my side.

Here's some info (I've munged IPs and phone numbers because I'm paranoid). Also, I apologize if this much is overkill, I'm trying to provide as much relevant data as possible.

Thanks for any help!!

Machine: Dell Poweredge 860 (Processor, 915, 2.8, 2X2MB Pentium D Presler, C1); Hard Drive, 80G, S2, 7.2K, 3.5 WD-UNIC x 2 in RAID 1 config.

Status output (ran update-script and fixes tonight):
Status Version 1.2.9 released on Date 042310
********************************************************************
* PBX in a Flash Version Daemon Status *
* Running Asterisk 1.4 *
********************************************************************
* Asterisk * ONLINE * Zaptel * ONLINE * MySQL * ONLINE *
* SSH * ONLINE * Apache * ONLINE * Iptables * ONLINE *
* Fail2ban * ONLINE * IP Connect* ONLINE * Ip6tables * ONLINE *
* BlueTooth * ONLINE * Hidd * ONLINE * NTPD * ONLINE *
* Sendmail * ONLINE * Samba * OFFLINE * Webmin * ONLINE *
* Ethernet0 * ONLINE * Ethernet1 * OFFLINE * Wlan0 * N/A *
********************************************************************
* Running Asterisk Version : Asterisk 1.4.21.2
* Asterisk Source Version : 1.4.21.2
* Zaptel Source Version : 1.4.12.1
* Libpri Source Version : 1.4.9
* Addons Source Version : 1.4.7
********************************************************************
pbx.local on 192.168.0.144 - eth0
CentOS release 5.2 (Final) :32 Bit Kernel: 2.6.18-92.1.22.el5

sip_general_custom.conf contains:
externip=166.70.24.75
localnet=192.168.0.0/255.255.255.0


Log file attached.

 

[blanchae]

I guess the question is what changes after an OS reboot? Or what gets cleared: iptables? log files? mysql?

It sounds like something is becoming full and then stops working. On reboot of the OS, it is cleared. Restarting Asterisk doesn't clear it only rebooting OS. Could it be the swap partition?

 

[randomng]

I've checked MySQL for anything that looked "big" and haven't seen anything there. Physical memory on the box looks fine -- I'll look at the swap partition. I'll check file handles too -- maybe connections aren't being closed all the way?

Any other ideas?

 

[Stewart]

When issues like that start happening to me, my SIP provider usually tells me that my box has sent it's registration using it's private IP address and not the public address and therefore the media stream cant be routed to be and gets lost in their network. Maybe yours is similar? I haven't found an exact workaround for it yet, though.

 

[jroper]

Hi

We had a similar issue on one of our servers, although your error messages look slightly different. Here is the text of an email exchange with a customer:-

I've just run cat /proc/net/sockstat

[root@ca:~]$ cat /proc/net/sockstat
sockets: used 4268
TCP: inuse 11 orphan 0 tw 0 alloc 19 mem 1
UDP: inuse 4131 mem 7
RAW: inuse 0
FRAG: inuse 0 memory 0
[root@ca:~]$

We cured the issue on the other server by doing an Asterisk upgrade to the latest stable, downtime is the time it takes to restart Asterisk.

After 48 hours of a lot of traffic, his system looks like this

sockets: used 227
TCP: inuse 12 orphan 0 tw 0 alloc 17 mem 0
UDP: inuse 24 mem 0
RAW: inuse 0
FRAG: inuse 0 memory 0

See that after some time, the cat /proc/net/sockstat command does not show high numbers of UDP in use.

Joe

 

[randomng]

Found some info -- is it too many open files?

Thanks for the ideas.

I looked at file handles. Each httpd process from asterisk (8 total) is using right around 128 files. Asterisk itself is at 211. Max open file descriptors is 1024 (ulimit -Sn and -Hn) and total system open is 2511 (lsof|wc -l). Is this my problem? I bumped up my limit to 5120 (ulimit -n 5120). Do I need to restart the OS for that to take effect? The problem didn't go away after changing the setting, and I don't want to restart without some confirmation that this could be the issue (tough to troubleshoot when it's working).

I'm checking with my provider now to see if they can see anything wrong with my registration. I'll post what I hear.

Also, I was so hopeful about connections, but that doesn't look like my problem:

sockets: used 123
TCP: inuse 9 orphan 0 tw 0 alloc 17 mem 2
UDP: inuse 12 mem 0
RAW: inuse 0
FRAG: inuse 0 memory 0


Memory/swap also looks good:
Cpu(s): 0.0%us, 0.0%sy, 0.0%ni, 99.3%id, 0.7%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 1035048k total, 667156k used, 367892k free, 189644k buffers
Swap: 2031608k total, 0k used, 2031608k free, 337504k cached

 

[Yahdie]

I've gone from Trixbox to PIAF. I've switched from cable to fiber (and changed ISPs)

I'm not gonna pretend to have the experience of the other responders but.....
At first I thought it might be an Iptables/firewall issue but that line sort of eliminates that, I think the problem is your trunk provider.
The fact that outbound calling works (ie the machine has and maintains its registration) and that you've changed just about every other piece of the puzzle?
Have you tried disabling and re-enabling the trunk when it happens?
And what is this entry in the log:

Received incoming SIP connection from unknown peer to

 

[randomng]

I tried disabling/reenabiling the trunk to no avail. I tried to stop process to get below the 1024 ulimit but I couldn't find a process that would stay down while leaving asterisk running correctly.

I ended up rebooting. Everything works fine now. I guess I'll wait to see if changing the ulimit helped. I'll report back here results. Also, if anyone comes across this and knows definitively, I'd love to know.

Thanks

 

[kmcdaniel]

What is your FreePBX version?

 

[randomng]

Sorry for not responding sooner -- here are version numbers from my modules admin page:

Core: 2.5.2.5
FreePBX Framework: 2.5.2.4

This has continued to happen. It seems to coincide with spending a lot of time using the service (on the phone a lot). It has happened twice in the last week. Reboot always fixes it.

Even if I could stop/start asterisk or something else to fix it, that would be much better -- rebooting the box takes minutes. Getting that down to seconds would be more tolerable.

 

[fmood123]

Router issue?

I have a similar issue and think it is my router stopping the port forwarding possibly when dhcp release expires? Reboot of machine fixes all the time. Haven't tried to reboot router but will as I'm troubleshooting. Not sure, but this is what I am thinking at this point in my case.

Hope it helps

 

[randomng]

Resolution!

In case anyone else experiences something similar - here's what finally resolved this issue.

In my sip_additional.conf, I originally had the following:

[ServiceName]
disallow=alltype=peer
trustrpid=yes
sendrpid=yes
qualify=yes
port=5060

host=***.**.**.***
dtmfmode=rfc2833
context=from-pstn
allow=ulaw

After adding "canreinvite=no" and changing "qualify=yes" to "qualify=no", everything started working correctly. I'm a bit baffled why it would work after a restart for months then start having issues on every call, but admittedly I don't understand the canreinvite functionality. Anyhow, thanks for everyone's help. Hopefully this helps someone in the future.