Performance keeps degrades over days...

[Lip]

Calling all Experts...

..or maybe even a novice who can point out the likely obvious mistake I'm making!

It's my first posting asking for help. I've tried, searched, read, and attempted many things, but there's a lot of (distracting) info out there! I keep editing this, and now think I should break it in multiple smaller posts...and ask for help more often vs ...

Symptoms
Seems like routing issues - audio poor(crackling, chopping) moreso for inbound side versus internal participant, but neither is perfect, but performance varies from near acceptable degrading to loss of connectivity, typically over days. I reboot every few days to help this.

[1] Testing Questions
a) Is there a way to pinpoint the problem (component)?
b) Is there then a way to confirm performance (won't degrade)?

[2] Setup Info & Questions
Any advice is appreciated! Provider
Voip.ms - seems decent but issues mainly from our connectivity, their internal extra free pbx features, or their value lines, otherwise decent.
Redundancy - ordering something.. Vitelity, didforsale..
Internet
10Mb Cable - solid, tests well, static IPs
Redundancy - ordered DSL
Router
Zyxel ZyWall 2WG - SUSPECT?
- ALG(vs QOS)
- rebooted on it's own a few times
- Opened ports ala NV remote extensions article
- Seeing IPs trying TCP 5025 to the pbx?
Zeroshell router
- likely replacement, installed and configuring now(it's tricky!), failover to DSL etc
Switch/LAN
DLink DGS1248 - solid, low traffic.
Phones
Grandstream gxv3140 x 10 - they're ok, seem pretty solid, but will swap if necessary
Softphone and iphones - work for remote static range (as per NV remotes article), but Travlin was working and stopped doing it iptables/permit magic?! (need to try it again)
Hardware
Travla C134 - SUSPECT? EPIA-M (VIA C3, VIA CLE266 + VIA VT8235M, DDR266 1GB) HDD 2.5" 60GB www.travla.com/product_d.php?id=0000000010
Suitable for 10-15 user SOHO? I have a bunch on hand but will get the recommended hardware if required?
Software
Status Version 1.2.9 released on Date 042310
**********************************************
* PBX in a Flash Version Daemon Status *
* Running Asterisk 1.4 *
**********************************************
* Asterisk * ONLINE * Dahdi * ONLINE * MySQL * ONLINE *
* SSH * ONLINE * Apache * ONLINE * Iptables * ONLINE *
* Fail2ban * ONLINE * IP Connect* ONLINE * Ip6tables * ONLINE *
* BlueTooth * ONLINE * Hidd * ONLINE * NTPD * ONLINE *
* Sendmail * ONLINE * Samba * OFFLINE * Webmin * LOADING *
* Ethernet0 * ONLINE * Ethernet1 * N/A * Wlan0 * N/A *
**********************************************
* Running Asterisk Version : Asterisk 1.4.33.1
* Asterisk Source Version : 1.4.33.1
* Dahdi Source Version : 2.3.0.1+2.3.0
* Libpri Source Version : 1.4.11.2
* Addons Source Version : 1.4.11
**********************************************
pbx.local on 192.168.1.8 - eth0
CentOS release 5.5 (Final) :32 Bit Kernel: 2.6.18-194.8.1.el5
**********************************************
-bash: statustofile: command not found

Upgrading
I've run update-source, update-scripts, update-fixes many times and I cannot get the system to upgrade beyond this?
It eventually end with this message "I am sorry. One or more of the files did not download from http://downloads.digium.com ... " not sure if that's a problem? I ran the incrediblepbx script when it came out. I shouldn't run it again after running update-sources should I?

SIP Settings
Many edits, but I think all config files all back to default and settings all in FreePBX>Asterisk SIP Settings>
NAT:yes IP:static ExtIP:fromprovider Local:192.168.1.0
Codecs: ulaw, alaw, gsm t38:no video:disabled
Reinvite: no RTP: 30, 300, 15
MWI:10 Ring:yes Hold:yes
Reg:20, 0 Times:60,3600,120
Jitter:disabled
Language, default context, bind addr, port, = blank
Allow sip guest:yes
SRV:disabled
Other: blank

FAX
With the frequent poor quality faxes fail, but they seem to work ok when line is clear. Maybe the settings above could be better for faxing (system for inbound, linksys PAP2T+ standard fax machine for outbound).

Trunk
Voip.ms says this is the ideal trunk context, just wanted to confirm with you guys?:
Peer:
canreinvite=nonat
nat=yes
context=from-trunk
host=VoipmsServerIP
username=VoipmsSubacct
secret=mypass
type=friend
disallow=all
allow=ulaw
fromuser=VoipmsSubacct
trustrpid=yes
sendrpid=yes
insecure=port,invite

User:blank

In advance, THANK YOU SO MUCH for anyone who helps me, I am dying to find the problem(s)!

 

[jroper]

Hi

Can you post the results of:-

cat /proc/net/sockstat

Joe

 

[Lip]

cat /proc/net/sockstat
sockets: used 145
TCP: inuse 14 orphan 0 tw 0 alloc 22 mem 1
UDP: inuse 17 mem 0
RAW: inuse 0
FRAG: inuse 0 memory 0

*If relevant, the system is not acting up right now because it was recently restarted. For example tested the fax and works fine right now because call quality is acceptable. It does seem like some (memory)build up occurs in the system or router which causes degradation.

 

[blanchae]

I've run update-source, update-scripts, update-fixes many times and I cannot get the system to upgrade beyond this?

I don't know why you would update-source. This seems to be causing a lot of problems from what I've read lately. I'm not one of the developers.... but I'm thinking that it might be better to remove that script or put a BLATANTLY LARGE message that asks if the user is ABSOLUTELY sure they want to continue to do that.

 

[Lip]

I'm pretty new to PIAF (around 'Orgasamtron'), and I read UW warnings about update-source. BUT, this weeks article, PIAF 1775 seemed to change that. UW replied in the article comments to run these for a complete(ly safe) upgrade! No such luck for me, but I'm still holding out that there is a simple solution.
I would like to try *1.6 and FPBX 2.6 or 2.7, etc. see if it helps my issues/learning... anyway I only tried the update last night in desperation but I've had this issue since the beginning, stupid me tried to read and fix it on my own without simply asking for help!

 

[jroper]

Hi

I've seen a number of Asterisk 1.6 servers suffer from lack of UDP ports recently, an upgrade seems to cure it.

Basically, the system runs out of available ports, the processor works overtime to find a clear port for RTP, and affects the call quality.

a reboot brings the UDP ports down to manageable levels, but over the course of the next few hours, days, weeks, the open UDP ports stay open, and quality drops, then asterisk fails to make any SIP calls.

Joe

 

[Lip]

I think I would prefer if I were running 1.6 but update-source seems to keep me at 1.4. But that sounds like the type of thing I thought this could be, reminds me of what happens to a lot of routers when running torrent without proper settings eventually all those sessions start to bring the hardware down. So are you suggesting I run that command periodically monitoring the results to compare to times of poor performance?

 

[wardmundy]

With 1.7.5.5 there is no reason (yet) to run update-source since everything is up to date. And it can't be used to switch from Asterisk 1.4 to 1.6.2. Sorry for any confusion.

 

[Lip]

At the bottom of the NV PIAF 1.7.5.5 article, someone asked/complained about the ISO procedure to upgrade to 1.7.5.5. Maybe I misunderstood, your replied something like, 'No, just run update..'

The PIAF box I need help with was ISO installed following the orgasmatron article, then upgraded(script/fix) for the incrediblepbx article, now upgraded(source, script, fix -maybe a mistake?) for the PIAF1755 article.

Maybe I should reinstall from the latest ISO?

I'm having a few issues, but the main one, is irrespective of updates it was since the beginning. I think I either need to replace the router or maybe the PIAF hardware, or am missing something obvious in settings like one simple reinvite setting or something tiny and sip/routing related.

Just need some guidance here and any words of advice or encouragement are greatly appreciated and likely helpful.

Cheers.

 

[darmock]

Based on your status output you are not running 1.7.5.5 you are running an earlier unknown version of PIAF. Start by downloading the current ISO of PIAF and then reinstall using it and choose asterisk 1.4 or 1.6 for your type of install gold silver or bronze.

NEVER RUN UPDATE SOURCE UNLESS THERE IS A REAL REASON. JUST BECAUSE IT IS THERE DOES NOT MEAN YOU SHOULD RUN IT EVER.

Of course we firmly believe people have the right to go to hell in their own way so this is why the upgrade-source program is included. Removing it from circulation is not an option we would consider. As for reliability it contains the very same steps as are used by install-pbx which installs PIAF initially. We have used it in its current version for several hundred times with no problems in the lab or in some of our commercial installs.

Upgrading to the very latest bleeding edge versions of asterisk/freepbx is YOUR responsibility and NOT ours. Will running bleeding edge asterisk/freepbx cause problems? Yes it will! YOU need to be prepared to debug this when it happens or DO NOT USE UPDATE-SOURCE.

Does this clarify things?

Also there is NO UPGRADE PATH BETWEEN 1.4 and 1.6. This means if you start out with Asterisk 1.4 no matter how many times you run update-source you will still only get asterisk 1.4. Nor can you downgrade from an Asterisk 1.6 back to an Asterisk 1.4 version.

Is this is clear enough?

If you want PBX in a Flash 1.7.5.5 then you have to install it from scratch using the new ISO.

Is the base software better than older versions of PIAF nope it is exactly the same. If you have a PIAF install that works then use it and don't bother trying the new version.

The major differences in PIAF 1755 is in the behind the scenes install and extra programs which have tighter integration and better redundancy.


Tom

 

[Lip]

My concern is unstable (audio/sip/routing) performance, not the updating. I am running the NV recommended software, nothing else. Please look at my original post (I edited it), I would love your input!

 

[jroper]

Hi

So are you suggesting I run that command periodically monitoring the results to compare to times of poor performance?

Yes, that is my suggestion, I would keep an eye on this, along with such stats as memory and processor load when you experience problems.

Useful commands are "top" and "ps aux", have a look now what they look like when everything is fine, so you get a feel for what it looks like when it's going pear shaped.

Joe

 

[Lip]

Looks normal?

top - 18:59:54 up 2:10, 2 users, load average: 0.00, 0.00, 0.00
Tasks: 88 total, 1 running, 87 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.7%us, 0.3%sy, 0.0%ni, 99.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 1002596k total, 538352k used, 464244k free, 35132k buffers
Swap: 779144k total, 0k used, 779144k free, 325732k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2180 mysql 15 0 135m 18m 3980 S 0.3 1.9 0:30.83 mysqld
1 root 15 0 2152 636 552 S 0.0 0.1 0:01.00 init
2 root RT -5 0 0 0 S 0.0 0.0 0:00.00 migration/0
3 root 39 19 0 0 0 S 0.0 0.0 0:00.00 ksoftirqd/0
4 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
5 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 events/0
6 root 19 -5 0 0 0 S 0.0 0.0 0:00.01 khelper
7 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kthread
10 root 10 -5 0 0 0 S 0.0 0.0 0:00.02 kblockd/0
11 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 kacpid
83 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 cqueue/0
86 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 khubd
88 root 10 -5 0 0 0 S 0.0 0.0 0:00.00 kseriod
145 root 18 0 0 0 0 S 0.0 0.0 0:00.00 khungtaskd
146 root 25 0 0 0 0 S 0.0 0.0 0:00.00 pdflush
147 root 15 0 0 0 0 S 0.0 0.0 0:00.07 pdflush
148 root 20 -5 0 0 0 S 0.0 0.0 0:00.00 kswapd0

 

[Lip]

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 2152 636 ? Ss 16:49 0:01 init [3]
root 2 0.0 0.0 0 0 ? S< 16:49 0:00 [migration/0]
root 3 0.0 0.0 0 0 ? SN 16:49 0:00 [ksoftirqd/0]
root 4 0.0 0.0 0 0 ? S< 16:49 0:00 [watchdog/0]
root 5 0.0 0.0 0 0 ? S< 16:49 0:00 [events/0]
root 6 0.0 0.0 0 0 ? S< 16:49 0:00 [khelper]
root 7 0.0 0.0 0 0 ? S< 16:49 0:00 [kthread]
root 10 0.0 0.0 0 0 ? S< 16:49 0:00 [kblockd/0]
root 11 0.0 0.0 0 0 ? S< 16:49 0:00 [kacpid]
root 83 0.0 0.0 0 0 ? S< 16:49 0:00 [cqueue/0]
root 86 0.0 0.0 0 0 ? S< 16:49 0:00 [khubd]
root 88 0.0 0.0 0 0 ? S< 16:49 0:00 [kseriod]
root 145 0.0 0.0 0 0 ? S 16:49 0:00 [khungtaskd]
root 146 0.0 0.0 0 0 ? S 16:49 0:00 [pdflush]
root 147 0.0 0.0 0 0 ? S 16:49 0:00 [pdflush]
root 148 0.0 0.0 0 0 ? S< 16:49 0:00 [kswapd0]
root 149 0.0 0.0 0 0 ? S< 16:49 0:00 [aio/0]
root 309 0.0 0.0 0 0 ? S< 16:49 0:00 [kpsmoused]
root 330 0.0 0.0 0 0 ? S< 16:49 0:00 [kstriped]
root 339 0.0 0.0 0 0 ? S< 16:49 0:00 [kjournald]
root 364 0.0 0.0 0 0 ? S< 16:49 0:00 [kauditd]
root 397 0.0 0.0 2372 632 ? S<s 16:49 0:00 /sbin/udevd -d
root 1015 0.0 0.0 0 0 ? S< 16:49 0:00 [kgameportd]
root 1162 0.0 0.0 0 0 ? S< 16:50 0:00 [kmpathd/0]
root 1163 0.0 0.0 0 0 ? S< 16:50 0:00 [kmpath_handle]
root 1187 0.0 0.0 0 0 ? S< 16:50 0:00 [kjournald]
root 1619 0.0 0.0 12620 764 ? S<sl 16:50 0:00 auditd
root 1621 0.0 0.0 12156 680 ? S<sl 16:50 0:00 /sbin/audispd
root 1651 0.0 0.0 1808 576 ? Ss 16:50 0:00 syslogd -m 0
root 1654 0.0 0.0 1756 404 ? Ss 16:50 0:00 klogd -x
rpc 1723 0.0 0.0 1904 528 ? Ss 16:50 0:00 portmap
dbus 1747 0.0 0.0 2836 928 ? Ss 16:50 0:00 dbus-daemon --s
root 1760 0.0 0.0 2248 768 ? Ss 16:50 0:00 /usr/sbin/hcid
root 1766 0.0 0.0 1824 512 ? Ss 16:50 0:00 /usr/sbin/sdpd
root 1781 0.0 0.0 0 0 ? S< 16:50 0:00 [krfcommd]
root 1828 0.0 0.1 12944 1304 ? Ssl 16:50 0:00 pcscd
root 1842 0.0 0.0 1752 540 ? Ss 16:50 0:00 /usr/sbin/acpid
68 1960 0.0 0.3 5788 3784 ? Ss 16:50 0:02 hald
root 1961 0.0 0.0 3252 980 ? S 16:50 0:00 hald-runner
68 1968 0.0 0.0 2100 812 ? S 16:50 0:00 hald-addon-acpi
root 1996 0.0 0.0 2000 460 ? Ss 16:50 0:00 /usr/bin/hidd -
root 2023 0.0 0.1 27356 1360 ? Ssl 16:50 0:00 automount
root 2047 0.0 0.1 7180 1068 ? Ss 16:50 0:00 /usr/sbin/sshd
root 2060 0.0 0.2 10228 2304 ? Ss 16:50 0:00 cupsd
root 2077 0.0 0.0 2828 856 ? Ss 16:50 0:00 xinetd -stayali
ntp 2093 0.0 0.4 4496 4496 ? SLs 16:50 0:00 ntpd -u ntp:ntp
root 2130 0.0 0.1 4620 1212 ? S 16:50 0:00 /bin/sh /usr/bi
mysql 2180 0.3 1.9 139072 19396 ? Sl 16:50 0:31 /usr/libexec/my
root 2222 0.0 0.1 9336 1900 ? Ss 16:50 0:00 sendmail: accep
smmsp 2230 0.0 0.1 8248 1520 ? Ss 16:50 0:00 sendmail: Queue
root 2253 0.0 0.8 25980 8760 ? Ss 16:50 0:00 /usr/sbin/httpd
root 2269 0.0 0.0 1920 472 ? S 16:50 0:00 nasd -b -local
root 2282 0.0 0.1 5376 1112 ? Ss 16:50 0:00 crond
asterisk 2304 0.1 1.6 38060 16884 ? S 16:50 0:09 /usr/sbin/httpd
asterisk 2305 0.1 1.6 37744 16512 ? S 16:50 0:09 /usr/sbin/httpd
asterisk 2306 0.0 1.7 38084 17376 ? S 16:50 0:06 /usr/sbin/httpd
asterisk 2307 0.0 1.6 37664 16800 ? S 16:50 0:06 /usr/sbin/httpd
asterisk 2308 0.1 1.6 37864 16660 ? S 16:50 0:08 /usr/sbin/httpd
asterisk 2309 0.0 1.6 37412 16680 ? S 16:50 0:06 /usr/sbin/httpd
asterisk 2310 0.1 1.6 37748 16540 ? S 16:50 0:08 /usr/sbin/httpd
asterisk 2311 0.0 1.6 37832 16628 ? S 16:50 0:07 /usr/sbin/httpd
xfs 2319 0.0 0.1 3664 1284 ? Ss 16:50 0:00 xfs -droppriv -
root 2338 0.1 0.4 113012 4012 ? Sl 16:50 0:13 /usr/bin/python
root 2340 0.0 0.1 2768 1160 ? S 16:50 0:00 /usr/libexec/ga
root 2410 0.0 0.0 2352 432 ? Ss 16:50 0:00 /usr/sbin/atd
avahi 2448 0.0 0.1 2688 1256 ? Ss 16:50 0:00 avahi-daemon: r
avahi 2449 0.0 0.0 2688 312 ? Ss 16:50 0:00 avahi-daemon: c
root 2766 0.0 1.0 26064 10368 ? SN 16:50 0:00 /usr/bin/python
root 2803 0.0 0.0 4620 556 ? S 16:50 0:00 /bin/sh /usr/sb
asterisk 2814 0.2 1.4 35636 14404 ? Sl 16:50 0:18 /usr/sbin/aster
asterisk 2884 0.0 0.0 4620 560 ? S 16:51 0:00 -bash -c cd /va
asterisk 2885 0.0 0.1 4568 1032 ? S 16:51 0:00 sh /var/www/htm
asterisk 2887 0.5 0.8 12652 8740 ? S 16:51 0:39 /usr/bin/perl /
root 2893 0.0 0.0 1604 220 ? S 16:51 0:00 /sbin/tuncfg
root 2912 0.0 0.0 3156 800 ? S 16:51 0:00 /usr/bin/hamach
root 2942 0.0 0.0 3604 512 ? S 16:51 0:00 /usr/sbin/smart
root 2947 0.0 0.8 10888 8524 ? Ss 16:51 0:00 /usr/bin/perl /
root 2950 0.0 0.0 1740 424 tty1 Ss+ 16:51 0:00 /sbin/mingetty
root 2951 0.0 0.0 1740 428 tty2 Ss+ 16:51 0:00 /sbin/mingetty
root 2952 0.0 0.0 1740 424 tty3 Ss+ 16:51 0:00 /sbin/mingetty
root 2953 0.0 0.0 1740 420 tty4 Ss+ 16:51 0:00 /sbin/mingetty
root 2954 0.0 0.0 1740 428 tty5 Ss+ 16:51 0:00 /sbin/mingetty
root 2959 0.0 0.0 1740 428 tty6 Ss+ 16:51 0:00 /sbin/mingetty
root 3015 0.0 0.2 10032 2860 ? Ss 16:53 0:00 sshd: root@pts/
root 3017 0.0 0.1 4624 1412 pts/0 Ss+ 16:54 0:00 -bash
root 3996 0.1 0.2 10032 2872 ? Ss 18:58 0:00 sshd: root@pts/
root 3999 0.0 0.1 4624 1480 pts/1 Ss 18:58 0:00 -bash
root 4243 0.0 0.0 4344 960 pts/1 R+ 19:01 0:00 ps aux

 

[Lip]

Thanks for suggestion. I have been doing that from FPBX and usually looks okay except physical memory used to be high at 97% so upgraded 512MBV to 1GB.

 

[jroper]

Generally, high memory usage in Linux is not an issue - if it isn't being used, it does not seem worth buying it - however looking at processes that are using an unusually high amount of memory can be useful for diagnosis.

Joe

PS - you can attach files on this forum - logs are easier to read as attachments.

 

[blanchae]

At this point, if it is not disruptive, I would suggest re-install but don't update-source as per the point that was subtly made in an earlier response.

This is definitely an interesting problem and should be monitored. But if you are trying to run a PBX then I would leave it up to those with a lab environment to solve. One thing you can do is capture the logs and info that Joe suggested the next time the problem occurs.

I've seen PiaF slowly appear to use up RAM until it is at 97% but most of the memory is cached that can be available when needed..

 

[Lip]

getting there...

Replaced Zywall 2WG with DLink DIR-825.

I need a way to test/verify performance (isn't going to degrade)?

So far seems decent except...

DTMF not registering from external callers, not in IVR, VM, or FAX. Something simple?

I'll still try Zeroshell but this seemed an easier, more reliable test.

Next I'll compare provider quality (Voip.ms to another).

If I still have degrading performance issues, maybe other hardware with fresh software.

If I do a reinstall I'll need to know I can easily/reliably bring over messages etc. Moving from (grrr) Trixbox was a mess - won't get into that..