RECOMMENDATIONS PIAF: Open channels overload

[drdrew]

Hi everyone,

I have recently run into an issue at one of my company's larger customers - before I explain it, let me give you a summary of the system in question:

Hardware:

HP Proliant DL380 G4:
Intel Xeon 3.2GHz (2x)
2GB PC2-3200 DDR2
3x 73GB SAS HDD (RAID 5)


Software:


PIAF-Green 2.0.6.4
FreePBX Version: 2.11.0.11
Asterisk Version: 11.5.1
OS: CentOS 6.4 Final
Kernel Version: 2.6.32-358.23.2.el6.i686 (32 bit)



System:

Number of active extensions: 70
Avg. concurrent calls: 30-50

Using SIP
Call recordings are enabled (wav49 format, stored on the same drive as the system)
Audio codec: g729

The issue I'm having is that at some point during the day, something, somewhere in the system snaps, and Asterisk stops closing the open channels when calls are completed, causing these channels to stack up until they reach ~1000, at which point the CPU usage on the system is at 100%, the system falls over and is unable to open any more channels until I do a forced amportal restart, or reboot the server (if the amportal doesn't work). When under load (before falling over) with 30-50 concurrent calls, CPU usage sits at ~50%.

The customer mentioned that their previous VoIP provider (also using Asterisk) had encountered a similar issue, and after performing diagnostics, determined that the storage of the call recordings was being bottlenecked by the hard drives (too many IOPS), and due to the recordings not being able to finish, Asterisk was not closing the channels created for the recordings. They solved this issue by setting up a separate server to handle the call recordings (linking the two using an IAX trunk, I'm not too certain of the details).

However, I would prefer to attempt to resolve this issue by making changes on the current system before resorting to the method above, and so far have come up with the following changes which I'm going to implement and test tomorrow:

- Disable ringback (generated by our provider, and is causing the recordings to start before the actual call starts (as they are set to record on answer, but the system sees the generated ringback as an answer))
- Enable SIP reinvites (to remove some of the load from Asterisk although I'm not certain how much of a difference this will make)
- Try recording in standard uncompressed 16-bit WAV and not wav49 (wav49 compression could be generating more overhead as well, although this will make storage space an issue)

Do you think that the above changes will lessen the load on the system enough to solve the issue, and do you have any other recommendations as to changes I could make?

One other issue that caught my attention, which I've seen at one of our other customers, is a warning generated by Asterisk, which appears at a rate of several a second when logged into the Asterisk CLI in verbose mode:

WARNING[xxxx]: translate.c:xxx framein: no samples for g729tolin

After researching the issue, I discovered that it was as a result of the dialler that the customer was using, trying to send audio using lin, but Asterisk refusing it (as our systems only use g729) and forcing it to retransmit - what I'd like to know is whether this process could be generating a significant amount of overhead or not (as it doesn't seem to be causing any errors).

Thanks in advance! I've given all the information that I think is relevant to the issue, but if I've missed something, please let me know and I'll be happy to provide it.

 

[phoneguy]

What kind of dialer and how are they sending the calls to Asterisk. Secondly are you sure Asterisk is not transcoding the G729 calls. If it is good luck with 50 concurrent calls being software transcoded. I would never try and transcode with software more then 10-15 calls at the most. Also for the amount of calls with Call recording you need alot more memory. I suggest 8GB of memory if not more.

Asterisk Call Recording is not efficient and once again we never recommend trying to record more then 15-20 calls at a time.

 

[drdrew]

Hi Tony,

Thank you for the speedy response. The dialler they are using is called Conaito - as to how it sends the calls to Asterisk, as far as I can tell, the software package the dialler is part of includes a softphone, which registers to the system, through which the calls are then made.

How can I check whether Asterisk is doing the transcoding? And if this is the case, what options do I have for doing hardware transcoding? In the meantime I will look into getting the memory on that server upgraded.

Lastly, are there any alternatives to Asterisk Call Recording which are more efficient?

Thanks again, your feedback is greatly appreciated!

 

[drdrew]

We have been doing an average of 50 concurrent calls with recording in wav49 on here for quite a while. It only starts giving issues when they launch an outbound call campaign which then pushes concurrent calls to over a 100. What we are going to do now is as follows:
1) put in a new, more powerful server with
a) 8gb of RAM
b) 2 x quad core CPU
c) Two separate drive arrays - one for the OS and asterisk and one for the recordings
2) Disable jitter buffer
3) force recordings to only start when a call is answered (struggling with this as our provider generates a ringback that tricks our PBX into thinking the call was answered. This means that regardless of what I set the outbound call recording option to (record when answered or immediately) it records all ringing).

3) I'm also considering:
a) Using a seperate PBX just for recordings
b) Changing the recordings to wav(normal uncompressed wav) although this uses 10 times more disk space which is a problem as this company makes 20,000 calls per day....

Please add any further suggestions to help reduce resource usage!