TIPS Hosting Performance Issues

[atsak]

Historically I've built Incredible PBX (2025) on Digital ocean droplets. I noticed in some situations there's a race condition and asterisk stops answering calls when the node gets a little busy that it's hosted on and it can't get enough CPU resources. I've moved this to dedicated CPU instances which seems to have fixed it, but it's expensive around $100 a month or something. It's mostly a problem in Toronto and NYC it seems.

In practice, is anyone out there running a slightly busier PBX (ie maybe a hundred or more calls a day) that they're sure isn't refusing calls (carriers tell me there's just no answer to the invite, but it's not a firewall issue, I can see the traffic pass to asterisk but it doesn't answer) and uses a cloud provider? Which ones?

Thanks.

 

[wardmundy]

Never had or heard of a problem with CrownCloud.

 

[atsak]

Never had or heard of a problem with CrownCloud.

Thanks - for this application Atlanta might work , also checking something in Canada maybe Montreal or Toronto.

 

[wardmundy]

Thanks - for this application Atlanta might work , also checking something in Canada maybe Montreal or Toronto.

Atlanta is an awesome site that is right beside the internet backbone. It also happened to be a block from my old office. Let me know if you have any issues. They are great guys to work with. They even host a couple of our older ISOs.

 

[Halea]

You might want to consider a VPS at a data center in Buffalo. Most data centers over there are relatively recent with equally recent hardware and not heavily saturated. Plus it would probably give you better ping times.
That said my approach is to double up the CPU and RAM requirements without going for dedicated CPUs. The extra resource allows to go over the bumps when the actual hardware is too stressed.
Also, are you sure that your PBX is not doing some media conversion ?

 

[atsak]

You might want to consider a VPS at a data center in Buffalo. Most data centers over there are relatively recent with equally recent hardware and not heavily saturated. Plus it would probably give you better ping times.
That said my approach is to double up the CPU and RAM requirements without going for dedicated CPUs. The extra resource allows to go over the bumps when the actual hardware is too stressed.
Also, are you sure that your PBX is not doing some media conversion ?

Yep, looking in Buffalo as well - there's a couple options there. The latency to Atlanta though is sub 50ms so that's fine, I usually have a jitter buffer on for 200ms or so and that's always been OK in terms of voice clarity; the issue is simply not servicing calls as you know.

It's virtually no CPU utilization when the issue exists - I've watched in real time. CPU wait time (or whatever it is called in Linux world) increases rapidly but the actual CPU utilization sits around 1%. I have always used 2vCPU and at least 2GB ram so OK there, since Asterisk is single threaded you can only use 1 but the other takes care of system processes (of which voipbl reloads are intensive).

 

[kenn10]

Keep in mind, looking at CPU utilization on the VM is deceiving if looking at it inside the VM. The host box CPU could be maxed out and your VM could show low activity.

 

[Halea]

... since Asterisk is single threaded you can only use 1 but the other takes care of system processes (of which voipbl reloads are intensive).

Actually even Asterisk benefits from multiple vCPUs as QEMU which is at the foundation of the KVM hypervisor breaks the client thread apart and assigns sub-threads to it. Generally a good portion of these sub-threads can run concurrently.
But based on your description of the issue the VPS seems to be struggling with I/O issues, probably at the storage level, which eventually throw some unhandled or ill-timed exceptions and killing the calls. I never saw this happening on asterisk but I did see it on other legacy software designs (with I/O timed through software loops rather than hardware interrupts).

 

[atsak]

Actually even Asterisk benefits from multiple vCPUs as QEMU which is at the foundation of the KVM hypervisor breaks the client thread apart and assigns sub-threads to it. Generally a good portion of these sub-threads can run concurrently.
But based on your description of the issue the VPS seems to be struggling with I/O issues, probably at the storage level, which eventually throw some unhandled or ill-timed exceptions and killing the calls. I never saw this happening on asterisk but I did see it on other legacy software designs (with I/O timed through software loops rather than hardware interrupts).

ya I don't see any IO wait on the server - just CPU wait. The drives are SSD on the VPS I'm using. It's quite intermittent though - it might happen just a few times over a week and then not again for a few months.

 

[chris_c_]

@atsak Install the sysstat package (sudo apt install sysstat or sudo dnf install sysstat) and run:

mpstat -P ALL 1 30

This tracks %steal across all vCPUs every second for 30 seconds.
Steal time is the percentage of time your virtual CPU (vCPU) wanted to run but had to wait for the physical CPU which was busy serving other VPS customers on the physical server. If this number regularly exceeds 5%, your physical host is oversubscribed, get a different server.

 

[atsak]

@atsak Install the sysstat package (sudo apt install sysstat or sudo dnf install sysstat) and run:

mpstat -P ALL 1 30

This tracks %steal across all vCPUs every second for 30 seconds.
Steal time is the percentage of time your virtual CPU (vCPU) wanted to run but had to wait for the physical CPU which was busy serving other VPS customers on the physical server. If this number regularly exceeds 5%, your physical host is oversubscribed, get a different server.

Thanks - this is actually what I did and noticed it doesn't regularly exceed 5%, but when the race condition / no answer happens, that number spikes. I frankly forgot when I made the post how I had done the troubleshooting so didn't include the detail, but I'm glad you did because this will be super helpful in the future to others.

 

[chris_c_]

You can also test IO for subpar speed due to the physical server having too much activity from other customers on the same machine.
We use a command from the same package:
sudo apt install sysstat
Run the command:

iostat -x 1 5

Look at the %util and await columns.
If %util is near 100% and await times are more than 10 ms for SSDs (or 50 ms for HDDs), your disk is waiting in a shared queue, which points to a heavy I/O from other users. Good reason to find a new host.

Check for heavy network traffic from other users.

sudo apt install mtr
mtr -c 100 -w 1.1.1.1

Loss %: Consistent packet loss at the first few hops (your host's gateway router) shows consistent network saturation.
StDev (Standard Deviation): High deviation (e.g., pings jumping from 2ms to 120ms) proves high network jitter.
Both of these can impede Asterisk's ability to respond quickly to incoming calls.

Check for dropped packets.

ip -s link show eth0

Replace eth0 with your network interface name that Asterisk is using.
Look at the drop counters. If the RX drop counter is increasing while your personal traffic is low, the host hypervisor's network ring buffer is overflowing due to physical node traffic from other users.

sudo ethtool -S eth0

rx_speed_drop or rx_missed_errors increment when the host hypervisor drops a packet before your VM even gets it.
rx_queue_0_drops: If you see multi-queue drops increasing, your local virtual CPU cores aren't pulling packets out of the ring buffer fast enough because the physical CPU is busy with other tenants.

If the network card is fine but the OS network stack is overwhelmed by neighbor traffic spikes, packets will drop at the socket buffer layer. Check this with:

ss -s

If you see high numbers or want to see UDP socket drops specifically (common during heavy traffic), run

netstat -su

and look for "packet receive errors" or "receive buffer errors".

Any of these could delay Asterisk's ability to get SIP INVITE messages for incoming calls immediately, process them, and respond right away.


Test your Max Throughput

sudo apt install iperf3

Find a public iperf3 server near your location and run:

iperf3 -c [public_iperf_server] -R

For North and South America it's at Hurricane Electric:

iperf3 -c iperf.he.net -R

The -R flag (for receive) tests your download speed.
If you have a 1 Gbps port and your VM's struggling to maintain 50 Mbps, and your CPU/RAM usage is idle, the physical node's network port is likely getting saturated by other VMs on the same node. Good reason to find a new node, or find a new host.