How to trace call quality issues

[phinphan]

So we have been having lousy calls lately. We think it is our internet service in the office (spectrum). When I do normal speedtests I am getting 115mbps down and 12mbps up and pings of 40 ms or so. Ping times from my office to Rentpbx are around 34ms . there is occasional packet loss of around 2%. I run voip quality tests like megapath which reports DOWNLOAD 114.50 Mbps UPLOAD
11.31Mbps LATENCY 45ms Jitter 3ms. Ping times from rentpbx to vitelity are 51ms with mdev of 0450 ms and 0% packet loss.

I opened a ticket with vitelity and send them most of that information as well as a traceroute and pcap session.

Any ideas on where else to try before I have to argue with Spectrum? Is there a way to continuously monitor the connection between office and rentpbx to check for problems?

As I was typing this, the connection dropped from my office to the internet.

How can i open up pings from my office (only) to my server?

 

[Eliad]

I have Spectrum too. They suck. I think they somehow fake those speed tests.
I have a backup DSL with CenturyLink. They suck too since they offer just 3mbs for the area.
I have an Opnsense firewall with 2 WAN. With this setup I end up with a better overall internet connection and less call issues.
I am curious myself to see how I could monitor the actual performance of Spectrum.

 

[phinphan]

Eliad - we really seem to have limited options. I am thinking of adding a 4glte backup but that will need a new expensive firewall so I am still hashing around different ideas.

On the plus side, I found a network monitoring tool http://www.nirsoft.net/utils/multiple_ping_tool.html which shows me that I cannot ping the server always. I am having about 1.34% failure rate for pings. Dropouts occur pretty often. See below. I have the program set to ping google dns at the same time and it has the same error rate and timing so I think that means its our internet not the server. Now to a discussion with Spectrum.

1/10/2020 11:11:37 AM Request Timeout
1/10/2020 11:11:26 AM Request Timeout
1/10/2020 11:11:15 AM Request Timeout
1/10/2020 11:04:58 AM Request Timeout
1/10/2020 11:04:47 AM Request Timeout
1/10/2020 11:04:36 AM Request Timeout
1/10/2020 10:56:57 AM Request Timeout
1/10/2020 10:56:46 AM Request Timeout
1/10/2020 10:56:35 AM Request Timeout
1/10/2020 10:56:24 AM Request Timeout
1/10/2020 10:56:14 AM Request Timeout
1/10/2020 10:55:42 AM Request Timeout
1/10/2020 10:55:31 AM Request Timeout
1/10/2020 10:55:20 AM Request Timeout
1/10/2020 10:55:10 AM Request Timeout
1/10/2020 10:44:37 AM Request Timeout
1/10/2020 10:44:26 AM Request Timeout
1/10/2020 10:44:15 AM Request Timeout
1/10/2020 10:44:04 AM Request Timeout
1/10/2020 10:43:54 AM Request Timeout
1/10/2020 10:34:40 AM Request Timeout
1/10/2020 10:33:48 AM Request Timeout
1/10/2020 10:15:28 AM Request Timeout
1/10/2020 10:12:13 AM Request Timeout
1/10/2020 10:12:01 AM Request Timeout
1/10/2020 10:11:50 AM Request Timeout
1/10/2020 10:11:40 AM Request Timeout
1/10/2020 10:11:18 AM Request Timeout
1/10/2020 10:11:08 AM Request Timeout
1/10/2020 9:59:25 AM Request Timeout
1/10/2020 9:59:14 AM Request Timeout
1/10/2020 9:58:43 AM Request Timeout
1/10/2020 9:58:32 AM Request Timeout
1/10/2020 9:58:21 AM Request Timeout
1/10/2020 9:54:06 AM Request Timeout
1/10/2020 9:53:45 AM Request Timeout
1/10/2020 9:53:34 AM Request Timeout
1/10/2020 9:53:23 AM Request Timeout
1/10/2020 9:53:12 AM Request Timeout
1/10/2020 9:53:01 AM Request Timeout
1/10/2020 9:52:50 AM Request Timeout
1/10/2020 9:52:39 AM Request Timeout
1/10/2020 9:52:08 AM Request Timeout
1/10/2020 9:51:56 AM Request Timeout
1/10/2020 9:35:49 AM Request Timeout
1/10/2020 9:35:38 AM Request Timeout
1/10/2020 8:43:33 AM Request Timeout
1/10/2020 8:43:22 AM Request Timeout
1/10/2020 8:43:11 AM Request Timeout
1/10/2020 8:33:31 AM Request Timeout
1/10/2020 8:33:19 AM Request Timeout
1/10/2020 8:33:08 AM Request Timeout
1/10/2020 8:32:57 AM Request Timeout
1/10/2020 8:32:46 AM Request Timeout
1/10/2020 8:32:24 AM Request Timeout
1/10/2020 8:32:14 AM Request Timeout
1/10/2020 8:32:02 AM Request Timeout
1/10/2020 8:31:52 AM Request Timeout
1/10/2020 8:31:40 AM Request Timeout
1/10/2020 8:31:29 AM Request Timeout
1/10/2020 8:31:18 AM Request Timeout
1/10/2020 8:31:07 AM Request Timeout
1/10/2020 8:30:57 AM Request Timeout
1/10/2020 8:30:45 AM Request Timeout
1/10/2020 8:30:34 AM Request Timeout
1/10/2020 8:30:23 AM Request Timeout
1/10/2020 8:30:12 AM Request Timeout
1/10/2020 8:30:01 AM Request Timeout
1/10/2020 8:29:50 AM Request Timeout
1/10/2020 8:29:40 AM Request Timeout
1/10/2020 8:29:28 AM Request Timeout
1/10/2020 8:29:17 AM Request Timeout
1/10/2020 8:29:06 AM Request Timeout
1/10/2020 8:28:55 AM Request Timeout
1/10/2020 8:28:45 AM Request Timeout
1/10/2020 8:28:33 AM Request Timeout
1/10/2020 8:15:40 AM Request Timeout
1/10/2020 8:15:28 AM Request Timeout
1/10/2020 8:15:17 AM Request Timeout
1/10/2020 8:15:06 AM Request Timeout
1/10/2020 8:14:25 AM Request Timeout
1/10/2020 8:14:14 AM Request Timeout
1/10/2020 8:14:03 AM Request Timeout
1/10/2020 8:13:52 AM Request Timeout
1/9/2020 5:23:36 PM Request Timeout
1/9/2020 5:23:25 PM Request Timeout
1/9/2020 5:23:14 PM Request Timeout
1/9/2020 5:23:03 PM Request Timeout

 

[phinphan]

Oddly, the timeouts occur during the work day only. Last one at 5:23pm EST and didnt start back up until 813AM this morning. Could the congestion on Spectrum be that bad?

Spectrum no help whatsoever. Customer service script reader says we can talk to your modem so it is your fault and the fault of your equipment and it we come out and check, we will charge you. Funny, during the phone call with Spectrum it dropped out and even though I gave them a call back number, they didnt call back.
Time to try twitter customer service. They usually have smarter people on that end.

 

[billsimon]

Speed tests are good for only one thing: telling you how many calls your connection could support.

In fact they are not speed tests at all but throughput tests.

A 12Mbps upload capacity may be limiting you if you are saturating it with other common internet traffic.

If you can get a visual of network traffic throughout the day, you might start seeing that you are up against your capacity at certain times. That would call for an increase in network service if possible, or configuration of QOS on your router.

A simple tool that you can use is "mtr" which you can keep running for a time to get a good idea about latency, loss, and jitter between you and your PBX. Just run it and step away for a while and come back and see what it looks like. If there are large deviations in latency (leading to high jitter), or any packet loss at all, your calls will sound bad.

 

[phinphan]

thanks Bill, I will give it a try. There is almost no one in the office (no usage except email and phone) and we have had only 2 concurrent calls but I will happily try another monitoring method. I will also ask my IT guy to set up QoS for our phones.

on another note, the dropouts are funny. We can hear the other party just fine but they cannot hear us. At the moment that happened last, I was on the phone with spectrum and my assistant was on the phone with another office. Neither party could hear us and we could hear them. I checked the logs which reported my ping to my server and to google dns failed as destination host unreachable but the reply address was the spectrum gateway for our fixed ip address. Looks to me (networking is not my strong suit) like the spectrum gateway responded. I think that means that our modem is working but there is a glitch between spectrum gateway and my server. On the other hand, the vast majority of the failed pings just say host unreachable with no reply IP address whatsoever.

 

[dicko]

Eliad - we really seem to have limited options. I am thinking of adding a 4glte backup but that will need a new expensive firewall so I am still hashing around different ideas.

On the plus side, I found a network monitoring tool http://www.nirsoft.net/utils/multiple_ping_tool.html which shows me that I cannot ping the server always. I am having about 1.34% failure rate for pings. Dropouts occur pretty often. See below. I have the program set to ping google dns at the same time and it has the same error rate and timing so I think that means its our internet not the server. Now to a discussion with Spectrum.

1/10/2020 11:11:37 AM Request Timeout
1/10/2020 11:11:26 AM Request Timeout
1/10/2020 11:11:15 AM Request Timeout
1/10/2020 11:04:58 AM Request Timeout
1/10/2020 11:04:47 AM Request Timeout
1/10/2020 11:04:36 AM Request Timeout
1/10/2020 10:56:57 AM Request Timeout
1/10/2020 10:56:46 AM Request Timeout
1/10/2020 10:56:35 AM Request Timeout
1/10/2020 10:56:24 AM Request Timeout
1/10/2020 10:56:14 AM Request Timeout
1/10/2020 10:55:42 AM Request Timeout
1/10/2020 10:55:31 AM Request Timeout
1/10/2020 10:55:20 AM Request Timeout
1/10/2020 10:55:10 AM Request Timeout
1/10/2020 10:44:37 AM Request Timeout
1/10/2020 10:44:26 AM Request Timeout
1/10/2020 10:44:15 AM Request Timeout
1/10/2020 10:44:04 AM Request Timeout
1/10/2020 10:43:54 AM Request Timeout
1/10/2020 10:34:40 AM Request Timeout
1/10/2020 10:33:48 AM Request Timeout
1/10/2020 10:15:28 AM Request Timeout
1/10/2020 10:12:13 AM Request Timeout
1/10/2020 10:12:01 AM Request Timeout
1/10/2020 10:11:50 AM Request Timeout
1/10/2020 10:11:40 AM Request Timeout
1/10/2020 10:11:18 AM Request Timeout
1/10/2020 10:11:08 AM Request Timeout
1/10/2020 9:59:25 AM Request Timeout
1/10/2020 9:59:14 AM Request Timeout
1/10/2020 9:58:43 AM Request Timeout
1/10/2020 9:58:32 AM Request Timeout
1/10/2020 9:58:21 AM Request Timeout
1/10/2020 9:54:06 AM Request Timeout
1/10/2020 9:53:45 AM Request Timeout
1/10/2020 9:53:34 AM Request Timeout
1/10/2020 9:53:23 AM Request Timeout
1/10/2020 9:53:12 AM Request Timeout
1/10/2020 9:53:01 AM Request Timeout
1/10/2020 9:52:50 AM Request Timeout
1/10/2020 9:52:39 AM Request Timeout
1/10/2020 9:52:08 AM Request Timeout
1/10/2020 9:51:56 AM Request Timeout
1/10/2020 9:35:49 AM Request Timeout
1/10/2020 9:35:38 AM Request Timeout
1/10/2020 8:43:33 AM Request Timeout
1/10/2020 8:43:22 AM Request Timeout
1/10/2020 8:43:11 AM Request Timeout
1/10/2020 8:33:31 AM Request Timeout
1/10/2020 8:33:19 AM Request Timeout
1/10/2020 8:33:08 AM Request Timeout
1/10/2020 8:32:57 AM Request Timeout
1/10/2020 8:32:46 AM Request Timeout
1/10/2020 8:32:24 AM Request Timeout
1/10/2020 8:32:14 AM Request Timeout
1/10/2020 8:32:02 AM Request Timeout
1/10/2020 8:31:52 AM Request Timeout
1/10/2020 8:31:40 AM Request Timeout
1/10/2020 8:31:29 AM Request Timeout
1/10/2020 8:31:18 AM Request Timeout
1/10/2020 8:31:07 AM Request Timeout
1/10/2020 8:30:57 AM Request Timeout
1/10/2020 8:30:45 AM Request Timeout
1/10/2020 8:30:34 AM Request Timeout
1/10/2020 8:30:23 AM Request Timeout
1/10/2020 8:30:12 AM Request Timeout
1/10/2020 8:30:01 AM Request Timeout
1/10/2020 8:29:50 AM Request Timeout
1/10/2020 8:29:40 AM Request Timeout
1/10/2020 8:29:28 AM Request Timeout
1/10/2020 8:29:17 AM Request Timeout
1/10/2020 8:29:06 AM Request Timeout
1/10/2020 8:28:55 AM Request Timeout
1/10/2020 8:28:45 AM Request Timeout
1/10/2020 8:28:33 AM Request Timeout
1/10/2020 8:15:40 AM Request Timeout
1/10/2020 8:15:28 AM Request Timeout
1/10/2020 8:15:17 AM Request Timeout
1/10/2020 8:15:06 AM Request Timeout
1/10/2020 8:14:25 AM Request Timeout
1/10/2020 8:14:14 AM Request Timeout
1/10/2020 8:14:03 AM Request Timeout
1/10/2020 8:13:52 AM Request Timeout
1/9/2020 5:23:36 PM Request Timeout
1/9/2020 5:23:25 PM Request Timeout
1/9/2020 5:23:14 PM Request Timeout
1/9/2020 5:23:03 PM Request Timeout

New but no so expensive firewall with 4g lte built in

https://mikrotik.com/product/wap_4g_kit

 

[billsimon]

We can hear the other party just fine but they cannot hear us.

QOS may help you here. This makes me suspect the imbalanced throughput of your ADSL/cable connection. You have 115Mbps throughput down and 12Mbps throughput up. If your uplink is full then you can't send any more traffic but you can still receive some with your bigger downlink. So you can hear them (download traffic), but they can't hear you (upload traffic). Just a hunch.

 

[phinphan]

They just called me back and said they found a problem where our modem is timing out frequently. Fingers crossed they can fix it. I would be fine with a qos fix as well. I just dont think there was any upload surge or use at that time but I dont know what our exchange server was doing at that moment.

 

[Eliad]

Eliad - we really seem to have limited options. I am thinking of adding a 4glte backup but that will need a new expensive firewall so I am still hashing around different ideas.

On the plus side, I found a network monitoring tool http://www.nirsoft.net/utils/multiple_ping_tool.html which shows me that I cannot ping the server always. I am having about 1.34% failure rate for pings. Dropouts occur pretty often. See below. I have the program set to ping google dns at the same time and it has the same error rate and timing so I think that means its our internet not the server. Now to a discussion with Spectrum.

1/10/2020 11:11:37 AM Request Timeout
1/10/2020 11:11:26 AM Request Timeout
1/10/2020 11:11:15 AM Request Timeout
1/10/2020 11:04:58 AM Request Timeout
1/10/2020 11:04:47 AM Request Timeout
1/10/2020 11:04:36 AM Request Timeout
1/10/2020 10:56:57 AM Request Timeout
1/10/2020 10:56:46 AM Request Timeout
1/10/2020 10:56:35 AM Request Timeout
1/10/2020 10:56:24 AM Request Timeout
1/10/2020 10:56:14 AM Request Timeout
1/10/2020 10:55:42 AM Request Timeout
1/10/2020 10:55:31 AM Request Timeout
1/10/2020 10:55:20 AM Request Timeout
1/10/2020 10:55:10 AM Request Timeout
1/10/2020 10:44:37 AM Request Timeout
1/10/2020 10:44:26 AM Request Timeout
1/10/2020 10:44:15 AM Request Timeout
1/10/2020 10:44:04 AM Request Timeout
1/10/2020 10:43:54 AM Request Timeout
1/10/2020 10:34:40 AM Request Timeout
1/10/2020 10:33:48 AM Request Timeout
1/10/2020 10:15:28 AM Request Timeout
1/10/2020 10:12:13 AM Request Timeout
1/10/2020 10:12:01 AM Request Timeout
1/10/2020 10:11:50 AM Request Timeout
1/10/2020 10:11:40 AM Request Timeout
1/10/2020 10:11:18 AM Request Timeout
1/10/2020 10:11:08 AM Request Timeout
1/10/2020 9:59:25 AM Request Timeout
1/10/2020 9:59:14 AM Request Timeout
1/10/2020 9:58:43 AM Request Timeout
1/10/2020 9:58:32 AM Request Timeout
1/10/2020 9:58:21 AM Request Timeout
1/10/2020 9:54:06 AM Request Timeout
1/10/2020 9:53:45 AM Request Timeout
1/10/2020 9:53:34 AM Request Timeout
1/10/2020 9:53:23 AM Request Timeout
1/10/2020 9:53:12 AM Request Timeout
1/10/2020 9:53:01 AM Request Timeout
1/10/2020 9:52:50 AM Request Timeout
1/10/2020 9:52:39 AM Request Timeout
1/10/2020 9:52:08 AM Request Timeout
1/10/2020 9:51:56 AM Request Timeout
1/10/2020 9:35:49 AM Request Timeout
1/10/2020 9:35:38 AM Request Timeout
1/10/2020 8:43:33 AM Request Timeout
1/10/2020 8:43:22 AM Request Timeout
1/10/2020 8:43:11 AM Request Timeout
1/10/2020 8:33:31 AM Request Timeout
1/10/2020 8:33:19 AM Request Timeout
1/10/2020 8:33:08 AM Request Timeout
1/10/2020 8:32:57 AM Request Timeout
1/10/2020 8:32:46 AM Request Timeout
1/10/2020 8:32:24 AM Request Timeout
1/10/2020 8:32:14 AM Request Timeout
1/10/2020 8:32:02 AM Request Timeout
1/10/2020 8:31:52 AM Request Timeout
1/10/2020 8:31:40 AM Request Timeout
1/10/2020 8:31:29 AM Request Timeout
1/10/2020 8:31:18 AM Request Timeout
1/10/2020 8:31:07 AM Request Timeout
1/10/2020 8:30:57 AM Request Timeout
1/10/2020 8:30:45 AM Request Timeout
1/10/2020 8:30:34 AM Request Timeout
1/10/2020 8:30:23 AM Request Timeout
1/10/2020 8:30:12 AM Request Timeout
1/10/2020 8:30:01 AM Request Timeout
1/10/2020 8:29:50 AM Request Timeout
1/10/2020 8:29:40 AM Request Timeout
1/10/2020 8:29:28 AM Request Timeout
1/10/2020 8:29:17 AM Request Timeout
1/10/2020 8:29:06 AM Request Timeout
1/10/2020 8:28:55 AM Request Timeout
1/10/2020 8:28:45 AM Request Timeout
1/10/2020 8:28:33 AM Request Timeout
1/10/2020 8:15:40 AM Request Timeout
1/10/2020 8:15:28 AM Request Timeout
1/10/2020 8:15:17 AM Request Timeout
1/10/2020 8:15:06 AM Request Timeout
1/10/2020 8:14:25 AM Request Timeout
1/10/2020 8:14:14 AM Request Timeout
1/10/2020 8:14:03 AM Request Timeout
1/10/2020 8:13:52 AM Request Timeout
1/9/2020 5:23:36 PM Request Timeout
1/9/2020 5:23:25 PM Request Timeout
1/9/2020 5:23:14 PM Request Timeout
1/9/2020 5:23:03 PM Request Timeout

I tried TMobile for backup LTE and did not work well. I think T mobile while posting good speedtests did not provide enough broad band. Crappy DSL still was better

 

[krzykat]

Eliad - we really seem to have limited options. I am thinking of adding a 4glte backup but that will need a new expensive firewall so I am still hashing around different ideas.

On the plus side, I found a network monitoring tool http://www.nirsoft.net/utils/multiple_ping_tool.html which shows me that I cannot ping the server always. I am having about 1.34% failure rate for pings. Dropouts occur pretty often. See below. I have the program set to ping google dns at the same time and it has the same error rate and timing so I think that means its our internet not the server. Now to a discussion with Spectrum.

I recommend PingPlotter Pro - many companies use it and when I've had to get into pissing contests with cable companies, and I provide them with graphs and details from PPP, they tend to own up to the issue much more quickly.
Also, it will not only should you dropped packets, but also latency/Jitter issues which are typically the bigger issue.

 

[phinphan]

krzykat - thanks very much. I will run that and see what it shows.

 

[phinphan]

Finally found the problem. After being lied to about 15 times by customer service, twitter customer service admitted that they have a problem upstream from us which is affecting my whole area. It has been mostly fixed. I will keep my fingers crossed.