Tags: SIP, RTP, MTR, Latency, Packet Loss, PacketLoss, VOIP, trixbox, Asterisk, CenturyLink, DSL, NetworkAnalysis, Network Analysis, NetworkTroubleshooting, Network Troubleshooting, OneWayAudio, One Way Audio
QUESTION SUMMARY: My office is having one-way audio quality issues to multiple remote parties over both incoming and outgoing calls. We in the office always hear the remote parties clearly. We are not experiencing dropped calls. Troubleshooting seems to point to our Internet connection via CenturyLink DSL. I am looking for feedback on the latest MTR reports from our IP PBX (I suggest the Excel file for easiest viewing) and whether you think they still indicate an issue, and if not what do you think is causing our problem? Suggestions for further troubleshooting steps are also greatly appreciated.
I am in the 5th day of our business phone system essentially being down. We are experiencing variable yet usually unintelligible one-way audio quality issues with the remote party (i.e. not us in the office) on both inbound and outbound calls. We can always hear the remote party perfectly (call quality inbound to our office is very good). All outbound calls connect fine and all inbound calls to the best of our knowledge connect fine as well. We have not experienced any dropped calls - just very poor one-way audio with the remote party not being able to understand our garbled/dropped speech. Call quality is excellent extension to extension and extension to voice mail (i.e. all within our LAN). This phone system in its current configuration has been very stable for years (with the exception of one HDD failure in the server - was RAID 1 so not a real issue - and a serious hack a couple years ago).
I have called Fonality, whose product trixbox Pro we use as our local hosted IP PBX and also through whom we have a support contract, and they say everything looks good on our server and that it must be a carrier issue (e.g. our virtual SIP trunk provider 8x8). Fonality's tech support has been helpful and knowledgeable in the past. I have called 8x8 and they said they were not having any service issues. They placed a call directly from their data center to our IP PBX, essentially testing solely the network connection between our IP PBX and 8x8, and the call quality was similarly poor. This eliminates the likelihood that it is an upstream carrier issue. 8x8 Tier 2 support was very nice and suggested I run MTR's to their east and west coast data centers after power cycling all my equipment (which I had already done multiple times).
The MTR's (Matt's traceroute, AKA My traceroute) appeared to show significant packet loss and latency both across CenturyLink's network and Level3. I called CenturyLink and got the run around, but eventually got to a Tier 4 Supervisor who has been helping me. He said yesterday that their network ops center had done some reconfiguration and to test the WAN connection again. I have attached to this question the most recent MTR reports in both text format and MS Excel format (much easier to read) from the trixbox Pro server, which is running a slimmed down/custom version of CentOS 4.4 (Final). These MTR's were run via CLI/Bash on the IP PBX server, whose NIC is set to a static IP, connected via a short patch cord to a newly replaced (yesterday) CenturyLink ZyXEL PK5001Z DSL modem/wireless router in full bridge mode (wireless is of course disabled). I normally have a ZyXEL ZyWALL 2 Pro firewall in between the modem and server, set in transparent/bridge mode, serving only as a simple firewall, but removed it for the purpose of all MTR tests, both initial and after the changes CenturyLinks ops center supposedly made. There is no router, no switch, or other intervening device between the IP PBX server and the DSL modem. Our DSL line is dedicated to our phone system.
The MTR reports do show significant improvement in packet loss and latency since CenturyLink's ops center supposedly made their changes, but our one way audio issue to remote parties is still about as bad as it was before. For comparison, I have attached MTR reports to the same hosts as I tested from the IP PBX server, but this time using WinMTR on my laptop connected to our office LAN/switch/SonicWALL via Comcast Business cable. They actually look a lot worse than the current CenturyLink reports. Any thoughts on why this might be?
My question has two parts: what are your thoughts on the latest MTR reports attached hereto, and also what do you think could be causing our one-way audio quality issues to remote parties?
Thank you very much for any insights or thoughts, and I do apologize for the long explanation/question.