SOLVED SRTCP unprotect failed (Unhold issues)

[TylerP]

We just upgraded our phone servers to the new Incredible PBX (Asterisk Version: 18.2.1). I can't find any other form online that has offered a suggestion for a fix to a hold issue we are having.

When a call comes in, audio on both ends works until that call is placed on hold from our end for what seems to be a certain amount of time; When the call is retrieved from hold, there is no audio on our end, but sometimes the other end can still hear us. I can pretty reliably reproduce the issue by calling from my cell phone into our phone server, placing myself on hold, and then waiting to retrieve that call for about 15-20+ minutes (anytime the call is retrieved before that time, audio resumes okay on both ends). Our phone server at that time then continuously produces the following logs until the call is terminated by either side:

19243 [2022-10-04 16:44:21] VERBOSE[21514][C-0000001c] res_srtp.c: SRTCP unprotect failed on SSRC 1667512219 because of replay check failed (index too old)
19244 [2022-10-04 16:44:21] VERBOSE[21514][C-0000001c] res_srtp.c: SRTP unprotect failed on SSRC 1667512219 because of authentication failure 160

This message sometimes also appears:

5904 [2022-10-04 11:21:18] WARNING[27806] res_pjsip_outbound_registration.c: No response received from 'sip:**our-telephony-provider.com**:5060' on registration attempt to 'sip:70511898@**our-telephony-provider.com**:5060', retrying in '60'

Our telephony provider looked a bit into this and said it may be a media session timeout while on hold. They also could not tell from a PCAP of the issue if it was our sever or the softphone we are using (Linphone). They pointed us to contact Asterisk support.
Not sure if this is the right place for this, but would someone be able to help me troubleshoot this?

 

[dallas]

Are you playing music on hold to the caller? It appears to be due to a lack of RTP (SRTP) activity.

 

[TylerP]

Yes we are playing music on hold from an external source. Would that cause the lack of RTP (SRTP) activity?

 

[kenn10]

Searching about this using Google, I see that there have been a number of issues like this. Digium says Asterisk is working correctly and points to the softphone or the carrier as the issue. One anecdotal fix said that they had reduced the number of CODECs in Asterisk from ten or more to just an essential two or three and the issue went away but I only saw one instance of that.

Current Asterisk release is at 19.6.0 but I'm not sure that upgrading would change anything. Many replies indicated that the softphone lost track of the SRTP encryption sequence after long hold times. Using hard phones, no one reported this type of issue.

I'd recommend trying a VOIP phone to test this theory and if that is the issue, test a different soft phone client.

 

[Samot]

When you put someone on hold there are two options really for it. You either null the audio in the stream or you end up with sendonly meaning that your side is sending the audio, such as music on hold. There is no *incoming audio* so playing hold music don't really mean anything since outbound audio isn't having issues.

When you pickup a call that was on hold the call is setup just like it was before. There are INVITEs, codecs negotiated and all that good stuff. This is just like those problems where someone gets an inbound call, can't hear the person puts them on hold/takes off hold and boom, they have audio. Because INVITEs were done again and this time any NAT issues have cleared up. That's how this is working, they have audio until they put the call on hold and then have to pick it up. When the INVITEs are handled from that, looks like NAT is being nasty.

Are the softphones are on the same network as the PBX? If they are then this is more likely an issue with the firewall in front of the PBX. If they are not then you need to determine if its the audio on the softphone side or the audio on the provider side (the channels) that are having the issue. If you can make internal calls and keep doing hold/off hold and audio is fine but calls coming in from the PSTN still have the issue, it's your firewall most likely.

 

[TylerP]

Searching about this using Google, I see that there have been a number of issues like this. Digium says Asterisk is working correctly and points to the softphone or the carrier as the issue. One anecdotal fix said that they had reduced the number of CODECs in Asterisk from ten or more to just an essential two or three and the issue went away but I only saw one instance of that.

Current Asterisk release is at 19.6.0 but I'm not sure that upgrading would change anything. Many replies indicated that the softphone lost track of the SRTP encryption sequence after long hold times. Using hard phones, no one reported this type of issue.

I'd recommend trying a VOIP phone to test this theory and if that is the issue, test a different soft phone client.

Thank you for your reply. I also came across that or a similar article while googling this issue. I narrowed it down to use only one codec: G722. I also tried several others individually like G729, G719, GSM, alaw, and several others that were compatible with SRTP, but the issues still persisted. I might consider upgrading to 19 if all else fails.

I don't remember if I specifically tried hard phones, but we primarily use our phone server with soft phones so people can still have the option to work remotely.

Other soft phone options require TLS when using SRTP with PJSIP, which is not setup on our server. My initial attempts to get TLS were unsuccessful; maybe I need to get TLS setup instead of UDP. (I have also tried TCP)

Are there any other free alternatives to Linphone that play nice with Incredible PBX?

 

[TylerP]

When you put someone on hold there are two options really for it. You either null the audio in the stream or you end up with sendonly meaning that your side is sending the audio, such as music on hold. There is no *incoming audio* so playing hold music don't really mean anything since outbound audio isn't having issues.

When you pickup a call that was on hold the call is setup just like it was before. There are INVITEs, codecs negotiated and all that good stuff. This is just like those problems where someone gets an inbound call, can't hear the person puts them on hold/takes off hold and boom, they have audio. Because INVITEs were done again and this time any NAT issues have cleared up. That's how this is working, they have audio until they put the call on hold and then have to pick it up. When the INVITEs are handled from that, looks like NAT is being nasty.

Are the softphones are on the same network as the PBX? If they are then this is more likely an issue with the firewall in front of the PBX. If they are not then you need to determine if its the audio on the softphone side or the audio on the provider side (the channels) that are having the issue. If you can make internal calls and keep doing hold/off hold and audio is fine but calls coming in from the PSTN still have the issue, it's your firewall most likely.

Thanks for quick reply! In my testing, when I take the call off hold and get no audio, putting the call back on hold and unholding the call does not resume the audio (even after doing that multiple times on the same call).

The phone server is behind a firewall in GCP. Our previous phone server was on our LAN. What NAT setting on our GCP firewall should I be looking for specifically?

 

[wardmundy]

Are there any other free alternatives to Linphone that play nice with Incredible PBX?

Have you tried Zoiper?

 

[Samot]

Thanks for quick reply! In my testing, when I take the call off hold and get no audio, putting the call back on hold and unholding the call does not resume the audio (even after doing that multiple times on the same call).

The phone server is behind a firewall in GCP. Our previous phone server was on our LAN. What NAT setting on our GCP firewall should I be looking for specifically?

So your PBX is in the cloud/data center not where the phones are. Here's what you do, you go in and turn call recording on for a call then make a test call into the PBX, answer the call, confirm there is audio put call on hold and take back off hold (which should be no incoming audio) then end the call. Once that is all done you go in and listen to said call recording, if there is two way audio of the call at the PBX level then there is an issue where the phones are with NAT/firewall. If there is no inbound audio after the hold is taken off at the PBX then there is something between the PBX and the carrier with the call.

 

[dallas]

There is no *incoming audio* so playing hold music don't really mean anything since outbound audio isn't having issues.

Well spotted! Excellent explanation.

 

[TylerP]

Have you tried Zoiper?

We have not because they require TLS which we don't have setup on our server

 

[TylerP]

So your PBX is in the cloud/data center not where the phones are. Here's what you do, you go in and turn call recording on for a call then make a test call into the PBX, answer the call, confirm there is audio put call on hold and take back off hold (which should be no incoming audio) then end the call. Once that is all done you go in and listen to said call recording, if there is two way audio of the call at the PBX level then there is an issue where the phones are with NAT/firewall. If there is no inbound audio after the hold is taken off at the PBX then there is something between the PBX and the carrier with the call.

Ha, funny thing is that was the one of the first things I checked when we first noticed the issue. We already had the call recording option enabled and the recording picked up both ends; However only the outside caller could hear the our end, but we couldn't hear them. This leads me to believe you are right about the firewall/NAT on our GCP. Thank you! I will investigate that further!

 

[Samot]

Ha, funny thing is that was the one of the first things I checked when we first noticed the issue. We already had the call recording option enabled and the recording picked up both ends; However only the outside caller could hear the our end, but we couldn't hear them. This leads me to believe you are right about the firewall/NAT on our GCP. Thank you! I will investigate that further!

That's not what I said. If the PBX is getting both streams of the audio then the issue is the firewall/NAT that the softphones are behind. That's why I asked if they were on the same network or remote, the latter means an additional layer of NAT before it gets to the softphone app. So PSTN -> PBX audio is fine. PBX -> softphone, no incoming audio. It would be related to the NAT the softphone is behind.

 

[TylerP]

That's not what I said. If the PBX is getting both streams of the audio then the issue is the firewall/NAT that the softphones are behind. That's why I asked if they were on the same network or remote, the latter means an additional layer of NAT before it gets to the softphone app. So PSTN -> PBX audio is fine. PBX -> softphone, no incoming audio. It would be related to the NAT the softphone is behind.

Oh apologies for the misunderstanding. Thank you! I will investigate our local firewall/NAT settings

 

[wardmundy]

We have not because they require TLS which we don't have setup on our server

We use Zoiper all the time without TLS. The PRO version also supports TLS, but it's not required.

 

[TylerP]

I tried getting Zoiper to work with UDP and encryption (SRTP) with no success. After several failed attempts, I reached out to their support and unfortunately they made it clear that TLS was required for encryption (SRTP).

 

[wardmundy]

Is SRTP a necessity? Do both the other end of your call and your provider support it? Are these PSTN terminated calls?

 

[dallas]

unfortunately they made it clear that TLS was required for encryption (SRTP).

Of course... What's the point of SRTP if your call signalling is not encrypted? In any case you can't guarantee encryption end to end.

 

[TylerP]

Is SRTP a necessity? Do both the other end of your call and your provider support it? Are these PSTN terminated calls?

Sorry I am not too familiar with Asterisk Terminology. what is meant by PSTN terminated calls?

 

[kenn10]

Calls terminated on the analog/TDM switched telephone network. This would be the switching offices that handle all the copper lines to people's homes and businesses as opposed to digital switched SIP services.