SOLVED GV stopped connecting today

[yozh]

Upgrade script has bunch of errors, have to go thru them, but otherways I think its good ... Reverting snapshot now to test again. Im wondering what changed on google side and broke this.

 

[billsimon]

I don't think we have a good understanding of the root cause. DSLReports VoIP Tech forum has reports from Obihai owners getting locked out of Google Voice yesterday and Asterisk users as well. Some Asterisk users updated their systems and the problem went away. Others claim to have done nothing and the problem went away.

My own perspective on it is that it is important to have the most up-to-date TLS, because Google is one of the leaders in deprecating old SSL/TLS suites that are less secure, as well as OAUTH2, because Google has been pushing for this for several years. I will not be surprised when Google forces the issue requiring the most current TLS schemes as well as OAUTH2 and abandoning user/pass auth.

 

[yozh]

I don't think we have a good understanding of the root cause. DSLReports VoIP Tech forum has reports from Obihai owners getting locked out of Google Voice yesterday and Asterisk users as well. Some Asterisk users updated their systems and the problem went away. Others claim to have done nothing and the problem went away.

My own perspective on it is that it is important to have the most up-to-date TLS, because Google is one of the leaders in deprecating old SSL/TLS suites that are less secure, as well as OAUTH2, because Google has been pushing for this for several years. I will not be surprised when Google forces the issue requiring the most current TLS schemes as well as OAUTH2 and abandoning user/pass auth.

Intresting. Let me go back to my original box again and see if it just works without an upgrade.

 

[Alex1]

How can I upgrade on green from 11.4 to 11.22 ?

EDIT: Trying with the script from here http://pbxinaflash.com/community/threads/original-asterisk-upgrade-script.12560/

Had to edit the file a bit was getting error: "Sorry. Asterisk compile from source not supported by SHMZ OS."

PIAF Installed Version = 2.0.6.4 under *VMWARE* x
x FreePBX Version = 2.11.0.31 x
x 2.11 x
x Running Asterisk Version = UNKNOWN x
x Asterisk Source Version = 11.4.0 x
x Dahdi Source Version = 2.6.2 x
x Libpri Source Version = 1.4.12 x
x IP Address = XXXXX on eth0 x
x Operating System = CentOS release 6.4 (Final) x
x Kernel Version = 2.6.32-358.6.2.el6.i686 - 32 Bit

Would you mind sharing the edits you made to make this upgrade work? Bracing myself to upgrade from 11.12 to 11.22.

 

[yozh]

Would you mind sharing the edits you made to make this upgrade work? Bracing myself to upgrade from 11.12 to 11.22.

I just changed the line:
if [ -d "/etc/schmooze" ]; to if [ -d "/etc/schmooze1" ]; so it would go thru.

 

[yozh]

WOW, so here is the thing

Intresting. Let me go back to my original box again and see if it just works without an upgrade.

This is amazing !!!! Went back to my original box, and guess what, its working !!!! So whatever it is google was doing it's either done, or there is another wave coming....

Now I`m not sure if 11.4 upgrade to 11.22 fixed it or was it fixed on google side before that... (meaning when 11.4 was not working and 11.22 was or were they both working)

 

[Huckda]

WOW, so here is the thing



This is amazing !!!! Went back to my original box, and guess what, its working !!!! So whatever it is google was doing it's either done, or there is another wave coming....

Now I`m not sure if 11.4 upgrade to 11.22 fixed it or was it fixed on google side before that... (meaning when 11.4 was not working and 11.22 was or were they both working)

I confirm this, on my system right now...
and all I can say is, ... I don't understand it, but that's not actually required(though often desired)... but work it does.

 

[JRoque]

Hi. I posted on this earlier. My GV trunks have had this issue on/off for a few weeks now. No changes made to cause or fix, it just goes away after a while. Because Jabber (?) retries so furiously, it opens all the available sockets in my firewall until it can't route anything else through. For that reason, I deleted my GV trunks. I'm now upgraded to 11.22 and will try again but whatever is flipping, it's on Google's end.
Regards,
JR

 

[rob-pbiaf]

Bill Simon directed me to this thread when I first reached out to him when I was experiencing the problem yesterday on my Tomatoware Asterisk. I commented out all my XMPP entries for the day and made other arrangements, and today tried them again and all is well. I have made no upgrades to asterisk or any library.

For what it is worth:
Asterisk 11.9.0
# openssl version
OpenSSL 1.0.1g 7 Apr 2014

With all the XMPP errors my asterisk process crashed and was unable to log in to the box that hosts it, which is just a router. The router did function as a router without asterisk.

In my humble opinion the XMPP implementation is broken to allow this sort of problem to take out asterisk, and hurt any part of the host system. My asterisk configuration also uses other SIP services and nothing was working until I commented out my XMPP configuration parts.

I am quite grateful to be able to make such good use of Google Voice as I do with the configuration I am running but quite worried about the next wave of trouble.

By the way, another independent line running through an Obi device stayed working just fine.

 

[wardmundy]

Just reactivated four GV trunks on our oldest PBX (Asterisk 1.8.11-cert1), and everything is back to normal. Chalk it up to another GV hiccup.

 

[tycho]

... a hiccup that apparently got a lot of Obi users to frantically buy the one-way-ticket to non-returnable OAUTH2 firmware.

 

[silox]

I registered just to post in this thread and add to the conversation. I'm running OpenWRT Backfire on a Netgear WNDR3700 and Asterisk 1.8 using gtalk/Google Voice. I've been using this setup successfully for years and I started having problems a couple of days ago like everyone else (TLS error with jabber module).

I read here that users of newer software weren't impacted so I decided to fire up an old Pogo Plug Pro, install a new uboot, install new kernel w/ Debian and Incredible PBX. It's installing now currently compiling Asterisk 13.

Funny now that I check this thread folks indicate it's working again and sure enough, it is for me too. I'll likely still be using the Incredible PBX install anyways once it's installed/configured, tons more features and more modern versions Plus I love making use out of older hardware that's been idle for awhile!

 

[gavimobile]

Please note that this post of mine was actually written before i posted it on this thread. I was waiting for the admin to approve my membership, so i started writing this up in advance.Also, because this issue has been resolved, my post is not so relevant. i will still post anyways for extra feedback for the community.

My PBX with google voice was working fine up until June 14th 2016 until early AM (GMT+2). As you can see in my logs below, some guy tried to authenticate at 3:58 AM (not related to this post, this is just to point out the time difference between his hack attempt and when i started getting these xmpp errors.) On its own at 4:03 AM these errors started happening and were showing up in the CLI till late yesterday June 15th 2016. I unloaded the xmpp module so my would stop displaying the message and would enable it when running a test.

[2016-06-14 03:58:10] NOTICE[1365][C-0000019e] chan_sip.c: Failed to authenticate device 1001<sip:[email protected]>;tag=ed76be85
[2016-06-14 04:03:49] WARNING[1359] res_xmpp.c: JABBER: socket read error
[2016-06-14 04:03:50] WARNING[1360] res_xmpp.c: JABBER: socket read error
[2016-06-14 04:03:50] ERROR[1359] res_xmpp.c: TLS connection for client 'google' cannot be established. OpenSSL initialization failed.
[2016-06-14 04:03:50] WARNING[1359] res_xmpp.c: Parsing failure: Hook returned an error.
[2016-06-14 04:03:50] WARNING[1359] res_xmpp.c: JABBER: socket read error
[2016-06-14 04:03:50] ERROR[1359] res_xmpp.c: TLS connection for client 'google' cannot be established. OpenSSL initialization failed.
[2016-06-14 04:03:50] WARNING[1359] res_xmpp.c: Parsing failure: Hook returned an error.
[2016-06-14 04:03:50] WARNING[1359] res_xmpp.c: JABBER: socket read error
[2016-06-14 04:03:50] ERROR[1360] res_xmpp.c: TLS connection for client 'google' cannot be established. OpenSSL initialization failed.

I’ve logged into goggle to make sure it’s not asking for a captcha as ariban mentioned in the beginning of this post and less protected apps option is enabled. I have a vm clone of my pbx with google voice from several months ago when everything was working just fine, which I booted up just to verify that this was not a configuration problem on my end. The vm was posting the same errors in the CLI immediately on boot. Ive rebooted my router just in case of anything fishy with my router. I’ve temporarily disabled net-filters on my PBX. I’ve updated openssl along with all the other outdated apps on my centos 6 pbx system. I was considering upgrading from asterisk 11 to asterisk 12 until I discovered that you all seem to be experiencing a similar or exact issue. I ran xmpp set debug on however I didn’t find any useful information there. There was no output of failure or error in the debug log.

I’ve made test calls to my incoming google voice line and incoming calls work. When I dial out using google voice to known working numbers it rings and rings but that’s all. I tried again right after to the same working number using a pots line to the same working number and they work and get answered immediately.

Im wondering what this has to do with jabber if im using motif.conf, xmpp.conf and NOT using jabber.conf

After starting to believe that the issue was not on my end, I checked later (June 16th 2016) to see if the issue was resolved. sure enough the errors were no longer displaying in my CLI. Today the 16th of June 2016 i've tested both incoming and outgoing calls using google voice and its back and working. thanks everyone for contributing their experience with this issue.

 

[tycho]

I registered just to post in this thread and add to the conversation.

Welcome to the circus, @silox...

The lesson I take from everything that happened (and that has similarly happened on a couple of prior occasions) is to do nothing for a few days to see if things automagically correct themselves. If things go back to normal, fantastic! If not, well, then you can follow the path blazed by the early adopters who groped for solutions in the dark, oftentimes heading down the wrong path.

2 years ago, in May of 2014, the sky was falling because Google was dropping the hammer and everything was supposed to break. I stayed pat with my Obi100 and my Motif PBX trunks and the world did not come to an end. The same scenario has replayed several times since then. This time around many folks expended a lot of effort upgrading Asterisk, SSL, building completely new PBXes, upgrading Obi devices to firmware that locks-out manual GV set-up and that only permits GV use if connected to Obihai's ObiTalk portal, etc. 36 hours later the XMPP error corrected itself.

While things were SNAFU my incoming GV worked fine. I don't terminate incoming GV to Google Chat-connected devices. Such calls instead simul-forward to my cell (Free via RingPlus!) and a free CallCentric NY DID registered to an old PAP2T plugged (along with my router and cable modem) into a hefty UPS. To make outgoing calls I used my free R+ cell (which sends GV number) and, for a brief period of time, I amused myself by using the old-school "call-back" method from the GV portal. Or, I used FireRTC (which spoofs my GV number). A bit later I flipped service to my Dockstar PBX running the PogoPlug image with Motif GV trunks. When that also broke I went back to cell and call-back. Multiple paid VoIP providers that charge virtually nothing (much less than a penny per minute -- can you say CircleNet?) were waiting in the wings, but I never had to go there.

There are many smart people around here. I am not one of them. So I take a vacation until the smart people figure it all out. Doing so saves me a great deal of grief as well as the impact of possibly irreversible actions.

 

[gavimobile]

Welcome to the circus, @silox...

The lesson I take from everything that happened (and that has similarly happened on a couple of prior occasions) is to do nothing for a few days to see if things automagically correct themselves. If things go back to normal, fantastic! If not, well, then you can follow the path blazed by the early adopters who groped for solutions in the dark, oftentimes heading down the wrong path.

2 years ago, in May of 2014, the sky was falling because Google was dropping the hammer and everything was supposed to break. I stayed pat with my Obi100 and my Motif PBX trunks and the world did not come to an end. The same scenario has replayed several times since then. This time around many folks expended a lot of effort upgrading Asterisk, SSL, building completely new PBXes, upgrading Obi devices to firmware that locks-out manual GV set-up and that only permits GV use if connected to Obihai's ObiTalk portal, etc. 36 hours later the XMPP error corrected itself.

While things were SNAFU my incoming GV worked fine. I don't terminate incoming GV to Google Chat-connected devices. Such calls instead simul-forward to my cell (Free via RingPlus!) and a free CallCentric NY DID registered to an old PAP2T plugged (along with my router and cable modem) into a hefty UPS. To make outgoing calls I used my free R+ cell (which sends GV number) and, for a brief period of time, I amused myself by using the old-school "call-back" method from the GV portal. Or, I used FireRTC (which spoofs my GV number). A bit later I flipped service to my Dockstar PBX running the PogoPlug image with Motif GV trunks. When that also broke I went back to cell and call-back. Multiple paid VoIP providers that charge virtually nothing (much less than a penny per minute -- can you say CircleNet?) were waiting in the wings, but I never had to go there.

There are many smart people around here. I am not one of them. So I take a vacation until the smart people figure it all out. Doing so saves me a great deal of grief as well as the impact of possibly irreversible actions.

Thanks for the welcome! I remember with 1.8 I had problems connecting to gv and after i upgrade to 10 things We're working again. Maybe that too was just a matter of time. If I remembered correctly I had an easier time configuring motif than jabber

 

[[email protected]]

So I wish I was a bit more patient and saw this before. Or maybe not . . .
Yesterday I couldn't get Google Voice to work on my Asterisk 11 box and so decided to upgrade to IncrediblePBX3 using the ISO. This installs on SL 6.7 32bit. After many hours (~20 hours) of manually installing and transferring settings and complications with PJSIP on trunks and endpoints, I finally have most of it working including Google Voice with OAUTH2. Only one problem remains. I use Schmoozecom's Endpoint Manager to manage Grandstream GXV3240 phones (among others not supported by OSS Endpoint Manger) and because Commercial Modules is not supported, I cannot get it installed. This URL (http://wiki.freepbx.org/display/FPG/Installation+on+CentOS+and+RHEL+based+systems) explains how to install Commercial Modules on FreePBX distros including SL based but it does not work. Apparently SL 6.7 is not a supported OS and queries to the Schmooze-Commercial.repo fail. Any suggestions from anyone before I abandon and start over with Centos from scratch? I do like Asterisk 13 and prefer to stay up to date on the software.

 

[yozh]

So after going back to the original box, I started getting a lot of errors like below,

pbx asterisk[28793]: WARNING[28813]: res_xmpp.c:3640 in xmpp_client_thread: JABBER: socket read error

It would still stay connected, but errors would popup. I setup splunk to kill asterisks when I get this errors, so I had to go back to 11.22 clone. Which seems to be stable, but once in awhile the error above pops up. The accounts actually stay connected (have a script that checks connections prior to killing asterisks process), so I dont understand where the errors coming from ?

 

[billsimon]

So after going back to the original box, I started getting a lot of errors like below,

pbx asterisk[28793]: WARNING[28813]: res_xmpp.c:3640 in xmpp_client_thread: JABBER: socket read error

It would still stay connected, but errors would popup. I setup splunk to kill asterisks when I get this errors, so I had to go back to 11.22 clone. Which seems to be stable, but once in awhile the error above pops up. The accounts actually stay connected (have a script that checks connections prior to killing asterisks process), so I dont understand where the errors coming from ?

That seems to be normal and not a reason for concern. Asterisk reconnects if the connection is broken. As long as it is reconnecting successfully you don't have to do anything.

As for why the connection breaks, there are a lot of possibilities, but remember that this is a TCP connection so you will see every network error, as opposed to UDP (probably most or all of your SIP traffic) where you are only notified of problems after a lot of retries.

 

[yozh]

That seems to be normal and not a reason for concern. Asterisk reconnects if the connection is broken. As long as it is reconnecting successfully you don't have to do anything.

As for why the connection breaks, there are a lot of possibilities, but remember that this is a TCP connection so you will see every network error, as opposed to UDP (probably most or all of your SIP traffic) where you are only notified of problems after a lot of retries.

So I have to figure out what to set splunk to alert to and kill asterisks... maybe "cannot be established. OpenSSL initialization failed." ?

 

[wardmundy]

Looks like Google Voice is experiencing quirks again today: