TIPS Dead stream should trigger restart of streamer

[herbertshades]

Know how we can stream moh? If the stream breaks for whatever reason, the ffmpeg (or other) program never quits so there's no music. So I normally do a fwconsole restart, which I am sure is overkill.

Is there a way to detect that the stream failed and bounce the player program?

 

[dallas]

Is there a way to detect that the stream failed and bounce the player program?

I found this on Reddit. Disclaimer, I have no experience with ffmpeg so this may not be of any use.

Continue stream output when input is stopped​

I think I have come up with a workable solution of my own in case anyone hits this issue.
Put the ffmpeg command line in a looping bash script.
Add the -stimeout option with a pretty short timeout, say 5 seconds to make ffmpeg giveup and terminate relatively quickly when the incoming stream dies.
Add a second ffmpeg command inside the loop after the above one that streams the placeholder video/image/whatever for 30 seconds.
This should result in the stream working until an interruption and then streaming the placeholder for 30 seconds over and over again until the input stream becomes available again - with some brief interruptions each time the original stream is retried after each placeholder stream.

 

[dicko]

Try opening the stream with socat, it has -t and -T
timeout options, pipe its stdout through whatever you use

 

[herbertshades]

It could be mpg123 or ffmpeg or other...

 

[dicko]

For ffmpeg, this is one solution using the BBC and some ugly shell scripting


The moh stream to stdout, piped through pv which is forcing output of only the stream rate but redirecting stderr to a file whilst then inline replacing return ('\r)' with a linefeed ('\n')

ffmpeg -loglevel error  -i    https://stream.live.vc.bbcmedia.co.uk/bbc_world_service  -vn -ar 8000 -ac 1 -f s16le  - | pv -fr -i 30   2> >(stdbuf -oL tr '\r' '\n' > rates.log)

You can then shell friendly monitor `rates.log` with

tail -f rates.log|while read LINE;do echo $LINE|sed 's/\[\([0-9\.]*\).*/\1/g'
;done

In this skeleton , then with a granularity of 30 seconds, when the $LINE includes a rate of presumably 0.0, you will know it's time to HUP ffmpeg

 

[dicko]

Thinking about this and prevent the need to reread moh with asterisk , then a simpler solution is to ensure you have socat installedb and change your moh 'application' to

/usr/bin/socat  udp-recv:42042 -

and run from a shell (or systemd or however) a

while true ;do
{whatever was your working ffmpeg|mpg123|other' application}  | socat -T 10 - udp-datagram:localhost:42042
sleep 1 # so you can ctl-c whilst debugging 
done

in my demo case

while : ;do
 /usr/bin/ffmpeg -loglevel error  -i    https://stream.live.vc.bbcmedia.co.uk/bbc_world_service  -vn -ar 8000 -ac  1 -f s16le - | /usr/bin/socat -T 10 - udp-datagram:localhost:42042
done

both socat's udp-datagram and udp-recv are non-blocking and fully isolated from each other, the -T 10 means that if the moh process stops sending bytes for more than 10 seconds, it will be restarted.

 

[wardmundy]

May also need to whitelist UDP 42042.

 

[dicko]

May also need to whitelist UDP 42042.

Probably not as the traffic is all on the lo interface in this case ( but you could also use any multicast address:randomport ;-) ) why not choose 20000 and have everyone fix their currently wrong rdp port range ? ;-)

You could also of course have a whole jukebox running multiple streams on any number of address port pairs from another routeably connected machine but ensure your bandwidth is adequate.

Just adding that this give you the chance to simply change your moh contemporaneously and without touching any asterisk/freepbx setup.

 

[dicko]

So you don't get a bill from the license owner of any streamed recordings and although I am not Swiss

play -t mp3 http://relay.publicdomainproject.org/jazz_swing.mp3
play -t mp3 http://relay.publicdomainproject.org/classical.mp3

both claim to be unimpedimented.

 

[Halea]

Causing either ffmpeg or mpg123 to exit when the streaming is interrupted is quite easy!

You need to use a timeout parameter of a few seconds. I typically set it to 10 seconds.
For mpg123 it's the command line parameter --timeout 10 (notice double dash) and for ffmpeg it would be -timeout 10000000 (notice single dash and lots of zeros).
Note that for mpg123 you enter seconds, but for ffmpeg you enter micro-seconds, thus the extra 6 zeros to go from micro to second.

That said I am not sure you would want to exit those players if the stream broke down! Why? Simply because asterisk/FreePBX would immediately call the same command line again mindlessly. If that happened your DNS look ups will go through the roof and CPU utilization jump to saturation.

The smart way to address this issue is to take the control away from asterisk/FreePBX.

You can do it by creating a bash script which we will call for instance "playstream" and make sure that it actually never exits!

In asterisk/FreePBX you would simply call playstream on the command line where you usually enter your ffmpeg or mpg123 line.
Inside the playstream script file you would run one or more MOH command lines like the following:

/opt/ffmpeg/bin/ffmpeg -f s16le -ac 1 -ar 8000 -vol 32 pipe:1 -timeout 10000000 -i http://ais-sa1.streamon.fm/7833_128k.aac;

/usr/bin/mpg123 --timeout 10 -q -s --mono -r 8000 -f 8192 -b 0 http://stream.live.vc.bbcmedia.co.uk/bbc_world_service;

(The first one streams WHUD the second is for BBC)

So, if the first one ever interrupted for more than 10 seconds, the script would go to the next line and that would run until it stopped streaming for more than 10 seconds.

Of course you should have enough sources and the proper logic so that this script file actually never exits!
You can use loops, counters, random calls, etc so that you get a constant flow of feeds.

You can also use the case statement so that you can pass to the script file a radio channel parameter, like the call letters WHUD, KABL, WINS, WCBS and connect to the proper ffmpeg or mpg123 instruction inside the script file. Then again, you would supplement the command in such a way that if that stream is interrupted the script starts another (maybe) random stream or even a static music file to prevent the script from exiting.

So if you can do basic scripting the sky is the limit.

 

[dicko]

Causing either ffmpeg or mpg123 to exit when the streaming is interrupted is quite easy!

You need to use a timeout parameter of a few seconds. I typically set it to 10 seconds.
For mpg123 it's the command line parameter --timeout 10 and for ffmpeg it would be --timeout 10000000
Note that for mpg123 you enter seconds, but for ffmpeg you enter micro-seconds, thus the extra 6 zeros to go from micro to second.

That said I am not sure you would want to exit those players if the stream broke down! Why? Simply because asterisk/FreePBX would immediately call the same command line again mindlessly. If that happened your DNS look ups will go through the roof and CPU utilization jump to saturation.

The smart way to address this issue is to take the control away from asterisk/FreePBX.

You can do it by creating a bash script which we will call for instance "playstream" and make sure that it actually never exits!

In asterisk/FreePBX you would simply call playstream on the command line where you usually enter your ffmpeg or mpg123 line.
Inside the playstream script file you would run one or more MOH command lines like the following:

/opt/ffmpeg/bin/ffmpeg -f s16le -ac 1 -ar 8000 -vol 32 pipe:1 -timeout 10000000 -i http://ais-sa1.streamon.fm/7833_128k.aac;

/usr/bin/mpg123 --timeout 10 -q -s --mono -r 8000 -f 8192 -b 0 http://stream.live.vc.bbcmedia.co.uk/bbc_world_service;

(The first one streams WHUD the second is for BBC)

So, if the first one ever interrupted for more than 10 seconds, the script would go to the next line and that would run until it stopped streaming for more than 10 seconds.

Of course you should have enough sources and the proper logic so that this script file actually never exits!
You can use loops, counters, random calls, etc so that you get a constant flow of feeds.

You can also use the case statement so that you can pass to the script file a radio channel parameter, like the call letters WHUD, KABL, WINS, WCBS and connect to the proper ffmpeg or mpg123 instruction inside the script file. Then again, you would supplement the command in such a way that if that stream is interrupted the script starts another (maybe) random stream or even a static music file to prevent the script from exiting.

So if you can do basic scripting the sky is the limit.

Pretty well nailed, but what better way to decouple any problems and simplify Asterisk's musiconhold by adding a class with a mode 'custom' where the application is simply "/usr/bin/socat udp-recv:{PORT} -" which will just wait for UDP streams on PORT and so play them (by default) as 'sln' , it will never need to 'restart' or 'reread' the .conf file.

Then you can pipe any of your solutions that result in good old sln through

socat -T 10 STDIO  UDP_DATAGRAM:localhost:[PORT]

whereby any lack of activity on STDIN for 10 seconds will just 'return' with an error so you will know to move on.

As a personal preference I go for sox being as it's the simplest way to get to single channel 8k rate 16 wide signed raw SLN fomat from any arbitrary URL but all these also work. You won't need to worry about timeouts in any one as the later pipe will handle that

ffmpeg
mpg123
gstreamer
curl
vlc
winamp?