TIPS Multilingual Voicemail Speech to Text

[Richo]

I think this will be a case of setting up a PBX for each language and forwarding calls but is it possible to setup "fwconsole voicemail" per extension? I assume not as it is in global setting (I am assuming this is what calls the sendmailmp3 file). The outcome I am looking for is having multiple inbound voicemail extensions, one for each language and then run separate voice to text via Watson. I can modify sendmailmp3 per language but it looks like it accepts a file stream so it cannot tell what number and hence language the data stream is coming from. Since I cannot do extension recognition in the sendmailmp3 file, I would have to do it in the PBX but it is set at a global level.
I assume that I will have to make the numbers redirect to separate servers and then I can modify the sendmailmp3 on each server. Overkill but the only way I can see to achieve this.

Any suggestions?

 

[kenn10]

I don't see how without custom programming. You can set a language per voice mailbox in the voicemail settings but the sendmailmp3 is system-wide. If you could figure out how to extract the voice mailbox language from the /etc/asterisk/voicemail.conf or the database and then act on it within the sendmailmp3, you might pull it off.

 

[Richo]

So I have been digging a bit further and the stream sent to sendmailmp3 is the whole email. If I had the time, inclination and skill, I could extract the line extension from the email as long as I had a specific format for the text in the email. I have instead decided to just duplicate the conversion section in the sendmailmp3 to add all translations to the one email. Then the end user can just scroll down to the translation that is correct for that language. It will of course have spurious data trying to convert one language in to five but it is the easiest work around I can see. Very clunky but it will have to do.

 

[Richo]

I went down the rabbit hole and got it all working by modifying the script. Based on the fact that I made heavy modifications to the original script, I would suggest you do not use this in production, but at least it will give someone else an idea on how to achieve the same thing.

It strips the input stream apart to find the extension (based on a specific entry in Asterisk) and does conversion based on which language is set for that extension. I have also included translation of the text on top of the transcription, using the IBM Watson service as well, with English as the default output language.

I could not see an option to post code so I have attached the file.

Enjoy, but don't blame me

 

[ostridge]

I could not see an option to post code so I have attached the file

Its easy to use \[ code=php] etc
look for the 3dots (more) icon above the posting window and choose the '</>' icon.

[/code]

 

[wardmundy]

@Richo: Attachment was missing. Be sure to tar or zip it up before uploading. Thanks.

 

[kenn10]

@billsimon did you pull your post about Azure transcription? I was reading it and went back and it was gone. It appears that Azure text to speech is a lot easier to set up than IBM Watson.

 

[Richo]

No problem Ward, I will use ostridge's method. Applogies that it is not well documented.

#! /bin/sh
############## start modify ##############
# Transcription
TRANSC_API_KEY=""
TRANSC_URL=""
# Translation
TRANSL_API_KEY=""
TRANSL_URL=""
# A line in the email configuration within "Settings", "Voicemail", "Email Config" must contain (without the external double quotes): "New voicemail message for '${VM_NAME}' (${VM_MAILBOX})"
EXTSTRING="New voicemail message for"
DE=802
JP=803
FR=804
NL=805
############### end modify ###############
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
pushd .
TMPDIR=$(mktemp -d)
cd $TMPDIR
cat >> stream.org
BOUNDARY=`grep "boundary=" stream.org | cut -d'"' -f 2`
awk '/'$BOUNDARY'/{i++}{print > "stream.part"i}' stream.org
PLAINTEXT=`cat stream.part1 | grep 'plain'`
if [ "$PLAINTEXT" != "" ]
then
  cat stream.org > stream.new
else
  sed '7,$d' stream.part3 > stream.part3.wav.head
  sed '1,6d' stream.part3 > stream.part3.wav.base64
  dos2unix -o stream.part3.wav.base64
  base64 -di stream.part3.wav.base64 > stream.part3.wav
  lame --abr 24 -mm -h -c --resample 22.050 stream.part3.wav stream.part3.mp3
  base64 stream.part3.mp3 > stream.part3.mp3.base64
  sed 's/x-wav/mpeg/g' stream.part3.wav.head | sed 's/.wav/.mp3/g' > stream.part3.mp3.head
  mv stream.part stream.new
  cat stream.part1 >> stream.new
  cat stream.part2 >> stream.new
  EXT=`cat stream.part2 | grep "$EXTSTRING" | sed 's/^.*(//' | sed 's/)//'`
    echo "Extension: $EXT"
  case "$EXT" in
    "$DE")
      curl -s -k -u "apikey:$TRANSC_API_KEY" -X POST \
        --limit-rate 40000 \
        --header "Content-Type: audio/wav" \
        --data-binary @stream.part3.wav \
        "$TRANSC_URL/v1/recognize?model=de-DE_NarrowbandModel" 1>audio.txt
      TRANSCRIPT=`cat audio.txt | grep transcript | sed 's#^.*"transcript": "##g' | sed 's# "##g'`
      TRANSCONF=`cat audio.txt | grep confidence | sed 's#^.*"confidence": ##g'`
            #TRANSC_TRANSL=`cat audio.txt`
      LANGUAGE="German Transcription: "
            DATA='{"text": ["'$TRANSCRIPT'"], "model_id":"de-en"}'
      curl -X POST -s -k -u "apikey:$TRANSL_API_KEY" \
        --header "Content-Type: application/json" \
        --data "$DATA" \
        "$TRANSL_URL/v3/translate?version=2018-05-01" 1>translation.txt
            TRANSLATION=`grep "\"translation\" :" translation.txt | sed 's/^.*"translation" : "//' | sed 's/"//'`
            echo "English Translation: " >> stream.new
            echo "$TRANSLATION" >> stream.new
            echo "" >> stream.new
            echo "Transcription Confidence: $TRANSCONF" >> stream.new
            echo "$LANGUAGE" >> stream.new
            echo "$TRANSCRIPT" >> stream.new
            echo "German"
    ;;
    "$JP")
      curl -s -k -u "apikey:$TRANSC_API_KEY" -X POST \
        --limit-rate 40000 \
        --header "Content-Type: audio/wav" \
        --data-binary @stream.part3.wav \
        "$TRANSC_URL/v1/recognize?model=ja-JP_NarrowbandModel" 1>audio.txt
      TRANSCRIPT=`cat audio.txt | grep transcript | sed 's#^.*"transcript": "##g' | sed 's# "##g'`
      TRANSCONF=`cat audio.txt | grep confidence | sed 's#^.*"confidence": ##g'`
            DATA='{"text": ["'$TRANSCRIPT'"], "model_id":"jp-en"}'
      curl -X POST -s -k -u "apikey:$TRANSL_API_KEY" \
        --header "Content-Type: application/json" \
        --data "$DATA" \
        "$TRANSL_URL/v3/translate?version=2018-05-01" 1>translation.txt
            TRANSLATION=`grep "\"translation\" :" translation.txt | sed 's/^.*"translation" : "//' | sed 's/"//'`
            echo "English Translation: " >> stream.new
            echo "$TRANSLATION" >> stream.new
            echo "" >> stream.new
            echo "Transcription Confidence: $TRANSCONF" >> stream.new
            echo "Japanese Transcription: " >> stream.new
            echo "$TRANSCRIPT" >> stream.new
            echo "Japanese"
    ;;
    "$FR")
      curl -s -k -u "apikey:$TRANSC_API_KEY" -X POST \
        --limit-rate 40000 \
        --header "Content-Type: audio/wav" \
        --data-binary @stream.part3.wav \
        "$TRANSC_URL/v1/recognize?model=fr-FR_NarrowbandModel" 1>audio.txt
      TRANSCRIPT=`cat audio.txt | grep transcript | sed 's#^.*"transcript": "##g' | sed 's# "##g'`
      TRANSCONF=`cat audio.txt | grep confidence | sed 's#^.*"confidence": ##g'`
            DATA='{"text": ["'$TRANSCRIPT'"], "model_id":"fr-en"}'
      curl -X POST -s -k -u "apikey:$TRANSL_API_KEY" \
        --header "Content-Type: application/json" \
        --data "$DATA" \
        "$TRANSL_URL/v3/translate?version=2018-05-01" 1>translation.txt
            TRANSLATION=`grep "\"translation\" :" translation.txt | sed 's/^.*"translation" : "//' | sed 's/"//'`
            echo "English Translation: " >> stream.new
            echo "$TRANSLATION" >> stream.new
            echo "" >> stream.new
            echo "Transcription Confidence: $TRANSCONF" >> stream.new
            echo "French Transcription: " >> stream.new
            echo "$TRANSCRIPT" >> stream.new
            echo "French"
    ;;
    "$NL")
      curl -s -k -u "apikey:$TRANSC_API_KEY" -X POST \
        --limit-rate 40000 \
        --header "Content-Type: audio/wav" \
        --data-binary @stream.part3.wav \
        "$TRANSC_URL/v1/recognize?model=nl-NL_NarrowbandModel" 1>audio.txt
      TRANSCRIPT=`cat audio.txt | grep transcript | sed 's#^.*"transcript": "##g' | sed 's# "##g'`
      TRANSCONF=`cat audio.txt | grep confidence | sed 's#^.*"confidence": ##g'`
            DATA='{"text": ["'$TRANSCRIPT'"], "model_id":"nl-en"}'
      curl -X POST -s -k -u "apikey:$TRANSL_API_KEY" \
        --header "Content-Type: application/json" \
        --data "$DATA" \
        "$TRANSL_URL/v3/translate?version=2018-05-01" 1>translation.txt
            TRANSLATION=`grep "\"translation\" :" translation.txt | sed 's/^.*"translation" : "//' | sed 's/"//'`
            echo "English Translation: " >> stream.new
            echo "$TRANSLATION" >> stream.new
            echo "" >> stream.new
            echo "Transcription Confidence: $TRANSCONF" >> stream.new
            echo "Dutch Transcription: " >> stream.new
            echo "$TRANSCRIPT" >> stream.new
            echo "Dutch"
    ;;
    *)
      curl -s -k -u "apikey:$TRANSC_API_KEY" -X POST \
        --limit-rate 40000 \
        --header "Content-Type: audio/wav" \
        --data-binary @stream.part3.wav \
        "$TRANSC_URL/v1/recognize?model=en-US_NarrowbandModel" 1>audio.txt
      TRANSCRIPT=`cat audio.txt | grep transcript | sed 's#^.*"transcript": "##g' | sed 's# "##g'`
      TRANSCONF=`cat audio.txt | grep confidence | sed 's#^.*"confidence": ##g'`
      LANGUAGE="English Transcription: "
            echo "Transcription Confidence: $TRANSCONF" >> stream.new
            echo "$LANGUAGE" >> stream.new
            echo "$TRANSCRIPT" >> stream.new
            echo "English - Catch All"
    ;;
  esac
  cat stream.part3.mp3.head >> stream.new
  dos2unix -o stream.new
  unix2dos -o stream.part3.mp3.base64
  cat stream.part3.mp3.base64 >> stream.new
  echo "" >> stream.tmp
  echo "" >> stream.tmp
  cat stream.part4 >> stream.tmp
  dos2unix -o stream.tmp
  cat stream.tmp >> stream.new
fi

cat stream.new | sendmail -t
popd
rm -Rf $TMPDIR

 

[billsimon]

@billsimon did you pull your post about Azure transcription? I was reading it and went back and it was gone. It appears that Azure text to speech is a lot easier to set up than IBM Watson.

Yes, it's a breeze to set up. I pulled it because my script does not address OP's issue. Azure can handle language detection BUT my script does not. I don't have time right now to change it to use the version of API where it will auto detect the language -- if you use mine, it's English.

Anyway, for the record, Azure transcription: https://github.com/simontelephonics/transcribe-with-azure but note, more work needed if you want it to do the language auto detection.

 

[kenn10]

@billsimon Thanks!