High latency with real-time STT models

[abdelmasieh]

I'm creating a dialplan in Asterisk which is like an AI Voice Assistant. The steps are as follow:

1. Get audio input from the caller using real-time websocket STT.
2. Give the output of the STT to an NLP model that generates a text response.
3. Get the output from the NLP back to a TTS model and then play the audio with the response.

The issue is that we get high latency between when the caller stops speaking to the moment the AI replies back.

What I tried so far:

• Record the audio of the user and then it to whisper API and get the response back to the NLP then the TTS, but it resulted in about 8 seconds or more of silence.
• Use a library whisper-streaming to get the output of whisper in real-time and redo the same steps which results in even more latency as it would send multiple requests to the server to get the response.
• Use Vosk, an STT real-time solution which resulted in about 6 to 10 seconds of latency.

Currently, I'm looking for a library to also make the TTS model also real-time, but the latency issue is mostly on the STT side, can anyone suggest solutions to enhance the latency of this dialplan?

 

[dicko]

vosk is not AI, you get to pay a premium for premium AI STT, too soon for recommendations yet I think, but it will come very quickly , someone needs to throw a lot of 'computes' at it to get the latency down and so make a buck ;-)

 

[Halea]

Between the time the caller stops speaking and (s)he gets a reply from your system three things happen:
1) His/her speech is converted to text
2) The text is submitted to the Natural Language Processing engine (I'm guessing that's what you call NLP) and an output gets issued in text mode.
3) The output from NLP gets converted to speech and played back to the caller (in one step but it could also be two steps depending on your tools)

You have to be careful with your terminology. You used the word "latency" all over but you in fact have 2 components to the amount of time that it takes to achieve each step from the above list.

Indeed there is the "processing" time of the action, and there is the "latency", which is the additional delay induced if that action is executed remotely on a different system. The latency is typically related to various data transport timings that you can try to minimize but never completely eliminate unless you run all your actions on local machines.

Thus before you find a fix to your problem you need to understand where the delay is coming from, and whether it's reasonable or not. For instance if 90% of the delay is coming from the step 2 and the bulk of it is because of the actual latency then your only alternative is to run the NLP locally. If the NLP engine actually takes most of the time to come up with an answer to the caller's request then there isn't much you can do.

You mentioned that "the latency issue is mostly on the STT side". If that's indeed the case does it represent the bulk of the "latency between when the caller stops speaking to the moment the AI replies back" ?

The application that you are trying to build is not a trivial one. All three steps require substantial processing power. Running those steps on your own local hardware might not be possible. Getting them executed remotely makes life more difficult when it comes to timing issues. You may pay top dollars to your process vendors to get things run faster at their end and use fat pipes to get the data go back and forth with minimal latency.

The reason for which I am taking time to get into the nitty gritty of your seemingly simple project is that some time ago I got into a similar one.

My application used the same three steps mentioned at the top. My NLP was a language translator.
We attempted to build a two way system so that when two people spoke to each other over the phone the "system" translated the languages spoken by the two parties on the fly.
Well, it actually turned out to be wishful thinking. Our actual one way translation delay turned out to be 12 seconds. In some instanced that was acceptable, in others it was not.

My point is, you need to further dig into your timings, make them much more granular, understand clearly what's processing time what's actual latency. Then you need to figure out what kind of response delay is acceptable. In the end your project might not be viable, at least not at this time with free or inexpensive tools.

We, for instance, were able to compute the cost of going from 12 seconds of delay to 6 seconds. It was substantial, therefore we decide to put the project in the back burner for now.

 

[wardmundy]

We've actually had this running with ChatGPT for the past six months. We use a dedicated DID locked down to a CallerID Number in Inbound Routes that points to the 2428 code in extensions_custom.conf:

exten => 2428,1,Answer()
exten => 2428,n,Wait(1)
exten => 2428,n,System(rm -f /tmp/*.sln)
;exten => 2428,n,AGI(picotts.agi,"After the beep, Say your query, then Press the pound key.",en-US)
exten => 2428,n,Playback(custom/say-chatgtp-query)
exten => 2428,n,Set(RANDFILE=${RAND(8000,8899)})
exten => 2428,n,Record(/tmp/${RANDFILE}.wav,3,60)
;exten => 2428,n,AGI(picotts.agi,"I think you said:",en-US)
exten => 2428,n,Playback(custom/i-think-you-said)
exten => 2428,n,Playback(/tmp/${RANDFILE})
;exten => 2428,n,AGI(picotts.agi,"If this is correct, press one, after the beep.",en-US)
exten => 2428,n,Playback(custom/satisfied)
exten => 2428,n,Read(MYCHOICE,beep,1)
exten => 2428,n,GotoIf($["foo${MYCHOICE}" = "foo1"]?chatproceed:chatdone)
;exten => 2428,n(chatproceed),AGI(picotts.agi,"One moment please while we contact Chat G P T.",en-US)
exten => 2428,n(chatproceed),Playback(custom/stay-on-line)
exten => 2428,n,AGI(chatgpt.sh,${RANDFILE})
exten => 2428,n,Set(QUERY=${FILE(/tmp/${RANDFILE}.txt)})
exten => 2428,n,Set(ChatGPT=${FILE(/tmp/results4u.txt)})
exten => 2428,n,Playback(/tmp/results4u)
;exten => 2428,n(chatdone),AGI(picotts.agi,"Have a nice day. Good bye.",en-US)
exten => 2428,n(chatdone),Playback(custom/googbye-from-chatgpt)
;exten => 2428,n,System(rm -f /tmp/${RANDFILE}.wav)
;exten => 2428,n,System(rm -f /tmp/${RANDFILE}.txt)
exten => 2428,n,hangup

The processing is actually handled by the chargpt.sh AGI script:

#!/bin/bash

# chatgpt.sh v.1.0 (c) Copyright 2017-2024, Ward Mundy & Associates LLC. All rights reserved.

# Consume all variables sent by Asterisk
while read VAR && [ -n ${VAR} ] ; do : ; done

# the script expects a random file name to use for STT conversation with IBM Watson
thisfile="$1"
lang= "en"
# please insert your OpenAI Key below
OPENAI_KEY="sk-sPCProB85H**YOUR OPENAI KEY GOES HERE**M5xCXWT"

# please insert your IBM Watson STT credentials below
API_USERNAME="apikey"
API_KEY="nK850-aW6vV5**YOUR WATSON API KEY GOES HERE**uGB1uE"
API_URL="https://api.us-south.speech-to-text.watson.cloud.ibm.com/instances/23790771-73e6-478f-b2cb-c05d0fe07f4e"
# make no changes below here

rm /tmp/*.txt
rm /tmp/results4u.*

curl -k -u $API_USERNAME:$API_KEY -X POST --limit-rate 80000 --header "Content-Type: audio/wav" --data-binary @/tmp/$thisfile.wav $API_URL"/v1/recognize?continuous=true&model=en-US_NarrowbandModel" 1>/tmp/$thisfile.txt

QUERY=`cat /tmp/$thisfile.txt | grep transcript | cut -f 2 -d ":" | cut -f 2 -d '"' | sed 's| *$||' | sed -e "s/\b\(.\)/\u\1/g"`

echo $QUERY > /tmp/$thisfile.txt

/usr/bin/sed -i 's|%HESITATION||g' /tmp/$thisfile.txt

size="$(wc -c <"/tmp/$thisfile.txt")"

if [ "$size" -lt "5" ]; then
 exit
fi

export PATH=$PATH:/usr/local/bin:/usr/bin:/usr/local/sbin

/usr/local/sbin/chat "$(/usr/bin/cat /tmp/$thisfile.txt)" > /tmp/results4u.txt
/usr/bin/sed -i 's|,|:|g' /tmp/results4u.txt
/usr/bin/sed -i 's|\\n\\n| Option |g' /tmp/results4u.txt
/usr/local/bin/gtts-cli -l "en" -f /tmp/results4u.txt  | /usr/bin/sox -t mp3 - -r 8000 /tmp/results4u.wav

rm /tmp/8*.wav
exit 0

Finally, you'll need a few sound files to talk to the caller during ChatGPT sessions (attached).

See the Nerd Vittles tutorial for more info.

 

[KNERD]

If you are running locally, to reduce latency, you will need to use an nVidia GPU instead of CPU.

 

[abdelmasieh]

vosk is not AI, you get to pay a premium for premium AI STT, too soon for recommendations yet I think, but it will come very quickly , someone needs to throw a lot of 'computes' at it to get the latency down and so make a buck ;-)

As of now we are testing before we get into production, so we are trying not to burn a lot of money in the process.

 

[abdelmasieh]

Between the time the caller stops speaking and (s)he gets a reply from your system three things happen:
1) His/her speech is converted to text
2) The text is submitted to the Natural Language Processing engine (I'm guessing that's what you call NLP) and an output gets issued in text mode.
3) The output from NLP gets converted to speech and played back to the caller (in one step but it could also be two steps depending on your tools)

You have to be careful with your terminology. You used the word "latency" all over but you in fact have 2 components to the amount of time that it takes to achieve each step from the above list.

Indeed there is the "processing" time of the action, and there is the "latency", which is the additional delay induced if that action is executed remotely on a different system. The latency is typically related to various data transport timings that you can try to minimize but never completely eliminate unless you run all your actions on local machines.

Thus before you find a fix to your problem you need to understand where the delay is coming from, and whether it's reasonable or not. For instance if 90% of the delay is coming from the step 2 and the bulk of it is because of the actual latency then your only alternative is to run the NLP locally. If the NLP engine actually takes most of the time to come up with an answer to the caller's request then there isn't much you can do.

You mentioned that "the latency issue is mostly on the STT side". If that's indeed the case does it represent the bulk of the "latency between when the caller stops speaking to the moment the AI replies back" ?

The application that you are trying to build is not a trivial one. All three steps require substantial processing power. Running those steps on your own local hardware might not be possible. Getting them executed remotely makes life more difficult when it comes to timing issues. You may pay top dollars to your process vendors to get things run faster at their end and use fat pipes to get the data go back and forth with minimal latency.

The reason for which I am taking time to get into the nitty gritty of your seemingly simple project is that some time ago I got into a similar one.

My application used the same three steps mentioned at the top. My NLP was a language translator.
We attempted to build a two way system so that when two people spoke to each other over the phone the "system" translated the languages spoken by the two parties on the fly.
Well, it actually turned out to be wishful thinking. Our actual one way translation delay turned out to be 12 seconds. In some instanced that was acceptable, in others it was not.

My point is, you need to further dig into your timings, make them much more granular, understand clearly what's processing time what's actual latency. Then you need to figure out what kind of response delay is acceptable. In the end your project might not be viable, at least not at this time with free or inexpensive tools.

We, for instance, were able to compute the cost of going from 12 seconds of delay to 6 seconds. It was substantial, therefore we decide to put the project in the back burner for now.

We will be testing on a VM that has 8 CPUs and 8 GPUs with 56 GB of ram, I think this will be sufficient to see if we can get the processing delay to a minimum of the Vosk step. But in case of the other steps the NLP engine and the TTS, I just learned they take about 4 to 5 seconds together, so we will be trying to get the response in real-time as well to see if it will reduce the delay.
A delay between the caller's voice and the AI response of about 3 or 4 seconds is acceptable in our case.

 

[abdelmasieh]

We've actually had this running with ChatGPT for the past six months. We use a dedicated DID locked down to a CallerID Number in Inbound Routes that points to the 2428 code in extensions_custom.conf:

exten => 2428,1,Answer()
exten => 2428,n,Wait(1)
exten => 2428,n,System(rm -f /tmp/*.sln)
;exten => 2428,n,AGI(picotts.agi,"After the beep, Say your query, then Press the pound key.",en-US)
exten => 2428,n,Playback(custom/say-chatgtp-query)
exten => 2428,n,Set(RANDFILE=${RAND(8000,8899)})
exten => 2428,n,Record(/tmp/${RANDFILE}:wav,3,60)
;exten => 2428,n,AGI(picotts.agi,"I think you said:",en-US)
exten => 2428,n,Playback(custom/i-think-you-said)
exten => 2428,n,Playback(/tmp/${RANDFILE})
;exten => 2428,n,AGI(picotts.agi,"If this is correct, press one, after the beep.",en-US)
exten => 2428,n,Playback(custom/satisfied)
exten => 2428,n,Read(MYCHOICE,beep,1)
exten => 2428,n,GotoIf($["foo${MYCHOICE}" = "foo1"]?chatproceed:chatdone)
;exten => 2428,n(chatproceed),AGI(picotts.agi,"One moment please while we contact Chat G P T.",en-US)
exten => 2428,n(chatproceed),Playback(custom/stay-on-line)
exten => 2428,n,AGI(chatgpt.sh,${RANDFILE})
exten => 2428,n,Set(QUERY=${FILE(/tmp/${RANDFILE}.txt)})
exten => 2428,n,Set(ChatGPT=${FILE(/tmp/results4u.txt)})
exten => 2428,n,Playback(/tmp/results4u)
;exten => 2428,n(chatdone),AGI(picotts.agi,"Have a nice day. Good bye.",en-US)
exten => 2428,n(chatdone),Playback(custom/googbye-from-chatgpt)
;exten => 2428,n,System(rm -f /tmp/${RANDFILE}.wav)
;exten => 2428,n,System(rm -f /tmp/${RANDFILE}.txt)
exten => 2428,n,hangup

The processing is actually handled by the chargpt.sh AGI script:

#!/bin/bash

# chatgpt.sh v.1.0 (c) Copyright 2017-2024, Ward Mundy & Associates LLC. All rights reserved.

# Consume all variables sent by Asterisk
while read VAR && [ -n ${VAR} ] ; do : ; done

# the script expects a random file name to use for STT conversation with IBM Watson
thisfile="$1"
lang= "en"
# please insert your OpenAI Key below
OPENAI_KEY="sk-sPCProB85H**YOUR OPENAI KEY GOES HERE**M5xCXWT"

# please insert your IBM Watson STT credentials below
API_USERNAME="apikey"
API_KEY="nK850-aW6vV5**YOUR WATSON API KEY GOES HERE**uGB1uE"
API_URL="https://api.us-south.speech-to-text.watson.cloud.ibm.com/instances/23790771-73e6-478f-b2cb-c05d0fe07f4e"
# make no changes below here

rm /tmp/*.txt
rm /tmp/results4u.*

curl -k -u $API_USERNAME:$API_KEY -X POST --limit-rate 80000 --header "Content-Type: audio/wav" --data-binary @/tmp/$thisfile.wav $API_URL"/v1/recognize?continuous=true&model=en-US_NarrowbandModel" 1>/tmp/$thisfile.txt

QUERY=`cat /tmp/$thisfile.txt | grep transcript | cut -f 2 -d ":" | cut -f 2 -d '"' | sed 's| *$||' | sed -e "s/\b\(.\)/\u\1/g"`

echo $QUERY > /tmp/$thisfile.txt

/usr/bin/sed -i 's|%HESITATION||g' /tmp/$thisfile.txt

size="$(wc -c <"/tmp/$thisfile.txt")"

if [ "$size" -lt "5" ]; then
 exit
fi

export PATH=$PATH:/usr/local/bin:/usr/bin:/usr/local/sbin

/usr/local/sbin/chat "$(/usr/bin/cat /tmp/$thisfile.txt)" > /tmp/results4u.txt
/usr/bin/sed -i 's|,|:|g' /tmp/results4u.txt
/usr/bin/sed -i 's|\\n\\n| Option |g' /tmp/results4u.txt
/usr/local/bin/gtts-cli -l "en" -f /tmp/results4u.txt  | /usr/bin/sox -t mp3 - -r 8000 /tmp/results4u.wav

rm /tmp/8*.wav
exit 0

Finally, you'll need a few sound files to talk to the caller during ChatGPT sessions (attached).

See the Nerd Vittles tutorial for more info.

View attachment 5347

Could you please share specific details about the delay between user silence and AI response? Also server hardware specs if possible.

 

[Samot]

exten => 2428,n,Record(/tmp/${RANDFILE}:wav,3,60)

This is incorrect filename format. You're create /tmp/8003:wav instead of /tmp/8003.wav it is possible it's recording the file in wav format and playing it back..but if you try to move or delete that file later in another script or further in the dialplan like you have commented out here....it's going to fail.

;exten => 2428,n,System(rm -f /tmp/${RANDFILE}.wav)

 

[KNERD]

Could you please share specific details about the delay between user silence and AI response? Also server hardware specs if possible.

You cannot see it is using IBM and OpenAI servers to produce the results? They are running a massive server farm. Any delay you will see it be contingent on your distance to their servers, and the time it takes for them to process your request.

 

[Halea]

If you are running locally, to reduce latency, you will need to use an nVidia GPU instead of CPU.

The very best tools when it comes to text to speech, speech to text and natural language processing, including so called "AI banks" are all provided as a service (free tier and paid) rather than a software package that you can run locally on your own hardware (physical, VM, whatever). It has to do with the protection of the intellectual property of the developer, but also the required processing power to make things work properly.
Therefore the latency issue is absolutely inescapable.
Nonetheless during the countless hours of testing and retesting what we found to be the most major impediment was still the processing time at the NLP level.
This thread turned out to be quite useful to me. I now wonder if ChatGPT could do plain vanilla language translation for our app.
Anyone has tried to ask ChatGPT to translate something?

 

[kenn10]

ChatGPT translated your message above to German:

Die allerbesten Werkzeuge für Text-zu-Sprache, Sprache-zu-Text und natürliche Sprachverarbeitung, einschließlich sogenannter "KI-Banken", werden alle als Service angeboten (kostenlose Stufe und kostenpflichtig), anstatt als Softwarepaket, das Sie lokal auf Ihrer eigenen Hardware (physisch, VM, was auch immer) ausführen können. Das hat mit dem Schutz des geistigen Eigentums des Entwicklers zu tun, aber auch mit der erforderlichen Rechenleistung, um die Dinge ordnungsgemäß zum Laufen zu bringen. Daher ist das Latenzproblem absolut unvermeidlich. Trotzdem war während der unzähligen Stunden des Testens und Wiederholens das, was wir als größtes Hindernis empfanden, immer noch die Verarbeitungszeit auf der NLP-Ebene. Dieser Thread stellte sich für mich als ziemlich nützlich heraus. Ich frage mich jetzt, ob ChatGPT eine einfache Übersetzung für unsere App durchführen könnte. Hat jemand versucht, ChatGPT zu bitten, etwas zu übersetzen?

Google translate seems to approve of the translation as well and translates it back to English flawlessly.

 

[dicko]

As of now we are testing before we get into production, so we are trying not to burn a lot of money in the process.

For a narrow set oof grammars and a limited dictionary, I find that vosk is very fast and accurate and of course, free

 

[wardmundy]

This is incorrect filename format. You're create /tmp/8003:wav instead of /tmp/8003.wav it is possible it's recording the file in wav format and playing it back..but if you try to move or delete that file later in another script or further in the dialplan like you have commented out here....it's going to fail.

Colon used to be a separator in the old days. Digium "improved" things, and I was late to the party. Thanks. Fixed.

 

[Samot]

Colon used to be a separator in the old days. Digium "improved" things, and I was late to the party. Thanks. Fixed.

Actually, it used to be | as the delimiter. The colon was used in Record() as the marker for the extension of the filename. The change was made almost 20 years ago. The | as the delimiter was changed 15-16 years ago....

Prior to ca. 2004-11-01
Record(filename:format[|silence][|maxduration][|option])

After 2004-11-01 (cvs) and in Asterisk v1.2
Record(filename.format[|silence][|maxduration][|option])

 

[prynze]

Yeah, that Whisper setup sounds like a lot of waiting around for a live dialplan. What needs to be checked is whether you're sending partial results to NLP or waiting for the final transcript. That alone can help cut down the delay.

 

[KNERD]

I use Whisper cpp with a old circa 2016 12 core Xeon. Despite it having 24 threads, I have Whisper using 12 threads and it is able to to do the STT in real time.