Let's build a voice assistant
On Thursday I hosted a 60-minute session in the Liquid AI Discord Community. There I shared a few tips and tricks for adding audio features into your software using a small-yet-powerful Audio Language Model, LFM2-Audio-1.5B.
If you missed the session, you can watch it here:
Github repo with the code is HERE
The whole premise is that you can add ears and mouth to your software, without having to connect (and pay) for external cloud services.
The model is so small, that when you couple it with a powerful inference engine like llama.cpp, things go FAST and cheap.
What can we build with this model?
On Thursday I showed how to build a simple Audio to Speech transcription CLI. You provide a piece of audio, and the model transcribes it into text in real-time.
Not bad.
But the thing is, this model is way more than just an Audio-to-Text mapper. As a starter, it can go the other way around. You provide text and the model transforms it into audio.
Not bad.
But you know what? It can do more than that. It can actually process what you are saying, either in text or voice form, and generate a response either in text or voice.
In other words, it is a speech-to-speech model.
What’s under the hood?
LFM2-Audio-1.5B is a speech-to-speech model built on top of LFM2-1.2B, which is a powerful text-to-text language model.
This means that both text tokens or audio tokens are embedded in a common vector space. From there, LFM2-1.2B maps them into a text completion, that can be decoded as an audio too.
Let me show you with an example
Hands-on example → Building a local voice assistant
On Friday I decided to build a voice chat application that uses the LFM2-Audio-1.5B model to generate conversational audio responses.
Record your voice, send it for processing, and receive an audio response that plays automatically.
Attention
This application could work 100% locally, but the liquid-audio library for inference requires CUDA. This is why the model is wrapped inside a Modal function and deployed to a serverless GPU environment with CUDA, so you can run it even if you (like me) don’t have an NVIDIA GPU at home. As soon as the model gets a better support by llama.cpp, I will switch the inference to llama.cpp.
How does it work?
The application works by:
Recording your voice question from the microphone (with auto-stop on silence)
Uploading the audio to a Modal volume
Processing the audio with LFM2-Audio-1.5B on a GPU instance to generate an interleaved text and audio response
Downloading the generated audio response
Playing the response through your speakers
The model generates responses that can include both text and audio tokens, creating a natural conversational experience.
Environment setup
Install uv if you don’t have it:
curl -LsSf https://astral.sh/uv/install.sh | shSet up Modal:
Create a Modal account at modal.com
Install the Modal CLI:
uv add modalAuthenticate:
uv run modal token new
Ensure microphone permissions are granted to your terminal/IDE
How to run it?
Deploy the server (first time only, or when server code changes):
make deploy-serverRun the client:
make runThe application will start recording when you run it. Speak your question into the microphone and wait around 15 seconds. The generated audio response will be downloaded and played automatically
What’s next?
As I said, this is just a starter.
This voice assistant works, but it is very slow, due to container startup times on Modal, network latency and PyTorch poor inference speed.
But there is one way to solve this. Its name is llama.cpp.
Once we get better support with llama.cpp we will be able to run everything locally. Which means 0 network latency, 0 container startup time, and way faster forward pass than PyTorch.
The future is bright. Let’s get there together :-)
Pau