Build a Voice AI Assistant with OpenAI Whisper & Python

Voice interfaces are transforming how humans interact with operating systems, developer terminals, and customer support channels. Building a voice assistant previously required stitch-together proprietary speech-to-text cloud services that suffered from high latency, privacy leaks, and poor accent recognition.

OpenAI Whisper is an open-source, multilingual automatic speech recognition (ASR) neural network trained on over 680,000 hours of diverse audio. By pairing Whisper (for local speech transcription) with a fast local LLM (like Llama 3.1) and a lightweight text-to-speech synthesizer, you can build a 100% offline, privacy-first desktop voice assistant.

Voice AI Pipeline Stages and Latency Profile

Prerequisites & Installation

Whisper requires FFmpeg installed in your system PATH to decode and process incoming audio streams.

# 1. Install system audio library (Ubuntu/Debian):
sudo apt install ffmpeg portaudio19-dev

# 2. Install Python packages:
pip install openai-whisper sounddevice numpy scipy pyttsx3

Pipeline Stage	Component / Engine	Audio / Data Format	Typical Execution Latency
1. Audio Capture	Sounddevice / PyAudio	16kHz Mono PCM Audio Stream	Real-time stream buffer
2. Speech-to-Text (STT)	OpenAI Whisper (Base / Small)	Mel-Spectrogram Spectrograms	150ms - 400ms (GPU accelerated)
3. Reasoning Engine	Local LLM via Ollama / vLLM	Text Prompt -> Response Token Stream	200ms - 600ms (Streaming tokens)
4. Text-to-Speech (TTS)	pyttsx3 / Piper TTS	Synthesized Waveform Audio	100ms - 250ms (Zero-latency playback)

Building the Local Audio Processing Engine

The voice assistant pipeline operates as a continuous finite state machine: capturing microphone audio when voice activity is detected, saving the buffer, transcribing audio to text via Whisper, and triggering speech synthesis.

Using Whisper's base or small model provides an optimal trade-off between word error rate (WER) and sub-second transcription latency on consumer hardware.

Complete Offline Voice AI Assistant Script in Python

import whisper
import pyttsx3
import sounddevice as sd
import numpy as np
import scipy.io.wavfile as wav
import tempfile
import os

# 1. Initialize Whisper Model and TTS Engine
print("Loading Whisper ASR model...")
asr_model = whisper.load_model("base")

tts_engine = pyttsx3.init()
tts_engine.setProperty("rate", 175) # Natural speaking rate

def speak(text: str):
    """Synthesize text to audio output."""
    print(f"Assistant: {text}")
    tts_engine.say(text)
    tts_engine.runAndWait()

def record_audio(duration_seconds: int = 5, sample_rate: int = 16000) -> str:
    """Record microphone audio to a temporary WAV file."""
    print(f"\n[Listening] Speak now ({duration_seconds}s)...")
    audio_data = sd.rec(int(duration_seconds * sample_rate), samplerate=sample_rate, channels=1, dtype="int16")
    sd.wait()
    
    with tempfile.NamedTemporaryFile(suffix=".wav", delete=False) as tmp_file:
        wav.write(tmp_file.name, sample_rate, audio_data)
        return tmp_file.name

def transcribe_audio_file(audio_path: str) -> str:
    """Transcribe WAV audio file using local Whisper neural network."""
    result = asr_model.transcribe(audio_path, fp16=False)
    os.remove(audio_path)
    return result["text"].strip()

def process_command(user_text: str):
    """Basic intent processing logic."""
    print(f"User Input: '{user_text}'")
    if "time" in user_text.lower():
        import datetime
        current_time = datetime.datetime.now().strftime("%I:%M %p")
        speak(f"The current time is {current_time}.")
    elif "status" in user_text.lower():
        speak("All systems are operational and running locally.")
    else:
        speak(f"I received your command: {user_text}")

if __name__ == "__main__":
    speak("iLab Academy voice assistant initialized and ready.")
    audio_file = record_audio(duration_seconds=4)
    command = transcribe_audio_file(audio_file)
    if command:
        process_command(command)
    else:
        speak("No speech detected.")

Audio Optimization and Production Hardening

• Voice Activity Detection (VAD): Rather than recording fixed duration windows, integrate Silero VAD to dynamically start recording when speech begins and stop automatically after 800ms of silence.
• Sampling Rate Compliance: Whisper expects single-channel 16,000Hz (16kHz) audio. Recording at other sample rates without resampling will cause hallucinated gibberish transcriptions.
• FP16 vs FP32 Inference: If running on CPU, always specify fp16=False in transcribe() to prevent PyTorch half-precision CPU warnings.

Frequently Asked Questions

Q: Can Whisper transcribe audio in languages other than English?
A: Yes. Whisper is natively trained on 99 languages and automatically detects the spoken language from audio waveform features.

Q: How can I make the assistant respond faster?
A: Use Whisper's tiny or base models along with Piper TTS (a neural TTS engine written in C++) for lightning-fast voice responses.

Related Guides and Tutorials on iLab Academy

• How to install Python on Windows
• How to Get IP Address and Geolocation Information Using Python
• 10 VS Code Extensions that make your coding Better
• 11 Practical AI Tools for Coders That Actually Save Time