# Voice & multimodal interfaces

URL: https://justuju.org/concept/voice-multimodal/
Layer 9 · Applications & Interfaces

Talking to AI out loud and showing it images or video, through phones, cars, speakers and glasses.

## D1 · Story

You can talk to some AIs out loud, and they talk back.
You can also show them a photo, or point a camera at something, and ask what it is.
1. You talk or show: voice, a photo, a camera
2. The AI understands: what it hears and sees
3. It replies: out loud or in words
Like: A phone call that can switch to video when you need to show something.

## D2 · Picture

The older design is a pipeline: speech-to-text turns your voice into words, a text model replies, and text-to-speech reads the reply aloud. Each step adds delay, and tone of voice is lost.
Newer speech-to-speech models take audio in and produce audio out directly, so they can reply faster, cope with interruptions and keep emphasis and emotion.
Images and video work in a similar way: a vision-capable model turns pictures into tokens alongside the text. Voice and cameras make AI usable hands-free, in cars, on glasses and for people who find typing hard.
Like: A phone call that can switch to video. Where it breaks: A friend on video watches continuously and remembers what they saw. Many assistants look at a single photo or a few frames, and can confidently describe things that are not there.

## D5 · Frontier

Voice feels like the most natural interface, but speech adds timing, identity and accuracy problems that text never had.
Turn-taking. Pipelines of separate speech and text models add seconds of delay and ignore overlaps and interruptions; a full-duplex model that listens while it speaks reached about 200 ms (Moshi, 2024).
Uneven recognition. Five commercial recognisers averaged a word error rate of 0.35 for Black speakers against 0.19 for white speakers (Koenecke et al., 2020).
Made-up transcripts. About 1% of one widely used recogniser's transcripts contained invented phrases, 38% of them harmful (Koenecke et al., 2024).
Voice cloning. A short clip of a relative's voice is enough for scammers to fake an emergency call (FTC, 2023), and always-on microphones and cameras raise privacy questions of their own.

## Examples as of 2026-10-30

- ChatGPT voice mode
- Gemini Live
- OpenAI Realtime API
- ElevenLabs
- Whisper (speech recognition)

## Sources

- Robust Speech Recognition via Large-Scale Weak Supervision (Whisper): https://arxiv.org/abs/2212.04356
- Moshi: a speech-text foundation model for real-time dialogue: https://arxiv.org/abs/2410.00037
- GPT-4o System Card: https://arxiv.org/abs/2410.21276
- Universals and cultural variation in turn-taking in conversation (Stivers et al., PNAS): https://doi.org/10.1073/pnas.0903616106
- Racial disparities in automated speech recognition (Koenecke et al., PNAS): https://doi.org/10.1073/pnas.1915768117
- Careless Whisper: Speech-to-Text Hallucination Harms: https://arxiv.org/abs/2402.08021
- Scammers use AI to enhance their family emergency schemes (US Federal Trade Commission): https://consumer.ftc.gov/consumer-alerts/2023/03/scammers-use-ai-enhance-their-family-emergency-schemes


---
Justuju · https://justuju.org/ · Content CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote with a link to the page.