JustujuAI, layer by layer.Open in the interactive map

Voice & multimodal interfaces

Talking to AI out loud and showing it images or video, through phones, cars, speakers and glasses.

Layer 9 · Applications & Interfaces · Run time · reviewed 2026-10-30

Read first: Tokens & tokenization.

D1 · StoryFor young children and total beginners

Predict: why does a slow reply feel worse when speaking than typing?

You can talk to some AIs out loud, and they talk back.

You can also show them a photo, or point a camera at something, and ask what it is.

Talking and showing, not just typing

  1. You talk or show: voice, a photo, a camera
  2. The AI understands: what it hears and sees
  3. It replies: out loud or in words

Like: A phone call that can switch to video when you need to show something.

D2 · PictureFor curious beginners

The older design is a pipeline: speech-to-text turns your voice into words, a text model replies, and text-to-speech reads the reply aloud. Each step adds delay, and tone of voice is lost.

Newer speech-to-speech models take audio in and produce audio out directly, so they can reply faster, cope with interruptions and keep emphasis and emotion.

Images and video work in a similar way: a vision-capable model turns pictures into tokens alongside the text. Voice and cameras make AI usable hands-free, in cars, on glasses and for people who find typing hard.

Like: A phone call that can switch to video.
Where it breaks: A friend on video watches continuously and remembers what they saw. Many assistants look at a single photo or a few frames, and can confidently describe things that are not there.

D3 · PlayFor motivated learners

In conversation people keep the silence between turns short. A study of ten languages found every language's average gap within a quarter of a second of the overall average. A voice assistant has to fit several steps into that silence.

In a pipeline the steps add up: wait to be sure you stopped talking, send the audio, turn it into text, let the model start, write the reply and turn it back into speech. Streaming starts speaking after the first sentence instead of the whole reply. A speech-to-speech model skips the two conversion steps.

Try the interactive version in the map.

Things to try

D5 · FrontierFor experts

Voice feels like the most natural interface, but speech adds timing, identity and accuracy problems that text never had.

Turn-taking. Pipelines of separate speech and text models add seconds of delay and ignore overlaps and interruptions; a full-duplex model that listens while it speaks reached about 200 ms (Moshi, 2024).

Uneven recognition. Five commercial recognisers averaged a word error rate of 0.35 for Black speakers against 0.19 for white speakers (Koenecke et al., 2020).

Made-up transcripts. About 1% of one widely used recogniser's transcripts contained invented phrases, 38% of them harmful (Koenecke et al., 2024).

Voice cloning. A short clip of a relative's voice is enough for scammers to fake an emergency call (FTC, 2023), and always-on microphones and cameras raise privacy questions of their own.

Examples as of 2026-10-30

Sources

  1. Robust Speech Recognition via Large-Scale Weak Supervision (Whisper) (2022)
  2. Moshi: a speech-text foundation model for real-time dialogue (2024)
  3. GPT-4o System Card (2024)
  4. Universals and cultural variation in turn-taking in conversation (Stivers et al., PNAS) (2009)
  5. Racial disparities in automated speech recognition (Koenecke et al., PNAS) (2020-03-23)
  6. Careless Whisper: Speech-to-Text Hallucination Harms (2024-02-12)
  7. Scammers use AI to enhance their family emergency schemes (US Federal Trade Commission) (2023-03-20)

How it connects

More in Layer 9 · Applications & Interfaces