Voice & multimodal interfaces
Talking to AI out loud and showing it images or video, through phones, cars, speakers and glasses.
Read first: Tokens & tokenization.
D1 · StoryFor young children and total beginners
You can talk to some AIs out loud, and they talk back.
You can also show them a photo, or point a camera at something, and ask what it is.
Like: A phone call that can switch to video when you need to show something.
D2 · PictureFor curious beginners
The older design is a pipeline: speech-to-text turns your voice into words, a text model replies, and text-to-speech reads the reply aloud. Each step adds delay, and tone of voice is lost.
Newer speech-to-speech models take audio in and produce audio out directly, so they can reply faster, cope with interruptions and keep emphasis and emotion.
Images and video work in a similar way: a vision-capable model turns pictures into tokens alongside the text. Voice and cameras make AI usable hands-free, in cars, on glasses and for people who find typing hard.
Like: A phone call that can switch to video.
Where it breaks: A friend on video watches continuously and remembers what they saw. Many assistants look at a single photo or a few frames, and can confidently describe things that are not there.
D5 · FrontierFor experts
Open problems: natural turn-taking and interruptions, the privacy of always-listening microphones and cameras, voice cloning used for fraud, and accents and languages that speech recognisers handle less well.