Module 3: Image, audio and video models · 8 min
Multimodal, audio and video AI
A multimodal model can handle more than one kind of data, such as text plus images plus sound. It can read a photo of a handwritten note, describe a chart, or answer a spoken question.
- Speech to text: turns voice into written words (subtitles, meeting notes).
- Text to speech: reads text aloud in a natural voice.
- Video models: extend the diffusion idea across many frames so that objects stay consistent in time.
- Music models: generate melodies and songs from a text description.
Video is much harder than images because every frame must agree with the next one. Expect mistakes such as extra fingers, bending objects and unreadable text.
← How image models work: diffusion | What image and video AI cannot do well →