Nexus Learn

Log in · Create free account

Module 3: Image, audio and video models · 8 min

Multimodal, audio and video AI

A multimodal model can handle more than one kind of data, such as text plus images plus sound. It can read a photo of a handwritten note, describe a chart, or answer a spoken question.

  • Speech to text: turns voice into written words (subtitles, meeting notes).
  • Text to speech: reads text aloud in a natural voice.
  • Video models: extend the diffusion idea across many frames so that objects stay consistent in time.
  • Music models: generate melodies and songs from a text description.

Video is much harder than images because every frame must agree with the next one. Expect mistakes such as extra fingers, bending objects and unreadable text.

(free account needed to save progress)

← How image models work: diffusion  |  What image and video AI cannot do well →