A traveler photographs a parking sign in a language she cannot read and asks her phone whether she can leave the car there until 6 pm. A support agent receives a blurry screenshot of an error dialog instead of a description. A shop owner talks to an assistant while pointing the camera at a label. All three are using the same kind of system: an AI model that works with pictures and sound as well as words.
This post explains what multimodal AI and a "modality" are, how these models turn images and audio into something a language model can use, what vision-language models do, where they fail and what to check before you upload a photo.
What is multimodal AI?
Multimodal AI is an AI model or system that works with more than one type of data, such as text, images, audio or video, as input, as output or both. A text-only chatbot is unimodal: words go in, words come out. A model that looks at a photo of your fridge and suggests a recipe, or hears a question and answers aloud, is multimodal.
The word itself just means "using several modes", which is why it also appears in transport (one shipment by truck, rail and sea), in medicine (a treatment that combines several methods) and in education, where multimodal learning means teaching through text, pictures and sound at once. This post stays with the AI meaning.
Many chat assistants now accept images and voice. Underneath, most are built around a large language model, a model trained on huge amounts of text to predict the next piece of text, with extra parts that let it handle other kinds of data.
What is a modality in AI?
A modality is a type of data, defined by how the information is captured or experienced: language, images, video, sound. A widely cited survey of multimodal machine learning by Baltrusaitis, Ahuja and Morency (2017) defines modality as "the way in which something happens or is experienced" and calls a problem multimodal when it involves several of them.
The survey focuses on three: natural language, written or spoken; visual signals such as images and video; and vocal signals, which carry sounds and the tone of a voice. Specialist fields add others, such as sensor readings or medical scans.
How does multimodal AI work?
A multimodal model turns every input into the same currency, vectors in a shared space, and lets one model reason over all of them. A typical model works in five steps:
- Send each input to a matching encoder. Text is split into tokens, small pieces of words; images go to a vision encoder and audio to an audio encoder, each a neural network trained for its kind of data.
- Cut the input into pieces. A vision transformer cuts an image into small square patches and treats them like words in a sentence. Audio is often turned into a spectrogram, a picture of how the sound's frequencies change over time, and read in short slices.
- Map the pieces into the language model's space. Each patch or slice becomes a vector, and a small connector network, sometimes a single layer, moves these vectors into the space the language model uses for word tokens.
- Reason over one combined sequence. The language model reads image vectors, audio vectors and text tokens as one sequence. Its attention mechanism, the part that weighs which pieces matter to each other, can link "the red one" in your question to the right patch of the photo.
- Produce the output. Usually that is text. Models that also speak or draw add decoders that turn output tokens back into sound or pixels.
The shared space is the key idea. OpenAI's CLIP model, described in the paper Learning Transferable Visual Models From Natural Language Supervision (2021), learned one through a simple task: predicting which caption goes with which image, over 400 million image-text pairs collected from the internet. Afterwards, a photo of a dog and the words "a photo of a dog" sit close together as vectors, and later models reuse CLIP's image encoder as their eyes.
What is the difference between early and late fusion?
Fusion is the step where information from different modalities is combined, and the difference is when it happens. The survey above describes the classic split:
- Early fusion joins the features of each modality right after they are extracted.
- Late fusion lets each modality reach its own decision first, then combines the decisions.
- Hybrid fusion mixes the two.
Late fusion is easy to build from existing parts but misses links between modalities, such as a caption that is sarcastic only because of the picture. Google's technical report on Gemini, by contrast, says the models were trained jointly across image, audio, video and text, and calls them natively multimodal. The opposite design is a pipeline: speech recognition turns your voice into text, a text-only model answers and a synthesizer reads the reply, losing tone of voice and pauses at the first step.
What kinds of data can multimodal AI handle?
Multimodal models most often handle four modalities:
| Modality | Typical input | Example task |
|---|---|---|
| Text | Questions, documents, code | Summarize a contract |
| Image | Photos, screenshots, charts, scans | Describe a photo, read a chart |
| Audio | Speech, calls, recordings | Answer a spoken question |
| Video | Clips, screen recordings | Find the moment something appears |
Documents such as PDFs sit in between: a model can read their text layer, or treat each page as an image when layout, tables or handwriting matter.
Input-multimodal or output-multimodal?
A model can be multimodal on the input side, the output side or both, and "multimodal" on a product page rarely says which.
- Multimodal input, text output. The model understands images, audio or video but answers only in text. Vision-language models work this way.
- Text input, other output. Text-to-image and text-to-speech models cross two modalities in one direction; not everyone calls them multimodal.
- Multimodal in and out. OpenAI's system card for GPT-4o (2024) describes a model that accepts text, audio, image and video and generates text, audio and images, with one neural network processing all of them.
List the modalities that go in and the ones that come out; that tells you more than the label.
What is a vision-language model?
A vision-language model (VLM) is a multimodal model that takes images and text as input and answers in text. It is the most common kind of multimodal AI, and it is what runs when you attach a picture to a chat and ask about it.
The usual recipe joins a pretrained vision encoder, a connector and a pretrained language model. The LLaVA paper (Visual Instruction Tuning, 2023) is a clear public example: it used the CLIP vision encoder, a simple linear layer to carry image features into the language model's word space, and an open language model called Vicuna. Its training conversations were written by a text-only GPT-4 that never saw the images; it was given their captions and object boxes as text.
Reading text from an image is an older and narrower job called optical character recognition; What Is OCR? explains how it works and when a dedicated OCR engine is still the better tool.
What can you do with multimodal AI?
Multimodal AI helps wherever a question depends on something you can see or hear. Common examples:
- Describe a photo: alt text for a website, a description for a blind or low-vision user, a caption for a product image.
- Read a chart or dashboard: upload a screenshot and ask which month had the drop, then check the number yourself.
- Ask about a document: a scanned contract, a page full of tables, a form filled in by hand.
- Talk by voice: a spoken conversation with follow-up questions, sometimes while showing the camera what you mean.
- Search inside video: find the minute in a two-hour recording where a slide or a product appears.
- Work on a screen: agents that operate a browser or desktop read screenshots to decide where to click, so they are multimodal too.
You probably meet it already: the image and microphone buttons in chat apps, phone cameras that translate menus, meeting tools that summarize a recording with its slides.
What are the limits of multimodal AI?
Multimodal models make the same mistakes as text models, plus some that belong to images, sound and video:
- Describing what is not there. A model can name an object, word or number that is not in the image, in the same confident tone as a correct answer. This is the visual form of AI hallucination, and the defense is the same: ask for evidence and check it.
- Counting and fine detail. A 2024 study titled "Vision language models are blind" tested four state-of-the-art VLMs on seven very simple tasks and found them failing at questions such as whether two circles overlap, how often two lines cross or which letter in a word is circled.
- Small and rotated text. Large images are scaled down before the model sees them, so fine print can become unreadable. Blurry, rotated or tiny images raise the error rate.
- Bias. Image-text pairs from the web carry the web's imbalances: some languages, scripts, regions and groups of people appear far more often than others.
- Not a fake detector. A general model cannot reliably tell whether an image was made by AI.
- Cost. Images and audio use many tokens. In Anthropic's vision documentation for its Claude models, an image is cut into 28 by 28 pixel patches, each one a visual token, so a 1000 by 1000 pixel image costs 1,296 tokens before you type a word.
How should you handle photos, faces and privacy?
Treat a photo as personal data whenever it shows a person or something that identifies one. A picture can show faces, people in the background, a house number, a car plate or an ID card on the table, and the file's metadata can record where and when it was taken.
The law is strict about faces. Under the EU's General Data Protection Regulation, facial images processed with specific technical means to identify a person are biometric data. Article 9 prohibits processing biometric data to uniquely identify someone unless one of its exceptions applies, and Recital 51 adds that an ordinary photograph is not automatically in that category. In Türkiye, Article 6 of the Personal Data Protection Law (KVKK) also lists biometric data as special category personal data.
A business needs a lawful basis before customer or employee photos go into an AI tool, and should check whether the provider trains on uploads. Face recognition needs a clear legal ground of its own.
Where does the training data for multimodal AI come from?
Multimodal models learn from very large sets of paired data, such as images with captions, audio with transcripts and video with subtitles, much of it from public web pages; CLIP's 400 million pairs came from the internet.
Coverage matters as much as size. A model that has seen menus, signs and receipts in only a few scripts will misread the rest, and a test set built from one country's pages hides that. Image-heavy pages are often localized too: shops and travel sites show different photos, banners and text inside images depending on the visitor's country. Collecting those pages as a local visitor sees them is where a Residential Proxy fits, since each request leaves through a home connection in the country and city you choose.
Collect public pages only, check the license of images before training on them, follow robots.txt and site terms, keep request rates low and leave out faces and personal data you do not need. Teams that would rather not run the collection themselves can ask our sales team about managed page extraction through our Web Scraping API.
Common mistakes
- Trusting counts and measurements. Treat a VLM's count of items on a shelf as an estimate.
- Sending a huge screenshot to read fine print. It is scaled down first; crop the part that matters.
- Taking a fluent description as verified. Check numbers, names and quotes against the image.
- Uploading other people's faces or IDs casually. They are personal data, and faces can be special category data.
- Testing on clean samples only. Try poor lighting, other scripts, rotated pages and noisy audio first.
- Paying for resolution you do not need. Image tokens add up quickly across thousands of files.
Decision guide
| Need | Use |
|---|---|
| Read printed text from many scans | Dedicated OCR, VLM for hard cases |
| Ask about a chart or screenshot | A vision-language model, then check numbers |
| Hands-free voice conversation | A speech-native model or a speech pipeline |
| Find a moment in a long video | A model with video input or a transcript |
| Exact counts or measurements | Classic computer vision or a human check |
| Photos with faces or ID cards | Redact first; legal basis for business use |
| Test data from many countries | Public pages collected per country, licenses checked |
Frequently asked questions
Is ChatGPT multimodal?
Yes. OpenAI's GPT-4o System Card (2024) describes GPT-4o, the model OpenAI used for ChatGPT's Advanced Voice Mode, as accepting text, audio, image and video and producing text, audio and images, with one network trained end-to-end across them. The models and features in ChatGPT change often and differ by plan, so check what your version offers.
What is the difference between multimodal AI and generative AI?
Generative AI produces new content, while multimodal AI handles more than one kind of data. An assistant that looks at a photo and writes a reply is both. A classifier that flags harmful posts from their text and image is multimodal but not generative; a text-only chatbot is the reverse.
What is a vision-language model?
A vision-language model is a multimodal model that takes images and text as input and answers in text. It usually combines a vision encoder, a small connector and a large language model, and it powers most "ask about this picture" features.
Can multimodal AI understand video?
Yes, mostly by sampling frames and reading them alongside the audio track or a transcript. Google's Gemini report describes encoding video as a sequence of frames in the model's context window. Because only some frames are read, a quick event between two samples can be missed, and long videos use many tokens.
Is it multimodal or multi-model AI?
Both terms exist and mean different things. Multimodal refers to several kinds of data; multi-model refers to several models working together, such as a speech recognizer, a language model and a voice synthesizer in a chain. A system can be both.
Is it safe to upload photos to an AI tool?
For ordinary pictures it is reasonably safe if you check the tool's data settings and keep sensitive details out. Do not upload ID cards, bank documents, medical records or photos of other people without their consent, crop what the task does not need and assume the upload may be stored. Businesses should also confirm whether uploads are used for training.
Summary
Multimodal AI works with more than one kind of data. Encoders turn images, sound and video into vectors in the same space as text, so one model can reason over all of them and answer in text, speech or images. Vision-language models, the most common form, let you question photos, charts and documents. They still miscount, misread small text and describe things that are not there, and photos of people are personal data. Check the answers, crop before you upload, and test models on public pages collected where their users are.




