Building with Multimodal AI

The gap between "multimodal AI exists" and "I built something useful with it" is smaller than it's ever been — you can call a vision-language model, a speech recognizer, or an image generator with an API and a few lines of code. But building well with multimodal AI means knowing what these models can and can't reliably do, how to combine them, and where the pitfalls are. This closing post is the practical one: how to actually build multimodal applications, and where the whole field is heading — toward general, any-to-any models.

By Pratik Dhanave · June 24, 2026 · 9 min read

#Multimodal AI #AI Engineering #Machine Learning #Generative AI

Having covered the modalities and the ideas behind them, this final post is about building with multimodal AI: the practical landscape of using multimodal models, common application patterns, key considerations and pitfalls, and the trajectory toward any-to-any general models. It ties the series together and points at the future. The goal is to leave you able to reason about using multimodal AI, not just understanding it.

The practical landscape

Building with multimodal AI today is largely about using powerful pretrained models (via APIs or open models), not training them from scratch — which makes these capabilities widely accessible:

Use pretrained models, don’t train from scratch. The multimodal models this series described (VLMs, image generators, speech models) are expensive to train and mostly built by large labs — but they’re accessible to use via APIs (hosted services) or open-source models you can run. Building with multimodal AI means leveraging these pretrained models, not building them. This is like building on LLMs: you use the model, you don’t create it. It makes multimodal capabilities available to any developer.
The core building blocks. The practical toolkit: vision-language models (understand images + text — for analysis, VQA, document understanding), image generation (text-to-image — for creating/editing visuals), speech recognition and synthesis (voice interfaces), embeddings/CLIP-style models (for multimodal search and matching), and increasingly general multimodal LLMs that do many of these in one model. Knowing these building blocks and what each does is the basis for building. You compose them into applications.
APIs and open models both viable. You can use hosted APIs (convenient, powerful, pay-per-use) or open models (run yourself, more control/privacy, especially the on-device angle from the on-device-AI series). The choice depends on needs (cost, privacy, latency, control) — the same build-vs-buy and hosted-vs-self-hosted considerations as with LLMs. Both make multimodal capabilities practically available.

Building with multimodal AI is mostly using accessible pretrained models (via APIs or open models) as building blocks — VLMs, image generation, speech, multimodal embeddings, and general multimodal LLMs — composed into applications. It’s accessible to any developer, like building on LLMs. The next question is what to build — the application patterns.

Application patterns

Multimodal AI enables a range of application patterns, combining the capabilities from across the series:

Understanding visual/audio content. Applications that understand images, documents, or audio: document processing (extract/understand data from documents, forms, screenshots — highly practical), image analysis (describe, categorize, answer questions about images), accessibility (describe visuals, transcribe audio), and content moderation/analysis. VLMs and speech recognition power these. “Make sense of non-text content” is a huge application category.
Generating content. Applications that create images, audio, or video: design and creative tools (generate/edit images), content creation, marketing visuals, voice generation (TTS for narration/assistants), and more. Generative multimodal models power these. “Create visual/audio content from descriptions” is a major category, especially for creative and content workflows.
Multimodal search and retrieval. Applications using shared-embedding-space matching (CLIP-style): search images by text (or vice versa), find similar content across modalities, and multimodal retrieval for RAG (retrieving relevant images/documents to ground responses — connecting to retrieval/RAG systems). Cross-modal search unlocks finding content by meaning across modalities.
Multimodal assistants and agents. Applications where an assistant can see, hear, and respond — the multimodal-chat experience (show it an image, ask questions), voice assistants (speak in, speak out), and multimodal agents that perceive and act (connecting to the agent series). General multimodal models increasingly enable rich assistants that work across modalities. This is the integrated, general multimodal-AI experience.
Domain-specific multimodal. Applications in specific domains combining modalities: medical (images + reports), robotics (perception + action), scientific analysis, etc. — bringing multimodal AI to specialized problems (from the “beyond core modalities” post). Combining domain modalities (e.g. an image and its text description) for domain tasks is a rich application area.

The application patterns — understanding visual/audio content, generating content, multimodal search/retrieval, multimodal assistants/agents, and domain-specific combinations — cover most of what’s built with multimodal AI. Recognizing which pattern fits your problem (and which building blocks it needs) is how you go from capability to application. But building well requires attention to real considerations and pitfalls.

Considerations and pitfalls

Building well with multimodal AI means knowing the models’ limitations and the practical pitfalls — because these models are powerful but imperfect:

They make mistakes and hallucinate. Multimodal models err — VLMs can misread images or “hallucinate” details not present, image generators produce artifacts or wrong details, ASR mis-transcribes. Like LLMs, they’re powerful but not reliable in the sense of always-correct. Build with this in mind: verify critical outputs, don’t assume the model saw/heard/generated correctly, and design for the possibility of error (especially for high-stakes uses). Overtrusting multimodal output is a common pitfall.
Know each capability’s reliability. Different multimodal tasks have different reliability — some (basic transcription, image captioning) are quite reliable; others (precise detail extraction, complex visual reasoning, perfectly accurate generation) are less so. Understand what the model does reliably vs unreliably for your use case, and design accordingly (e.g. don’t rely on a VLM to perfectly extract every number from a complex document without verification). Match usage to real reliability, not hoped-for reliability.
Cost and latency. Multimodal processing (especially images/video and generation) can be computationally expensive and slower than text — images are large, video larger, generation involves many steps (recall diffusion’s iterative process and video’s scale). Consider cost and latency in design — they can be significant, especially at scale or for video/generation. This is the same test-time-compute cost/latency tradeoff from the reasoning series, in the multimodal domain.
Ethical and safety concerns. Multimodal generation raises real concerns: deepfakes (fake images/video/voice — impersonation, misinformation), generating harmful or copyrighted content, privacy (processing images/audio of people), and bias (models reflecting biases in training data). These are serious, and building responsibly means considering misuse, consent, provenance, and bias. (The voice-cloning/deepfake concern from the audio post is one example.) The generative power carries real societal risks to build around thoughtfully.
Data and privacy. Images and audio are often sensitive (photos of people, private documents, voice) — processing them (especially via external APIs) raises privacy concerns. Consider data handling, and where privacy matters, on-device or self-hosted models (the on-device-AI series). Multimodal data is often more sensitive than text.

Building well with multimodal AI means accounting for models’ mistakes/hallucinations (verify critical outputs), matching usage to each capability’s real reliability, managing cost and latency (significant for images/video/generation), taking ethical/safety concerns seriously (deepfakes, harmful/copyrighted content, bias), and handling sensitive multimodal data with privacy in mind. These considerations separate robust applications from naive ones.

The any-to-any future

Finally, where multimodal AI is heading: toward general, any-to-any models — a fitting close to the series:

Unified models handling all modalities. The clear trajectory is toward single, general models that handle many modalities — text, images, audio, video — as input and output, natively and together (rather than separate specialized models). Frontier AI is increasingly multimodal by default, moving toward models that seamlessly work across all modalities. The separate-systems era is giving way to unified multimodal models.
“Any-to-any” generation and understanding. The aspiration is any-to-any: models that can take any modality (or combination) as input and produce any modality as output — understand an image and respond in speech, generate video from text and audio, translate freely across modalities. This general cross-modal fluency — any input, any output — is the direction, extending the cross-modal translations covered throughout to a general capability. It’s multimodal AI’s endpoint: no fixed input/output modality, just meaning flowing across forms.
Toward more general intelligence. Multimodal integration — grounding language in perception, connecting what’s read/seen/heard, reasoning across modalities — is part of building more general AI, closer to how humans integrate senses and modalities. Combining multimodal understanding with reasoning (the reasoning-models series) and action (agents) points toward more capable, general systems. Multimodality is a key ingredient of more general intelligence, not just a feature.
The recipe’s generality is the enabler. As the series showed, the same recipe (sequences, transformers, scale, shared representations, shared generation techniques) works across modalities — which is why unified any-to-any models are feasible: one architecture and approach can handle everything. The convergence on a general recipe across modalities is what makes the general multimodal future possible. Understanding that recipe is understanding where AI is going.

Building with multimodal AI is mostly using accessible pretrained models as building blocks, composed into application patterns (understanding, generating, searching, assisting, domain-specific), with careful attention to reliability, cost, ethics, and privacy — and the field is heading toward general, any-to-any models that handle all modalities together, a key ingredient of more general AI. That completes the series: from what multimodal AI is, through vision, cross-modal connection, VLMs, generation, audio, and video, to building with it and its future. Multimodal AI is how AI engages the full, multi-sensory richness of the world — and it’s rapidly becoming the default.

Key takeaways

Building with multimodal AI is mostly using accessible pretrained models (via APIs or open models — they’re expensive to train but easy to use), composing building blocks — vision-language models, image generation, speech recognition/synthesis, CLIP-style embeddings, and general multimodal LLMs — into applications, like building on LLMs.
The main application patterns are: understanding visual/audio content (document processing, image analysis, accessibility), generating content (images/audio/video for creative and content work), multimodal search/retrieval (CLIP-style cross-modal matching, multimodal RAG), multimodal assistants/agents (see/hear/respond), and domain-specific combinations (medical, robotics, scientific).
Build well by accounting for pitfalls: multimodal models make mistakes and hallucinate (verify critical outputs, don’t overtrust), each capability has different real reliability (match usage to it), cost and latency can be significant (images/video/generation are expensive — the test-time-compute tradeoff), and multimodal data is often sensitive (privacy).
Take ethical/safety concerns seriously — deepfakes (fake images/video/voice for impersonation/misinformation), harmful or copyrighted generated content, bias reflecting training data, and consent/privacy — as the generative power carries real societal risks to build around thoughtfully.
The field is heading toward general, any-to-any models — single unified models handling all modalities (text, image, audio, video) as input and output natively — enabled by the multimodal recipe’s generality (one architecture/approach across modalities), and multimodal integration (grounding language in perception, reasoning across modalities) is a key ingredient of more general, human-like AI.