Spring AI Multimodality: Send Images to a Chat Model

A vision-capable chat model can answer a question about an image as well as read text. Spring AI sends both through the same user message, so an application can attach a photo without constructing the provider’s image request manually.

Here, OrderDesk examines a parcel photo and returns either a description or a DamageReport record. We configure a separate local vision model, accept uploaded images, and explain the limits of the resulting report. The example uses Spring Boot, Spring AI, Java 21, and Ollama.

1. How Image Input Works in Spring AI

A modality is a kind of input, such as text, an image, or audio. A multimodal model accepts more than one kind. A vision model is a multimodal model that accepts text and images.

ChatClient prepares the messages and delegates the request to a configured model. In this example, the system message provides instructions and the user message contains a question plus an image.

The user message includes both text and media. Each media item identifies its content and MIME type, such as image/jpeg or image/png, so the provider can interpret it correctly.

In code, a media item is a Media object (org.springframework.ai.content.Media). We rarely create it ourselves, because ChatClient builds it for us when we call user(u -> u.text(…).media(…)).

This example asks for text output. For a Java object, entity() requests structured text and converts the response into the target type. Reading an image and generating an image are separate capabilities.

Media goes only into user messages. The system message stays text.

1.1. Choose a Model That Accepts Images

Spring AI supports image input for these providers, as long as the chosen model is a vision model:

1. Anthropic (Claude models).
2. Amazon Bedrock, through the Converse API.
3. Google Gemini.
4. Mistral AI, with its Pixtral models.
5. OpenAI.
6. Ollama, with local vision models such as LLaVA, Llama vision and qwen2.5vl.

Check image-input support for the specific model, not just the provider. The ordinary qwen2.5:3b model used elsewhere in OrderDesk is text-only; the vision endpoints select a different model.

2. Setting Up the Vision Model

This article is part of the OrderDesk series, where every Spring AI tutorial adds one feature to the same app. OrderDesk is the support assistant of a demo online store. The code for this article belongs to the com.howtodoinjava.orderdesk.vision package. Every new class goes in a sub-package of com.howtodoinjava.orderdesk, the package of the main class, because Spring Boot looks for beans only in that package and below it. The full app is on GitHub in the orderdesk folder.

The basic setup (Ollama, the chat model qwen2.5:3b, and a Spring Boot project with Spring AI) is the same for every article in the series. Getting Started with Spring AI explains each of these step by step. This article adds three things on top: a vision model, a second ChatClient bean, and a sample image. The pom.xml needs no new dependency, because the Ollama starter already sends media.

2.1. Download a Vision Model

The chat model qwen2.5:3b reads only text. If we send it an image, Ollama rejects the request (Section 4.1 shows the error). So we need a vision model too. For OrderDesk, we use qwen2.5vl:3b, a small vision model from the same Qwen family. The download is about 3.2 GB:

ollama pull qwen2.5vl:3b

After the download, ollama list shows qwen2.5vl:3b next to qwen2.5:3b.

We keep qwen2.5:3b as the default model in application.properties, so only the image endpoints use the vision model. The vision model needs more memory than the chat model. With 8 GB of RAM, both models may not fit at the same time, and calls get slow (Section 4.2).

2.2. Add a Second ChatClient Bean

Next, we tell Spring AI which requests go to the vision model. We do that with a second ChatClient bean (a Spring-managed object that other classes can inject). Only the damage endpoints use it:

package com.howtodoinjava.orderdesk.vision;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.ollama.api.OllamaChatOptions;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class VisionConfig {

    // qwen2.5:3b reads only text, so images go to a vision model
    @Bean
    ChatClient visionClient(ChatClient.Builder builder) {
        return builder
            .defaultSystem("You check photos of parcels for the OrderDesk support team.")
            .defaultOptions(OllamaChatOptions.builder().model("qwen2.5vl:3b").temperature(0.1))
            .build();
    }
}

Let’s go through it line by line:

1. @Configuration marks VisionConfig as a class that declares beans.
2. @Bean ChatClient visionClient(…) creates a bean named visionClient. The bean name comes from the method name, and we use it later to inject this client.
3. ChatClient.Builder is the builder that Spring AI’s auto-configuration creates for us. The settings below change only visionClient.
4. defaultSystem(…) sets the system message that goes with every request from this client.
5. defaultOptions(…) sets the model options for every call of visionClient. OllamaChatOptions holds the Ollama options. model(“qwen2.5vl:3b”) replaces the default model from application.properties, and temperature(0.1) makes the answers less random, which we want for a report. The missing build() after temperature(0.1) is not a mistake. In Spring AI 2.0, defaultOptions() has a version that takes the options builder (ChatOptions.Builder) and builds it itself.
6. build() returns the finished ChatClient.

We set the model name in Java here. You can also move it to a property of your own and inject it with @Value.

The application now has multiple ChatClient beans. The existing controller can resolve supportClient by its parameter name where name-based injection is available, but an explicit @Qualifier(“supportClient”) makes the intended client clearer and avoids relying on that naming behavior.

2.3. Add the Sample Image

Download damaged-parcel.jpg from the images folder on GitHub, or use any JPEG photo of a parcel. Put it in src/main/resources/images and name it damaged-parcel.jpg. Files in src/main/resources end up on the classpath, so the code can load the image by the path images/damaged-parcel.jpg. Our image is a drawn test image (900 x 700 pixels).

3. Sending Images to the Model

Now we build DamageController in three steps. First it describes an image from the classpath. Then it accepts a photo that the customer uploads. Finally it returns the answer as a DamageReport record.

Start Ollama and the app as shown in Getting Started with Spring AI. Restart the app after adding the code of each step.

Step 1: Describe an Image from the Classpath

Let’s start with the smallest request: one fixed image and one question. Spring loads files through the Resource interface (org.springframework.core.io.Resource), which gives access to the bytes of a file wherever it lives. Our image is on the classpath, so we use the ClassPathResource implementation. The class below already has all the imports of the finished file, and the methods of Steps 2 and 3 use MimeType, PostMapping, RequestParam, and MultipartFile:

package com.howtodoinjava.orderdesk.vision;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.core.io.ClassPathResource;
import org.springframework.util.MimeType;
import org.springframework.util.MimeTypeUtils;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.multipart.MultipartFile;

@RestController
public class DamageController {

    private final ChatClient visionClient;

    DamageController(@Qualifier("visionClient") ChatClient visionClient) {
        this.visionClient = visionClient;
    }

    // Step 1: an image from the classpath and a question about it
    @GetMapping("/damage/sample")
    String describeSample() {
        var photo = new ClassPathResource("images/damaged-parcel.jpg");
        return visionClient.prompt()
            .user(u -> u.text("Describe this parcel in two sentences. Is it damaged?")
                        .media(MimeTypeUtils.IMAGE_JPEG, photo))
            .call()
            .content();
    }
}

OrderDesk has more than one ChatClient bean, so @Qualifier(“visionClient”) picks this one.

The request adds a question with text() and an image with media(). The call().content() chain executes synchronously and returns the generated description. The client’s configured model must support the supplied media.

MimeTypeUtils.IMAGE_JPEG is the constant for image/jpeg. Spring AI reads the bytes of the Resource and encodes the image for us.

Call the endpoint:

curl localhost:8080/damage/sample

The model reads the label and sees damage:

The parcel is a cardboard box with a label that reads "OrderDesk Order: ORD-1042 FRAGILE." The box appears to be damaged, as there is a noticeable tear on the side.

The order number is right, but the tear is on the top, not the side (Section 4.4). The first call can take minutes while Ollama loads the model (Section 4.2).

Step 2: Accept an Uploaded Photo

Next, we take the photo from the customer instead of the classpath. A browser or curl sends a file as a multipart request, an HTTP request whose body has several named parts, such as form fields and files. Spring MVC gives us each uploaded file as a MultipartFile, which holds the file’s bytes, its name, and its content type. Add this method to DamageController:

    // Step 2: an image that the customer uploads
    @PostMapping("/damage/describe")
    String describe(@RequestParam("photo") MultipartFile photo) {
        return visionClient.prompt()
            .user(u -> u.text("Describe this parcel in two sentences. Is it damaged?")
                        .media(MimeType.valueOf(photo.getContentType()), photo.getResource()))
            .call()
            .content();
    }

Two things changed from Step 1. The photo.getResource() call gives the uploaded bytes as a Resource, so media() accepts it like the classpath file. The photo.getContentType() call gives the MIME type that the client sent, and MimeType.valueOf() turns that String into a MimeType.

We send the file with curl -F, which builds a multipart form request. Run the command from the project folder, because the file path is relative to it. The name before = must match @RequestParam(“photo”). The ;type=image/jpeg part sets the content type of the file:

curl -F "photo=@src/main/resources/images/damaged-parcel.jpg;type=image/jpeg" localhost:8080/damage/describe

The parcel is a cardboard box with a label that reads "OrderDesk Order: ORD-1042 FRAGILE". The box appears to be damaged as there is a noticeable tear on the side.

The sample trusts the submitted content type. In an application, validate both the declared type and actual image content, reject missing or unsupported types, and bound file size and decoded dimensions before inference. A content-type label alone does not prove that a file is an image.

Step 3: Get a DamageReport Record

Finally, we want data that our code can use, not two sentences. The support team needs the order number, the kind of damage, and whether the label is readable. Let’s create a record for that:

package com.howtodoinjava.orderdesk.vision;

public record DamageReport(String orderNumber, String damageType, boolean labelReadable,
                           String summary) {
}

Then add the third method to DamageController. It’s the upload from Step 2 with a new question, and it ends with entity() instead of content():

    // Step 3: the same upload, returned as a Java record
    @PostMapping("/damage/report")
    DamageReport report(@RequestParam("photo") MultipartFile photo) {
        return visionClient.prompt()
            .user(u -> u.text("""
                    Fill a damage report for this parcel. Read the order number
                    from the label. damageType is one of: none, crushed, torn, wet.""")
                        .media(MimeType.valueOf(photo.getContentType()), photo.getResource()))
            .call()
            .entity(DamageReport.class);
    }

In ChatClient, the entity() method returns the answer as a Java object instead of a String. Calling entity(DamageReport.class) adds format instructions to our text, which tell the model to reply with JSON that has the fields of DamageReport. When the reply comes back, Spring AI converts the JSON into the record. Spring AI Structured Output Converters explains this conversion in detail.

Field names and prompt instructions guide the response, but they do not enforce business rules. The example asks for a limited set of damageType values; validate the returned value before comparing it in application logic. A free-form String can still contain an unexpected value.

Send the same photo to the new endpoint:

curl -F "photo=@src/main/resources/images/damaged-parcel.jpg;type=image/jpeg" localhost:8080/damage/report

Spring MVC writes the record back as JSON:

{"orderNumber":"ORD-1042","damageType":"crushed","labelReadable":true,"summary":"The parcel is damaged and labeled as fragile."}

The order number is correct, and the label is readable. The damageType field says “crushed”, but the box is also torn (Section 4.4).

4. Limits and Traps

The code works, but images bring problems that text prompts don’t have.

4.1. A Text-Only Model Rejects Images

The supportClient bean from Getting Started uses the default model qwen2.5:3b. If we send the same request through supportClient, Ollama refuses it:

org.springframework.ai.retry.NonTransientAiException: HTTP 400 - {"error":"Multimodal data provided, but model does not support multimodal requests."}

Spring AI doesn’t retry this error. The fix is to send images only to a vision model. Before you pick an Ollama model, check its page in the Ollama library for the “vision” label. For hosted providers, check the provider’s model list for image input.

4.2. The First Call After Switching Models Is Slow

On our test machine, the first call to visionClient took 6 minutes. Every call after that took about 16 seconds.

The first request may include loading the model into memory. Later requests can be faster while it remains loaded. Unloading behavior depends on Ollama’s keep-alive configuration, available memory, and other activity, so do not treat one five-minute idle interval as a fixed application rule.

Memory matters too. The vision model needs more memory than the chat model, and both may not fit in 8 GB of RAM. In that case, Ollama swaps them: it unloads one model to load the other. Each swap is a slow call.

In development, call /damage/sample once after startup to load the model. On a server, keep the model loaded or run it on a machine with enough memory for both models.

4.3. Image Size Costs Tokens

Image processing contributes to the request workload and, for hosted models, may contribute to billable usage. How dimensions translate to image tokens depends on the model and provider. Resize unnecessarily large photos while preserving the detail needed for the task.

A photo from a phone camera has many times more pixels than our test image. So resize large photos on the server before sending them to the model.

Spring Boot also limits the upload size. By default, one uploaded file can be at most 1 MB. A larger photo fails before it reaches our method. To accept bigger files, set spring.servlet.multipart.max-file-size in application.properties.

4.4. The Model Can Be Wrong

The box has a torn hole on the top, a pushed-in corner and a brown stain on the front.

The sample reads the order number and identifies damage, but places the tear incorrectly and reports only “crushed.” A single damageType field also forces multiple visible defects into one label. Model quality and the output schema both affect how useful the report is.

So treat the report as a suggestion, not a decision. Let a person look at the photo before OrderDesk approves a refund or a replacement. If the report must list every kind of damage, change damageType to a List<String> and say so in the question.

4.5. Generating Images and Reading Audio

Multimodal input in this article means images in and text out. To create images from a text prompt, use ImageModel, as shown in Spring AI ImageModel Example. To turn audio into text, see Spring AI Speech to Text.

5. FAQs

5.1. Can I send more than one image in one message?

Yes. Call media() once for each image inside the same user() call. Each image adds tokens, so the request gets slower. We tested OrderDesk with one image only.

5.2. Can the chat model send back an image?

This endpoint produces text or a record, not an image. For the separate Spring AI image-generation API, use ImageModel as described in Section 4.5. Other output capabilities depend on the chosen provider and integration.

5.3. Which image formats can I send?

JPEG and PNG work with most providers. Some providers accept more types, such as GIF or WebP, so check your provider’s documentation. We tested OrderDesk with JPEG only.

5.4. Do I always need a second ChatClient for images?

No. Many hosted models read both text and images, so one model can serve the whole app. We need two beans in OrderDesk because the small local chat model reads only text.

5.5. Does Spring AI send my photo anywhere?

It sends the photo to the model provider that the ChatClient uses. In OrderDesk, that is Ollama on localhost, so the photo stays on our computer. With a hosted provider, the photo leaves your server, so check what the provider stores before you send customer photos.

6. Conclusion

Attach image media and the question to the user message, then send them through a vision-capable ChatClient. Use content() for a description or entity() for a typed report, and validate the result before acting on it.

Check model capabilities, upload limits, and memory requirements before deployment. A successful response can still misidentify details, so retain a way to review the image alongside the report.

Use the generated report to support review. It should not, on its own, authorize a refund or establish what happened to the parcel.

All articles in order: Spring AI Tutorial

7. References

• Spring AI Reference: Multimodality API
• Ollama library: qwen2.5vl

Drop me your questions related to Spring AI multimodality in the comments.

Happy Learning !!

Source Code on Github