Spring AI with Docker Model Runner: Local LLM Example

Docker Model Runner can serve a local language model through an OpenAI-compatible endpoint. Spring AI can call it using the OpenAI integration, while the same Spring Boot application retains a separate Ollama client.

This tutorial explains the endpoint, model and client configuration, and common connection errors. The recorded answers use Ollama because Docker Model Runner was not available in the original environment. They demonstrate the calling code rather than a live Docker Model Runner result.

1. How Docker Model Runner Works with Spring AI

Docker Model Runner (DMR) provides a way to download and serve AI models with Docker tooling. A local application calls its HTTP endpoint rather than managing model execution directly in Java.

Models are distributed as named artifacts, including those in Docker Hub’s ai namespace. After downloading a model, local inference can use it without sending prompts to a hosted model provider.

For this integration, DMR exposes an OpenAI-compatible chat endpoint. A request contains a model name and role-based messages, and is sent to a path ending in /chat/completions. Spring AI’s OpenAI client constructs that request.

Compatibility lets us reuse the client for this chat-completions example. It does not mean that every OpenAI endpoint, option, or feature is available in DMR.

We use spring-ai-starter-model-openai and point its base URL at DMR. A separate provider implementation is unnecessary for this basic call. The Spring AI DMR reference describes the setup.

No live DMR response is included. Section 5 uses Ollama’s OpenAI-compatible endpoint to exercise the OpenAiChatModel path and compares it with the native Ollama client. This demonstrates the client flow against Ollama, rather than Docker installation, DMR inference, or its performance.

1.1. Choosing Between DMR and Ollama

Both can serve models locally without a hosted inference charge. Local compute requirements and software or model license conditions still apply. The choice depends on model support, hardware, deployment needs, and your existing tooling:

• Ollama is a separate program with its own model library. The other OrderDesk articles use it, and Spring AI has a dedicated Ollama module for it.
• DMR integrates with Docker tooling. Check the supported setup for your operating system and hardware.

In this article, we use both in one app.

2. Enable DMR and Make a Model Available

Use Docker’s current DMR setup guide to check platform and hardware support. The Spring AI reference’s Docker Desktop for Mac example is not a complete list of supported environments. The command below illustrates the Desktop setup used by this configuration.

Enable Model Runner’s TCP endpoint on port 12434 so an application running on the host can reach it:

docker desktop enable model-runner --tcp 12434

The model must also be available to DMR. Download it with the Docker model tooling for your installation. This example selects ai/gemma3; an explicit tag such as ai/gemma3:4B-F16 selects a particular artifact. Use the same model name in DMR and the application properties.

For integration tests, the Spring AI documentation also shows a Testcontainers option, DockerModelRunnerContainer, which we didn’t run.

3. Configure the Spring Boot Application

The example is a Spring Boot app called orderdesk-dmr. You can find it in the spring-ai-examples repository on GitHub. The app uses the qwen2.5:3b model in Ollama as its second model. If you need help installing Ollama and downloading the model, follow the Getting Started with Spring AI tutorial.

The app uses Spring Boot 4.1, Spring AI 2.0.1, and Java 21, with all classes in the com.howtodoinjava.orderdesk.dmr package.

3.1. Dependencies

Here is the full pom.xml. It imports the Spring AI BOM (bill of materials), which sets the version of every Spring AI module:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>4.1.1</version>
    <relativePath/>
  </parent>

  <groupId>com.howtodoinjava</groupId>
  <artifactId>orderdesk-dmr</artifactId>
  <version>1.0.0</version>

  <properties>
    <java.version>21</java.version>
  </properties>

  <dependencyManagement>
    <dependencies>
      <dependency>
        <groupId>org.springframework.ai</groupId>
        <artifactId>spring-ai-bom</artifactId>
        <version>2.0.1</version>
        <type>pom</type>
        <scope>import</scope>
      </dependency>
    </dependencies>
  </dependencyManagement>

  <dependencies>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-webmvc</artifactId>
    </dependency>
    <dependency>
      <groupId>org.springframework.ai</groupId>
      <artifactId>spring-ai-starter-model-ollama</artifactId>
    </dependency>
    <dependency>
      <groupId>org.springframework.ai</groupId>
      <artifactId>spring-ai-starter-model-openai</artifactId>
    </dependency>
  </dependencies>

  <build>
    <plugins>
      <plugin>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-maven-plugin</artifactId>
      </plugin>
    </plugins>
  </build>
</project>

The spring-boot-starter-webmvc dependency is the Spring MVC web starter in Spring Boot 4. The spring-ai-starter-model-openai starter creates the OpenAiChatModel bean, and this bean calls DMR.

3.2. Set the Endpoint, Model, and Client Options

Here is the full application.properties:

spring.application.name=orderdesk-dmr
server.port=8087

# Two chat models in one app, so we build the ChatClients ourselves
spring.ai.chat.client.enabled=false

# The local model, as in the other OrderDesk articles
spring.ai.ollama.base-url=http://localhost:11434
spring.ai.ollama.chat.model=qwen2.5:3b
spring.ai.ollama.chat.temperature=0.2

# Docker Model Runner speaks the OpenAI API on localhost; any key works
spring.ai.openai.api-key=test
spring.ai.openai.base-url=http://localhost:12434/engines/v1
spring.ai.openai.chat.model=ai/gemma3
# Model Runner has no embedding model here; use Ollama's
spring.ai.model.embedding=ollama
spring.ai.openai.chat.temperature=0.2

The OpenAI-prefixed properties configure the client library used to reach DMR; they do not mean the application sends requests to OpenAI. The important settings are:

1. spring.ai.openai.api-key=test supplies the client library’s key setting for this local endpoint. This is a placeholder, not an OpenAI account credential.
2. spring.ai.openai.base-url points to the DMR API base, including /engines/v1. The client appends the chat-completions path.
3. spring.ai.openai.chat.model selects the model available in DMR, here ai/gemma3.
4. spring.ai.model.embedding=ollama selects Ollama for embedding auto-configuration. This endpoint does not use embeddings; the setting prevents unrelated embedding configuration from targeting the DMR chat endpoint.

The application has both OpenAiChatModel and OllamaChatModel. It disables the auto-configured builder with spring.ai.chat.client.enabled=false and constructs a ChatClient for each model.

4. Create Separate DMR and Ollama Clients

Each backend has its own ChatModel and ChatClient. The controller uses the same fluent call with either client:

ChatConfig creates one client per model, with the same system prompt:

package com.howtodoinjava.orderdesk.dmr;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.ollama.OllamaChatModel;
import org.springframework.ai.openai.OpenAiChatModel;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class ChatConfig {

    static final String SUPPORT = "You are OrderDesk, the support assistant of an online store. "
        + "Answer in at most three sentences.";

    // The model in Docker Model Runner, reached through the OpenAI API
    @Bean
    ChatClient runnerClient(OpenAiChatModel chatModel) {
        return ChatClient.builder(chatModel).defaultSystem(SUPPORT).build();
    }

    // The model in Ollama, as in the other OrderDesk articles
    @Bean
    ChatClient ollamaClient(OllamaChatModel chatModel) {
        return ChatClient.builder(chatModel).defaultSystem(SUPPORT).build();
    }
}

The DMR client wraps OpenAiChatModel because it uses the OpenAI-compatible protocol. The configured base URL determines the destination. This basic request can also target Ollama’s compatible endpoint by changing the address and model. Other services may need additional credentials or options.

Injecting the concrete model classes identifies the backend each client wraps. The controller uses qualifiers to select the corresponding ChatClient beans.

The controller picks a client from the model request parameter. It returns the answer with the model name and the token counts:

package com.howtodoinjava.orderdesk.dmr;

// The answer plus the facts that differ between models
public record Reply(String model, String answer, Integer inputTokens, Integer outputTokens) {
}

package com.howtodoinjava.orderdesk.dmr;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.metadata.ChatResponseMetadata;
import org.springframework.ai.chat.model.ChatResponse;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class SupportController {

    private final ChatClient runnerClient;
    private final ChatClient ollamaClient;

    SupportController(@Qualifier("runnerClient") ChatClient runnerClient,
                      @Qualifier("ollamaClient") ChatClient ollamaClient) {
        this.runnerClient = runnerClient;
        this.ollamaClient = ollamaClient;
    }

    // model=runner (default) or model=ollama
    @GetMapping("/ask")
    Reply ask(@RequestParam String question,
              @RequestParam(defaultValue = "runner") String model) {
        ChatClient client = "ollama".equals(model) ? ollamaClient : runnerClient;
        ChatResponse response = client.prompt().user(question).call().chatResponse();
        ChatResponseMetadata metadata = response.getMetadata();
        return new Reply(metadata.getModel(), response.getResult().getOutput().getText(),
            metadata.getUsage().getPromptTokens(), metadata.getUsage().getCompletionTokens());
    }
}

@Qualifier tells Spring which bean to inject when multiple beans match the required type. In this example, it identifies the bean by the name of the @Bean method. The chatResponse() method returns the full response, and its getMetadata() gives us the model name and the token counts (Usage).

The main class is the usual Spring Boot one:

package com.howtodoinjava.orderdesk.dmr;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class DmrApplication {

    public static void main(String[] args) {
        SpringApplication.run(DmrApplication.class, args);
    }
}

5. Call the Endpoint and Read the Recorded Results

For a DMR-backed request, start DMR with the configured model available. Start Ollama as well for the native Ollama branch, then launch the application with mvn spring-boot:run. Omitting the model parameter selects the client named for the runner:

curl -G localhost:8087/ask --data-urlencode "question=Can I return a kettle after 5 days?"

With the DMR configuration and a working endpoint, the request targets ai/gemma3. No output from that backend is included. The recorded alternative below uses Ollama.

5.1. Reproduce the Client Flow with Ollama

Ollama exposes an OpenAI-compatible endpoint at http://localhost:11434/v1. The recorded example launches the application with overrides for that address and qwen2.5:3b, allowing the runner client to call Ollama through OpenAiChatModel:

java -jar target/orderdesk-dmr-1.0.0.jar \
  --spring.ai.openai.base-url=http://localhost:11434/v1 \
  --spring.ai.openai.chat.model=qwen2.5:3b

Recorded Ollama response through the default runner branch (OpenAiChatModel):

{"model":"qwen2.5:3b","answer":"Yes, you can return a kettle within 5 days, but it's best to check your store's specific return policy as some policies may vary.","inputTokens":43,"outputTokens":31}

Recorded Ollama response through the native model=ollama branch (OllamaChatModel):

{"model":"qwen2.5:3b","answer":"Yes, you can return a kettle within 5 days of receiving it, as many online stores offer a 7-day return policy, which typically includes the first 5 days. Please check your specific store's return policy for any variations.","inputTokens":43,"outputTokens":49}

Both responses use Ollama’s qwen2.5:3b. The difference is the client protocol: OpenAI-compatible chat completions in the first path and the native Ollama integration in the second. The results demonstrate neither DMR answer quality nor DMR latency.

The small system prompt does not include OrderDesk’s ten-day return window. The model cannot reliably apply it until the application supplies the policy through instructions, retrieval, or tools.

5.2. Inspect the Request Format for DMR

This body was captured by a recording server at POST /engines/v1/chat/completions. It shows how Spring AI builds the DMR-targeted request without implying that a live DMR instance processed it:

{"messages":[{"content":"You are OrderDesk, the support assistant of an online store. Answer in at most three sentences.","role":"system"},{"content":"Can I return a kettle after 5 days?","role":"user"}],"model":"ai/gemma3","temperature":0.2}

Notice two things. First, the OpenAI module adds /chat/completions to the base URL, which is why the base URL must end with /engines/v1 for DMR.

Second, Spring AI sends the system prompt as the first message, with the role system. The model name and the temperature come from application.properties.

5.3. Diagnose a Connection Failure

The client does not establish a model connection at startup. If the endpoint is unavailable, the error appears when the first request attempts to connect:

com.openai.errors.OpenAIIoException: Request failed
...
java.net.ConnectException: Connection refused

The package is com.openai, because the Spring AI OpenAI module uses the official openai-java SDK (Spring AI OpenAI chat page). Connection refused means nothing is listening on port 12434. Check that DMR is turned on with the –tcp 12434 option from Section 2.

5.4. Diagnose an Incorrect API Path

We saw this error in the Ollama stand-in. If the base URL is missing /v1 (for example http://localhost:11434), the call fails even though the server is running:

com.openai.errors.NotFoundException: 404

The OpenAI module sent the request to /chat/completions, but that path doesn’t exist on the server at the URL it was given. The DMR base URL from the Spring AI documentation already ends with /v1, so make sure you configure the base URL correctly and let the client build the rest of the path. Don’t accidentally duplicate or omit part of the URL.

Use the error type to narrow the investigation. A refused connection suggests no service is accepting requests at the host and port. A 404 means an HTTP service responded but did not recognize the route or resource. Check the complete address and path before changing model code.

All articles in order: Spring AI Tutorial

Drop me your questions related to Spring AI with Docker Model Runner in the comments.

Happy Learning !!

Source Code on Github