Testing Spring AI Apps: Mocks, Testcontainers and Evaluators

Testing a Spring AI app means testing two things: our own code around the model, and the answers that the model gives. In this example, we test OrderDesk, the support assistant of a demo online store, at three levels: unit tests with a fake model, integration tests with Ollama in Docker, and answer checks with a model as the judge. We’ll use Spring Boot 4.1, Spring AI 2.0, JUnit 5, and Testcontainers.

1. How Testing an AI App Works

A model’s answer changes from run to run, so a test can’t compare it with a fixed string. But most of an AI app is ordinary code: building prompts, parsing the answer, guarding the database, and choosing the next step. That code can be tested like any other code, with a fake model that always returns the same text.

So the tests form three levels:

1. Unit tests replace the ChatModel with a lambda. They test our code in milliseconds and give the same result every time.
2. Integration tests start a real model in a Docker container. They check that the beans, the prompt, and the model work together.
3. Evaluation tests send a real answer to a second model and ask it whether the answer is right. Spring AI calls these evaluators.

Most tests should sit at the bottom.

2. The Example App

This article adds tests to OrderDesk, the app of this series. The tests are in src/test/java and belong to the com.howtodoinjava.orderdesk.sql and com.howtodoinjava.orderdesk.it packages. The full app is on GitHub in the orderdesk folder. The basic setup (Ollama, the chat model qwen2.5:3b, and the Spring Boot project) is in Getting Started with Spring AI.

The spring-boot-starter-test starter (JUnit 5, AssertJ, and Mockito) is part of every Spring Boot project. The integration tests need four more test dependencies, shown in section 4.

2.1. The Code Under Test

The tests check code from earlier articles. Copy these files from GitHub; this article doesn’t explain them again:

1. OrderQueryController.java, OrderQueryService.java, and SqlGuard.java: the text-to-SQL feature from Query a Database with Natural Language using Spring AI. The model writes SQL, SqlGuard cleans it and allows only one SELECT, and OrderQueryService runs it. The unit test in Section 3 tests these three classes.
2. ChatClientConfig.java: the supportClient bean from Getting Started, shown below. The integration test in Section 4 calls it.
3. RagConfig.java: the questionAnswerAdvisor bean, from Spring AI RAG Advisors: QuestionAnswer vs RetrievalAugmentation. It needs the spring-ai-vector-store-advisor dependency.
4. SimpleVectorStoreConfig.java: the VectorStore bean, from Spring AI VectorStore Example. It needs the embedding model nomic-embed-text (ollama pull nomic-embed-text and the property spring.ai.ollama.embedding.options.model=nomic-embed-text).
5. PolicyIngestion.java and the policies folder: loads the four policy files into the vector store, from Spring AI ETL Pipeline Example. returns.md says “You can return most items within 10 days of delivery.”

Only the evaluation test in Section 5 needs items 3 to 5.

The SQL code and its unit test need a database. OrderDesk uses HSQLDB, a small Java database, so the pom.xml has its driver:

<dependency>
  <groupId>org.hsqldb</groupId>
  <artifactId>hsqldb</artifactId>
  <scope>runtime</scope>
</dependency>

The test also needs spring-boot-starter-jdbc for JdbcTemplate. OrderDesk gets it through its chat memory starter, spring-ai-starter-model-chat-memory-repository-jdbc. A project without chat memory adds spring-boot-starter-jdbc itself, as the text-to-SQL article shows.

The supportClient bean holds the store’s rules. One rule says that OrderDesk delivers only within India, and the integration test checks it:

@Configuration
public class ChatClientConfig {

    static final String SUPPORT_RULES = """
        You are OrderDesk, the support assistant of an online store.
        Answer in at most three sentences.
        Delivery: Pune, Mumbai and Delhi in 2 days; every other city in India in 5 days.
        Delivery is free above INR 499; below that it costs INR 49.
        We do not deliver outside India.
        """;

    @Bean
    ChatClient supportClient(ChatClient.Builder builder) {
        return builder
            .defaultSystem(SUPPORT_RULES)
            .build();
    }

3. Unit Tests with a Fake ChatModel

ChatModel is an interface with one method to implement: it takes a Prompt and returns a ChatResponse. So a lambda can play the model, and ChatClient.create(chatModel) builds a client on top of it without Spring.

Step 1: Fake the Model

OrderQueryController calls the model twice: once to write the SQL and once to explain the rows. Our fake model returns the next reply from a queue:

package com.howtodoinjava.orderdesk.sql;

import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;

import java.util.ArrayDeque;
import java.util.Deque;
import java.util.List;

import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.messages.AssistantMessage;
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.chat.model.ChatResponse;
import org.springframework.ai.chat.model.Generation;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.datasource.SimpleDriverDataSource;

// A unit test without a real model: a fake ChatModel returns fixed answers
class OrderQueryControllerTest {

    private final Deque<String> modelReplies = new ArrayDeque<>();
    private OrderQueryController controller;

    @BeforeEach
    void setUp() {
        ChatModel fakeModel = prompt -> new ChatResponse(
            List.of(new Generation(new AssistantMessage(modelReplies.pop()))));
        ChatClient fakeClient = ChatClient.create(fakeModel);

        JdbcTemplate jdbc = new JdbcTemplate(new SimpleDriverDataSource(
            new org.hsqldb.jdbc.JDBCDriver(), "jdbc:hsqldb:mem:unit", "SA", ""));
        jdbc.execute("DROP TABLE IF EXISTS orders");
        jdbc.execute("CREATE TABLE orders (order_id VARCHAR(10), city VARCHAR(30))");
        jdbc.update("INSERT INTO orders VALUES ('ORD-1031', 'Pune'), ('ORD-1032', 'Mumbai')");

        controller = new OrderQueryController(fakeClient, fakeClient, new OrderQueryService(jdbc));
    }

1. The lambda is the whole fake model. Each call takes the next text from modelReplies and wraps it in a ChatResponse.
2. The controller’s constructor takes two clients (sqlClient writes SQL, supportClient explains the rows) and the service. We pass the same fake client twice.
3. An in-memory HSQLDB table with two orders replaces the real database, so the SQL from the fake model really runs.

Without Spring, @Transactional(readOnly = true) on OrderQueryService has no effect, so this connection isn’t read-only. We test the read-only part separately (see the end of this section).

Step 2: Test the Code Around the Model

Each test queues the model’s replies and checks what our code does with them. The controller.ask() method returns an OrderAnswer record with the fields question, sql, rows, and answer:

    @Test
    void runsTheSqlAndExplainsTheRows() {
        modelReplies.add("```sql\nSELECT COUNT(*) FROM orders WHERE city = 'Pune';\n```");
        modelReplies.add("1 order went to Pune.");

        var answer = controller.ask("How many orders went to Pune?");

        assertThat(answer.sql()).isEqualTo("SELECT COUNT(*) FROM orders WHERE city = 'Pune'");
        assertThat(answer.rows()).hasSize(1);
        assertThat(answer.answer()).isEqualTo("1 order went to Pune.");
    }

    @Test
    void refusesSqlThatChangesData() {
        modelReplies.add("DELETE FROM orders");

        assertThatThrownBy(() -> controller.ask("Delete all orders"))
            .isInstanceOf(IllegalArgumentException.class)
            .hasMessageStartingWith("Only SELECT statements are allowed");
    }
}

1. The first test sends SQL wrapped in a Markdown code fence with a semicolon, as a real model often does. SqlGuard.clean() must strip both before the query runs.
2. The second test makes the “model” write a DELETE. SqlGuard.check() throws an IllegalArgumentException whose message starts with “Only SELECT statements are allowed”. A real model rarely does this on demand; a fake model makes it one line.

mvn test -Dtest=OrderQueryControllerTest

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.661 s -- in com.howtodoinjava.orderdesk.sql.OrderQueryControllerTest

This is the level to test the safety rules on. The SqlSafetyTest in the repo goes one step further: it proves that the read-only connection refuses a DELETE even if one got past SqlGuard.

4. Integration Tests with Testcontainers

A unit test can’t tell whether the real model follows our system prompt. For that, we start Ollama in Docker with Testcontainers, a library that starts containers from a JUnit test and stops them afterwards. This level needs Docker running on the build machine (docker info prints the server details when it is), but no Ollama.

Step 1: Add the Test Dependencies

<dependency>
  <groupId>org.springframework.ai</groupId>
  <artifactId>spring-ai-spring-boot-testcontainers</artifactId>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-testcontainers</artifactId>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.testcontainers</groupId>
  <artifactId>testcontainers-ollama</artifactId>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.testcontainers</groupId>
  <artifactId>testcontainers-junit-jupiter</artifactId>
  <scope>test</scope>
</dependency>

1. testcontainers-ollama has the OllamaContainer class, and testcontainers-junit-jupiter has the @Testcontainers and @Container annotations.
2. With spring-boot-testcontainers and spring-ai-spring-boot-testcontainers, @ServiceConnection sets Spring AI’s Ollama URL to the container. You don’t need a URL property.

The Spring AI BOM and the Spring Boot parent set all four versions.

Step 2: Start Ollama in a Container

package com.howtodoinjava.orderdesk.it;

import static org.assertj.core.api.Assertions.assertThat;

import java.io.IOException;

import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.boot.testcontainers.service.connection.ServiceConnection;
import org.testcontainers.DockerClientFactory;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import org.testcontainers.ollama.OllamaContainer;
import org.testcontainers.utility.DockerImageName;

// Starts Ollama in Docker, so the test needs no Ollama on the build machine
@SpringBootTest(properties = {
    "spring.ai.ollama.chat.options.model=" + SupportClientIT.MODEL,
    "spring.datasource.url=jdbc:hsqldb:mem:it",
    "spring.ai.mcp.server.enabled=false"})
@Testcontainers
class SupportClientIT {

    static final String MODEL = "qwen2.5:0.5b";
    // Ollama image with the model already inside, created on the first run
    static final String CACHED_IMAGE = "orderdesk-ollama:qwen2.5-0.5b";

    @Container
    @ServiceConnection
    static OllamaContainer ollama = new OllamaContainer(imageName());

    static DockerImageName imageName() {
        boolean cached = !DockerClientFactory.instance().client()
            .listImagesCmd().withImageNameFilter(CACHED_IMAGE).exec().isEmpty();
        return cached
            ? DockerImageName.parse(CACHED_IMAGE).asCompatibleSubstituteFor("ollama/ollama")
            : DockerImageName.parse("ollama/ollama:0.40.6");
    }

    @BeforeAll
    static void pullModelOnce() throws IOException, InterruptedException {
        if (ollama.getDockerImageName().equals(CACHED_IMAGE)) {
            return;
        }
        ollama.execInContainer("ollama", "pull", MODEL);
        ollama.commitToImage(CACHED_IMAGE);
    }

    @Autowired
    @Qualifier("supportClient")
    ChatClient supportClient;

    @Test
    void answersFromTheSupportRules() {
        String answer = supportClient.prompt()
            .user("Do you deliver to London?")
            .call()
            .content();

        System.out.println("Answer: " + answer);
        assertThat(answer).containsIgnoringCase("India");
    }
}

1. @Container starts the container once for the class, and @ServiceConnection sets Spring AI’s Ollama URL to the container’s random port.
2. A fresh Ollama image has no model, and downloading one takes minutes. So on the first run, pullModelOnce() pulls qwen2.5:0.5b and saves the container as a new local image with commitToImage(). Later runs start that image, with the model inside.
3. imageName() picks the image. listImagesCmd().withImageNameFilter(CACHED_IMAGE) asks Docker whether the saved image exists. asCompatibleSubstituteFor(“ollama/ollama”) tells Testcontainers that our own image is an Ollama image; OllamaContainer refuses other image names.
4. The property spring.ai.ollama.chat.options.model switches the model to the small qwen2.5:0.5b (about 400 MB) instead of the app’s qwen2.5:3b, to keep the container fast. The test checks the answer for one word, “India”, and not for an exact sentence.
5. The other two properties switch off parts of the full OrderDesk app that the test doesn’t need. The database becomes an in-memory one, and OrderDesk’s Model Context Protocol (MCP) server is turned off; it’s the endpoint at /mcp that offers OrderDesk’s tools to other AI apps. In a project without these features, leave them out.
6. The finished OrderDesk app has more than one ChatClient bean (supportClient, summaryClient, sqlClient, and others), so @Qualifier(“supportClient”) names the one we want.

The class name ends in IT. The mvn test command runs Maven’s Surefire plugin, which by default runs only classes whose names start with Test or end with Test, Tests, or TestCase. So mvn test skips the slow tests. Run them by name:

mvn test -Dtest=SupportClientIT

Answer: No, I do not deliver to London. I can only provide delivery options for cities in India, including Pune, Mumbai, and Delhi. If you're looking for delivery to London, I recommend contacting a local courier service or checking their website for more information.
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.62 s -- in com.howtodoinjava.orderdesk.it.SupportClientIT

The test passes because it checks only the fact that matters (India), not the wording.

5. Evaluating Answers with a Model as the Judge

An assertion on one word works for a short fact. To check whether a longer answer is true to the documents, Spring AI has evaluators: classes that send the answer to a model and ask it for a yes-or-no verdict.

1. FactCheckingEvaluator asks whether the answer (the claim) is supported by the documents.
2. RelevancyEvaluator asks whether the answer fits the question and the documents.

Both are in spring-ai-client-chat, which the Ollama starter brings.

Step 1: Write the Evaluation Test

The test checks answers about the store’s return policy. One test checks an answer built with retrieval-augmented generation (RAG), where the app first finds the policy sections that match the question and adds them to the prompt. The test uses Ollama on the local machine, with qwen2.5:3b both as the app’s model and as the judge. So Ollama must be running, with qwen2.5:3b and nomic-embed-text pulled:

package com.howtodoinjava.orderdesk.it;

import static org.assertj.core.api.Assertions.assertThat;

import java.util.List;

import org.junit.jupiter.api.Disabled;
import org.junit.jupiter.api.Test;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.client.ChatClientResponse;
import org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor;
import org.springframework.ai.chat.evaluation.FactCheckingEvaluator;
import org.springframework.ai.chat.evaluation.RelevancyEvaluator;
import org.springframework.ai.document.Document;
import org.springframework.ai.evaluation.EvaluationRequest;
import org.springframework.ai.evaluation.EvaluationResponse;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.test.context.SpringBootTest;

import com.howtodoinjava.orderdesk.etl.PolicyIngestion;

// Uses the Ollama on this machine (qwen2.5:3b) both to answer and to judge.
// Run with: mvn test -Dtest=RagAnswerEvaluationIT
@SpringBootTest(properties = {
    "spring.datasource.url=jdbc:hsqldb:mem:eval",
    "spring.ai.mcp.server.enabled=false"})
class RagAnswerEvaluationIT {

    static final String QUESTION = "How many days do I have to return a jacket?";

    @Autowired @Qualifier("supportClient") ChatClient supportClient;
    @Autowired QuestionAnswerAdvisor questionAnswerAdvisor;
    @Autowired PolicyIngestion policyIngestion;
    @Autowired ChatClient.Builder judgeBuilder;

    // The evaluators compare the reply with "yes" exactly, so the judge must
    // answer with one word; a small model adds an explanation otherwise
    ChatClient.Builder judge() {
        return judgeBuilder.clone()
            .defaultSystem("Reply with one word only: yes or no. No explanation.")
            .defaultOptions(org.springframework.ai.ollama.api.OllamaChatOptions.builder()
                .temperature(0.0));
    }

    @Test
    @Disabled("qwen2.5:3b answers NO even for a correct answer; use a larger judge model")
    void ragAnswerIsRelevantToTheRetrievedPolicy() throws Exception {
        policyIngestion.ingest();
        ChatClientResponse response = supportClient.prompt()
            .user(QUESTION)
            .advisors(questionAnswerAdvisor)
            .call()
            .chatClientResponse();
        String answer = response.chatResponse().getResult().getOutput().getText();
        @SuppressWarnings("unchecked")
        List<Document> documents = (List<Document>) response.context()
            .get(QuestionAnswerAdvisor.RETRIEVED_DOCUMENTS);

        EvaluationResponse relevancy = new RelevancyEvaluator(judge())
            .evaluate(new EvaluationRequest(QUESTION, documents, answer));

        System.out.println("Answer: " + answer);
        System.out.println("Relevant: " + relevancy.isPass());
        assertThat(relevancy.isPass()).isTrue();
    }

    @Test
    void factCheckCatchesAnInventedPolicy() throws Exception {
        List<Document> policy = List.of(new Document(
            "Return window\nYou can return most items within 10 days of delivery."));
        FactCheckingEvaluator factChecker = FactCheckingEvaluator.builder(judge()).build();

        EvaluationResponse right = factChecker.evaluate(new EvaluationRequest(
            QUESTION, policy, "You have 10 days from delivery to return the jacket."));
        EvaluationResponse wrong = factChecker.evaluate(new EvaluationRequest(
            QUESTION, policy, "You have 30 days from delivery to return the jacket."));

        System.out.println("10 days supported: " + right.isPass());
        System.out.println("30 days supported: " + wrong.isPass());
        assertThat(right.isPass()).isTrue();
        assertThat(wrong.isPass()).isFalse();
    }
}

1. @SpringBootTest starts the whole app with the same two properties as SupportClientIT, but without a container.
2. judgeBuilder is the auto-configured ChatClient.Builder, so the judge uses the app’s model, qwen2.5:3b. judge() changes a copy of it (Step 2).
3. ragAnswerIsRelevantToTheRetrievedPolicy() loads the policy files with PolicyIngestion and asks the question with questionAnswerAdvisor. The advisor puts the policy sections it found into the response context under QuestionAnswerAdvisor.RETRIEVED_DOCUMENTS. RelevancyEvaluator then gets the question, those sections and the answer. This test is disabled (Step 3).
4. factCheckCatchesAnInventedPolicy() gives FactCheckingEvaluator one policy Document and two claims. We test a right claim and a wrong claim. If we tested only the wrong one, a judge that always says “no” would also pass.

Step 2: Make the Judge Answer in One Word

With FactCheckingEvaluator.builder(judgeBuilder), without judge(), both claims fail:

10 days supported: false
30 days supported: false

To see the judge’s raw reply, we added Spring AI’s SimpleLoggerAdvisor to the judge’s builder (.defaultAdvisors(new SimpleLoggerAdvisor())). It logs every request and reply at DEBUG level, which the property logging.level.org.springframework.ai.chat.client.advisor=DEBUG turns on. The judge did say yes to the right claim, but with an explanation:

"text" : "Yes\n\nThe document states that you can return most items within 10 days of delivery. While the document does not specifically mention \"jackets,\" it covers a wide range of items,

FactCheckingEvaluator compares the whole reply with “yes” using equalsIgnoreCase(), so any extra word makes it a fail:

The judge() method fixes it with two settings on a copy of the builder: a system prompt that allows one word, and a temperature of 0 for a steady verdict. Calling clone() keeps the app’s own ChatClient.Builder unchanged.

In Spring AI 2.0, defaultOptions() takes a ChatOptions.Builder, so the options need no build() call.

mvn test -Dtest=RagAnswerEvaluationIT

10 days supported: true
30 days supported: false
Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 8.381 s -- in com.howtodoinjava.orderdesk.it.RagAnswerEvaluationIT

The skipped test is the relevancy check.

Step 3: Know When the Judge Is Too Weak

Before we disabled it, the relevancy test printed a right answer and a “no” from the judge:

Answer: The return window for most items is within 10 days of delivery. If you choose to return the jacket, you should follow the return process within this timeframe. For a refund, the process typically takes 5 to 7 working days after we receive the item. If you prefer store credit, it will be added to your OrderDesk wallet at once. If you have any specific questions or need further assistance, feel free to ask.
Relevant: false

The judge’s raw reply was the single word “NO”. So the reply format was not the problem; qwen2.5:3b judged the answer wrong. That’s why the test has @Disabled with the reason in the code.

A judge must be better than the model it judges, or at least as good. A 3B model is fine for a fact check against one short document, but too weak for relevancy. Use a larger model as the judge: pull it with Ollama and add .model() with its name to the OllamaChatOptions builder in judge(). The app keeps answering with qwen2.5:3b.

6. What to Run in CI

Each level fits a different place in the build:

Level	Needs	Time in OrderDesk	When to run
Unit tests with a fake ChatModel	Nothing	Under 1 second	Every build: mvn test
Integration tests with Testcontainers	Docker	About 16 seconds (cached image)	Every pull request
Evaluation tests	A capable judge model	About 8 seconds for one fact check	Nightly, or after a prompt or model change

The mvn test command already leaves the IT classes out. Run them in a separate CI step, with a pattern for the class names:

mvn test -Dtest='*IT'

Evaluation results are a trend, not a gate: a failure means “look at this answer”, not always “the code is broken”.

7. Common Traps

7.1. A Talkative Judge

When every evaluator check fails, look at the judge’s raw reply with SimpleLoggerAdvisor (Section 5, Step 2).

7.2. The First Testcontainers Run in CI

The first run of SupportClientIT pulls the Ollama image and the model, which takes minutes and needs internet access from Docker. In CI, cache the committed image orderdesk-ollama:qwen2.5-0.5b between builds, so every later run starts with the model inside.

8. FAQs

8.1. Can I use Mockito instead of a lambda?

Yes. ChatModel is an interface, so Mockito.mock(ChatModel.class) with when(…).thenReturn(…) works too. The lambda is shorter when the test needs only a list of replies.

8.2. Does Spring AI have a test module?

Yes, spring-ai-test has helpers for evaluation tests. OrderDesk uses only the evaluators in spring-ai-client-chat and plain JUnit.

8.3. Can the integration test use a hosted model instead of Docker?

Yes, with an API key in the CI secrets. But each run then costs money, and a provider’s outage fails the build. Ollama in Docker costs nothing and needs no network after the first pull.

8.4. Why does the test use a different model than the app?

Speed. The qwen2.5:0.5b model starts and answers faster than qwen2.5:3b, and SupportClientIT checks only that the wiring and the system prompt work. The evaluation test answers with the app’s real model, qwen2.5:3b, because it checks answer quality.

9. Conclusion

A Spring AI app is mostly ordinary code, and a fake ChatModel tests that code in under a second. Testcontainers adds a real model for the wiring, and evaluators let a second model judge the answers.

The one trap to remember is the judge. An evaluator is only as good as the model behind it, so check it with a right and a wrong answer before you trust it.

All articles in order: Spring AI Tutorial

10. References

• Spring AI Reference: Evaluation Testing
• Spring AI Reference: Testcontainers
• Testcontainers: Ollama Module

Drop me your questions related to testing Spring AI apps in the comments.

Happy Learning !!

Source Code on Github