Testing Spring AI: Unit Tests, Testcontainers and Evaluators

A Spring AI application needs checks for two different things: the Java code that controls model access, and the quality of generated answers. These should not rely on the same testing strategy.

This article tests OrderDesk with deterministic model responses, an Ollama integration in Testcontainers, and model-based evaluators. We explain what each level proves, how the recorded results were obtained, and why an evaluator’s verdict can itself be wrong.

1. Separate Application Tests from Model Evaluation

Exact-string assertions are often unsuitable for generated prose, but most surrounding behavior is ordinary Java. Test prompt construction, parsing, branching, and database safeguards with controlled responses. Use real-model tests to investigate the behavior that a fake cannot establish.

So the tests form three levels:

1. Unit tests replace the ChatModel with a lambda. They test our code in milliseconds and give the same result every time.
2. Integration tests start a real model in a Docker container. They check that the beans, the prompt, and the model work together.
3. Evaluation tests ask a model to assess an answer against specified criteria or evidence. They add a quality signal rather than proving that the answer is correct.

Keep routine control-flow tests fast and repeatable. Run a smaller set of real-model and evaluation checks where their additional cost and variability are justified.

2. The Example App

This article adds tests to OrderDesk, the app of this series. The tests are in src/test/java and belong to the com.howtodoinjava.orderdesk.sql and com.howtodoinjava.orderdesk.it packages. The full app is on GitHub in the orderdesk folder. The basic setup (Ollama, the chat model qwen2.5:3b, and the Spring Boot project) is in Getting Started with Spring AI.

The example uses spring-boot-starter-test for its test framework and assertion libraries. Add it when creating a project that does not already include test dependencies. The integration setup adds the four dependencies shown in Section 4.

2.1. The Code Under Test

The tests check code from earlier articles. Copy these files from GitHub; this article doesn’t explain them again:

1. OrderQueryController.java, OrderQueryService.java, and SqlGuard.java: the text-to-SQL feature from Query a Database with Natural Language using Spring AI. The model writes SQL, SqlGuard cleans it and allows only one SELECT, and OrderQueryService runs it. The unit test in Section 3 tests these three classes.
2. ChatClientConfig.java: the supportClient bean from Getting Started, shown below. The integration test in Section 4 calls it.
3. RagConfig.java: the questionAnswerAdvisor bean, from Spring AI RAG Advisors: QuestionAnswer vs RetrievalAugmentation. It needs the spring-ai-vector-store-advisor dependency.
4. SimpleVectorStoreConfig.java: the VectorStore bean, from Spring AI VectorStore Example. It needs the embedding model nomic-embed-text (ollama pull nomic-embed-text and the property spring.ai.ollama.embedding.options.model=nomic-embed-text).
5. PolicyIngestion.java and the policies folder: loads the four policy files into the vector store, from Spring AI ETL Pipeline Example. returns.md says “You can return most items within 10 days of delivery.”

Only the evaluation test in Section 5 needs items 3 to 5.

The SQL code and its unit test need a database. OrderDesk uses HSQLDB, a small Java database, so the pom.xml has its driver:

<dependency>
  <groupId>org.hsqldb</groupId>
  <artifactId>hsqldb</artifactId>
  <scope>runtime</scope>
</dependency>

The test also needs spring-boot-starter-jdbc for JdbcTemplate. OrderDesk gets it through its chat memory starter, spring-ai-starter-model-chat-memory-repository-jdbc. A project without chat memory adds spring-boot-starter-jdbc itself, as the text-to-SQL article shows.

The supportClient bean holds the store’s rules. One rule says that OrderDesk delivers only within India, and the integration test checks it:

@Configuration
public class ChatClientConfig {

    static final String SUPPORT_RULES = """
        You are OrderDesk, the support assistant of an online store.
        Answer in at most three sentences.
        Delivery: Pune, Mumbai and Delhi in 2 days; every other city in India in 5 days.
        Delivery is free above INR 499; below that it costs INR 49.
        We do not deliver outside India.
        """;

    @Bean
    ChatClient supportClient(ChatClient.Builder builder) {
        return builder
            .defaultSystem(SUPPORT_RULES)
            .build();
    }

3. Test Java Behavior with a Controlled Model

ChatModel is an interface with one method to implement: it takes a Prompt and returns a ChatResponse. So a lambda can play the model, and ChatClient.create(chatModel) builds a client on top of it without Spring.

Step 1: Fake the Model

OrderQueryController calls the model twice: once to write the SQL and once to explain the rows. Our fake model returns the next reply from a queue:

package com.howtodoinjava.orderdesk.sql;

import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;

import java.util.ArrayDeque;
import java.util.Deque;
import java.util.List;

import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.messages.AssistantMessage;
import org.springframework.ai.chat.model.ChatModel;
import org.springframework.ai.chat.model.ChatResponse;
import org.springframework.ai.chat.model.Generation;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.jdbc.datasource.SimpleDriverDataSource;

// A unit test without a real model: a fake ChatModel returns fixed answers
class OrderQueryControllerTest {

    private final Deque<String> modelReplies = new ArrayDeque<>();
    private OrderQueryController controller;

    @BeforeEach
    void setUp() {
        ChatModel fakeModel = prompt -> new ChatResponse(
            List.of(new Generation(new AssistantMessage(modelReplies.pop()))));
        ChatClient fakeClient = ChatClient.create(fakeModel);

        JdbcTemplate jdbc = new JdbcTemplate(new SimpleDriverDataSource(
            new org.hsqldb.jdbc.JDBCDriver(), "jdbc:hsqldb:mem:unit", "SA", ""));
        jdbc.execute("DROP TABLE IF EXISTS orders");
        jdbc.execute("CREATE TABLE orders (order_id VARCHAR(10), city VARCHAR(30))");
        jdbc.update("INSERT INTO orders VALUES ('ORD-1031', 'Pune'), ('ORD-1032', 'Mumbai')");

        controller = new OrderQueryController(fakeClient, fakeClient, new OrderQueryService(jdbc));
    }

1. The lambda is the whole fake model. Each call takes the next text from modelReplies and wraps it in a ChatResponse.
2. The controller’s constructor takes two clients (sqlClient writes SQL, supportClient explains the rows) and the service. We pass the same fake client twice.
3. An in-memory HSQLDB table with two orders replaces the real database, so the SQL from the fake model really runs.

Without Spring, @Transactional(readOnly = true) on OrderQueryService has no effect, so this connection isn’t read-only. We test the read-only part separately (see the end of this section).

Step 2: Test the Code Around the Model

Each test queues the model’s replies and checks what our code does with them. The controller.ask() method returns an OrderAnswer record with the fields question, sql, rows, and answer:

    @Test
    void runsTheSqlAndExplainsTheRows() {
        modelReplies.add("```sql\nSELECT COUNT(*) FROM orders WHERE city = 'Pune';\n```");
        modelReplies.add("1 order went to Pune.");

        var answer = controller.ask("How many orders went to Pune?");

        assertThat(answer.sql()).isEqualTo("SELECT COUNT(*) FROM orders WHERE city = 'Pune'");
        assertThat(answer.rows()).hasSize(1);
        assertThat(answer.answer()).isEqualTo("1 order went to Pune.");
    }

    @Test
    void refusesSqlThatChangesData() {
        modelReplies.add("DELETE FROM orders");

        assertThatThrownBy(() -> controller.ask("Delete all orders"))
            .isInstanceOf(IllegalArgumentException.class)
            .hasMessageStartingWith("Only SELECT statements are allowed");
    }
}

1. The first test sends SQL wrapped in a Markdown code fence with a semicolon, as a real model often does. SqlGuard.clean() must strip both before the query runs.
2. The second test makes the “model” write a DELETE. SqlGuard.check() throws an IllegalArgumentException whose message starts with “Only SELECT statements are allowed”. A real model rarely does this on demand; a fake model makes it one line.

mvn test -Dtest=OrderQueryControllerTest

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.661 s -- in com.howtodoinjava.orderdesk.sql.OrderQueryControllerTest

Use controlled model responses to exercise rejection paths deliberately. A fake can return a DELETE every time, allowing the test to check the guard independently of a model’s willingness to produce it. The repository also includes SqlSafetyTest for the separate database-level safeguard.

4. Check Real Model Integration with Testcontainers

A fake cannot establish that the real model endpoint, configuration, and prompt work together. This example uses Testcontainers to start Ollama in Docker. Docker must be available, but a separately installed Ollama process is unnecessary for this test.

Step 1: Add the Test Dependencies

<dependency>
  <groupId>org.springframework.ai</groupId>
  <artifactId>spring-ai-spring-boot-testcontainers</artifactId>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-testcontainers</artifactId>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.testcontainers</groupId>
  <artifactId>testcontainers-ollama</artifactId>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.testcontainers</groupId>
  <artifactId>testcontainers-junit-jupiter</artifactId>
  <scope>test</scope>
</dependency>

1. testcontainers-ollama has the OllamaContainer class, and testcontainers-junit-jupiter has the @Testcontainers and @Container annotations.
2. With spring-boot-testcontainers and spring-ai-spring-boot-testcontainers, @ServiceConnection sets Spring AI’s Ollama URL to the container. You don’t need a URL property.

The Spring AI BOM and the Spring Boot parent set all four versions.

Step 2: Start Ollama in a Container

package com.howtodoinjava.orderdesk.it;

import static org.assertj.core.api.Assertions.assertThat;

import java.io.IOException;

import org.junit.jupiter.api.BeforeAll;
import org.junit.jupiter.api.Test;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.boot.testcontainers.service.connection.ServiceConnection;
import org.testcontainers.DockerClientFactory;
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.junit.jupiter.Testcontainers;
import org.testcontainers.ollama.OllamaContainer;
import org.testcontainers.utility.DockerImageName;

// Starts Ollama in Docker, so the test needs no Ollama on the build machine
@SpringBootTest(properties = {
    "spring.ai.ollama.chat.options.model=" + SupportClientIT.MODEL,
    "spring.datasource.url=jdbc:hsqldb:mem:it",
    "spring.ai.mcp.server.enabled=false"})
@Testcontainers
class SupportClientIT {

    static final String MODEL = "qwen2.5:0.5b";
    // Ollama image with the model already inside, created on the first run
    static final String CACHED_IMAGE = "orderdesk-ollama:qwen2.5-0.5b";

    @Container
    @ServiceConnection
    static OllamaContainer ollama = new OllamaContainer(imageName());

    static DockerImageName imageName() {
        boolean cached = !DockerClientFactory.instance().client()
            .listImagesCmd().withImageNameFilter(CACHED_IMAGE).exec().isEmpty();
        return cached
            ? DockerImageName.parse(CACHED_IMAGE).asCompatibleSubstituteFor("ollama/ollama")
            : DockerImageName.parse("ollama/ollama:0.40.6");
    }

    @BeforeAll
    static void pullModelOnce() throws IOException, InterruptedException {
        if (ollama.getDockerImageName().equals(CACHED_IMAGE)) {
            return;
        }
        ollama.execInContainer("ollama", "pull", MODEL);
        ollama.commitToImage(CACHED_IMAGE);
    }

    @Autowired
    @Qualifier("supportClient")
    ChatClient supportClient;

    @Test
    void answersFromTheSupportRules() {
        String answer = supportClient.prompt()
            .user("Do you deliver to London?")
            .call()
            .content();

        System.out.println("Answer: " + answer);
        assertThat(answer).containsIgnoringCase("India");
    }
}

1. @Container starts the container once for the class, and @ServiceConnection sets Spring AI’s Ollama URL to the container’s random port.
2. A fresh Ollama image has no model, and downloading one takes minutes. So on the first run, pullModelOnce() pulls qwen2.5:0.5b and saves the container as a new local image with commitToImage(). Later runs start that image, with the model inside.
3. imageName() picks the image. listImagesCmd().withImageNameFilter(CACHED_IMAGE) asks Docker whether the saved image exists. asCompatibleSubstituteFor(“ollama/ollama”) tells Testcontainers that our own image is an Ollama image; OllamaContainer refuses other image names.
4. The property spring.ai.ollama.chat.options.model switches the model to the small qwen2.5:0.5b (about 400 MB) instead of the app’s qwen2.5:3b, to keep the container fast. The test checks the answer for one word, “India”, and not for an exact sentence.
5. The other two properties switch off parts of the full OrderDesk app that the test doesn’t need. The database becomes an in-memory one, and OrderDesk’s Model Context Protocol (MCP) server is turned off; it’s the endpoint at /mcp that offers OrderDesk’s tools to other AI apps. In a project without these features, leave them out.
6. The finished OrderDesk app has more than one ChatClient bean (supportClient, summaryClient, sqlClient, and others), so @Qualifier(“supportClient”) names the one we want.

The class name ends in IT. The mvn test command runs Maven’s Surefire plugin, which by default runs only classes whose names start with Test or end with Test, Tests, or TestCase. So mvn test skips the slow tests. Run them by name:

mvn test -Dtest=SupportClientIT

Answer: No, I do not deliver to London. I can only provide delivery options for cities in India, including Pune, Mumbai, and Delhi. If you're looking for delivery to London, I recommend contacting a local courier service or checking their website for more information.
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.62 s -- in com.howtodoinjava.orderdesk.it.SupportClientIT

Checking for “India” avoids depending on exact wording, but it remains a narrow assertion: a response could contain that word and still be wrong. Treat this as a wiring check, not a complete evaluation of delivery-policy accuracy.

5. Evaluate Generated Answers Against Evidence

Longer answers need more than a keyword check. Spring AI evaluators can send the question, evidence, and response to a judge model. Keep the evaluation criterion explicit and verify the judge on examples whose expected outcomes are known.

1. FactCheckingEvaluator asks whether the answer (the claim) is supported by the documents.
2. RelevancyEvaluator asks whether the answer fits the question and the documents.

Both are in spring-ai-client-chat, which the Ollama starter brings.

Step 1: Write the Evaluation Test

The test checks answers about the store’s return policy. One test checks an answer built with retrieval-augmented generation (RAG), where the app first finds the policy sections that match the question and adds them to the prompt. The test uses Ollama on the local machine, with qwen2.5:3b both as the app’s model and as the judge. So Ollama must be running, with qwen2.5:3b and nomic-embed-text pulled:

package com.howtodoinjava.orderdesk.it;

import static org.assertj.core.api.Assertions.assertThat;

import java.util.List;

import org.junit.jupiter.api.Disabled;
import org.junit.jupiter.api.Test;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.client.ChatClientResponse;
import org.springframework.ai.chat.client.advisor.vectorstore.QuestionAnswerAdvisor;
import org.springframework.ai.chat.evaluation.FactCheckingEvaluator;
import org.springframework.ai.chat.evaluation.RelevancyEvaluator;
import org.springframework.ai.document.Document;
import org.springframework.ai.evaluation.EvaluationRequest;
import org.springframework.ai.evaluation.EvaluationResponse;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.test.context.SpringBootTest;

import com.howtodoinjava.orderdesk.etl.PolicyIngestion;

// Uses the Ollama on this machine (qwen2.5:3b) both to answer and to judge.
// Run with: mvn test -Dtest=RagAnswerEvaluationIT
@SpringBootTest(properties = {
    "spring.datasource.url=jdbc:hsqldb:mem:eval",
    "spring.ai.mcp.server.enabled=false"})
class RagAnswerEvaluationIT {

    static final String QUESTION = "How many days do I have to return a jacket?";

    @Autowired @Qualifier("supportClient") ChatClient supportClient;
    @Autowired QuestionAnswerAdvisor questionAnswerAdvisor;
    @Autowired PolicyIngestion policyIngestion;
    @Autowired ChatClient.Builder judgeBuilder;

    // The evaluators compare the reply with "yes" exactly, so the judge must
    // answer with one word; a small model adds an explanation otherwise
    ChatClient.Builder judge() {
        return judgeBuilder.clone()
            .defaultSystem("Reply with one word only: yes or no. No explanation.")
            .defaultOptions(org.springframework.ai.ollama.api.OllamaChatOptions.builder()
                .temperature(0.0));
    }

    @Test
    @Disabled("qwen2.5:3b answers NO even for a correct answer; use a larger judge model")
    void ragAnswerIsRelevantToTheRetrievedPolicy() throws Exception {
        policyIngestion.ingest();
        ChatClientResponse response = supportClient.prompt()
            .user(QUESTION)
            .advisors(questionAnswerAdvisor)
            .call()
            .chatClientResponse();
        String answer = response.chatResponse().getResult().getOutput().getText();
        @SuppressWarnings("unchecked")
        List<Document> documents = (List<Document>) response.context()
            .get(QuestionAnswerAdvisor.RETRIEVED_DOCUMENTS);

        EvaluationResponse relevancy = new RelevancyEvaluator(judge())
            .evaluate(new EvaluationRequest(QUESTION, documents, answer));

        System.out.println("Answer: " + answer);
        System.out.println("Relevant: " + relevancy.isPass());
        assertThat(relevancy.isPass()).isTrue();
    }

    @Test
    void factCheckCatchesAnInventedPolicy() throws Exception {
        List<Document> policy = List.of(new Document(
            "Return window\nYou can return most items within 10 days of delivery."));
        FactCheckingEvaluator factChecker = FactCheckingEvaluator.builder(judge()).build();

        EvaluationResponse right = factChecker.evaluate(new EvaluationRequest(
            QUESTION, policy, "You have 10 days from delivery to return the jacket."));
        EvaluationResponse wrong = factChecker.evaluate(new EvaluationRequest(
            QUESTION, policy, "You have 30 days from delivery to return the jacket."));

        System.out.println("10 days supported: " + right.isPass());
        System.out.println("30 days supported: " + wrong.isPass());
        assertThat(right.isPass()).isTrue();
        assertThat(wrong.isPass()).isFalse();
    }
}

1. @SpringBootTest starts the whole app with the same two properties as SupportClientIT, but without a container.
2. judgeBuilder is the auto-configured ChatClient.Builder, so the judge uses the app’s model, qwen2.5:3b. judge() changes a copy of it (Step 2).
3. ragAnswerIsRelevantToTheRetrievedPolicy() loads the policy files with PolicyIngestion and asks the question with questionAnswerAdvisor. The advisor puts the policy sections it found into the response context under QuestionAnswerAdvisor.RETRIEVED_DOCUMENTS. RelevancyEvaluator then gets the question, those sections and the answer. This test is disabled (Step 3).
4. factCheckCatchesAnInventedPolicy() gives FactCheckingEvaluator one policy Document and two claims. We test a right claim and a wrong claim. If we tested only the wrong one, a judge that always says “no” would also pass.

Step 2: Make the Judge Answer in One Word

With FactCheckingEvaluator.builder(judgeBuilder), without judge(), both claims fail:

10 days supported: false
30 days supported: false

To see the judge’s raw reply, we added Spring AI’s SimpleLoggerAdvisor to the judge’s builder (.defaultAdvisors(new SimpleLoggerAdvisor())). It logs every request and reply at DEBUG level, which the property logging.level.org.springframework.ai.chat.client.advisor=DEBUG turns on. The judge did say yes to the right claim, but with an explanation:

"text" : "Yes\n\nThe document states that you can return most items within 10 days of delivery. While the document does not specifically mention \"jackets,\" it covers a wide range of items,

FactCheckingEvaluator compares the whole reply with “yes” using equalsIgnoreCase(), so any extra word makes it a fail:

The judge() method adds a one-word instruction and sets temperature to zero on a cloned builder. In the recorded fact-check examples, this resolves the response-format problem. It does not establish that the judge will reliably follow the format or assess every future claim correctly.

In Spring AI 2.0, defaultOptions() takes a ChatOptions.Builder, so the options need no build() call.

mvn test -Dtest=RagAnswerEvaluationIT

10 days supported: true
30 days supported: false
Tests run: 2, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 8.381 s -- in com.howtodoinjava.orderdesk.it.RagAnswerEvaluationIT

The skipped test is the relevancy check.

Step 3: Know When the Judge Is Too Weak

Before we disabled it, the relevancy test printed a right answer and a “no” from the judge:

Answer: The return window for most items is within 10 days of delivery. If you choose to return the jacket, you should follow the return process within this timeframe. For a refund, the process typically takes 5 to 7 working days after we receive the item. If you prefer store credit, it will be added to your OrderDesk wallet at once. If you have any specific questions or need further assistance, feel free to ask.
Relevant: false

The judge’s raw reply was the single word “NO”. So the reply format was not the problem; qwen2.5:3b judged the answer wrong. That’s why the test has @Disabled with the reason in the code.

Select the judge by measuring its agreement with reliable expected judgments on your task. A larger or different model may improve results, but size alone does not establish evaluator quality. This example’s failed relevancy check demonstrates a limitation of the chosen setup, rather than a universal rule about all 3B models.

6. What to Run in CI

Each level fits a different place in the build:

Level	Needs	Time in OrderDesk	When to run
Unit tests with a fake ChatModel	Nothing	Under 1 second	Every build: mvn test
Integration tests with Testcontainers	Docker	About 16 seconds (cached image)	Every pull request
Evaluation tests	A capable judge model	About 8 seconds for one fact check	Nightly, or after a prompt or model change

The mvn test command already leaves the IT classes out. Run them in a separate CI step, with a pattern for the class names:

mvn test -Dtest='*IT'

Decide how evaluation results affect delivery based on the judge’s reliability and the consequence of an error. Some checks can be monitored as trends; a validated evaluation can also support a release gate. Investigate failures rather than assuming they always indicate either broken code or a bad judge.

7. Common Traps

7.1. A Talkative Judge

When every evaluator check fails, look at the judge’s raw reply with SimpleLoggerAdvisor (Section 5, Step 2).

7.2. The First Testcontainers Run in CI

The first run of SupportClientIT pulls the Ollama image and the model, which takes minutes and needs internet access from Docker. In CI, cache the committed image orderdesk-ollama:qwen2.5-0.5b between builds, so every later run starts with the model inside.

8. FAQs

8.1. Can I use Mockito instead of a lambda?

Yes. ChatModel is an interface, so Mockito.mock(ChatModel.class) with when(…).thenReturn(…) works too. The lambda is shorter when the test needs only a list of replies.

8.2. Does Spring AI have a test module?

Yes, spring-ai-test has helpers for evaluation tests. OrderDesk uses only the evaluators in spring-ai-client-chat and plain JUnit.

8.3. Can the integration test use a hosted model instead of Docker?

Yes, with valid provider access and credentials supplied through CI secrets. Consider request cost, quotas, data handling, and network failures. Local Ollama avoids a hosted inference charge, but Docker resources and model downloads still have operational costs.

8.4. Why does the test use a different model than the app?

Speed. The qwen2.5:0.5b model starts and answers faster than qwen2.5:3b, and SupportClientIT checks only that the wiring and the system prompt work. The evaluation test answers with the app’s real model, qwen2.5:3b, because it checks answer quality.

9. Conclusion

Use a fake ChatModel to verify deterministic Java behavior, Testcontainers to check the real model connection, and evaluators to assess generated answers against explicit criteria. Each layer answers a different question.

Validate the evaluator as well as the application. Include correct and incorrect reference answers, inspect unexpected verdicts, and preserve the evidence needed to understand failures.

All articles in order: Spring AI Tutorial

10. References

• Spring AI Reference: Evaluation Testing
• Spring AI Reference: Testcontainers
• Testcontainers: Ollama Module

Drop me your questions related to testing Spring AI apps in the comments.

Happy Learning !!

Source Code on Github