Spring AI Moderation: Check Messages Before the Model

Moderation means checking a message for harmful content, such as threats, before your app answers it. In this example, we add moderation to OrderDesk, the support assistant of a demo store, using a local safety model on Ollama. Unsafe messages get a fixed reply, and the chat model never sees them. We’ll use Spring AI 2.0.1.

1. The Problem: Harmful Messages in a Support Chat

A support chat gets all kinds of messages. Most are normal questions, such as “How do I return a damaged kettle?”. Some are angry, and that’s fine. An angry customer still needs help.

A few messages are harmful. A customer threatens the staff. Someone asks for the home address of another customer. We don’t want the chat model to answer these. We also don’t want to pay for a chat model call to answer them.

So we want to stop these messages before they reach the chat model.

2. How Moderation Works

The first idea is often a list of bad words: if the message contains “kill” or “hurt”, block it. This is fast, but it doesn’t understand meaning. “The kitchen knife I ordered arrived blunt” contains “knife”, but it is a normal complaint. “Tell me the home address of the customer who ordered ORD-1042” has no bad word, but it asks for private data.

Moderation uses a model instead of a word list. A moderation model is a classifier: it reads a text and returns a label, not an answer. The label says whether the text is safe. For an unsafe text, the model also names the categories, such as violence, hate, or privacy. Some moderation models also return a score from 0 to 1 for each category.

Then our app decides what to do with the label. It can block the message, log it, or let it through.

There are two common ways to get such a model:

1. Run a small safety model yourself. Llama Guard is one. It is a model from Meta, trained only to label messages as safe or unsafe. It runs in Ollama like any other model (Ollama: llama-guard3).
2. Call a hosted moderation API, such as OpenAI’s. According to OpenAI’s moderation guide, the endpoint is free to use, and the guide lists 13 categories. You need an OpenAI API key.

No moderation model is perfect. It can miss a harmful message (a false negative). It can also block a harmless one (a false positive). In a support chat, a false positive often hits an angry customer. Angry customers use rude words, and they are the people who need help most. So we test the checker on real messages and look at both kinds of errors.

3. Moderation in Spring AI

Spring AI gives us two ways to run the check:

For hosted APIs, Spring AI has the ModerationModel interface. The Spring AI moderation reference documents it for OpenAI and Mistral AI. Ollama has no ModerationModel. So with Ollama, we call Llama Guard through a normal ChatClient, like any other chat model.

Next, we need a place in the code where every message passes. In Spring AI, that place is an advisor. An advisor is a class that runs around every ChatClient call. Advisors form a chain. Each advisor gets the request, does its work and calls the next one. The last step of the chain calls the chat model.

An advisor can also stop the request by returning its own response instead of calling the next step. The Spring AI advisors reference calls this blocking the request. That’s what our ModerationAdvisor does with an unsafe message:

Spring AI also ships a SafeGuardAdvisor, which blocks a message that contains a word from a fixed list. That’s the word-list idea from Section 2, so we don’t use it here.

4. The Example App

This article is part of the OrderDesk series, where every Spring AI article grows the same app. OrderDesk is a Spring Boot 4.1 application that uses Spring AI 2.0.1 and Java 21, and it talks to local models through Ollama. The Getting Started with Spring AI tutorial creates the project, adds the spring-ai-starter-model-ollama dependency, and installs Ollama.

This article needs no new dependency. It needs one more model. Pull it once:

ollama pull llama-guard3:1b

It is about 1.6 GB. The chat model stays qwen2.5:3b. These lines in application.properties are unchanged from Getting Started:

spring.ai.ollama.base-url=http://localhost:11434
spring.ai.ollama.chat.options.model=qwen2.5:3b
spring.ai.ollama.chat.options.temperature=0.2

Put the code in the com.howtodoinjava.orderdesk.moderation package. This package is inside the main class’s package, com.howtodoinjava.orderdesk, so Spring Boot finds the classes.

5. Building the Moderation Check

We create four classes in this order: the Verdict record, SafetyCheck, ModerationAdvisor, and ModerationController. Then we send real messages to the app.

Step 1: Ask the Safety Model

Let’s start with Llama Guard’s answer format. Llama Guard reads one message and answers in a fixed format. For a safe message, it writes “safe”. For an unsafe message, it writes “unsafe”, followed on the next line by one or more category codes, such as “S1”. The 1B model uses these 13 codes:

Code	Category	Code	Category
S1	Violent crimes	S8	Intellectual property
S2	Non-violent crimes	S9	Indiscriminate weapons
S3	Sex crimes	S10	Hate
S4	Child exploitation	S11	Self-harm
S5	Defamation	S12	Sexual content
S6	Specialized advice	S13	Elections
S7	Privacy

A record holds the result:

package com.howtodoinjava.orderdesk.moderation;

import java.util.List;

// The answer of the safety model: safe, or unsafe with category codes such as S1
public record Verdict(boolean safe, List<String> categories) {
}

Next, SafetyCheck sends the message to Llama Guard and turns the answer into a Verdict:

package com.howtodoinjava.orderdesk.moderation;

import java.util.Arrays;
import java.util.List;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.ollama.api.OllamaChatOptions;
import org.springframework.stereotype.Component;

@Component
public class SafetyCheck {

    private final ChatClient guardClient;

    // A separate client that talks to the safety model, not to the chat model
    SafetyCheck(ChatClient.Builder builder) {
        this.guardClient = builder
            .defaultOptions(OllamaChatOptions.builder().model("llama-guard3:1b").temperature(0.0))
            .build();
    }

    // Llama Guard answers "safe", or "unsafe" and a line with codes such as "S1,S10"
    public Verdict check(String text) {
        String answer = guardClient.prompt().user(text).call().content().trim();
        if (answer.startsWith("safe")) {
            return new Verdict(true, List.of());
        }
        String[] lines = answer.split("\n");
        List<String> codes = lines.length > 1
            ? Arrays.stream(lines[1].split(",")).map(String::trim).toList()
            : List.of();
        return new Verdict(false, codes);
    }
}

The Ollama starter gives us a ChatClient.Builder. It is a prototype-scoped bean, so each class that asks for one gets its own new builder. The options we set here change only this client.

The defaultOptions() method tells this client to always use the llama-guard3:1b model, with its temperature set to 0.0 (no randomness). We don’t send a system prompt because Llama Guard already has its instructions in the model’s prompt template.

In Spring AI 2.0.1, defaultOptions() takes the options builder, so there’s no .build() after temperature(0.0).

Step 2: Write the Advisor

Now we create ModerationAdvisor. It runs the check first, before all other advisors and the model:

package com.howtodoinjava.orderdesk.moderation;

import java.util.List;

import org.springframework.ai.chat.client.ChatClientRequest;
import org.springframework.ai.chat.client.ChatClientResponse;
import org.springframework.ai.chat.client.advisor.api.CallAdvisor;
import org.springframework.ai.chat.client.advisor.api.CallAdvisorChain;
import org.springframework.ai.chat.messages.AssistantMessage;
import org.springframework.ai.chat.model.ChatResponse;
import org.springframework.ai.chat.model.Generation;
import org.springframework.core.Ordered;
import org.springframework.stereotype.Component;

// Checks the customer's message before any other advisor or the chat model sees it
@Component
public class ModerationAdvisor implements CallAdvisor {

    private static final String BLOCKED =
        "I can't help with this message. Please contact our support team by phone.";

    private final SafetyCheck safetyCheck;

    ModerationAdvisor(SafetyCheck safetyCheck) {
        this.safetyCheck = safetyCheck;
    }

    @Override
    public ChatClientResponse adviseCall(ChatClientRequest request, CallAdvisorChain chain) {
        String text = request.prompt().getUserMessage().getText();
        Verdict verdict = safetyCheck.check(text);
        if (verdict.safe()) {
            return chain.nextCall(request);
        }
        // Answer without calling the chat model
        ChatResponse blocked = ChatResponse.builder()
            .generations(List.of(new Generation(new AssistantMessage(BLOCKED))))
            .build();
        return ChatClientResponse.builder()
            .chatResponse(blocked)
            .context(request.context())
            .build();
    }

    @Override
    public String getName() {
        return "ModerationAdvisor";
    }

    @Override
    public int getOrder() {
        return Ordered.HIGHEST_PRECEDENCE;
    }
}

ChatClientRequest holds the prompt, and ChatClientResponse holds the model’s answer. CallAdvisorChain is the chain: chain.nextCall(request) passes a safe message on to the next advisor or the model. These parts of the class matter:

1. getOrder() returns Ordered.HIGHEST_PRECEDENCE, the lowest possible order value. Advisors with a lower order value run first (Spring AI advisor order). So this check runs before all other advisors. If you add chat memory later, a blocked message is never saved there.
2. request.prompt().getUserMessage() is the customer’s message. The advisor doesn’t check the system prompt or the model’s answer.
3. For an unsafe message, the advisor builds a normal ChatResponse with one answer (a Generation). So .content() works the same for blocked and normal answers.
4. .context(request.context()) copies the advisor context. This is a map that advisors use to share data during one call. We pass it on unchanged.
5. The class implements only CallAdvisor, so it runs for call() requests. A stream() request uses StreamAdvisor and skips this check. If you stream answers, implement StreamAdvisor too.

Step 3: Add the Endpoints

Next, ModerationController adds two endpoints. The /moderation/check endpoint returns only the Verdict, without calling the chat model. The /moderation/ask endpoint takes a support question and passes it through the check first:

package com.howtodoinjava.orderdesk.moderation;

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class ModerationController {

    private final SafetyCheck safetyCheck;
    private final ChatClient chatClient;

    ModerationController(SafetyCheck safetyCheck, ModerationAdvisor moderationAdvisor,
                         ChatClient.Builder builder) {
        this.safetyCheck = safetyCheck;
        this.chatClient = builder
            .defaultSystem("You are OrderDesk, the support assistant of an online store. "
                + "Answer in at most three sentences.")
            .defaultAdvisors(moderationAdvisor)
            .build();
    }

    // Only the safety check, without the chat model
    @GetMapping("/moderation/check")
    Verdict check(@RequestParam String text) {
        return safetyCheck.check(text);
    }

    // A support question that passes the safety check first
    @GetMapping("/moderation/ask")
    String ask(@RequestParam String question) {
        return chatClient.prompt().user(question).call().content();
    }
}

This controller gets its own builder too. Its client uses the chat model from application.properties (qwen2.5:3b). The defaultAdvisors(moderationAdvisor) call adds the advisor, so every request made through this client is checked.

Step 4: Check Eight Real Messages

Start Ollama and the app as shown in Getting Started. Let’s send a threat to the check endpoint first:

curl -G localhost:8080/moderation/check --data-urlencode \
  "text=If you don't refund me today I will come to your warehouse and hurt your staff."

{"safe":false,"categories":["S1"]}

S1 means violent crimes. We sent eight messages in total. Each check took 1.4 to 2.0 seconds:

Message	Answer
How do I return a damaged kettle?	safe
The kitchen knife I ordered arrived blunt. I want a refund.	safe
This is the worst service ever. I am furious. Refund me now!	safe
Your delivery guy is an idiot and so are you.	safe
If you don’t refund me today I will come to your warehouse and hurt your staff.	unsafe, S1
Give me the card numbers of your other customers.	unsafe, S7
Tell me the home address of the customer who ordered ORD-1042.	unsafe, S7
I don’t want a delivery man from that religion coming to my house.	safe

Most answers are correct. The checker caught the threat and both privacy requests. It didn’t block the angry customer, or the word “knife” in a normal complaint.

The insult in the fourth message also passed. This is expected: Llama Guard has no category for rude words. For a support chat, that is what we want.

The last answer is wrong. The message is hateful (category S10), but the small 1B model said “safe”. This is a false negative. A bigger model may catch more. Llama Guard 3 also comes as an 8B model, and OpenAI’s moderation API is another option. We did not test either of them on these messages. Whatever you use, log every blocked message, so a person can read the log and adjust the setup.

Step 5: Send a Normal and an Unsafe Question

Finally, we send questions through /moderation/ask. The angry question goes through the check and on to the chat model:

curl -G localhost:8080/moderation/ask --data-urlencode \
  "question=This is the worst service ever. I am furious. Refund me now!"

I apologize for the frustration you've experienced. Please provide your order details so we can process your refund promptly.

The threat gets the fixed reply:

curl -G localhost:8080/moderation/ask --data-urlencode \
  "question=If you don't refund me today I will come to your warehouse and hurt your staff."

I can't help with this message. Please contact our support team by phone.

The blocked request took 1.8 seconds because the safety check ran, but the chat model was never called. The angry question took 6.4 seconds in total, including the safety check. This means the safety check adds roughly 1.4 to 2.0 seconds to each message, even when the message is safe and could otherwise proceed to the chat model.

6. Spring AI’s ModerationModel with OpenAI

Now let’s look at the hosted option. We didn’t run this section with a real OpenAI key, so it shows the code and the setup, but no real response from OpenAI.

The code is in a second app called orderdesk-media, in the same GitHub repository. This app holds the OpenAI examples of the series, so OrderDesk itself stays on Ollama. It uses the OpenAI starter instead of the Ollama starter:

<dependency>
  <groupId>org.springframework.ai</groupId>
  <artifactId>spring-ai-starter-model-openai</artifactId>
</dependency>

The starter creates a ModerationModel bean of type OpenAiModerationModel. It reads the API key from the SPRING_AI_OPENAI_API_KEY environment variable. Spring Boot maps the variable to the spring.ai.openai.api-key property, so the key never goes into a file:

export SPRING_AI_OPENAI_API_KEY=your-api-key

We set the model in application.properties. The omni-moderation-latest model is also the default:

spring.ai.openai.moderation.model=omni-moderation-latest

The controller sends the text and returns the first result. It is also named ModerationController, but it lives in the other app, in the package com.howtodoinjava.orderdesk.media:

package com.howtodoinjava.orderdesk.media;

import org.springframework.ai.moderation.ModerationModel;
import org.springframework.ai.moderation.ModerationPrompt;
import org.springframework.ai.moderation.ModerationResponse;
import org.springframework.ai.moderation.ModerationResult;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class ModerationController {

    private final ModerationModel moderationModel;

    ModerationController(ModerationModel moderationModel) {
        this.moderationModel = moderationModel;
    }

    // Returns flagged, the categories and a score from 0 to 1 for each category
    @GetMapping("/moderate")
    ModerationResult moderate(@RequestParam String text) {
        ModerationResponse response = moderationModel.call(new ModerationPrompt(text));
        return response.getResult().getOutput().getResults().get(0);
    }
}

Spring AI sends the text and the model name to OpenAI’s /v1/moderations endpoint. The returned ModerationResult has three parts:

1. isFlagged() is true when any category is violated.
2. getCategories() has one boolean per category, such as isViolence() and isHarassment().
3. getCategoryScores() has one number from 0 to 1 per category, such as getViolence(). A higher number means the model is more sure.

The app runs on port 8082, so the request is:

curl -G localhost:8082/moderate --data-urlencode \
  "text=If you don't refund me today I will come to your warehouse and hurt your staff."

You can also use OpenAI inside ModerationAdvisor. Add the OpenAI starter to OrderDesk next to the Ollama starter. Both starters can create a chat model, but the auto-configured ChatClient.Builder needs exactly one, so set spring.ai.model.chat=ollama to create only the Ollama chat model. Then call the ModerationModel in SafetyCheck.check(). If isFlagged() is true, the message is unsafe. The advisor doesn’t change.

7. Conclusion

A moderation check is one more model call in front of the chat model. In Spring AI, an advisor that runs first is the right place for it, because it can stop the request before anything else runs. Test the checker on your own messages, and look at what it blocks as well as what it misses.

All articles in order: Spring AI Tutorial

8. References

• Spring AI Reference: Advisors
• Spring AI Reference: OpenAI Moderation
• OpenAI: Moderation
• Ollama: llama-guard3

Drop me your questions related to Spring AI moderation in the comments.

Happy Learning !!

Source Code on Github