A customer asks your AI assistant a simple question, but the answer takes 15 seconds. Is the model slow? Is the application searching too many documents? Or did it call an order service several times? Spring AI observability helps you find out by measuring the steps behind each answer.
In this guide, we will read token counts, measure response time, and follow a slow request in Zipkin. We will also use the token counts to estimate the cost of a hosted model. The example is OrderDesk, an online store’s support assistant, built with Spring AI 2.0.1, Spring Boot 4.1.1, Java 21, and a local model in Ollama.
1. Understand What You Are Measuring
Spring AI records an observation when an operation runs. Think of it as a record of one piece of work: what ran, how long it took, and details such as the model name. Operations include asking a model for an answer, searching documents, and executing a tool.
Micrometer connects these observations to monitoring tools. Actuator lets us inspect measurements through HTTP endpoints. Zipkin displays the steps of a request as a timeline. We will configure these connections below.
You will see three terms throughout the article:
| Signal | Question it answers | Example |
|---|---|---|
| Metrics | What happened across many requests? | Total tokens, number of calls, average response time |
| Traces | Where did one request spend its time? | Time spent searching documents, running tools, and generating the answer |
| Logs | What detail helps explain a problem? | An error message from a failed operation |
These are the names to look for in OrderDesk’s measurements:
| Name | What it measures |
|---|---|
spring.ai.chat.client | The overall ChatClient call, including the work it performs |
spring.ai.advisor | An advisor and the steps it calls, such as document retrieval |
gen_ai.client.operation | A model operation, such as chat or embedding |
gen_ai.client.token.usage | Token usage reported for model calls |
db.vector.client.operation | Vector-store operations |
spring.ai.tool | Tool execution |
One request can produce several measurements. For example, a ChatClient call may search documents and then call the model twice. Its total time includes those steps; the model timer measures each model call separately. See the Spring AI observability reference.
2. Overview of the Example Application
OrderDesk answers delivery questions, reads store policies, and looks up orders. We will compare these three endpoints:
| Endpoint | Work performed |
|---|---|
GET /ask | Sends a question to the chat model |
GET /rag/qa | Retrieves store-policy passages before generating an answer |
GET /tools/ask | Makes an order lookup available as a tool |
Start with /ask: it is enough to see model-call timing and token usage. The other endpoints add more work. RAG means finding relevant documents and giving them to the model with the question. A tool is an application method the model can ask the application to execute, such as looking up an order.
The OrderDesk project contains the application configuration and endpoints. If you are starting with a new project, follow Getting Started with Spring AI first.
To try the RAG endpoint too, prepare its policy search. An embedding model turns text into numbers that can be compared, and a vector store uses those numbers to find similar text. OrderDesk selects them with these properties:
spring.ai.ollama.embedding.options.model=nomic-embed-text
spring.ai.vectorstore.type=simpleThe model must be available in Ollama:
ollama pull nomic-embed-textAfter starting the application, load the policy documents so the search has something to find:
curl -X POST http://localhost:8080/ingestThis endpoint belongs to OrderDesk; it is not an endpoint automatically supplied by Spring AI. The ETL tutorial explains how the documents are read, split, and stored.
3. Read Token Counts in Actuator
3.1. Add the Actuator Dependency
Add the Actuator starter to an existing Spring AI application. Its version is managed by the Spring Boot parent or dependency-management configuration:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>OrderDesk already includes this dependency. You do not need to add it a second time when using the example project.
3.2. Make the Measurements Available over HTTP
Actuator’s metrics endpoint lets us read measurements in the browser or with curl. Make it available over HTTP:
management.endpoints.web.exposure.include=health,metricsRestart the application and send a question to /ask. Then list the available measurements:
curl http://localhost:8080/actuator/metricsLook for gen_ai.client.operation, gen_ai.client.token.usage, and spring.ai.chat.client. Actuator calls a named measurement a meter. Some meters appear only after the operation runs, so send a request before looking for its metrics.
The endpoint shows the current measurements; it does not store a history or draw a dashboard. Keep it accessible only to the people and monitoring services that need it. See the Actuator endpoint documentation.
3.3. Read Input and Output Tokens Separately
A token is a piece of text processed by the model, often a word or part of a word. Input tokens come from the request; output tokens come from the generated response. Read the input count with:
curl "http://localhost:8080/actuator/metrics/gen_ai.client.token.usage?tag=gen_ai.operation.name:chat&tag=gen_ai.token.type:input"For output tokens:
curl "http://localhost:8080/actuator/metrics/gen_ai.client.token.usage?tag=gen_ai.operation.name:chat&tag=gen_ai.token.type:output"Each tag in these URLs is a filter. The chat filter selects chat-model usage. The input or output filter selects the side of the exchange we want to count.
In the JSON response, find measurements and its COUNT value. For this token metric, COUNT means the number of tokens recorded so far. It does not mean the number of questions or model calls.
Always select a token type. If a call uses 100 input tokens and 20 output tokens, its total is 120. Adding all three values gives 240, which counts the same tokens twice. Keep input and output counts separate when estimating cost.
If you use several models, filter by model too. The response’s availableTags field lists the filters and values you can use. Without a model filter, the result can include usage from several models. See the metrics endpoint documentation.
3.4. Why Three Questions Produced Four Model Calls
The previously published OrderDesk run recorded the following chat usage across three HTTP requests:
| Request | Model call | Input tokens | Output tokens |
|---|---|---|---|
| Plain question | Generate an answer | 89 | 12 |
| RAG question | Answer using retrieved policy text | 247 | 47 |
| Tool question | Request the order lookup | 367 | 26 |
| Tool question | Answer using the tool result | 470 | 57 |
| Total | 4 model calls | 1,173 | 142 |
These are recorded example results, not expected output for every machine or request. Model responses, conversation context, and configuration can change token counts.
The tool question needed two calls. In the first, the model asked the application to look up the order. After the lookup, the application sent the result back so the model could write its answer. That second request included the earlier context and the lookup result, which explains its larger input count.
This is why one customer question can cost more than one model call. A workflow with several tool lookups or retries may make even more calls.
4. Find Out How Long Model Calls Take
Latency means the time an operation takes. Read the chat-model timing with:
curl "http://localhost:8080/actuator/metrics/gen_ai.client.operation?tag=gen_ai.operation.name:chat"In the recorded example, the relevant measurements were:
COUNT 4
TOTAL_TIME 43.198048743 seconds
MAX 12.666572848 secondsHere, COUNT means four model calls completed. TOTAL_TIME adds their durations together: about 43.2 seconds. Divide that total by the call count to find the average:
43.198048743 / 4 = about 10.8 seconds per model callThe 10.8-second average describes one model call. It does not tell us how long a customer waited. Their request may also include document search and tool execution, or call the model more than once. To see the full request, open its trace as shown next.
MAX shows a maximum duration, about 12.7 seconds in this example. Depending on the metrics registry, it may cover a recent time window rather than all calls since startup. See the Micrometer timer documentation.
5. Find the Slow Step with a Trace
A trace is a timeline of one request. Each timed step is called a span. For example, one span can represent document search and another can represent the model call. Zipkin displays these spans so we can see which step took the most time.
5.1. Send Traces to Zipkin
For this Spring Boot 4.1 example, add the Zipkin starter:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-zipkin</artifactId>
</dependency>OrderDesk already includes the starter, but keeps trace export disabled in its base configuration:
management.tracing.export.enabled=falseIts application-observability.properties profile enables export:
management.tracing.export.enabled=true
management.tracing.sampling.probability=1.0
management.tracing.export.zipkin.endpoint=http://localhost:9411/api/v2/spansThe value 1.0 means every request is selected for tracing. That makes local testing easier because the request you just sent should appear in Zipkin. In production, you can trace a smaller share of requests to reduce the amount of data collected.
Start Zipkin and run OrderDesk with the profile:
docker run -d --name zipkin -p 9411:9411 openzipkin/zipkinmvn spring-boot:run -Dspring-boot.run.profiles=observabilityIf a container named zipkin already exists, start or reuse that container instead of creating another with the same name. Open the Zipkin interface at http://localhost:9411.
These starter and property names match the example’s Spring Boot 4.1 configuration. Check the matching documentation when using another Boot release. See the Spring Boot tracing reference.
5.2. Include the Vector Search in the Trace
If the trace shows a model call but no vector search, check how the vector store was created. A store built in your own configuration may need you to pass in an ObservationRegistry. This Micrometer object connects the store’s observations to the application’s metrics and tracing setup.
OrderDesk creates its own SimpleVectorStore. Its configuration passes Micrometer’s ObservationRegistry explicitly:
package com.howtodoinjava.orderdesk.vectorstore;
import java.io.File;
import io.micrometer.observation.ObservationRegistry;
import org.springframework.ai.embedding.EmbeddingModel;
import org.springframework.ai.vectorstore.SimpleVectorStore;
import org.springframework.beans.factory.ObjectProvider;
import org.springframework.boot.autoconfigure.condition.ConditionalOnProperty;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
@ConditionalOnProperty(name = "spring.ai.vectorstore.type", havingValue = "simple")
public class SimpleVectorStoreConfig {
public static final File STORE_FILE = new File("data/vector-store.json");
@Bean
SimpleVectorStore vectorStore(
EmbeddingModel embeddingModel,
ObjectProvider<ObservationRegistry> observationRegistry) {
SimpleVectorStore store = SimpleVectorStore.builder(embeddingModel)
.observationRegistry(
observationRegistry.getIfUnique(() -> ObservationRegistry.NOOP))
.build();
if (STORE_FILE.exists()) {
store.load(STORE_FILE);
}
return store;
}
}Two parts of this class serve different purposes. STORE_FILE holds the saved vectors so the app can load them after a restart. .observationRegistry(...) connects the store to monitoring. The NOOP fallback means no observations are recorded when there is no unique registry to use.
The embedding model and vector store record their work separately. Seeing an embedding span therefore does not prove that the vector search itself has been recorded. Pass the registry to the store so both steps are visible.
For ChatClient, prefer Spring Boot’s configured ChatClient.Builder so the client retains the application’s observation setup and customizations.
5.3. Read the Trace without Counting Time Twice
Send a question that uses the indexed return policy:
curl -G http://localhost:8080/rag/qa --data-urlencode "question=How many days do I have to return a jacket?"In Zipkin, select the application service and open the corresponding request. The existing article’s captured run shows a request lasting about 13.3 seconds:
The recorded timings included:
| Operation | Recorded duration | What it tells us |
|---|---|---|
| HTTP request | 13.288 s | Overall server-request duration |
| Chat model | 12.058 s | Most of this request was spent in generation |
| Vector-store query | 0.206 s | Retrieval was a smaller part of this example |
| Query embedding within the search | 0.189 s | A child operation included in retrieval time |
Some spans contain other spans. Here, the 0.189-second embedding step is part of the 0.206-second search. It is not another 0.189 seconds to add on top. The complete search and model call are also included in the overall HTTP request. Read the nesting before adding durations.
For this request, the model used about 12.1 of the total 13.3 seconds. That makes generation the first place to investigate. Another request may spend most of its time searching documents or waiting for a tool, so compare several traces before changing the application.
The first search after startup also recorded an extra embedding call. Later requests did not show the same work. Compare a first request with later requests before treating a one-time startup delay as an ongoing problem.
6. Turn Token Counts into a Cost Estimate
Hosted models often charge different prices for input and output tokens. Once you have both counts, multiply each by its matching rate and add the results:
estimated cost = (input tokens / 1,000,000 × input price per million)
+ (output tokens / 1,000,000 × output price per million)Suppose input costs USD 0.40 per million tokens and output costs USD 1.60 per million tokens. These are example rates. Applying them to the recorded counts gives:
Input: 1,173 / 1,000,000 × 0.40 = USD 0.0004692
Output: 142 / 1,000,000 × 1.60 = USD 0.0002272
Total: USD 0.0006964The estimated total is about USD 0.0007. OrderDesk used a local Ollama model, so it did not receive this hosted-model charge. The calculation shows how to apply a rate card; running a local model still uses your own computing resources.
For a real estimate, use the prices for the actual provider, model, and billing period. Cached input and other charge types may have different rates. Compare your estimate with the provider’s usage report to explain any difference.
To estimate today’s spend, use today’s token usage, not an application’s lifetime total. Counters can reset when an app restarts. A monitoring system can track those changes and combine measurements from several application instances.
7. Decide What Should Trigger an Alert
Actuator shows measurements, but an alert needs a monitoring system that stores them over time. Start with signals you know how to investigate:
| Signal | What to investigate |
|---|---|
| Rising HTTP or model-call latency | Slow generation, queues, retrieval, or downstream tools |
| Increased failures | Provider errors, timeouts, credentials, or unavailable models |
| Rising token consumption | Larger prompts, longer responses, retries, or extra model calls |
| Increased tools or model calls per request | A workflow change or repeated tool iterations |
| Increased estimated spend | More traffic, a model change, or higher usage per interaction |
Compare similar requests over the same period, such as order lookups during the last hour. A rise in tool calls may be normal if more customers ask about orders. It becomes more useful when you can see that each order question now runs the same tool several times.
Also check answer quality. A fast request with HTTP status 200 can still give the wrong answer. Performance metrics and answer evaluations tell you different things.
8. Debug without Logging Customer Conversations
You can measure timing and token usage without recording the full prompt or answer. Those bodies may contain customer messages, private policy documents, or order details returned by a tool.
Keep full-content logging off during normal operation. If you need it to investigate a problem, limit who can read the logs, remove private information where possible, and keep the records only as long as needed.
Avoid putting customer IDs or conversation IDs in metric labels. Monitoring systems keep a separate series of measurements for each label combination. Thousands of unique IDs can therefore create thousands of series. If you need usage per customer, store those usage records separately in your application.
9. Common Problems and What to Check
9.1. Why Are AI Metrics Missing?
Send a request first, because some meters are created only when their operation runs. If they are still missing, check that Actuator is installed, metrics is exposed, and the model request completed.
If model observations exist but the ChatClient observations do not, inspect how the client was constructed. A manually created client may bypass the configuration applied by Spring Boot’s builder.
If the vector-store meter is missing, inspect the store’s registry configuration as shown in section 5.2.
9.2. Why Are There No Traces in Zipkin?
Check these in order: Zipkin is running, OrderDesk started with the observability profile, trace export is enabled, and the application can reach the export URL.
localhost refers to the environment where the application runs. If the application is inside a container, it does not automatically refer to Zipkin running on the host or in another container.
Check the sampling setting too. If only some requests are selected for tracing, a missing trace may be expected even though the request updated the metrics.
9.3. Why Are Token Counts Missing or Unexpected?
Check whether the provider returned token-usage information, then check the token-type and model filters in your query. No reported usage does not necessarily mean the request used zero tokens.
For a streamed response, token counts may arrive near the end. If the stream is interrupted, the provider may not send the final usage information.
If the count is higher than expected, look beyond the customer’s question. Conversation history, retrieved passages, tool descriptions, and tool results can all add input tokens. Count every model call in the request.
9.4. Do We Need OpenTelemetry as Well as Zipkin?
This example uses Spring Boot’s Zipkin starter. An application can instead use Spring Boot’s OpenTelemetry integration and export to a compatible collector or backend.
Use the tracing setup your application needs. You do not need both approaches to follow this example, and neither trace exporter is needed just to read token counts from Actuator.
10. Conclusion
Use metrics to see how many tokens the application consumes and how long its calls take. When a request is slow, open its trace to find the step responsible. In OrderDesk’s recorded run, three questions produced four model calls, and the chat model took most of the time in the captured RAG request.
These measurements help you choose what to improve first: a long prompt, a slow search, repeated tool calls, or model generation. Use token counts to estimate cost, then compare that estimate with the provider’s usage report.
Continue with the Spring AI Tutorial.
Happy Learning !!
