A large language model (LLM) processes tokens and generates an answer one token at a time. The terms in this glossary describe what goes into a request, how the model generates output and how an application supplies data, memory and tools.
For Java developers, these concepts explain Spring AI and LangChain4j configuration as well as provider limits. Use the sections below to understand context-length errors, variable answers, retrieval behavior and token costs without treating framework names as new concepts.
The following example counts tokens with the JTokkit library, which implements the tokenizers that OpenAI models use. Other providers use their own tokenizers, so the counts differ slightly between models.
Encoding encoding = Encodings.newDefaultEncodingRegistry().getEncoding(EncodingType.O200K_BASE);
String sentence = "Spring Boot makes Java microservices easy to build.";
int tokens = encoding.countTokens(sentence); // tokens = 10 (8 words)
IntArrayList ids = encoding.encode("unbelievably"); // 3 tokens: un, bel, ievablyNotice that a token is not a word. Common words are often one token, while longer or rarer words are split into pieces. In the rest of the article, we group the terms the way we meet them in code, from the request itself to cost and safety, and finish with a table that maps each term to its Spring AI and LangChain4j class.
1. How an LLM Request Works
Every call to a chat model follows the same path, whether we use Spring AI, LangChain4j or plain HTTP. Our application sends messages, the provider turns them into tokens, the model generates new tokens until it stops, and the provider turns those tokens back into text.
A model uses the context supplied for each generation. An application or provider can retain conversation history and include it in later calls; the model weights are not updated by ordinary chat. This is why chat memory and retrieved context consume part of the request budget.
2. Model Terms
We meet the model terms when we choose a provider or a model name in the configuration.
| Term | What it means |
|---|---|
| Large language model (LLM) | A model trained on large amounts of text and code that generates text by predicting the next token |
| Parameters (weights) | The numbers the model learned during training; the model size, such as 8 billion parameters, counts them |
| Training | The phase in which the model learns its parameters from data; it happens once, before we use the model |
| Inference | Running the trained model to answer a request; every API call is an inference call |
| Base model | A model after pre-training only, good at continuing text but not at following instructions |
| Instruct or chat model | A base model trained further to follow instructions and hold a conversation; chat APIs use these |
| Fine-tuning | Training an existing model further on our own examples to change its style or teach it a narrow task |
| Open-weight model | A model whose weights we can download and run ourselves, for example with Ollama, a tool that runs models locally |
| Multimodal model | A model that also accepts or produces images, audio or video, not only text |
| Knowledge cutoff | The date after which the model has no training data, so it does not know about newer events or library versions |
| Quantization | Storing the weights with fewer bits, which makes an open-weight model small enough to run on a laptop at some cost in quality |
| Reasoning model | A model that generates internal reasoning tokens before the answer, which improves hard tasks but costs more tokens and time |
3. Prompt and Token Terms
A tokenizer maps text to token IDs. A token can represent a word, a word fragment, whitespace or punctuation. English word-to-token ratios are rough estimates; use the target model’s tokenizer or reported usage for limits and cost calculations.
The following example shows how one tokenizer splits a word and a line of Java code. The name O200K_BASE refers to one of the tokenizer encodings that JTokkit ships, and each token has an integer ID, so decoding the IDs one by one shows the pieces.
Encoding encoding = Encodings.newDefaultEncodingRegistry().getEncoding(EncodingType.O200K_BASE);
IntArrayList ids = encoding.encode("unbelievably"); // ids.size() = 3
List<String> pieces = IntStream.range(0, ids.size())
.mapToObj(i -> {
IntArrayList one = new IntArrayList();
one.add(ids.get(i));
return encoding.decode(one);
})
.toList(); // [un, bel, ievably]
int codeTokens = encoding.countTokens("List<String> names = new ArrayList<>();"); // codeTokens = 10The remaining terms cover what goes into a request and how long the answer may get.
| Term | What it means |
|---|---|
| Context window | The maximum number of tokens the model handles in one request, input and output together; large hosted models offer about 128,000 to 1 million tokens, and small local models often much less |
| Prompt | Everything we send to the model in one request |
| System message (system prompt) | Instructions that set the model’s role and rules, such as “Answer only questions about our product” |
| User message | The question or input from the user |
| Assistant message | A previous answer from the model, sent back as part of the chat history |
| Prompt template | A prompt with placeholders, such as Summarize {text} in {count} bullet points, that we fill in at runtime |
| Max tokens | The limit on how many tokens the model may generate for the answer; a cut-off answer often means this limit was too low |
| Streaming | Receiving the answer token by token as the model generates it, instead of waiting for the full text |
| Zero-shot and few-shot prompting | Asking without examples (zero-shot) or with a few worked examples in the prompt (few-shot) |
4. Terms for Controlling the Output
For every next token, the model gives a score, called a logit, to each token in its vocabulary (the set of all tokens it knows), and the scores are turned into probabilities. Temperature changes how those probabilities are spread. A low temperature makes the model pick the most likely token almost every time, whereas a high temperature gives the less likely tokens a real chance.
The following example uses four made-up scores for the word after “The coffee is” and converts them into probabilities at three temperatures. Real models do the same calculation over their whole vocabulary.
static double[] softmax(double[] scores, double temperature) {
double[] scaled = Arrays.stream(scores).map(s -> Math.exp(s / temperature)).toArray();
double sum = Arrays.stream(scaled).sum();
return Arrays.stream(scaled).map(s -> s / sum).toArray();
}
double[] scores = {4.0, 3.0, 2.5, 0.5}; // hot, ready, strong, purple
double[] low = softmax(scores, 0.2); // hot 99%, ready 1%, strong 0%, purple 0%
double[] normal = softmax(scores, 1.0); // hot 62%, ready 23%, strong 14%, purple 2%
double[] high = softmax(scores, 2.0); // hot 44%, ready 27%, strong 21%, purple 8%The demonstration divides logits by a positive temperature, exponentiates and normalizes them. Lower temperatures concentrate probability on higher-scoring tokens; higher temperatures flatten the distribution. Providers commonly interpret zero as greedy selection, but implementation details mean repeatable settings do not guarantee identical output.
| Term | What it means |
|---|---|
| Temperature | Controls randomness; low values for facts, code and extraction, higher values for brainstorming; the allowed range differs by provider, for example 0.0 to 1.0 for Anthropic models |
| Top-p (nucleus sampling) | Picks the next token only from the smallest group of most likely tokens (the nucleus) whose probabilities add up to p, so 0.1 keeps the top 10% of the probability; providers recommend changing either top-p or temperature, not both |
| Top-k | Picks the next token only from the k most likely tokens |
| Stop sequence | A string that ends generation when the model produces it |
| Structured output | Asking the model for JSON that matches a schema, so we can map the answer to a Java record or class |
| Deterministic output | The same answer for the same input; temperature 0 makes answers more repeatable but does not guarantee identical output |
Generation controls vary by model and API. A parameter supported by one model may be restricted, ignored or rejected by another. Check the chosen provider’s model-specific reference before copying temperature, top-p or top-k settings.
5. Embeddings, Vector Search and RAG Terms
An embedding is an array of floating-point numbers (a vector) that represents the meaning of a text. An embedding model creates it, and texts with similar meaning get vectors that point in similar directions, so we can find related text by comparing vectors instead of matching keywords.
The following example compares three made-up vectors with four dimensions each, while real embedding models return hundreds or thousands of dimensions. Cosine similarity returns a value close to 1 for vectors that point the same way.
static double cosineSimilarity(float[] a, float[] b) {
double dot = 0, normA = 0, normB = 0;
for (int i = 0; i < a.length; i++) {
dot += a[i] * b[i];
normA += a[i] * a[i];
normB += b[i] * b[i];
}
return dot / (Math.sqrt(normA) * Math.sqrt(normB));
}
float[] cat = {0.9f, 0.8f, 0.1f, 0.0f};
float[] kitten = {0.85f, 0.75f, 0.2f, 0.05f};
float[] invoice = {0.05f, 0.1f, 0.9f, 0.8f};
double catKitten = cosineSimilarity(cat, kitten); // 0.995
double catInvoice = cosineSimilarity(cat, invoice); // 0.147A vector store saves these vectors and finds the closest ones to a query vector. A RAG pipeline splits documents into chunks, stores an embedding for each chunk, and at question time embeds the question, finds the closest chunks and adds them to the prompt. The nearest-vector search is the retrieval step of that pipeline, and the existing Spring AI EmbeddingModel example shows the same idea with a real embedding model.
| Term | What it means |
|---|---|
| Embedding | A vector of numbers that represents the meaning of a text, image or other input |
| Dimensions | The length of the embedding vector, fixed for each embedding model |
| Cosine similarity | A measure of how close two vectors point, from -1 to 1, where values near 1 mean similar meaning |
| Vector store (vector database) | A database that stores embeddings and returns the most similar ones, such as PGvector, Chroma or Redis |
| Chunking | Splitting long documents into smaller pieces before creating embeddings, so each piece fits and matches precisely |
| Retrieval augmented generation (RAG) | Finding the document chunks related to a question and adding them to the prompt, so the model answers from our data |
| Top-K retrieval and similarity threshold | How many of the closest chunks the vector store returns, and the minimum similarity a chunk needs to be included |
| Re-ranking | A second step that sorts the retrieved chunks again with a more precise model before they go into the prompt |
| Grounding | Making the model answer from given sources instead of from what it learned in training |
| Hallucination | A confident answer that is false, because the model predicts likely tokens rather than checking facts |
6. Tool, Agent and Memory Terms
Tool calling lets a model request an operation by returning a tool name and arguments. Application code validates and authorizes the request, executes the operation and supplies the result in a later model call. The model proposes the call; it does not execute Java code itself.
| Term | What it means |
|---|---|
| Tool calling (function calling) | The model returns the name and arguments of a tool instead of text, our code runs the tool, and the result goes back to the model |
| Agent | An application where the model decides, step by step, which tools to call, reads each result and repeats this loop until the task is done |
| Model Context Protocol (MCP) | An open standard for connecting AI applications to external tools and data; an MCP server exposes tools, data and prompts, and an MCP client inside the AI application calls them, so one server works with many applications |
| Chat memory | The part of our application that stores past messages and sends them with each new request |
| Guardrails | Checks before or after the model call that block unsafe input or output, such as personal data in an answer |
7. Cost, Quality and Safety Terms
Token pricing usually distinguishes input from output and may include cache, tool or other charges. Count all calls needed to finish a task, including retries and agent loops. A token count is an input to the price calculation, not a price by itself.
| Term | What it means |
|---|---|
| Token usage | The input and output token counts that the provider returns with each response; we log them to track cost |
| Prompt caching | Many providers charge less for repeated input, such as a long system prompt, that they have cached from an earlier request |
| Rate limit | The maximum requests or tokens per minute that a provider allows for our account |
| Latency and time to first token | The total time until the full answer, and the time until the first streamed token appears |
| Evaluation (evals) | Automated tests that check model answers, for example for relevance or factual correctness |
| LLM-as-a-judge | Using a second model call to grade the answer of the first one |
| Prompt injection | Text in user input or retrieved documents that tries to override our instructions; the OWASP Top 10 for LLM Applications lists it first |
8. LLM Terms and Their Java Classes
Both main Java libraries use the same concepts under slightly different names. The table maps the terms to the types we use in Spring AI and LangChain4j.
| Term | Spring AI | LangChain4j |
|---|---|---|
| Chat model call | ChatClient, ChatModel | ChatModel, AiServices |
| Prompt and messages | Prompt, SystemMessage, UserMessage | ChatRequest, SystemMessage, UserMessage |
| Prompt template | PromptTemplate | PromptTemplate |
| Temperature, max tokens | ChatOptions | ChatRequest builder methods |
| Structured output | BeanOutputConverter, entity() | Return types of an AiServices interface |
| Embeddings | EmbeddingModel | EmbeddingModel |
| Vector store | VectorStore | EmbeddingStore |
| Chunking | TokenTextSplitter | DocumentSplitter |
| RAG | QuestionAnswerAdvisor, RetrievalAugmentationAdvisor | EmbeddingStoreContentRetriever |
| Tool calling | @Tool | @Tool |
| Chat memory | ChatMemory, MessageChatMemoryAdvisor | MessageWindowChatMemory |
| Streaming | ChatClient stream(), returns a Flux | StreamingChatModel |
| Token usage | Usage in the ChatResponse metadata | Token usage in the ChatResponseMetadata |
| MCP | MCP client and server Boot starters | langchain4j-mcp module, McpToolProvider |
The Spring AI tutorial and getting started with LangChain4j show these types in working Spring Boot applications.
9. LLM Terms FAQs
These questions connect the glossary to common API behavior.
9.1. What Is the Difference Between a Token and a Word?
A token is a piece of text that the tokenizer defines, which can be a whole word, part of a word, a space or a symbol. In English, 100 words are roughly 130 tokens, and each tokenizer splits text a little differently.
9.2. What Is the Difference Between RAG and Fine-Tuning?
RAG adds our documents to the prompt at request time, so the data can change every day without retraining. Fine-tuning changes the model itself with training examples, which suits a fixed style or task better than changing facts.
9.3. What Does Temperature 0 Mean?
Temperature 0 tells the model to pick the most likely token almost every time. The answers become more repeatable, which suits extraction and classification, but providers do not guarantee identical output for identical requests.
9.4. Why Does My Request Fail With a Context Length Error?
The request exceeds the model’s applicable context or output limits. Check input size, reserved output and any reasoning or protocol overhead. Reduce unnecessary history or retrieved material, lower the output allowance where appropriate, or use a suitable model with a larger supported budget.
10. Conclusion
Follow a single request to connect the terms: messages become tokens, the context budget limits what can be processed and generation settings affect the output. Embeddings and retrieval select supporting data; tools and memory extend the application workflow. None of those additions removes the need to check the answer.
With these terms in place, the Spring AI and LangChain4j docs read much faster, because each class maps to one idea from this page.
11. References
- Spring AI Concepts
- Spring AI OpenAI Chat Options
- Spring AI Anthropic Chat Options
- LangChain4j RAG Tutorial
- What Is the Model Context Protocol
- OWASP Top 10 for LLM Applications
- JTokkit
Happy Learning !!
