RAG Explained: Retrieval-Augmented Generation in Java

Retrieval-augmented generation (RAG) gives an LLM relevant source material before it answers a question. The application searches its documents, adds selected passages to the prompt and asks the model to answer using that context. The model is not retrained.

This tutorial explains indexing, retrieval and generation, then implements them in plain Java 17. A local embedding model finds recipe passages and Ollama generates an answer. The recorded comparison shows how the right passage corrects a wrong baking time, followed by examples where retrieval or generation still fails.

1. Why an LLM Doesn’t Know Your Documents

A model cannot reliably know a private document unless the application supplies it or connects it to a source that can. Training data may contain public and other licensed material; it is not a live index of your files. Without the recipe in context, an answer may be a plausible guess.

Let’s say we keep our family recipes as text files, and one of them is Nani’s Lemon Semolina Cake. The recipe says to bake the cake at 170 C for 38 minutes. We ask a small local model, qwen2.5:0.5b on Ollama, about it with no other text in the prompt:

Question: How long do I bake Nani's Lemon Semolina Cake, and at what temperature?

Answer: To bake Nani's Lemon Semolina Cake, you should bake it at 350°F (180°C) for 30 minutes.

In the recorded response, both numbers are wrong. RAG adds the recipe as evidence without changing the model weights. That makes a correct answer possible, but the application must still retrieve the right passage and check that the generated answer follows it.

2. How Retrieval Augmented Generation Works in Three Phases

The example has an indexing phase before questions arrive, then retrieval and generation for each question. Re-index changed documents so later searches use the updated content.

1. Index: we split every document into small pieces called chunks, turn each chunk into a list of numbers with an embedding model, and save the chunk and its numbers in a vector store.
2. Retrieve: when a question arrives, we turn the question into numbers with the same embedding model, and the vector store returns the few chunks whose numbers are closest to the question’s.
3. Generate: we put an instruction, the question and the retrieved chunks into one prompt, and send that prompt to the LLM, which writes the answer.

The name follows the phases: we retrieve text, augment (extend) the prompt with it, and let the model generate the answer.

An open-book exam has the same structure: the student (the LLM) has never read our textbook, so a helper finds the two matching pages and puts them on the desk. A support chatbot that answers from a product’s help pages works the same way.

3. Parts of a RAG Pipeline

Each part below shows up as one line or one class in the Java example of section 4.

A chunk is a piece of a document small enough to be about one thing, such as one paragraph of a recipe. We search chunks, not whole documents, because a whole document fills the prompt with text the question doesn’t need. A chunk should also make sense when read alone; section 5 shows what happens when it doesn’t.

An embedding model maps text to a numeric vector. Nearby vectors can identify related wording, which helps match a question to a passage even when they use different words. The two-dimensional arrays below are invented illustrations; the real model produces 384 dimensions.

The all-MiniLM-L6-v2 model we use works the same way with 384 numbers instead of two. It reads at most 256 tokens (pieces of words, counted by the model’s own tokenizer) and cuts off the rest.

The question and the chunks must be embedded by the same model, because each model places meanings in its own way, and vectors from two different models don’t line up.

A vector store keeps every chunk next to its embedding and finds the stored vectors closest to a given one. Production systems use a vector database such as PostgreSQL with pgvector; for a few hundred chunks, a plain Java list does the same job.

Cosine similarity compares vector direction using the dot product divided by both vector magnitudes. Its numeric range is -1 to 1. A high value means aligned vectors; whether that means a relevant passage depends on the embedding model and data. Zero does not prove unrelated meaning, and -1 does not mean linguistic opposites.

A score like 0.84 has no fixed meaning on its own, and there is no universal “good” threshold. Higher means closer, and the score is useful for ranking the chunks of one index against each other.

Top-k is the number of best-scoring chunks we keep from the search. A small k keeps the prompt short but can miss the chunk with the answer, and a large k puts more unrelated text in front of the model. Our example uses k = 2. For your own data, start with 3 to 5, print what comes back for a few real questions, and adjust.

The model has a context budget covering the request and generated output. Reserve room for the answer after accounting for instructions, history, the question and retrieved chunks. Provider-specific token accounting can include additional overhead.

4. Building a Minimal RAG in Plain Java

The following steps build the Recipe Box example as a small Maven project: first the project and the documents, then the vector store and the Ollama client, and finally the class that runs the three phases. The only AI library is LangChain4j’s in-process embedding model, so the chunking, the vector store, the similarity search and the prompt are all our own code.

Step 1: Create the Maven Project

Let’s start with an empty Maven project. Every class goes into the package com.howtodoinjava.rag, and the recipes go under src/main/resources so the program can read them from the classpath:

rag-explained/
  pom.xml
  src/main/java/com/howtodoinjava/rag/
    InMemoryVectorStore.java    (Step 3)
    OllamaClient.java           (Step 4)
    AskWithoutRag.java          (Step 4)
    RecipeBoxRag.java           (Step 5)
    ConsoleFormat.java          (console helpers, on GitHub)
    RagTrap.java                (section 5, on GitHub)
  src/main/resources/recipes/
    lemon-semolina-cake.txt
    banana-walnut-bread.txt
    tomato-rasam.txt
    sunday-pancakes.txt

The project needs one dependency, langchain4j-embeddings-all-minilm-l6-v2, which holds the model file and the code that runs it inside the JVM, so embedding text needs no network call or API key. It also brings in Jackson, the JSON library we use for the Ollama call.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
                             https://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.howtodoinjava</groupId>
  <artifactId>rag-explained</artifactId>
  <version>1.0</version>

  <properties>
    <maven.compiler.release>17</maven.compiler.release>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <exec.mainClass>com.howtodoinjava.rag.RecipeBoxRag</exec.mainClass>
  </properties>

  <dependencies>
    <!-- Offline embedding model: all-MiniLM-L6-v2 runs inside the JVM, no API key -->
    <dependency>
      <groupId>dev.langchain4j</groupId>
      <artifactId>langchain4j-embeddings-all-minilm-l6-v2</artifactId>
      <version>1.22.0-beta32</version>
    </dependency>
  </dependencies>

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-compiler-plugin</artifactId>
        <version>3.16.0</version>
      </plugin>
      <plugin>
        <groupId>org.codehaus.mojo</groupId>
        <artifactId>exec-maven-plugin</artifactId>
        <version>3.6.4</version>
      </plugin>
    </plugins>
  </build>
</project>

The first build downloads about 190 MB, most of it the model and its runtime. The exec.mainClass property sets the class that mvn compile exec:java starts.

Step 2: Add the Recipe Documents

Next, we add the documents that the model has never seen: four plain text files with a title on the first line, then one paragraph each for the ingredients, the method and the serving notes. The lemon cake file is the one our questions are about:

Nani's Lemon Semolina Cake

Ingredients: 1 cup fine semolina, 3/4 cup sugar, 1 cup yogurt, 1/2 cup
melted butter, zest of 2 lemons, 1 teaspoon baking powder and a pinch of
cardamom.

Method: Mix the semolina, sugar and yogurt, and let the batter rest for
20 minutes so the semolina softens. Stir in the butter, zest, cardamom
and baking powder. Bake at 170 C for 38 minutes in a 9-inch tin.

Serving: Pour warm lemon syrup over the cake while it is still hot. The
cake keeps for 3 days in an airtight box at room temperature.

The other three files have the same shape and are on GitHub. The banana bread matters, because its method paragraph also says “Bake at … in a tin”, so the search has a near-miss to get wrong.

Step 3: Create the Vector Store

Now we create the vector store. It is a list of entries, each holding one chunk and its embedding, and a search() method that scores every entry against the question’s embedding with cosine similarity, sorts by score and keeps the top-k:

package com.howtodoinjava.rag;

import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;

/**
 * A vector store kept in a plain list: every chunk is saved next to its embedding,
 * and a search compares the question's embedding with all of them.
 */
public class InMemoryVectorStore {

    public record Entry(String chunk, float[] embedding) {
    }

    public record Match(String chunk, double score) {
    }

    private final List<Entry> entries = new ArrayList<>();

    public void add(String chunk, float[] embedding) {
        entries.add(new Entry(chunk, embedding));
    }

    public int size() {
        return entries.size();
    }

    /** Similarity search: score every stored chunk and keep the top-k. */
    public List<Match> search(float[] questionEmbedding, int topK) {
        return entries.stream()
                .map(e -> new Match(e.chunk(), cosineSimilarity(questionEmbedding,
                        e.embedding())))
                .sorted(Comparator.comparingDouble(Match::score).reversed())
                .limit(topK)
                .toList();
    }

    /** Cosine similarity = (A . B) / (|A| * |B|), a number from -1 to 1. */
    static double cosineSimilarity(float[] a, float[] b) {
        double dot = 0;
        double normA = 0;
        double normB = 0;
        for (int i = 0; i < a.length; i++) {
            dot += a[i] * b[i];
            normA += a[i] * a[i];
            normB += b[i] * b[i];
        }
        return dot / (Math.sqrt(normA) * Math.sqrt(normB));
    }
}

Comparing the question with every stored vector is called a brute-force search; vector databases add an index that skips most of the comparisons.

Step 4: Create the Ollama Client

Then we need a way to call the LLM. Ollama is a tool that downloads open models and runs them on our own machine behind an HTTP API on port 11434. If you haven’t installed it yet, the article Run LLMs Locally for Java covers it; after that, one command downloads the model we use, about 400 MB:

ollama pull qwen2.5:0.5b

The client sends our prompt to Ollama’s /api/chat endpoint and returns the answer text, with timeouts so a stopped server can’t hang the program:

package com.howtodoinjava.rag;

import java.io.IOException;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.time.Duration;
import java.util.List;
import java.util.Map;

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;

/**
 * Sends one prompt to a local Ollama server and returns the model's text answer.
 * This is the "generate" phase of RAG; the class knows nothing about retrieval.
 */
public class OllamaClient {

    /** The same instruction for every call, with or without RAG. */
    static final String SYSTEM_MESSAGE =
            "You are a cooking assistant. Answer in one or two sentences.";

    private static final ObjectMapper JSON = new ObjectMapper();

    private final HttpClient http = HttpClient.newBuilder()
            .connectTimeout(Duration.ofSeconds(5))
            .build();
    private final String baseUrl;
    private final String model;

    public OllamaClient(String baseUrl, String model) {
        this.baseUrl = baseUrl;
        this.model = model;
    }

    /** Reads OLLAMA_URL and OLLAMA_MODEL, with the Ollama defaults as fallback. */
    public static OllamaClient fromEnvironment() {
        String url = System.getenv()
                .getOrDefault("OLLAMA_URL", "http://localhost:11434");
        String model = System.getenv().getOrDefault("OLLAMA_MODEL", "qwen2.5:0.5b");
        return new OllamaClient(url, model);
    }

    public String generate(String prompt) throws IOException, InterruptedException {
        Map<String, Object> body = Map.of(
                "model", model,
                "stream", false,
                "messages", List.of(
                        Map.of("role", "system", "content", SYSTEM_MESSAGE),
                        Map.of("role", "user", "content", prompt)),
                // temperature 0 and a fixed seed give the same answer on every run;
                // num_predict caps the answer length in tokens
                "options", Map.of("temperature", 0, "seed", 42, "num_predict", 200));

        HttpRequest request = HttpRequest.newBuilder(URI.create(baseUrl + "/api/chat"))
                .timeout(Duration.ofMinutes(2))
                .header("Content-Type", "application/json")
                .POST(HttpRequest.BodyPublishers.ofString(JSON.writeValueAsString(body)))
                .build();

        HttpResponse<String> response =
                http.send(request, HttpResponse.BodyHandlers.ofString());
        if (response.statusCode() != 200) {
            throw new IOException("Ollama returned " + response.statusCode()
                    + ": " + response.body());
        }
        JsonNode json = JSON.readTree(response.body());
        return json.path("message").path("content").asText().strip();
    }
}

Temperature 0 and the fixed seed make comparisons less variable, without guaranteeing identical answers. num_predict limits the output to 200 tokens so this small model cannot continue repeating indefinitely.

The AskWithoutRag class is the quickest test of the client, and it is the program that printed the wrong answer in section 1:

package com.howtodoinjava.rag;

/**
 * Asks the model about a private recipe with no retrieved text in the prompt.
 * The model has never seen our Recipe Box, so it can only guess.
 */
public class AskWithoutRag {

    public static void main(String[] args) throws Exception {
        String question = "How long do I bake Nani's Lemon Semolina Cake, "
                + "and at what temperature?";

        OllamaClient llm = OllamaClient.fromEnvironment();
        String answer = llm.generate(question);

        System.out.println("Question: " + question);
        System.out.println("Answer:");
        System.out.println(answer);
    }
}

Step 5: Run Index, Retrieve and Generate

Finally, we create the class that runs the three phases. The main() method follows the order of section 2, and each phase is marked with a comment. The two console helpers it calls, ConsoleFormat.shorten() (one line per chunk) and ConsoleFormat.wrap() (line breaks in the answer), only format the output and are on GitHub.

package com.howtodoinjava.rag;

import java.io.IOException;
import java.io.InputStream;
import java.io.UncheckedIOException;
import java.net.ConnectException;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
import java.util.Locale;

import dev.langchain4j.model.embedding.EmbeddingModel;
import dev.langchain4j.model.embedding.onnx.allminilml6v2.AllMiniLmL6V2EmbeddingModel;

/**
 * A minimal RAG pipeline for a private Recipe Box: index, retrieve, generate.
 */
public class RecipeBoxRag {

    static final List<String> RECIPE_FILES = List.of(
            "lemon-semolina-cake.txt",
            "banana-walnut-bread.txt",
            "tomato-rasam.txt",
            "sunday-pancakes.txt");

    static final int TOP_K = 2;

    public static void main(String[] args) throws Exception {
        String question = args.length > 0
                ? String.join(" ", args)
                : "How long do I bake Nani's Lemon Semolina Cake, "
                        + "and at what temperature?";

        // embedding model: runs inside the JVM, no API key, 384 numbers per text
        EmbeddingModel embeddingModel = new AllMiniLmL6V2EmbeddingModel();
        InMemoryVectorStore store = new InMemoryVectorStore();

        // Phase 1: index. Chunk every document, embed every chunk, store both.
        for (String file : RECIPE_FILES) {
            for (String chunk : chunk(loadRecipe(file))) {
                float[] vector = embeddingModel.embed(chunk).content().vector();
                store.add(chunk, vector);
            }
        }
        System.out.println("Indexed " + store.size() + " chunks");

        // Phase 2: retrieve. Embed the question, then run a similarity search.
        float[] questionVector = embeddingModel.embed(question).content().vector();
        List<InMemoryVectorStore.Match> matches = store.search(questionVector, TOP_K);

        System.out.println("Question: " + question);
        System.out.println("Top-" + TOP_K + " chunks:");
        for (InMemoryVectorStore.Match match : matches) {
            System.out.printf(Locale.ROOT, "  %.3f  %s%n", match.score(),
                    ConsoleFormat.shorten(match.chunk()));
        }

        // Phase 3: generate. Put the retrieved text into the prompt, then call the LLM.
        String prompt = buildPrompt(question, matches);
        System.out.println();
        System.out.println("Prompt sent to the model:");
        System.out.println(prompt);
        System.out.println();
        System.out.println("Answer:");
        System.out.println(ConsoleFormat.wrap(generateOrExplain(prompt)));
    }

    /** Splits a recipe into chunks: one per paragraph, each starting with the title. */
    static List<String> chunk(String recipe) {
        String[] paragraphs = recipe.strip().split("\\n\\s*\\n");
        String title = paragraphs[0].strip();
        List<String> chunks = new ArrayList<>();
        for (int i = 1; i < paragraphs.length; i++) {
            chunks.add(title + "\n" + paragraphs[i].strip());
        }
        return chunks;
    }

    /** The augmented prompt: instruction, question, then the retrieved chunks. */
    static String buildPrompt(String question, List<InMemoryVectorStore.Match> matches) {
        List<String> chunks = matches.stream()
                .map(InMemoryVectorStore.Match::chunk)
                .toList();
        return """
                Answer the question using only the context below.
                If the context does not contain the answer,
                say "The Recipe Box does not say."

                Question: %s

                Context:
                %s""".formatted(question, String.join("\n\n", chunks));
    }

    static String generateOrExplain(String prompt) throws InterruptedException {
        try {
            return OllamaClient.fromEnvironment().generate(prompt);
        } catch (ConnectException e) {
            return "(no Ollama server found; start one to see the generated answer)";
        } catch (IOException e) {
            return "(Ollama call failed: " + e.getMessage() + ")";
        }
    }

    static String loadRecipe(String file) {
        String path = "/recipes/" + file;
        try (InputStream in = RecipeBoxRag.class.getResourceAsStream(path)) {
            if (in == null) {
                throw new IllegalStateException("Recipe not found: " + file);
            }
            return new String(in.readAllBytes(), StandardCharsets.UTF_8);
        } catch (IOException e) {
            throw new UncheckedIOException(e);
        }
    }
}

The chunk() method makes one chunk per paragraph and starts every chunk with the recipe title. The title looks redundant, but it is what makes the method paragraph findable, as section 5.1 shows.

The buildPrompt() method puts the question before the chunks; with the chunks first, this small model answered with the 20-minute resting time instead of the baking time.

Step 6: Run the Program

Now we start Ollama (the desktop app starts it, or ollama serve in a terminal) and run the program:

mvn compile exec:java

The program prints the two chunks it retrieved with their scores, the exact prompt it sent, and the answer. You may also see a few harmless warning lines from the embedding runtime and the logging library, which are left out here:

Indexed 12 chunks
Question: How long do I bake Nani's Lemon Semolina Cake, and at what temperature?
Top-2 chunks:
  0.840  Nani's Lemon Semolina Cake | Method: Mix the semolina, su...
  0.788  Nani's Lemon Semolina Cake | Serving: Pour warm lemon syr...

Prompt sent to the model:
Answer the question using only the context below.
If the context does not contain the answer,
say "The Recipe Box does not say."

Question: How long do I bake Nani's Lemon Semolina Cake, and at what temperature?

Context:
Nani's Lemon Semolina Cake
Method: Mix the semolina, sugar and yogurt, and let the batter rest for
20 minutes so the semolina softens. Stir in the butter, zest, cardamom
and baking powder. Bake at 170 C for 38 minutes in a 9-inch tin.

Nani's Lemon Semolina Cake
Serving: Pour warm lemon syrup over the cake while it is still hot. The
cake keeps for 3 days in an airtight box at room temperature.

Answer:
  To bake Nani's Lemon Semolina Cake, you should bake it at 170 C for 38
  minutes.

Both retrieved chunks belong to the right recipe, the method paragraph ranks first (0.840, ahead of 0.788), and the answer now matches the recipe: 170 C for 38 minutes. If Ollama is not running, the program still prints the chunks and the prompt, and says that no Ollama server was found in place of the answer.

5. Why RAG Still Gives Wrong Answers and How to Fix Them

RAG fails in two places: the search returns the wrong chunks, or the model adds facts that are not in the chunks. The RagTrap class runs both cases against the same recipes. This section shows only the lines that differ from RecipeBoxRag; the complete runnable class is RagTrap.java on GitHub, and we run it with:

mvn compile exec:java -Dexec.mainClass=com.howtodoinjava.rag.RagTrap

5.1. Retrieving the Wrong Chunk

The first case uses a chunker that splits each recipe into paragraphs but drops the title, which is the obvious way to write one:

/** The broken chunker: one chunk per paragraph, and the title is dropped. */
static List<String> chunkWithoutTitle() {
    List<String> chunks = new ArrayList<>();
    for (String file : RecipeBoxRag.RECIPE_FILES) {
        String recipe = RecipeBoxRag.loadRecipe(file);
        String[] paragraphs = recipe.strip().split("\\n\\s*\\n");
        for (int i = 1; i < paragraphs.length; i++) {
            chunks.add(paragraphs[i].strip());
        }
    }
    return chunks;
}

We ask “How long do I bake the lemon cake, and at what temperature?” with top-1 retrieval (k = 1), once against an index built from these chunks and once against an index built from the chunks of Step 5:

Trap 1, question: How long do I bake the lemon cake, and at what temperature?
Chunks WITHOUT the title:
  0.687  Serving: Pour warm lemon syrup over the cake while it is ...
  Answer: To bake the lemon cake, you should pour warm lemon syrup over the
  cake while it is still hot. The cake keeps for 3 days in an airtight box
  at room temperature.
Chunks WITH the title:
  0.585  Nani's Lemon Semolina Cake | Method: Mix the semolina, su...
  Answer: To bake the lemon cake, you should bake it at 170 C for 38
  minutes.

Compare the rank of the useful passage in each index. Although both searches use cosine similarity, changing chunk content changes the vectors, so the two top scores alone cannot tell us which index is better. Check whether the returned text actually contains the answer.

Rank	Without the title	Score	With the title	Score
1st	Lemon cake, serving notes	0.687	Lemon cake, method (has the answer)	0.585
2nd	Lemon cake, method (has the answer)	0.432	Lemon cake, serving notes	0.542
3rd	Banana bread, method	0.413	Lemon cake, ingredients	0.368

Without the title, the method paragraph never says “lemon” or “cake”; it says semolina, yogurt and a 9-inch tin. The serving paragraph says “lemon syrup over the cake”, so it ranks first, and the model turns it into a wrong answer about pouring syrup. With the title on every chunk, the method paragraph ranks first and the answer is right.

Every chunk should carry the name of the document it came from, such as the title, the file name or the section heading, because the search sees only the chunk. The chunk should also stay at one topic and under the 256-token limit of the embedding model. And top-k should leave some margin: with k = 1, one near-miss in the ranking loses the answer, so a few chunks are safer as long as they fit in the context window.

5.2. Answering Beyond the Retrieved Text

The second case asks a question the Recipe Box can’t answer: “Can I freeze Nani’s Lemon Semolina Cake?”. The recipe only says the cake keeps for 3 days at room temperature. We send the same retrieved chunks with two prompts, the strict one from buildPrompt() and a loose one that differs only in its first line:

/** The broken prompt: it lets the model add what it "knows" to the context. */
static String loosePrompt(String question, List<InMemoryVectorStore.Match> matches) {
    List<String> chunks = matches.stream()
            .map(InMemoryVectorStore.Match::chunk)
            .toList();
    return """
            Use the context below to help you answer.

            Question: %s

            Context:
            %s""".formatted(question, String.join("\n\n", chunks));
}

Trap 2, question: Can I freeze Nani's Lemon Semolina Cake?
Loose prompt answer:
  Yes, you can freeze Nani's Lemon Semolina Cake. The cake keeps for 3 days
  in an airtight box at room temperature.
Strict prompt answer:
  The Recipe Box does not say.

With the loose prompt, the model says yes and mixes in a sentence from the chunk, so the made-up part reads as if it came from our recipe. The strict prompt tells the model to answer only from the context and gives it an exact sentence for a missing answer, and our code can check for that sentence.

The strict instruction helped in this recorded comparison. It remains a prompt-level control, so test it with missing facts, conflicting passages and misleading documents. Showing sources helps readers verify the answer; application checks are still needed when an answer triggers an action.

6. RAG With LangChain4j and Spring AI

In an application, a framework provides the same pieces we wrote by hand. In LangChain4j, InMemoryEmbeddingStore replaces our list and EmbeddingStoreContentRetriever runs the similarity search with maxResults (our top-k); the LangChain4j RAG tutorial walks through them, and a LangChain4j RAG Example with Embedding Store article is coming on this site. In Spring, the Spring AI RAG chatbot example builds the same pipeline with Spring AI’s EmbeddingModel and VectorStore.

RAG is one of three ways to make an LLM work with your own data, next to fine-tuning and prompt engineering. Which one to pick is the topic of the article Fine-Tuning vs RAG vs Prompt Engineering for LLM Apps.

7. FAQs

7.1. What is the difference between RAG and fine-tuning?

RAG puts our documents in the prompt at question time, and fine-tuning changes the model’s weights by training it further on our data. A changed document takes effect in RAG as soon as it is re-indexed, while a fine-tuned model has to be trained again.

7.2. Do I need a vector database for RAG?

No. A list in memory, like the InMemoryVectorStore in this article, works for a small document set. A vector database becomes useful when the index must survive a restart or comparing every vector gets too slow.

7.3. Can RAG run without an API key or internet connection?

Yes. The example in this article runs fully offline after the first build: all-MiniLM-L6-v2 runs inside the JVM, and Ollama runs the LLM on the same machine. Only the first Maven build and the ollama pull need a network connection.

7.4. Can I use PDF or Word files instead of text files?

Yes. A document loader and a parser turn the file into plain text first, and the rest of the pipeline stays the same. LangChain4j and Spring AI both ship document loaders for this.

7.5. Does RAG stop LLM hallucinations?

No, it reduces them. A hallucination is an answer the model makes up, and as section 5.2 shows, a model can still add facts that are not in the retrieved text.

8. Conclusion

A minimal RAG pipeline indexes documents, retrieves relevant chunks and generates an answer from them. In this example, adding recipe titles helped retrieve the right baking instructions. Keep checking both halves of the pipeline: an excellent prompt cannot recover a passage that was never retrieved, and a correct passage does not force a faithful answer.

9. References

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020)
• LangChain4j documentation: RAG
• LangChain4j documentation: in-process embedding models
• all-MiniLM-L6-v2 model card
• Ollama API reference

Drop me your questions related to RAG in the comments.

Happy Learning !!

Source Code on Github