Apache Lucene: Index and Search Text Files in Java

To index and search text files with Apache Lucene, we turn each file into a Lucene Document, write it to an index folder with IndexWriter, and search that folder with IndexSearcher and a query from QueryParser. Lucene builds an inverted index, which maps every word to the files that contain it, so a search reads only the matching entries instead of scanning every file.

We use Lucene when an app needs full-text search over its own files without running a search server, for example a desktop note-taking tool, a log viewer or a help desk tool that searches hundreds of troubleshooting notes.

The following example indexes every file in the notes folder and searches the index for files that contain both vpn and windows. The example uses Lucene 10.5.2 on Java 25.

try (Directory directory = FSDirectory.open(Path.of("index"));
     IndexWriter writer = new IndexWriter(directory, new IndexWriterConfig(new StandardAnalyzer()));
     Stream<Path> files = Files.list(Path.of("notes"))) {
  for (Path file : files.toList()) {
    Document doc = new Document();
    doc.add(new StringField("path", file.toString(), Field.Store.YES));          // stored, matched as one value
    doc.add(new TextField("contents", Files.readString(file), Field.Store.NO));   // analyzed, not stored
    writer.updateDocument(new Term("path", file.toString()), doc);                // replaces an older copy
  }
}

try (Directory directory = FSDirectory.open(Path.of("index"));
     DirectoryReader reader = DirectoryReader.open(directory)) {
  IndexSearcher searcher = new IndexSearcher(reader);
  Query query = new QueryParser("contents", new StandardAnalyzer()).parse("vpn AND windows");
  TopDocs topDocs = searcher.search(query, 10);                                   // 1 hit
  for (ScoreDoc scoreDoc : topDocs.scoreDocs) {
    String path = searcher.storedFields().document(scoreDoc.doc).get("path");    // notes/vpn-setup.txt
  }
}

Notice that the file content goes into a TextField, which Lucene splits into words, whereas the path goes into a StringField, which Lucene keeps as one exact value. Only stored fields come back in the search results, so we get the path of the file and not its text.

Next, we see how Lucene analyzes text and builds the index. After that, we build a reusable indexer and searcher, try the query syntax on real files, handle invalid queries and keep the index in sync when files change or disappear.

1. How Lucene Indexes and Searches Text Files

A plain text file has no structure that Lucene understands, so we wrap it in a Document, which is a list of named fields. A field either holds text that Lucene splits into words or holds one exact value, such as a file path.

An analyzer turns the text of a field into terms (the words that Lucene indexes). The StandardAnalyzer splits the text at spaces and punctuation and converts every word to lowercase. IndexWriter writes the terms into an inverted index on disk, where each term points to the documents that contain it.

For example, a help desk team keeps its troubleshooting notes as .txt files in a shared folder. A search for windows in a folder with 500 notes should not open 500 files. With the inverted index, Lucene reads the entry for the term windows and gets the two matching files at once.

Searching is the same process in reverse. QueryParser reads the query text, runs it through the same analyzer and builds a Query object. IndexSearcher looks up the query terms in the index, gives each matching document a relevance score and returns the best matches as TopDocs.

The same analyzer must process the text at indexing time and at search time. If the index has windows and the query asks for Windows without lowercasing, the terms differ and the search finds nothing, as section 6.1 shows.

2. Maven Dependencies

The Lucene core artifact has IndexWriter, IndexSearcher and StandardAnalyzer. QueryParser is in a separate artifact, and lucene-analysis-common adds analyzers for other languages, such as EnglishAnalyzer.

<dependencies>
  <dependency>
    <groupId>org.apache.lucene</groupId>
    <artifactId>lucene-core</artifactId>
    <version>10.5.2</version>
  </dependency>
  <dependency>
    <groupId>org.apache.lucene</groupId>
    <artifactId>lucene-queryparser</artifactId>
    <version>10.5.2</version>
  </dependency>
  <dependency>
    <groupId>org.apache.lucene</groupId>
    <artifactId>lucene-analysis-common</artifactId>
    <version>10.5.2</version>
  </dependency>
</dependencies>

Lucene 10 needs Java 21 or newer. On Java 25, Lucene prints two warnings at startup, one about the incubating Vector API and one about restricted native methods. Both are harmless, and we can enable the features with two JVM options.

java --add-modules jdk.incubator.vector --enable-native-access=ALL-UNNAMED -cp ... com.howtodoinjava.demo.lucene.textfiles.TextFileSearchDemo

3. Indexing a Folder of Text Files

Before writing the indexer, we decide which fields each file gets and which of them Lucene stores. An indexed field is searchable. A stored field keeps its original value in the index, so the search results can return it. A field can be indexed, stored or both.

Field	Lucene type	Indexed as	Stored	Used for
path	StringField	one exact term	yes	showing the file in results, updating and deleting its document
modified	LongField	number	yes	range queries such as “changed in the last 24 hours”
contents	TextField	analyzed terms	no	full-text search

We don’t store contents, because the text is already on disk and storing it would copy every file into the index. When the results must show the matching text, we store the field, as section 6.2 explains.

3.1. The Indexer Class

The following example is an indexer for the help desk notes. The method indexFolder() finds every .txt file under a folder with Files.walk(), and indexFile() creates one Document per file.

public static int indexFolder(Path docsDir, Path indexDir) throws IOException {
  List<Path> files;
  try (Stream<Path> paths = Files.walk(docsDir)) {
    files = paths.filter(Files::isRegularFile)
        .filter(p -> p.getFileName().toString().endsWith(".txt"))
        .sorted()
        .toList();
  }

  IndexWriterConfig config = new IndexWriterConfig(new StandardAnalyzer());
  config.setOpenMode(IndexWriterConfig.OpenMode.CREATE_OR_APPEND);

  try (Directory directory = FSDirectory.open(indexDir);
       IndexWriter writer = new IndexWriter(directory, config)) {
    for (Path file : files) {
      indexFile(writer, file);
    }
    writer.commit();
  }
  return files.size();
}

The open mode CREATE_OR_APPEND creates a new index when the folder is empty and adds to the existing index otherwise. The commit() call makes the changes visible to new readers, and closing the writer commits too.

static void indexFile(IndexWriter writer, Path file) throws IOException {
  String path = file.toString();
  long modified = Files.getLastModifiedTime(file).toMillis();

  try (BufferedReader reader = Files.newBufferedReader(file, StandardCharsets.UTF_8)) {
    Document doc = new Document();
    doc.add(new StringField("path", path, Field.Store.YES));        // exact value, stored
    doc.add(new LongField("modified", modified, Field.Store.YES));  // number, stored
    doc.add(new TextField("contents", reader));                     // analyzed, not stored

    writer.updateDocument(new Term("path", path), doc);             // delete old version + add
    System.out.println("Indexed " + path);
  }
}

The TextField constructor that takes a Reader streams the file content into the analyzer, so a large file never sits in memory as one String. A TextField built from a Reader is never stored.

The call updateDocument(term, doc) first deletes every document whose path field equals the given term and then adds the new document. So running the indexer twice on the same folder does not create duplicates. After two runs, reader.numDocs() still returns 3.

The demo creates three notes in the notes folder and indexes them.

Indexed notes/printer-offline.txt
Indexed notes/reset-password.txt
Indexed notes/vpn-setup.txt
Files indexed: 3

The index folder contains the files of one segment (a self-contained part of the index) plus the segments_1 file, which lists the committed segments. The write.lock file stays in the folder after the writer closes, which is normal.

_0.cfe
_0.cfs
_0.si
segments_1
write.lock

3.2. What the Analyzer Does With the Text

The analyzer decides which words a search can match. For example, StandardAnalyzer turns the first sentence of vpn-setup.txt into lowercase words and keeps every word. Since Lucene 8, the no-argument StandardAnalyzer uses no stop words, so common words such as the and on are indexed too.

EnglishAnalyzer also removes English stop words and reduces each word to its stem, so a search for window also finds windows.

List<String> standard = tokens(new StandardAnalyzer(), "Install the VPN client on Windows or macOS.");
// [install, the, vpn, client, on, windows, or, macos]

List<String> english = tokens(new EnglishAnalyzer(), "Install the VPN client on Windows or macOS.");
// [instal, vpn, client, window, maco]

The helper tokens() reads the terms from analyzer.tokenStream(), and the class AnalyzerTokens in the example project has its code. We pick StandardAnalyzer for notes that mix product names, error codes and words in several languages, and EnglishAnalyzer when the files are English prose and users type word forms such as installing or printers.

4. Searching the Indexed Files

The searcher opens a DirectoryReader, which is a point-in-time view of the index. Changes that the writer commits after we open the reader are not visible to that reader, so a long-running app opens a new reader after each commit.

4.1. The Searcher Class

The following example searches the contents field and returns the path and the score of each matching file. SearchHit is a small record with these two values.

public static List<SearchHit> search(Path indexDir, String userQuery, int maxHits) throws IOException {
  try (Directory directory = FSDirectory.open(indexDir);
       DirectoryReader reader = DirectoryReader.open(directory)) {

    IndexSearcher searcher = new IndexSearcher(reader);
    Query query = parse(userQuery);
    TopDocs topDocs = searcher.search(query, maxHits);
    StoredFields storedFields = searcher.storedFields();

    List<SearchHit> hits = new ArrayList<>();
    for (ScoreDoc scoreDoc : topDocs.scoreDocs) {
      Document doc = storedFields.document(scoreDoc.doc);
      hits.add(new SearchHit(doc.get("path"), scoreDoc.score));
    }
    return hits;
  }
}

Each ScoreDoc holds an internal document number and a score. We pass the number to storedFields().document() to load the stored fields of the document. In Lucene 9 and earlier, the same call was searcher.doc(id), which Lucene 10 removed.

4.2. Query Syntax With Real Results

The classic QueryParser supports a search-box syntax that users already know from other search engines. Each row shows a query against the three notes, with the parsed query and the real scores.

Query	Meaning	Parsed query	Matching files (score)
password	one word	contents:password	reset-password.txt (0.2964), vpn-setup.txt (0.2209)
vpn AND windows	both words	+contents:vpn +contents:windows	vpn-setup.txt (0.6820)
“network cable”	exact phrase	contents:”network cable”	printer-offline.txt (0.8523)
*print**	words starting with print	*contents:print**	printer-offline.txt (1.0000)
pasword~	similar spelling	contents:pasword~2	reset-password.txt (0.2541), vpn-setup.txt (0.1894)
windows -vpn	windows but not vpn	contents:windows -contents:vpn	printer-offline.txt (0.2042)

A few rules explain the results.

• Words without an operator are joined with OR, so vpn windows finds files with either word.
• The operators AND, OR and NOT must be in upper case. A lowercase and is a normal search word.
• Wildcard queries such as *print** give every match the same constant score of 1.0, because Lucene does not rank the expanded terms.
• The fuzzy query pasword~ matches terms up to two edits away, so password matches although the query misses one letter.

The score comes from the BM25 ranking formula. A file scores higher when the query term is rare across all files and appears in a short field. For example, reset-password.txt scores higher than vpn-setup.txt for password because the term appears twice in it.

4.3. Handling Invalid Queries From Users

When the text in a search box breaks the query syntax, QueryParser.parse() throws a checked ParseException. For example, a user types vpn AND and presses Enter before typing the second word.

Query query = new QueryParser("contents", new StandardAnalyzer()).parse("vpn AND");
// ParseException: Cannot parse 'vpn AND': Encountered "<EOF>" at line 1, column 7.

A search box should never answer with an error page because of a missing word. So parse() catches the exception and searches the text as plain words with QueryBuilder, which ignores the query syntax.

static Query parse(String userQuery) {
  StandardAnalyzer analyzer = new StandardAnalyzer();
  try {
    return new QueryParser("contents", analyzer).parse(userQuery);
  } catch (ParseException e) {
    Query plainWords = new QueryBuilder(analyzer).createBooleanQuery("contents", userQuery);
    return plainWords != null ? plainWords : new MatchNoDocsQuery();
  }
}

The method createBooleanQuery() returns null when the analyzer finds no terms in the text, for example for ???, so we return a MatchNoDocsQuery in that case. With the fallback, vpn AND finds all three files, and vpn-setup.txt comes first.

Query: vpn AND
  notes/vpn-setup.txt          score 0.5238
  notes/reset-password.txt     score 0.0615
  notes/printer-offline.txt    score 0.0580

The two lower scores come from the word and, which the fallback searches as a normal word and which appears in both other notes.

5. Keeping the Index in Sync With the Folder

The index is a copy of the folder at the time of the last indexing run. When someone edits a note, the index still has the old words. When someone deletes a note, the search still returns its path.

Running indexFolder() again handles changed and new files, because updateDocument() replaces the document of each file. A deleted file is different, since the indexer never sees it. So we delete its document with the same path term.

public static void removeFile(Path indexDir, Path file) throws IOException {
  IndexWriterConfig config = new IndexWriterConfig(new StandardAnalyzer());
  try (Directory directory = FSDirectory.open(indexDir);
       IndexWriter writer = new IndexWriter(directory, config)) {
    writer.deleteDocuments(new Term("path", file.toString()));
    writer.commit();
  }
}

The test in the example project changes vpn-setup.txt to a note about Linux laptops and runs the indexer again. After that, linux finds vpn-setup.txt, and windows finds only printer-offline.txt. After removeFile(), the search for printer returns no files.

Only one IndexWriter can hold the write.lock of an index at a time, and a second writer on the same folder throws a LockObtainFailedException. For a folder that changes all day, we keep one writer open for the life of the app, and we use the modified field to re-index only the files whose time stamp changed.

6. Lucene Text File Search FAQs

6.1. Why Does My Lucene Search Return No Results?

In most cases, the query terms differ from the indexed terms. StandardAnalyzer indexes Windows as windows, and a TermQuery skips the analyzer, so it searches for the exact term we pass.

int upperHits = searcher.count(new TermQuery(new Term("contents", "Windows")));   // 0 (indexed as "windows")
int lowerHits = searcher.count(new TermQuery(new Term("contents", "windows")));   // 2

There are two more common causes.

• The query searches a StringField such as path through QueryParser. The parser analyzes the text, but the field holds one exact term, so we search a StringField with a TermQuery and the full value.
• The index and the query use different analyzers, for example EnglishAnalyzer at indexing time and StandardAnalyzer in QueryParser.

6.2. How Do I Show the Matching Text in the Search Results?

We store the content with new TextField(“contents”, text, Field.Store.YES), because a field that is not stored comes back as null. For example, storedFields().document(id).get(“contents”) returns null in our example. With the stored text, the Lucene highlighter cuts out the fragments around the matched words.

6.3. Can Lucene Index PDF or Word Files?

No, not by itself. Lucene indexes text that we give it, so we first extract the text from a PDF or a Word file with a library such as Apache Tika and add that text to a TextField. The rest of the indexer stays the same.

6.4. Should I Use Lucene or Elasticsearch?

We use Lucene when the search runs inside one Java app and the index fits on one machine, such as the help desk notes in this example. Elasticsearch runs Lucene inside a separate server cluster and adds a REST API, replication and scaling across machines, at the cost of running and monitoring that cluster. For entities stored in a database, Hibernate Search keeps a Lucene index in sync with the tables.

7. Conclusion

Lucene searches text files through an inverted index. We turn each file into a Document with a stored StringField for the path and an analyzed TextField for the content, and IndexWriter.updateDocument() keeps one document per file across indexing runs.

For searching, QueryParser gives users the familiar syntax with AND, phrases, wildcards and fuzzy matching, and storedFields() returns the path of each hit in Lucene 10. Invalid user input throws a ParseException, so the search falls back to plain words instead of failing.

The analyzer must be the same at indexing and at search time, and the index needs a delete for every removed file. If you are new to Lucene, the Lucene getting started tutorial covers the basic classes with an in-memory index.

8. References

• Apache Lucene 10 Core API
• IndexWriter JavaDoc
• IndexSearcher JavaDoc
• Classic QueryParser syntax
• StandardAnalyzer JavaDoc
• Lucene demo: IndexFiles and SearchFiles
• Migrating to Lucene 10

Happy Learning !!

Source Code on Github