Learn to read small and large files from the filesystem using the Java NIO APIs Path, FileChannel, ByteBuffer and MappedByteBuffer.
- We are using the RandomAccessFile instance that behaves like a large array of bytes stored in the file system. It uses file pointers that act as a cursor to maintain the current read location in the file.
- A ByteBuffer represents the buffered bytes in the memory during the read/write operations.
- A MappedByteBuffer is a direct byte buffer whose content is a memory-mapped region of a file.
1. Reading Small Files with ByteBuffer and FileChannel
Use this technique to read a small file. The idea is to create a ByteBuffer large enough where all the file content fits into the buffer, and the file can be read in a single read() operation.
try(RandomAccessFile aFile = new RandomAccessFile("test.txt", "r");
FileChannel inChannel = aFile.getChannel();) {
long fileSize = inChannel.size();
//Create buffer of the file size
ByteBuffer buffer = ByteBuffer.allocate((int) fileSize);
inChannel.read(buffer);
buffer.flip();
// Verify the file content
for (int i = 0; i < fileSize; i++) {
System.out.print((char) buffer.get());
}
} catch (IOException e) {
e.printStackTrace();
}
2. Reading Large Files with ByteBuffer and FileChannel
Use this technique to read a large file where all the file content will not fit into the buffer at a time. To avoid OutOfMemory issues, we can read the file in chunks with a fixed size small buffer.
try (RandomAccessFile aFile = new RandomAccessFile("test.txt", "r");
FileChannel inChannel = aFile.getChannel();) {
//Buffer size is 1024
ByteBuffer buffer = ByteBuffer.allocate(1024);
while (inChannel.read(buffer) > 0) {
buffer.flip();
for (int i = 0; i < buffer.limit(); i++) {
System.out.print((char) buffer.get());
}
buffer.clear(); // do something with the data and clear/compact it.
}
} catch (IOException e) {
e.printStackTrace();
}
3. Reading a File using MappedByteBuffer
MappedByteBuffer
extends the ByteBuffer
class with operations that are specific to memory-mapped file regions.
try (RandomAccessFile aFile = new RandomAccessFile("test.txt", "r");
FileChannel inChannel = aFile.getChannel();) {
MappedByteBuffer buffer = inChannel
.map(FileChannel.MapMode.READ_ONLY, 0, inChannel.size());
buffer.load();
for (int i = 0; i < buffer.limit(); i++) {
System.out.print((char) buffer.get());
}
buffer.clear(); // do something with the data and clear/compact it.
} catch (IOException e) {
e.printStackTrace();
}
All the above techniques will read the content of the file and print it to the console.
Happy Learning !!
MappedByteBuffer buffer = inChannel.map(FileChannel.MapMode.READ_ONLY, 0, inChannel.size());
—-> inChannel.size()
it looks like size of the file. What will happen if file is having 8-10 GB. Will mappedbytebuffer is suggestible and how internally it will work without out of memory
It is size of the region to be mapped. It is not buffer size.
https://docs.oracle.com/javase/7/docs/api/java/nio/channels/FileChannel.html#map%28java.nio.channels.FileChannel.MapMode,%20long,%20long%29
Nice examples. I just wonder that on your first example “1) Read a small file in buffer of file size” why did you not use finally{} block for aFile resource?
When you close the reader it will close the file for you, theoretically you would miss the case when the reader throws an exception in the constructor after the file has opened, but I believe the only exceptions that could occur at that time would be things like out of memory and similar which would forcibly close down your JVM anyways.
Hi Lokesh,
I want to divide an xml file according to size using NIO. We can read char or byte only from buffer, so how can i check whether the xml file is proper or not.
Animesh
Brilliant question. And that means I do not know the answer 🙂 I will try to find a solution for this problem when time permits. I you get any solution, then please update me as well. Thanks !!
Hi,
Could you please explain why in the example “1) Read a small file in buffer of file size”, you write:
20 buffer.rewind();
21 buffer.flip();
Wouldn’t be enough to use buffer.flip(), like in the example “2) Read a large file in chunks with fixed size buffer”?
You are right. I must have forget to comment that line. Actually both methods are almost similar. In fact, you can use rewind( ) to go back and reread the data in a buffer that has already been flipped.
How multiple threads could read single file parallely
Hi Lokesh, thanks for the informative write-up. The key difference is that in NIO you are reading a buffer not a stream. And NIO is by design non-blocking. So the thread is not blocked.
Just a question. I have to process huge text files (billions of records, each) line by line. Should I read them with IO, NIO, or NIO2? Which class?
IO is not good solution. Use ReadFileWithFixedSizeBuffer example. In java 8, you can use https://howtodoinjava.com/java8/read-file-line-by-line/
Sorry but the old famous way do not the same – here is reading line by line – and with a Reader.
The three other sources reading byte by byte – with a stream.
No good!
Dear friend, it’s not about lines or bytes or streams. It’s about how it was and how it is.
very useful
thanks