Java MappedByteBuffer

Learn about Java memory-mapped files and learn to read and write content from a memory mapped file with the help of RandomAccessFile and MemoryMappedBuffer.

1. Java Memory-mapped IO

If you know how java IO works at lower level, then you will be aware of buffer handling, memory paging and other such concepts. For conventional file I/O, in which user processes issue read() and write() system calls to transfer data, there is almost always one or more copy operations to move the data between these filesystem pages in kernel space and a memory area in user space. This is because there is not usually a one-to-one alignment between filesystem pages and user buffers.

There is, however, a special type of I/O operation supported by most operating systems that allows user processes to take maximum advantage of the page-oriented nature of system I/O and completely avoid buffer copies. This is called memory-mapped I/O and we are going to learn few things here around memory-mapped files.

2. Java Memory-Mapped Files

Memory-mapped I/O uses the filesystem to establish a virtual memory mapping from user space directly to the applicable filesystem pages. With a memory-mapped file, we can pretend that the entire file is in memory and that we can access it by simply treating it as a very large array. This approach greatly simplifies the code we write in order to modify the file.

Read More : Working With Buffers

To do both writing and reading in memory mapped files, we start with a RandomAccessFile, get a channel for that file. Memory mapped byte buffers are created via the FileChannel.map() method. This class extends the ByteBuffer class with operations that are specific to memory-mapped file regions.

A mapped byte buffer and the file mapping that it represents remain valid until the buffer itself is garbage-collected. Note that you must specify the starting point and the length of the region that you want to map in the file; this means that you have the option to map smaller regions of a large file.

Example 1: Writing to a memory mapped file

import java.io.RandomAccessFile;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;

public class MemoryMappedFileExample 
{
	static int length = 0x8FFFFFF; 	

	public static void main(String[] args) throws Exception 
	{
		try(RandomAccessFile file = new RandomAccessFile("howtodoinjava.dat", "rw")) 
		{
			MappedByteBuffer out = file.getChannel()
										.map(FileChannel.MapMode.READ_WRITE, 0, length);
			
			for (int i = 0; i < length; i++) 
			{
				out.put((byte) 'x');
			}
			
			System.out.println("Finished writing");
		}
	}
}

The file created with the above program is 128 MB long, which is probably larger than the space your OS will allow. The file appears to be accessible all at once because only portions of it are brought into memory, and other parts are swapped out. This way a very large file (up to 2 GB) can easily be modified.

3. File Mapping Modes

Like conventional file handles, file mappings can be writable or read-only.

  • The first two mapping modes, MapMode.READ_ONLY and MapMode.READ_WRITE, are fairly obvious. They indicate whether you want the mapping to be read-only or to allow modification of the mapped file.
  • The third mode, MapMode.PRIVATE, indicates that you want a copy-on-write mapping. This means that any modifications you make via put() will result in a private copy of the data that only the MappedByteBuffer instance can see.

    No changes will be made to the underlying file, and any changes made will be lost when the buffer is garbage collected. Even though a copy-on-write mapping prevents any changes to the underlying file, you must have opened the file for read/write to set up a MapMode.PRIVATE mapping. This is necessary for the returned MappedByteBuffer object to allow put()s.

You’ll notice that there is no unmap() method. Once established, a mapping remains in effect until the MappedByteBuffer object is garbage collected.

Also, mapped buffers are not tied to the channel that created them. Closing the associated FileChannel does not destroy the mapping; only disposal of the buffer object itself breaks the mapping.

A MemoryMappedBuffer has a fixed size, but the file it’s mapped to is elastic. Specifically, if a file’s size changes while the mapping is in effect, some or all of the buffer may become inaccessible, undefined data could be returned, or unchecked exceptions could be thrown.

Be careful about how files are manipulated by other threads or external processes when they are memory-mapped.

4. Benefits of Memory Mapped Files

Memory-Mapped IO have several advantages over normal I/O:

  1. The user process sees the file data as memory, so there is no need to issue read() or write() system calls.
  2. As the user process touches the mapped memory space, page faults will be generated automatically to bring in the file data from disk. If the user modifies the mapped memory space, the affected page is automatically marked as dirty and will be subsequently flushed to disk to update the file.
  3. The virtual memory subsystem of the operating system will perform intelligent caching of the pages, automatically managing memory according to system load.
  4. The data is always page-aligned, and no buffer copying is ever needed.
  5. Very large files can be mapped without consuming large amounts of memory to copy the data.

5. How to read a Memory-Mapped File

To read a file using memory mapped IO, use below code template:

import java.io.File;
import java.io.RandomAccessFile;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;

public class MemoryMappedFileReadExample 
{
	private static String bigExcelFile = "bigFile.xls";

	public static void main(String[] args) throws Exception 
	{
		try (RandomAccessFile file = new RandomAccessFile(new File(bigExcelFile), "r"))
		{
			//Get file channel in read-only mode
			FileChannel fileChannel = file.getChannel();
	        
	        //Get direct byte buffer access using channel.map() operation
	        MappedByteBuffer buffer = fileChannel.map(FileChannel.MapMode.READ_ONLY, 0, fileChannel.size());
	        
	        // the buffer now reads the file as if it were loaded in memory. 
	        System.out.println(buffer.isLoaded()); 	//prints false
	        System.out.println(buffer.capacity());	//Get the size based on content size of file
	        
	        //You can read the file from this buffer the way you like.
	        for (int i = 0; i < buffer.limit(); i++)
	        {
	            System.out.print((char) buffer.get()); //Print the content of file
	        }
		}
	}
}

6. How to write into a Memory-Mapped File

To write data into a file using memory mapped IO, use below code template:

import java.io.File;
import java.io.RandomAccessFile;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;

public class MemoryMappedFileWriteExample {
	private static String bigTextFile = "test.txt";

	public static void main(String[] args) throws Exception 
	{
		// Create file object
		File file = new File(bigTextFile);
		
		//Delete the file; we will create a new file
		file.delete();
					
		try (RandomAccessFile randomAccessFile = new RandomAccessFile(file, "rw"))
		{
			// Get file channel in read-write mode
			FileChannel fileChannel = randomAccessFile.getChannel();

			// Get direct byte buffer access using channel.map() operation
			MappedByteBuffer buffer = fileChannel.map(FileChannel.MapMode.READ_WRITE, 0, 4096 * 8 * 8);

			//Write the content using put methods
			buffer.put("howtodoinjava.com".getBytes());
		}
	}
}

Drop me your comments and thoughts in the comments section.

Happy Learning !!

Was this post helpful?

Join 7000+ Fellow Programmers

Subscribe to get new post notifications, industry updates, best practices, and much more. Directly into your inbox, for free.

13 thoughts on “Java MappedByteBuffer”

  1. You really should update your code to show how to close the RandomAccessFile and FileChannel (e.g. in try-with-resources), and set the MappedByteBuffer reference to null (to encourage GC), once you have finished with the file.

    Reply
  2. Hi,
    Is there any way to access memory files like buffers , streams or … with an address ?
    in an android project ,
    for example I want to access to these memory objects like access to any file in phone storage with an its address

    Reply
  3. i am fetching data of around 15GB from a Database and creating a file from that. Will memory mapped file help me ?

    Reply
  4. Must mention that releasing the memory does not work on some unices as expected. Then you need explicitely call sun package to release memory. Off course the warning for usin sun internal api eill show up.

    Reply
  5. Hi,
    I like to check how to read a file more that 2GB. Can I define the offset and split the file and make more than one read? I was not successful doing it. Can you please throw your ideas on it.
    Thanks
    Sudhagar C

    Reply
  6. Thanks for the clear tutorial, it helped me a lot.

    I was searching for a way to read quickly bytes in large file and I was using a ReadableByteChannel. I compared both methods and it seems that the MappedByteBuffer is faster.

    Reply

Leave a Comment

HowToDoInJava

A blog about Java and its related technologies, the best practices, algorithms, interview questions, scripting languages, and Python.