Manage system log files not to exceed N GB in linux using java

Today, most applications are required to manage their own log files included removing them when they reach at a particular size limit. In this article, i will try to propose a solution for this kind of log file management. I will also suggest the way to capture the output of linux process and log it into some separate log file.

This article is in continuation of my previous post: Auto reloading of configuration, in which i discussed about how an application can take care of its configuration reloading without restarting the application, whenever someone changes the configuration file in file system.

Sections in this post:

• Determining the approach
• Writing the bash script
• Writing script executor
• Adding stream eater
• Review all

Determining the approach

I believe, to get the job done in hand, we will need a bash script for sure. Linux has some very useful built in commands which can do this job in no time when executing though command window. Advantage of bash script includes the decoupling of file handler code of actual application, which can be modified if any platform specific change is needed.

Writing the bash script

I have written a script for demonstration. You can use it if you want :

#!/bin/bash
###############################################################################
#
# discCleanUp.sh
#
################################################################################

DIR_PATH=$1
NUM_OF_DAYS=$2
DIR_SIZE=$3
SIZE=1024

DIR_SIZE=$(($DIR_SIZE * $SIZE * $SIZE))

# Command to find the File older then NUM_OF_DAYS and removing them from directory DIR_PATH
find $DIR_PATH -mtime +$NUM_OF_DAYS -exec rm {} ;

#Go to specified directory
cd $DIR_PATH

# Command to get the current directory size.
CURR_DIR_SIZE=`du -sk | cut -f1`

while [[ $CURR_DIR_SIZE -gt DIR_SIZE ]];

do

echo $CURR_DIR_SIZE

FILE=`ls -1tra | head -1`

rm -rf $FILE

# Command to get the current directory size.
CURR_DIR_SIZE=`du -sk | cut -f1`

done

exit 0

In above script, DIR_PATH, NUM_OF_DAYS, DIR_SIZE are command line parameters. Whereas SIZE is a constant for calculation purpose. In above script, line number 16 will delete the log files older than n number of days.

In Line number 22, script checks the current size of directory with du command. If the size exceeds the DIR_SIZE (in GB), script will start picking oldest files using ‘ls -1tra | head -1‘ command and start removing them one by one. It will continue until directory size does not come under desired limit.

Writing script executor

As for as this article, core component will remain above bash script, but still you need a way to run .sh file from your application. Best approach will be using a thread  (You can use Executors if want). This thread will execute above bash script periodically at a configurable rate. I will not write that part of code. If you face any issue in executing code, write down a comment.

Let me give you a sample code ::

package corejava.dischandler;

import java.io.IOException;

/**
* Title: DiskFileManager.java
* Description :- This class provides API to check file size and file time threshold and
* if threshold is crossed then deletes file to come within threshold.
*/
public final class DiskFileManager
{

	private static DiskFileManager diskFileManager=new DiskFileManager();
	private DiskFileManager()
	{
	}

	/**
	* <pre>
	* <b>checkAndDeleteFiles</b>
	* <b>Description:This method is called to clean the files from the file system.</b>
	* </pre>
	* @throws InterruptedException
	* @throws IOException
	* @throws InstantiationException
	*/
	public   void checkAndDeleteFiles() throws InterruptedException, IOException, InstantiationException
	{
		try
		{
			StringBuffer outStr = new StringBuffer();
			StringBuffer errStr = new StringBuffer();
			String scriptFileName = "./discfilehandler.sh";
			String command = scriptFileName + "/logs 1 1";

			System.out.println("Command to be executed  :- " + command);

			Process output = Runtime.getRuntime().exec(command);
			StreamEater errorEater = new StreamEater(output.getErrorStream(),errStr);
			StreamEater outputEater = new StreamEater(output.getInputStream(),outStr);
			errorEater.start();
			outputEater.start();
			try
			{
				int ret = output.waitFor();
				errorEater.join();
				outputEater.join();

				System.out.println();

				//logger.info("execute(): Error Stream:" + errStr + " ; execute(): Output Stream:" + outStr + " ; execute(): ExitValue:" + ret);

			} catch (InterruptedException e)
			{
				throw e;
			}
		} catch (IOException e)
		{
			throw e;
		}
	}
	/**
	* <pre>
	* <b>getInstance</b>
	* <b>Description:This method is used to get the instance of Disk File Manager.</b>
	* </pre>
	* @return
	*/
	public static DiskFileManager getInstance()
	{
		return diskFileManager;
	}
}

Above code will locate the script file in class path and pass the required parameter. As in our case, “/logs 1 1” are 3 parameters, meaning

• directory to monitor is /logs,
• delete one day old log files and
• do not store more than 1 GB of log files at any time.

Adding stream eater

So for we have added an bash script and an executor code for running the script. You have to write your own Thread to run periodically above executor.

Now we will look into StreamEater which actually collects command output and gives to your application code, so application can either log it or do whatever it wants.

We have already used StreamEater in above executor, now i am giving its code:;

package corejava.dischandler;

import java.io.BufferedReader;
import java.io.InputStream;
import java.io.InputStreamReader;

public class StreamEater extends Thread
{

	/**
	* Stream Eater thread to eat up every thing that is the output to the
	* execute command
	*/
	private InputStream  iStream;

	public InputStream getiStream() {
		return iStream;
	}

	public void setiStream(InputStream iStream) {
		this.iStream = iStream;
	}

	private StringBuffer stringBuffer;

	public StringBuffer getStringBuffer() {
		return stringBuffer;
	}

	public void setStringBuffer(StringBuffer stringBuffer) {
		this.stringBuffer = stringBuffer;
	}

	/**
	* This is the constructor.
	*
	* @param is - It is the input stream
	* @param type - type of input stream
	* @param redirect - string buffer to contain the output.
	* @throws InstantiationException
	*/

	public StreamEater(InputStream is, StringBuffer redirect) throws InstantiationException
	{
		this.iStream = is;
		if (redirect == null)
		{
			throw new InstantiationException("String Buffer Reference , second param can not be null");
		}
		this.stringBuffer = redirect;
	}

	/**
	* This is method called when the thread is started. It captures the stream
	* output and and store it into the buffer.
	*/
	public void run()
	{
		InputStreamReader isr = null;
		BufferedReader br = null;
		try
		{

			isr = new InputStreamReader(iStream);
			br = new BufferedReader(isr);
			String line;
			while ((line = br.readLine()) != null)
			{
				stringBuffer.append(line).append(System.getProperty("line.separator"));
			}

		}
		catch (Exception ex)
		{
			ex.printStackTrace();
		}
		finally
		{
		if (isr != null)
		{
			try
			{
				isr.close();
			}
			catch (Exception e)
			{
				e.printStackTrace();
			}
		}
		if (br != null)
		{
			try
			{
				br.close();
			}
			catch (Exception e)
			{
				e.printStackTrace();
			}
		}
		}
	}

}

As you see these are small threads waiting to collect the process output, that all.

Review all

Using above classes and given bash script, you can easily develop some application component which will ensure the size of log files do not exceed some predefined number. As you application is taking care of deleting the actual log files, this can be customized at any level.

Hope you got some useful information in this post. Please write down to me if need any help or have any query.

Happy Learning !!