HowToDoInJava

  • Python
  • Java
  • Spring Boot
  • Dark Mode
Home / Linux / Manage system log files not to exceed N GB in linux using java

Manage system log files not to exceed N GB in linux using java

Today, most applications are required to manage their own log files included removing them when they reach at a particular size limit. In this article, i will try to propose a solution for this kind of log file management. I will also suggest the way to capture the output of linux process and log it into some separate log file.

This article is in continuation of my previous post: Auto reloading of configuration, in which i discussed about how an application can take care of its configuration reloading without restarting the application, whenever someone changes the configuration file in file system.

Sections in this post:

  • Determining the approach
  • Writing the bash script
  • Writing script executor
  • Adding stream eater
  • Review all

Determining the approach

I believe, to get the job done in hand, we will need a bash script for sure. Linux has some very useful built in commands which can do this job in no time when executing though command window. Advantage of bash script includes the decoupling of file handler code of actual application, which can be modified if any platform specific change is needed.

Writing the bash script

I have written a script for demonstration. You can use it if you want :

#!/bin/bash
###############################################################################
#
# discCleanUp.sh
#
################################################################################

DIR_PATH=$1
NUM_OF_DAYS=$2
DIR_SIZE=$3
SIZE=1024

DIR_SIZE=$(($DIR_SIZE * $SIZE * $SIZE))

# Command to find the File older then NUM_OF_DAYS and removing them from directory DIR_PATH
find $DIR_PATH -mtime +$NUM_OF_DAYS -exec rm {} ;

#Go to specified directory
cd $DIR_PATH

# Command to get the current directory size.
CURR_DIR_SIZE=`du -sk | cut -f1`

while [[ $CURR_DIR_SIZE -gt DIR_SIZE ]];

do

echo $CURR_DIR_SIZE

FILE=`ls -1tra | head -1`

rm -rf $FILE

# Command to get the current directory size.
CURR_DIR_SIZE=`du -sk | cut -f1`

done

exit 0

In above script, DIR_PATH, NUM_OF_DAYS, DIR_SIZE are command line parameters. Whereas SIZE is a constant for calculation purpose. In above script, line number 16 will delete the log files older than n number of days.

In Line number 22, script checks the current size of directory with du command. If the size exceeds the DIR_SIZE (in GB), script will start picking oldest files using ‘ls -1tra | head -1‘ command and start removing them one by one. It will continue until directory size does not come under desired limit.

Writing script executor

As for as this article, core component will remain above bash script, but still you need a way to run .sh file from your application. Best approach will be using a thread  (You can use Executors if want). This thread will execute above bash script periodically at a configurable rate. I will not write that part of code. If you face any issue in executing code, write down a comment.

Let me give you a sample code ::

package corejava.dischandler;

import java.io.IOException;

/**
* Title: DiskFileManager.java
* Description :- This class provides API to check file size and file time threshold and
* if threshold is crossed then deletes file to come within threshold.
*/
public final class DiskFileManager
{

	private static DiskFileManager diskFileManager=new DiskFileManager();
	private DiskFileManager()
	{
	}

	/**
	* <pre>
	* <b>checkAndDeleteFiles</b>
	* <b>Description:This method is called to clean the files from the file system.</b>
	* </pre>
	* @throws InterruptedException
	* @throws IOException
	* @throws InstantiationException
	*/
	public   void checkAndDeleteFiles() throws InterruptedException, IOException, InstantiationException
	{
		try
		{
			StringBuffer outStr = new StringBuffer();
			StringBuffer errStr = new StringBuffer();
			String scriptFileName = "./discfilehandler.sh";
			String command = scriptFileName + "/logs 1 1";

			System.out.println("Command to be executed  :- " + command);

			Process output = Runtime.getRuntime().exec(command);
			StreamEater errorEater = new StreamEater(output.getErrorStream(),errStr);
			StreamEater outputEater = new StreamEater(output.getInputStream(),outStr);
			errorEater.start();
			outputEater.start();
			try
			{
				int ret = output.waitFor();
				errorEater.join();
				outputEater.join();

				System.out.println();

				//logger.info("execute(): Error Stream:" + errStr + " ; execute(): Output Stream:" + outStr + " ; execute(): ExitValue:" + ret);

			} catch (InterruptedException e)
			{
				throw e;
			}
		} catch (IOException e)
		{
			throw e;
		}
	}
	/**
	* <pre>
	* <b>getInstance</b>
	* <b>Description:This method is used to get the instance of Disk File Manager.</b>
	* </pre>
	* @return
	*/
	public static DiskFileManager getInstance()
	{
		return diskFileManager;
	}
}

Above code will locate the script file in class path and pass the required parameter. As in our case, “/logs 1 1” are 3 parameters, meaning

  • directory to monitor is /logs,
  • delete one day old log files and
  • do not store more than 1 GB of log files at any time.

Adding stream eater

So for we have added an bash script and an executor code for running the script. You have to write your own Thread to run periodically above executor.

Now we will look into StreamEater which actually collects command output and gives to your application code, so application can either log it or do whatever it wants.

We have already used StreamEater in above executor, now i am giving its code:;

package corejava.dischandler;

import java.io.BufferedReader;
import java.io.InputStream;
import java.io.InputStreamReader;

public class StreamEater extends Thread
{

	/**
	* Stream Eater thread to eat up every thing that is the output to the
	* execute command
	*/
	private InputStream  iStream;

	public InputStream getiStream() {
		return iStream;
	}

	public void setiStream(InputStream iStream) {
		this.iStream = iStream;
	}

	private StringBuffer stringBuffer;

	public StringBuffer getStringBuffer() {
		return stringBuffer;
	}

	public void setStringBuffer(StringBuffer stringBuffer) {
		this.stringBuffer = stringBuffer;
	}

	/**
	* This is the constructor.
	*
	* @param is - It is the input stream
	* @param type - type of input stream
	* @param redirect - string buffer to contain the output.
	* @throws InstantiationException
	*/

	public StreamEater(InputStream is, StringBuffer redirect) throws InstantiationException
	{
		this.iStream = is;
		if (redirect == null)
		{
			throw new InstantiationException("String Buffer Reference , second param can not be null");
		}
		this.stringBuffer = redirect;
	}

	/**
	* This is method called when the thread is started. It captures the stream
	* output and and store it into the buffer.
	*/
	public void run()
	{
		InputStreamReader isr = null;
		BufferedReader br = null;
		try
		{

			isr = new InputStreamReader(iStream);
			br = new BufferedReader(isr);
			String line;
			while ((line = br.readLine()) != null)
			{
				stringBuffer.append(line).append(System.getProperty("line.separator"));
			}

		}
		catch (Exception ex)
		{
			ex.printStackTrace();
		}
		finally
		{
		if (isr != null)
		{
			try
			{
				isr.close();
			}
			catch (Exception e)
			{
				e.printStackTrace();
			}
		}
		if (br != null)
		{
			try
			{
				br.close();
			}
			catch (Exception e)
			{
				e.printStackTrace();
			}
		}
		}
	}

}

As you see these are small threads waiting to collect the process output, that all.

Review all

Using above classes and given bash script, you can easily develop some application component which will ensure the size of log files do not exceed some predefined number. As you application is taking care of deleting the actual log files, this can be customized at any level.

Hope you got some useful information in this post. Please write down to me if need any help or have any query.

Happy Learning !!

Was this post helpful?

Let us know if you liked the post. That’s the only way we can improve.

Share this:

  • Twitter
  • Facebook
  • LinkedIn
  • Reddit

About Lokesh Gupta

A family guy with fun loving nature. Love computers, programming and solving everyday problems. Find me on Facebook and Twitter.

Feedback, Discussion and Comments

  1. arthur

    February 21, 2018

    Hello,

    this is intressting topic, but why you shouldn’t use logrotate?
    https://linux.die.net/man/8/logrotate

    or just
    man logrotate

    It is simply and can do more than deleting files.

  2. pepito

    July 3, 2014

    hi, what if i need to find directories bigger then 20gb with 30 days old and delete them ?

Comments are closed on this article!

Search Tutorials

Meta Links

  • About Me
  • Contact Us
  • Privacy policy
  • Advertise
  • Guest and Sponsored Posts

Recommended Reading

  • 10 Life Lessons
  • Secure Hash Algorithms
  • How Web Servers work?
  • How Java I/O Works Internally?
  • Best Way to Learn Java
  • Java Best Practices Guide
  • Microservices Tutorial
  • REST API Tutorial
  • How to Start New Blog

Copyright © 2020 · HowToDoInjava.com · All Rights Reserved. | Sitemap

  • Java 15 New Features
  • Sealed Classes and Interfaces
  • EdDSA (Ed25519 / Ed448)