In Spring batch jobs, the best approach to delete the flat files after read or processing is to create separate Tasklet
and execute it at the end of job, when processing is complete.
Tasklet for deleting files
This is an example of such Tasklet
which will delete all CSV files from c:/temp/input/
location at the end of job.
import java.io.File; import org.springframework.batch.core.StepContribution; import org.springframework.batch.core.UnexpectedJobExecutionException; import org.springframework.batch.core.scope.context.ChunkContext; import org.springframework.batch.core.step.tasklet.Tasklet; import org.springframework.batch.repeat.RepeatStatus; import org.springframework.beans.factory.InitializingBean; import org.springframework.core.io.Resource; import org.springframework.util.Assert; public class FileDeletingTasklet implements Tasklet, InitializingBean { private Resource[] resources; public RepeatStatus execute(StepContribution contribution, ChunkContext chunkContext) throws Exception { for(Resource r: resources) { File file = r.getFile(); boolean deleted = file.delete(); if (!deleted) { throw new UnexpectedJobExecutionException("Could not delete file " + file.getPath()); } } return RepeatStatus.FINISHED; } public void setResources(Resource[] resources) { this.resources = resources; } public void afterPropertiesSet() throws Exception { Assert.notNull(resources, "directory must be set"); } }
Archive file
Feel free to modify the logic inside FileDeletingTasklet
to archive the files to different location or implement your own archiving logic.
It’s simple IO operation.
How to use FileDeletingTasklet
Create another Step
to be executed after main step and execute the FileDeletingTasklet
.
import org.springframework.batch.core.Job; import org.springframework.batch.core.Step; import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing; import org.springframework.batch.core.configuration.annotation.JobBuilderFactory; import org.springframework.batch.core.configuration.annotation.StepBuilderFactory; import org.springframework.batch.core.launch.support.RunIdIncrementer; import org.springframework.batch.item.file.FlatFileItemReader; import org.springframework.batch.item.file.MultiResourceItemReader; import org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper; import org.springframework.batch.item.file.mapping.DefaultLineMapper; import org.springframework.batch.item.file.transform.DelimitedLineTokenizer; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.core.io.Resource; import com.howtodoinjava.demo.model.Employee; @Configuration @EnableBatchProcessing public class BatchConfig { @Autowired private JobBuilderFactory jobBuilderFactory; @Autowired private StepBuilderFactory stepBuilderFactory; @Value("file:c:/temp/input/inputData*.csv") private Resource[] inputResources; @Bean public Job readCSVFilesJob() { return jobBuilderFactory .get("readCSVFilesJob") .incrementer(new RunIdIncrementer()) .start(step1()) .next(step2()) .build(); } @Bean public Step step1() { return stepBuilderFactory.get("step1").<Employee, Employee>chunk(5) .reader(multiResourceItemReader()) .writer(writer()) .build(); } @Bean public Step step2() { FileDeletingTasklet task = new FileDeletingTasklet(); task.setResources(inputResources); return stepBuilderFactory.get("step2") .tasklet(task) .build(); } @Bean public MultiResourceItemReader<Employee> multiResourceItemReader() { MultiResourceItemReader<Employee> resourceItemReader = new MultiResourceItemReader<Employee>(); resourceItemReader.setResources(inputResources); resourceItemReader.setDelegate(reader()); return resourceItemReader; } @SuppressWarnings({ "rawtypes", "unchecked" }) @Bean public FlatFileItemReader<Employee> reader() { // Create reader instance FlatFileItemReader<Employee> reader = new FlatFileItemReader<Employee>(); // Set number of lines to skips. Use it if file has header rows. reader.setLinesToSkip(1); // Configure how each line will be parsed and mapped to different values reader.setLineMapper(new DefaultLineMapper() { { // 3 columns in each row setLineTokenizer(new DelimitedLineTokenizer() { { setNames(new String[] { "id", "firstName", "lastName" }); } }); // Set values in Employee class setFieldSetMapper(new BeanWrapperFieldSetMapper<Employee>() { { setTargetType(Employee.class); } }); } }); return reader; } @Bean public ConsoleItemWriter<Employee> writer() { return new ConsoleItemWriter<Employee>(); } }
Now run the application and watch the logs.
2018-07-11 12:30:00 INFO - Job: [SimpleJob: [name=readCSVFilesJob]] launched with the following parameters: [{JobID=1531292400004}] 2018-07-11 12:30:00 INFO - Executing step: [step1] Employee [id=1, firstName=Lokesh, lastName=Gupta] Employee [id=2, firstName=Amit, lastName=Mishra] Employee [id=3, firstName=Pankaj, lastName=Kumar] Employee [id=4, firstName=David, lastName=Miller] Employee [id=5, firstName=Ramesh, lastName=Gupta] Employee [id=6, firstName=Vineet, lastName=Mishra] Employee [id=7, firstName=Amit, lastName=Kumar] Employee [id=8, firstName=Dav, lastName=Miller] Employee [id=9, firstName=Vikas, lastName=Kumar] Employee [id=10, firstName=Pratek, lastName=Mishra] Employee [id=11, firstName=Brian, lastName=Kumar] Employee [id=12, firstName=David, lastName=Cena] 2018-07-11 12:30:00 INFO - Executing step: [step2] Deleted file :: c:\temp\input\inputData1.csv Deleted file :: c:\temp\input\inputData2.csv Deleted file :: c:\temp\input\inputData3.csv 2018-07-11 12:30:00 INFO - Job: [SimpleJob: [name=readCSVFilesJob]] completed with the following parameters: [{JobID=1531292400004}] and the following status: [COMPLETED]
Drop me your questions in comments section.
Happy Learning !!
G. D. Srinath Gamage
In a real situation files in folder path can changes over time and, what is to gurantee that you are deleting only the processed files ?
vikas ramireddy
Thank you so much for providing tutorials.
Hope you can help me out
https://stackoverflow.com/questions/58879713/reading-multiple-files-in-a-folder-and-parsing-it-and-writing-to-another-folder
Ravi
How do you delete the single file at a time ?