Spring Batch FlatFileItemWriter: CSV File Writer Example

Learn to write CSV data using FlatFileItemWriter in a Spring batch application. The FlatFileItemWriter is an ItemWriter implementation that writes data to a flat file or stream. The location of the output file is defined by a Resource and must represent a writable file.

1. Maven

Include the Sprint batch starter module in the Spring boot application. Spring batch requires a database to store the Job execution information so we will use H2 database for the demo.

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-batch</artifactId>
</dependency>
<dependency>
  <groupId>com.h2database</groupId>
  <artifactId>h2</artifactId>
  <scope>runtime</scope>
</dependency>

2. Spring Batch FlatFileItemWriter

The FlatFileItemWriter class is specifically designed for writing data to a flat file in a batch-processing scenario. Flat files are files where each line represents a record, and the fields within the record are delimited by a specific character, such as a comma or tab (for example CSV files). It offers configuration options for defining the output file, specifying the format, and supporting optional features such as headers and footers.

The following is a simple FlatFileItemWriter configuration that writes the Person data to a file ‘outputData.csv‘.

The BeanWrapperFieldExtractor classes take the field names of the model object (e.g. Person in our case) and extract the values from an instance to finally pass to DelimitedLineAggregator.
The DelimitedLineAggregator arranges the field data in a flat-file format in which all fields are separated by a delimiter.
Finally, the delimited data is written to a WriteableResource.

Resource resource = new FileSystemResource("c:/temp/outputData.csv");

@Bean
public FlatFileItemWriter<Person> writer() {

  //Create writer instance
  FlatFileItemWriter<Person> writer = new FlatFileItemWriter<>();

  //Set output file location
  writer.setResource(resource);

  //All job repetitions should "append" to same output file
  writer.setAppendAllowed(true);

  //Name field values sequence based on object properties
  writer.setLineAggregator(new DelimitedLineAggregator<>() {
    {
      setDelimiter(",");
      setFieldExtractor(new BeanWrapperFieldExtractor<>() {
        {
          setNames(new String[]{"firstName", "lastName", "age", "active"});
        }
      });
    }
  });
  return writer;
}

3. Output File Creation

When a new batch Job starts, it does the following:

If the file does not exist then create a new file, and write data to it.
If the file exists then write the data to it. It will overwrite any previous data written in the file. If you want to append the data in this file, the use writer.setAppendAllowed(true) method.

Also, we are using the same output file everytime and we do not want to duplicate the records, then we can use the write.shouldDeleteIfExists(true) method to delete the existing file (if there is one) every time a new Job is started. In case the Job is restarted due to a processing error, it won’t delete the file.

Drop me your questions in the comments section.

Happy Learning !!

Source Code on Github

i have this code

package com.rbs.laas_service.batch;


import com.rbs.laas_service.dto.Flatfile;
import lombok.extern.slf4j.Slf4j;
import org.springframework.batch.item.ExecutionContext;
import org.springframework.batch.item.ItemWriter;
import org.springframework.batch.item.file.FlatFileItemWriter;
import org.springframework.batch.item.file.transform.BeanWrapperFieldExtractor;
import org.springframework.batch.item.file.transform.DelimitedLineAggregator;
import org.springframework.context.annotation.Bean;
import org.springframework.core.io.ClassPathResource;
import org.springframework.stereotype.Component;

import java.util.List;

@Slf4j
@Component
public class FlatFileWriter implements ItemWriter<Flatfile> {

@Bean
    public FlatFileItemWriter<Flatfile> faltwriter(){

        System.out.println("writer flatfile");
        FlatFileItemWriter <Flatfile> writer=new FlatFileItemWriter<>();
        writer.setResource(new ClassPathResource("OutputLAAS.txt"));

        writer.setLineAggregator(new DelimitedLineAggregator<Flatfile>(){{
            setFieldExtractor(new BeanWrapperFieldExtractor<Flatfile>(){{
                setNames(new String[]{"firstname"});
            }});
        }});
        return writer;
    }
    @Override
    public void write(List<? extends Flatfile> list) throws Exception {
//        faltwriter().open(new ExecutionContext());
////faltwriter().write(list);
        for(Flatfile file:list){
            file.getFirstname();
            System.out.println(file.getFirstname());
        }
        faltwriter().write(list);

    }


}

but it gives me writer not open error

Lokesh Gupta

March 6, 2024 at 3:08 pm

Try adding writer.open(new ExecutionContext()); before returning writer from faltwriter() bean.
Reply

Thank you for this tutorial, it really helped a lot

I am just having issues testing the FlatFileWriter, my file will have (header,body and footer)
I can write to the file successfully the only issue comes when i need to get total record of items written to the file using stepExecution.getWriteCount(), stepExecution is always null even though i set it using the beforeStep(StepExecution stepExecution)


Error i am getting :java.lang.NullPointerException: Cannot invoke "org.springframework.batch.core.StepExecution.getWriteCount()" because "this.stepExecution" is null
and i am setting the stepExecution 
        StepExecution stepExecution = MetaDataInstanceFactory.createStepExecution("dataStep", SystemTime.SYSTEM.milliseconds());
        batchConfiguration.trailerCallback().beforeStep(stepExecution);

Below is my code


SpringBatchConfiguration.java
@Slf4j
@Configuration
@EnableBatchProcessing
@RequiredArgsConstructor
public class BatchConfiguration {

    private final StepBuilderFactory stepBuilderFactory;

    @Bean
    @StepScope
    public HeaderCallback headerCallback() {
        return new HeaderCallback();
    }

    @Bean
    @StepScope
    public TrailerCallback trailerCallback() {
        return new TrailerCallback();
    }


    @Bean
    public Step step1(FlatFileItemWriter fileItemWriter) {
        return stepBuilderFactory.get("step1")
                .chunk(100)
                .writer(fileItemWriter)
                .build();
    }

    @Bean
    @StepScope
    public FlatFileItemWriter fileItemWriter(@Value("#{jobParameters['filePath']") String filePath) {
        var path = new FileSystemResource(filePath);
        log.info("Writing file to {}", kv("filePath",path));
        return new FlatFileItemWriterBuilder()
                .name("eiwDataTransactionRecordFlatFileItemWriter")
                .headerCallback(headerCallback())
                .lineAggregator(transactionRecordLineAggregator())
                .append(true)
                .resource(path)
                .footerCallback(trailerCallback())
                .shouldDeleteIfExists(false)
                .build();

    }

    private LineAggregator transactionRecordLineAggregator() {
        DelimitedLineAggregator lineAggregator = new DelimitedLineAggregator();
        lineAggregator.setDelimiter("");
        FieldExtractor fieldExtractor = transactionRecordFieldExtractor();
        lineAggregator.setFieldExtractor(fieldExtractor);

        return lineAggregator;
    }

    private FieldExtractor transactionRecordFieldExtractor() {
        BeanWrapperFieldExtractor fieldExtractor = new BeanWrapperFieldExtractor();
        fieldExtractor.setNames(new String[]{
                "recordId",
                "transactionDate",
                "transactionTime"
        });
        fieldExtractor.afterPropertiesSet();

        return fieldExtractor;
    }
}

Footer
public class TrailerCallback implements FlatFileFooterCallback, StepExecutionListener {
    private StepExecution stepExecution;

    @Override
    public void writeFooter(@NonNull Writer writer) throws IOException {
        String trailerRecord = TrailerRecord
                .builder()
                .trailerProcessingDate(LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyy-MM-dd")))
                .totalRecordCount(stepExecution.getWriteCount())
                .build()
                .toString();
        writer.write(trailerRecord);
    }

    @Override
    public void beforeStep(@NonNull StepExecution stepExecution) {
        this.stepExecution = stepExecution;
    }

    @Override
    public ExitStatus afterStep(@NonNull StepExecution stepExecution) {
        return ExitStatus.COMPLETED;
    }
}



Header
public class HeaderCallback implements FlatFileHeaderCallback {
    @Override
    public void writeHeader(final Writer writer) throws IOException {
        String headerRecord = HeaderRecord
                .builder()
                .headerProcessingDate(LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyy-MM-dd")))
                .build()
                .toString();
        writer.write(headerRecord);
    }
}

    @Test
    void testFlatFileItemWriter() throws Exception {
        FileSystemResource expectedFile = new FileSystemResource("sampleFile.txt");
        FileSystemResource actualFile = new FileSystemResource("testFile."+LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyyMMddHHmmss"))+".txt");
        var flatFileItemWriter = batchConfiguration.transactionRecordFlatFileItemWriter(actualFile.getPath());
        StepExecution stepExecution = MetaDataInstanceFactory.createStepExecution("dataStep", SystemTime.SYSTEM.milliseconds());
        batchConfiguration.trailerCallback().beforeStep(stepExecution);

        //when
        StepScopeTestUtils.doInStepScope(stepExecution, () -> {
            flatFileItemWriter.open(stepExecution.getExecutionContext());
            flatFileItemWriter.write(List.of(buildEiwDataTransactionRecord()));
            flatFileItemWriter.close();
            return null;
        });

        AssertFile.assertFileEquals(expectedFile, actualFile);

    }

Amit Banger

December 24, 2019 at 12:05 pm

Is there a way to set own values to header?
for Example: my header looks like {s.no;name;dept;rspcode}
In this first three column I am extracting and setting from csv file but for column ‘rspcode’ I want to set(00–>success, 11–>failure–when rows inserted successfully to DB).
Reply
Prateek

December 9, 2019 at 9:15 pm

I want to contribute for the Spring Batch tutorials. Please let me know if I can contribute in writing tutorials
Reply
- Lokesh Gupta
  
  December 10, 2019 at 12:08 pm
  
  Please drop an email to howtodoinjava@gmail.com
  Reply
Ankit SHarma

September 18, 2019 at 4:42 pm

Hi,

Could you please let me know how to add headers, name of the Columns in the output CSV file
Reply
gyan

November 14, 2018 at 2:07 pm

I want to read data from DB and write to txt or csv/ please write that program also
Reply
Devashree Kabra

October 10, 2018 at 4:54 pm

How to sort this list before writing it to file?
Reply
- Kirnan
  
  December 5, 2019 at 5:23 pm
  
  Collections.sort
  Reply

Spring Batch FlatFileItemWriter: CSV File Writer Example

1. Maven

2. Spring Batch FlatFileItemWriter

3. Output File Creation

Leave a Comment

Leave a Comment Cancel reply

Spring Batch MultiResourceItemReader Example

Spring Batch Delete Files After Processing

About Us

Tutorial Series

Meta Links

Our Blogs

Follow On: