Monthly Archives: March 2015

The second way to run Spring Batch job is by Spring Batch Admin. It is more powerful. By SBA, we can execute/stop/monitor our batch job in web UI. I downloaded the SBA sample code from Spring Batch website and made it more simple version.

I used the mysql to store the metadata. The metadata tables can be created by several ways. We can create it in normal spring batch job by adding below code in the config xml.

<jdbc:initialize-database data-source="dataSource">
      <jdbc:script location="org/springframework/batch/core/schema-drop-mysql.sql" />
      <jdbc:script location="org/springframework/batch/core/schema-mysql.sql" />
</jdbc:initialize-database>

Or we can manually copy the sql and run it from spring-batch-core.jar: /org/springframework/batch/core/schema-mysql.sql

Once we have meta tables in database, let me explain the following configuration.

batch-mysql.properties

# Placeholders batch.*
#    for MySQL:
batch.jdbc.driver=com.mysql.jdbc.Driver  //In the project, remember we should add this dependency in maven
batch.jdbc.url=jdbc:mysql://localhost:3306/spring_batch
batch.jdbc.user=username
batch.jdbc.password=password
batch.jdbc.testWhileIdle=true
batch.jdbc.validationQuery=SELECT 1    //test the database connection
batch.schema.script=classpath*:/org/springframework/batch/core/schema-mysql.sql  //to build the meta tables
batch.drop.script=classpath*:/org/springframework/batch/core/schema-drop-mysql.sql  //to swipte the meta tables
batch.business.schema.script=classpath:/business-schema-mysql.sql
batch.database.incrementer.class=org.springframework.jdbc.support.incrementer.MySQLMaxValueIncrementer

# Non-platform dependent settings that you might like to change
batch.data.source.init=false  //Always set false, or it will swipe and rebuild the meta tables

infinite-context.xml
The configuration file is different from normal one. In the config xml in SBA, we should delete the jobRepository, transacionManager and jobLauncher. We only define the job itself. Pay attention to the namespace version, spring-batch-admin and spring version. Sometimes, it may not correct! According to my experience, spring-batch-admin 1.2.1, spring3.0.5 are together good ones.

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:aop="http://www.springframework.org/schema/aop"
      xmlns:tx="http://www.springframework.org/schema/tx" xmlns:file="http://www.springframework.org/schema/integration/file"
      xmlns:integration="http://www.springframework.org/schema/integration" xmlns:p="http://www.springframework.org/schema/p"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:schemaLocation="http://www.springframework.org/schema/aop http://www.springframework.org/schema/aop/spring-aop.xsd
      http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch-2.1.xsd
      http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
      http://www.springframework.org/schema/integration http://www.springframework.org/schema/integration/spring-integration-1.0.xsd
      http://www.springframework.org/schema/integration/file http://www.springframework.org/schema/integration/file/spring-integration-file-1.0.xsd">

<job id="infinite" xmlns="http://www.springframework.org/schema/batch">
   <step id="step1" next="step1">
      <tasklet start-limit="100">
         <chunk commit-interval="1" reader="itemReader" writer="itemWriter" />
      </tasklet>
   </step>
</job>

<bean id="itemWriter" class="com.wfs.springbatch.springbatchadmin.helloworld.ExampleItemWriter"/>

<bean id="itemReader" class="com.wfs.springbatch.springbatchadmin.helloworld.ExampleItemReader" scope="step"/>

</beans>

In my case, I use mysql. So before run the batch job, we should set -DENVIRONMENT=mysql parameter for the JBoss or Tomcat server. If you use other databases, you should replace the batch-mysql.properties with batch-[other-databse-name].properties

My ppt summary for Spring Batch Admin.

source code: link

There are two ways to run your batch job.

First is to use console command. In order to do this, we need to compile the batch job, and saves all dependency jar in a lib file. We just write the following in the maven pom.xml(I refered this good example from mkyong.)

<plugin>
   <artifactId>maven-dependency-plugin</artifactId>
   <executions>
      <execution>
         <phase>package</phase>
         <goals>
            <goal>copy-dependencies</goal>
         </goals>
         <configuration>
            <outputDirectory>${project.build.directory}/lib/</outputDirectory>
         </configuration>
      </execution>
   </executions>
</plugin>

After that, we go to the target folder, and run following command:
java -cp “dependency-jars/*;SpringBatchExample.jar” org.springframework.batch.core.launch.support.CommandLineJobRunner spring\batch\jobs\job-hello-world.xml pliJob

Another way to run batch job is like we run normal main() in App.java. We compile it. And we use java -cp to call the App.class in the jar file:
java -cp SpringBatchExample.jar com.pli.project.sba.App

In last 2 weeks, I spent a lot of effort in researching in Spring Batch. This framework requires a lot of xml configuration. Below, I attached y passed code and explanation.
My Spring Batch project refers from mkyong.

database.xml
If we want to autimatically create the metadata in database, we should add <jdbc:initialize-database>

<beans xmlns="http://www.springframework.org/schema/beans"
   xmlns:jdbc="http://www.springframework.org/schema/jdbc" 
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://www.springframework.org/schema/beans 
      http://www.springframework.org/schema/beans/spring-beans-3.2.xsd
      http://www.springframework.org/schema/jdbc 
      http://www.springframework.org/schema/jdbc/spring-jdbc-3.2.xsd">

    <!-- connect to database -->
   <bean id="dataSource"
      class="org.springframework.jdbc.datasource.DriverManagerDataSource">
      <property name="driverClassName" value="com.mysql.jdbc.Driver" />
      <property name="url" value="jdbc:mysql://localhost:3306/test" />
      <property name="username" value="root" />
      <property name="password" value="" />
   </bean>

   <!-- create job-meta tables automatically -->
   <jdbc:initialize-database data-source="dataSource">
      <jdbc:script location="org/springframework/batch/core/schema-drop-mysql.sql" />
      <jdbc:script location="org/springframework/batch/core/schema-mysql.sql" />
   </jdbc:initialize-database>
   
</beans>

context.xml
In order to launch job from App.java. It requires jobRepository, transacionManager and jobLauncher. If job doesn’t has job-repository property, job will use the default jobRepository(the name is jobRepository).
JobRepository is responsible for save the execution inforation. One choice is to save it in memory. Another is to save it in database. Based on different option, the configuraion are different. Below shows both ways.

<beans xmlns="http://www.springframework.org/schema/beans"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="
      http://www.springframework.org/schema/beans 
      http://www.springframework.org/schema/beans/spring-beans-3.2.xsd">

   <!-- stored job-meta in memory -->
   <bean id="jobRepository"
      class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean">
      <property name="transactionManager" ref="transactionManager" />
   </bean>

    <!-- stored job-meta in database -->
   <bean id="jobRepository"
      class="org.springframework.batch.core.repository.support.JobRepositoryFactoryBean">
      <property name="dataSource" ref="dataSource" />
      <property name="transactionManager" ref="transactionManager" />
      <property name="databaseType" value="mysql" />
   </bean>
   
   <bean id="transactionManager"
      class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />

   <bean id="jobLauncher"
      class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
      <property name="jobRepository" ref="jobRepository" />
   </bean>
</beans>

job-hello-world.xml

<beans xmlns="http://www.springframework.org/schema/beans"
   xmlns:batch="http://www.springframework.org/schema/batch" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://www.springframework.org/schema/batch
      http://www.springframework.org/schema/batch/spring-batch-2.2.xsd
      http://www.springframework.org/schema/beans 
      http://www.springframework.org/schema/beans/spring-beans-3.2.xsd
   ">

   <import resource="../config/context.xml" />
   <import resource="../config/database.xml" />

   <bean id="report" class="batch.entity.Report" scope="prototype" />
   <bean id="itemProcessor" class="batch.model.CustomItemProcessor" />

   <batch:job id="helloWorldJob">
      <batch:step id="step1">
         <batch:tasklet>
            <batch:chunk reader="cvsFileItemReader" writer="myWriter" processor="itemProcessor"
               commit-interval="10">
            </batch:chunk>
         </batch:tasklet>
      </batch:step>
   </batch:job>

   <bean id="cvsFileItemReader" class="org.springframework.batch.item.file.FlatFileItemReader">
      <property name="resource" value="classpath:cvs/input/report.csv" />
      <property name="lineMapper">
         <bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
            <property name="lineTokenizer">
               <bean
                  class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
                  <property name="names" value="id,sales,qty,staffName,date" />
               </bean>
            </property>
            <property name="fieldSetMapper">
                <bean class="batch.model.ReportFieldSetMapper" />
            </property>
         </bean>
      </property>
   </bean>

   <bean id="myWriter" class="batch.model.MyWriter"/>
</beans>

MyWriter.java

package batch.model;

import batch.entity.Report;
import org.springframework.batch.item.ItemWriter;
import java.io.PrintWriter;
import java.util.List;

public class MyWriter implements ItemWriter {
    @Override
    public void write(List items) throws Exception {
        PrintWriter writer = new PrintWriter("c:\\Users\\lipeng\\_Main\\output.csv", "UTF-8");
        writer.println("hello");
        writer.close();
    }
}

CustomItemProcessor.java

package batch.model;
import batch.entity.Report;
import org.springframework.batch.item.ItemProcessor;

public class CustomItemProcessor implements ItemProcessor {
   @Override
   public Report process(Report item) throws Exception {
      System.out.println("Processing..." + item);
      return item;
   }
}

ReportFieldSetMapper.java
After reading from reader, Spring Batch uses Mapper to transform the read data to a expected bean.

package batch.model;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import org.springframework.batch.item.file.mapping.FieldSetMapper;
import org.springframework.batch.item.file.transform.FieldSet;
import org.springframework.validation.BindException;
import batch.entity.Report;

public class ReportFieldSetMapper implements FieldSetMapper {
   private SimpleDateFormat dateFormat = new SimpleDateFormat("dd/MM/yyyy");   
   @Override
   public Report mapFieldSet(FieldSet fieldSet) throws BindException {      
      Report report = new Report();
      report.setId(fieldSet.readInt(0));
      report.setSales(fieldSet.readBigDecimal(1));
      report.setQty(fieldSet.readInt(2));
      report.setStaffName(fieldSet.readString(3));      
      String date = fieldSet.readString(4);
      try {
         report.setDate(dateFormat.parse(date));
      } catch (ParseException e) {
         e.printStackTrace();
      }      
      return report;      
   }
}

App.java
When run a job and this running is recorded in memory or metadata, it is not allowed to run the job for the 2nd time, we should use JobParametersBuilder to add some parameter to discriminate each running. For example, we can use the timestamp as the parameter.

package batch;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.JobExecution;
import org.springframework.batch.core.JobParameters;
import org.springframework.batch.core.launch.JobLauncher;
import org.springframework.context.ApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;

public class App {
   public static void main(String[] args) {

      String[] springConfig  = 
         {
               "spring/batch/jobs/job-hello-world.xml"
         };
      
      ApplicationContext context = 
            new ClassPathXmlApplicationContext(springConfig);
      
      JobLauncher jobLauncher = (JobLauncher) context.getBean("jobLauncher");
      Job job = (Job) context.getBean("helloWorldJob");

      try {
         JobExecution execution = jobLauncher.run(job, new JobParameters());
         System.out.println("Exit Status : " + execution.getStatus());
      } catch (Exception e) {
         e.printStackTrace();
      }
   }
}

source code: link