Spring Batch_Multi-threaded Step_使用多线程的Step

spring官方文档:http://docs.spring.io/spring-batch/trunk/reference/html/scalability.html

The simplest way to start parallel processing is to add a TaskExecutor to your Step configuration, e.g. as an attribute of the tasklet:

<step id="loading">
    <tasklet task-executor="taskExecutor">...</tasklet>
</step>

In this example the taskExecutor is a reference to another bean definition, implementing the TaskExecutor interface. TaskExecutor is a standard Spring interface, so consult(查阅,参照) the Spring User Guide for details of available implementations. The simplest multi-threaded TaskExecutor is a SimpleAsyncTaskExecutor.

The result of the above configuration will be that the Step executes by reading, processing and writing each chunk of items (each commit interval) in a separate thread of execution. Note that this means there is no fixed order for the items to be processed, and a chunk might contain items that are non-consecutive(不连续的) compared to the single-threaded case. In addition to any limits placed by the task executor (e.g. if it is backed by a thread pool), there is a throttle limit in the tasklet configuration which defaults to 4. You may need to increase this to ensure that a thread pool is fully utilised, e.g.

<step id="loading"> <tasklet
    task-executor="taskExecutor"
    throttle-limit="20">...</tasklet>
</step>

Note also that there may be limits placed on concurrency by any pooled resources used in your step, such as a DataSource. Be sure to make the pool in those resources at least as large as the desired number of concurrent threads in the step.

There are some practical limitations of using multi-threaded Steps for some common Batch use cases. Many participants in a Step (e.g. readers and writers) are stateful(有状态的), and if the state is not segregated(分离,隔开的) by thread, then those components are not usable in a multi-threaded Step. In particular most of the off-the-shelf(现成的) readers and writers from Spring Batch are not designed for multi-threaded use. It is, however, possible to work with stateless or thread safe readers and writers, and there is a sample (parallelJob) in the Spring Batch Samples that show the use of a process indicator (see Section 6.12, “Preventing State Persistence”) to keep track of items that have been processed in a database input table.

Spring Batch provides some implementations of ItemWriter and ItemReader. Usually they say in the Javadocs if they are thread safe or not, or what you have to do to avoid problems in a concurrent environment. If there is no information in Javadocs, you can check the implementation to see if there is any state. If a reader is not thread safe, it may still be efficient(有效率的) to use it in your own synchronizing delegator. You can synchronize the call to read() and as long as the processing and writing is the most expensive part of the chunk your step may still complete much faster than in a single threaded configuration.

=================END=================

时间: 2024-08-01 22:47:30

Spring Batch_Multi-threaded Step_使用多线程的Step的相关文章

Spring Batch_使用JdbcPagingItemReader_多线程的Step

我们最经常使用的就是 JdbcCursorItemReader,使用游标的方式 逐条数据的读取.但是 从spring 官方文档我们知道 ,他不是线程安全的.在这里,我们使用 JdbcPagingItemReader 从数据库读取数据,并且是分页的读,而且这个类是线程安全的,那么我们就可以使用多线程的Step,从而提高 JOB 的执行效率. 下面是主要的配置文件: <beans xmlns="http://www.springframework.org/schema/beans"

spring定时任务.线程池,自定义多线程配置

定时任务及多线程配置xml <?xml version="1.0" encoding="UTF-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springfra

Spring Boot 定时任务单线程和多线程

Spring Boot 的定时任务: 第一种:把参数配置到.properties文件中: 代码: package com.accord.task; import java.text.SimpleDateFormat; import java.util.Date; import org.springframework.scheduling.annotation.Scheduled; import org.springframework.stereotype.Component; /** * 从配置

使用spring的多线程机制

多线程并发处理起来通常比較麻烦,假设你使用spring容器来管理业务bean,事情就好办了多了.spring封装了java的多线程的实现,你仅仅须要关注于并发事物的流程以及一些并发负载量等特性. 详细来说怎样使用spring来处理并发事务: 首先编写详细的事务逻辑,实现Runnable接口.比方说 package com.andy.threadDemo; public class ThreadTransCode implements Runnable{ @Override public void

Spring Batch Hello World Example

Spring Batch is a framework for batch processing – execution of a series of jobs. In Spring Batch, A job consists of many steps and each step consists of a READ-PROCESS-WRITE task or single operation task (tasklet). For “READ-PROCESS-WRITE” process,

Spring+Ibatis数据库水平分库

1.引言    笔者最近在做一个互联网的"类SNS"应用,应用中用户数量巨大(约4000万)左右,因此,简单的使用传统单一数据库存储肯定是不行的. 参考了业内广泛使用的分库分表,以及使用DAL数据访问层等的做法,笔者决定使用一种最简单的数据源路由选择方式来解决问题. 严格的说,目前的实现不能算是一个解决方案,只能是一种思路的简易实现,笔者也仅花了2天时间来完成(其中1.5天是在看资料和Spring/ibatis的源码).这里也只是为各位看官提供一个思路参考,顺便给自己留个笔记 2.系统

[Spring Batch 系列] 第一节 初识 Spring Batch

距离开始使用 Spring Batch 有一段时间了,一直没有时间整理,现在项目即将完结,整理下这段时间学习和使用经历. 官网地址:http://projects.spring.io/spring-batch/ 一.定义与特点 A lightweight, comprehensive batch framework designed to enable the development of robust batch applications vital for the daily operati

Spring Session 介绍及使用

spring Session的简易使用步骤 生成 step 1:后台业务模块使用Spring-Session生成一个session step 2:后台业务模块往session里设置信息 step 3:将session存到redis缓存中(支持持久化) step 4:将session id 返回给浏览器 step 5:浏览器根据cookie方式保存session id 使用 step 6:浏览器取出session id通过HTTP报文带给后台 step 7:后台根据session id从redis

Enable HTTPS in Spring Boot

Spring-boot-enable-ssl Enable HTTPS in Spring Boot APRIL 14, 2015DRISS AMRI This weekend I answered a question about enabling HTTPS in JHipster onstackoverflow that caught a lot of interest on Twitter so I decided to put a short post on it with some