求超大文件上传方案( HTML5 )

1,项目调研

因为需要研究下断点上传的问题。找了很久终于找到一个比较好的项目。

在GoogleCode上面,代码弄下来超级不方便,还是配置hosts才好,把代码重新上传到了github上面。

https://github.com/freewebsys/java-large-file-uploader-demo

效果:

上传中,显示进度,时间,百分比。

点击【Pause】暂停,点击【Resume】继续。

2,代码分析

原始项目:

https://code.google.com/p/java-large-file-uploader/

这个项目最后更新的时间是 2012 年,项目进行了封装使用最简单的方法实现了http的断点上传。

因为html5 里面有读取文件分割文件的类库,所以才可以支持断点上传,所以这个只能在html5 支持的浏览器上面展示。

同时,在js 和 java 同时使用 cr32 进行文件块的校验,保证数据上传正确。

代码在使用了最新的servlet 3.0 的api,使用了异步执行,监听等方法。

上传类UploadServlet

@Component("javaLargeFileUploaderServlet")

@WebServlet(name = "javaLargeFileUploaderServlet", urlPatterns = { "/javaLargeFileUploaderServlet" })

public class UploadServlet extends HttpRequestHandlerServlet

implements HttpRequestHandler {

private static final Logger log = LoggerFactory.getLogger(UploadServlet.class);

@Autowired

UploadProcessor uploadProcessor;

@Autowired

FileUploaderHelper fileUploaderHelper;

@Autowired

ExceptionCodeMappingHelper exceptionCodeMappingHelper;

@Autowired

Authorizer authorizer;

@Autowired

StaticStateIdentifierManager staticStateIdentifierManager;

@Override

public void handleRequest(HttpServletRequest request, HttpServletResponse response)

throws IOException {

log.trace("Handling request");

Serializable jsonObject = null;

try {

// extract the action from the request

UploadServletAction actionByParameterName =

UploadServletAction.valueOf(fileUploaderHelper.getParameterValue(request, UploadServletParameter.action));

// check authorization

checkAuthorization(request, actionByParameterName);

// then process the asked action

jsonObject = processAction(actionByParameterName, request);

// if something has to be written to the response

if (jsonObject != null) {

fileUploaderHelper.writeToResponse(jsonObject, response);

}

}

// If exception, write it

catch (Exception e) {

exceptionCodeMappingHelper.processException(e, response);

}

}

private void checkAuthorization(HttpServletRequest request, UploadServletAction actionByParameterName)

throws MissingParameterException, AuthorizationException {

// check authorization

// if its not get progress (because we do not really care about authorization for get

// progress and it uses an array of file ids)

if (!actionByParameterName.equals(UploadServletAction.getProgress)) {

// extract uuid

final String fileIdFieldValue = fileUploaderHelper.getParameterValue(request, UploadServletParameter.fileId, false);

// if this is init, the identifier is the one in parameter

UUID clientOrJobId;

String parameter = fileUploaderHelper.getParameterValue(request, UploadServletParameter.clientId, false);

if (actionByParameterName.equals(UploadServletAction.getConfig) && parameter != null) {

clientOrJobId = UUID.fromString(parameter);

}

// if not, get it from manager

else {

clientOrJobId = staticStateIdentifierManager.getIdentifier();

}

// call authorizer

authorizer.getAuthorization(

request,

actionByParameterName,

clientOrJobId,

fileIdFieldValue != null ? getFileIdsFromString(fileIdFieldValue).toArray(new UUID[] {}) : null);

}

}

private Serializable processAction(UploadServletAction actionByParameterName, HttpServletRequest request)

throws Exception {

log.debug("Processing action " + actionByParameterName.name());

Serializable returnObject = null;

switch (actionByParameterName) {

case getConfig:

String parameterValue = fileUploaderHelper.getParameterValue(request, UploadServletParameter.clientId, false);

returnObject =

uploadProcessor.getConfig(

parameterValue != null ? UUID.fromString(parameterValue) : null);

break;

case verifyCrcOfUncheckedPart:

returnObject = verifyCrcOfUncheckedPart(request);

break;

case prepareUpload:

returnObject = prepareUpload(request);

break;

case clearFile:

uploadProcessor.clearFile(UUID.fromString(fileUploaderHelper.getParameterValue(request, UploadServletParameter.fileId)));

break;

case clearAll:

uploadProcessor.clearAll();

break;

case pauseFile:

List<UUID> uuids = getFileIdsFromString(fileUploaderHelper.getParameterValue(request, UploadServletParameter.fileId));

uploadProcessor.pauseFile(uuids);

break;

case resumeFile:

returnObject =

uploadProcessor.resumeFile(UUID.fromString(fileUploaderHelper.getParameterValue(request, UploadServletParameter.fileId)));

break;

case setRate:

uploadProcessor.setUploadRate(UUID.fromString(fileUploaderHelper.getParameterValue(request, UploadServletParameter.fileId)),

Long.valueOf(fileUploaderHelper.getParameterValue(request, UploadServletParameter.rate)));

break;

case getProgress:

returnObject = getProgress(request);

break;

}

return returnObject;

}

List<UUID> getFileIdsFromString(String fileIds) {

String[] splittedFileIds = fileIds.split(",");

List<UUID> uuids = Lists.newArrayList();

for (int i = 0; i < splittedFileIds.length; i++) {

uuids.add(UUID.fromString(splittedFileIds[i]));

}

return uuids;

}

private Serializable getProgress(HttpServletRequest request)

throws MissingParameterException {

Serializable returnObject;

String[] ids =

new Gson()

.fromJson(fileUploaderHelper.getParameterValue(request, UploadServletParameter.fileId), String[].class);

Collection<UUID> uuids = Collections2.transform(Arrays.asList(ids), new Function<String, UUID>() {

@Override

public UUID apply(String input) {

return UUID.fromString(input);

}

});

returnObject = Maps.newHashMap();

for (UUID fileId : uuids) {

try {

ProgressJson progress = uploadProcessor.getProgress(fileId);

((HashMap<String, ProgressJson>) returnObject).put(fileId.toString(), progress);

}

catch (FileNotFoundException e) {

log.debug("No progress will be retrieved for " + fileId + " because " + e.getMessage());

}

}

return returnObject;

}

private Serializable prepareUpload(HttpServletRequest request)

throws MissingParameterException, IOException {

// extract file information

PrepareUploadJson[] fromJson =

new Gson()

.fromJson(fileUploaderHelper.getParameterValue(request, UploadServletParameter.newFiles), PrepareUploadJson[].class);

// prepare them

final HashMap<String, UUID> prepareUpload = uploadProcessor.prepareUpload(fromJson);

// return them

return Maps.newHashMap(Maps.transformValues(prepareUpload, new Function<UUID, String>() {

public String apply(UUID input) {

return input.toString();

};

}));

}

private Boolean verifyCrcOfUncheckedPart(HttpServletRequest request)

throws IOException, MissingParameterException, FileCorruptedException, FileStillProcessingException {

UUID fileId = UUID.fromString(fileUploaderHelper.getParameterValue(request, UploadServletParameter.fileId));

try {

uploadProcessor.verifyCrcOfUncheckedPart(fileId,

fileUploaderHelper.getParameterValue(request, UploadServletParameter.crc));

}

catch (InvalidCrcException e) {

// no need to log this exception, a fallback behaviour is defined in the

// throwing method.

// but we need to return something!

return Boolean.FALSE;

}

return Boolean.TRUE;

}

}

异步上传UploadServletAsync

@Component("javaLargeFileUploaderAsyncServlet")

@WebServlet(name = "javaLargeFileUploaderAsyncServlet", urlPatterns = { "/javaLargeFileUploaderAsyncServlet" }, asyncSupported = true)

public class UploadServletAsync extends HttpRequestHandlerServlet

implements HttpRequestHandler {

private static final Logger log = LoggerFactory.getLogger(UploadServletAsync.class);

@Autowired

ExceptionCodeMappingHelper exceptionCodeMappingHelper;

@Autowired

UploadServletAsyncProcessor uploadServletAsyncProcessor;

@Autowired

StaticStateIdentifierManager staticStateIdentifierManager;

@Autowired

StaticStateManager<StaticStatePersistedOnFileSystemEntity> staticStateManager;

@Autowired

FileUploaderHelper fileUploaderHelper;

@Autowired

Authorizer authorizer;

/**

* Maximum time that a streaming request can take.<br>

*/

private long taskTimeOut = DateUtils.MILLIS_PER_HOUR;

@Override

public void handleRequest(final HttpServletRequest request, final HttpServletResponse response)

throws ServletException, IOException {

// process the request

try {

//check if uploads are allowed

if (!uploadServletAsyncProcessor.isEnabled()) {

throw new UploadIsCurrentlyDisabled();

}

// extract stuff from request

final FileUploadConfiguration process = fileUploaderHelper.extractFileUploadConfiguration(request);

log.debug("received upload request with config: "+process);

// verify authorization

final UUID clientId = staticStateIdentifierManager.getIdentifier();

authorizer.getAuthorization(request, UploadServletAction.upload, clientId, process.getFileId());

//check if that file is not paused

if (uploadServletAsyncProcessor.isFilePaused(process.getFileId())) {

log.debug("file "+process.getFileId()+" is paused, ignoring async request.");

return;

}

// get the model

StaticFileState fileState = staticStateManager.getEntityIfPresent().getFileStates().get(process.getFileId());

if (fileState == null) {

throw new FileNotFoundException("File with id " + process.getFileId() + " not found");

}

// process the request asynchronously

final AsyncContext asyncContext = request.startAsync();

asyncContext.setTimeout(taskTimeOut);

// add a listener to clear bucket and close inputstream when process is complete or

// with

// error

asyncContext.addListener(new UploadServletAsyncListenerAdapter(process.getFileId()) {

@Override

void clean() {

log.debug("request " + request + " completed.");

// we do not need to clear the inputstream here.

// and tell processor to clean its shit!

uploadServletAsyncProcessor.clean(clientId, process.getFileId());

}

});

// then process

uploadServletAsyncProcessor.process(fileState, process.getFileId(), process.getCrc(), process.getInputStream(),

new WriteChunkCompletionListener() {

@Override

public void success() {

asyncContext.complete();

}

@Override

public void error(Exception exception) {

// handles a stream ended unexpectedly , it just means the user has

// stopped the

// stream

if (exception.getMessage() != null) {

if (exception.getMessage().equals("Stream ended unexpectedly")) {

log.warn("User has stopped streaming for file " + process.getFileId());

}

else if (exception.getMessage().equals("User cancellation")) {

log.warn("User has cancelled streaming for file id " + process.getFileId());

// do nothing

}

else {

exceptionCodeMappingHelper.processException(exception, response);

}

}

else {

exceptionCodeMappingHelper.processException(exception, response);

}

asyncContext.complete();

}

});

}

catch (Exception e) {

exceptionCodeMappingHelper.processException(e, response);

}

}

}

3,请求流程图:

主要思路就是将文件切分,然后分块上传。

详细代码可以参照我写的这篇文章:http://t.cn/AiHwnJaB

原文地址:https://www.cnblogs.com/songsu/p/12187552.html

时间: 2024-11-06 10:06:18

求超大文件上传方案( HTML5 )的相关文章

求超大文件上传方案( csharp )

HTML部分 <%@ Page Language="C#" AutoEventWireup="true" CodeBehind="index.aspx.cs" Inherits="up6.index" %> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xh

求超大文件上传方案( WebUploader )

文件夹数据库处理逻辑 public class DbFolder { JSONObject root; public DbFolder() { this.root = new JSONObject(); this.root.put("f_id", ""); this.root.put("f_nameLoc", "根目录"); this.root.put("f_pid", ""); thi

求超大文件上传方案( jsp )

1 背景 用户本地有一份txt或者csv文件,无论是从业务数据库导出.还是其他途径获取,当需要使用蚂蚁的大数据分析工具进行数据加工.挖掘和共创应用的时候,首先要将本地文件上传至ODPS,普通的小文件通过浏览器上传至服务器,做一层中转便可以实现,但当这份文件非常大到了10GB级别,我们就需要思考另一种形式的技术方案了,也就是本文要阐述的方案. 技术要求主要有以下几方面: 支持超大数据量.10G级别以上 稳定性:除网络异常情况100%成功 准确性:数据无丢失,读写准确性100% 效率:1G文件分钟级

求超大文件上传方案( vue )

java两台服务器之间,大文件上传(续传),采用了Socket通信机制以及JavaIO流两个技术点,具体思路如下: 实现思路: 1.服:利用ServerSocket搭建服务器,开启相应端口,进行长连接操作 2.服:使用ServerSocket.accept()方法进行阻塞,接收客户端请求 3.服:每接收到一个Socket就建立一个新的线程来处理它 4.客:利用Socket进行远程连接,询问已上传进度 5.客:使用FileInputStream.skip(long length)从指定位置读取文件

超大文件上传方案(B/S)

javaweb上传文件 上传文件的jsp中的部分 上传文件同样可以使用form表单向后端发请求,也可以使用 ajax向后端发请求 1.通过form表单向后端发送请求 <form id="postForm" action="${pageContext.request.contextPath}/UploadServlet" method="post" enctype="multipart/form-data"> <

超大文件上传方案( Java )

1.介绍enctype enctype 属性规定发送到服务器之前应该如何对表单数据进行编码. enctype作用是告知服务器请求正文的MIME类型(请求消息头content-type的作用一样) 1.1 enctype的取值有三种 值 描述 application/x-www-form-urlencoded 在发送前编码所有字符(默认) multipart/form-data 不对字符编码.每一个表单项分割为一个部件 text/plain 空格转换为 “+” 加号,但不对特殊字符编码. 1. 当

JavaScript超大文件上传

随着视频网站和大数据应用的普及,特别是高清视频和4K视频应用的到来,超大文件上传已经成为了日常的基础应用需求. 但是在很多情况下,平台运营方并没有大文件上传和断点续传的开发经验,往往在网上找一些简单的PHP或者Java程序来实现基本的上传功能,然而在实际使用中会发现,这些基于脚本语言实现的上传功能模块性能很弱,一是不支持2GB以上的内容上传:二是无法支持断点续传:三是效率极低,单台服务器最多支持几十个并发上传连接. 当前我们要搭建一个运营级的视频服务平台,在尝试了各种产品均无法满足要求,因此最后

超大文件上传到Azure Linux虚拟机最佳实践

客户在实际进行迁移的时候,往往碰到需要将本地数据中心的超大文件,比如单个200GB的文件,或者总共1TB的无数文件上传到Azure上的情况,尤其是传到Azure的Linux虚拟机的场景,这种场景包括: 大日志文件上传到Azure进行分析 数据库备份文件导入导出 大数据文件下载到本地等等 尤其是要将数据库备份文件导入到Linux虚拟机进行数据库恢复,一般用户常常会选择传统的FTP方式进行处理,这样的方式通常会花费用户十几个小时,甚至几天的时间进行文件传输,但是在云计算的时代,其实有非常多的工具和方

超大文件上传(web应用---SSH框架)

struts提供的上传文件功能,文件太大就报错. 需要解决超大文件(几个G)上传的问题.断点续传的功能. 技术选型,查百度之后.最终选择了这个stream插件. stream官方资料地址:http://www.twinkling.cn/ 去下载源码.源码地址.项目源码托管在git.oschina.net,地址:http://git.oschina.net/jiangdx/stream 解压安装包之后,在eclipse中导入此项目.(注,这是一个Maven项目,服务器用的是tomcat 不是jet