Java中比较不同的MD5计算方式

在项目中经常需要使用计算文件的md5，用作一些用途，md5计算算法，通常在网络上查询时，一般给的算法是读取整个文件的字节流，然后计算文件的md5,这种方式当文件较大，且有很大并发量时，则可能导致内存打爆掉。所以如下代码提供了几种方式。并通过计算一个323M的文件的md5和大小给出了，GC的一些信息

代码

/*
 * Copyright (C) 2016. All Rights Reserved.
 */
package me.nabil.mixed;

import org.apache.commons.codec.digest.DigestUtils;
import org.apache.commons.io.FileUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;

/**
 * md5加密类
 */
public class Md5Util {

    private static final Logger LOGGER = LoggerFactory.getLogger(Md5Util.class);

    protected static final char[] hexDigits = {‘0‘, ‘1‘, ‘2‘, ‘3‘, ‘4‘, ‘5‘, ‘6‘, ‘7‘, ‘8‘,
            ‘9‘, ‘a‘, ‘b‘, ‘c‘, ‘d‘, ‘e‘, ‘f‘};

    /**
     * 将二进制数据进行md5加密
     *
     * @param data 文件二进制数据
     * @return md5加密码
     */
    public static String getMd5ByByte(byte[] data) {
        try {
            char[] str;
            MessageDigest mdTemp = MessageDigest.getInstance("MD5");
            mdTemp.update(data);
            byte[] md = mdTemp.digest();
            int j = md.length;
            str = new char[j * 2];
            int k = 0;
            for (int i = 0; i < j; i++) {
                byte byte0 = md[i];
                str[k++] = hexDigits[byte0 >>> 4 & 0xf];
                str[k++] = hexDigits[byte0 & 0xf];
            }

            return new String(str);
        } catch (Exception e) {
            LOGGER.error("Error occurred when making MD5 for data file", e);
            return null;
        }
    }

    /**
     * 文件对象
     *
     * @param file
     * @return
     */
    public static String getMD5ByFile(File file) {
        FileInputStream fis = null;
        try {
            MessageDigest md = MessageDigest.getInstance("MD5");
            fis = new FileInputStream(file);
            byte[] buffer = new byte[8192];
            int length = -1;
            System.out.println("开始算");
            while ((length = fis.read(buffer)) != -1) {
                md.update(buffer, 0, length);
            }
            System.out.println("算完了");
            return bytesToString(md.digest());
        } catch (IOException ex) {
            LOGGER.info(ex.getMessage(), ex);
            return null;
        } catch (NoSuchAlgorithmException e) {
            LOGGER.info(e.getMessage(), e);
            return null;
        } finally {
            try {
                fis.close();
            } catch (IOException ex) {
                LOGGER.info(ex.getMessage(), ex);
            }
        }
    }

    /**
     * bytesToString
     *
     * @param data
     * @return
     */
    public static String bytesToString(byte[] data) {
        char hexDigits[] = {‘0‘, ‘1‘, ‘2‘, ‘3‘, ‘4‘, ‘5‘, ‘6‘, ‘7‘, ‘8‘, ‘9‘, ‘a‘, ‘b‘, ‘c‘, ‘d‘,
                ‘e‘, ‘f‘};
        char[] temp = new char[data.length * 2];
        for (int i = 0; i < data.length; i++) {
            byte b = data[i];
            temp[i * 2] = hexDigits[b >>> 4 & 0x0f];
            temp[i * 2 + 1] = hexDigits[b & 0x0f];
        }
        return new String(temp);
    }

    /**
     * main
     *
     * @param args
     * @throws IOException
     */
    public static void main(String[] args) throws IOException {
        String path = "/path/to/file"; // 一个323M的文件路径

        // 1. 流处理
        System.out.println(getMD5ByFile(new File(path)) + ":" + FileUtils.sizeOf(new File(path)));

        // 2. common-codec
        System.out.println(DigestUtils.md5Hex(new FileInputStream(path)) + ":" + FileUtils.sizeOf(new File(path)));

        // 3. 全部读进内存
        byte[] data = FileUtils.readFileToByteArray(new File(path));
        System.out.println(getMd5ByByte(data) + ":" + data.length);

    }
}

分析

以上三种方式，通过注释其他两种，只运行其中一种的方式来给出JVM的一些信息

运行时加入参数jvm参数：

-XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime

1. 8M-buffer流处理

0.151: Total time for which application threads were stopped: 0.0000535 seconds, Stopping threads took: 0.0000072 seconds
0.151: Total time for which application threads were stopped: 0.0000428 seconds, Stopping threads took: 0.0000083 seconds
Connected to the target VM, address: ‘127.0.0.1:63630‘, transport: ‘socket‘
0.314: Total time for which application threads were stopped: 0.0001357 seconds, Stopping threads took: 0.0000145 seconds
0.417: Total time for which application threads were stopped: 0.0001063 seconds, Stopping threads took: 0.0000169 seconds
开始算
1.442: Total time for which application threads were stopped: 0.0221453 seconds, Stopping threads took: 0.0219728 seconds
算完了
Disconnected from the target VM, address: ‘127.0.0.1:63630‘, transport: ‘socket‘
aa3858dfbc48daab4b891e112d5396fc:323251542
Heap
 PSYoungGen      total 38400K, used 6658K [0x0000000795580000, 0x0000000798000000, 0x00000007c0000000)
  eden space 33280K, 20% used [0x0000000795580000,0x0000000795c00b00,0x0000000797600000)
  from space 5120K, 0% used [0x0000000797b00000,0x0000000797b00000,0x0000000798000000)
  to   space 5120K, 0% used [0x0000000797600000,0x0000000797600000,0x0000000797b00000)
 ParOldGen       total 87552K, used 0K [0x0000000740000000, 0x0000000745580000, 0x0000000795580000)
  object space 87552K, 0% used [0x0000000740000000,0x0000000740000000,0x0000000745580000)
 Metaspace       used 3560K, capacity 4728K, committed 4864K, reserved 1056768K
  class space    used 386K, capacity 424K, committed 512K, reserved 1048576K

Process finished with exit code 0

2. apache common-codec的结果

0.184: Total time for which application threads were stopped: 0.0000615 seconds, Stopping threads took: 0.0000078 seconds
0.185: Total time for which application threads were stopped: 0.0000279 seconds, Stopping threads took: 0.0000056 seconds
0.640: Total time for which application threads were stopped: 0.0001445 seconds, Stopping threads took: 0.0000329 seconds
0.891: Total time for which application threads were stopped: 0.0001546 seconds, Stopping threads took: 0.0000383 seconds
1.904: Total time for which application threads were stopped: 0.0102225 seconds, Stopping threads took: 0.0101437 seconds
4.580: Total time for which application threads were stopped: 0.0001097 seconds, Stopping threads took: 0.0000165 seconds
aa3858dfbc48daab4b891e112d5396fc:323251542
Heap
 PSYoungGen      total 38400K, used 9508K [0x0000000795580000, 0x0000000798000000, 0x00000007c0000000)
  eden space 33280K, 28% used [0x0000000795580000,0x0000000795ec92b8,0x0000000797600000)
  from space 5120K, 0% used [0x0000000797b00000,0x0000000797b00000,0x0000000798000000)
  to   space 5120K, 0% used [0x0000000797600000,0x0000000797600000,0x0000000797b00000)
 ParOldGen       total 87552K, used 0K [0x0000000740000000, 0x0000000745580000, 0x0000000795580000)
  object space 87552K, 0% used [0x0000000740000000,0x0000000740000000,0x0000000745580000)
 Metaspace       used 3616K, capacity 4792K, committed 5120K, reserved 1056768K
  class space    used 391K, capacity 424K, committed 512K, reserved 1048576K
Disconnected from the target VM, address: ‘127.0.0.1:63674‘, transport: ‘socket‘

3. 全部读进内存再计算

0.161: Total time for which application threads were stopped: 0.0000584 seconds, Stopping threads took: 0.0000190 seconds
0.161: Total time for which application threads were stopped: 0.0000407 seconds, Stopping threads took: 0.0000054 seconds
0.356: Total time for which application threads were stopped: 0.0001168 seconds, Stopping threads took: 0.0000177 seconds
0.860: [GC (Allocation Failure) [PSYoungGen: 21892K->4950K(38400K)] 21892K->25438K(125952K), 0.0187028 secs] [Times: user=0.01 sys=0.01, real=0.02 secs]
0.879: Total time for which application threads were stopped: 0.0189020 seconds, Stopping threads took: 0.0000150 seconds
4.411: Total time for which application threads were stopped: 0.0445206 seconds, Stopping threads took: 0.0435764 seconds
4.694: Total time for which application threads were stopped: 0.0003998 seconds, Stopping threads took: 0.0000154 seconds
5.695: Total time for which application threads were stopped: 0.0001298 seconds, Stopping threads took: 0.0000841 seconds
aa3858dfbc48daab4b891e112d5396fc:323251542
Heap
 PSYoungGen      total 38400K, used 22181K [0x0000000795580000, 0x000000079a080000, 0x00000007c0000000)
  eden space 33280K, 51% used [0x0000000795580000,0x0000000796653bd0,0x0000000797600000)
  from space 5120K, 96% used [0x0000000797600000,0x0000000797ad58d0,0x0000000797b00000)
  to   space 5120K, 0% used [0x0000000799b80000,0x0000000799b80000,0x000000079a080000)
 ParOldGen       total 863744K, used 827683K [0x0000000740000000, 0x0000000774b80000, 0x0000000795580000)
  object space 863744K, 95% used [0x0000000740000000,0x0000000772848dd8,0x0000000774b80000)
 Metaspace       used 3663K, capacity 4792K, committed 5120K, reserved 1056768K
  class space    used 393K, capacity 424K, committed 512K, reserved 1048576K
Disconnected from the target VM, address: ‘127.0.0.1:63717‘, transport: ‘socket‘

Process finished with exit code 0

结论

可以看出前两种方案是可取的，第三种方式内存占用太高。

时间： 2024-12-09 10:50:36

Java中比较不同的MD5计算方式的相关文章

JAVA中集合输出的四种方式

在JAVA中Collection输出有四种方式,分别如下: 一) Iterator输出. 该方式适用于Collection的所有子类. public class Hello { public static void main(String[] args) throws Exception { Set<Person> javaProgramers = new HashSet<Person>(); javaProgramers.add(new Person("aaron&qu

java中设置代理的两种方式

1 前言 有时候我们的程序中要提供可以使用代理访问网络,代理的方式包括http.https.ftp.socks代理.比如在IE浏览器设置代理. 那我们在我们的java程序中使用代理呢,有如下两种方式.直接上代码. 2 采用设置系统属性 ? 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 import jav

JAVA中单例模式的几种实现方式

1 线程不安全的实现方法 首先介绍java中最基本的单例模式实现方式,我们可以在一些初级的java书中看到.这种实现方法不是线程安全的,所以在项目实践中如果涉及到线 程安全就不会使用这种方式.但是如果不需要保证线程安全,则这种方式还是不错的,因为所需要的开销比较小.下面是具体的实现代码: 转http://www.cnblogs.com/CodeGuy/p/3580486.html public Class Singleton { private static Singleton instance

java中获取路径的几种方式

总是忘记, 备份一下,方便下次用. 第一种: File directory = new File("");//参数为空 String courseFile = directory.getCanonicalPath() ;System.out.println(courseFile); 结果:C:\Documents and Settings\Administrator\workspace\projectName获取当前类的所在工程路径; 第二种: File f = new File(th

JAVA中实现多线程的四种方式

Java中多线程实现方式主要有四种:1<继承Thread类.2<实现Runnable接口.3<实现Callable接口通过FutureTask包装器来创建Thread线程.4<使用ExecutorService.Callable.Future实现有返回结果的多线程. 其中前两种方式线程执行完后都没有返回值,后两种是带返回值的. 1.继承Thread类创建线程 Thread类本质上是实现了Runnable接口的一个实例,代表一个线程的实例.启动线程的唯一方法就是通过Thread类的s

细说java中Map的两种迭代方式

以前对java中迭代方式总是迷迷糊糊的,今天总算弄懂了,特意的总结了一下,基本是算是理解透彻了. 1.再说Map之前先说下Iterator: Iterator主要用于遍历(即迭代访问)Collection集合中的元素,Iterator也称为迭代器.它仅仅只有三个方法:hasNext(),next()和remove() hasNext():如果仍有元素可以迭代,则返回 true.(换句话说,如果 next 返回了元素而不是 抛出异常,则返回 true). next():返回迭代的下一个元素. re

JAVA中创建字符串的两种方式的区别

我们知道,通常在Java中创建一个字符串会有两种方式,通过双引号直接赋值和通过构造器来创建. String x = "abcd"; String y = new String("abcd"); 然而,这两种方式之间的区别是什么?分别应用于哪些情况,之前还不是很懂. 1.双引号的方式 String x = "abcd"; String y = "abcd"; System.out.println(x==y);//true Sys

Java中实现多线程的两种方式之间的区别

Java提供了线程类Thread来创建多线程的程序.其实,创建线程与创建普通的类的对象的操作是一样的,而线程就是Thread类或其子类的实例对象.每个Thread对象描述了一个单独的线程.要产生一个线程,有两种方法: ◆需要从Java.lang.Thread类派生一个新的线程类,重载它的run()方法:  ◆实现Runnalbe接口,重载Runnalbe接口中的run()方法. 为什么Java要提供两种方法来创建线程呢?它们都有哪些区别?相比而言,哪一种方法更好呢? 在Java中,类仅支持单继承

java中终止线程的三种方式

在java中有三种方式可以终止线程.分别为: 1.  使用退出标志,使线程正常退出,也就是当run方法完成后线程终止.  2.  使用stop方法强行终止线程(这个方法不推荐使用,因为stop和suspend.resume一样,也可能发生不可预料的结果). 3.  使用interrupt方法中断线程. 下面我们来详细的介绍这三种方法. 1. 使用退出标志终止线程 当run方法执行完后,线程就会退出.但有时run方法是永远不会结束的.如在服务端程序中使用线程进行监听客户端请求,或是其他的需要循环处