hadoop中使用的Unsafe.java

今天查看hadoop源代码, 发现有个Unsafe.java

稍微总结下

优势 1 减少线程调度开销, Unsafe.java 通过采用非堵塞原子方式来减少线程调度开销
        2 传统线程通信通过wait,notify方法实现(会有信号量的堵塞队列),而Unsafe使用操作系统调度命令park,unpark, 减少信号量的开销,提高新能。

差不多和windows下的InterlockedDecrement差不多的概念, 只是Unsafe的功能更加强大,可以获取的类型更多,另外可以有C/C++方式的内存分配模式以及内存交换等更多操作

附源代码

/*** Eclipse Class Decompiler plugin, copyright (c) 2012 Chao Chen ([email protected]) ***/
package sun.misc;

import java.lang.reflect.Field;
import java.lang.reflect.Modifier;
import java.security.ProtectionDomain;
import sun.reflect.CallerSensitive;
import sun.reflect.Reflection;

public final class Unsafe
{
  private static final Unsafe theUnsafe;
  public static final int INVALID_FIELD_OFFSET = -1;
  public static final int ARRAY_BOOLEAN_BASE_OFFSET;
  public static final int ARRAY_BYTE_BASE_OFFSET;
  public static final int ARRAY_SHORT_BASE_OFFSET;
  public static final int ARRAY_CHAR_BASE_OFFSET;
  public static final int ARRAY_INT_BASE_OFFSET;
  public static final int ARRAY_LONG_BASE_OFFSET;
  public static final int ARRAY_FLOAT_BASE_OFFSET;
  public static final int ARRAY_DOUBLE_BASE_OFFSET;
  public static final int ARRAY_OBJECT_BASE_OFFSET;
  public static final int ARRAY_BOOLEAN_INDEX_SCALE;
  public static final int ARRAY_BYTE_INDEX_SCALE;
  public static final int ARRAY_SHORT_INDEX_SCALE;
  public static final int ARRAY_CHAR_INDEX_SCALE;
  public static final int ARRAY_INT_INDEX_SCALE;
  public static final int ARRAY_LONG_INDEX_SCALE;
  public static final int ARRAY_FLOAT_INDEX_SCALE;
  public static final int ARRAY_DOUBLE_INDEX_SCALE;
  public static final int ARRAY_OBJECT_INDEX_SCALE;
  public static final int ADDRESS_SIZE;

  private static native void registerNatives();

  @CallerSensitive
  public static Unsafe getUnsafe()
  {
    Class localClass = Reflection.getCallerClass();
    if (localClass.getClassLoader() != null)
      throw new SecurityException("Unsafe");
    return theUnsafe;
  }

  public native int getInt(Object paramObject, long paramLong);

  public native void putInt(Object paramObject, long paramLong, int paramInt);

  public native Object getObject(Object paramObject, long paramLong);

  public native void putObject(Object paramObject1, long paramLong, Object paramObject2);

  public native boolean getBoolean(Object paramObject, long paramLong);

  public native void putBoolean(Object paramObject, long paramLong, boolean paramBoolean);

  public native byte getByte(Object paramObject, long paramLong);

  public native void putByte(Object paramObject, long paramLong, byte paramByte);

  public native short getShort(Object paramObject, long paramLong);

  public native void putShort(Object paramObject, long paramLong, short paramShort);

  public native char getChar(Object paramObject, long paramLong);

  public native void putChar(Object paramObject, long paramLong, char paramChar);

  public native long getLong(Object paramObject, long paramLong);

  public native void putLong(Object paramObject, long paramLong1, long paramLong2);

  public native float getFloat(Object paramObject, long paramLong);

  public native void putFloat(Object paramObject, long paramLong, float paramFloat);

  public native double getDouble(Object paramObject, long paramLong);

  public native void putDouble(Object paramObject, long paramLong, double paramDouble);

  @Deprecated
  public int getInt(Object paramObject, int paramInt)
  {
    return getInt(paramObject, paramInt);
  }

  @Deprecated
  public void putInt(Object paramObject, int paramInt1, int paramInt2)
  {
    putInt(paramObject, paramInt1, paramInt2);
  }

  @Deprecated
  public Object getObject(Object paramObject, int paramInt)
  {
    return getObject(paramObject, paramInt);
  }

  @Deprecated
  public void putObject(Object paramObject1, int paramInt, Object paramObject2)
  {
    putObject(paramObject1, paramInt, paramObject2);
  }

  @Deprecated
  public boolean getBoolean(Object paramObject, int paramInt)
  {
    return getBoolean(paramObject, paramInt);
  }

  @Deprecated
  public void putBoolean(Object paramObject, int paramInt, boolean paramBoolean)
  {
    putBoolean(paramObject, paramInt, paramBoolean);
  }

  @Deprecated
  public byte getByte(Object paramObject, int paramInt)
  {
    return getByte(paramObject, paramInt);
  }

  @Deprecated
  public void putByte(Object paramObject, int paramInt, byte paramByte)
  {
    putByte(paramObject, paramInt, paramByte);
  }

  @Deprecated
  public short getShort(Object paramObject, int paramInt)
  {
    return getShort(paramObject, paramInt);
  }

  @Deprecated
  public void putShort(Object paramObject, int paramInt, short paramShort)
  {
    putShort(paramObject, paramInt, paramShort);
  }

  @Deprecated
  public char getChar(Object paramObject, int paramInt)
  {
    return getChar(paramObject, paramInt);
  }

  @Deprecated
  public void putChar(Object paramObject, int paramInt, char paramChar)
  {
    putChar(paramObject, paramInt, paramChar);
  }

  @Deprecated
  public long getLong(Object paramObject, int paramInt)
  {
    return getLong(paramObject, paramInt);
  }

  @Deprecated
  public void putLong(Object paramObject, int paramInt, long paramLong)
  {
    putLong(paramObject, paramInt, paramLong);
  }

  @Deprecated
  public float getFloat(Object paramObject, int paramInt)
  {
    return getFloat(paramObject, paramInt);
  }

  @Deprecated
  public void putFloat(Object paramObject, int paramInt, float paramFloat)
  {
    putFloat(paramObject, paramInt, paramFloat);
  }

  @Deprecated
  public double getDouble(Object paramObject, int paramInt)
  {
    return getDouble(paramObject, paramInt);
  }

  @Deprecated
  public void putDouble(Object paramObject, int paramInt, double paramDouble)
  {
    putDouble(paramObject, paramInt, paramDouble);
  }

  public native byte getByte(long paramLong);

  public native void putByte(long paramLong, byte paramByte);

  public native short getShort(long paramLong);

  public native void putShort(long paramLong, short paramShort);

  public native char getChar(long paramLong);

  public native void putChar(long paramLong, char paramChar);

  public native int getInt(long paramLong);

  public native void putInt(long paramLong, int paramInt);

  public native long getLong(long paramLong);

  public native void putLong(long paramLong1, long paramLong2);

  public native float getFloat(long paramLong);

  public native void putFloat(long paramLong, float paramFloat);

  public native double getDouble(long paramLong);

  public native void putDouble(long paramLong, double paramDouble);

  public native long getAddress(long paramLong);

  public native void putAddress(long paramLong1, long paramLong2);

  public native long allocateMemory(long paramLong);

  public native long reallocateMemory(long paramLong1, long paramLong2);

  public native void setMemory(Object paramObject, long paramLong1, long paramLong2, byte paramByte);

  public void setMemory(long paramLong1, long paramLong2, byte paramByte)
  {
    setMemory(null, paramLong1, paramLong2, paramByte);
  }

  public native void copyMemory(Object paramObject1, long paramLong1, Object paramObject2, long paramLong2, long paramLong3);

  public void copyMemory(long paramLong1, long paramLong2, long paramLong3)
  {
    copyMemory(null, paramLong1, null, paramLong2, paramLong3);
  }

  public native void freeMemory(long paramLong);

  @Deprecated
  public int fieldOffset(Field paramField)
  {
    if (Modifier.isStatic(paramField.getModifiers())) {
      return (int)staticFieldOffset(paramField);
    }
    return (int)objectFieldOffset(paramField);
  }

  @Deprecated
  public Object staticFieldBase(Class paramClass)
  {
    Field[] arrayOfField = paramClass.getDeclaredFields();
    for (int i = 0; i < arrayOfField.length; ++i) {
      if (Modifier.isStatic(arrayOfField[i].getModifiers())) {
        return staticFieldBase(arrayOfField[i]);
      }
    }
    return null;
  }

  public native long staticFieldOffset(Field paramField);

  public native long objectFieldOffset(Field paramField);

  public native Object staticFieldBase(Field paramField);

  public native boolean shouldBeInitialized(Class<?> paramClass);

  public native void ensureClassInitialized(Class paramClass);

  public native int arrayBaseOffset(Class paramClass);

  public native int arrayIndexScale(Class paramClass);

  public native int addressSize();

  public native int pageSize();

  public native Class defineClass(String paramString, byte[] paramArrayOfByte, int paramInt1, int paramInt2, ClassLoader paramClassLoader, ProtectionDomain paramProtectionDomain);

  @Deprecated
  @CallerSensitive
  public native Class defineClass(String paramString, byte[] paramArrayOfByte, int paramInt1, int paramInt2);

  public native Class defineAnonymousClass(Class paramClass, byte[] paramArrayOfByte, Object[] paramArrayOfObject);

  public native Object allocateInstance(Class paramClass)
    throws InstantiationException;

  public native void monitorEnter(Object paramObject);

  public native void monitorExit(Object paramObject);

  public native boolean tryMonitorEnter(Object paramObject);

  public native void throwException(Throwable paramThrowable);

  public final native boolean compareAndSwapObject(Object paramObject1, long paramLong, Object paramObject2, Object paramObject3);

  public final native boolean compareAndSwapInt(Object paramObject, long paramLong, int paramInt1, int paramInt2);

  public final native boolean compareAndSwapLong(Object paramObject, long paramLong1, long paramLong2, long paramLong3);

  public native Object getObjectVolatile(Object paramObject, long paramLong);

  public native void putObjectVolatile(Object paramObject1, long paramLong, Object paramObject2);

  public native int getIntVolatile(Object paramObject, long paramLong);

  public native void putIntVolatile(Object paramObject, long paramLong, int paramInt);

  public native boolean getBooleanVolatile(Object paramObject, long paramLong);

  public native void putBooleanVolatile(Object paramObject, long paramLong, boolean paramBoolean);

  public native byte getByteVolatile(Object paramObject, long paramLong);

  public native void putByteVolatile(Object paramObject, long paramLong, byte paramByte);

  public native short getShortVolatile(Object paramObject, long paramLong);

  public native void putShortVolatile(Object paramObject, long paramLong, short paramShort);

  public native char getCharVolatile(Object paramObject, long paramLong);

  public native void putCharVolatile(Object paramObject, long paramLong, char paramChar);

  public native long getLongVolatile(Object paramObject, long paramLong);

  public native void putLongVolatile(Object paramObject, long paramLong1, long paramLong2);

  public native float getFloatVolatile(Object paramObject, long paramLong);

  public native void putFloatVolatile(Object paramObject, long paramLong, float paramFloat);

  public native double getDoubleVolatile(Object paramObject, long paramLong);

  public native void putDoubleVolatile(Object paramObject, long paramLong, double paramDouble);

  public native void putOrderedObject(Object paramObject1, long paramLong, Object paramObject2);

  public native void putOrderedInt(Object paramObject, long paramLong, int paramInt);

  public native void putOrderedLong(Object paramObject, long paramLong1, long paramLong2);

  public native void unpark(Object paramObject);

  public native void park(boolean paramBoolean, long paramLong);

  public native int getLoadAverage(double[] paramArrayOfDouble, int paramInt);

  static
  {
    registerNatives();
    Reflection.registerMethodsToFilter(Unsafe.class, new String[] { "getUnsafe" });

    theUnsafe = new Unsafe();

    ARRAY_BOOLEAN_BASE_OFFSET = theUnsafe.arrayBaseOffset([Z.class);

    ARRAY_BYTE_BASE_OFFSET = theUnsafe.arrayBaseOffset([B.class);

    ARRAY_SHORT_BASE_OFFSET = theUnsafe.arrayBaseOffset([S.class);

    ARRAY_CHAR_BASE_OFFSET = theUnsafe.arrayBaseOffset([C.class);

    ARRAY_INT_BASE_OFFSET = theUnsafe.arrayBaseOffset([I.class);

    ARRAY_LONG_BASE_OFFSET = theUnsafe.arrayBaseOffset([J.class);

    ARRAY_FLOAT_BASE_OFFSET = theUnsafe.arrayBaseOffset([F.class);

    ARRAY_DOUBLE_BASE_OFFSET = theUnsafe.arrayBaseOffset([D.class);

    ARRAY_OBJECT_BASE_OFFSET = theUnsafe.arrayBaseOffset([Ljava.lang.Object.class);

    ARRAY_BOOLEAN_INDEX_SCALE = theUnsafe.arrayIndexScale([Z.class);

    ARRAY_BYTE_INDEX_SCALE = theUnsafe.arrayIndexScale([B.class);

    ARRAY_SHORT_INDEX_SCALE = theUnsafe.arrayIndexScale([S.class);

    ARRAY_CHAR_INDEX_SCALE = theUnsafe.arrayIndexScale([C.class);

    ARRAY_INT_INDEX_SCALE = theUnsafe.arrayIndexScale([I.class);

    ARRAY_LONG_INDEX_SCALE = theUnsafe.arrayIndexScale([J.class);

    ARRAY_FLOAT_INDEX_SCALE = theUnsafe.arrayIndexScale([F.class);

    ARRAY_DOUBLE_INDEX_SCALE = theUnsafe.arrayIndexScale([D.class);

    ARRAY_OBJECT_INDEX_SCALE = theUnsafe.arrayIndexScale([Ljava.lang.Object.class);

    ADDRESS_SIZE = theUnsafe.addressSize();
  }
}

hadoop中使用的Unsafe.java

时间: 2024-10-10 12:02:29

hadoop中使用的Unsafe.java的相关文章

hadoop中Text类 与 java中String类的区别

hadoop 中 的Text类与java中的String类感觉上用法是相似的,但两者在编码格式和访问方式上还是有些差别的,要说明这个问题,首先得了解几个概念: 字符集: 是一个系统支持的所有抽象字符的集合.字符是各种文字和符号的总称,包括各国家文字.标点符号.图形符号.数字等.例如 unicode就是一个字符集,它的目标是涵盖世界上所有国家的文字和符号: 字符编码:是一套法则,使用该法则能够对自然语言的字符的一个集合(如字母表或音节表),与其他东西的一个集合(如号码或电脉冲)进行配对.即在符号集

Hadoop中RPC协议小例子报错java.lang.reflect.UndeclaredThrowableException解决方法

最近在学习传智播客吴超老师的Hadoop视频,里面他在讲解RPC通信原理的过程中给了一个RPC的小例子,但是自己编写的过程中遇到一个小错误,整理如下: log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).log4j:WARN Please initialize the log4j system properly.log4j:WARN See

java.lang.Comparable, java.util.Compartor区别以及Hadoop中关于自定义类型中的compare方法

public interface Comparable<T> { public int compareTo(T o); } 规定了对象内部比较的方法 public interface Comparator<T> { int compare(T o1, T o2); boolean equals(Object obj); } 定义外部比较器的基本方法,其中equals是用来确定两个比较器是否相等. 关于对象内部比较和外部比较这两个接口的区别和使用场景如下: 个人总结: Compara

Hadoop中的CRC数据校验文件

错误分析: 今天在A程序中生成了数据,然后将A结果数据作为B的输入数据,在查看A结果数据时候,发现有一条数据不符合规则,遂将其删除,删除后执行B程序则报错.尝试打断点,发现程序连map过程都不执行,到了 job.waitForCompletion(true);这一步之后直接就报错了,错误信息为: org.apache.hadoop.fs.ChecksumException: Checksum error: file:/E:/hdpSapce/web_log/out/golden/part-m-0

Hadoop 中利用 mapreduce 读写 mysql 数据

Hadoop 中利用 mapreduce 读写 mysql 数据 有时候我们在项目中会遇到输入结果集很大,但是输出结果很小,比如一些 pv.uv 数据,然后为了实时查询的需求,或者一些 OLAP 的需求,我们需要 mapreduce 与 mysql 进行数据的交互,而这些特性正是 hbase 或者 hive 目前亟待改进的地方. 好了言归正传,简单的说说背景.原理以及需要注意的地方: 1.为了方便 MapReduce 直接访问关系型数据库(Mysql,Oracle),Hadoop提供了DBInp

结合手机上网流量业务来说明Hadoop中的自定义数据类型(序列化、反序列化机制)

大家都知道,Hadoop中为Key的数据类型必须实现WritableComparable接口,而Value的数据类型只需要实现Writable接口即可:能做Key的一定可以做Value,能做Value的未必能做Key.但是具体应该怎么应用呢?--本篇文章将结合手机上网流量业务进行分析. 先介绍一下业务场景:统计每个用户的上行流量和,下行流量和,以及总流量和. 本次描述所用数据: 日志格式描述: 日志flowdata.txt中的具体数据: 接下来贴出详细代码,代码中含有详细注释,从代码中可以看出,

Hadoop中WordCount代码-直接加载hadoop的配置文件

Hadoop中WordCount代码-直接加载hadoop的配置文件 在Myeclipse中,直接编写WordCount代码,代码中直接调用core-site.xml,hdfs-site.xml,mapred-site.xml配置文件 package com.apache.hadoop.function; import java.io.IOException; import java.util.Iterator; import java.util.StringTokenizer; import 

深度分析如何在Hadoop中控制Map的数量

深度分析如何在Hadoop中控制Map的数量 [email protected] 很多文档中描述,Mapper的数量在默认情况下不可直接控制干预,因为Mapper的数量由输入的大小和个数决定.在默认情况下,最终input 占据了多少block,就应该启动多少个Mapper.如果输入的文件数量巨大,但是每个文件的size都小于HDFS的blockSize,那么会造成 启动的Mapper等于文件的数量(即每个文件都占据了一个block),那么很可能造成启动的Mapper数量超出限制而导致崩溃.这些逻

Hadoop中的Speculative Task调度策略

一:背景 Speculative Task,又叫推测式任务,是指在分布式集群环境下,因为程序bug,负载不均衡或者资源分布不均,造成同一个job多个task运行速度不不一致,有的task运行速度明显要慢于其他task(比如:一个job的某个task进度只有10%,而其他所有task已经运行完毕),则这些task拖慢了了作业的整体执行进度,为了避免这种情况发生,Hadoop会为该task启动Speculative task,让该Speculative task与原始task同时运行,哪个先运行完,