Hadoop版本有很多,这里我选择的是cdh版本。cdh是Cloudera公司在Apache原版基础经过加工的东西。chd具体的下载地址为:
http://archive-primary.cloudera.com/cdh5/cdh/5/
版本信息如下:
Hadoop:Hadoop 2.3.0-cdh5.1.0
JDK:1.7.0_79
Maven:apache-maven-3.2.5(3.3.1以及更高版本必须在JDK1.7以上)
Protobuf:protobuf-2.5.0
Ant:1.7.1
1.安装Maven
Maven可以自己到Maven官网上下载(http://maven.apache.org/download.cgi)。需要注意的是:如果JDK是1.6及其以下的,那Maven最多只能下载到3.2.5,如果下载3.3.1及其更高版本就必须要1.7的JDK。
2.安装Protobuf
Protobuf的安装参见:http://www.tuicool.com/articles/jM7Nn2,安装过程也很简单。
3.安装其他需要的组件
还需要安装ant、gcc等组件,运行如下命令,会根据需要自己到网上下载安装。
yum install autoconf automake libtool ant gcc gcc-c++ make cmake openssl-devel ncurses-devel
如果失败就多试几次。如果一直失败可考虑自己上网下载然后手动安装。具体流程为:Windows上网下载ftp传给Linux在Linux上解压安装。
其中Ant的安装步骤可见:http://blog.csdn.net/zj7758/article/details/8437739
4.下载Hadoop源码并解压
源码下载地址为:http://archive-primary.cloudera.com/cdh5/cdh/5/
5.编译源码
运行:
mvn package -DskipTests -Pdist,native -Dtar
如果出现错误可以加上-X显示错误详细信息:
mvn -X package -DskipTests -Pdist,native –Dtar
重编译完成后,native包一般在如下的目录下:
../hadoop-2.3.0-cdh5.1.0/src/hadoop-common-project/hadoop-common/target/native/target/usr/local/lib/
如下图:
安装过程中常见的错误:
1、maven中的jar包下载失败。
Downloading: https://repository.cloudera.com/artifactory/cloudera-repos/com/cloudera/cdh/cdh-root/5.1.0/cdh-root-5.1.0.pom Downloading: http://repository.jboss.org/nexus/content/groups/public/com/cloudera/cdh/cdh-root/5.1.0/cdh-root-5.1.0.pom Downloading: https://repo.maven.apache.org/maven2/com/cloudera/cdh/cdh-root/5.1.0/cdh-root-5.1.0.pom [ERROR] The build could not read 1 project -> [Help 1] [ERROR] [ERROR] The project org.apache.hadoop:hadoop-main:2.3.0-cdh5.1.0 (/home/hbase/install/hadoop-2.3.0-cdh5.1.0/src/pom.xml) has 1 error [ERROR] Non-resolvable parent POM: Could not transfer artifact com.cloudera.cdh:cdh-root:pom:5.1.0 from/to cdh.repo (https://repository.cloudera.com/artifactory/cloudera-repos): repository.cloudera.com and 'parent.relativePath' points at wrong local POM @ line 19, column 11: Unknown host repository.cloudera.com -> [Help 2] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException [ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException [ERROR] Unresolveable build extension: Plugin org.apache.felix:maven-bundle-plugin:2.4.0 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.felix:maven-bundle-plugin:jar:2.4.0 @ at org.apache.maven.project.DefaultProjectBuilder.build(DefaultProjectBuilder.java:416) at org.apache.maven.DefaultMaven.collectProjects(DefaultMaven.java:672) at org.apache.maven.DefaultMaven.getProjectsForMavenReactor(DefaultMaven.java:663) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:253) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:584) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:216) at org.apache.maven.cli.MavenCli.main(MavenCli.java:160) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) [ERROR] [ERROR] The project org.apache.hadoop:hadoop-main:2.3.0 (/usr/local/release-2.3.0/pom.xml) has 1 error
原因找来找去,搞了一天,最后发现Linux集群无法上网(通过ping www.baidu.com不通)。最后修改了一下DNS就可以上网了。
2、编译失败
通过mvn package总是失败,可以考虑通过mvn compile来,如下:
mvn compile -Pdist,native -Dskiptests -Dtar
知识提示:Hadoop调试的开启和关闭
开启:export HADOOP_ROOT_LOGGER=DEBUG,console 关闭:export HADOOP_ROOT_LOGGER=INFO,console
全文完,转载请注明出处:http://blog.csdn.net/ghuil/article/details/45371977