链接：http://pan.baidu.com/s/1gfHnaVL 密码：7j12
mysql-connector version 5.1.32
若在安装版本过程遇到些问题，可参考http://dbspace.blog.51cto.com/6873717/1875955，其中一些问题的解决办法
下载并安装：
cd /usr/local/tar -zxvf sqoop2-1.99.3-cdh5.0.0.tar.gzmv sqoop2-1.99.3-cdh5.0.0 sqoop添加sqoop2到系统环境变量中：export SQOOP_HOME=/usr/local/sqoopexport CATALINA_BASE=$SQOOP_HOME/serverexport PATH=$PATH:/usr/local/sqoop/bin拷贝mysql驱动包到$SQOOP2_HOME/server/lib下cp mysql-connector-java-5.1.32-bin.jar /usr/local/sqloop/server/lib/ 修改配置文件：vim /usr/local/sqoop/server/conf/sqoop.propertiesorg.apache.sqoop.submission.engine.mapreduce.configuration.directory=/usr/local/hadoop/etc/hadoop#hadoop的配置文件路径vim /usr/local/sqoop/server/conf/catalina.properties把原来58行注释了，这里主要配置了hadoop的jar包的路径信息common.loader=${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar,${ca talina.home}/../lib/*.jar,/usr/local/hadoop/share/hadoop/common/*.jar,/usr/local/hadoop/share/hadoop/common/lib/*. jar,/usr/local/hadoop/share/hadoop/hdfs/*.jar,/usr/local/hadoop/share/hadoop/hdfs/lib/*.jar,/usr/local/hadoop/shar e/hadoop/mapreduce/*.jar,/usr/local/hadoop/share/hadoop/mapreduce/lib/*.jar,/usr/local/hadoop/share/hadoop/tools/* .jar,/usr/local/hadoop/share/hadoop/tools/lib/*.jar,/usr/local/hadoop/share/hadoop/yarn/*.jar,/usr/local/hadoop/sh are/hadoop/yarn/lib/*.jar启动\停止sqoop/usr/local/sqoop/sqoop2-server start/stop验证是否启动成功：方式一：jps查看进程: Bootstrap [[email protected] sqoop]# jps
25505 SqoopShell13080 SecondaryNameNode12878 NameNode26568 Jps方式二：方式二：http://192.168.1.114:12000/sqoop/version #SQOOP默认使用的端口为12000在/usr/local/sqoop/server/conf/server.xml中进行设置
####接下来测试mysql到hadoop存储的一个过程1、客户端登陆[[email protected] bin]# sqoop2-shell Sqoop home directory: /usr/local/sqoopSqoop Shell: Type ‘help‘ or ‘\h‘ for help.sqoop:000> 2、创建一个mysql链接，在这个版本create 就只有[connection|job],注意不同版本的添加链接方式是不同的. 查看支持的链接服务 sqoop:000> show connector+----+------------------------+-----------------+------------------------------------------------------+| Id | Name | Version | Class |+----+------------------------+-----------------+------------------------------------------------------+| 1 | generic-jdbc-connector | 1.99.3-cdh5.0.0 | org.apache.sqoop.connector.jdbc.GenericJdbcConnector |+----+------------------------+-----------------+------------------------------------------------------+##在1.99.7的版本显示的方式和服务更多。sqoop:000> create connection --cid 1Creating connection for connector with id 1Please fill following values to create new connection objectName: mysql_to_hadoop
Connection configuration
JDBC Driver Class: com.mysql.jdbc.DriverJDBC Connection String: jdbc:mysql://192.168.1.107:3306/sqoop #这里需要在1.107先添加好库sqoopUsername: sqoop##需要在数据库添加好链接的用户Password: *******JDBC Connection Properties: There are currently 0 values in the map:entry# Security related configuration optionsMax connections:New connection was successfully created with validation status ACCEPTABLE and persistent id 22、创建jobsqoop:000> create job --xid 2 --type import##注意 --xid 2为链接的id号Creating job for connection with id 2Please fill following values to create new job objectName: mysql_to_hadoop
Database configuration
Schema name: sqoop#MySQL的库名Table name: wangyuan#库下的表Table SQL statement: Table column names: Partition column name: Nulls in partition column: Boundary query:
Output configuration
Storage type: 0 : HDFSChoose: 0Output format: 0 : TEXT_FILE 1 : SEQUENCE_FILEChoose: 0Compression format: 0 : NONE 1 : DEFAULT 2 : DEFLATE 3 : GZIP 4 : BZIP2 5 : LZO 6 : LZ4 7 : SNAPPYChoose: 0Output directory: hdfs://192.168.1.114:9000/home/mysql_to_hdfs2#注意这个mysql_to_hdfs不能再hadoop的/home/已经存在的，但/home路径要存在,9000端口是在配置hadoop的时候配置，根据实际，或者通过WEB查看http:ip:50070----显示Overview ‘mycat:9000‘ (active)创建hdfs路径/usr/local/hadoop/bin/hadoop fs -mkidr /home查看创建目录：/usr/local/hadoop/bin/hadoop fs -ls /home 或者通过WEB查看http:ip:50070Throttling resourcesExtractors: Loaders: New job was successfully created with validation status FINE and persistent id 2sqoop:000> 启动jobsqoop:000> start job --jid 2Exception has occurred during processing command Exception: org.apache.sqoop.common.SqoopException Message: CLIENT_0001:Server has returned exception根本不知道这个提示说什么，通过修改设置：set option --name verbose --value truesqoop:000> start job --jid 2 Submission detailsJob ID: 2Server URL: http://localhost:12000/sqoop/Created by: rootCreation date: 2016-11-23 21:15:27 CSTLastly updated by: rootExternal ID: job_1479653943050_0007 http://haproxy:8088/proxy/application_1479653943050_0007/Connector schema: Schema{name=wangyuan,columns=[ FixedPoint{name=id,nullable=null,byteSize=null,unsigned=null}, Date{name=c_time,nullable=null,fraction=null,timezone=null}]}2016-11-23 21:15:27 CST: BOOTING - Progress is not available返回这样信息OK查看结果通过WEB
/usr/local/hadoop/bin/hadoop fs -ls /home/
时间： 2024-10-10 17:32:06