简介：
Apache Sqoop是专为Apache Hadoop和结构化数据存储如关系数据库之间的数据转换工具的有效工具。你可以使用Sqoop从外部结构化数据存储的数据导入到Hadoop分布式文件系统或相关系统如Hive和HBase。相反，Sqoop可以用来从Hadoop的数据提取和导出到外部结构化数据存储如关系数据库和企业数据仓库。
Sqoop专为大数据批量传输设计，能够分割数据集并创建Hadoop任务来处理每个区块。
下面介绍下安装部署的步骤：
1.下载安装包并解压
下载地址 作者使用的是sqoop-1.4.6-cdh5.7.0.tar.gz版本
# wget下载安装包 也可下载到本地 然后上传至Linux
[[email protected] software]$ pwd
/home/hadoop/software
[[email protected] software]$ wget http://archive.cloudera.com/cdh5/cdh/5/sqoop-1.4.6-cdh5.7.0.tar.gz
# 解压
[[email protected] software]$ tar -xzvf sqoop-1.4.6-cdh5.7.0.tar.gz -C /home/hadoop/app/
# 查看解压目录
[[email protected] sqoop-1.4.6-cdh5.7.0]$ ls -lh
total 1.9M
drwxr-xr-x 2 hadoop hadoop 4.0K Jul 3 16:00 bin --执行脚本目录
-rw-rw-r-- 1 hadoop hadoop 60K Mar 24 2016 build.xml
-rw-rw-r-- 1 hadoop hadoop 1.1K Mar 24 2016 cdh.build.properties
-rw-rw-r-- 1 hadoop hadoop 35K Mar 24 2016 CHANGELOG.txt
drwxr-xr-x 4 hadoop hadoop 4.0K Jul 3 16:00 cloudera
-rw-rw-r-- 1 hadoop hadoop 6.8K Mar 24 2016 cloudera-pom.xml
-rw-rw-r-- 1 hadoop hadoop 9.7K Mar 24 2016 COMPILING.txt
drwxr-xr-x 2 hadoop hadoop 4.0K Jul 3 16:00 conf --配置文件目录
drwxr-xr-x 5 hadoop hadoop 4.0K Jul 3 16:00 docs --文档
drwxr-xr-x 2 hadoop hadoop 4.0K Jul 3 16:00 ivy
-rw-rw-r-- 1 hadoop hadoop 17K Mar 24 2016 ivy.xml
drwxr-xr-x 2 hadoop hadoop 4.0K Jul 3 16:00 lib --lib依赖包
-rw-rw-r-- 1 hadoop hadoop 15K Mar 24 2016 LICENSE.txt
-rw-rw-r-- 1 hadoop hadoop 505 Mar 24 2016 NOTICE.txt
-rw-rw-r-- 1 hadoop hadoop 19K Mar 24 2016 pom-old.xml
-rw-rw-r-- 1 hadoop hadoop 1.1K Mar 24 2016 README.txt
-rw-rw-r-- 1 hadoop hadoop 1012K Mar 24 2016 sqoop-1.4.6-cdh5.7.0.jar --完整jar包
-rw-rw-r-- 1 hadoop hadoop 6.5K Mar 24 2016 sqoop-patch-review.py
-rw-rw-r-- 1 hadoop hadoop 641K Mar 24 2016 sqoop-test-1.4.6-cdh5.7.0.jar
drwxr-xr-x 7 hadoop hadoop 4.0K Mar 24 2016 src --源码
drwxr-xr-x 4 hadoop hadoop 4.0K Jul 3 16:00 testdata2.配置环境变量
# 添加sqoop环境变量 可加入全局 也可只配置个人环境变量
[[email protected] ~]$ sudo vi/etc/profile
export SQOOP_HOME=/home/hadoop/app/sqoop-1.4.6-cdh5.7.0
export PATH=$SQOOP_HOME/bin:$PATH
[[email protected] ~]$ source /etc/profile3.修改sqoop配置文件
[[email protected] conf]$ pwd
/home/hadoop/app/sqoop-1.4.6-cdh5.7.0/conf
[[email protected] conf]$ cp sqoop-env-template.sh sqoop-env.sh
# 添加hadoop及hive目录
[[email protected] conf]$ vi sqoop-env.sh
#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0
#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/home/hadoop/app/hadoop-2.6.0-cdh5.7.0
#set the path to where bin/hbase is available
#export HBASE_HOME=
#Set the path to where bin/hive is available
export HIVE_HOME=/home/hadoop/app/hive-1.1.0-cdh5.7.0
#Set the path for where zookeper config dir is
#export ZOOCFGDIR= 4.拷贝jdbc驱动包到sqoop/lib目录下
# 将Hive lib目录下的mysql驱动包拷贝过来
[[email protected] lib]$ pwd
/home/hadoop/app/sqoop-1.4.6-cdh5.7.0/lib
[[email protected] lib]$ cp /home/hadoop/app/hive-1.1.0-cdh5.7.0/lib/mysql-connector-java-5.1.46.jar .5.sqoop简单测试使用
# 查看命令帮助
[[email protected] ~]$ sqoop help
Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
18/07/03 16:23:05 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.0
usage: sqoop COMMAND [ARGS]
Available commands:
codegen Generate code to interact with database records
create-hive-table Import a table definition into Hive
eval Evaluate a SQL statement and display the results
export Export an HDFS directory to a database table
help List available commands
import Import a table from a database to HDFS
import-all-tables Import tables from a database to HDFS
import-mainframe Import datasets from a mainframe server to HDFS
job Work with saved jobs
list-databases List available databases on a server
list-tables List available tables in a database
merge Merge results of incremental imports
metastore Run a standalone Sqoop metastore
version Display version information
See ‘sqoop help COMMAND‘ for information on a specific command.
# 查看sqoop版本
[[email protected] ~]$ sqoop version
Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/hadoop/app/sqoop-1.4.6-cdh5.7.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
18/07/03 16:23:30 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.7.0
Sqoop 1.4.6-cdh5.7.0
git commit id
Compiled by jenkins on Wed Mar 23 11:30:51 PDT 2016
# 这里的警告是因为我没有配置hbase，zookeeper，HCatalog原文地址：http://blog.51cto.com/10814168/2135654
时间： 2024-10-27 14:29:58