环境下载
首先将下载的 sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz放到 /usr/hadoop/目录下（该目录可以自定义，一般为Hadoop集群安装目录），然后对安装包解压、修改文件名和修改用户权限。
[[email protected] /]$ cd /usr/hadoop/ [[email protected] hadoop]$ ls flume hadoop-2.6.0 [[email protected] hadoop]$ rz //上传安装包 [[email protected] hadoop]$ ls flume hadoop-2.6.0 sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz [[email protected] hadoop]$ tar -zxf sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz //解压 [[email protected] hadoop]$ mv sqoop-1.4.6.bin__hadoop-2.0.4-alpha sqoop //修改安装文件目录 [[email protected] hadoop]$ ls flume hadoop-2.6.0 sqoop sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz [[email protected] hadoop]$ rm -f sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz //删除安装包 [[email protected] hadoop]$ ls flume hadoop-2.6.0 sqoop
环境配置
1.配置hadoop版本号
切换到/sqoop/conf 目录下，执行以下命令修改对应Hadoop版本号为自己所安装的集群的版本号。
[[email protected] hadoop]$ cd sqoop/ [[email protected] sqoop]$ ls bin COMPILING.txt ivy LICENSE.txt README.txt sqoop-test-1.4.6.jar build.xml conf ivy.xml NOTICE.txt sqoop-1.4.6.jar src CHANGELOG.txt docs lib pom-old.xml sqoop-patch-review.py testdata [[email protected] sqoop]$ vi build.xml
修改为：
2.配置sqoop-env.sh文件
执行以下命令生成sqoop-env.sh配置文件。
[[email protected] sqoop]$ cd conf/ [[email protected] conf]$ ls oraoop-site-template.xml sqoop-env-template.sh sqoop-site.xml sqoop-env-template.cmd sqoop-site-template.xml [[email protected] conf]$ mv sqoop-env-template.sh sqoop-env.sh
然后使用 vi sqoop-env.sh 命令，打开文件添加如下内容。
如果数据读取不涉及hbase和hive，那么相关hbase和hive的配置可以不加；如果集群有独立的zookeeper集群，那么配置zookeeper，反之，不用配置。
3.配置环境变量
[[email protected] conf]$ su Password: [[email protected] conf]# vim /etc/profile //追加以下内容 #sqoop export SQOOP_HOME=/usr/hadoop/sqoop export PATH=$PATH:$SQOOP_HOME/bin [[email protected] conf]# source /etc/profile
4.上传jar包
将相关的驱动 jar 包拷贝到 sqoop/lib 目录下。数据库驱动 jar 包需要导入，这里我们使用的是 mysql 数据库，所以需要导入mysql-connector-java-5.1.32.jar包。
5.确定用户组
需要确定sqoop所属用户组是否为hadoop
[[email protected] hadoop]$ ll total 12 drwxrwxr-x. 9 hadoop hadoop 4096 Jun 19 15:30 flume drwxr-xr-x. 12 hadoop hadoop 4096 Jun 13 21:14 hadoop-2.6.0 drwxr-xr-x. 9 hadoop hadoop 4096 Jun 21 09:56 sqoop
如果不是请通过以下语句修改：
chown –R hadoop:hadoop sqoop
测试运行
//首先启动hadoop集群[[email protected] sqoop]$ cd ../hadoop-2.6.0/ [[email protected] hadoop-2.6.0]$ sbin/start-all.sh This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh Starting namenodes on [centpy] centpy: starting namenode, logging to /usr/hadoop/hadoop-2.6.0/logs/hadoop-hadoop-namenode-centpy.out centpy: starting datanode, logging to /usr/hadoop/hadoop-2.6.0/logs/hadoop-hadoop-datanode-centpy.out Starting secondary namenodes [0.0.0.0] 0.0.0.0: starting secondarynamenode, logging to /usr/hadoop/hadoop-2.6.0/logs/hadoop-hadoop-secondarynamenode-centpy.out starting yarn daemons starting resourcemanager, logging to /usr/hadoop/hadoop-2.6.0/logs/yarn-hadoop-resourcemanager-centpy.out centpy: starting nodemanager, logging to /usr/hadoop/hadoop-2.6.0/logs/yarn-hadoop-nodemanager-centpy.out [[email protected] hadoop-2.6.0]$ jps 2624 SecondaryNameNode 3088 Jps 2769 ResourceManager 2340 NameNode 2870 NodeManager 2439 DataNode//测试sqoop [[email protected] hadoop-2.6.0]$ cd ../sqoop/ [[email protected] sqoop]$ sqoop list-databases --connect jdbc:mysql://202.193.60.117/dataweb --username root --password 20134997 Warning: /usr/hadoop/sqoop/../hbase does not exist! HBase imports will fail. Please set $HBASE_HOME to the root of your HBase installation. Warning: /usr/hadoop/sqoop/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /usr/hadoop/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. Warning: /usr/hadoop/sqoop/../zookeeper does not exist! Accumulo imports will fail. Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation. 18/06/21 10:47:00 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6 18/06/21 10:47:00 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 18/06/21 10:47:01 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. information_schema dataweb mysql performance_schema test
sqoop 命令执行成功，代表安装成功。
sqoop list-databases --connect jdbc:mysql://202.193.60.117/dataweb --username root --password 20134997
该语句中使用我们上传的mysql-connector-java-5.1.32.jar包链接本地数据库，其中ip地址可以通过命令行中输入ipconfig进行查看。
执行该命令之前要先启动数据库服务，若出现2003错误时因为mysql服务没有成功启动，解决方案如下：
选中后右键开启服务即可。
到此，sqoop的安装和配置就结束了！
以上就是博主为大家介绍的这一板块的主要内容，这都是博主自己的学习过程，希望能给大家带来一定的指导作用，有用的还望大家点个支持，如果对你没用也望包涵，有错误烦请指出。如有期待可关注博主以第一时间获取更新哦，谢谢！
原文地址：https://www.cnblogs.com/zimo-jing/p/9208192.html
时间： 2024-10-05 04:39:34