Sqoop的安装和验证

　　Sqoop是一个用来完成Hadoop和关系型数据库中的数据相互转移的工具，它可以将关系型数据库中的数据导入到Hadoop的HDFS中，也可以将HDFS的数据导入到关系型数据库中。

　　Kafka是一个开源的分布式消息订阅系统

　　一、Sqoop的安装

　　1.http://www-eu.apache.org/dist/sqoop/1.4.7/下载sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz并解压到/home/jun下

[[email protected] sqoop-1.4.7.bin__hadoop-2.6.0]$ ls -l
total 2020
drwxr-xr-x. 2 jun jun    4096 Dec 19  2017 bin
-rw-rw-r--. 1 jun jun   55089 Dec 19  2017 build.xml
-rw-rw-r--. 1 jun jun   47426 Dec 19  2017 CHANGELOG.txt
-rw-rw-r--. 1 jun jun    9880 Dec 19  2017 COMPILING.txt
drwxr-xr-x. 2 jun jun     150 Dec 19  2017 conf
drwxr-xr-x. 5 jun jun     169 Dec 19  2017 docs
drwxr-xr-x. 2 jun jun      96 Dec 19  2017 ivy
-rw-rw-r--. 1 jun jun   11163 Dec 19  2017 ivy.xml
drwxr-xr-x. 2 jun jun    4096 Dec 19  2017 lib
-rw-rw-r--. 1 jun jun   15419 Dec 19  2017 LICENSE.txt
-rw-rw-r--. 1 jun jun     505 Dec 19  2017 NOTICE.txt
-rw-rw-r--. 1 jun jun   18772 Dec 19  2017 pom-old.xml
-rw-rw-r--. 1 jun jun    1096 Dec 19  2017 README.txt
-rw-rw-r--. 1 jun jun 1108073 Dec 19  2017 sqoop-1.4.7.jar
-rw-rw-r--. 1 jun jun    6554 Dec 19  2017 sqoop-patch-review.py
-rw-rw-r--. 1 jun jun  765184 Dec 19  2017 sqoop-test-1.4.7.jar
drwxr-xr-x. 7 jun jun      73 Dec 19  2017 src
drwxr-xr-x. 4 jun jun     114 Dec 19  2017 testdata

　　2.配置MySQL连接器

[[email protected] sqoop-1.4.7.bin__hadoop-2.6.0]$ cp /home/jun/Resources/mysql-connector-java-5.1.46/mysql-connector-java-5.1.46.jar   /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/lib/

　　3.配置Sqoop环境变量

　　编辑配置文件

[[email protected] lib]$ cd /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/conf/
[[email protected] conf]$ ls
oraoop-site-template.xml  sqoop-env-template.cmd  sqoop-env-template.sh  sqoop-site-template.xml  sqoop-site.xml
[[email protected] conf]$ cp sqoop-env-template.sh sqoop-env.sh
[[email protected] conf]$ gedit sqoop-env.sh

　　增加下面的配置

#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/home/jun/hadoop

#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/home/jun/hadoop

#set the path to where bin/hbase is available
export HBASE_HOME=/home/jun/hbase-1.2.6.1

#Set the path to where bin/hive is available
export HIVE_HOME=/home/jun/apache-hive-2.3.3-bin

#Set the path for where zookeper config dir is
export ZOOCFGDIR=/usr/local/zk

　　4.配置linux环境变量

#sqoop
export SQOOP_HOME=/home/jun/sqoop-1.4.7.bin__hadoop-2.6.0
export PATH=$PATH:$SQOOP_HOME/bin

　　5.启动Sqoop，如果出现下面的内容就说明安装成功

[[email protected] ~]$ sqoop-help
Warning: /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
18/07/23 15:56:36 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/jun/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/jun/hbase-1.2.6.1/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
usage: sqoop COMMAND [ARGS]

Available commands:
  codegen            Generate code to interact with database records
  create-hive-table  Import a table definition into Hive
  eval               Evaluate a SQL statement and display the results
  export             Export an HDFS directory to a database table
  help               List available commands
  import             Import a table from a database to HDFS
  import-all-tables  Import tables from a database to HDFS
  import-mainframe   Import datasets from a mainframe server to HDFS
  job                Work with saved jobs
  list-databases     List available databases on a server
  list-tables        List available tables in a database
  merge              Merge results of incremental imports
  metastore          Run a standalone Sqoop metastore
  version            Display version information

See ‘sqoop help COMMAND‘ for information on a specific command.

　　6.测试与MySQL的连接

　　（1）列出MySQL的所有数据库

[[email protected] ~]$ sqoop-list-databases --connect jdbc:mysql://localhost:3306 --username root -P
Warning: /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
18/07/23 16:03:01 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/jun/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/jun/hbase-1.2.6.1/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Enter password:
18/07/23 16:03:05 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Mon Jul 23 16:03:05 CST 2018 WARN: Establishing SSL connection without server‘s identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn‘t set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to ‘false‘. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
information_schema
hive_db
mysql
performance_schema
sys

　　（2）列出数据库下的所有数据表

[[email protected] ~]$ sqoop-list-tables --connect jdbc:mysql://localhost:3306/mysql --username root -P
Warning: /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
18/07/23 16:06:06 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/jun/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/jun/hbase-1.2.6.1/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Enter password:
18/07/23 16:06:09 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Mon Jul 23 16:06:09 CST 2018 WARN: Establishing SSL connection without server‘s identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn‘t set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to ‘false‘. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
columns_priv
db
engine_cost
event
func
general_log
gtid_executed
help_category
help_keyword
help_relation
help_topic
innodb_index_stats
innodb_table_stats
ndb_binlog_index
plugin
proc
procs_priv
proxies_priv
server_cost
servers
slave_master_info
slave_relay_log_info
slave_worker_info
slow_log
tables_priv
time_zone
time_zone_leap_second
time_zone_name
time_zone_transition
time_zone_transition_type
user

　　（3）执行MySQL的查询语句

[[email protected] ~]$ sqoop-eval --connect jdbc:mysql://localhost:3306/mysql --username root -P --query "select * from plugin"
Warning: /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/jun/sqoop-1.4.7.bin__hadoop-2.6.0/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
18/07/23 16:09:33 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/jun/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/jun/hbase-1.2.6.1/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Enter password:
18/07/23 16:09:36 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
Mon Jul 23 16:09:37 CST 2018 WARN: Establishing SSL connection without server‘s identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn‘t set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to ‘false‘. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
-----------------------------------------------
| name                 | dl                   |
-----------------------------------------------
| validate_password    | validate_password.so |
-----------------------------------------------

原文地址：https://www.cnblogs.com/BigJunOba/p/9355201.html

时间： 2024-11-05 14:38:28

Sqoop的安装和验证的相关文章

大数据学习之十五——sqoop的安装和使用

1.概念了解 sqoop主要用于hadoop与传统的数据库(mysql.postgresql...)间进行数据的传递,可以将一个关系型数据库(例如:MYSQL,Oracle,Postgrep等)中的数据导到hadoop的HDFS中,也可以将HDFS的数据导进到关系型数据库中. 2.sqoop的安装 (1)将压缩包sqoop-1.4.6.bin__hadoop-2.0.4-alpha.jar放在Linux的路径下,并修改配置文件/etc/profile export SQOOP_HOME=该压缩包

【sqoop】安装配置测试sqoop1

1.1.1 下载sqoop1:sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz 1.1.2 解压并查看目录: [[email protected] ~]$ tar -zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz --解压 [[email protected] ~]$ cd sqoop-1.4.7.bin__hadoop-2.6.0 [[email protected] sqoop-1.4.7.bin__hadoop-2.6.0]$ l

自动化测试环境搭建[Selenium-ide安装及验证]

[Selenium-ide安装及验证] 1.安装Firefox 2.下载selenium-ide 我下载的版本是selenium-ide-2.8.0.xpi selenium-ide安装:下载的selenium-ide-2.8.0.xpi拖到打开的Firefox,点击安装,根据提示重启Firefox浏览器 重启后,在菜单栏就可以看到Selenium IDE(没看到的到定制里面拖拽出来) 3.Selenium server的安装 下载selenium-server-standalone-2.45.

hadoop伪分布下的sqoop基本安装配置

1.环境工具版本介绍 centos6.4(Final) jdk-7u60-linux-i586.gz hadoop-1.1.2.tar.gz sqoop-1.4.3.bin__hadoop-1.0.0.tar.gz mysql-5.6.11.tar.gz 2.安装centos 参照网上Ultra的使用制作了U盘启动,直接格式化安装系统,具体做法网上资料很多,但注意最好不要在安装时改主机名称,也最好不要利用图形化界面添加用户,因为我出过问题重做了系统,这些terminal中都能完成的 3.安装jd

Sqoop环境安装

环境下载 首先将下载的 sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz放到 /usr/hadoop/目录下(该目录可以自定义,一般为Hadoop集群安装目录),然后对安装包解压.修改文件名和修改用户权限. [[email protected] /]$ cd /usr/hadoop/ [[email protected] hadoop]$ ls flume hadoop-2.6.0 [[email protected] hadoop]$ rz //上传安装包

Hive/Hbase/Sqoop的安装教程

Hive/Hbase/Sqoop的安装教程 HIVE INSTALL 1.下载安装包:https://mirrors.tuna.tsinghua.edu.cn/apache/hive/hive-2.3.3/2.上传到Linux指定目录,解压: mkdir hive mv apache-hive-2.3.3-bin.tar.gz hive tar -zxvf apache-hive-2.3.3-bin.tar.gz mv apache-hive-2.3.3-bin apache-hive-2.3.

Nginx 安装以及验证教程

为了证明今天的我还活着,啊哈哈... 那就来篇Nginx安装以及验证教程. 一.标题 二.按步骤来我都总结好了 三.接着按步骤 四.按步骤 五.按步骤 六.就完了..嘻嘻,就是这么简单...嘻嘻嘻嘻啊哈哈哈啊哈哈哈. 原文地址:https://www.cnblogs.com/money131/p/10849656.html

sqoop配置安装以及导入

安装sqoop的前提是已经具备java和hadoop的环境 1.上传并解压 (要导mysql的数据)得加入mysql的jdbc驱动包 接下来验证启动 Sqoop的数据导入 "导入工具"导入单个表从RDBMS到HDFS.表中的每一行被视为HDFS的记录.所有记录都存储为文本文件的文本数据(或者Avro.sequence文件等二进制数据) 语法 下面的语法用于将数据导入HDFS. $ sqoop import (generic-args) (import-args) 示例 表数据 在mys

sqoop的安装和使用

在sqoop使用前,应先安装好hive和zookeeper,还要在一台虚拟机里安装好mysql 1.先将zookeeper启动:zkServer.sh start,集群启动起来:start-all.sh 2.启动mysql:service mysql  start 然后进入mysql的客户端: 3.在windows下安装mysql的客户端(可在西西软件园下载) 下载完成后,进入客户端 4.接下来安装sqoop-1.4.6 具体代码可参考: tar -zxvf sqoop-1.4.6.bin__h