HBase（三）HBase集群Shell操作

一、进入HBase命令行

在你安装的随意台服务器节点上，执行命令：hbase shell，会进入到你的 hbase shell 客 户端

[[email protected] ~]$ hbase shell
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/module/hadoop-2.7.6/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
HBase Shell; enter ‘help<RETURN>‘ for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 1.2.6, rUnknown, Mon May 29 02:25:32 CDT 2017
hbase(main):001:0>

说明，先看一下提示。其实是不是有一句很重要的话：

HBase Shell; enter ‘help<RETURN>‘ for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell

讲述了怎么获得帮助，怎么退出客户端

help 获取帮助

　　help：获取所有命令提示

　　help "dml" ：获取一组命令的提示

　　help "put" ：获取一个单独命令的提示帮助

exit 退出 hbase shell 客户端

二、HBase表的操作

这些是关于HBase在表中操作的命令。

  • create: 创建一个表。
  • list: 列出HBase的所有表。
  • disable: 禁用表。
  • is_disabled: 验证表是否被禁用。
  • enable: 启用一个表。
  • is_enabled: 验证表是否已启用。
  • describe: 提供了一个表的描述。
  • alter: 改变一个表。
  • exists: 验证表是否存在。
  • drop: 从HBase中删除表。
  • drop_all: 丢弃在命令中给出匹配“regex”的表。
  • Java Admin API: 在此之前所有的上述命令，Java提供了一个通过API编程来管理实现DDL功能。在这个org.apache.hadoop.hbase.client包中有HBaseAdmin和HTableDescriptor 这两个重要的类提供DDL功能。

关于表的操作包括（创建create，查看表列表list。查看表的详细信息desc，删除表drop，清空表truncate，修改表的定义alter）

1、创建表create

可以输入以下命令进行查看帮助命令

hbase(main):001:0>  help ‘create‘

Creates a table. Pass a table name, and a set of column family
specifications (at least one), and, optionally, table configuration.
Column specification can be a simple string (name), or a dictionary
(dictionaries are described below in main help output), necessarily
including NAME attribute.
Examples:

Create a table with namespace=ns1 and table qualifier=t1
  hbase> create ‘ns1:t1‘, {NAME => ‘f1‘, VERSIONS => 5}

Create a table with namespace=default and table qualifier=t1
  hbase> create ‘t1‘, {NAME => ‘f1‘}, {NAME => ‘f2‘}, {NAME => ‘f3‘}
  hbase> # The above in shorthand would be the following:
  hbase> create ‘t1‘, ‘f1‘, ‘f2‘, ‘f3‘
  hbase> create ‘t1‘, {NAME => ‘f1‘, VERSIONS => 1, TTL => 2592000, BLOCKCACHE => true}
  hbase> create ‘t1‘, {NAME => ‘f1‘, CONFIGURATION => {‘hbase.hstore.blockingStoreFiles‘ => ‘10‘}}

Table configuration options can be put at the end.
Examples:

  hbase> create ‘ns1:t1‘, ‘f1‘, SPLITS => [‘10‘, ‘20‘, ‘30‘, ‘40‘]
  hbase> create ‘t1‘, ‘f1‘, SPLITS => [‘10‘, ‘20‘, ‘30‘, ‘40‘]
  hbase> create ‘t1‘, ‘f1‘, SPLITS_FILE => ‘splits.txt‘, OWNER => ‘johndoe‘
  hbase> create ‘t1‘, {NAME => ‘f1‘, VERSIONS => 5}, METADATA => { ‘mykey‘ => ‘myvalue‘ }
  hbase> # Optionally pre-split the table into NUMREGIONS, using
  hbase> # SPLITALGO ("HexStringSplit", "UniformSplit" or classname)
  hbase> create ‘t1‘, ‘f1‘, {NUMREGIONS => 15, SPLITALGO => ‘HexStringSplit‘}
  hbase> create ‘t1‘, ‘f1‘, {NUMREGIONS => 15, SPLITALGO => ‘HexStringSplit‘, REGION_REPLICATION => 2, CONFIGURATION => {‘hbase.hregion.scan.loadColumnFamiliesOnDemand
‘ => ‘true‘}}  hbase> create ‘t1‘, {NAME => ‘f1‘, DFS_REPLICATION => 1}

You can also keep around a reference to the created table:

  hbase> t1 = create ‘t1‘, ‘f1‘

Which gives you a reference to the table named ‘t1‘, on which you can then
call methods.
hbase(main):002 >

可以看到其中一条提示

hbase> create ‘t1‘, {NAME => ‘f1‘}, {NAME => ‘f2‘}, {NAME => ‘f3‘}

其中t1是表名，f1,f2,f3是列簇的名，如：

hbase(main):002:0> create ‘myHbase‘,{NAME => ‘myCard‘,VERSIONS => 5}
0 row(s) in 3.1270 seconds

=> Hbase::Table - myHbase
hbase(main):003:0>

创建了一个名为myHbase的表，表里面有1个列簇，名为myCard，保留5个版本信息

2、查看表列表list

可以输入以下命令进行查看帮助命令

hbase(main):003:0> help ‘list‘
List all tables in hbase. Optional regular expression parameter could
be used to filter the output. Examples:

  hbase> list
  hbase> list ‘abc.*‘
  hbase> list ‘ns:abc.*‘
  hbase> list ‘ns:.*‘
hbase(main):004:0>

直接输入list进行查看

hbase(main):004:0> list
TABLE
myHbase
1 row(s) in 0.0650 seconds

=> ["myHbase"]
hbase(main):005:0>

只有一条结果，就是刚刚创建的表myHbase

3、查看表详细信息desc

一个大括号，就相当于一个列簇。

hbase(main):006:0> desc ‘myHbase‘
Table myHbase is ENABLED
myHbase
COLUMN FAMILIES DESCRIPTION
{NAME => ‘myCard‘, BLOOMFILTER => ‘ROW‘, VERSIONS => ‘5‘, IN_MEMORY => ‘false‘, KEEP_DELETED_CELLS => ‘FALSE‘, D
ATA_BLOCK_ENCODING => ‘NONE‘, TTL => ‘FOREVER‘, COMPRESSION => ‘NONE‘, MIN_VERSIONS => ‘0‘, BLOCKCACHE => ‘true‘
, BLOCKSIZE => ‘65536‘, REPLICATION_SCOPE => ‘0‘}
1 row(s) in 0.2160 seconds

hbase(main):007:0>

4、修改表定义alter

添加一个列簇

hbase(main):007:0> alter ‘myHbase‘, NAME => ‘myInfo‘
Updating all regions with the new schema...
1/1 regions updated.
Done.
0 row(s) in 2.0690 seconds

hbase(main):008:0> desc ‘myHbase‘
Table myHbase is ENABLED 
myHbase 
COLUMN FAMILIES DESCRIPTION 
{NAME => ‘myCard‘, BLOOMFILTER => ‘ROW‘, VERSIONS => ‘5‘, IN_MEMORY => ‘false‘, KEEP_DELETED_CELLS => ‘FALSE‘, D
ATA_BLOCK_ENCODING => ‘NONE‘, TTL => ‘FOREVER‘, COMPRESSION => ‘NONE‘, MIN_VERSIONS => ‘0‘, BLOCKCACHE => ‘true‘
, BLOCKSIZE => ‘65536‘, REPLICATION_SCOPE => ‘0‘} 
{NAME => ‘myInfo‘, BLOOMFILTER => ‘ROW‘, VERSIONS => ‘1‘, IN_MEMORY => ‘false‘, KEEP_DELETED_CELLS => ‘FALSE‘, D
ATA_BLOCK_ENCODING => ‘NONE‘, TTL => ‘FOREVER‘, COMPRESSION => ‘NONE‘, MIN_VERSIONS => ‘0‘, BLOCKCACHE => ‘true‘
, BLOCKSIZE => ‘65536‘, REPLICATION_SCOPE => ‘0‘} 
2 row(s) in 0.0420 seconds

hbase(main):009:0>

删除一个列簇

hbase(main):009:0> alter ‘myHbase‘, NAME => ‘myCard‘, METHOD => ‘delete‘
Updating all regions with the new schema...
1/1 regions updated.
Done.
0 row(s) in 2.1920 seconds

hbase(main):010:0> desc ‘myHbase‘
Table myHbase is ENABLED
myHbase
COLUMN FAMILIES DESCRIPTION
{NAME => ‘myInfo‘, BLOOMFILTER => ‘ROW‘, VERSIONS => ‘1‘, IN_MEMORY => ‘false‘, KEEP_DELETED_CELLS => ‘FALSE‘, D
ATA_BLOCK_ENCODING => ‘NONE‘, TTL => ‘FOREVER‘, COMPRESSION => ‘NONE‘, MIN_VERSIONS => ‘0‘, BLOCKCACHE => ‘true‘
, BLOCKSIZE => ‘65536‘, REPLICATION_SCOPE => ‘0‘}
1 row(s) in 0.0290 seconds

hbase(main):011:0>

删除一个列簇也可以执行以下命令

alter ‘myHbase‘, ‘delete‘ => ‘myCard‘

添加列簇hehe同时删除列簇myInfo

hbase(main):011:0> alter ‘myHbase‘, {NAME => ‘hehe‘}, {NAME => ‘myInfo‘, METHOD => ‘delete‘}
Updating all regions with the new schema...
1/1 regions updated.
Done.
Updating all regions with the new schema...
1/1 regions updated.
Done.
0 row(s) in 3.8260 seconds

hbase(main):012:0> desc ‘myHbase‘
Table myHbase is ENABLED
myHbase
COLUMN FAMILIES DESCRIPTION
{NAME => ‘hehe‘, BLOOMFILTER => ‘ROW‘, VERSIONS => ‘1‘, IN_MEMORY => ‘false‘, KEEP_DELETED_CELLS => ‘FALSE‘, DAT
A_BLOCK_ENCODING => ‘NONE‘, TTL => ‘FOREVER‘, COMPRESSION => ‘NONE‘, MIN_VERSIONS => ‘0‘, BLOCKCACHE => ‘true‘,
BLOCKSIZE => ‘65536‘, REPLICATION_SCOPE => ‘0‘}
1 row(s) in 0.0410 seconds

hbase(main):013:0>

5、清空表truncate

hbase(main):013:0> truncate ‘myHbase‘
Truncating ‘myHbase‘ table (it may take a while):
 - Disabling table...
 - Truncating table...
0 row(s) in 3.6760 seconds

hbase(main):014:0>

6、删除表drop

hbase(main):014:0> drop ‘myHbase‘

ERROR: Table myHbase is enabled. Disable it first.

Here is some help for this command:
Drop the named table. Table must first be disabled:
  hbase> drop ‘t1‘
  hbase> drop ‘ns1:t1‘

hbase(main):015:0>

直接删除表会报错，根据提示需要先停用表

hbase(main):015:0> disable ‘myHbase‘
0 row(s) in 2.2620 seconds

hbase(main):016:0> drop ‘myHbase‘
0 row(s) in 1.2970 seconds

hbase(main):017:0> list
TABLE
0 row(s) in 0.0110 seconds

=> []
hbase(main):018:0>

三、HBase表中数据的操作

  • put: 把指定列在指定的行中单元格的值在一个特定的表。
  • get: 取行或单元格的内容。
  • delete: 删除表中的单元格值。
  • deleteall: 删除给定行的所有单元格。
  • scan: 扫描并返回表数据。
  • count: 计数并返回表中的行的数目。
  • truncate: 禁用，删除和重新创建一个指定的表。
  • Java client API: 在此之前所有上述命令，Java提供了一个客户端API来实现DML功能，CRUD（创建检索更新删除）操作更多的是通过编程，在org.apache.hadoop.hbase.client包下。 在此包HTable 的 Put和Get是重要的类。

关于数据的操作（增put，删delete，查get + scan,  改==变相的增加）

创建 user 表，包含 info、data 两个列簇

hbase(main):018:0> create ‘user_info‘,{NAME=>‘base_info‘,VERSIONS=>3 },{NAME=>‘extra_info‘,VERSIONS=>1 }
0 row(s) in 4.2670 seconds

=> Hbase::Table - user_info
hbase(main):019:0>

1、增put

查看帮助，需要传入表名，rowkey，列簇名、值等

hbase(main):019:0> help ‘put‘
Put a cell ‘value‘ at specified table/row/column and optionally
timestamp coordinates.  To put a cell value into table ‘ns1:t1‘ or ‘t1‘
at row ‘r1‘ under column ‘c1‘ marked with the time ‘ts1‘, do:

  hbase> put ‘ns1:t1‘, ‘r1‘, ‘c1‘, ‘value‘
  hbase> put ‘t1‘, ‘r1‘, ‘c1‘, ‘value‘
  hbase> put ‘t1‘, ‘r1‘, ‘c1‘, ‘value‘, ts1
  hbase> put ‘t1‘, ‘r1‘, ‘c1‘, ‘value‘, {ATTRIBUTES=>{‘mykey‘=>‘myvalue‘}}
  hbase> put ‘t1‘, ‘r1‘, ‘c1‘, ‘value‘, ts1, {ATTRIBUTES=>{‘mykey‘=>‘myvalue‘}}
  hbase> put ‘t1‘, ‘r1‘, ‘c1‘, ‘value‘, ts1, {VISIBILITY=>‘PRIVATE|SECRET‘}

The same commands also can be run on a table reference. Suppose you had a reference
t to table ‘t1‘, the corresponding command would be:

  hbase> t.put ‘r1‘, ‘c1‘, ‘value‘, ts1, {ATTRIBUTES=>{‘mykey‘=>‘myvalue‘}}
hbase(main):020:0>

向 user 表中插入信息，row key 为 user0001，列簇 base_info 中添加 name 列标示符，值为 zhangsan1

hbase(main):020:0> put ‘user_info‘, ‘user0001‘, ‘base_info:name‘, ‘zhangsan1‘
0 row(s) in 0.2900 seconds

hbase(main):021:0>

此处可以多添加几条数据

put ‘user_info‘, ‘zhangsan_20150701_0001‘, ‘base_info:name‘, ‘zhangsan1‘
put ‘user_info‘, ‘zhangsan_20150701_0002‘, ‘base_info:name‘, ‘zhangsan2‘
put ‘user_info‘, ‘zhangsan_20150701_0003‘, ‘base_info:name‘, ‘zhangsan3‘
put ‘user_info‘, ‘zhangsan_20150701_0004‘, ‘base_info:name‘, ‘zhangsan4‘
put ‘user_info‘, ‘zhangsan_20150701_0005‘, ‘base_info:name‘, ‘zhangsan5‘
put ‘user_info‘, ‘zhangsan_20150701_0006‘, ‘base_info:name‘, ‘zhangsan6‘
put ‘user_info‘, ‘zhangsan_20150701_0007‘, ‘base_info:name‘, ‘zhangsan7‘
put ‘user_info‘, ‘zhangsan_20150701_0008‘, ‘base_info:name‘, ‘zhangsan8‘

put ‘user_info‘, ‘zhangsan_20150701_0001‘, ‘base_info:age‘, ‘21‘
put ‘user_info‘, ‘zhangsan_20150701_0002‘, ‘base_info:age‘, ‘22‘
put ‘user_info‘, ‘zhangsan_20150701_0003‘, ‘base_info:age‘, ‘23‘
put ‘user_info‘, ‘zhangsan_20150701_0004‘, ‘base_info:age‘, ‘24‘
put ‘user_info‘, ‘zhangsan_20150701_0005‘, ‘base_info:age‘, ‘25‘
put ‘user_info‘, ‘zhangsan_20150701_0006‘, ‘base_info:age‘, ‘26‘
put ‘user_info‘, ‘zhangsan_20150701_0007‘, ‘base_info:age‘, ‘27‘
put ‘user_info‘, ‘zhangsan_20150701_0008‘, ‘base_info:age‘, ‘28‘

put ‘user_info‘, ‘zhangsan_20150701_0001‘, ‘extra_info:Hobbies‘, ‘music‘
put ‘user_info‘, ‘zhangsan_20150701_0002‘, ‘extra_info:Hobbies‘, ‘sport‘
put ‘user_info‘, ‘zhangsan_20150701_0003‘, ‘extra_info:Hobbies‘, ‘music‘
put ‘user_info‘, ‘zhangsan_20150701_0004‘, ‘extra_info:Hobbies‘, ‘sport‘
put ‘user_info‘, ‘zhangsan_20150701_0005‘, ‘extra_info:Hobbies‘, ‘music‘
put ‘user_info‘, ‘zhangsan_20150701_0006‘, ‘extra_info:Hobbies‘, ‘sport‘
put ‘user_info‘, ‘zhangsan_20150701_0007‘, ‘extra_info:Hobbies‘, ‘music‘

put ‘user_info‘, ‘baiyc_20150716_0001‘, ‘base_info:name‘, ‘baiyc1‘
put ‘user_info‘, ‘baiyc_20150716_0002‘, ‘base_info:name‘, ‘baiyc2‘
put ‘user_info‘, ‘baiyc_20150716_0003‘, ‘base_info:name‘, ‘baiyc3‘
put ‘user_info‘, ‘baiyc_20150716_0004‘, ‘base_info:name‘, ‘baiyc4‘
put ‘user_info‘, ‘baiyc_20150716_0005‘, ‘base_info:name‘, ‘baiyc5‘
put ‘user_info‘, ‘baiyc_20150716_0006‘, ‘base_info:name‘, ‘baiyc6‘
put ‘user_info‘, ‘baiyc_20150716_0007‘, ‘base_info:name‘, ‘baiyc7‘
put ‘user_info‘, ‘baiyc_20150716_0008‘, ‘base_info:name‘, ‘baiyc8‘

put ‘user_info‘, ‘baiyc_20150716_0001‘, ‘base_info:age‘, ‘21‘
put ‘user_info‘, ‘baiyc_20150716_0002‘, ‘base_info:age‘, ‘22‘
put ‘user_info‘, ‘baiyc_20150716_0003‘, ‘base_info:age‘, ‘23‘
put ‘user_info‘, ‘baiyc_20150716_0004‘, ‘base_info:age‘, ‘24‘
put ‘user_info‘, ‘baiyc_20150716_0005‘, ‘base_info:age‘, ‘25‘
put ‘user_info‘, ‘baiyc_20150716_0006‘, ‘base_info:age‘, ‘26‘
put ‘user_info‘, ‘baiyc_20150716_0007‘, ‘base_info:age‘, ‘27‘
put ‘user_info‘, ‘baiyc_20150716_0008‘, ‘base_info:age‘, ‘28‘

put ‘user_info‘, ‘baiyc_20150716_0001‘, ‘extra_info:Hobbies‘, ‘music‘
put ‘user_info‘, ‘baiyc_20150716_0002‘, ‘extra_info:Hobbies‘, ‘sport‘
put ‘user_info‘, ‘baiyc_20150716_0003‘, ‘extra_info:Hobbies‘, ‘music‘
put ‘user_info‘, ‘baiyc_20150716_0004‘, ‘extra_info:Hobbies‘, ‘sport‘
put ‘user_info‘, ‘baiyc_20150716_0005‘, ‘extra_info:Hobbies‘, ‘music‘
put ‘user_info‘, ‘baiyc_20150716_0006‘, ‘extra_info:Hobbies‘, ‘sport‘
put ‘user_info‘, ‘baiyc_20150716_0007‘, ‘extra_info:Hobbies‘, ‘music‘
put ‘user_info‘, ‘baiyc_20150716_0008‘, ‘extra_info:Hobbies‘, ‘sport‘

2、查get + scan

获取 user 表中 row key 为 user0001 的所有信息

hbase(main):022:0> get ‘user_info‘, ‘user0001‘
COLUMN                        CELL
 base_info:name               timestamp=1522320801670, value=zhangsan1
1 row(s) in 0.1310 seconds

hbase(main):023:0>

获取user表中row key为rk0001，info列簇的所有信息

hbase(main):025:0> get ‘user_info‘, ‘rk0001‘, ‘base_info‘
COLUMN                        CELL
 base_info:name               timestamp=1522321247732, value=zhangsan
1 row(s) in 0.0320 seconds

hbase(main):026:0>

查询user_info表中的所有信息

hbase(main):026:0> scan ‘user_info‘
ROW                           COLUMN+CELL
 rk0001                       column=base_info:name, timestamp=1522321247732, value=zhangsan
 user0001                     column=base_info:name, timestamp=1522320801670, value=zhangsan1
2 row(s) in 0.0970 seconds

hbase(main):027:0>

查询user_info表中列簇为base_info的信息

hbase(main):027:0> scan ‘user_info‘, {COLUMNS => ‘base_info‘}
ROW                           COLUMN+CELL
 rk0001                       column=base_info:name, timestamp=1522321247732, value=zhangsan
 user0001                     column=base_info:name, timestamp=1522320801670, value=zhangsan1
2 row(s) in 0.0620 seconds

hbase(main):028:0>

3、删delete

删除user_info表row key为rk0001，列标示符为base_info:name的数据

hbase(main):028:0> delete ‘user_info‘, ‘rk0001‘, ‘base_info:name‘
0 row(s) in 0.0780 seconds

hbase(main):029:0> scan ‘user_info‘, {COLUMNS => ‘base_info‘}
ROW                           COLUMN+CELL
 user0001                     column=base_info:name, timestamp=1522320801670, value=zhangsan1
1 row(s) in 0.0530 seconds

hbase(main):030:0>

删除user_info表row key为rk0001的全部数据：

hbase(main):030:0> delete ‘user_info‘, ‘rk0001‘

四、HBase常用命令

HBase常用命令status, version, table_help和whoami。本章将介绍了这些命令。

1、status

命令返回包括在系统上运行的服务器的细节和系统的状态。它的语法如下：

hbase(main):009:0> status

如果执行这个命令，它会返回下面的输出

hbase(main):009:0> status
1 active master, 1 backup masters, 3 servers, 0 dead, 1.3333 average load

2、version

该命令返回HBase系统使用的版本。它的语法如下：

hbase(main):010:0> version

如果执行这个命令，它会返回下面的输出。

hbase(main):009:0> version
1.2.6, rUnknown, Mon May 29 02:25:32 CDT 2017

3、table_help

此命令将引导如何使用表引用的命令。下面给出的是使用这个命令的语法。

hbase(main):02:0> table_help

当使用此命令时，它显示帮助主题表相关的命令。

hbase(main):002:0> table_help
Help for table-reference commands.

You can either create a table via ‘create‘ and then manipulate the table via commands like ‘put‘, ‘get‘, etc.
See the standard help information for how to use each of these commands.

However, as of 0.96, you can also get a reference to a table, on which you can invoke commands.
For instance, you can get create a table and keep around a reference to it via:

   hbase> t = create ‘t‘, ‘cf‘

Or, if you have already created the table, you can get a reference to it:

   hbase> t = get_table ‘t‘

You can do things like call ‘put‘ on the table:

  hbase> t.put ‘r‘, ‘cf:q‘, ‘v‘

which puts a row ‘r‘ with column family ‘cf‘, qualifier ‘q‘ and value ‘v‘ into table t.

To read the data out, you can scan the table:

  hbase> t.scan

which will read all the rows in table ‘t‘.

Essentially, any command that takes a table name can also be done via table reference.
Other commands include things like: get, delete, deleteall,
get_all_columns, get_counter, count, incr. These functions, along with
the standard JRuby object methods are also available via tab completion.

For more information on how to use each of these commands, you can also just type:

   hbase> t.help ‘scan‘

which will output more information on how to use that command.

You can also do general admin actions directly on a table; things like enable, disable,
flush and drop just by typing:

   hbase> t.enable
   hbase> t.flush
   hbase> t.disable
   hbase> t.drop

Note that after dropping a table, your reference to it becomes useless and further usage
is undefined (and not recommended).

4、whoami

该命令返回HBase用户详细信息。如果执行这个命令，返回当前HBase用户，如下图所示

hbase(main):008:0> whoami
hbase(main):008:0> whoami
admin (auth:SIMPLE)
groups: admin
 

原文地址：https://www.cnblogs.com/frankdeng/p/9310204.html

时间： 2024-10-13 23:30:49

HBase（三）HBase集群Shell操作的相关文章

Zookeeper集群shell操作

1.zookeeper Zookeeper可以用来保证数据在Zookeeper集群之间的数据事务一致性 2.启动集群与检查 分别在每台机器上启动 ./zkServer.sh start 检查状态 ./zkServer.sh status 3.客户端启动shell命令行 zkCli.sh 4.shell命令:输入help得到所有的命令列表 我的个人网站:http://www.caicongyang.com : 我的CSDN博客地址: http://blog.csdn.net/caicongyang

基于HBase Hadoop 分布式集群环境下的MapReduce程序开发

HBase分布式集群环境搭建成功后,连续4.5天实验客户端Map/Reduce程序开发,这方面的代码网上多得是,写个测试代码非常容易,可是真正运行起来可说是历经挫折.下面就是我最终调通并让程序在集群上运行起来的一些经验教训. 一.首先说一下我的环境: 1,集群的环境配置请见这篇博文. 2,开发客户机环境:操作系统是CentOS6.5,JDK版本是1.7.0-60,开发工具是Eclipse(原始安装是从google的ADT网站下载的ADT专用开发环境,后来加装了Java企业开发的工具,启动Flas

基于Hadoop的数据分析综合管理平台之Hadoop、HBase完全分布式集群搭建

能够将热爱的技术应用于实际生活生产中,是做技术人员向往和乐之不疲的事. 现将前期手里面的一个项目做一个大致的总结,与大家一起分享.交流.进步.项目现在正在线上运行,项目名--基于Hadoop的数据分析综合管理平台. 项目流程整体比较清晰,爬取数据(txt文本)-->数据清洗-->文本模型训练-->文本分类-->热点话题发现-->报表"实时"展示,使用到的技术也是当今互联网公司常用的技术:Hadoop.Mahout.HBase.Spring Data Had

Java接口对Hadoop集群的操作

Java接口对Hadoop集群的操作 首先要有一个配置好的Hadoop集群 这里是我在SSM框架搭建的项目的测试类中实现的 一.windows下配置环境变量 下载文件并解压到C盘或者其他目录. 链接:http://pan.baidu.com/s/1jHHPElg 密码:aufd 配置环境变量 1.配置HADOOP_HOME 2.配置PATH 在PATH中添加 %HADOOP_HOME%\bin 1 3.配置HADOOP_USER_NAME 这是Hadoop集群的用户名 HADOOP_USER_N

LVS+NGINX+TOMCAT_集群实施操作记录.docx

LVS IP: Eth0:192.168.100.115 Eth1:192.168.100.215 Vi  /etc/init.d./lvs #!/bin/sh # # lvs      Start lvs # # chkconfig: 2345 08 92 # description:  Starts, stops and saves lvs # SNS_VIP=192.168.100.215 SNS_RIP1=192.168.100.114 SNS_RIP2=192.168.100.113

优化cdh集群性能-可在安装集群前操作002

优化cdh集群性能-可在安装集群前操作002//读完cdh官方文档后,可知的优化操作 可在<03搭建cdh 生产环境前的Linux 优化(涉及到Linux内存参数优化)>https://blog.51cto.com/12445535/2365948 这步同时操作 讲解了:提供了一些性能问题的解决方案,并介绍了配置最佳实践. 1.禁止tuned 服务 //是内存分配管理//关于tuned服务是什么?RHEL/CentOS 在 6.3 版本以后引入了一套新的系统调优工具 tuned/tuned-a

Hadoop及Zookeeper+HBase完全分布式集群部署

Hadoop及HBase集群部署 一. 集群环境 系统版本 虚拟机:内存 16G CPU 双核心 系统: CentOS-7 64位 系统下载地址: http://124.202.164.6/files/417500000AB646E7/mirrors.163.com/centos/7/isos/x86_64/CentOS-7-x86_64-DVD-1708.iso 软件版本 hadoop-2.8.1.tar.gz hbase-1.3.1-bin.tar.gz zookeeper-3.4.10.t

学习搭建Hadoop+HBase+ZooKeeper分布式集群环境

一.环境配置 由于集群至少需要三台服务器,我就拿上次做的MongoDB Master, Slave, Arbiter环境来做Hadoop集群.服务器还是ibmcloud 免费提供的.其中Arbiter在这里做的也是slave的角色. Hostname IP  Server Type Master 192.168.0.28 Centos6.2 Slave 192.168.0.29 Ubuntu14.04 Arbiter 192.168.0.30 Ubuntu14.04 配置三台机器的Master

HBase集成Zookeeper集群部署

大数据集群为了保证故障转移,一般通过zookeeper来整体协调管理,当节点数大于等于6个时推荐使用,接下来描述一下Hbase集群部署在zookeeper上的过程: 安装Hbase之前首先系统应该做通用的集群环境准备工作,这些是必须的: 1.集群中主机名必须正确配置,最好有实际意义:并且主机名都在hosts文件中对应主机IP,一一对应,不可缺少 这里集群有6台服务器:bigdata1,bigdata2,bigdata3,bigdata4,bigdata5,bigdata6 这里是3台主机,分别对