cdh权限测试示例

1. 准备测试数据

cat /tmp/events.csv
10.1.2.3,US,android,createNote
10.200.88.99,FR,windows,updateNote
10.1.2.3,US,android,updateNote
10.200.88.77,FR,ios,createNote
10.1.4.5,US,windows,updateTag

2. 创建用户
2.1. 创建系统用户
在集群所有节点创建系统用户并设置密码

useradd user1
passwd user1
useradd user2
passwd user2
useradd user3
passwd user3

2.2. 创建kerberos用户

kadmin.local -q "addprinc user1"
kadmin.local -q "addprinc user2"
kadmin.local -q "addprinc user3"

3. 创建数据库和表
3.1. 创建数据库
admin为sentry的超级管理员，该用户配置权限时已设置

kinit admin

通过beeline连接 hiveserver2，运行下面命令创建hive库的超级管理员角色, 并将该角色赋予admin组，使admin有操作hive库的权力

beeline -u "jdbc:hive2://vmw208:10000/;principal=hive/[email protected]"
create role admin_role;
GRANT ALL ON SERVER server1 TO ROLE admin_role;
GRANT ROLE admin_role TO GROUP admin;

创建两个测试数据库

create database db1;
create database db2;

3.2. 创建表
在两个测试数据库中各创建一张测试表,并导入测试数据

create table db1.table1 (
ip STRING, country STRING, client STRING, action STRING
) ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘,‘;

create table db2.table1 (
ip STRING, country STRING, client STRING, action STRING
) ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘,‘;
create table db2.table2 (
ip STRING, country STRING, client STRING, action STRING
) ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘,‘;

load data local inpath ‘/home/iie/events.csv‘ overwrite into table db1.table1;
load data local inpath ‘/home/iie/events.csv‘ overwrite into table db2.table1;
load data local inpath ‘/home/iie/events.csv‘ overwrite into table db2.table2;

　　

4. 赋予用户权限
4.1. 给user1赋予db1的所有权限

create role user1_role;
GRANT ALL ON DATABASE db1 TO ROLE user1_role;
GRANT ROLE user1_role TO GROUP user1;

4.2. 给user2赋予db2的所有权限

create role user2_role;
GRANT ALL ON DATABASE db2 TO ROLE user2_role;
GRANT ROLE user2_role TO GROUP user2;

4.3. 给user3赋予db2.table1的所有权限

create role user3_role;
use db2;
GRANT select ON table table1 TO ROLE user3_role;
GRANT ROLE user3_role TO GROUP user3;

5. 测试用户权限
5.1. Hive测试
5.1.1. admin用户拥有整个hive库的权限

kinit admin
beeline -u "jdbc:hive2://vmw208:10000/;principal=hive/[email protected]"
show databases;
5.1.2.	user1用户只具有db1和default的权限
kinit user1
beeline -u "jdbc:hive2://vmw208:10000/;principal=hive/[email protected]"
0: jdbc:hive2://vmw208:10000/> show databases;
+----------------+--+
| database_name |
+----------------+--+
| db1 |
| default |
+----------------+--+

5.1.3. user2用户只具有db2和default的权限

kinit user2
beeline -u "jdbc:hive2://vmw208:10000/;principal=hive/[email protected]"
0: jdbc:hive2://vmw208:10000/> show databases;
+----------------+--+
| database_name |
+----------------+--+
| db2 |
| default |
+----------------+--+

5.1.4. user3用户只具有db2.table1和default的权限

kinit user2
beeline -u "jdbc:hive2://vmw208:10000/;principal=hive/[email protected]"
0: jdbc:hive2://vmw208:10000/> show databases;
+----------------+--+
| database_name |
+----------------+--+
| db2 |
| default |
+----------------+--+
0: jdbc:hive2://node0a17:10000/> use db2;
0: jdbc:hive2://node0a17:10000/> show tables;
INFO : OK
+-----------+--+
| tab_name |
+-----------+--+
| table1 |
+-----------+--+

5.2. Hdfs测试
配置hdfs acl与sentry同步后，hdfs权限与sentry监控的目录（/user/hive/warehouse）的权限同步
5.2.1. 切换到hive用户，查看hive库文件的权限
设置hdfs acl与sentry同步后，sentry监控的hive库的权限改动会同步到对应的hdfs文件权限

[[email protected] home]# kinit hive
[[email protected] home]# hdfs dfs -getfacl -R /user/hive/warehouse/
# file: /user/hive/warehouse
# owner: hive
# group: hive
user::rwx
user:hive:rwx
group::---
group:hive:rwx
mask::rwx
other::--x

# file: /user/hive/warehouse/db1.db
# owner: hive
# group: hive
user::rwx
user:hive:rwx
group:user1:rwx
group::---
group:hive:rwx
mask::rwx
other::--x

# file: /user/hive/warehouse/db1.db/table1
# owner: hive
# group: hive
user::rwx
user:hive:rwx
group:user1:rwx
group::---
group:hive:rwx
mask::rwx
other::--x

# file: /user/hive/warehouse/db1.db/table1/events.csv
# owner: hive
# group: hive
user::rwx
user:hive:rwx
group:user1:rwx
group::---
group:hive:rwx
mask::rwx
other::--x

# file: /user/hive/warehouse/db2.db
# owner: hive
# group: hive
user::rwx
user:hive:rwx
group:user2:rwx
group::---
group:hive:rwx
mask::rwx
other::--x

# file: /user/hive/warehouse/db2.db/table1
# owner: hive
# group: hive
user::rwx
user:hive:rwx
group:user2:rwx
group::---
group:hive:rwx
mask::rwx
other::--x

# file: /user/hive/warehouse/db2.db/table1/events.csv
# owner: hive
# group: hive
user::rwx
user:hive:rwx
group:user2:rwx
group::---
group:hive:rwx
mask::rwx
other::--x

　　

5.2.2. 切换到user1用户，查看hdfs文件

[[email protected] home]# kinit user1
Password for [email protected]:
[[email protected] home]# hdfs dfs -ls /user/hive/warehouse/db2.db
ls: Permission denied: user=user1, access=READ_EXECUTE, inode="/user/hive/warehouse/db2.db":hive:hive:drwxrwx—x
[[email protected] home]# hdfs dfs -cat /user/hive/warehouse/db2.db/table1/events.csv
cat: Permission denied: user=user1, access=READ, inode="/user/hive/warehouse/db2.db/table1/events.csv":hive:hive:-rwxrwx--x

[[email protected] home]# hdfs dfs -ls /user/hive/warehouse/db1.db
Found 1 items
drwxrwx--x+ - hive hive 0 2016-09-29 16:54 /user/hive/warehouse/db1.db/table1
[[email protected] home]# hdfs dfs -cat /user/hive/warehouse/db1.db/table1/events.csv
10.1.2.3,US,android,createNote
10.200.88.99,FR,windows,updateNote
10.1.2.3,US,android,updateNote
10.200.88.77,FR,ios,createNote
10.1.4.5,US,windows,updateTag

　　

5.2.3. 切换到user2用户，查看hdfs文件

[[email protected] home]# kinit user2
Password for [email protected]:
[[email protected] home]# hdfs dfs -cat /user/hive/warehouse/db1.db/table1/events.csv
cat: Permission denied: user=user2, access=READ, inode="/user/hive/warehouse/db1.db/table1/events.csv":hive:hive:-rwxrwx--x
[[email protected] home]# hdfs dfs -cat /user/hive/warehouse/db2.db/table1/events.csv
10.1.2.3,US,android,createNote
10.200.88.99,FR,windows,updateNote
10.1.2.3,US,android,updateNote
10.200.88.77,FR,ios,createNote
10.1.4.5,US,windows,updateTag

5.3. Spark测试
5.3.1. Spark读hive表数据并打印到控制台
（1） 切换到user1用户测试

[[email protected] xdf]# kinit user1
Password for [email protected]:
[[email protected] xdf]# spark-submit --class iie.hadoop.permission.QueryTable --master local /home/xdf/spark.jar db2 table1
……
Exception in thread "main" org.apache.hadoop.security.AccessControlException: Permission denied: user=user1, access=READ_EXECUTE, inode="/user/hive/warehouse/db2.db/table1":hive:hive:drwxrwx—x
[[email protected] xdf]# spark-submit --class iie.hadoop.permission.QueryTable --master local /home/xdf/spark.jar db1 table1
……
+------------+-------+-------+----------+
| ip|country| client| action|
+------------+-------+-------+----------+
| 10.1.2.3| US|android|createNote|
|10.200.88.99| FR|windows|updateNote|
| 10.1.2.3| US|android|updateNote|
|10.200.88.77| FR| ios|createNote|
| 10.1.4.5| US|windows| updateTag|
+------------+-------+-------+----------+

　　

（2） 切换到user2用户测试

[[email protected] xdf]# kinit user2
Password for [email protected]:
[[email protected] xdf]# spark-submit --class iie.hadoop.permission.QueryTable --master local /home/xdf/spark.jar db1 table1
……
Exception in thread "main" org.apache.hadoop.security.AccessControlException: Permission denied: user=user2, access=READ_EXECUTE, inode="/user/hive/warehouse/db1.db/table1":hive:hive:drwxrwx—x
[[email protected] xdf]# spark-submit --class iie.hadoop.permission.QueryTable --master local /home/xdf/spark.jar db2 table1
……
+------------+-------+-------+----------+
| ip|country| client| action|
+------------+-------+-------+----------+
| 10.1.2.3| US|android|createNote|
|10.200.88.99| FR|windows|updateNote|
| 10.1.2.3| US|android|updateNote|
|10.200.88.77| FR| ios|createNote|
| 10.1.4.5| US|windows| updateTag|
+------------+-------+-------+----------+

5.3.2. Spark读文件数据写入hive表中
调用工具程序spark.jar读本地文件/home/xdf/events.csv数据写到db2.table2
切换到user2用户测试

kinit user2
beeline -u "jdbc:hive2://vmw208:10000/;principal=hive/[email protected]"
use db2;
create table table2 (
ip STRING, country STRING, client STRING, action STRING
) ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘,‘;
[[email protected] xdf]# spark-submit --class iie.hadoop.permission.HCatWriterTest --master local /home/xdf/spark.jar /home/xdf/events.csv db2 table2

成功！
写到db1.table1报错，没有权限！

Exception in thread "main" org.apache.hive.hcatalog.common.HCatException : 2004 : HCatOutputFormat not initialized, setOutput has to be called. Cause : org.apache.hadoop.security.AccessControlException: Permission denied: user=user2, access=WRITE, inode="/user/hive/warehouse/db1.db/table1":hive:hive:drwxrwx--x

上面只是测试环境，因为kinit + 密码的方式有时效限制，不适合在生产环境运行，幸好spark提供了相关的参数：

spark-submit
……
--principal # 用户对应的kerberos principle
--keytab # 对应用户principle生成的密钥文件

spark的权限管理通过对hdfs/hive的文件目录设置权限来管理，不同的用户拥有不同的权限，用户在提交spark任务时，指定对应用户的kerberos principle和keytab来实现权限管理。任务提交命令如下：

spark-submit --class iie.hadoop.permission.QueryTable --master local [email protected] --keytab=/home/user1/user1.keytab /home/user1/spark.jar db1 table1

其中--principal 和--keytab与用户一一对应
5.4. Kafka测试
5.4.1. 认证
用户kafka为kafka权限控制的超级管理员

[[email protected] iie]#kinit -kt /home/iie/kafka.keytab kafka

5.4.2. 创建topic
创建topic1和topic2

[[email protected] iie]#kafka-topics --zookeeper node11:2181/kafka --create --topic topic1 --partitions 2 --replication-factor 1
[[email protected] iie]#kafka-topics --zookeeper node11:2181/kafka --create --topic topic2 --partitions 2 --replication-factor 1

5.4.3. 赋权
给user1用户附topic1的读写权限

[[email protected] iie]#kafka-acls --authorizer-properties zookeeper.connect=node11:2181/kafka --add --allow-principal User:user1 --allow-host node10  --producer --topic topic1 --group console-consumer-9175
[[email protected] iie]#kafka-acls --authorizer-properties zookeeper.connect=node11:2181/kafka --add --allow-principal User:user1 --allow-host node10  --consumer --topic topic1 --group console-consumer-9175

给user2用户附topic2的读写权限

[[email protected] iie]#kafka-acls --authorizer-properties zookeeper.connect=node11:2181/kafka --add --allow-principal User:user2 --allow-host node10  --producer --topic topic2 --group console-consumer-9175
[[email protected] iie]#kafka-acls --authorizer-properties zookeeper.connect=node11:2181/kafka --add --allow-principal User:user2 --allow-host node10  --consumer --topic topic2 --group console-consumer-9175

5.4.4. 查看权限

[[email protected] iie]#kafka-acls --authorizer-properties zookeeper.connect=node11:2181/kafka --list
Current ACLs for resource `Topic:topic1`:
User:user1 has Allow permission for operations: Write from hosts: node10
User:user1 has Allow permission for operations: Read from hosts: node10
Current ACLs for resource `Topic:topic2`:
User:user2 has Allow permission for operations: Read from hosts: node10
User:user2 has Allow permission for operations: Write from hosts: node10

5.4.5. 创建生产消费配置文件
创建consumer.properties

cat /etc/kafka/conf/consumer.properties
security.protocol=SASL_PLAINTEXT
sasl.mechanism=GSSAPI
sasl.kerberos.service.name=kafka
group.id=console-consumer-9175

创建producer.properties

cat /etc/kafka/conf/producer.properties
security.protocol=SASL_PLAINTEXT
sasl.mechanism=GSSAPI
sasl.kerberos.service.name=kafka

5.4.6. 生产数据
命令行生产数据

[[email protected] iie]#kinit user1
[[email protected] iie]#kafka-console-producer --broker-list node12:9092 --topic topic1 --producer.config /etc/kafka/conf/producer.properties
123123
123123

5.4.7. 消费数据
命令行消费数据

[[email protected] iie]#kinit user1
[[email protected] iie]#kafka-console-consumer --bootstrap-server node12:9092 --topic topic1 --new-consumer --from-beginning --consumer.config /etc/kafka/conf/consumer.properties
123123
123123

用户对topic没有权限时报错

[[email protected] iie]# kinit user2
Password for [email protected]:
[[email protected] iie]# kafka-console-consumer --bootstrap-server node12:9092 --topic topic1 --new-consumer --from-beginning --consumer.config /etc/kafka/conf/consumer.properties
[2016-10-12 15:38:01,599] ERROR Unknown error when running consumer:  (kafka.tools.ConsoleConsumer$)
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [topic1]

5.4.8. 移除权限
登陆管理员用户移除权限

[[email protected] iie]#kinit -kt /home/iie/kafka.keytab kafka

删除user1对topic1的消费权限

[[email protected] iie]# kafka-acls --authorizer-properties zookeeper.connect=node11:2181/kafka --remove --allow-principal User:user1 --allow-host node10 --consumer --topic topic1 --group console-consumer-92175
Are you sure you want to remove ACLs:
 	User:user1 has Allow permission for operations: Read from hosts: node10
	User:user1 has Allow permission for operations: Describe from hosts: node10
 from resource `Topic:topic1`? (y/n)
y
Are you sure you want to remove ACLs:
 	User:user1 has Allow permission for operations: Read from hosts: node10
 from resource `Group:console-consumer-92175`? (y/n)
y
Current ACLs for resource `Topic:topic1`:
 	User:Aluser1 has Allow permission for operations: Read from hosts: node10
	User:Aluser1 has Allow permission for operations: Describe from hosts: node10
	User:user1 has Allow permission for operations: Write from hosts: node10 

Current ACLs for resource `Group:console-consumer-92175`:

　　

测试user1消费topic1报错，说明权限已经移除

[[email protected] iie]# kinit user1
Password for [email protected]:
[[email protected] iie]# kafka-console-consumer --bootstrap-server node12:9092 --topic topic1 --new-consumer --from-beginning --consumer.config /etc/kafka/conf/consumer.properties
[2016-10-12 15:45:11,572] WARN The configuration sasl.mechanism = GSSAPI was supplied but isn‘t a known config. (org.apache.kafka.clients.consumer.ConsumerConfig)
[2016-10-12 15:45:11,914] WARN Not authorized to read from topic topic1. (org.apache.kafka.clients.consumer.internals.Fetcher)
[2016-10-12 15:45:11,916] ERROR Error processing message, terminating consumer process:  (kafka.tools.ConsoleConsumer$)
org.apache.kafka.common.errors.TopicAuthorizationException: Not authorized to access topics: [topic1]
[2016-10-12 15:45:11,920] WARN Not authorized to read from topic topic1. (org.apache.kafka.clients.consumer.internals.Fetcher)
[2016-10-12 15:45:11,921] ERROR Not authorized to commit to topics [topic1] for group console-consumer-9175 (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2016-10-12 15:45:11,922] WARN Auto offset commit failed for group console-consumer-9175: Not authorized to access topics: [topic1] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator)
[2016-10-12 15:45:11,927] WARN TGT renewal thread has been interrupted and will exit. (org.apache.kafka.common.security.kerberos.Login)
Processed a total of 0 messages

　　

时间： 2024-11-19 23:59:51

cdh权限测试示例的相关文章

【转】Oracle索引列NULL值引发执行计划该表的测试示例

有时开发进行表结构设计,对表字段是否为空过于随意,出现诸如id1=id2,如果允许字段为空,因为Oracle中空值并不等于空值,有可能得到意料之外的结果.除此之外,最关键的是,NULL会影响oracle的执行计划. 以下为NULL影响执行计划的测试示例. /*1.构建test表,其中create table方式建立的test表结构object_id非空*,走索引/ SELECT Count(*) FROM all_objects WHERE object_id IS NOT NULL; --41

Java中，权限修饰符的权限测试

============================================================================= 1. 1 /* 2 权限修饰符的权限测试: 3 4 权限修饰符 本类中 同一个包下的子类和无关类 不同包下的子类 不同包下的无关类 5 private Yes No No No 6 默认 Yes Yes No No 7 protected Yes Yes Yes No 8 public Yes Yes Yes Yes 9 */ 10 11 p

ASP.NET MVC：窗体身份验证及角色权限管理示例

ASP.NET MVC 建立 ASP.NET 基础之上,很多 ASP.NET 的特性(如窗体身份验证.成员资格)在 MVC 中可以直接使用.本文旨在提供可参考的代码,不会涉及这方面太多理论的知识. 本文仅使用 ASP.NET 的窗体身份验证,不会使用它的 成员资格(Membership) 和 角色管理 (RoleManager),原因有二:一是不灵活,二是和 MVC 关系不太. 一.示例项目 User.cs 是模型文件,其中包含了 User 类: public class User { publ

[有码有真相]python类私有属性等要点理解及测试示例代码

# encoding: utf-8 ''' 1. python约定类定义中"__"开头(至多一个"_"结尾)的属性为"相对"的私有属性 2. 私有属性在类及其子类定义中是"可见"的,在 其它范围正常访问是相对"不可见"的 3. 留下子类可以修改父类私有属性的口子方便debug测试等 4. 在类定义以外可以通过['_'+className+'私有属性名称']名称访问 5. 私有属性引入是避免类,实例等混淆和冲

Go语言表组测试示例

比基本测试多个循环... package main import ( "testing" "net/http" ) const checkMark = " OK! " const ballotX = " ERROR! " func TestDownload(t *testing.T) { var urls = []struct { url string statusCode int }{ { "http://loca

kubernetes容器集群管理启动一个测试示例

创建nginx 创建3个nginx副本 [[email protected] bin]# kubectl run nginx --image=nginx --replicas=3 kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create inste

2016最后一贴，终于调通一个测试示例，并发现一个BUG？？？

真的难点在于第一次调通.纠结五天,终于搞出界面. 也发现了一个书上代码,编辑用户时死活不通的情况,我将Links去了,改在data里,我X,,全OK了.. 原来的代码: onAdd: function(button, e, options){ this.createDialog(null); }, onEdit: function(button, e, options){ var me = this, records = me.getRecordsSelected(); if(records[0

利用EasyMock生成数据库连接简单测试示例

package demo.mock; import java.sql.Connection; import java.sql.PreparedStatement; import java.sql.ResultSet; import java.sql.SQLException; public class LoginAction{ public String login(Connection conn, String name, int pas) { PreparedStatement pst =

docker nodes 测试示例

1.在同级目录编辑app.js文件 文件内容如下 const http = require('http'); const os = require('os'); console.log("Kubia server starting...."); var handler = function(request, response) { console.log("Received request from " + request.connection.remoteAddr