最近线上的hive升级到了0.13，遇到不少问题。权限上面，设置了hive.security.authorization.createtable.owner.grants 在hive0.13中，用户自己创建的表也没有权限。通过对源码的分析和debug找到了rc并fix，下面记录下。
1.首先在hive0.11中和hive0.13中分别做建表测试，通过查看数据库中的元数据，发现在hive0.11中如果设置了owner的参数在表创建之后，用户会有这个表的all的权限（具体可以分析db_privs ，tbl_privs表，hive.security.authorization.createtable.owner.grants设置为了all），而在hive0.13中则为空的。
2.通过
hive -hiveconf hive.root.logger=DEBUG,console
对比前后的日志,发现在hive0.11的时候，创建的表的属性中有权限的设置
而在hive0.11中这个是为空的
hive0.11:
14/07/16 17:05:39 DEBUG hive.log: DDL: struct ericni4 { string a}14/07/16 17:05:39 DEBUG
lazy.LazySimpleSerDe: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with:
columnNames=[a] columnTypes=[string] separator=[[[email protected]] nullstring=\N
lastColumnTakesRest=false14/07/16 17:05:39 INFO metastore.HiveMetaStore: 0:
create_table: Table(tableName:ericni4, dbName:temp, owner:ericni2, createTime:1405501539,
lastAccessTime:0, retention:0,sd:StorageDescriptor(cols:[FieldSchema(name:a,
type:string, comment:null)], location:null,
inputFormat:org.apache.hadoop.mapred.SequenceFileInputFormat,
outputFormat:org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat,
compressed:false, numBuckets:-1,
serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy.
LazySimpleSerDe,parameters:{serialization.format=1}), bucketCols:[], sortCols:[],
parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[],
skewedColValueLocationMaps:{}), storedAsSubDirectories:false),partitionKeys:[],
parameters:{}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE,
privileges:PrincipalPrivilegeSet(userPrivileges:{ericni2=[PrivilegeGrantInfo(privilege:
all, createTime:-1, grantor:ericni2, grantorType:USER, grantOption:true)]},
groupPrivileges:null, rolePrivileges:null)) #hive0.11创建表的时候增加了这个设置
hive0.13
14/07/16 17:10:07 DEBUG hive.log: DDL: struct ericni4 { string a}14/07/16 17:10:07 DEBUG
lazy.LazySimpleSerDe: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe initialized with:
columnNames=[a] columnTypes=[string] separator=[[[email protected]] nullstring=\N
lastColumnTakesRest=false14/07/16 17:10:07 INFO metastore.HiveMetaStore: 0:
create_table: Table(tableName:ericni4, dbName:temp, owner:ericni1, createTime:1405501807,
lastAccessTime:0, retention:0,sd:StorageDescriptor(cols:[FieldSchema(name:a,
type:string, comment:null)], location:null,
inputFormat:org.apache.hadoop.mapred.SequenceFileInputFormat,
outputFormat:org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat,
compressed:false, numBuckets:-1,
serdeInfo:SerDeInfo(name:null, serializationLib:org.apache.hadoop.hive.serde2.lazy
.LazySimpleSerDe,parameters:{serialization.format=1}), bucketCols:[], sortCols:[],
parameters:{}, skewedInfo:SkewedInfo(skewedColNames:[], skewedColValues:[],
skewedColValueLocationMaps:{}), storedAsSubDirectories:false),partitionKeys:[],
parameters:{}, viewOriginalText:null, viewExpandedText:null, tableType:MANAGED_TABLE)
3.会不会是参数没有生效？在hive0.13中注释掉这个设置做对比，发现create_table的属性并没有改变，说明至少在现在的0.13环境下这个参数并没有生效。而在0.11中前后是有变化的。
4.在hive0.11中我们设置一个错误的值来获取堆栈的调用信息：
得到0.11的调用栈：
at org.apache.hadoop.hive.ql.session.CreateTableAutomaticGrant.checkPrivilege(CreateTableAutomaticGrant.java:118) at org.apache.hadoop.hive.ql.session.CreateTableAutomaticGrant.getGrantorInfoList(CreateTableAutomaticGrant.java:97) at org.apache.hadoop.hive.ql.session.CreateTableAutomaticGrant.create(CreateTableAutomaticGrant.java:52) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:275) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:278) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:670) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
再来分析源码
在hive0.11中，在初始化session时，调用SessionState的start方法，会直接根据CreateTableAutomaticGrant类设置createTableGrants属性。
SessionState.start:
try {
startSs. authenticator = HiveUtils.getAuthenticator(
startSs.getConf(),HiveConf.ConfVars.HIVE_AUTHENTICATOR_MANAGER);
startSs. authorizer = HiveUtils.getAuthorizeProviderManager(
startSs.getConf(), HiveConf.ConfVars.HIVE_AUTHORIZATION_MANAGER,
startSs. authenticator);
startSs. createTableGrants = CreateTableAutomaticGrant.create(startSs
.getConf()); //设置owner的相关权限
} catch (HiveException e) {
throw new RuntimeException(e);
}
然后在创建表的时候，调用Hive.createTable的方法
(public void createTable(Table tbl, boolean ifNotExists))
public CreateTableAutomaticGrant getCreateTableGrants() {
return createTableGrants;
}
而在hive0.13中，即使设置错误也不会有报错产生，通过debug源码发现这里的调用关系有些变化，SessionState的getCreateTableGrants方法调用的是setupAuth方法。
public CreateTableAutomaticGrant getCreateTableGrants() {
setupAuth(); //调用setupAuth方法设置createTableGrants
return createTableGrants;
}
其中setupAuth方法如下：
private void setupAuth() {
if (authenticator != null) {
// auth has been initialized
return;
}
try {
authenticator = HiveUtils.getAuthenticator( conf,
HiveConf.ConfVars.HIVE_AUTHENTICATOR_MANAGER);
authenticator.setSessionState( this);
authorizer = HiveUtils.getAuthorizeProviderManager( conf,
HiveConf.ConfVars.HIVE_AUTHORIZATION_MANAGER, authenticator, true );
if (authorizer == null) { //在authorizer 的取值为null时，才会生成权限
// if it was null, the new authorization plugin must be specified in
// config
HiveAuthorizerFactory authorizerFactory = HiveUtils.getAuthorizerFactory(conf ,
HiveConf.ConfVars.HIVE_AUTHORIZATION_MANAGER);
authorizerV2 = authorizerFactory.createHiveAuthorizer(new HiveMetastoreClientFactoryImpl(),
conf, authenticator);
authorizerV2.applyAuthorizationConfigPolicy(conf );
// create the create table grants with new config
createTableGrants = CreateTableAutomaticGrant.create( conf);
}
} catch (HiveException e) {
throw new RuntimeException(e);
}
if(LOG.isDebugEnabled()){
Object authorizationClass = getAuthorizationMode() == AuthorizationMode. V1 ?
getAuthorizer() : getAuthorizerV2();
LOG.debug("Session is using authorization class " + authorizationClass.getClass());
}
return;
}
可以看到在在authorizer 的取值为null时，才会有生产ower权限的操作（CreateTableAutomaticGrant.create(conf)）
authorizer 的值由参数hive.security.authorization.manager获得，在hive0.13里面这个是有默认值的：
HIVE_AUTHORIZATION_MANAGER("hive.security.authorization.manager",
"org.apache.hadoop.hive.ql.security.authorization.DefaultHiveAuthorizationProvider"),
这样就导致在setupAuth方法中并不会通过CreateTableAutomaticGrant.create(conf)来设置owner的权限，rc找到了。fix也比较简单。
try {
authenticator = HiveUtils.getAuthenticator( conf,
HiveConf.ConfVars.HIVE_AUTHENTICATOR_MANAGER);
authenticator.setSessionState( this);
authorizer = HiveUtils.getAuthorizeProviderManager( conf,
HiveConf.ConfVars.HIVE_AUTHORIZATION_MANAGER, authenticator, true );
createTableGrants = CreateTableAutomaticGrant.create(conf); //不管authorizer 是否为空都设置相关权限
if (authorizer == null) {
.......
bug fix之后，使用错误的参数，堆栈如下，说明生效了：
14/07/18 12:33:38 WARN session.SessionState: authenticator is null ,return,auth has been initialized 14/07/18 12:33:38 WARN session.CreateTableAutomaticGrant: Privilege is null FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException: Privilege alldddd is not found. 14/07/18 12:33:38 ERROR ql.Driver: FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException: Privilege alldddd is not found. java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Privilege alldddd is not found. at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:410) at org.apache.hadoop.hive.ql.session.SessionState.getAuthorizationMode(SessionState.java:979) at org.apache.hadoop.hive.ql.session.SessionState.isAuthorizationModeV2(SessionState.java:990) at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:508) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:461) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:322) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:975) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1040) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:359) at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:456) at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:466) at org.apache.hadoop.hive.cli.CliDriver.processInitFiles(CliDriver.java:502) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:739) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:208) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Privilege alldddd is not found. at org.apache.hadoop.hive.ql.session.CreateTableAutomaticGrant.validatePrivilege(CreateTableAutomaticGrant.java:129) at org.apache.hadoop.hive.ql.session.CreateTableAutomaticGrant.getGrantorInfoList(CreateTableAutomaticGrant.java:105) at org.apache.hadoop.hive.ql.session.CreateTableAutomaticGrant.create(CreateTableAutomaticGrant.java:57) at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:392) ... 24 more
时间： 2024-07-30 10:14:29