从 "org.apache.hadoop.security.AccessControlException:Permission denied: user=..." 看Hadoop 的用户登陆认证

假设远程提交任务给Hadoop 可能会遇到?"org.apache.hadoop.security.AccessControlException:Permission denied: user=..." , 当然,假设是spark over YARN, 也相同会遇到相似的问题,比如:

?An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext.

: org.apache.hadoop.security.AccessControlException: Permission denied: user=abel, access=WRITE, inode="/user/abel/.sparkStaging/application_1460633311001_0032":hdfs:hdfs:drwxr-xr-x

hadoop 的用户鉴权是基于JAAS的。当中hadoop.security.authentication属性 有simple 和kerberos 等方式。假设hadoop.security.authentication等于”kerberos”,那么是“hadoop-user-kerberos”或者“hadoop-keytab-kerberos”。否则是“hadoop-simple”。 当用户登陆的时候。若org.apache.hadoop.security.User为空,那么说明尚未登录过,调用静态方法getLoginUser()创建org.apache.hadoop.security.UserGroupInformatio实例,在getLoginUser()中又会调用HadoopLoginModule的login()和commit()方法。

在使用了kerberos的情况下。从javax.security.auth.kerberos.KerberosPrincipal的实例获取username。在没有使用kerberos时。首先读取hadoop 的系统环境变量。假设没有的话。对于windows 从com.sun.security.auth.NTUserPrincipal 获取username。对于类unix 从com.sun.security.auth.UnixPrincipal 中获得username,然后再看该用户属于哪个group,从而完毕登陆认证。

基本理解了问题的根源,那么这个“org.apache.hadoop.security.AccessControlException:Permission denied: user=...”异常信息是怎么产生的呢?远程提交,假设没有hadoop 的系统环境变量。就会读取当前主机的username,结果Hadoop集群中没有该用户。所以杯具了。

至于问题的解决,以mac 为例。 将

export HADOOP_USER_NAME = hdfs?

加入到 ~/.bash_profile 中。

?$ source ?~/.bash_profile?

接下来。继续提交任务到 hadoop 集群,everything is OK。?

原文地址:https://www.cnblogs.com/llguanli/p/8728170.html

时间: 2024-11-09 04:42:44

从 "org.apache.hadoop.security.AccessControlException:Permission denied: user=..." 看Hadoop 的用户登陆认证的相关文章

Win下Eclipse提交Hadoop程序出错:org.apache.hadoop.security.AccessControlException: Permission denied: user=D

描述:在Windows下使用Eclipse进行Hadoop的程序编写,然后Run on hadoop 后,出现如下错误: 11/10/28 16:05:53 INFO mapred.JobClient: Running job: job_201110281103_000311/10/28 16:05:54 INFO mapred.JobClient: map 0% reduce 0%11/10/28 16:06:05 INFO mapred.JobClient: Task Id : attemp

kylin cube测试时,报错:org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x

异常: org.apache.hadoop.security.AccessControlException: Permission denied: user=root, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.DefaultAuthorizationProvider.checkFsPermission(DefaultAuthoriza

Exception in thread "main" org.apache.hadoop.security.AccessControlException: Permission denied: user=Mypc, access=WRITE, inode="/":fan:supergroup:drwxr-xr-x

在window上编程提示没有写Hadoop的权限 Exception in thread "main" org.apache.hadoop.security.AccessControlException: Permission denied: user=Mypc, access=WRITE, inode="/":fan:supergroup:drwxr-xr-x 曾经踩过的坑: 保存结果到hdfs上没有写的权限* 通过修改权限将文件写入到指定的目录下* * $HAD

异常-Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=hdfs, access=WRITE, inode="/hbase":root:supergroup:drwxr-xr-x

1 详细异常 Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=hdfs, access=WRITE, inode="/hbase":root:supergroup:drwxr-xr-x at org.apache.hadoop.hdfs.server.namenode.DefaultAu

org.apache.hadoop.security.AccessControlException: Permission denied: user=?, access=WRITE, inode="/":hadoop:supergroup:drwxr-xr-x 异常解决

进行如下更改: vim /usr/local/hadoop/etc/hadoop/hdfs-site.xml [我的hadoop目录在/usr/local下,具体的是修改你的hadoop目录中的/etc/hadoop/hdfs-site.xml]添加一个property:<property>      <name>dfs.permissions</name>      <value>false</value></property> 然

org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException)

在运行hadoop的程序时,向hdfs中写文件时候,抛出异常信息如下: Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=Administrator, access=WRITE, inode="/user":root:supergroup:drwxr-xr-x 原因:Hdfs中的/user

org.apache.hadoop.security.AccessControlException

Windows|Eclipse 运行HDFS程序之后,报:org.apache.Hadoop.security.AccessControlException: Permission denied: user=sunqw, access=WRITE, inode="":hadoop:supergroup:rwxr-xr-x. 或者 Windows|Eclipse 运行HDFS程序之后,报:org.apache.hadoop.security.AccessControlException:

Eclipse远程连接hadoop时 报 Permission denied:

用Eclipse远程连接hadoop时报org.apache.hadoop.security.AccessControlException: Permission denied: user=DrWho, access=READ_EXECUTE, inode="system":hadoop:supergroup:rwx-wx-wx错误,解决方法: 在hdfs-site.xml 中取消权限校验,即加入以下配置: <property>    <name>dfs.per

hive之权限问题AccessControlException Permission denied: user=root, access=WR

问题描述:在集群上,用hive分析数据出现如下错误 FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Got exception: org.apache.hadoop.security.AccessControlException Permission denied: user=root, access=WRITE, inode="/d