1.安装CDH5.12.x

安装方式
安装前准备
安装步骤
安装过程
修改/etc/hosts
设置ssh 互信
修改linux 系统设置
安装JDK1.8
安装python2.7
安装mysql/postgreysql数据库
安装ntp
设置本地yum源
下载CDH parcels包
安装CM
使用yum安装CM
安装agent
进入CDH
添加节点
使用CM添加节点
手动安装agent
远程yum源安装节点
服务
安装中出现的问题

安装方式

CDH有三种安装方式

  • parcels 二进制程序包,包含了CDH组件中的依赖\版本等信息,可以方便的切换CDH版本,CM调用yum来安装parcels,非常方便.cloudera 推荐使用此种方式安装.
  • packges rpm方式,缺点是每次只能安装一个版本的rpm,升级\回滚都必须重装rpm,因此升级\回滚存在包不兼容风险.CM调用yum来安装rpm.不推荐.
  • tarball 手动下载tar包安装,无法使用yum安装,十分麻烦,而且不支持通过CM在线升级CDH,必须完全手动升级.不建议.

    本质上parcels和tarball都是CDH组件的压缩包,但是cloudera对tarball没做支持.

    本次以parcels方式安装,不使用CDH自动安装,手动下载parcels包安装

  • 自动安装 设置CDH yum资源库,然后可以直接用yum 安装jdk CM agent
  • 手动安装 手动安装JDK,下载CDH的CM agent,配置本地yum源安装CM和agent

安装前准备

  • OS版本 centos6.8
  • JDK 1.8
  • python 2.7
  • 安装账户root CM依赖root账户
  • 安装路径 CDH默认路径/opt/cloudera/parcls即可
  • 数据库 mysql,不要使用内置的postgrey

安装步骤

  • 在vbox上安装centos6.8 略
  • 设置ssh互信 至少CM节点要能ssh登录其它节点
  • 修改linux系统设置

    修改unlimit

    关闭iptables

    关闭selinux

    关闭swape 略

    关闭ipv6 略

    设置静态ip

    修改/etc/hosts,加入CDH所有节点的ip

  • 安装JDK1.8
  • 安装python2.7
  • 安装mysql数据库,并创建数据库和用户
  • 安装ntp
  • 安装CM
  • 安装agent
  • 安装CDH组件

安装过程

修改/etc/hosts

echo "127.0.0.1 localhost" > /etc/hosts
echo "192.168.0.20 CM"     >> /etc/hosts
echo "192.168.0.21 cdh1"   >> /etc/hosts
echo "192.168.0.21 cdh2"   >> /etc/hosts

设置ssh 互信

参CM节点上执行:

[[email protected] ~]# ssh-keygen -t rsa -P ‘‘

然后将key复制到其它节点:

ssh-copy-id -i ~/.ssh/id_rsa.pub 192.168.0.21
ssh-copy-id -i ~/.ssh/id_rsa.pub 192.168.0.22

这样CM节点就能ssh到192.168.0.21 192.168.0.22两台机器上了

修改linux 系统设置

1.修改unlimit 略

2.关闭iptables

[[email protected] ~]# service iptables stop
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Unloading modules:                               [  OK  ]
[[email protected] ~]# chkconfig iptables off

3.关闭swap

虚拟机内存不足,略

4.禁用ipv6

这个好麻烦的,略

安装JDK1.8

下载对应版本的JDK1.8,解压到/opt/jdk1.8下

设置java环境变量:

在/etc/profile中设置java环境变量

[[email protected] opt]# tar -zxvf jdk-7u80-linux-x64.tar.gz
[[email protected] opt]# rm jdk-7u80-linux-x64.tar.gz
[[email protected] jdk1.7.0_80]# vi /etc/profile #添加以下内容
JAVA_HOME=/opt/jdk1.7.0_80
PATH=$JAVA_HOME/bin:$PATH

export JAVA_HOME
export PATH

然后执行java 命令测试是否安装成功.

安装python2.7

为了方便,直接安装acconda,再修改系统的python即可.

此外centos6自带的python2.6.6也是可以的

安装mysql/postgreysql数据库

这里选择安装postgreysql,因为mysql的安装包实在太TM的大了

[[email protected] ~]# yum isntall postgresql #安装后数据目录在 /var/lib/pgsql
[[email protected] ~]# chkconfig --list #找到pg的服务
…
postgresql         0:off    1:off    2:off    3:off    4:off    5:off    6:off
..
[[email protected] ~]# service postgresql initdb #初始化pg数据库
Initializing database:                                     [  OK  ]
[[email protected] ~]# chkconfig postgresql on #开机启动
[[email protected] ~]# service postgresql start #启动pg
Starting postgresql service:                               [  OK  ]
[[email protected] ~]# sudo -u postgres psql
could not change directory to "/root"
psql (8.4.20)
Type "help" for help.

postgres=# create user cdh1;
CREATE ROLE
postgres=# select * from pg_users;
ERROR:  relation "pg_users" does not exist
LINE 1: select * from pg_users;
                      ^
postgres=# select * from pg_user;
 usename  | usesysid | usecreatedb | usesuper | usecatupd |  passwd  | valuntil | useconfig
----------+----------+-------------+----------+-----------+----------+----------+-----------
 postgres |       10 | t           | t        | t         | ******** |          |
 cdh1     |    16384 | f           | f        | f         | ******** |          |
(2 rows)

postgres=# alter user cdh1 with password ‘cdh1‘;
ALTER ROLE

postgres=# create database cdh1 owner cdh1 ENCODING ‘UTF-8‘;
CREATE DATABASE

[[email protected] ~]# vi  /var/lib/pgsql/data/pg_hba.conf #改成如下
# "local" is for Unix domain socket connections only
local   all         all                               truest
# IPv4 local connections:
host    all         all         127.0.0.1/32          md5
host    all         all         192.168.0.0/24        md5
# IPv6 local connections:
host    all         all         ::1/128               ident
[[email protected] data]# vi postgresql.conf #改成
listen_addresses = ‘*‘
[[email protected] ~]# service postgresql restart
Stopping postgresql service:                               [  OK  ]
Starting postgresql service:                               [  OK  ]
[[email protected] ~]# psql -d cdh1 -U cdh1
Password for user cdh1:
psql (8.4.20)
Type "help" for help.

cdh1=>

安装ntp

[[email protected] ~]# yum install ntp

设置本地yum源

使用yum安装CM时,可以使用cloudera的远程yum源,或者把CM安装包下载到本地,并设置本地yum源.

[[email protected] ~]# mkdir -p cloudera_software/RPMS/x86_64
[[email protected] ~]# mkdir -p cloudera_software/repodata
[[email protected] ~]# cd cloudera_software/RPMS/x86_64
[[email protected] x86_64]# wget https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.12.0/RPMS/x86_64/cloudera-manager-agent-5.12.0-1.cm5120.p0.120.el6.x86_64.rpm
[[email protected] x86_64]# wget https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.12.0/RPMS/x86_64/cloudera-manager-daemons-5.12.0-1.cm5120.p0.120.el6.x86_64.rpm
[[email protected] x86_64]# wget https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.12.0/RPMS/x86_64/cloudera-manager-server-5.12.0-1.cm5120.p0.120.el6.x86_64.rpm
[[email protected] x86_64]# wget https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.12.0/RPMS/x86_64/cloudera-manager-server-db-2-5.12.0-1.cm5120.p0.120.el6.x86_64.rpm
[[email protected] x86_64]# wget https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5.12.0/RPMS/x86_64/enterprise-debuginfo-5.12.0-1.cm5120.p0.120.el6.x86_64.rpm
[[email protected] x86_64]# #wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5/RPMS/x86_64/jdk-6u31-linux-amd64.rpm
[[email protected] x86_64]# #wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5/RPMS/x86_64/oracle-j2sdk1.7-1.7.0+update67-1.x86_64.rpm
[[email protected] x86_64]# cd ../../repodata
[[email protected] repodata]# wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5/repodata/filelists.xml.gz
[[email protected] repodata]# wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5/repodata/filelists.xml.gz.asc
[[email protected] repodata]# wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5/repodata/other.xml.gz
[[email protected] repodata]# wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5/repodata/other.xml.gz.asc
[[email protected] repodata]# wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5/repodata/primary.xml.gz
[[email protected] repodata]# wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5/repodata/primary.xml.gz.asc
[[email protected] repodata]# wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5/repodata/repomd.xml
[[email protected] repodata]# wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5/repodata/repomd.xml.asc
[[email protected] repodata]# cd ..
[[email protected] cloudera_software]# wget http://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera
  • 配置本地yum源
[[email protected] cloudera_software]# cd /etc/yum.repos.d/ #添加
[cloudera-manager-local]
# Packages for Cloudera Manager, Version 5, on RedHat or CentOS 5 x86_64
name=cloudera-manager-local
baseurl=file:///root/cloudera_software/
gpgkey =file:///root/cloudera_software/RPM-GPG-KEY-cloudera
gpgcheck = 1

下载CDH parcels包

到 https://archive.cloudera.com/cdh5/parcels/ 下载对应版本的parcels包:

这里选择下载https://archive.cloudera.com/cdh5/parcels/latest/ 下的最新的CDH:

https://archive.cloudera.com/cdh5/parcels/latest/CDH-5.12.0-1.cdh5.12.0.p0.29-el6.parcel
https://archive.cloudera.com/cdh5/parcels/latest/CDH-5.12.0-1.cdh5.12.0.p0.29-el6.parcel.sha1
https://archive.cloudera.com/cdh5/parcels/latest/manifest.json

三个文件缺少一个都会导致找不到parcels包,如果新加入了文件,要重启CM进程.

放到/opt/cloudera/parcel-repo下,为什么放这里?因为CDH的parcels包默认就放这里,上图也可以看到.

[[email protected] ~]# mkdir -p /opt/cloudera/parcel-repo
[[email protected] ~]# mkdir -p /opt/cloudera/parcels

安装CM

使用yum安装CM

[[email protected] cloudera_software]# yum install cloudera-manager-agent cloudera-manager-daemons cloudera-manager-server

一开始下成了redhat5的CM包,删除重新下.记得要执行yum clean

上一步完成之后,会有这些文件:

[[email protected] x86_64]# ls /etc/cloudera-scm-server
db.properties  log4j.properties

修改其中的db.properties文件:

# Copyright (c) 2012 Cloudera, Inc. All rights reserved.
#
# This file describes the database connection.
#

# The database type
# Currently ‘mysql‘, ‘postgresql‘ and ‘oracle‘ are valid databases.
com.cloudera.cmf.db.type=postgresql

# The database host
# If a non standard port is needed, use ‘hostname:port‘
com.cloudera.cmf.db.host=localhost

# The database name
com.cloudera.cmf.db.name=cmf

# The database user
com.cloudera.cmf.db.user=cdh1

# The database user‘s password
com.cloudera.cmf.db.password=cdh1

# The db setup type
# By default, it is set to INIT
# If scm-server uses Embedded DB then it is set to EMBEDDED
# If scm-server uses External DB then it is set to EXTERNAL
com.cloudera.cmf.db.setupType=EXTERNAL

启动CM

[[email protected] ~]# service cloudera-scm-server start
Starting cloudera-scm-server:                              [FAILED]
[[email protected] ~]# more /var/log/cloudera-scm-server/cloudera-scm-server.out
+======================================================================+
|      Error: JAVA_HOME is not set and Java could not be found         |
+----------------------------------------------------------------------+
| Please download the latest Oracle JDK from the Oracle Java web site  |
|  > http://www.oracle.com/technetwork/java/javase/index.html <        |
|                                                                      |
| Cloudera Manager requires Java 1.6 or later.                         |
| NOTE: This script will find Oracle Java whether you install using    |
|       the binary or the RPM based installer.                         |
+======================================================================+
[[email protected] ~]# echo $JAVA_HOME
/opt/jdk1.7.0_80

简直是睁眼说瞎话.网上查了一下 http://community.cloudera.com/t5/Cloudera-Manager-Installation/Error-JAVA-HOME-is-not-set-and-Java-could-not-be-found/td-p/18974/page/3

按照要求设置JAVA_HOME

[[email protected] ~]# vi /etc/default/cloudera-scm-server
export JAVA_HOME=/opt/jdk1.7.0_80
然后执行:
[[email protected] ~]# service cloudera-scm-server start
Starting cloudera-scm-server:                              [  OK  ]

我有一句mmp不知当讲不当讲!

启动成功了却进不去 192.168.0.20:7180

打开日志一看:

2017-08-20 18:19:25,182 WARN com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0:com.mchange.v2.resourcepool.BasicResourcePool: [email protected]3 -- Acquisition Attempt Failed!!! Clearing pending acquires. Whil
e trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (5). Last acquisition attempt exception:
org.postgresql.util.PSQLException: FATAL: database "cmf" does not exist
    at org.postgresql.core.v3.ConnectionFactoryImpl.readStartupMessages(ConnectionFactoryImpl.java:469)
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:112)

卧槽,我以为CM会自动创建数据库呢.

再次启动报一堆表找不到,然后CM自动建表了了,可是过一会又挂了

2017-08-20 18:38:48,481 ERROR ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Error while attempting to retrieve repository info for repo https://archive.cloudera.com/sqoop-connectors/parcels/latest/
java.io.IOException: Closed
    at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.doConnect(NettyAsyncHttpProvider.java:873)
    at com.ning.http.client.providers.netty.NettyAsyncHttpProvider.execute(NettyAsyncHttpProvider.java:858)
    at com.ning.http.client.AsyncHttpClient.executeRequest(AsyncHttpClient.java:512)

CM一直在找远程资源库有没有?!

再次重启,竟然启动了!what the fuck!

安装agent

一上步就直接安装了agent了

安装完需要修改agent的配置文件使其能找到CM

vi /etc/cloudera-scm-agent/config.ini
server_host=192.168.0.20

启动agent

进入CDH

默认使用admin/admin进入

一开始不要选择新加主机,因为还没有装CM的其它服务,如host monitor

由于已经安装了JDK,就不让CM再安装了.

不装单用户

选key

选择CM节点的pub key

配置正确会出现CDH5.12.1的parcels,下一步,直接到parcels分发完成

完成后会显示没有host monitor

创建cmservice数据库create database cmservice owner cdh1 ENCODING ‘UTF-8‘;

这一步死活连不上数据库.查了很多资料,试了很多方法都不行.

添加节点

添加节点的本质是:

在节点上安装agent,然后agent连接cm,那么这个节点就添加进集群了.再由CM分发parcels包,再安装服务

使用CM添加节点

1.

2.

3.选择第二项可以加入自己的yum源

4.自动添加主机时,会设置节点上的yum源

我们到 https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/cloudera-manager.repo 看看这里写的是啥

[cloudera-manager]
# Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64
name=Cloudera Manager
baseurl=https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5/
gpgkey =https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera
gpgcheck = 1

其实就是cloudera的在线yum源，这个源里的东西是：

其实就是agent和cm的安装包有没有？

鉴于刷新yum元数据实在太慢了，放弃

或者在上面的步骤中设置自己yum源，不用CDH,这样快很多

手动安装agent

这时就不用选择"这样的操作了"，直接在节点上安装agent就好了

从cm上复制agent的安装包,并设置本地yum源并安装

[[email protected] cloudera_software]# scp -r /root/cloudera_software [email protected]:/root/
[[email protected] ~]# vi /etc/yum.repos.d/CentOS-Media.repo #添加
[cloudera-manager-local]
# Packages for Cloudera Manager, Version 5, on RedHat or CentOS 5 x86_64
name=cloudera-manager-local
baseurl=file:///root/cloudera_software/
gpgkey =file:///root/cloudera_software/RPM-GPG-KEY-cloudera
gpgcheck = 1
[[email protected] ~]# yum install cloudera-manager-agent
这时竟然要联网下一堆依赖，不过很快。怎么创建局域网源？
[[email protected] ~]# vi /etc/cloudera-scm-agent/config.ini
server_host=CM
[[email protected] ~]# service cloudera-scm-agent start

agent启动完成后，以CM就能看到节点了。

但此时并没有和host monitor连接。

在CM中重启host monitor后就正常了：

远程yum源安装节点

[[email protected] cloudera_software]# yum isntall httpd
………..
[[email protected] ~]# vi /etc/httpd/conf/httpd.conf
DocumentRoot "/root/cloudera_software"
……
[[email protected] ~]# chown -R apache.apache /root/cloudera_software
#删除默认主页
# [[email protected] ~]# mv /etc/httpd/conf.d/welcome.conf /etc/httpd/conf.d/welcome.conf.bk
[[email protected] ~]# service httpd start

等我会用httpd了再说吧，mzdd!

等节点加入到主机后，选择"向集群添加新主机" "管理当前主机"，选择刚加上的主机，下一步分发parcels。

服务

安装中出现的问题

1.

WARN [[email protected]:[email protected]]
[email protected] failed
to locate nozzleHOST_MONITORING
com.cloudera.cmon.MgmtServiceLocatorException: Could not find a
HOST_MONITORING
nozzle from SCM.
at
com.cloudera.cmon.MgmtServiceLocator.getNozzleIPC(MgmtServiceLocator.java:147)
at
com.cloudera.server.cmf.tsquery.NozzleRequest.<init>(NozzleRequest.java:50)

在安装时添加主机,出现了这个问题,如果只选CM节点,不出现这个问题.

2.

将jdk修改为1.8,之前是1.7

下载postgresql的jdbc驱动到/usr/share/java下,修改/etc/default/cloudera-scm-server的jar路径,添加postgresql的jdbc驱动.并在该文件中export JAVA_HOME

修改日志级别:

export CMF_ROOT_LOGGER="INFO,LOGFILE"

3.

INFO [JvmPauseMonitor:[email protected]] Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 1039ms: no GCs detected.
+ cat
+======================================================================+
| Error: JAVA_HOME is not set and Java could not be found |
+----------------------------------------------------------------------+
| Please download the latest Oracle JDK from the Oracle Java web site |
| > h  t t p : /  / w w w .or acl e . c o m/technetwork/java/javase/index.html < |
| |
| Cloudera Manager requires Java 1.6 or later. |
| NOTE: This script will find Oracle Java whether you install using |
| the binary or the RPM based installer. |
+======================================================================+
+ exit 1

修改成jdk1.8,然后并没有任务卵用.

到主机界面设置一下JDK

以上问题,用JAVA1.8重新安装CM AGENT后都解决了!!

来自为知笔记(Wiz)

时间： 2024-12-26 17:55:58

1.安装CDH5.12.x的相关文章

0025-CENTOS6.5安装CDH5.12.1(二)

温馨提示:要看高清无码套图,请使用手机打开并单击图片放大查看. 5.快速组件服务验证 5.1HDFS验证(mkdir+put+cat+get) mkdir操作: [[email protected]~]# hadoop fs -mkdir -p /fayson/test [[email protected] ~]# hadoop fs -ls / Found 3 items drwxr-xr-x - root supergroup 0 2017-09-0506:16 /fayson drwxrw

Ubuntu14.04用apt安装CDH5.1.2[Apache Hadoop 2.3.0]

--------------------------------------- 博文作者:迦壹 博客名称:Ubuntu14.04用apt安装CDH5.1.2[Apache Hadoop 2.3.0] 博客地址:http://idoall.org/home.php?mod=space&uid=1&do=blog&id=558 转载声明:可以转载, 但必须以超链接形式标明文章原始出处和作者信息及版权声明,谢谢合作! -----------------------------------

CDH5.12.0 中扩容增加计算节点

CDH5.12.0 中扩容增加计算节点 标签(空格分隔): 大数据平台构建 一: 环境准备 二: 增加节点环境系统初始化 三: 增加节点关于CDH5.12.0环境配置 四: 在CM 上面节点增加 一:环境的概述与准备 概述: 很多时候,企业的大数据环境(CDH5.12.0),根据使用的时间越来越长,空间会有不足的情况,集群的计算 能力也因此下降, 此时需要对将大数据的 集群环境进行扩容 增加计算节点. 1.1 系统逻辑部署图 1.2 安装文件详细信息列表: CM: cloudera-manage

CDH5.12.1 安装部署

###通过http://192.168.50.200:7180/cmf/login 访问CM控制台 4.CDH安装 4.1CDH集群安装向导 1.admin/admin登陆到CM 2.同意license协议,点击继续 3.选择60试用,点击继续 4.点击"继续" 5.输入主机IP或者名称,点击搜索找到主机名后点击继续 6.点击"继续" 7.使用parcel选项,点击"更多选项",点击"-"删除其他所有的地址,输入http://

Hadoop部署实践: &nbsp; 离线安装 CDH5.1 &nbsp; (待完成)

配置主机映射关系 [[email protected] ~]$ cat /etc/hosts 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4 10.15.5.200 master.hadoop 10.15.5.201 slave01.hadoop 10.15.5.202 slave02.hadoop 10.15.5.203 slave03.hadoop 如上有4台主机,每台hosts都已

Cloudrea manager5安装CDH5文档

一.主机规划.存储规划 服务器配置信息:CentOS6.5 最小化安装+development tools组包,其余组件yum安装即可. 二.系统设置如下: 1.服务器信息如下(/etc/hosts文件): 192.168.100.231 db01.chavin.king db01 192.168.100.232 db02.chavin.king db02 192.168.100.233 db03.chavin.king db03 192.168.100.234 db04.chavin.king

centos7.4下离线安装CDH5.7

(一)安装前的规划 (1)操作系统版本:centos7.4(64bit) [[email protected] etc]# more /etc/centos-release CentOS Linux release 7.4.1708 (Core) [[email protected] installPackage]# cat /proc/version Linux version 3.10.0-693.el7.x86_64 ([email protected]) (gcc version 4.8

大数据平台CentOS7+CDH5.12.1集群搭建

1.CM(Cloudera Manager)介绍 1.1 简介 Cloudera Manager是一个拥有集群自动化安装.中心化管理.集群监控.报警功能的一个工具,使得安装集群从几天的时间缩短在几个小时内,运维人员从数十人降低到几人以内,极大的提高集群管理的效率. 对比Apache / CDH / HDP: (1)Apache:运维麻烦,组件间兼容性需要自己调研.(一般大厂使用,技术实力雄厚,有专业的运维人员)(2)CDH:国内使用最多的版本,但CM不开源,但其实对中.小公司使用来说没有影响(建

Linux - Ubuntu中文输入法安装（12.04）

Ubuntu中文输入法安装(12.04) 本文地址:http://blog.csdn.net/caroline_wendy Ubuntu作为Linux常见的操作系统,是需要熟练使用的. 在安装过程中,输入法有时会出现问题,需要手动安装. 点击System Settings:找到Language Support. Install/Remove Language,找到Chinese(simplified),并点击Apply Changes,即可. 安装完毕之后,Keyboard input meth