CDH大数据集群环境搭建

笔者安装CDH集群是参照官方文档:https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_install_path_b.html

1.在集群中所有服务器（包含CM管理节点和各个agent节点）中安装JDK

wget http://download.oracle.com/otn/java/jdk/7u80-b15/jdk-7u80-linux-x64.tar.gz?AuthParam=1528156044_59d0d3a22c59b5ac6d9f0dddd4418808

export JAVA_HOME=/usr/local/jdk1.7.0_80
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export PATH=$JAVA_HOME/bin:$PATH

2.安装mysql

wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
rpm -ivh mysql-community-release-el7-5.noarch.rpm
yum update
yum install –y mysql-community-server
service mysqld start
systemctl enable mysqld.service

3.修改mysql的密码

mysqladmin password

4.修改/etc/my.cnf配置如下

[mysqld]
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
# symbolic-links = 0

key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1

max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M

#log_bin should be on a disk with enough free space. Replace ‘/var/lib/mysql/mysql_binary_log‘ with an appropriate path for your system
#and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log

# For MySQL version 5.1.8 or later. For older versions, reference MySQL documentation for configuration help.
binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

sql_mode=STRICT_ALL_TABLES

5.配置mysql驱动

mkdir -p /usr/share/java/
cd /usr/share/java/
wget https://cdn.mysql.com//Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
tar -xvf mysql-connector-java-5.1.46.tar.gz
cp mysql-connector-java-5.1.46.jar /usr/share/java/mysql-connector-java.jar

6.配置Cloudera Manager需要访问的mysql用户名和密码

其对应的sql脚本如下

create database amon DEFAULT CHARACTER SET utf8;
grant all on amon.* TO ‘amon‘@‘%‘ IDENTIFIED BY ‘amon‘;

create database rman DEFAULT CHARACTER SET utf8;
grant all on rman.* TO ‘rman‘@‘%‘ IDENTIFIED BY ‘rman!‘;

create database hive DEFAULT CHARACTER SET utf8;
grant all on hive.* TO ‘hive‘@‘%‘ IDENTIFIED BY ‘hive‘;

create database sentry DEFAULT CHARACTER SET utf8;
grant all on sentry.* TO ‘sentry‘@‘%‘ IDENTIFIED BY ‘sentry‘;

create database nav DEFAULT CHARACTER SET utf8;
grant all on nav.* TO ‘nav‘@‘%‘ IDENTIFIED BY ‘nav‘;

create database navms DEFAULT CHARACTER SET utf8;
grant all on navms.* TO ‘navms‘@‘%‘ IDENTIFIED BY ‘navms‘;

create database oozie DEFAULT CHARACTER SET utf8;
grant all on oozie.* TO ‘oozie‘@‘%‘ IDENTIFIED BY ‘oozie‘;

create database hue DEFAULT CHARACTER SET utf8;
grant all on hue.* TO ‘hue‘@‘%‘ IDENTIFIED BY ‘hue‘;

create database cmf DEFAULT CHARACTER SET utf8;
grant all on cmf.* TO ‘cmf‘@‘%‘ IDENTIFIED BY ‘cmf‘;

flush privileges;

7.安装Cloudera Manager

yum install cloudera-manager-daemons cloudera-manager-server

修改cloudera server db配置/etc/cloudera-scm-server/ db.properties如下
[[email protected] cloudera-scm-server]# cat db.properties
com.cloudera.cmf.db.type=mysql:3306
com.cloudera.cmf.db.host=cdh1
com.cloudera.cmf.db.name=cmf
com.cloudera.cmf.db.user=cmf
com.cloudera.cmf.db.password=cmf
com.cloudera.cmf.db.setupType=EXTERNAL

8.初始化Cloudera Manager Mysql脚本

/usr/share/cmf/schema/scm_prepare_database.sh mysql cmf cmf cmf

9.启动Cloudera Manager Server

service cloudera-scm-server start

10.登录到cmf:   http://cdh:7180/cmf

原文地址：https://www.cnblogs.com/qizhelongdeyang/p/9141531.html