GreenPlum 大数据平台--安装

1. 环境准备

  01, 安装包准备:

    Greenplum :  >>>>链接地址

    Pgadmin客户端 :  >>>链接地址

    greenplum-cc-web监控: >>>>链接地址

  02,节点说明

服务器ip 主机名 角色 系统版本
192.168.0.221 greenplum01 master centos7
192.168.0.222 greenplum02 Segment/standby centos7
192.168.0.223 greenplum03 Segment centos7

  03,host文件配置(All)

echo -e "192.168.0.221 greenplum01\n192.168.0.222 greenplum02\n192.168.0.223 greenplum03" >>/etc/hosts

scp /etc/hosts 192.168.0.222:/etc/hosts
scp /etc/hosts 192.168.0.223:/etc/hosts

  04,创建用户(ALL) 

[root@greenplum01 ~]# groupadd -g 530 gpadmin
[root@greenplum01 ~]# useradd -g 530 -m -d /home/gpadmin -s /bin/bash gpadmin
[root@greenplum01 ~]# passwd gpadmin
Changing password for user gpadmin.
New password:
BAD PASSWORD: The password is shorter than 8 characters
Retype new password:
passwd: all authentication tokens updated successfully.

[root@greenplum02 ~]# groupadd -g 530 gpadmin
[root@greenplum02 ~]# useradd -g 530 -m -d /home/gpadmin -s /bin/bash gpadmin
[root@greenplum02 ~]# passwd gpadmin
Changing password for user gpadmin.
New password:
BAD PASSWORD: The password is shorter than 8 characters
Retype new password:
passwd: all authentication tokens updated successfully.

[root@greenplum03 ~]# groupadd -g 530 gpadmin
[root@greenplum03 ~]# useradd -g 530 -m -d /home/gpadmin -s /bin/bash gpadmin
[root@greenplum03 ~]# passwd gpadmin
Changing password for user gpadmin.
New password:
BAD PASSWORD: The password is shorter than 8 characters
Retype new password:
passwd: all authentication tokens updated successfully.

  05,内核参数修改(ALL)

echo ‘
kernel.shmmax = 500000000
kernel.shmmni = 4096
kernel.shmall = 4000000000
kernel.sem = 250 512000 100 2048
kernel.sysrq = 1
kernel.core_uses_pid = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.msgmni = 2048
net.ipv4.tcp_syncookies = 1
net.ipv4.ip_forward = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.conf.all.arp_filter = 1
net.ipv4.ip_local_port_range = 1025 65535
net.core.netdev_max_backlog = 10000
net.core.rmem_max = 2097152
net.core.wmem_max = 2097152
vm.overcommit_memory = 2
‘>>/etc/sysctl.confsysctl -

  06.打开数限制(ALL)

echo ‘
* soft nofile 65536
* hard nofile 65536
* soft nproc 131072
* hard nproc 131072
‘>>/etc/security/limits.conf

  07,关闭防火墙,selinux(ALL)

systemctl disable firewalld.service
systemctl stop firewalld.service
setenforce 0

  08,创建需要的目录(ALL)

mkdir /greenplum
chown -R gpadmin:gpadmin /greenplum

二,安装greenplum(master节点)

  01,安装  

[root@greenplum01 ~]# ls
anaconda-ks.cfg  greenplum-cc-web-4.6.1-LINUX-x86_64      initial-setup-ks.cfg  Templates
Desktop          greenplum-cc-web-4.6.1-LINUX-x86_64.zip  Music                 Videos
Documents        greenplum-db-5.16.0-rhel7-x86_64.rpm     Pictures
Downloads    Public
[root@greenplum01 ~]# rpm -ivh greenplum-db-5.16.0-rhel7-x86_64.rpm
Preparing...                          ################################# [100%]
Updating / installing...
   1:greenplum-db-5.16.0-1            ################################# [100%]

  02,配置greeplum

chown -R gpadmin:gpadmin /usr/local/
su - gpadmin
mkdir -p /home/gpadmin/conf
vi /home/gpadmin/conf/hostlist
greenplum01
greenplum02
greenplum03

  再创建一个seg_hosts ,包含所有的Segment Host的主机名:

vi /home/gpadmin/conf/seg_hosts
greenplum02
greenplum03

   03,配置链接

[gpadmin@greenplum01 ~]$ source /usr/local/greenplum-db/greenplum_path.sh
[gpadmin@greenplum01 ~]$ gpssh-exkeys -f /home/gpadmin/conf/hostlist
[STEP 1 of 5] create local ID and authorize on local host

[STEP 2 of 5] keyscan all hosts and update known_hosts file

[STEP 3 of 5] authorize current user on remote hosts
  ... send to greenplum02
  ***
  *** Enter password for greenplum02:
  ... send to greenplum03

[STEP 4 of 5] determine common authentication file content

[STEP 5 of 5] copy authentication files to all remote hosts
  ... finished key exchange with greenplum02
  ... finished key exchange with greenplum03

[INFO] completed successfully

    验证链接:

[gpadmin@greenplum01 ~]$ ssh greenplum02
[gpadmin@greenplum02 ~]$ exit
logout
Connection to greenplum02 clsed.
[gpadmin@greenplum01 ~]$ ssh greenplum03
[gpadmin@greenplum02 ~]$ exit
logout
Connection to greenplum03 clsed.

三,安装其他节点的DB

  01,权限授予

chown -R gpadmin:gpadmin /usr/local
chown -R gpadmin:gpadmin /greenplum

  02,数据打包

cd /usr/localtar zcvf gp.tar greenplum-db-5.16.0/
gpscp -f /home/gpadmin/conf/seg_hosts gp.tar =:/usr/local/

  03,解压缩

[gpadmin@greenplum01 ~]$ source /usr/local/greenplum-db/greenplum_path.sh
[gpadmin@greenplum01 ~]$ gpssh -f /home/gpadmin/conf/seg_hosts
=> cd /usr/local
[greenplum03]
[greenplum02]
=> tar -xf gp.tar
[greenplum03]
[greenplum02]
=> ln -s ./greenplum-db-5.16.0/ greenplum-db
[greenplum03]
[greenplum02]
=> ll
[greenplum03] total 271720
[greenplum03] drwxr-xr-x.  2 gpadmin gpadmin         6 Nov  5  2016 bin
[greenplum03] drwxr-xr-x.  2 gpadmin gpadmin         6 Nov  5  2016 etc
[greenplum03] drwxr-xr-x.  2 gpadmin gpadmin         6 Nov  5  2016 games
[greenplum03] -rw-rw-r--   1 gpadmin gpadmin 278239886 Mar 12 23:18 gp.tar
[greenplum03] lrwxrwxrwx   1 gpadmin gpadmin        22 Mar 12 23:21 greenplum-db -> ./greenplum-db-5.16.0/
[greenplum03] drwxr-xr-x  11 gpadmin gpadmin       138 Mar 12 23:03 greenplum-db-5.16.0
[greenplum03] drwxr-xr-x.  2 gpadmin gpadmin         6 Nov  5  2016 include
[greenplum03] drwxr-xr-x.  2 gpadmin gpadmin         6 Nov  5  2016 lib
[greenplum03] drwxr-xr-x.  2 gpadmin gpadmin         6 Nov  5  2016 lib64
[greenplum03] drwxr-xr-x.  2 gpadmin gpadmin         6 Nov  5  2016 libexec
[greenplum03] drwxr-xr-x.  2 gpadmin gpadmin         6 Nov  5  2016 sbin
[greenplum03] drwxr-xr-x.  5 gpadmin gpadmin        49 Feb 18 22:35 share
[greenplum03] drwxr-xr-x.  2 gpadmin gpadmin         6 Nov  5  2016 src
[greenplum03] drwxrwxr-x   3 gpadmin gpadmin        19 Mar 12 23:21 usr
[greenplum02] total 271720
[greenplum02] drwxr-xr-x.  2 gpadmin gpadmin         6 Nov  5  2016 bin
[greenplum02] drwxr-xr-x.  2 gpadmin gpadmin         6 Nov  5  2016 etc
[greenplum02] drwxr-xr-x.  2 gpadmin gpadmin         6 Nov  5  2016 games
[greenplum02] -rw-rw-r--   1 gpadmin gpadmin 278239886 Mar 12 23:18 gp.tar
[greenplum02] lrwxrwxrwx   1 gpadmin gpadmin        22 Mar 12 23:21 greenplum-db -> ./greenplum-db-5.16.0/
[greenplum02] drwxr-xr-x  11 gpadmin gpadmin       138 Mar 12 23:03 greenplum-db-5.16.0
[greenplum02] drwxr-xr-x.  2 gpadmin gpadmin         6 Nov  5  2016 include
[greenplum02] drwxr-xr-x.  2 gpadmin gpadmin         6 Nov  5  2016 lib
[greenplum02] drwxr-xr-x.  2 gpadmin gpadmin         6 Nov  5  2016 lib64
[greenplum02] drwxr-xr-x.  2 gpadmin gpadmin         6 Nov  5  2016 libexec
[greenplum02] drwxr-xr-x.  2 gpadmin gpadmin         6 Nov  5  2016 sbin
[greenplum02] drwxr-xr-x.  5 gpadmin gpadmin        49 Feb 18 22:35 share
[greenplum02] drwxr-xr-x.  2 gpadmin gpadmin         6 Nov  5  2016 src
[greenplum02] drwxrwxr-x   3 gpadmin gpadmin        19 Mar 12 23:21 usr

四,初始化数据库

  01,创建需要的资源目录

[gpadmin@greenplum01 ~]$ source /usr/local/greenplum-db/greenplum_path.sh
[gpadmin@greenplum01 ~]$ gpssh -f /home/gpadmin/conf/hostlist
=> mkdir -p /greenplum/data/master
[greenplum01]
[greenplum02]
[greenplum03]
=> mkdir -p /greenplum/data/primary
[greenplum01]
[greenplum02]
[greenplum03]
=> mkdir -p /greenplum/data/mirror
[greenplum01]
[greenplum02]
[greenplum03]
=> mkdir -p /greenplum/data2/primary
[greenplum01]
[greenplum02]
[greenplum03]
=> mkdir -p /greenplum/data2/mirror
[greenplum01]
[greenplum02]
[greenplum03]
=>

  02,节点配置环境(all)

[gpadmin@greenplum01 ~]$ vim .bash_profile
source /usr/local/greenplum-db/greenplum_path.sh
export MASTER_DATA_DIRECTORY=/greenplum/data/master/gpseg-1
export GPPORT=5432
export PGDATABASE=gp_db
[gpadmin@greenplum01 ~]$ scp .bash_profile greenplum02:/home/gpadmin/
.bash_profile                                                              100%  344   738.4KB/s   00:00
[gpadmin@greenplum01 ~]$ scp .bash_profile greenplum03:/home/gpadmin/
.bash_profile                                                              100%  344   828.8KB/s   00:00
[gpadmin@greenplum01 ~]$ source .bash_profile
[gpadmin@greenplum02 ~]$ source .bash_profile
[gpadmin@greenplum03 ~]$ source .bash_profile

  03,初始化检查

    

  

  

  

  

原文地址:https://www.cnblogs.com/kingle-study/p/10522076.html

时间: 2024-10-11 21:59:03

GreenPlum 大数据平台--安装的相关文章

GreenPlum 大数据平台--segment 失效问题恢复

1,问题检查 [[email protected] conf]$ psql -c "select * from gp_segment_configuration where status='d'" dbid | content | role | preferred_role | mode | status | port | hostname | address | replication_por t ------+---------+------+----------------+--

大数据平台一键安装OS【定制化OS镜像制作】

 定制化 大数据平台一键安装OS 大数据平台一键安装OS系列 大数据平台一键安装OS[搭建脚本篇] 定制化OS镜像制作 1.操作环境 操作环境:VMware Workstarion 9 and vSphere client 系统:CentOS -6.8-x86_64 工具:gconf-editor anaconda repodata createrepo mkisofs rsync 2.主要思路 定制化是通过kickstart脚本来实现的,linux系统安装完毕后在root目录下会生成anaco

大数据平台一键安装OS【搭建脚本篇】

定制化大数据平台一键安装OS 大数据平台一键安装OS系列 大数据平台一键安装OS[定制化OS镜像制作] 搭建脚本篇 脚本命名随便,只要能看懂就行 镜像包说明 os1.1.iso 用于安装服务端和客户端操作系统 hdp.iso 用于安装hdp服务 1.架构图: 服务端 客户端 2.原理分析 安装系统-安装本地yum源-安装http源-安装并配置ntp服务-配置秘钥-安装jdk-安装mysql-编译安装ambari平台 3.服务端脚本分析 (1)create_local_yum.sh #安装本地yu

大数据平台HDP-2.6.4安装与配置

大数据平台HDP-2.6.4 安装 与配置 标签(空格分隔): 大数据平台构建 一: 关于HDP的介绍 二: 系统环境介绍 三: 系统环境初始化 四: 安装HDP-2.6.4平台 五: 启用NameNode HA 与 YARN HA 一:关于HDP的介绍 1.1:hdp 是什么 HDP全称叫做Hortonworks Data Platform. Hortonworks数据平台是一款基于Apache Hadoop的是开源数据平台,提供大数据云存储,大数据处理和分析等服务.该平台是专门用来应对多来源

大数据平台CDH5.14.2 的安装配置

大数据平台CDH5.14.2 的安装配置 标签(空格分隔): 大数据平台构建 一:系统环境初始化 二:安装CDH5.14.2 平台 三:分配主机与分配角色 一: 系统环境初始化 1.1: 系统环境介绍 系统: CentOS7.5X64 cat /etc/hosts --- 172.17.100.11 node-01.flyfish 172.17.100.12 node-02.flyfish 172.17.100.13 node-03.flyfish 172.17.100.14 node-04.f

大数据平台CDH6.1.0 安装配置

大数据平台 CDH6.1.0 安装配置 标签(空格分隔): 大数据平台构建 一: 关于CDH6.1.0 的介绍 二: 系统环境初始化 三: 安装CDH6.1.0所需环境 四: 配置CDH6.1.0的cloudera-manager安装 一:关于CDH6.1.0 的介绍 Cloudera在前天12月19日,对外宣布正式发布Cloudera Enterprise 6.1包含大数据生态版本如下: 1.组件版本:Kafka 2.0, Spark 2.4, HBase 2.1.1, Accumulo 1.

ambari 搭建hadoop大数据平台系列4-配置ambari-server

ambari 搭建hadoop大数据平台系列4-配置ambari-server,分为三部分: 官网:  https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-installation/content/download_the_ambari_repo_lnx7.html 安装ambari-server  配置ambari-server  命令;ambari-server setup 启动ambari-server 命令

Spark大型项目实战:电商用户行为分析大数据平台

本项目主要讲解了一套应用于互联网电商企业中,使用Java.Spark等技术开发的大数据统计分析平台,对电商网站的各种用户行为(访问行为.页面跳转行为.购物行为.广告点击行为等)进行复杂的分析.用统计分析出来的数据,辅助公司中的PM(产品经理).数据分析师以及管理人员分析现有产品的情况,并根据用户行为分析结果持续改进产品的设计,以及调整公司的战略和业务.最终达到用大数据技术来帮助提升公司的业绩.营业额以及市场占有率的目标. 1.课程研发环境 开发工具: Eclipse Linux:CentOS 6

大数据平台在互联网行业的应用

本文是58同城信息系统部高级经理余中洋对大数据在互联网行业应用的总结.以实际经验讲解了大数据从雏形到发展,以及发展过程中传统数据仓库到大数据平台的转化和大数据的可视化应用. 大数据显雏形--发展 所有公司在一开始时,数据的建设都是比较落后的,但随着互联网的环境推动,以及公司计划2013年在纽约交易所上市,这时候面临一个很大的问题.在上市之前,公司的数据,无论是流量数据还是财务产品数据都需要经过非常严格的审计.为此,我们开始整理数据,做数据报表,花费一年多的时间做数据整理.数据规范,慢慢的大数据的