Centos7使用kubeadm 安装多主高可用kubernets:v.1.11集群

实验环境介绍：

本次实验环境是5个节点 3台master 2台node节点：

k8smaster01 192.168.111.128 软件：etcd k8smaster haproxy keepalived
k8smaster02 192.168.111.129 软件：etcd k8smaster haproxy keepalived
k8smaster03 192.168.111.130 软件：etcd k8smaster haproxy keepalived
k8snode01 192.168.111.131  软件：k8snode
k8snode02 192.168.111.132  软件：k8snode

VIP： 192.168.111.100

系统优化（在所有节点上操作）

关闭防火墙

systemctl stop firewalld.service
systemctl disable firewalld.service

关闭SELINUX和swap，优化内核参数

 sed -i ‘s/SELINUX=enforcing/SELINUX=disabled/‘ /etc/selinux/config
 setenforce 0

# 临时关闭swap # 永久关闭 注释/etc/fstab文件里swap相关的行 swapoff -a

# 配置转发相关参数，否则可能会出错cat <<EOF >  /etc/sysctl.d/k8s.confnet.bridge.bridge-nf-call-ip6tables = 1net.bridge.bridge-nf-call-iptables = 1vm.swappiness=0EOFsysctl --system

# 加载ipvs相关内核模块# 如果重新开机，需要重新加载modprobe ip_vsmodprobe ip_vs_rrmodprobe ip_vs_wrrmodprobe ip_vs_shmodprobe nf_conntrack_ipv4lsmod | grep ip_vs

配置yum源

wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo
mv /etc/yum.repos.d/epel.repo /etc/yum.repos.d/epel.repo.backup
mv /etc/yum.repos.d/epel-testing.repo /etc/yum.repos.d/epel-testing.repo.backup
wget -O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

yum clean all && yum makecache 

sudo yum install -y yum-utils device-mapper-persistent-data lvm2

配置hosts解析

192.168.111.128 k8smaster01
192.168.111.129 k8smaster02
192.168.111.130 k8smaster03
192.168.111.131 k8snode01
192.168.111.132 k8snode02

安装docker

v1.11.1版本推荐使用docker v17.03,v1.11,v1.12,v1.13, 也可以使用，再高版本官网不推荐使用，但是可以忽略。

这里安装18.06.0-ce

yum -y install docker-ce
systemctl enable docker && systemctl restart docker

安装 kubeadm, kubelet 和 kubectl(所有节点)

yum install -y kubelet kubeadm kubectl ipvsadm
systemctl enable kubelet && systemctl start kubelet

配置haproxy代理和keepalived（如下操作在所有master节点上操作）

# 拉取haproxy镜像
docker pull haproxy:1.7.8-alpine
cat >/etc/haproxy/haproxy.cfg<<EOF
global
  log 127.0.0.1 local0 err
  maxconn 5000
  uid 99
  gid 99
  #daemon
  nbproc 1
  pidfile haproxy.pid

defaults
  mode http
  log 127.0.0.1 local0 err
  maxconn 5000
  retries 3
  timeout connect 5s
  timeout client 30s
  timeout server 30s
  timeout check 2s

listen admin_stats
  mode http
  bind 0.0.0.0:1080
  log 127.0.0.1 local0 err
  stats refresh 30s
  stats uri     /haproxy-status
  stats realm   Haproxy\ Statistics
  stats auth    will:will
  stats hide-version
  stats admin if TRUE

frontend k8s-https
  bind 0.0.0.0:8443
  mode tcp
  #maxconn 50000
  default_backend k8s-https

backend k8s-https
  mode tcp
  balance roundrobin
  server k8smaster01 192.168.111.128:6443 weight 1 maxconn 1000 check inter 2000 rise 2 fall 3
  server k8smaster02 192.168.111.129:6443 weight 1 maxconn 1000 check inter 2000 rise 2 fall 3
  server k8smaster03 192.168.111.130:6443 weight 1 maxconn 1000 check inter 2000 rise 2 fall 3
EOF
# 启动haproxy
docker run -d --name my-haproxy -v /etc/haproxy:/usr/local/etc/haproxy:ro -p 8443:8443 -p 1080:1080 --restart always haproxy:1.7.8-alpine
# 拉取keepalived镜像
docker pull osixia/keepalived:1.4.4

# 启动
# 载入内核相关模块
lsmod | grep ip_vs
modprobe ip_vs

# 启动keepalived
# ens33为本次实验192.168.111.0/24网段的所在网卡
docker run --net=host --cap-add=NET_ADMIN -e KEEPALIVED_INTERFACE=ens33 -e KEEPALIVED_VIRTUAL_IPS="#PYTHON2BASH:[‘192.168.111.100‘]" -e KEEPALIVED_UNICAST_PEERS="#PYTHON2BASH:[‘192.168.111.128‘,‘192.168.111.129‘,‘192.168.111.130‘]" -e KEEPALIVED_PASSWORD=hello --name k8s-keepalived --restart always -d osixia/keepalived:1.4.4

# 此时会配置 192.168.111.100 到其中一台机器
# ping测试
ping  192.168.111.100

# 如果失败后清理后，重新实验
#docker rm -f k8s-keepalived
#ip a del 192.168.111.100/32 dev ens33

配置kubelet(所有节点操作)

# 配置kubelet使用国内pause镜像
# 配置kubelet的cgroups

cat >/etc/sysconfig/kubelet<<EOF
KUBELET_EXTRA_ARGS="--cgroup-driver=cgroupfs  --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.1"
EOF

# 启动
systemctl daemon-reload
systemctl enable kubelet && systemctl restart kubelet

配置k8smaster01(192.168.111.128上操作)

cd /etc/kubernetes
# 生成配置文件
cat >kubeadm-master.config<<EOF
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.1
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers

apiServerCertSANs:
- "k8smaster01"
- "k8smaster02"
- "k8smaster03"
- "192.168.111.128"
- "192.168.111.129"
- "192.168.111.130"
- "192.168.111.100"
- "127.0.0.1"

api:
  advertiseAddress: 192.168.111.128
  controlPlaneEndpoint: 192.168.111.100:8443

etcd:
  local:
    extraArgs:
      listen-client-urls: "https://127.0.0.1:2379,https://192.168.111.128:2379"
      advertise-client-urls: "https://192.168.111.128:2379"
      listen-peer-urls: "https://192.168.111.128:2380"
      initial-advertise-peer-urls: "https://192.168.111.128:2380"
      initial-cluster: "k8smaster01=https://192.168.111.128:2380"
    serverCertSANs:
      - k8smaster01
      - 192.168.111.128
    peerCertSANs:
      - k8smaster01
      - 192.168.111.128

controllerManagerExtraArgs:
  node-monitor-grace-period: 10s
  pod-eviction-timeout: 10s

networking:
  podSubnet: 10.244.0.0/16

kubeProxy:
  config:
    mode: ipvs
    # mode: iptables
EOF

# 提前拉取镜像
# 如果执行失败 可以多次执行
kubeadm config images pull --config kubeadm-master.config

# 初始化
# 注意保存返回的 join 命令
kubeadm init --config kubeadm-master.config

# 初始化失败时使用
#kubeadm reset

# 将ca相关文件传至其他master节点cd /etc/kubernetes/pki/
USER=root
CONTROL_PLANE_IPS="k8smaster02 k8smaster03"
for host in ${CONTROL_PLANE_IPS}; do　　 ssh "${USER}"@$host "mkdir -p /etc/kubernetes/pki/etcd"
    scp ca.crt ca.key  sa.key  sa.pub front-proxy-ca.crt front-proxy-ca.key  "${USER}"@$host:/etc/kubernetes/pki/
    scp etcd/ca.crt etcd/ca.key "${USER}"@$host:/etc/kubernetes/pki/etcd/　　scp ../admin.conf "${USER}"@$host:/etc/kubernetes/

done

kubeadm init失败解决：

将阿里云image tag成官方的image，即可解决init失败问题。(v1.11.0有此问题)

docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver-amd64:v1.11.1 k8s.gcr.io/kube-apiserver-amd64:v1.11.1docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy-amd64:v1.11.1 k8s.gcr.io/kube-proxy-amd64:v1.11.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/etcd-amd64:3.2.18 k8s.gcr.io/etcd-amd64:3.2.18
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler-amd64:v1.11.1 k8s.gcr.io/kube-scheduler-amd64:v1.11.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager-amd64:v1.11.1 k8s.gcr.io/kube-controller-manager-amd64:v1.11.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.1.3 k8s.gcr.io/coredns:1.1.3
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.1 k8s.gcr.io/pause-amd64:3.1
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.1 k8s.gcr.io/pause:3.1

配置k8smaster02(192.168.111.129上操作)

cd /etc/kubernetes
# 生成配置文件
cat >kubeadm-master.config<<EOF
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.1
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers

apiServerCertSANs:
- "k8smaster01"
- "k8smaster02"
- "k8smaster03"
- "192.168.111.128"
- "192.168.111.129"
- "192.168.111.130"
- "192.168.111.100"
- "127.0.0.1"

api:
  advertiseAddress: 192.168.111.129
  controlPlaneEndpoint: 192.168.111.100:8443

etcd:
  local:
    extraArgs:
      listen-client-urls: "https://127.0.0.1:2379,https://192.168.111.129:2379"
      advertise-client-urls: "https://192.168.111.129:2379"
      listen-peer-urls: "https://192.168.111.129:2380"
      initial-advertise-peer-urls: "https://192.168.111.129:2380"
      initial-cluster: "k8smaster01=https://192.168.111.128:2380,k8smaster02=https://192.168.111.129:2380"
      initial-cluster-state: existing
    serverCertSANs:
      - k8smaster02
      - 192.168.111.129
    peerCertSANs:
      - k8smaster02
      - 192.168.111.129

controllerManagerExtraArgs:
  node-monitor-grace-period: 10s
  pod-eviction-timeout: 10s

networking:
  podSubnet: 10.244.0.0/16

kubeProxy:
  config:
    mode: ipvs
    # mode: iptables
EOF

# 配置kubelet
kubeadm alpha phase certs all --config kubeadm-master.config
kubeadm alpha phase kubelet config write-to-disk --config kubeadm-master.config
kubeadm alpha phase kubelet write-env-file --config kubeadm-master.config
kubeadm alpha phase kubeconfig kubelet --config kubeadm-master.config
systemctl restart kubelet

# 添加etcd到集群中
export KUBECONFIG=/etc/kubernetes/admin.conf
kubectl exec -n kube-system etcd-k8smaster01 -- etcdctl --ca-file /etc/kubernetes/pki/etcd/ca.crt --cert-file /etc/kubernetes/pki/etcd/peer.crt --key-file /etc/kubernetes/pki/etcd/peer.key --endpoints=https://192.168.111.128:2379 member add k8smaster02 https://192.168.111.129:2380

# 提前拉取镜像
kubeadm config images pull --config kubeadm-master.config

# 部署
kubeadm alpha phase kubeconfig all --config kubeadm-master.config
kubeadm alpha phase controlplane all --config kubeadm-master.config
kubeadm alpha phase mark-master --config kubeadm-master.config

配置k8smaster03(192.168.111.130上操作)

cd /etc/kubernetes
# 生成配置文件
cat >kubeadm-master.config<<EOF
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.1
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers

apiServerCertSANs:
- "k8smaster01"
- "k8smaster02"
- "k8smaster03"
- "192.168.111.128"
- "192.168.111.129"
- "192.168.111.130"
- "192.168.111.100"
- "127.0.0.1"

api:
  advertiseAddress: 192.168.111.130
  controlPlaneEndpoint: 192.168.111.100:8443

etcd:
  local:
    extraArgs:
      listen-client-urls: "https://127.0.0.1:2379,https://192.168.111.130:2379"
      advertise-client-urls: "https://192.168.111.130:2379"
      listen-peer-urls: "https://192.168.111.130:2380"
      initial-advertise-peer-urls: "https://192.168.111.130:2380"
      initial-cluster: "k8smaster01=https://192.168.111.128:2380,k8smaster02=https://192.168.111.129:2380,k8smaster03=https://192.168.111.130:2380"
      initial-cluster-state: existing
    serverCertSANs:
      - k8smaster03
      - 192.168.111.130
    peerCertSANs:
      - k8smaster03
      - 192.168.111.130

controllerManagerExtraArgs:
  node-monitor-grace-period: 10s
  pod-eviction-timeout: 10s

networking:
  podSubnet: 10.244.0.0/16

kubeProxy:
  config:
    mode: ipvs
    # mode: iptables
EOF
# 配置kubelet
kubeadm alpha phase certs all --config kubeadm-master.config
kubeadm alpha phase kubelet config write-to-disk --config kubeadm-master.config
kubeadm alpha phase kubelet write-env-file --config kubeadm-master.config
kubeadm alpha phase kubeconfig kubelet --config kubeadm-master.config
systemctl restart kubelet

# 添加etcd到集群中
KUBECONFIG=/etc/kubernetes/admin.conf
kubectl exec -n kube-system etcd-k8smaster01 -- etcdctl --ca-file /etc/kubernetes/pki/etcd/ca.crt --cert-file /etc/kubernetes/pki/etcd/peer.crt --key-file /etc/kubernetes/pki/etcd/peer.key --endpoints=https://192.168.111.128:2379 member add k8smaster03 https://192.168.111.130:2380

# 提前拉取镜像
kubeadm config images pull --config kubeadm-master.config

# 部署
kubeadm alpha phase kubeconfig all --config kubeadm-master.config
kubeadm alpha phase controlplane all --config kubeadm-master.config
kubeadm alpha phase mark-master --config kubeadm-master.config

配置使用kubectl (master 任意节点执行)

rm -rf $HOME/.kube
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# 查看node节点

　kubectl get nodes

# 只有网络插件也安装配置完成之后，才能会显示为ready状态

# 设置master允许部署应用pod，参与工作负载，现在可以部署其他系统组件

配置使用网络插件（任意master节点上操作）

# 下载配置
cd /etc/kubernetes
mkdir flannel && cd flannel
wget https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml

# 修改配置
# 此处的ip配置要与上面kubeadm的pod-network一致
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }

# 修改镜像
image: registry.cn-shanghai.aliyuncs.com/gcr-k8s/flannel:v0.10.0-amd64

# 如果Node有多个网卡的话，参考flannel issues 39701，
# https://github.com/kubernetes/kubernetes/issues/39701
# 目前需要在kube-flannel.yml中使用--iface参数指定集群主机内网网卡的名称，
# 否则可能会出现dns无法解析。容器无法通信的情况，需要将kube-flannel.yml下载到本地，
# flanneld启动参数加上--iface=<iface-name>
    containers:
      - name: kube-flannel
        image: registry.cn-shanghai.aliyuncs.com/gcr-k8s/flannel:v0.10.0-amd64
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        - --iface=ens33

# 启动
kubectl apply -f kube-flannel.yml

# 查看
kubectl get pods --namespace kube-system
kubectl get svc --namespace kube-system

配置node节点加入集群（所有的node节点上操作）

以下上master生成的，与你环境可能不符合  kubeadm join 192.168.111.100:8443 --token uf9oul.7k4csgxe5p7upvdb --discovery-token-ca-cert-hash sha256:36bc173b46eb0545fc30dd5db2d27dab70a257bd406fd791647d991a69454595

node节点报错处理办法：

tail -f /var/log/message

Jul 19 07:52:21 localhost kubelet: E0726 07:52:21.336281   10018 summary.go:102] Failed to get system container stats for "/system.slice/kubelet.service": failed to get cgroup stats for "/system.slice/kubelet.service": failed to get container info for "/system.slice/kubelet.service": unknown container "/system.slice/kubelet.service"

在kubelet配置文件追加以下配置

/etc/sysconfig/kubelet

# Append configuration in Kubelet
--runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice

这样一个集群环境配置完成里，其余的是自己添加附件吧。

原文地址：https://www.cnblogs.com/xzkzzz/p/9502502.html

时间： 2024-08-06 17:37:58

Centos7使用kubeadm 安装多主高可用kubernets:v.1.11集群的相关文章

高可用，多路冗余GFS2集群文件系统搭建详解

2014.06 标签:GFS2 multipath 集群文件系统 cmirror 实验拓扑图: 实验原理: 实验目的:通过RHCS集群套件搭建GFS2集群文件系统,保证不同节点能够同时对GFS2集群文件系统进行读取和写入,其次通过multipath实现node和FC,FC和Share Storage之间的多路冗余,最后实现存储的mirror复制达到高可用. GFS2:全局文件系统第二版,GFS2是应用最广泛的集群文件系统.它是由红帽公司开发出来的,允许所有集群节点并行访问.元数据通常会保存在共享

通过keepalived搭建高可用的LVS负载均衡集群

一.keepalived软件简介 keepalived是基于vrrp协议实现高可用功能的一种软件,它可以解决单点故障的问题,通过keepalived搭建一个高可用的LVS负载均衡集群时,keepalived还能检测后台服务器的运行状态. 二.vrrp协议原理简介 vrrp(虚拟路由器冗余协议),是为了解决网络上静态路由出现的单点故障的问题,举个例子,如下图 主机A和B均在同一个局域网内,C和D均是该局域网的网关,即A和B想与外网通信,需指网关到C或D,那究竟指向C好还是指向D好呢?都不好!当指向

高可用结合gfs2,，实现集群文件系统以及集群逻辑卷。

为什么要集群文件系统,在什么场景中适用我就用一句话来概括,当多个节点需要读写同一个文件系统时,需要使用集群文件系统,它可以将文件系统持有的锁信息传递到各个节点. 实验一.将iSCSI共享出来的磁盘,做成gfs2文件系统,实现多个节点可挂载同一个文件系统,保证数据同步 实验平台:RHEL6 环境拓扑: ansible配置前面文章有讲到    略 iSCSI服务器配置    略 用控制端让三个节点安装需要的程序包. ansible all -m shell -a 'yum install cman

构建高可用的LVS负载均衡集群 入门篇

一.LVS简介 LVS是Linux Virtual Server的简称,也就是Linux虚拟服务器, 是一个由章文嵩博士发起的自由软件项目,它的官方站点是www.linuxvirtualserver.org.现在LVS已经是 Linux标准内核的一部分,在Linux2.4内核以前,使用LVS时必须要重新编译内核以支持LVS功能模块,但是从Linux2.4内核以后,已经完全内置了LVS的各个功能模块,无需给内核打任何补丁,可以直接使用LVS提供的各种功能. LVS 集群采用IP负载和基于内容请求分

构建高可用的LVS负载均衡集群 进阶篇

一.lvs组件介绍 lvs的组件由两部分组成:工作在内核空间的ipvs模块和工作在用户空间ipvsadm工具.其中ipvsadm是规则生成工具,而ipvs是一个使规则生效的工具. 二.ipvsadm详解 构建高可用的LVS负载均衡集群 进阶篇,布布扣,bubuko.com

kubeadm创建高可用kubernetes v1.12.0集群

节点规划 主机名 IP Role k8s-master01 10.3.1.20 etcd.Master.Node.keepalived k8s-master02 10.3.1.21 etcd.Master.Node.keepalived k8s-master03 10.3.1.25 etcd.Master.Node.keepalived VIP 10.3.1.29 None 版本信息: OS::Ubuntu 16.04 Docker:17.03.2-ce k8s:v1.12 来自官网的高可用架构

使用Nginx+Keepalived组建高可用负载平衡Web server集群

一,首先说明一下网络拓扑结构: 1,Nginx 反向代理Server(HA):      ①Nginx master:192.168.1.157      ②Nginx backup:192.168.1.158         虚拟IP统一为:192.168.1.110   2,web服务器:      192.168.1.160 ,192.168.1.161,192.168.1.162     即web服务器,已配置好 Tomcat(Jboss等皆可)和java程序 3,mysql 数据库Se

keepalived + nginx组建高可用负载平衡Web server集群

1 nginx负载均衡高可用 1.1 什么是负载均衡高可用 nginx作为负载均衡器,所有请求都到了nginx,可见nginx处于非常重点的位置,如果nginx服务器宕机后端web服务将无法提供服务,影响严重. 为了屏蔽负载均衡服务器的宕机,需要建立一个备份机.主服务器和备份机上都运行高可用(High Availability)监控程序,通过传送诸如"I am alive"这样的信息来监控对方的运行状况.当备份机不能在一定的时间内收到这样的信息时,它就接管主服务器的服务IP并继续提供负

LVS + Keepalived 实现高可用、负载均衡 Web 集群

简介: LVS 是 Linux Virtual Server 的简写,Linux 虚拟服务器的意思,是一个虚拟的服务器集群系统,此项目由章文嵩博士于 1998 年 5 月成立,是中国最早出现的自由软件项目之一. LVS 负载均衡集群系统的优点: 1.提高吞吐量 想获得更高的吞吐量,在 LVS 中只需增加 Real-server 即可,其开销只是线性增长.如选择更换一台更高性能的服务器来获得相当的吞吐量,开销要大很多. 2.冗余 如果 LVS 中某台 Real-server 由于需要升级或其它原因