kubeadm部署k8s1.9高可用集群--4部署master节点

部署master节点

kubernetes master 节点包含的组件：

  • kube-apiserver
  • kube-scheduler
  • kube-controller-manager

本文档介绍部署一个三节点高可用 master 集群的步骤，分别命名为k8s-host1、k8s-host2、k8s-host3：

  • k8s-host1：172.16.120.154
  • k8s-host2：172.16.120.155
  • k8s-host3：172.16.120.156

安装docker

在每台主机安装docker。建议使用docker v1.12 , 但是 v1.11, v1.13 and 17.03 也可以正常工作. 版本 17.06+ 可能工作, 但是没有被k8s团队测试过。
PS：kubelet设置的cgroup driver应该和Docker设置的cgroup driver一致。这边统一为systemd。

yum install -y docker
systemctl enable docker && systemctl start docker

安装kubeadm, kubelet 和 kubectl

cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF
setenforce 0
yum install -y kubelet kubeadm kubectl
systemctl enable kubelet && systemctl start kubelet
  • 通过运行命令setenforce 0 禁用selinux，主要为了允许容器可以访问主机文件系统和pod networks的需要。
  • Some users on RHEL/CentOS 7 have reported issues with traffic being routed incorrectly due to iptables being bypassed。所以需要执行下面命令：
cat <<EOF >  /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
sysctl --system

安装master

kubeadm init

kubeadm init --apiserver-advertise-address=172.16.120.200 --pod-network-cidr=192.168.0.0/16
  • apiserver-advertise-address该参数一般指定为haproxy+keepalived 的vip。
  • pod-network-cidr 主要是在搭建pod network（calico）时候需要在init时候指定。

出现如下错误：

[[email protected] ~]# kubeadm init --apiserver-advertise-address=172.16.120.200 --pod-network-cidr=192.168.0.0/16

[init] Using Kubernetes version: v1.9.0
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
        [WARNING FileExisting-crictl]: crictl not found in system path
[preflight] Some fatal errors occurred:
        [ERROR Swap]: running with swap on is not supported. Please disable swap
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

解决办法：
1：kubelet启动参数增加--fail-swap-on=false

[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"
Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"
Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"
Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"
Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"
Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"
Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki"
Environment="KUBELET_SWAP_ARGS=--fail-swap-on=false"
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS $KUBELET_SWAP_ARGS

2：kubeadm init 参数增加ignore-preflight-errors

 kubeadm init --apiserver-advertise-address=172.16.120.200  --pod-network-cidr=192.168.0.0/16 www.bajieyy.org--ignore-preflight-errors ‘Swap‘

3：在物理机上可能并不会出现该问题。

最终安装成功

[[email protected] manifests]# kubeadm init --apiserver-advertise-address=172.16.120.200  --pod-network-cidr=192.168.0.0/16 --ignore-preflight-errors ‘Swap‘
[init] Using Kubernetes version: v1.9.0
[init] Using Authorization modes: [Node RBAC]
[preflight] Running pre-flight checks.
        [WARNING Swap]: running with swap on is not supported. Please disable swap
        [WARNING FileExisting-crictl]: crictl not found in system path
[preflight] Starting the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [k8s-host1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.16.120.200]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf"
[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".
[init] This might take a minute or longer if the control plane images have to be pulled.
[apiclient] All control plane components are healthy after 30.002346 seconds
[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[markmaster] Will mark node k8s-host1 as master by adding a label and a taint
[markmaster] Master k8s-host1 tainted and labelled with key/value: node-role.kubernetes.io/master=""
[bootstraptoken] Using token: 26ea24.cb987cb5cf6f29a6
[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: kube-dns
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join www.dijiuyy.com--token 26ea24.cb987cb5cf6f29a6 172.16.120.200:6443 --discovery-token-ca-cert-hash sha256:a2d5e0391695bb122d4a43dc951f93934a2aaa1360e00556401a49ea0ec2bdab

PS 需要记下 kubeadm join --token 26ea24.cb987cb5cf6f29a6 172.16.120.200:6443 --discovery-token-ca-cert-hash sha256:a2d5e0391695bb122d4a43dc951f93934a2aaa1360e00556401a49ea0ec2bdab，便于以后node加入到k8s集群中。

将kube-apiserver指向上一节部署的etcd集群

首先需要将上节生成的etcd ca等证书cp到mater主机的/etc/etcd/ssl/下

默认kubeadm安装了一个etcd节点，非集群，非加密，所以我们必须对kube-apiserver.yaml做修改。
修改后的kube-apiserver.yaml：

apiVersion: v1
kind: Pod
metadata:
  annotations:
    scheduler.alpha.kubernetes.io/critical-pod: ""
  labels:
    component: kube-apiserver
    tier: control-plane
  name: kube-apiserver
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-apiserver
    - --allow-privileged=true
    - --requestheader-username-headers=X-Remote-User
    - --requestheader-group-headers=X-Remote-Group
    - --requestheader-extra-headers-prefix=X-Remote-Extra-
    - --service-cluster-ip-range=10.96.0.0/12
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key
    - --enable-bootstrap-token-auth=true
    - --service-account-key-file=/etc/kubernetes/pki/sa.pub
    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt
    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key
    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt
    - --insecure-port=0
    - --admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota
    - --requestheader-allowed-names=front-proxy-client
    - --advertise-address=172.16.120.200
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --secure-port=6443
    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
    - --authorization-mode=Node,RBAC
    - --etcd-servers=https://172.16.120.151:2379,https://172.16.120.152:2379,https://172.16.120.153:2379
    - --etcd-cafile=/etc/etcd/ssl/ca.pem
    - --etcd-certfile=/etc/etcd/ssl/etcd.pem
    - --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem
    image: gcr.io/google_containers/kube-apiserver-amd64:v1.9.0
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 172.16.120.200
        path: /healthz
        port: 6443
        scheme: HTTPS
      initialDelaySeconds: 15
      timeoutSeconds: 15
    name: kube-apiserver
    resources:
      requests:
        cpu: 250m
    volumeMounts:
    - mountPath: /etc/kubernetes/pki
      name: k8s-certs
      readOnly: true
    - mountPath: /etc/ssl/certs
      name: ca-certs
      readOnly: true
    - mountPath: /etc/pki
      name: ca-certs-etc-pki
      readOnly: true
    - mountPath: /etc/etcd/ssl
      name: ca-certs-etc-etcd-ssl
      readOnly: true
  hostNetwork: true
  volumes:
  - hostPath:
      path: /etc/kubernetes/pki
      type: DirectoryOrCreate
    name: k8s-certs
  - hostPath:
      path: /etc/ssl/certs
      type: DirectoryOrCreate
    name: ca-certs
  - hostPath:
      path: /etc/pki
      type: DirectoryOrCreate
    name: ca-certs-etc-pki
  - hostPath:
      path: /etc/etcd/ssl
      type: DirectoryOrCreate
    name: ca-certs-etc-etcd-ssl
status: {}

注意添加了以下四项：

--etcd-cafile=/etc/etcd/ssl/ca.pem
--etcd-certfile=/etc/etcd/ssl/etcd.pem
--etcd-keyfile=/etc/etcd/ssl/etcd-key.pem
--etcd-servers=https://172.16.120.151:2379,https://172.16.120.152:2379,https://172.16.120.153:2379
# 以及挂载
 - mountPath: /etc/etcd/ssl
      name: ca-certs-etc-etcd-ssl
      readOnly: true

 - hostPath:
      path: /etc/etcd/ssl
      type: DirectoryOrCreate
    name: ca-certs-etc-etcd-ssl

确认第一台master三大组件都成功启动

kubectl get componentstatuses
NAME                 STATUS    MESSAGE              ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-1               Healthy   {"health": "true"}
etcd-0               Healthy   {"health": "true"}
etcd-2               Healthy   {"health": "true"}

安装其他主机

我一般的做法是，依旧使用kubeadm init --apiserver-advertise-address=172.16.120.200 --pod-network-cidr=192.168.0.0/16 --ignore-preflight-errors ‘Swap‘
主要目的是下载k8s相关组件docker镜像等，其实就是让kubeadm把容易忽略的工作都做了。

然后将第一台/etc/kubernetes/下所有文件拷贝到其他master主机对应目录下，直至其他机器master三大组件启动成功。

至此一个完整的master高可用集群搭建完毕。

原文地址：https://www.cnblogs.com/tianshifu/p/8127831.html

时间： 2024-11-11 09:56:36

kubeadm部署k8s1.9高可用集群--4部署master节点的相关文章

部署redis主从高可用集群

部署redis主从高可用集群本文部署的redis集群是一主一从,这两台服务器都设置了哨兵进程,另外再加一台哨兵做仲裁,建议哨兵数量为基数172.16.1.187    redis主+哨兵172.16.1.188    redis从+哨兵172.16.1.189    哨兵以上系统均为CentOS6 在187,188,189上部署redis过程如下:(1)redis使用编译安装方式,所以需要安装编译基本组件# yum -y install gcc gcc-c++ make cmake cpp gl

ProxySQL Cluster 配置详解 以及 高可用集群方案部署记录（完结篇）

早期的ProxySQL若需要做高可用,需要搭建两个实例,进行冗余.但两个ProxySQL实例之间的数据并不能共通,在主实例上配置后,仍需要在备用节点上进行配置,对管理来说非常不方便.但是ProxySQl 从1.4.2版本后,ProxySQL支持原生的Cluster集群搭建,实例之间可以互通一些配置数据,大大简化了管理与维护操作. ProxySQL是一个非中心化代理,在拓扑中,建议将它部署在靠近应用程序服务器的位置处.ProxySQL节点可以很方便地扩展到上百个节点,因为它支持runtime修改配

CentOS 7.3 部署LVS + Keepalived 高可用集群

双击热备 在这个高度信息化的IT时代,企业的生产系统,业务运营,销售个支持 以及日常管理环节越来越依赖计算机信息和服务,使得对高可用(HA) 技术的应用需求大量上升,以便提供持续的,不间断计算机或网络 vs服务器和web-server他们必须在同一个网段内,因为LVS转发包的时候,是直接修改了包目标的MAC地址,直接扔给了rs,基于MAC地址的修改是活动在OSI二层数据链路层的,工作在数据链路层的网络设备就是交换机了,所以必须在一个交换机下面,也就是一个局域网内为啥抑制real-server的A

部署数据库的高可用集群和性能调优

IP规划角色 IP地址 主机名Master 数据库服务器 192.168.4.51 master51备用 1 master 数据库服务器 192.168.4.52 master52备用 2 master 数据库服务器 192.168.4.53 master53第 1 台 slave 服务器 192.168.4.54 slave54第 2 台 slave 服务器 192.168.4.55 slave55Mha_manager 服务器 192.168.4.56 mgm56VIP 地址 192.168

高可用集群的基本部署

高可用集群的部署 实验环境的准备: 准备三台rhel6.5的虚拟机三台,真机作测试,做好解析. 解析 [[email protected] ~]# cat /etc/hosts 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6 172

Ansible自动化部署k8s-1.16.0版集群

Ansible自动化部署k8s二进制集群 Ansible是一种IT自动化工具.它可以配置系统,部署软件以及协调更高级的IT任务,例如持续部署,滚动更新.Ansible适用于管理企业IT基础设施. 这里我通过Ansible来实现Kubernetes v1.16 高可用集群自动部署(离线版) (但是还是需要网络,因为这里需要去部署flannel,coredns,ingress,dashboard插件,需要拉取镜像 Ansible自动化部署k8s-1.16.0版集群介绍 使用ansible自动化部署k

Redis Cluster 4.0高可用集群安装、在线迁移操作记录

之前介绍了redis cluster的结构及高可用集群部署过程,今天这里简单说下redis集群的迁移.由于之前的redis cluster集群环境部署的服务器性能有限,需要迁移到高配置的服务器上.考虑到是线上生产环境,决定在线迁移,迁移过程,不中断服务.操作过程如下: 一.机器环境 1 2 3 4 5 6 7 8 9 10 11 12 13 迁移前机器环境 ----------------------------------------------------------------------

linux高可用集群(HA)原理详解

高可用集群 一.什么是高可用集群 高可用集群就是当某一个节点或服务器发生故障时,另一个节点能够自动且立即向外提供服务,即将有故障节点上的资源转移到另一个节点上去,这样另一个节点有了资源既可以向外提供服务.高可用集群是用于单个节点发生故障时,能够自动将资源.服务进行切换,这样可以保证服务一直在线.在这个过程中,对于客户端来说是透明的. 二.高可用集群的衡量标准 高可用集群一般是通过系统的可靠性(reliability)和系统的可维护性(maintainability)来衡量的.通常用平均无故障时间

Keepalived高可用集群应用场景与配置

1.Keepalived单实例主备模式集群方案 这是最简单的模式,不只考虑高可用集群,先不考虑后方的Nginx负载均衡集群,即后端的服务器集群,参考下面的图示: 其对应的Keepalived核心配置如下: lb01 global_defs { notification_email { [email protected] [email protected] [email protected] } notification_email_from [email protected] smtp_serv