kubernetes中部署Heketi和GlusterFS(二)

kubernetes中部署Heketi和GlusterFS(二)
在上一节中，Heketi的部署方式还不能用于生产环境，因为Heketi Pod的数据并没有持久化，容易导致heketi的数据丢失，Heketi的数据保存在/var/lib/heketi/heketi.db文件中，因此需要把此目录挂载到GlusterFS分布式存储中。

按照上一节的步骤，执行heketi-cli topology load --json=topology-sample.json

$ echo $HEKETI_CLI_SERVER
http://10.254.49.43:8080

$ heketi-cli topology load --json=topology-sample.json
Creating cluster ... ID: 5b930ef6081fd22e895c25a3dfb0c516
    Allowing file volumes on cluster.
    Allowing block volumes on cluster.
    Creating node 10.30.1.15 ... ID: b120572be40db6c1d979c3903876430b
        Adding device /dev/sdb ... OK
    Creating node 10.30.1.16 ... ID: 7ce13ffc5eabe64a3791e93233fd3c1a
        Adding device /dev/sdb ... OK
    Creating node 10.30.1.17 ... ID: f9abdc2e5d4cfa17c035a97f984a1a3b
        Adding device /dev/sdb ... OK

紧接着执行：将生成heketi-storage.json文件。

$ heketi-cli setup-openshift-heketi-storage
Saving heketi-storage.json

执行完后，查看Pod deploy-heketi日志信息，看看做了哪些事：

#只截取了部分日志，基本操作就是进入到各个glusterfs Pod创建brick目录及创建一个副本为3的Replicate volume, volume名为heketidbstorage
[kubeexec] DEBUG 2018/07/09 07:07:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.17 Pod: glusterfs-8qrpt Command: mkdir -p /var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6
Result:
[kubeexec] DEBUG 2018/07/09 07:07:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.15 Pod: glusterfs-c4859 Command: mkdir -p /var/lib/heketi/mounts/vg_19584b16bc8f21b87662b27b551652fb/brick_abcb32853351840ee82a95693cbb63b4
Result:
[kubeexec] DEBUG 2018/07/09 07:07:23 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.16 Pod: glusterfs-25cm8 Command: mkdir -p /var/lib/heketi/mounts/vg_9534f15dd9f0822ad454140d13c660a5/brick_ba4091b858d94a088b21a582d8d4abaa

[kubeexec] DEBUG 2018/07/09 07:07:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.17 Pod: glusterfs-8qrpt Command: mkdir /var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6/brick

[kubeexec] DEBUG 2018/07/09 07:07:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.17 Pod: glusterfs-8qrpt Command: mkdir /var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6/brick

[kubeexec] DEBUG 2018/07/09 07:07:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.15 Pod: glusterfs-c4859 Command: mkdir /var/lib/heketi/mounts/vg_19584b16bc8f21b87662b27b551652fb/brick_abcb32853351840ee82a95693cbb63b4/brick

[kubeexec] DEBUG 2018/07/09 07:07:26 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.16 Pod: glusterfs-25cm8 Command: mkdir /var/lib/heketi/mounts/vg_9534f15dd9f0822ad454140d13c660a5/brick_ba4091b858d94a088b21a582d8d4abaa/brick
Result:
[cmdexec] INFO 2018/07/09 07:07:26 Creating volume heketidbstorage replica 3
[kubeexec] DEBUG 2018/07/09 07:07:27 /src/github.com/heketi/heketi/executors/kubeexec/kubeexec.go:246: Host: 10.30.1.16 Pod: glusterfs-25cm8 Command: gluster --mode=script volume create heketidbstorage replica 3 10.30.1.16:/var/lib/heketi/mounts/vg_9534f15dd9f0822ad454140d13c660a5/brick_ba4091b858d94a088b21a582d8d4abaa/brick 10.30.1.17:/var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6/brick 10.30.1.15:/var/lib/heketi/mounts/vg_19584b16bc8f21b87662b27b551652fb/brick_abcb32853351840ee82a95693cbb63b4/brick
Result: volume create: heketidbstorage: success: please start the volume to access data

进入任意GlusterFS Pod查看卷信息：

$ kubectl exec glusterfs-25cm8 -it bash
[[email protected] /]# gluster volume info

Volume Name: heketidbstorage
Type: Replicate
Volume ID: c8da2a4a-3066-4708-a59d-201d22decd92
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.30.1.16:/var/lib/heketi/mounts/vg_9534f15dd9f0822ad454140d13c660a5/brick_ba4091b858d94a088b21a582d8d4abaa/brick
Brick2: 10.30.1.17:/var/lib/heketi/mounts/vg_a146220fd3f761e8da2be784523ce07e/brick_6f0ce82692e70ce5ae2ec55a60f237c6/brick
Brick3: 10.30.1.15:/var/lib/heketi/mounts/vg_19584b16bc8f21b87662b27b551652fb/brick_abcb32853351840ee82a95693cbb63b4/brick
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
[[email protected] /]#

分析下heketi-storage.json：

#将创建如下资源信息：
Endpoints：
    name：heketi-storage-endpoints
         10.30.1.16:1 10.30.1.15:1 10.30.1.17:1
Service：
    name: heketi-storage-endpoints
job：
   name: heketi-storage-copy-job
   images: heketi/heketi:dev
   声明了 volume：heketi-storage
         "volumes": [
                                {
                                    "name": "heketi-storage",
                                    "glusterfs": {
                                        "endpoints": "heketi-storage-endpoints",
                                        "path": "heketidbstorage"
                                    }
                                },
     挂载到 /heketi：
             volumeMounts": [
                                            {
                                                "name": "heketi-storage",
                                                "mountPath": "/heketi"
                                            },

   启动时执行命令：cp /db/heketi.db /heketi
   #由此可知，此job的作用就是复制heketi中的数据文件到 /heketi，而/heketi目录挂载在了卷heketi-storage中，而heketi-storage volume是前面执行"heketi-cli setup-openshift-heketi-storage"时创建好了的

创建之：

$ kubectl create -f heketi-storage.json
secret "heketi-storage-secret" created
endpoints "heketi-storage-endpoints" created
service "heketi-storage-endpoints" created
job "heketi-storage-copy-job" created

当Job执行完后就可以删除它了：

$ kubectl get job
NAME                      DESIRED   SUCCESSFUL   AGE
heketi-storage-copy-job   1         1            1m

等到job完成后，删除bootstrap Heketi实例相关的组件：

#把之前由heketi-bootstrap.json创建的资源删除
$ kubectl delete all,service,jobs,deployment,secret --selector="deploy-heketi"
deployment "deploy-heketi" deleted
job "heketi-storage-copy-job" deleted
pod "deploy-heketi-69bfbd4bbd-q8tsk" deleted
service "deploy-heketi" deleted
secret "heketi-storage-secret" deleted

之前创建的名为deploy-heketi的pod,service已经删除了：

$ kubectl get pod
NAME              READY     STATUS              RESTARTS   AGE
glusterfs-25cm8   1/1       Running             1          1h
glusterfs-8qrpt   1/1       Running             1          1h
glusterfs-c4859   1/1       Running             1          1h

$ kubectl get svc
NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
heketi-storage-endpoints      ClusterIP   10.254.191.233   <none>        1/TCP      4m

最后，使用heketi-deployment.json文件重新部署heketi

$ cat heketi-deployment.json
{
  "kind": "List",
  "apiVersion": "v1",
  "items": [
    {
      "kind": "Secret",
      "apiVersion": "v1",
      "metadata": {
        "name": "heketi-db-backup",
        "labels": {
          "glusterfs": "heketi-db",
          "heketi": "db"
        }
      },
      "data": {
      },
      "type": "Opaque"
    },
    {
      "kind": "Service",
      "apiVersion": "v1",
      "metadata": {
        "name": "heketi",
        "labels": {
          "glusterfs": "heketi-service",
          "deploy-heketi": "support"
        },
        "annotations": {
          "description": "Exposes Heketi Service"
        }
      },
      "spec": {
        "selector": {
          "name": "heketi"
        },
        "ports": [
          {
            "name": "heketi",
            "port": 8080,
            "targetPort": 8080
          }
        ]
      }
    },
    {
      "kind": "Deployment",
      "apiVersion": "extensions/v1beta1",
      "metadata": {
        "name": "heketi",
        "labels": {
          "glusterfs": "heketi-deployment"
        },
        "annotations": {
          "description": "Defines how to deploy Heketi"
        }
      },
      "spec": {
        "replicas": 1,
        "template": {
          "metadata": {
            "name": "heketi",
            "labels": {
              "name": "heketi",
              "glusterfs": "heketi-pod"
            }
          },
          "spec": {
            "serviceAccountName": "heketi-service-account",
            "containers": [
              {
                "image": "heketi/heketi:dev",
                "imagePullPolicy": "Always",
                "name": "heketi",
                "env": [
                  {
                    "name": "HEKETI_EXECUTOR",
                    "value": "kubernetes"
                  },
                  {
                    "name": "HEKETI_DB_PATH",
                    "value": "/var/lib/heketi/heketi.db"
                  },
                  {
                    "name": "HEKETI_FSTAB",
                    "value": "/var/lib/heketi/fstab"
                  },
                  {
                    "name": "HEKETI_SNAPSHOT_LIMIT",
                    "value": "14"
                  },
                  {
                    "name": "HEKETI_KUBE_GLUSTER_DAEMONSET",
                    "value": "y"
                  }
                ],
                "ports": [
                  {
                    "containerPort": 8080
                  }
                ],
                "volumeMounts": [
                  {
                    "mountPath": "/backupdb",
                    "name": "heketi-db-secret"
                  },
                  {
                    "name": "db",
                    "mountPath": "/var/lib/heketi"
                  },
                  {
                    "name": "config",
                    "mountPath": "/etc/heketi"
                  }
                ],
                "readinessProbe": {
                  "timeoutSeconds": 3,
                  "initialDelaySeconds": 3,
                  "httpGet": {
                    "path": "/hello",
                    "port": 8080
                  }
                },
                "livenessProbe": {
                  "timeoutSeconds": 3,
                  "initialDelaySeconds": 30,
                  "httpGet": {
                    "path": "/hello",
                    "port": 8080
                  }
                }
              }
            ],
            "volumes": [
              {
                "name": "db",
                "glusterfs": {
                  "endpoints": "heketi-storage-endpoints",
                  "path": "heketidbstorage"
                }
              },
              {
                "name": "heketi-db-secret",
                "secret": {
                  "secretName": "heketi-db-backup"
                }
              },
              {
                "name": "config",
                "secret": {
                  "secretName": "heketi-config-secret"
                }
              }
            ]
          }
        }
      }
    }
  ]
}

heketi-deployment.json 文件 创建了如下资源：

Service:
     name: heketi
     port: 8080
Deployment：
    name: heketi
    replicas: 1
    image: heketi/heketi:dev
    volumeMounts:
           name: db   mountPath: /var/lib/heketi
   volumes:
      endpoints: heketi-storage-endpoints   #由heketi-storage.json文件创建
       path：heketidbstorage  #这是gluster volume名，此volume是由"heketi-cli setup-openshift-heketi-storage"自动创建。
    # 将heketi容器内的/var/lib/heketi 目录挂载到了GlusterFS volume “heketidbstorage”中。

部署之：

$ kubectl create -f heketi-deployment.json
secret "heketi-db-backup" created
service "heketi" created
deployment "heketi" created

$ kubectl get deployment
NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
heketi    1         1         1            1           45s
$ kubectl get svc
NAME                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
heketi                        ClusterIP   10.254.239.189   <none>        8080/TCP   51s
heketi-storage-endpoints      ClusterIP   10.254.191.233   <none>        1/TCP      31m

验证heketi是否在用在用gluster volume：

$ kubectl get pod
NAME                      READY     STATUS              RESTARTS   AGE
glusterfs-25cm8           1/1       Running             1          1h
glusterfs-8qrpt           1/1       Running             1          1h
glusterfs-c4859           1/1       Running             1          1h
heketi-7898db85dd-nb6kn   1/1       Running             0          1m

$ kubectl exec heketi-7898db85dd-nb6kn -it bash
[[email protected] /]# mount |grep heketi
10.30.1.15:heketidbstorage on /var/lib/heketi type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

至此，heketi db已正确配置了GlusterFS卷。

原文地址：http://blog.51cto.com/newfly/2139393

时间： 2024-10-08 21:17:40

kubernetes中部署Heketi和GlusterFS(二)的相关文章

Helm, 在Kubernetes中部署应用的利器

一.背景 Kubernetes(k8s)是一个基于容器技术的分布式架构领先方案.它在Docker技术的基础上,为容器化的应用提供部署运行.资源调度.服务发现和动态伸缩等一系列完整功能,提高了大规模容器集群管理的便捷性.在容器云环境及容器化服务在业界开始大规模部署应用的前提下,Kubernetes在业界的实际应用情况又是怎样的呢?在今年召开的JFrog SwampUp用户大会上,Codefresh公司为大家展示了一些有意思的数据.如下图: 据Codefresh公司统计,在目前JFrog的企业用户当

在虚拟机环境(CentOS7系统)下将kubernetes中部署服务成功，但在虚拟机外部无法访问到服务

在CentOS7环境下,kubernetes单机版环境,成功部署一个服务,在虚拟机中访问服务没问题,下面这样: curl http://172.27.73.26:8888/eureka-server/default/master {"name":"eureka-server","profiles":["default"],"label":"master","version&qu

Kubernetes中，通过Service访问Pod快速入门

一.背景 理想状态下,我们可以认为Kubernetes Pod是健壮的.但是,理想与现实的差距往往是非常大的.很多情况下,Pod中的容器可能会因为发生故障而死掉.Deployment等Controller会通过动态创建和销毁Pod来保证应用整体的健壮性.众所周知,每个Pod都拥有自己的IP地址,当新的Controller用新的Pod替代发生故障的Pod时,我们会发现,新的IP地址可能跟故障的Pod的IP地址可能不一致.此时,客户端如何访问这个服务呢?Kubernetes中的Service应运而生

[译]Kubernetes 分布式应用部署和人脸识别 app 实例

原文地址:KUBERNETES DISTRIBUTED APPLICATION DEPLOYMENT WITH SAMPLE FACE RECOGNITION APP 原文作者:skarlso 译文出自:掘金翻译计划 好的,伙计,让我们静下心来.下面将会是一个漫长但充满希望和有趣的旅程. 我将使用 Kubernetes 部署分布式应用程序.我试图创建一个类似于真实世界 app 的应用程序.显然,由于时间和精力有限,我不得不忽略一些细节部分. 我的重点将放在 Kubernetes 和应用部署上.

在kubernetes集群中部署mysql主从

本文介绍在kubernetes环境中部署mysql主从集群,数据持久化采用nfs. 一.环境介绍Mysql版本:5.7 Mysql master节点: 主机名:vm1IP地址:192.168.115.5/24 Mysql slave节点: 主机名:vm2IP地址:192.168.115.6/24 NFS节点:主机名:vm2IP地址:192.168.115.6/24共享目录:/home/mysql_master./home/mysql_slave 二.准备mysql主从的镜像环境dockerfil

在kubernetes集群中部署php应用

本文将介绍在kubernetes环境中部署一套php应用系统.前端web采用nginx.中间件php以fastcgi的方式运行,后台数据库由mysql主从提供支撑.各服务组件之间的调用采用dns解析服务名的方式进行,数据和配置文件持久化采用pv和pvc(基于nfs). 一.通过dockerfile创建php镜像文件 # cat dockerfile FROM docker.io/openshift/base-centos7:latest MAINTAINER ylw "[email protec

部署中遇到的问题-UWSGI(二)

pip-installed uWSGI ./python_plugin.so error 使用xml格式的uwsgi配置 uwsgi -x /home/zoueswebsite/zoueswebsite.xml的命令显示如下error: open("./python_plugin.so"): No such file or directory [core/utils.c line 3653] !!! UNABLE to load uWSGI plugin: ./python_plugi

Kubernetes应用部署策略实践

几个概念: Pod:是Kubernetes最基本的部署调度单元,可以包含container,逻辑上表示某种应用的一个实例.比如一个web站点应用由前端.后端及数据库构建而成,这三个组件将运行在各自的容器中,那么我们可以创建包含三个container的pod. node: 是 Kubernetes的worker节点,通常也称作为Minion node.除了运行一些kubernetes的组件以外(kubelet, kube-proxy等),还承担着运行容器服务的重任. ReplicationCont

Centos下Kubernetes+Flannel部署（新）

一.准备工作 1) 三台centos主机 k8s master: 10.11.151.97  tc-151-97 k8s node1: 10.11.151.100  tc-151-100 k8s node2: 10.11.151.101  tc-151-101 2)程序下载(百度网盘) k8s-1.1.3,Docker-1.8.2,ETCD-2.2.1,Flannel-0.5.5 二.ETCD集群部署 ETCD是k8s集群的基础,可以单结点也可以以集群的方式部署.本文以三台主机组成ETCD集群进