k8s使用Glusterfs动态生成pv

一、环境介绍

[[email protected] ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

172.31.250.152 k8s-m
172.31.250.153 node1
172.31.250.154 node2

[[email protected] ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
k8s-m Ready master 41m v1.13.1
node1 Ready <none> 27m v1.13.1
node2 Ready <none> 28m v1.13.1

#删除k8s集群master的污点(我这边就三台服务器,需要使用master,生产环境千万别这么干)
[[email protected] ~]# kubectl taint node k8s-m node-role.kubernetes.io/master-
node/k8s-m untainted

所有存储服务器下载

yum install centos-release-gluster -y

yum install glusterfs-server -y

二、Heketi安装

所有版本:https://github.com/heketi/heketi/releases

#下载安装
wget https://github.com/heketi/heketi/releases/download/v8.0.0/heketi-v8.0.0.linux.amd64.tar.gz
tar xf heketi-v8.0.0.linux.amd64.tar.gz
mkdir  -p /data/heketi/{bin,conf,data}
cp heketi/heketi.json /data/heketi/conf/
cp heketi/{heketi,heketi-cli} /data/heketi/bin/

#我们glusterFS部署在k8s集群外,所以heketi通过ssh管理glusterFS。需要创建免秘钥登陆到所有glusterFS节点。
ssh-keygen -f /data/heketi/conf/heketi_key -t rsa -N ‘‘
#将公钥放到所有GlusterFS节点
ssh-copy-id  -i /data/heketi/conf/heketi_key.pub  [email protected]m
ssh-copy-id  -i /data/heketi/conf/heketi_key.pub  [email protected]
ssh-copy-id  -i /data/heketi/conf/heketi_key.pub  [email protected]

2、heketi.json配置文件

[[email protected] ~]# cat /data/heketi/conf/heketi.json
{
  "_port_comment": "Heketi Server Port Number",
  "port": "18080",

	"_enable_tls_comment": "Enable TLS in Heketi Server",
	"enable_tls": false,

	"_cert_file_comment": "Path to a valid certificate file",
	"cert_file": "",

	"_key_file_comment": "Path to a valid private key file",
	"key_file": "",

  "_use_auth": "Enable JWT authorization. Please enable for deployment",
  "use_auth": true,

  "_jwt": "Private keys for access",
  "jwt": {
    "_admin": "Admin has access to all APIs",
    "admin": {
      "key": "adminkey"
    },
    "_user": "User only has access to /volumes endpoint",
    "user": {
      "key": "userkey"
    }
  },

  "_backup_db_to_kube_secret": "Backup the heketi database to a Kubernetes secret when running in Kubernetes. Default is off.",
  "backup_db_to_kube_secret": false,

  "_glusterfs_comment": "GlusterFS Configuration",
  "glusterfs": {
    "_executor_comment": [
      "Execute plugin. Possible choices: mock, ssh",
      "mock: This setting is used for testing and development.",
      "      It will not send commands to any node.",
      "ssh:  This setting will notify Heketi to ssh to the nodes.",
      "      It will need the values in sshexec to be configured.",
      "kubernetes: Communicate with GlusterFS containers over",
      "            Kubernetes exec api."
    ],
    "executor": "ssh",

    "_sshexec_comment": "SSH username and private key file information",
    "sshexec": {
      "keyfile": "/data/heketi/conf/heketi_key",
      "user": "root",
      "port": "22",
      "fstab": "/etc/fstab",
      "backup_lvm_metadata": false
    },

    "_kubeexec_comment": "Kubernetes configuration",
    "kubeexec": {
      "host" :"https://kubernetes.host:8443",
      "cert" : "/path/to/crt.file",
      "insecure": false,
      "user": "kubernetes username",
      "password": "password for kubernetes user",
      "namespace": "OpenShift project or Kubernetes namespace",
      "fstab": "Optional: Specify fstab file on node.  Default is /etc/fstab",
      "backup_lvm_metadata": false
    },

    "_db_comment": "Database file name",
    "db": "/data/heketi/data/heketi.db",

     "_refresh_time_monitor_gluster_nodes": "Refresh time in seconds to monitor Gluster nodes",
    "refresh_time_monitor_gluster_nodes": 120,

    "_start_time_monitor_gluster_nodes": "Start time in seconds to monitor Gluster nodes when the heketi comes up",
    "start_time_monitor_gluster_nodes": 10,

    "_loglevel_comment": [
      "Set log level. Choices are:",
      "  none, critical, error, warning, info, debug",
      "Default is warning"
    ],
    "loglevel" : "debug",

    "_auto_create_block_hosting_volume": "Creates Block Hosting volumes automatically if not found or exsisting volume exhausted",
    "auto_create_block_hosting_volume": true,

    "_block_hosting_volume_size": "New block hosting volume will be created in size mentioned, This is considered only if auto-create is enabled.",
    "block_hosting_volume_size": 500,

    "_block_hosting_volume_options": "New block hosting volume will be created with the following set of options. Removing the group gluster-block option is NOT recommended. Additional options can be added next to it separated by a comma.",
    "block_hosting_volume_options": "group gluster-block"
  }
}

  

3、创建heketi启动脚本

[[email protected] ~]# cat /usr/lib/systemd/system/heketi.service
[Unit]
Description=RESTful based volume management framework for GlusterFS
Before=network-online.target
After=network-online.target
Documentation=https://github.com/heketi/heketi
[Service]
Type=simple
LimitNOFILE=65536
ExecStart=/data/heketi/bin/heketi --config=/data/heketi/conf/heketi.json
KillMode=process
Restart=on-failure
RestartSec=5
SuccessExitStatus=15
StandardOutput=syslog
StandardError=syslog
[Install]
WantedBy=multi-user.target

#启动heketi
systemctl start heketi
systemctl enable heketi
systemctl status heketi

4、Heketi添加cluster

[[email protected] ~]# /data/heketi/bin/heketi-cli --user admin --server http://k8s-m:18080 --secret adminkey --json  cluster create
{"id":"aa378f94299c29bbd0224fc902c0cbd8","nodes":[],"volumes":[],"block":true,"file":true,"blockvolumes":[]}

5、#将3个glusterfs节点作为node添加到cluster

[[email protected] ~]# /data/heketi/bin/heketi-cli --server http://k8s-m:18080   --user "admin" --secret "adminkey"   node add --cluster "aa378f94299c29bbd0224fc902c0cbd8"  --management-host-name  k8s-m --storage-host-name 172.31.250.152 --zone 1
Node information:
Id: 9280cde1c2640bf3fef483509a04bff2
State: online
Cluster Id: aa378f94299c29bbd0224fc902c0cbd8
Zone: 1
Management Hostname k8s-m
Storage Hostname 172.31.250.152

[[email protected]-m ~]# /data/heketi/bin/heketi-cli --server http://k8s-m:18080   --user "admin" --secret "adminkey"   node add --cluster "aa378f94299c29bbd0224fc902c0cbd8"  --management-host-name  node1 --storage-host-name 172.31.250.153 --zone 1
Node information:
Id: 19838aea8104aee077b0ea67cd6ea012
State: online
Cluster Id: aa378f94299c29bbd0224fc902c0cbd8
Zone: 1
Management Hostname node1
Storage Hostname 172.31.250.153
[[email protected]-m ~]# /data/heketi/bin/heketi-cli --server http://k8s-m:18080   --user "admin" --secret "adminkey"   node add --cluster "aa378f94299c29bbd0224fc902c0cbd8"  --management-host-name  node2 --storage-host-name 172.31.250.154 --zone 1
Node information:
Id: 171aeb876968bf9ef1fab28e6a31e919
State: online
Cluster Id: aa378f94299c29bbd0224fc902c0cbd8
Zone: 1
Management Hostname node2
Storage Hostname 172.31.250.154

6、添加device

#机器只是作为gluster的运行单元,volume是基于device创建的。同时需要特别说明的是,目前heketi仅支持使用裸分区或裸磁盘(未格式化)添加为device,不支持文件系统

[[email protected] ~]# /data/heketi/bin/heketi-cli --server http://k8s-m:18080   --user "admin" --secret "adminkey"    --json device add --name="/dev/vdb" --node "9280cde1c2640bf3fef483509a04bff2"
Device added successfully

[[email protected]-m ~]# /data/heketi/bin/heketi-cli --server http://k8s-m:18080   --user "admin" --secret "adminkey"    --json device add --name="/dev/vdb" --node "19838aea8104aee077b0ea67cd6ea012"
Device added successfully

[[email protected]-m ~]# /data/heketi/bin/heketi-cli --server http://k8s-m:18080   --user "admin" --secret "adminkey"    --json device add --name="/dev/vdb" --node "171aeb876968bf9ef1fab28e6a31e919"
Device added successfully

##查看
[[email protected]-m ~]# /data/heketi/bin/heketi-cli --server http://k8s-m:18080   --user "admin" --secret "adminkey"   node list
Id:171aeb876968bf9ef1fab28e6a31e919    Cluster:aa378f94299c29bbd0224fc902c0cbd8
Id:19838aea8104aee077b0ea67cd6ea012    Cluster:aa378f94299c29bbd0224fc902c0cbd8
Id:9280cde1c2640bf3fef483509a04bff2    Cluster:aa378f94299c29bbd0224fc902c0cbd8

7、添加volume

创建一个大小为3G,副本为3的volume
[[email protected] ~]# /data/heketi/bin/heketi-cli --server http://k8s-m:18080   --user "admin" --secret "adminkey"  volume create --size 3 --replica 3
Name: vol_2aab01a26b3a2dfc1286eed16c8fcb62
Size: 3
Volume Id: 2aab01a26b3a2dfc1286eed16c8fcb62
Cluster Id: aa378f94299c29bbd0224fc902c0cbd8
Mount: 172.31.250.154:vol_2aab01a26b3a2dfc1286eed16c8fcb62
Mount Options: backup-volfile-servers=172.31.250.153,172.31.250.152
Block: false
Free Size: 0
Reserved Size: 0
Block Hosting Restriction: (none)
Block Volumes: []
Durability Type: replicate
Distributed+Replica: 3

  

8、创建storageclass

[[email protected] ~]# cat   storageclass-glusterfs.yaml
apiVersion: v1
kind: Secret
metadata:
  name: heketi-secret
  namespace: default
data:
  # base64 encoded password. E.g.: echo -n "mypassword" | base64
  key: YWRtaW5rZXk=
type: kubernetes.io/glusterfs

---
apiVersion: storage.k8s.io/v1beta1
kind: StorageClass
metadata:
  name: glusterfs
provisioner: kubernetes.io/glusterfs
allowVolumeExpansion: true
parameters:
  resturl: "http://172.31.250.152:18080"
  clusterid: "aa378f94299c29bbd0224fc902c0cbd8"
  restauthenabled: "true"
  restuser: "admin"
  #secretNamespace: "default"
  #secretName: "heketi-secret"
  restuserkey: "adminkey"
  gidMin: "40000"
  gidMax: "50000"
  volumetype: "replicate:3"

#导入
[[email protected]-m ~]# kubectl apply -f    storageclass-glusterfs.yaml
secret/heketi-secret created
storageclass.storage.k8s.io/glusterfs created

9、创建一个 statefulset测试是否会自动生成pv

#查看pv和pvc
[[email protected]-m ~]# kubectl get pv --all-namespaces
No resources found.
[[email protected]-m ~]# kubectl get pvc --all-namespaces
No resources found.

#statefulset文件
[[email protected]-m ~]# cat   mystatefulset.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  ports:
  - port: 80
    name: web
  clusterIP: None
  selector:
    app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nginx
spec:
  selector:
    matchLabels:
      app: nginx # has to match .spec.template.metadata.labels
  serviceName: "nginx"
  replicas: 3 # by default is 1
  template:
    metadata:
      labels:
        app: nginx # has to match .spec.selector.matchLabels
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - metadata:
      name: www
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "glusterfs"
      resources:
        requests:
          storage: 2Gi

#查看pv和pvc(根据pvc自动创建了pv)
[[email protected]-m ~]# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                 STORAGECLASS   REASON   AGE
pvc-0afdc43a-1567-11e9-b8f6-00163e024454   2Gi        RWO            Delete           Bound    default/www-nginx-0   glusterfs               6s
[[email protected]-m ~]# kubectl get pvc
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
www-nginx-0   Bound    pvc-0afdc43a-1567-11e9-b8f6-00163e024454   2Gi        RWO            glusterfs      16s

原文地址:https://www.cnblogs.com/zhangb8042/p/10254983.html

时间: 2024-10-30 04:15:06

k8s使用Glusterfs动态生成pv的相关文章

C# 动态生成WebService,无需添加引用

C#项目调用WebService是很常见的现象,但一旦修改链接地址就需要重新更新引用很是麻烦,这里跟大家分享一个通过地址,无需添加引用动态生成Webservice的小方法 方法类: 1 using System; 2 using System.Collections.Generic; 3 using System.Linq; 4 using System.Text; 5 using System.ServiceModel; 6 using System.ServiceModel.Channels

C++实现根据类名动态生成类对象

在开发后台服务的过程中,我们常常需要从数据库中取数据,并将数据缓存在本地中,另外,我们的服务还需要有更新数据的能力:包括定时的主动更新以及数据库数据更新时服务收到通知的被动更新. 之前在需要用到以上功能的时候,模仿着组内通用的数据Cache部分的代码来写,十分方便,基本上只需要自己写两个类:一个是取数据并缓存数据的类XXXData,一个是扇出数据的类XXXFetcher. 在需要使用数据的时候,通过: FetcherFactory::getFetcher<XXXFetcher>() 即可获取一

[搬运自我的CSDN博客] python抓取javascript动态生成HTML内容的实践

<注:CSDN博客在美国访问特别卡,所以转移到cnblogs来发文章> 本实验在Ubuntu14.04上完成.使用的浏览器是火狐(Firefox 33.0),python版本是2.7.6. 大家都知道用urllib配合正则表达式抓取静态HTML的内容很方便,但是如果网页中有javascript动态生成的内容,urllib就无能为力了. 此时我们要借助一个额外的工具:selenium.它的工作原理是操纵(火狐)浏览器浏览目标网页,等待网页中的javascript全部执行完毕后再对HTML源码进行

动态生成二级菜单

现在越来越多的用到二级甚至多级菜单,前台菜单的显示,手动指定也越来越不能满足要求,所以,动态生成菜单是必须的 思路 + 示例代码(以二级菜单为例) 先取出一级菜单内容值,接下来遍历一级菜单,将其id当做本次检索的parentid,将与之对应的二级菜单值获取到, 并加入到当前数组中(后台) 二层循环,当获取一个值时,检查其对于的二级菜单项是否有数据,有的话,则输出来,没有则跳过(前台) 以PHP后台为例 $res = mysql_query('*** where parentid = 0');  

nginx利用image_filter动态生成缩略图

原文:http://www.open-open.com/lib/view/open1416193847945.html "我现在是有些图片需要生成缩略图,这个现在加了image_filter这个已经实现了,但我不知道怎么样才能访问我上传的原图" 刚开始觉得也不太好弄,让他用程序区处理,实际上稍微动脑筋分析一下也可以不修改程序实现动态生成缩略图且能够访问原图. 前提是需要定好图片的访问规则. 先来看一下什么是nginx的image filter模块. HttpImageFilterMod

动态生成随机背景色表格

<!DOCTYPE html><html> <head> <meta charset="UTF-8"> <title>动态生成随机背景色表格</title> <style> table{margin-top:20px;width:800px;border:1px solid #ddd;border-collapse:collapse;} td{border:1px solid #ddd;padding:

Android动态生成表格

最近刚刚学习完Android的五大布局,现在我们进一步深入学习,尝试做一个动态生成表格功能的例子 样式布局代码如下: 1 <LinearLayout xmlns:android="http://schemas.android.com/apk/res/android" 2 xmlns:tools="http://schemas.android.com/tools" 3 android:layout_width="match_parent" 4

js动态生成数据列表

我们通常会使用table标签来展示数据内容,由于需要展示的数据内容是随时更换的,所以不可能将展示的数据列表写死在html写死在页面中,而是需要我们根据后台传来的数据随时更换,这个时候就需要我们使用js来动态生成表格. 首先我们需要先写好页面的样式. html部分 <!DOCTYPE html> <html> <head> <meta http-equiv="Content-Type" content="text/html; chars

用Aspose.Words for .NET动态生成word文档中的数据表格

1.概述 最近项目中有一个这样的需求:导出word 文档,要求这个文档的格式不是固定的,用户可以随便的调整,导出内容中的数据表格列是动态的,例如要求导出姓名和性别,你就要导出这两列的数据,而且这个文档不是导出来之后再调整而是导出来后已经是调整过了的.看到这里,您也许马上想到用模板导出!而且.NET中自带有这个组件:Microsoft.Office.Interop.Word,暂且可以满足需求吧.但这个组件也是有局限性的,例如客户端必须装 office组件,而且编码复杂度高.最麻烦的需求是后面那个-