一、k8s对接外部ceph存储
1、k8s对接ceph存储的六种方式
1）直接使用ceph的文件系统
2）直接使用ceph的块存储
3）使用社区提供的cephfs做持久化数据卷
4）使用社区提供的RBD做pod的持久化存储
5）使用官方ceph-csi的cephfs方式
6）使用官方ceph-csi的rbd方式
2、分为三大类
1）社区
1.cephfs做持久化存储
2.RBD做pod的持久化存储
2）官方ceph-csi
1.cephfs方式
2.rbd方式
3）直接使用ceph的文件系统或者块存储
3、k8s可以通过两种方式使用ceph做volume:cephfs和rbd
1.一个ceph集群仅支持一个cephFS
2.cephfs方式支持k8s的pv的3种访问模式ReadWriteOnce、ReadOnlyMany、ReadWriteMany
3.RBD支持ReadWriteOnce、ReadOnlyMany两种方式
注意：访问模式只是能力描述，并不是强制执行，对于没有按PVC声明的方式使用pv，存储提供者应该负责访问时的运行错误。例如如果设置PVC的访问模式为ReadOnlyMany，pod挂载后依然可写，如果需要真正的不可写，申请pvc是需要指定readOnly:true参数
二、直接使用ceph的块存储（直接使用静态PV）
1、k8s使用volume的方法
1.直接方式：volume->backend
2.静态配备：volume->PersistentVolumeClaim->PersistenVolume->Backend
3.动态配备：volume->PersistentVolumeClaim->StorageClass->Backend
2、静态PV(rbd)方式
1.所有k8s节点安装依赖组件
cat > /etc/yum.repos.d/ceph.repo << EOF [ceph] name=ceph baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/ gpgcheck=0 priority=1 [ceph-noarch] name=cephnoarch baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/noarch/ gpgcheck=0 priority=1 [ceph-source] name=Ceph source packages baseurl=http://mirrors.aliyun.com/ceph/rpm-nautilus/el7/SRPMS gpgcheck=0 priority=1 EOF cat > /etc/yum.repos.d/epel.repo << EOF [epel] name=Extra Packages for Enterprise Linux 7 - $basearch baseurl=http://mirrors.aliyun.com/epel/7/$basearch failovermethod=priority enabled=1 gpgcheck=0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 [epel-debuginfo] name=Extra Packages for Enterprise Linux 7 - $basearch - Debug baseurl=http://mirrors.aliyun.com/epel/7/$basearch/debug failovermethod=priority enabled=0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 gpgcheck=0 [epel-source] name=Extra Packages for Enterprise Linux 7 - $basearch - Source baseurl=http://mirrors.aliyun.com/epel/7/SRPMS failovermethod=priority enabled=0 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-7 gpgcheck=0 EOF
2.配置免密
ssh-copy-id 192.168.112.110 ssh-copy-id 192.168.112.111 ssh-copy-id 192.168.112.112
3.同步配置文件
ceph-deploy --overwrite-conf admin ceph1 ceph2 ceph3 k8s-master k8s-node1 k8s-node2
4.创建存储池并开启rbd功能
#创建kube池给k8s ceph osd pool create kube 128 128
5.创建ceph用户，提供给k8s使用
#查看ceph集群中的认证用户及相关的key ceph auth list #删除集群中的一个认证用户 ceph auth del osd.0（不执行，需要再执行） #创建集群用户 ceph auth get-or-create client.kube mon 'allow r' osd 'allow class-read object_prefix rbd_children,allow rwx pool=kube'
6.创建secret资源
ceph auth get-key client.admin | base64 ceph auth get-key client.kube | base64 #base64单向加密，k8s不以明文方式存储账号密码 mkdir jtpv && cd jtpv cat > ceph-admin-secret.yaml << EOF apiVersion: v1 kind: Secret metadata: name: ceph-admin-secret namespace: default data: key: QVFEOHhqVmhBMXE5THhBQXRFbzZZUWtkbzRjREdkbG9kdGl6NHc9PQ== #（admin的key） type: kubernetes.io/rbd EOF cat > ceph-kube-secret.yaml << EOF apiVersion: v1 kind: Secret metadata: name: ceph-kube-secret namespace: default data: key: QVFEZzZUNWhXS09OQnhBQXdtQmxlWlozWmtJVDRUWm1tUXUrcXc9PQ== #（kube的key） type: kubernetes.io/rbd EOF
7.创建pv
cat > pv.yaml << EOF
apiVersion: v1
kind: PersistentVolume
metadata:
name: ceph-pv-test
spec:
capacity:
storage: 5Gi
#指定访问模式
accessModes: #访问模式有三种，ReadWriteOnce,ReadOnlyMany,ReadWriteMany
- ReadWriteOnce
rbd:
monitors:
- 192.168.112.130:6789
- 192.168.112.131:6789
- 192.168.112.132:6789
pool: kube
image: ceph-image
user: admin
secretRef:
name: ceph-admin-secret
fsType: ext4
readOnly: false
#指定pv的回收策略,即pvc资源释放后的事件,回收策略有三种:Retain,Recyle,Delete
persistentVolumeReclaimPolicy: Retain
EOF8.发布创建
[root@k8s-master jtpv]# kubectl apply -f pv.yaml
persistentvolume/ceph-pv-test created
[root@k8s-master jtpv]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
ceph-pv-test 5Gi RWO Retain Available 9s
[root@k8s-master jtpv]# kubectl describe pv ceph-pv-test
Name: ceph-pv-test
Labels:
Annotations:
Finalizers: [kubernetes.io/pv-protection]
StorageClass:
Status: Available
Claim:
Reclaim Policy: Retain
Access Modes: RWO
VolumeMode: Filesystem
Capacity: 5Gi
Node Affinity:
Message:
Source:
Type: RBD (a Rados Block Device mount on the host that shares a pod's lifetime)
CephMonitors: [192.168.112.130:6789 192.168.112.131:6789 192.168.112.132:6789]
RBDImage: ceph-image #未创建
FSType: ext4
RBDPool: kube
RadosUser: admin
Keyring: /etc/ceph/keyring
SecretRef: &SecretReference{Name:ceph-admin-secret,Namespace:,}
ReadOnly: false
Events:9.创建镜像，划出一块空间给他用
#创建一个名为ceph-image的镜像，大小5G rbd create -p kube -s 5G ceph-image #查看镜像 rbd ls -p kube #查看详情 rbd info ceph-image -p kube rbd image 'ceph-image': size 5 GiB in 1280 objects order 22 (4 MiB objects) snapshot_count: 0 id: d9aa5d2335ec block_name_prefix: rbd_data.d9aa5d2335ec format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Mon Sep 13 14:31:46 2021 access_timestamp: Mon Sep 13 14:31:46 2021 modify_timestamp: Mon Sep 13 14:31:46 2021
10.创建一个pvc
cat > pvc.yaml << EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ceph-test-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
EOF
kubectl apply -f pvc.yaml11.创建测试pod
cat > pod.yaml << EOF
apiVersion: v1
kind: Pod
metadata:
name: ceph-pod
spec:
containers:
- name: test-pod
image: busybox:1.24
command: ["sleep","60000"]
volumeMounts:
- name: pvc
mountPath: "/usr/share/busybox"
readOnly: false
volumes:
- name: pvc
persistentVolumeClaim:
claimName: ceph-test-claim
EOF
kubectl apply -f pod.yaml
#验证
kubectl exec -it ceph-pod -- df -h|grep /dev/rbd012.解决报错
如果pod详情报这个错
MountVolume.WaitForAttach failed for volume "ceph-pv-test" : rbd: map failed exit status 6, rbd output: rbd: sysfs write failed
RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable kube/ceph-image object-map fast-diff deep-flatten".
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (6) No such device or address
原因：内核3.10版本只支持layering，其他特性需要高版本内核支持
layering: 支持分层
striping: 支持条带化 v2
exclusive-lock: 支持独占锁
object-map: 支持对象映射（依赖 exclusive-lock ）
fast-diff: 快速计算差异（依赖 object-map ）
deep-flatten: 支持快照扁平化操作
journaling: 支持记录 IO 操作（依赖独占锁）
解决办法：你可以禁用内核不支持的特性。 rbd feature disable kube/ceph-image object-map fast-diff deep-flatten
