We have set up our OpenShift system, more than likely using HA if you are interested in Ceph, it’s time to get the storage online. You can do a SNO setup, then add additional workers. Or even you can do one system, with Ceph running storage… for some reason…

A very quick brief on Ceph. Ceph is an open-source shared storage system. It is comparable to VMware vSAN. Ceph has many knobs you can tune for performance. Once set up, you have several options for storage. You can do file-layer, POSIX-compatible storage (CephFS) that lets pods access files in a shared folder; S3 storage (RGW) that pods connect to over an HTTP endpoint; or block storage (RBD). For VMs we will be focusing on block storage. These will be our virtual disks. Since this is networked storage, you will want a fast interconnect between nodes, usually a minimum of 10gb/s. Depending on your disks and traffic, you can easily saturate your network here. Having a dedicated network for your nodes to talk over, and/or overprovisioning can help. After a node reboots, Ceph will resync whatever has changed while it was offline and compare itself to other cluster members. This will go as fast as your storage will let it.

When using OpenShift and wanting Ceph, you have two options. You can run the officially supported Red Hat OpenShift Data Foundation. This is a nicer version of Ceph that has a clean interface you can add as a plugin within OpenShift. The downside is it wants a ton of resources and is a licensed product. The other large downside is the requirements; it WILL NOT ALLOW YOU TO INSTALL without 30 CPUs (a Kubernetes measurement) and 72GB of RAM available across your nodes. For a home lab, that’s probably too much.

The alternative is Rook. Rook is a Kubernetes wrapper around Ceph and gives you controls for running Ceph within Kubernetes. This is what I will be discussing deploying. Rook is a project of the Cloud Native Computing Foundation and has lots of support. There is a getting started guide specifically for OpenShift, OpenShift – Rook Ceph Documentation, but I think it’s written more for people with experience.

Installing Rook

Since this is a generic open-source project and not an OpenShift-specific one, this installation will be oc-command heavy.

Note: OpenShift marks your installation disk as not available, Ceph will attempt to use any other disks in your hosts. If you have data on disks you do not want to lose, remove those disks. There is a chance Ceph will see the data and refuse to use the disk until you erase it. In that case, if Ceph does not grab your disk, I dd zeros to the first few MB of the drive, and then it will see it as blank.

• git clone https://github.com/rook/rook.git to a workstation which has the oc commands present.
• oc create namespace rook-ceph
• oc apply -f ./rook/deploy/examples/crds.yaml -f ./rook/deploy/examples/common.yaml
  • These are low level prereqs for Rook
• oc apply -f ./rook/deploy/examples/csi-operator.yaml
• oc apply -f ./rook/deploy/examples/operator-openshift.yaml
• oc apply -f ./rook/deploy/examples/cluster.yaml

• # Wait for pods to come up
• oc -n rook-ceph get service

The system should start grabbing disks and you will see a bunch of OSD pods come online. Each hard drive gets an OSD pod that will be in charge of it. This is replicated, distributed storage; if it says you have 10TB, and you have replication of 2 on, then you will only be able to store 5TB of actual data.

With the backend initializing, we need to do some configuration. We will start with the block storage that is our main use case.

1_configure_pool.yaml

apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
  name: cephblockpool
  namespace: rook-ceph
spec:
  failureDomain: host
  replicated:
    size: 2

oc apply -f 1_configure_pool.yaml

Note here I am saying keep 2 copies of each block of data in our ceph block pool and make sure they are stored on different hosts. You can set this higher to keep 3 copies or live on the edge with 1. If you are running SNO: set failureDomain: osd (or size: 1), or the system will not be able to start. I am running 3 nodes, so I purposefully set my replica size for block data to 2, and metadata above to 3. Block storage tends to be large; keeping only 2 copies allows for a backup and for use when a cluster node is rebooting for updates.

2_configure_ceph_block.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
   name: rook-ceph-block
provisioner: rook-ceph.rbd.csi.ceph.com
parameters:
    clusterID: rook-ceph
    pool: cephblockpool
    imageFormat: "2"
    imageFeatures: layering
    csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
    csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
    csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
    csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
    csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
    csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
reclaimPolicy: Delete
allowVolumeExpansion: true

oc apply -f 2_configure_ceph_block.yaml

If you want to make this your default storage run:

oc patch storageclass rook-ceph-block -p '{"metadata":{"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Now that we have our block storage configured, you may want CephFS also. It’s a shared folder where applications within OpenShift can mount and share files. This is file-based like a NAS. Whereas the block storage is block-based like a SAN.

3_configure_cephfs.yaml

apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
  name: cephfspool
  namespace: rook-ceph
spec:
  metadataPool:
    replicated:
      size: 3
      requireSafeReplicaSize: false
  dataPools:
    - name: replicated
      replicated:
        size: 3
        requireSafeReplicaSize: false
      compressionMode: none
  preserveFilesystemOnDelete: false
  metadataServer:
    activeCount: 3
    activeStandby: false

oc apply -f 3_configure_cephfs.yaml

4_configure_ceph_storage_class.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: rook-cephfs
provisioner: rook-ceph.cephfs.csi.ceph.com
allowVolumeExpansion: true
parameters:
  clusterID: rook-ceph
  fsName: cephfspool
  pool: cephfspool-replicated
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
reclaimPolicy: Delete

oc apply -f 4_configure_ceph_storage_class.yaml

For VMs and wanting to snapshot I added:

oc apply -f rook/deploy/examples/csi/rbd/snapshotclass.yaml

oc create -f rook/deploy/examples/csi/cephfs/snapshotclass.yaml

The last thing I would add, this is optional but makes life easier, is the web dashboard for Ceph.

oc -n rook-ceph edit cephcluster will pop up a text editor, then add

spec:
  dashboard:
    enabled: true
    ssl: true

That will start the dashboard on a node, but you may want to access it from outside the cluster itself:

oc apply -f ./rook/deploy/examples/dashboard-external-https.yaml

That exposes it on all nodes; you can then run the following to get the port number:

oc get services -n rook-ceph
NAME                                     TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
rook-ceph-exporter                       ClusterIP      172.30.120.33    <none>        9926/TCP            229d
rook-ceph-mgr                            ClusterIP      172.30.52.78     <none>        9283/TCP            229d
rook-ceph-mgr-dashboard                  ClusterIP      172.30.238.190   <none>        8443/TCP            229d
rook-ceph-mgr-dashboard-external-https   NodePort       172.30.254.103   <none>        8443:32088/TCP      229d
rook-ceph-mon-a                          ClusterIP      172.30.58.172    <none>        6789/TCP,3300/TCP   229d
rook-ceph-mon-b                          ClusterIP      172.30.10.72     <none>        6789/TCP,3300/TCP   229d
rook-ceph-mon-c                          ClusterIP      172.30.164.174   <none>        6789/TCP,3300/TCP   229d

In this example, that “32088” is the port we connect on. You will need to get the admin password as a secret within the cluster, and you will need to base64 decode it, on Linux / Mac:

oc -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode

On Windows:

oc -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}"
[System.Text.Encoding]::UTF8.GetString([System.Convert]::FromBase64String("mylittlestringfromfirststep="))

The dashboard comes with a bunch of metrics and configuration options out of the box. Optionally, you can enable additional performance monitoring or connect Ceph to Grafana for metrics.

Some helpful info

One helpful thing to enable is the toolbox for Ceph. This is an additional pod that will run and have all the Ceph tools ready for you.

oc apply -f ./rook/deploy/examples/toolbox.yaml

After the pod starts run:
oc exec -it -n rook-ceph deployment/rook-ceph-tools -- bash

Welcome to the world of Ceph. Commands like ceph status become your friend. The dashboard has most of this information in GUI format, but the commands have all the power to change settings. Ceph has anonymous metrics you can enable, at https://telemetry-public.ceph.com/ they have public Grafana dashboards of all the clusters checking in worldwide!

After struggling through getting it to work the first time, I found a blog which lays a similar workflow out nicely. I’ll leave it here in case their notes help anyone!

We now have our SNO node stood up, and we have our networking how we want it; it’s time to get storage configured to host VMs. OpenShift is Kubernetes with some opinions. Like picking a Linux distribution, there are many paths and OpenShift helps put you on one, making it easier to start with Kubernetes. OpenShift also comes with multiple Operators, Operators are like plugins, to help you manage different aspects of the system. Storage can be done multiple ways, and OpenShift supports multiple ways. In this post we will be discussing using the LVM Operator to have local storage.

Once you have installed a SNO node, and then deployed LVM here, congrats you will have everything you need to start deploying workloads. Before setting up storage, we have no place to persistently store files. The OpenShift node has a primary OS disk, which it uses for its own settings and storing container images, but this storage will be used for workload/pod data itself.

A few notes: If you are running HA OpenShift, you do not want this because storage cannot be shared amongst the different nodes. Also worth noting, during this process the system may try to format any drives that the OpenShift system itself is not using, make sure you are ready for these drives to be formatted.

Configuration Steps

• Go to the OpenShift web console, if you need to setup OpenShift I suggest checking out either my SNO guide or HA Guide.
• Click Operators -> OperatorHub.
• Install “LVM Storage” Operator.

• You will be prompted to create an “LVMCluster”, click the button.
• Name it something like “local-lvm-storage”.
• Open the “storage” menu.
  • Open “deviceClasses”.
    • Check default, if you want this to be the default storage.
    • It will say 90% of the space will be used for LVM, you can increase if you want more space, by default it leaves a little in case you need more for metadata storage. LVM has data and metadata storage as two different sub pools within your LVM.
    • Open “deviceSelector”.
      • forceWipeDevicesAndDestroyAllData if you dare / want it to format the drive first.
      • Open “paths”, here you must add the drives you want to use, this will go to each node, you can use serial numbers or for my single node I will just use “/dev/sda” since I know I have an empty SATA drive I want to use.
      • Note: If you used the partition script from the SNO article, then you need to put “/dev/sda5”, or “/dev/nvme0n1p5”, or whichever drive you used, if you run $ lsblk from Compute -> Nodes -> <your node> you will see the partition layout.
• Click “Create”.

• If the drive sets up correctly you will see Status: Ready.

If you install OpenShift Virtualization, it will automatically populate some of the templates to the default storage, also it will allow you to “Create PersistentVolumeClaim” “With Data upload form”.

• Going to Compute -> Nodes -> <your node> -> Terminal, you can run: chroot /host, lsblk and see all the goings on with lvm.
• Going to the same terminal, you can enter sudo lvs -a vg1 to verify lvm is configured and space available. I have started testing on OpenShift 4.20 nodes, and have noticed the host lvm.conf filters may hide the volumes, and you can use lvs --config 'devices { filter=["a|.*|"] }' to override that filter.

Types of Storage in Kubernetes

Now that LVM is setup it is worth quickly giving an overview on more layers of Storage within OpenShift and Kubernetes as a whole. There are two types of storage for Kubernetes / OpenShift, and within those types of access for files and data.

First, there is file system storage, this type of storage is a folder or files that are needed by a pod, think of storage such as NFS. Second there is block, this is large binary blobs; this is used a lot with VMs. Block will be both your ISO images for templates, and disk images you boot off of. This storage is just “here is 30GB”, x VM can request to edit any of the bytes in that block.

For access types there is RWO and RWX, biggest difference there is how many systems can access that storage at once. All our LVM will be RWO, because it’s local to this box. When you do Ceph storage, that can be RWX. There is also Read Only (ROX) and Read write once pod (RWOP) but we will not worry about that here.

These types of storage come into play later when you will want to add ISOs or VM hard drives within KubeVirt. The next post will be about setting up Ceph for a cluster. LVM storage is great for a single node you want to develop with; it’s simple, and there is very little overhead. When we get into multinode, we enter the world of hyper-converged infrastructure with all its pluses and minuses; but we need that to migrate VMs and have redundancy. After Ceph, we will finally discuss installing and managing KubeVirt / OpenShift Virtualization.