HX-CSI ReadWriteMany Block Device Example

Specialized applications can need to read/write directly to a block device. The main use case are databases. A filesystem is more “human-readable” and it adds extra overhead. Instead of the filesystem, block devices are chosen more often. More information about Kubernetes can be found HERE.

After the installation of Kubernetes and the HyperFlex Container Storage Interface, you can proceed with this post.

Storage Class YAML File

I am using the default Storage Class of the HX-CSI example directory. Nothing fancy.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
    name: csi-hxcsi-default
provisioner: csi-hxcsi
parameters:
    datastore: default-ds
    datastoreSize: "-1"

Persistent Volume Claim YAML File.

After creating the Storage Class, create a PVC as a Block Device with ReadWriteMany because different containers wants to read and write to the same shared storage.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: demo-block-rwx
spec:
storageClassName: csi-hxcsi-default
volumeMode: Block
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Gi

Deployement YAML File

In this example I’m using the Ubuntu container image. Because it’s a block device, devicePath is now used instead of volumeMount.

apiVersion: apps/v1
kind: Deployment
metadata:
name: demo-block
labels:
app: demo-block
spec:
replicas: 1
selector:
matchLabels:
app: demo-block
template:
metadata:
labels:
app: demo-block
name: demo-block
spec:
volumes:
- name: blockvolume
persistentVolumeClaim:
claimName: demo-block-rwx
containers:
- name: demo-block
image: ubuntu
command: [ "/bin/bash", "-c", "--" ]
args: [ "while true; do sleep 30; done;" ]
ports:
- containerPort: 22
volumeDevices:
- devicePath: /dev/block
name: blockvolume

When the SC, PVC and the container are deployed, verification if everything is working can be done.
To verify if the block device connected in the container:

kubectl exec demo-block-757cfdc75c-59dbz – bash -c "ls -l /dev/"
total 0
brw-rw-- – 1 root disk 8, 32 Mar 11 19:48 block
lrwxrwxrwx 1 root root 11 Mar 11 19:21 core -> /proc/kcore
lrwxrwxrwx 1 root root 13 Mar 11 19:21 fd -> /proc/self/fd
crw-rw-rw- 1 root root 1, 7 Mar 11 19:21 full
drwxrwxrwt 2 root root 40 Mar 11 19:21 mqueue
crw-rw-rw- 1 root root 1, 3 Mar 11 19:21 null
lrwxrwxrwx 1 root root 8 Mar 11 19:21 ptmx -> pts/ptmx
drwxr-xr-x 2 root root 0 Mar 11 19:21 pts
crw-rw-rw- 1 root root 1, 8 Mar 11 19:21 random
drwxrwxrwt 2 root root 40 Mar 11 19:21 shm
lrwxrwxrwx 1 root root 15 Mar 11 19:21 stderr -> /proc/self/fd/2
lrwxrwxrwx 1 root root 15 Mar 11 19:21 stdin -> /proc/self/fd/0
lrwxrwxrwx 1 root root 15 Mar 11 19:21 stdout -> /proc/self/fd/1
-rw-rw-rw- 1 root root 0 Mar 11 19:21 termination-log
crw-rw-rw- 1 root root 5, 0 Mar 11 19:21 tty
crw-rw-rw- 1 root root 1, 9 Mar 11 19:21 urandom
crw-rw-rw- 1 root root 1, 5 Mar 11 19:21 zero

There is a /dev/block and you can see it’s a block device, because the first letter on the line is a “b”
Let’s write some data on the block device:

kubectl exec demo-block-757cfdc75c-59dbz – bash -c "echo "I_Am_Joost_._com" | dd conv=unblock of=/dev/block"

Verify what is written on the storage device:

kubectl exec demo-block-757cfdc75c-59dbz – bash -c "od -An -c -N 20 /dev/block"
jvdmade@K8S-M:~$ kubectl exec demo-block-757cfdc75c-59dbz – bash -c "od -An -c -N 20 /dev/block" 
I _ A m _ J o o s t _ . _ c o m

It is possible to create some Replica’s, but I want to have a complete “new” server connect to this RWX Block device, so I’ve created another deployment. Basically it’s the same YAML file, but with different names.

Here you see the Server2-demo pod is running:

jvdmade@K8S-M:~/block$ kubectl get pods
NAME READY STATUS RESTARTS AGE
csi-attacher-hxcsi-0 2/2 Running 0 3d21h
csi-nodeplugin-hxcsi-2w4cg 2/2 Running 2 3d21h
csi-nodeplugin-hxcsi-h2pfz 2/2 Running 2 3d21h
csi-nodeplugin-hxcsi-vcbjq 2/2 Running 2 3d21h
csi-provisioner-hxcsi-0 2/2 Running 0 3d21h
csi-resizer-hxcsi-7979549b8f-wh58n 2/2 Running 0 3d21h
demo-block-757cfdc75c-59dbz 1/1 Running 0 2d23h
server2-block-fcd48df5-4tctb 1/1 Running 0 15s
test-647b4c8874-l5g5d 1/1 Running 0 3d20h

To see if the block device is really has the same content of the other container:

kubectl exec server2-block-fcd48df5-4tctb – bash -c "od -An -c -N 16 /dev/block"

And the result is what the other container wrote to the device.

I _ A m _ J o o s t _ . _ c o m

Via Server2 we’re going to overwrite this data:

kubectl exec server2-block-fcd48df5-4tctb – bash -c "echo "This_Is_Server_2_Writing_Something_To_The_Block_Device" | dd conv=unblock of=/dev/block"

Lets see if the content can be read from the other server with the command:

kubectl exec demo-block-757cfdc75c-59dbz – bash -c "od -An -c -N 75 /dev/block"

And here is the result:

T h i s _ I s _ S e r v e r _ 2
_ W r i t i n g _ S o m e t h i
n g _ T o _ T h e _ B l o c k _
D e v i c e \n \0 \0 \0 \0 \0 \0 \0 \0 \0
\0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0

Hooray! It’s working.


Here are other HyperFlex / Technical Posts:

  • Barocco MD770 Keyboard Macros
    For a year I was using a Koolertron Split Keyboard and I was pretty happy about it. The biggest disadvantage was the arrow keys, but those I didn’t use it often. Now I have the Barocco MD770 Suddenly my right keyboard thought it was a left keyboard and all the keys were wrong. Trying to
  • HX-CSI ReadWriteMany Block Device Example
    Specialized applications can need to read/write directly to a block device. The main use case are databases. A filesystem is more “human-readable” and it adds extra overhead. Instead of the filesystem, block devices are chosen more often. More information about Kubernetes can be found HERE. After the installation of Kubernetes and the HyperFlex Container Storage
  • HyperFlex 4.5 Container Storage Interface (HX-CSI) Installation
    In a containerized environment, you need persistent volumes . With HyperFlex 4.5, iSCSI thisis possible. What are the difference between using iSCSI or CSI in a container. Information about Kubernetes Persistent Volumes can be found HERE. iSCSI can only do: ReadWriteOnce and ReadOnlyMany. HX-CSI can do both AND also ReadWriteMany. (Block) There is a very
  • Kubernetes Installation.
    For the HyperFlex Container Storage Interface (HX-CSI) you will need a Kubernetes cluster. Here is the start of an image that have the right software on it and which you can clone before configuring Kubernetes. Start with a normal Ubuntu 18.04 installation. I just want to have certain versions of software and that is why
  • Cutting MP4 Into smaller Files.
    To make some awesome videos, I’m using my GoPro Hero8 with 2K7 (2704×1520) resolution. This is because I’m using a green screen and although the end videos are in FullHD, with this resolution I can easily crop, resize parts of the video without losing any quality. Cutting MP4 into smaller files would make life easier.

Leave a Comment