HX-CSI ReadWriteMany Block Device Example

Specialized applications can need to read/write directly to a block device. The main use case are databases. A filesystem is more “human-readable” and it adds extra overhead. Instead of the filesystem, block devices are chosen more often. More information about Kubernetes can be found HERE.

After the installation of Kubernetes and the HyperFlex Container Storage Interface, you can proceed with this post.

Storage Class YAML File

I am using the default Storage Class of the HX-CSI example directory. Nothing fancy.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
    name: csi-hxcsi-default
provisioner: csi-hxcsi
parameters:
    datastore: default-ds
    datastoreSize: "-1"

Persistent Volume Claim YAML File.

After creating the Storage Class, create a PVC as a Block Device with ReadWriteMany because different containers wants to read and write to the same shared storage.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: demo-block-rwx
spec:
storageClassName: csi-hxcsi-default
volumeMode: Block
accessModes:
- ReadWriteMany
resources:
requests:
storage: 10Gi

Deployement YAML File

In this example I’m using the Ubuntu container image. Because it’s a block device, devicePath is now used instead of volumeMount.

apiVersion: apps/v1
kind: Deployment
metadata:
name: demo-block
labels:
app: demo-block
spec:
replicas: 1
selector:
matchLabels:
app: demo-block
template:
metadata:
labels:
app: demo-block
name: demo-block
spec:
volumes:
- name: blockvolume
persistentVolumeClaim:
claimName: demo-block-rwx
containers:
- name: demo-block
image: ubuntu
command: [ "/bin/bash", "-c", "--" ]
args: [ "while true; do sleep 30; done;" ]
ports:
- containerPort: 22
volumeDevices:
- devicePath: /dev/block
name: blockvolume

When the SC, PVC and the container are deployed, verification if everything is working can be done.
To verify if the block device connected in the container:

kubectl exec demo-block-757cfdc75c-59dbz – bash -c "ls -l /dev/"
total 0
brw-rw-- – 1 root disk 8, 32 Mar 11 19:48 block
lrwxrwxrwx 1 root root 11 Mar 11 19:21 core -> /proc/kcore
lrwxrwxrwx 1 root root 13 Mar 11 19:21 fd -> /proc/self/fd
crw-rw-rw- 1 root root 1, 7 Mar 11 19:21 full
drwxrwxrwt 2 root root 40 Mar 11 19:21 mqueue
crw-rw-rw- 1 root root 1, 3 Mar 11 19:21 null
lrwxrwxrwx 1 root root 8 Mar 11 19:21 ptmx -> pts/ptmx
drwxr-xr-x 2 root root 0 Mar 11 19:21 pts
crw-rw-rw- 1 root root 1, 8 Mar 11 19:21 random
drwxrwxrwt 2 root root 40 Mar 11 19:21 shm
lrwxrwxrwx 1 root root 15 Mar 11 19:21 stderr -> /proc/self/fd/2
lrwxrwxrwx 1 root root 15 Mar 11 19:21 stdin -> /proc/self/fd/0
lrwxrwxrwx 1 root root 15 Mar 11 19:21 stdout -> /proc/self/fd/1
-rw-rw-rw- 1 root root 0 Mar 11 19:21 termination-log
crw-rw-rw- 1 root root 5, 0 Mar 11 19:21 tty
crw-rw-rw- 1 root root 1, 9 Mar 11 19:21 urandom
crw-rw-rw- 1 root root 1, 5 Mar 11 19:21 zero

There is a /dev/block and you can see it’s a block device, because the first letter on the line is a “b”
Let’s write some data on the block device:

kubectl exec demo-block-757cfdc75c-59dbz – bash -c "echo "I_Am_Joost_._com" | dd conv=unblock of=/dev/block"

Verify what is written on the storage device:

kubectl exec demo-block-757cfdc75c-59dbz – bash -c "od -An -c -N 20 /dev/block"
jvdmade@K8S-M:~$ kubectl exec demo-block-757cfdc75c-59dbz – bash -c "od -An -c -N 20 /dev/block" 
I _ A m _ J o o s t _ . _ c o m

It is possible to create some Replica’s, but I want to have a complete “new” server connect to this RWX Block device, so I’ve created another deployment. Basically it’s the same YAML file, but with different names.

Here you see the Server2-demo pod is running:

jvdmade@K8S-M:~/block$ kubectl get pods
NAME READY STATUS RESTARTS AGE
csi-attacher-hxcsi-0 2/2 Running 0 3d21h
csi-nodeplugin-hxcsi-2w4cg 2/2 Running 2 3d21h
csi-nodeplugin-hxcsi-h2pfz 2/2 Running 2 3d21h
csi-nodeplugin-hxcsi-vcbjq 2/2 Running 2 3d21h
csi-provisioner-hxcsi-0 2/2 Running 0 3d21h
csi-resizer-hxcsi-7979549b8f-wh58n 2/2 Running 0 3d21h
demo-block-757cfdc75c-59dbz 1/1 Running 0 2d23h
server2-block-fcd48df5-4tctb 1/1 Running 0 15s
test-647b4c8874-l5g5d 1/1 Running 0 3d20h

To see if the block device is really has the same content of the other container:

kubectl exec server2-block-fcd48df5-4tctb – bash -c "od -An -c -N 16 /dev/block"

And the result is what the other container wrote to the device.

I _ A m _ J o o s t _ . _ c o m

Via Server2 we’re going to overwrite this data:

kubectl exec server2-block-fcd48df5-4tctb – bash -c "echo "This_Is_Server_2_Writing_Something_To_The_Block_Device" | dd conv=unblock of=/dev/block"

Lets see if the content can be read from the other server with the command:

kubectl exec demo-block-757cfdc75c-59dbz – bash -c "od -An -c -N 75 /dev/block"

And here is the result:

T h i s _ I s _ S e r v e r _ 2
_ W r i t i n g _ S o m e t h i
n g _ T o _ T h e _ B l o c k _
D e v i c e \n \0 \0 \0 \0 \0 \0 \0 \0 \0
\0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0

Hooray! It’s working.


Here are other HyperFlex / Technical Posts:

  • Ubuntu Automated Installation (For K8S)
    Because I am doing a lot of testing it can be rewarding to do a complete re-installation of a Kubernetes cluster. As a result the only thing I disliked was the manual process of the Ubuntu installation.Here is a procedure to automate a complete Ubuntu installation and customize it. Requirements Before you can start, make
  • My First 3D Printer (Ender 3 V2)
    For my “early” birthday, my wife gave me my first 3d printer. An Creality Ender 3 V2. It is a nice printer to start with. Content: Hiccups Changing thing on the 3D printer Upgrading the 3D printer Conclusion Hiccups My first print from the dog, which is on the default SD-Card went without any problems.
  • License Registration for HyperFlex
    After the installation of HyperFlex, you will have an evaluation period. During this time you have to register the cluster with a valid license. There are features that won’t be running anymore after the evaluation period is over. How to register your HyperFlex cluster? In HyperFlex Connect there is information about the license of the
  • Barocco MD770 Keyboard Macros
    For a year I was using a Koolertron Split Keyboard and I was pretty happy about it. The biggest disadvantage was the arrow keys, but those I didn’t use it often. Now I have the Barocco MD770 Suddenly my right keyboard thought it was a left keyboard and all the keys were wrong. Trying to
  • HX-CSI ReadWriteMany Block Device Example
    Specialized applications can need to read/write directly to a block device. The main use case are databases. A filesystem is more “human-readable” and it adds extra overhead. Instead of the filesystem, block devices are chosen more often. More information about Kubernetes can be found HERE. After the installation of Kubernetes and the HyperFlex Container Storage

Leave a Comment