Introduction
You have the desire that your application needs to scale across multiple nodes for load balancing, but needs access to a common PVC? For this purpose, you need a PVC that is RWX-enabled. As part of our Managed Kubernetes Cluster, it is possible to create a CSI Cinder block storage. However, due to technical limitations, this is only ReadWriteOnce (RWO) capable. We will show you here how we can still create a RWX-capable PVC using a workaround!
PVCs are explained in the tutorial Creating Persistent Volumes in Kubernetes, on which this tutorial builds.
Take, for example, a web server that has HTML pages, images, JS etc. stored on its PVC. These files are to be uploaded once and made available to all instances of the web server simultaneously. To better distribute the load, the deployment is scaled accordingly. Thus, several pods of the same type run on different servers.
To ensure that all pods – across host boundaries – have access to the files, we create a RWX Storageclass with the nfs-ganesha-server-and-external-provisioner. The basis for this is an NWS Managed Kubernetes Cluster. After completing the workaround, we have the option of creating PVCs that can be read and written to simultaneously by multiple pods.
A note beforehand: the setup described is not HA capable!
Prerequisites
The tutorial assumes the following:
- NWS Managed Kubernetes Cluster
- kubectl
- helm
You can find more information on these topics in our NWS Docs
Setup
With the following command we add the Helm Repository, which provides us with the nfs-ganesha-server-and-external-provisioner, called NFS Provisioner in the following.
$ helm repo add nfs-ganesha-server-and-external-provisioner https://kubernetes-sigs.github.io/nfs-ganesha-server-and-external-provisioner/
After that we can immediately start with the installation. Note the settings that follow the –set parameter. These result in:
-
persistence.enabled=true This parameter ensures that our data is stored on a persistent volume and persists even after the pod restarts.
-
persistence.storageClass=default Here it is specified that the storage class "standard" should be used for the persistent data.
-
persistence.size=200Gi This parameter specifies the size of the PVC that the NFS Provisioner fetches for the files.
Note that the PVC size specified with persistence.size is shared for all NFS PVCs that are obtained from the NFS Provisioner. There are many other configuration options with which the NFS Provisioner can be adapted to your own needs. You can find these here
$ helm install nfs-server nfs-ganesha-server-and-external-provisioner/nfs-server-provisioner --set persistence.enabled=true,persistence.storageClass=standard,persistence.size=200Gi
If the NFS Provisioner could be installed successfully, an output like the following appears:
NAME: nfs-server
LAST DEPLOYED: Mon May 22 14:41:58 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The NFS Provisioner service has now been installed.
A storage class named 'nfs' has now been created and is available to provision dynamic volumes.
You can use this storageclass by creating a `PersistentVolumeClaim` with the correct storageClassName attribute. For example:
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: test-dynamic-volume-claim
spec:
storageClassName: "nfs"
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Mi
Now let’s take a look at the NFS Provisioner Pod we just created and see if it’s running:
kubectl get pods NAME READY STATUS RESTARTS AGE nfs-server-nfs-server-provisioner-0 1/1 Running 0 36m
And of course the associated storage class for the automatic provision of NFS RWX PVCs. We can now use this to dynamically create and use PVCs of the RWX type.
$ kubectl get storageclass NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE encrypted cinder.csi.openstack.org Delete Immediate true 3h22m encrypted-high-iops cinder.csi.openstack.org Delete Immediate true 3h22m high-iops cinder.csi.openstack.org Delete Immediate true 3h22m nfs cluster.local/nfs-server-nfs-server-provisioner Delete Immediate true 39m nws-storage cinder.csi.openstack.org Delete Immediate true 3h22m standard (default) cinder.csi.openstack.org Delete Immediate true 3h22m
Web server example
We now have a PVC with the property RWX which we can provision dynamically. Now let’s take a look at how this is integrated into a deployment.
We create two files; one for our web server deployment and one for the RWX PVC:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: webserver
name: webserver
spec:
replicas: 1
selector:
matchLabels:
app: webserver
template:
metadata:
creationTimestamp: null
labels:
app: webserver
spec:
containers:
- image: nginx:latest
name: nginx
volumeMounts:
- mountPath: /files
name: files
volumes:
- name: files
persistentVolumeClaim:
claimName: nfs-pvc
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: nfs-pvc
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
storageClassName: nfs
Our deployment specifies that we want to run exactly one replica of the NGiNX Container. This integrates the dynamically created NFS PVC under /files. Now we have to feed the definitions to Kubernetes using kubectl:
$ kubectl apply -f nfs-pvc.yaml nginx-deployment.yaml persistentvolumeclaim/nfs-pvc created deployment.apps/webserver created
As we can see a Webserver pod is running:
$ kubectl get pods NAME READY STATUS RESTARTS AGE nfs-server-nfs-server-provisioner-0 1/1 Running 0 54m webserver-5486dd9cf5-hfhnd 1/1 Running 0 114s
But since this is far from sufficient for our load, we will expand the deployment to 4 replications:
$ kubectl scale deployment webserver --replicas=4 deployment.apps/webserver scaled
A small check is recommended and lo and behold, all 4 pods are running happily:
$ kubectl get pods NAME READY STATUS RESTARTS AGE nfs-server-nfs-server-provisioner-0 1/1 Running 0 55m webserver-5486dd9cf5-hfhnd 1/1 Running 0 3m9s webserver-5486dd9cf5-nh9fl 1/1 Running 0 18s webserver-5486dd9cf5-ss27f 1/1 Running 0 18s webserver-5486dd9cf5-xl2lj 1/1 Running 0 18s
Now let’s check if the NFS is also working properly between the pods.
A first command shows us that /files is empty. Using a second command, we create the nfs-is-rwx file. From the third issue we can see that we have been successful. We created in one pod and the file immediately existed in another pod.
$ kubectl exec webserver-5486dd9cf5-hfhnd -- ls /files $ kubectl exec webserver-5486dd9cf5-nh9fl -- touch /files/nfs-is-rwx $ kubectl exec webserver-5486dd9cf5-xl2lj -- ls /files nfs-is-rwx
Summary
You have now set up an NFS server, which uses a CSI Cinder block storage in the background. The NFS server uses the block storage in the background to provide an RWX PVC for your pods via NFS. As a result, we have circumvented the technical limitation of a CSI Cinder block device and you can now use your NWS Managed Kubernetes Cluster for more use cases.
