This guide walks you through setting up a 5-node multi-AZ Kubernetes cluster, distributed across two compute availability zones in the NETWAYS Cloud (HetznerNBG4, NorisNBG3), to provide cross-data-center redundancy within a single OpenStack region—all managed by Claudie.
This article was created in collaboration with Samuel Stoličný of Berops. You can find him on LinkedIn if you’d like to learn more about Claudie.
Why this setup?
Operating a single cluster across two NWS data centers provides you with geographic redundancy without leaving a single OpenStack project. If one data center fails, the control plane maintains the quorum from the other data center; workloads running in the healthy data center remain available.
All cross-data-center network traffic flows over a dedicated connection between the two data centers and is additionally encrypted using WireGuard.
How does NWS Multi-AZ work?
NWS has launched two Compute Availability Zones within its region: HetznerNBG4 (Hetzner Nuremberg) and NorisNBG3 (Noris Nuremberg). Both AZs share a Keystone endpoint and a Neutron network, so a single application credential and a single provider entry in InputManifest are sufficient—the only difference between the node pools is the “ zone ” field:
providers:
- name: netways-1
secretRef: { name: netways-secret, ... }
nodePools:
dynamic:
- { name: nbg4-ctrl, providerSpec: { ..., zone: HetznerNBG4 } }
- { name: noris-ctrl, providerSpec: { ..., zone: NorisNBG3 } }Use the following command to check which AZs your OpenStack project supports:
openstack availability zone list --compute
+-------------+-------------+
| Zone Name | Zone Status |
+-------------+-------------+
| HetznerNBG4 | available |
| NorisNBG3 | available |
+-------------+-------------+What You Can Achieve with This Claudie Setup
- 3 control-plane nodes: 2 at
HetznerNBG4+ 1 atNorisNBG3. With 3 control planes spread across 2 data centers, etcd can withstand either a failure atNorisNBG3or a node failure atHetznerNBG4. A complete loss ofHetznerNBG4takes the cluster offline because etcd loses its quorum. See considerations below. - 2 compute nodes: 1 at
HetznerNBG4+ 1 atNorisNBG3. - A WireGuard VPN that spans all five nodes.
- Cilium CNI and Longhorn Storage, installed by Claudie.
Prerequisites
- A Kubernetes cluster running Claudie (the “management cluster”).
kubectl, configured for this cluster.- An NWS account with a quota in the
HetznerNBG4region (which covers both Compute AZs) for at least 5 VMs, 3 floating IPs, and 250 GB of storage.
Step 1 – Installing Claudie
Install cert-manager and Claudie:
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.19.3/cert-manager.yaml
kubectl apply -f https://github.com/berops/claudie/releases/latest/download/claudie.yamlCheck to see if the pods are running:
kubectl get pods -n claudie
NAME READY STATUS RESTARTS AGE
ansibler-7b5f8c6d49-qvz2w 1/1 Running 0 60s
claudie-operator-59dd646bcf-bjsdr 1/1 Running 0 60s
kube-eleven-bdc799684-xlwws 1/1 Running 0 60s
kuber-7cc7cb797d-bvsrv 1/1 Running 0 60s
manager-69f96bb548-l5wqr 1/1 Running 0 60s
minio-0 1/1 Running 0 60s
minio-1 1/1 Running 0 60s
minio-2 1/1 Running 0 60s
minio-3 1/1 Running 0 60s
mongodb-858994c5cb-x8w77 1/1 Running 0 60s
nats-0 2/2 Running 0 60s
nats-1 2/2 Running 0 60s
nats-2 2/2 Running 0 60s
terraformer-667b9f4556-dkx4x 1/1 Running 0 60sYou can find more details in the comprehensive Claudie Guide.
Step 2 – Creating Provider Credentials
NWS authentication is typically federated via the NWS ID (OIDC). Claudie requires static credentials, so the process involves activating the OpenStack project user in the NWS Customer Interface and then creating an application credential using that user.
After you have enabled the Project User and noted down its username and password, authenticate yourself with the OpenStack CLI and create the application credential:
export OS_AUTH_URL=https://cloud.netways.de:5000/v3/
export OS_IDENTITY_API_VERSION=3
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_DOMAIN_NAME=Default
export OS_USERNAME=<your-project-user>
export OS_PROJECT_NAME=<your-project-name>
read -s OS_PASSWORD; export OS_PASSWORD
openstack application credential create --role member claudie -f shell
id="..."
secret="..."
project_id="..."
openstack project show $(openstack token issue -c project_id -f value) \
-c domain_id -f value
defaultThe application credential is limited to the region you authenticated against (HetznerNBG4 if you used the default project user), and it can communicate with any AZ within that region, including both HetznerNBG4 and NorisNBG3. One credential, one secret.
Step 3 – Creating Kubernetes Secrets
kubectl create namespace claudie-secrets
kubectl create secret generic netways-secret \
--namespace=claudie-secrets \
--from-literal=authurl='https://cloud.netways.de:5000/v3/' \
--from-literal=domainid='default' \
--from-literal=projectid='<your-nws-project-id>' \
--from-literal=applicationcredentialid='<your-credential-id>' \
--from-literal=applicationcredentialsecret='<your-credential-secret>'Step 4 – Creating the Claudie manifest file
Save the following as netways-multi-az.yaml:
apiVersion: claudie.io/v1beta1
kind: InputManifest
metadata:
name: netways-multi-az
labels:
app.kubernetes.io/part-of: claudie
spec:
providers:
- name: netways-1
providerType: openstack
secretRef:
name: netways-secret
namespace: claudie-secrets
nodePools:
dynamic:
- name: nbg4-ctrl
providerSpec:
name: netways-1
region: HetznerNBG4
zone: HetznerNBG4
externalNetworkName: public-network
count: 2
serverType: s1.medium
image: "Ubuntu Noble 24.04 LTS"
storageDiskSize: 50
- name: noris-ctrl
providerSpec:
name: netways-1
region: HetznerNBG4
zone: NorisNBG3
externalNetworkName: public-network
count: 1
serverType: s1.medium
image: "Ubuntu Noble 24.04 LTS"
storageDiskSize: 50
- name: nbg4-cmp
providerSpec:
name: netways-1
region: HetznerNBG4
zone: HetznerNBG4
externalNetworkName: public-network
count: 1
serverType: s1.medium
image: "Ubuntu Noble 24.04 LTS"
storageDiskSize: 50
- name: noris-cmp
providerSpec:
name: netways-1
region: HetznerNBG4
zone: NorisNBG3
externalNetworkName: public-network
count: 1
serverType: s1.medium
image: "Ubuntu Noble 24.04 LTS"
storageDiskSize: 50
kubernetes:
clusters:
- name: netways-multi-az
version: v1.32.0
network: 192.168.2.0/24
pools:
control:
- nbg4-ctrl
- noris-ctrl
compute:
- nbg4-cmp
- noris-cmpNotes:
- All four node pools share
region: HetznerNBG4because both AZs are located in this region. Onlyzoneis different. serverType: s1.medium(4 vCPUs, 4 GB RAM, 50 GB local disk) boots directly from the image. Any NWS flavor with a local disk is compatible.storageDiskSize: 50Adds an additional Cinder volume for Longhorn storage per worker. Controlplane nodes ignore this field.- The cluster network
192.168.2.0/24is the WireGuard overlay, separate from the NWS Neutron network.
Step 5 – Creating the Cluster
kubectl apply -f netways-multi-az.yaml
inputmanifest.claudie.io/netways-multi-az created
kubectl get inputmanifest -n claudie
NAME STATUS
netways-multi-az IN_PROGRESSA multi-AZ build with 5 nodes typically takes 8 to 12 minutes for creation. Once it starts, the InputManifest updates its status:
kubectl get inputmanifest -n claudie
NAME STATUS
netways-multi-az WATCHING_FOR_CHANGESStep 6 – Accessing the Cluster
kubectl get secrets -n claudie -l claudie.io/output=kubeconfig \
-o jsonpath='{.items[0].data.kubeconfig}' | base64 -d > netways-multi-az.kubeconfig
export KUBECONFIG=$PWD/netways-multi-az.kubeconfig
kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP OS-IMAGE
nbg4-ctrl-xxxxxxx-01 Ready control-plane 6m v1.32.0 192.168.2.1 Ubuntu 24.04.4 LTS
nbg4-ctrl-xxxxxxx-02 Ready control-plane 6m v1.32.0 192.168.2.2 Ubuntu 24.04.4 LTS
noris-ctrl-yyyyyyy-01 Ready control-plane 6m v1.32.0 192.168.2.3 Ubuntu 24.04.4 LTS
nbg4-cmp-zzzzzzz-01 Ready <none> 4m v1.32.0 192.168.2.4 Ubuntu 24.04.4 LTS
noris-cmp-wwwwwww-01 Ready <none> 4m v1.32.0 192.168.2.5 Ubuntu 24.04.4 LTSEach node has a INTERNAL-IP in the WireGuard overlay. Cross-data-center traffic remains encrypted via these addresses, regardless of which physical NWS data center it passes through.
For workloads that must remain in a single data center, apply labels and taints to pods for a topology zone (topology.kubernetes.io/zone=HetznerNBG4 or NorisNBG3) so that the scheduler respects the placement.
Step 7 – Cleaning Up the Cluster
kubectl delete inputmanifest netways-multi-az -n claudieThe teardown deletes all VMs, networks, routers, security groups, and floating IPs across both availability zones. This usually takes 2 to 5 minutes.
Considerations
- Loss of the control plane due to a complete outage of the HetznerNBG4 data center. With 2 HetznerNBG4s and 1 NorisNBG3, the loss of a HetznerNBG4 brings the etcd quorum below the required threshold, and the control plane becomes read-only (workloads continue to run, but no new scheduling takes place).
- Cross-data-center latency. The round-trip time for HetznerNBG4-NorisNBG3 is typically 6–9 ms in our test builds, which is within a normal, etcd-friendly range.
Additional Information and Materials
– Claudie documentation for reference and instructions.
– OpenStack Provider Reference.
– Load Balancing Guide for deploying services across availability zones using a load balancer managed by Claudie.
– Autoscaling Guide for scaling compute node pools independently by AZ.
0 Comments