Automatic Fedora CoreOS Updates for your Kubernetes

Automatic Fedora CoreOS Updates for your Kubernetes

You want automated Fedora CoresOS updates for your Kubernetes? And what do Zincati and libostree have to do with it? Here you will quickly see an overview!

Fedora CoreOS is used as the operating system for many Kubernetes clusters. This operating system, which specializes in containers, scores particularly well with simple, automatic updates. Unlike usual, it is not updated package by package. Fedora CoreOS first creates a new, updated image of the system and finalizes the update with a reboot. rpm-ostree in combination with Cincinnati and Zincati ensures a smooth process.

Before we take a closer look at the components, let’s first clarify how you can enable automatic updates for your NWS Kubernetes cluster.

 

How do you activate automatic updates for your NWS Kubernetes Cluster?

 

In the NWS portal you can easily choose between different update mechanisms. Click on “Update Fedora CoreOS” in the context menu of your Kubernetes cluster and choose between immediate, periodic and lock-based.

Shows settings for automatic periodic updatesImmediate applies updates immediately to all of your Kubernetes nodes and finalizes the update with a reboot.

Periodic updates your nodes only during a freely selectable maintenance window. In addition to the days of the week, you can also specify the start time and the length of the maintenance window.

Lock-based uses the FleetLock protocol to coordinate the updates. Here, a lock manager is used to coordinate the finalization of updates. This ensures that nodes do not finalize and reboot updates at the same time. In addition, the update process is stopped in the event of problems and other nodes do not perform an update.

Disable deactivates automatic updates.

So far, so good! But what is rpm-ostree and Zincati?

 

 

Updates but different!

 

The introduction of container-based applications has also made it possible to standardize and simplify the underlying operating systems. Reliable, automatic updates and the control of these – by the operator of the application – additionally reduce the effort for maintenance and coordination.

 

rpm-ostree creates the images

rpm-ostree is a hybrid of libostree and libdnf and therefore a mixture of image and package system. libostree describes itself as a git for operating system binaries, with each commit containing a bootable file tree. A new release of Fedora CoreOS therefore corresponds to an rpm-ostree commit, maintained and provided by the CoreOS team. libdnf provides the familiar package management features, making the base provided by libostree extensible by users.

Taints and Tolerations Nodes on which containers cannot be started or are unreachable are given a so-called taint by Kubernetes (e.g. not-ready or unreachable). As a counterpart, pods on such nodes are given a toleration. This also happens during a Fedora CoreOS update. Pods are automatically marked with tolerationSeconds=300 on reboot, which will restart your pods on other nodes after 5 minutes. Of course, you can find more about taints and tolerations in the Kubernetes documentation.

 

Cincinnati and Zincati distribute the updates

To distribute the rpm-ostree commits, Cincinnati and Zincati are used. The latter is a client that regularly asks the Fedora CoreOS Cincinnati server for updates. As soon as a suitable update is available, rpm-ostree prepares a new, bootable file tree. Depending on the chosen strategy, Zincati finalizes the update by rebooting the node.

 

 

What are the advantages?

 

Easy rollback

With libostree it is easy to restore the old state. For this, you just have to boot into the previous rpm-ostree commit. This can also be found as an entry in the grub bootloader menu.

Low effort

Fedora CoreOS can update itself without manual intervention. In combination with Kubernetes, applications are also automatically moved to the currently available nodes.

Flexible Configuration

Zincati offers a simple and flexible configuration that will hopefully allow any user to find a suitable update strategy.

Better Quality

The streamlined image-based approach makes it easier and more accurate to test each version as a whole.

 

Only time will tell whether this hybrid of image and package-based operating system will prevail. Fedora CoreOS – as the basis for our NMS Managed Kubernetes – significantly simplifies the update process while still providing our customers with straightforward control.

Openstack made easy – Volume Backups

Openstack made easy – Volume Backups

Du willst zusätzliche Backups für Deine Volumes? Natürlich am besten ein einem weiteren unabhängigen Storage? Hier erfährst du in wenigen Minuten wie du tägliche Volume Backups in der NWS-Cloud konfigurierst.

Für die Datensicherung von Volumes bietet NWS eine einfache Backup-Funktionalität an. Dadurch werden ausgewählte Volumes in ein zweites unabhängiges Storage kopiert, um deine wichtigsten Daten vor kleinen Dummheiten und großen Katastrophen zu schützen.

Tägliche Backups automatisch erstellen

Tägliche Backups für Deine Volumes lassen sich mit einem Klick im internen Bereich von nws.netways.de aktivieren. Dort kannst du individuell für jedes Volume entscheiden ob ein automatisches nächtliches Backup erstellt wird. Zudem kannst du die maximale Anzahl festlegen, ist diese erreicht wird mit jedem neuen Backup auch das Älteste gelöscht. Somit hast du auch die Kosten im Griff.

Ansicht zur Verwaltung der Backups auf nws.netways.de

Die Konfiguration von täglichen Backups ist damit ähnlich einfach wie die der Snapshots. Aber worin liegt der Unterschied und wieso kann beides Sinn machen?  

Volume Snapshot vs. Volume Backup

Bei einem Volume Snapshot wird im zentralen Openstack Storage ein Snapshot angelegt. Im Falle eines Falles ist dieser auch schnell verfügbar. Der Snapshot wird dreifach über zwei Standorte repliziert. Durch das Replizieren sind Deine Daten somit gegen tägliche Störungen wie Hardwaredefekte gesichert.  Auch vor größeren Katastrophen wie Feuer oder Hochwasser an einem Standort sichern dich die Snapshots ab.  Wieso ist ein Backup trotzdem sinnvoll? 

Schutz gegen menschliche Fehler:

Wird versehentlich ein falsches Volume gelöscht so werden alle aktuellen Daten und alle dazugehörigen Snapshots gelöscht. Hingegen ist ein vorhandenes Backup davon nicht betroffen und das Volume kann wieder hergestellt werden. Natürlich kannst du hoffen, dass dein Team und Du keine Fehler machen, aber nach ein paar Jahren in der IT, hat vermutlich jeder schon die ein oder andere Panne verursacht.

Schutz gegen Bugs:

Auch wenn aktuelle Storage Systeme eine hohe Qualität aufweisen und über jahrelange Erfahrung zuverlässig und sicher laufen, können Fehler in der Software zu Datenverlust führen. Ein zweites unabhängiges Storage für Ihr Backup vermindert dieses Risiko erheblich. 

 

Beim lesen ist dir sicherlich aufgefallen, dass immer die Rede von Volume Snapshot und Backups war. Aber was ist mit virtuellen Server ohne Volume? 

 

Virtuelle Server ohne Volume

Beim starten eines virtuellen Server in der NWS-Cloud hast du die Möglichkeit die root-Partition auf einem Volume abzulegen oder eben auch nicht. Im ersten Fall werden alle Daten im zentralen Storage gespeichert und dadurch auch dreifach über zwei Standorte repliziert und die Backups kannst du wie oben beschrieben verwalten. 

Verzichtest du auf das Volume liegt die root-Partition des virtuellen Server direkt auf einem Hypervisor. Erstellst du in diesem Fall einen Snapshot werden alle Daten vom Hypervisor in das zentrale Storage kopiert. Dort wird es, wie auch die Volumes, dreifach über zwei Standorte repliziert. Somit sind die Daten bereits in zwei unabhängigen Systemen gespeichert und somit vor kleinen Dummheiten und großen Katastrophen sicher. Wie du automatische tägliche Snapshots für Deine virtuellen Server konfigurierst, erfährst du hier.

Du hast bei alle dem replizieren und kopieren die Übersicht verloren? Bei Fragen helfen gerne weiter!

X-Forward-For and Proxy-Protocol

X-Forward-For and Proxy-Protocol

You want to know how to get the IP addresses of your clients in your Kubernetes cluster? In five minutes you have an overview!

From HTTP client to application

In the nginx-Ingress-Controller tutorial, we showed how to make an application publicly accessible. In the case of the NETWAYS Cloud, your Kubernetes cluster uses an Openstack load balancer, which forwards the client requests to an nginx ingress controller in the Kubernetes cluster. This then distributes all requests to the corresponding pods.

With all the pushing around and forwarding of requests, the connection details of the clients get lost without further configuration. Since the problem has not only arisen since Kubernetes, the tried and tested solutions X-Forward-For or Proxy-Protocol are used.

In order not to lose track in the buzzword bingo between service, load balancer, ingress, proxy, client and application, you can look at the path of an HTTP request from the client to the application through the components of a Kubernetes cluster in this example.

Der Weg vom HTTP-Request zur Anwendung im Kubernetes-Cluster

 

Client IP Addresses with X-Forward-For

If you use HTTP, the client IP address can be stored in the X-Forward-For (XFF) and transported further. XFF is an entry in the HTTP header and is supported by most proxy servers. In this example, the load balancer places the client IP address in the XFF entry and forwards the request. All other proxy servers and the applications can therefore recognise in the XFF entry from which address the request was originally sent.

In Kubernetes, the load balancer is configured via annotations in the service object. If you set loadbalancer.openstack.org/x-forwarded-for: true there, the load balancer is configured accordingly. Of course, it is also important that the next proxy does not overwrite the X-Forward-For header again. In the case of nginx, you can set the option use-forwarded-headers in its ConfigMap.

Service

---
kind: Service
apiVersion: v1
metadata:
  name: loadbalanced-service
  annotations:
    loadbalancer.openstack.org/x-forward-for: "true"
spec:
  selector:
    app: echoserver
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP

nginx ConfigMap

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx
data:
  use-forwarded-headers: "true"

Since the HTTP header is used, it is not possible to enrich HTTPS connections with the client IP address. Here, one must either terminate the TLS/SSL protocol at the load balancer or fall back on the proxy protocol.

 

Client Information with Proxy Protocol

If you use X-Forwarded-For, you are obviously limited to HTTP. In order to enable HTTPS and other applications behind load balancers and proxies to access the connection option of the clients, the so-called proxy protocol was invented. Technically, a small header with the client’s connection information is added by the load balancer. The next hop (here nginx) must of course also understand the protocol and handle it accordingly. Besides classic proxies, other applications such as MariaDB or postfix also support the proxy protocol.

To activate the proxy protocol, you must add the annotation loadbalancer.openstack.org/proxy-protocol to the service object. The protocol must also be activated for the accepting proxy.

Service Loadbalancer

---
kind: Service
apiVersion: v1
metadata:
  name: loadbalanced-service
  annotations:
    loadbalancer.openstack.org/proxy-protocol: "true"
spec:
  selector:
    app: echoserver
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 8080Helm Up
    protocol: TCP

Nginx ConfigMap

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx
data:
  use-proxy-protocol: "true"

In most cases, however, you will fall back on the Helm chart of the nginx ingress controller. There, a corresponding configuration is even easier.

 

nginx-Ingress-Controller and Helm

If you use Helm to install the nginx-ingress-controller, the configuration is very clear. The proxy protocol is activated for both the nginx and the load balancer via the Helm values file:

nginx-ingress.values:

---
controller:
  config:
    use-proxy-protocol: "true"
  service:
    annotations:
      loadbalancer.openstack.org/proxy-protocol: true
    type: LoadBalancer

 

$ helm install my-ingress stable/nginx-ingress -f nginx-ingress.values

The easiest way to test whether everything works as expected is to use the Google Echoserver. This is a small application that simply returns the HTTP request to the client. As described in the nginx-Ingress-Controller tutorial, we need a deployment with service and ingress. The former starts the echo server, the service makes it accessible in the cluster and the ingress configures the nginx so that the requests are forwarded to the deployment.

 

Deployment

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echoserver
spec:
  selector:
    matchLabels:
      app: echoserver
  replicas: 1
  template:
    metadata:
      labels:
        app: echoserver
    spec:
      containers:
      - name: echoserver
        image: gcr.io/google-containers/echoserver:1.8
        ports:
          - containerPort: 8080
Service

---
apiVersion: v1
kind: Service
metadata:
  name: echoserver-svc
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
    name: http
  selector:
    app: echoserver

Ingress

---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: echoserver-ingress
spec:
  rules:
  - host: echoserver.nws.netways.de
    http:
      paths:
        - backend:
            serviceName: echoserver-svc
            servicePort: 80

For testing purposes, it’s best to fake your /etc/hosts so that echoserver.nws.netways.de points to the public IP address of your nginx ingress controller. curl echoserver.nws.netways.de will then show you everything that the echo server knows about your client, including the IP address in the X-Forward-For header.

 

Conclusion

In the Kubernetes cluster, the proxy protocol is probably the better choice for most use cases. The well-known Ingress controllers support the proxy protocol and TLS/SSL connections can be configured and terminated in the K8s cluster. The quickest way to find out what information arrives at your application is to use Google’s echo server.

Logging with Loki and Grafana in Kubernetes

Logging with Loki and Grafana in Kubernetes

You already know the most important building blocks for starting your application from our Tutorial-Serie. Are you still missing metrics and logs for your applications? After this blog post, you can tick off the latter.

Logging with Loki and Grafana in Kubernetes – an Overview

One of the best-known, heavyweight solutions for collecting and managing your logs is also available for Kubernetes. This usually consists of Logstash or Fluentd for collecting, paired with Elasticsearch for storing and Kibana or Graylog for visualising your logs.

In addition to this classic combination, a new, more lightweight stack has been available for a few years now with Loki and Grafana! The basic architecture hardly differs from the familiar setups.

Promtail collects the logs of all containers on each Kubernetes node and sends them to a central Loki instance. This aggregates all logs and writes them to a storage back-end. Grafana is used for visualisation, which fetches the logs directly from the Loki instance.

The biggest difference to the known stacks is probably the lack of Elasticsearch. This saves resources and effort, and therefore no triple-replicated full-text index has to be stored and administered. And especially when you start to build up your application, a lean and simple stack sounds appealing. As the application landscape grows, individual Loki components are scaled up to spread the load across multiple servers.

No full text index? How does it work?

Of course, Loki does not do without an index for quick searches, but only metadata (similar to Prometheus) is indexed. This greatly reduces the effort required to run the index. For your Kubernetes cluster, Labels are therefore mainly stored in the index and your logs are automatically organised using the same metadata as your applications in your Kubernetes cluster. Using a time window and the Labels, Loki quickly and easily finds the logs you are looking for.

To store the index, you can choose from various databases. Besides the two cloud databases BigTable and DynamoDB, Loki can also store its index locally in Cassandra or BoltDB. The latter does not support replication and is mainly suitable for development environments. Loki offers another database, boltdb-shipper, which is currently still under development. This is primarily intended to remove dependencies on a replicated database and regularly store snapshots of the index in chunk storage (see below).

A quick example

A pod produces two log streams with stdout and stderr. These log streams are split into so-called chunks and compressed as soon as a certain size has been reached or a time window has expired.

A chunk therefore contains compressed logs of a stream and is limited to a maximum size and time unit. These compressed data records are then stored in the chunk storage.

Label vs. Stream

A combination of exactly the same labels (including their values) defines a stream. If you change a label or its value, a new stream is created. For example, the logs from stdout of an nginx pod are in a stream with the labels: pod-template-hash=bcf574bc8, app=nginx and stream=stdout.

In Loki’s index, these chunks are linked with the stream’s labels and a time window. A search in the index must therefore only be filtered by labels and time windows. If one of these links matches the search criteria, the chunk is loaded from the storage and the logs it contains are filtered according to the search query.

Chunk Storage

The compressed and fragmented log streams are stored in the chunk storage. As with the index, you can also choose between different storage back-ends. Due to the size of the chunks, an object store such as GCS, S3, Swift or our Ceph object store is recommended. Replication is automatically included and the chunks are automatically removed from the storage based on an expiry date. In smaller projects or development environments, you can of course also start with a local file system.

Visualisation with Grafana

Grafana is used for visualisation. Preconfigured dashboards can be easily imported. LogQL is used as the query language. This proprietary creation of Grafana Labs leans heavily on PromQL from Prometheus and is just as quick to learn. A query consists of two parts:
First, you filter for the corresponding chunks using labels and the Log Stream Selector. With = you always make an exact comparison and =~ allows the use of regex. As usual, the selection is negated with !
After you have limited your search to certain chunks, you can expand it with a search expression. Here, too, you can use various operators such as |= and |~ to further restrict the result. A few examples are probably the quickest way to show the possibilities:

Log Stream Selector:

{app = "nginx"}
{app != "nginx"}
{app =~ "ngin.*"}
{app !~ "nginx$"}
{app = "nginx", stream != "stdout"}
Search Expression:


{app = "nginx"} |= "192.168.0.1"
{app = "nginx"} != "192.168.0.1"
{app = "nginx"} |~ "192.*" 
{app = "nginx"} !~ "192$"

Further possibilities such as aggregations are explained in detail in the official documentation of LogQL.

After this short introduction to the architecture and functionality of Grafana Loki, we will of course start right away with the installation. A lot more information and possibilities for Grafana Loki are of course available in the official documentation.

Get it running!

You would like to just try out Loki?

With the NWS Managed Kubernetes Cluster you can do without the details! With just one click you can start your Loki Stack and always have your Kubernetes Cluster in full view!

 

As usual with Kubernetes, a running example is deployed faster than reading the explanation. Using Helm and a few variables, your lightweight logging stack is quickly installed. First, we initialise two Helm repositories. Besides Grafana, we also add the official Helm stable charts repository. After two short helm repo add commands we have access to the required Loki and Grafana charts.

Install Helm

$ brew install helm
$ apt install helm
$ choco install kubernetes-helm

You don’t have the right sources? On helm.sh you will find a brief guide for your operating system.

 

$ helm repo add loki https://grafana.github.io/loki/charts
$ helm repo add stable https://kubernetes-charts.storage.googleapis.com/

 

Install Loki and Grafana

For your first Loki stack you do not need any further configuration. The default values fit very well and helm install does the rest. Before installing Grafana, we first set its configuration using the well-known helm values files. Save them with the name grafana.values.

In addition to the password for the administrator, Loki that has just been installed is also set as the data source. For visualisation, we import a dashboard and the required plugins. And hence you install a Grafana configured for Loki and can get started directly after the deploy.

grafana.values:

---
adminPassword: supersecret

datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
    - name: Loki
      type: loki
      url: http://loki-headless:3100
      jsonData:
        maxLines: 1000

plugins:
  - grafana-piechart-panel

dashboardProviders:
  dashboardproviders.yaml:
    apiVersion: 1
    providers:
      - name: default
        orgId: 1
        folder:
        type: file
        disableDeletion: true
        editable: false
        options:
          path: /var/lib/grafana/dashboards/default

dashboards:
  default:
    Logging:
      gnetId: 12611
      revison: 1
      datasource: Loki

 

The actual installation is done with the help of helm install. The first parameter is a freely selectable name. With its help, you can also quickly get an overview:

$ helm install loki loki/loki-stack
$ helm install loki-grafana stable/grafana -f grafana.values
$ kubectl get all -n kube-system -l release=loki

 

After deployment, you can log in as admin with the password supersecret. To be able to access the Grafana Webinterface directly, you still need a port-forward:

$ kubectl --namespace kube-system port-forward service/loki-grafana 3001:80

The logs of your running pods should be immediately visible in Grafana. Try the queries under Explore and discover the dashboard!

 

Logging with Loki and Grafana in Kubernetes – the Conclusion

With Loki, Grafana Labs offers a new approach to central log management. The use of low-cost and easily available object stores makes the time-consuming administration of an Elasticsearch cluster superfluous. The simple and fast deployment is also ideal for development environments. While the two alternatives Kibana and Graylog offer a powerful feature set, for some administrators Loki with its streamlined and simple stack may be more enticing.

Creating Persistent Volumes in Kubernetes

Creating Persistent Volumes in Kubernetes

You want to create a persistent volume in Kubernetes? Here you can learn how it works with Openstack Cinder in a NWS Managed Kubernetes plan.

Pods and containers are by definition more or less temporary components in a Kubernetes cluster and are created and destroyed as needed. However, many applications such as databases can rarely be operated meaningfully without long-lived storage. With the industry-standard Container Storage Interface (CSI), Kubernetes offers a uniform integration for different storage backends for the integration of persistent volumes. For our Managed Kubernetes solution, we use the Openstack component Cinder to provide persistent volumes for pods. The CSI Cinder controller is already active for NWS Kubernetes from version 1.18.2 and you can use persistent volumes with only a few K8s objects.

 

Creating Persistent Volumes with CSI Cinder Controller

Before you can create a volume, a StorageClass must be created with Cinder as the provisioner. As usual, the K8s objects are sent to your cluster in YAML format and kubectl apply:

storageclass.yaml:

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cinderStorage
provisioner: cinder.csi.openstack.org
allowVolumeExpansion: true
You don’t have any experience with K8s and kubectl? Here you get help for your first steps with Kubernetes (click)

 

With get and describe you can check whether the creation has worked:

$ kubectl apply -f storageclass.yaml
$ kubectl get storageclass
$ kubectl describe storageclass cinderStorage

Based on this storage class, you can now create as many volumes as you like.

 

Persistent Volume (PV) and Persistent Volume Claim (PVC)

You can create a new volume with the help of a peristentVolumeClaim. The PVC claims a persistentVolume resource for you. If no suitable PV resource is available, it is created dynamically by the CSI Cinder Controller. PVC and PV are bound to each other and are exclusively available for you. Without further configuration, a dynamically created PV is immediately deleted when the associated PVC is deleted. This behaviour can be overridden in the StorageClass defined above with the help of the reclaimPolicy.

pvc.yaml:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nginx-documentroot
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: cinderStorage

 

In addition to the name, other properties such as size and accessMode are defined in the PVC-Objekt. After you have created the PVC in the cluster with kubectl apply a new volume is created in the storage backend in the background. In the case of our NETWAYS Managed Kubernetes, Cinder creates a volume as RBD in the Ceph cluster. In the next step, your new volume is mounted in the document root of an Nginx pod.

 

Pods and persistent Volumes

Usually, volumes are defined in the context of a pod and therefore have the same life cycle as them. However, if you want to use a volume that is independent of the pod and container, you can reference the PVC you just created in the volumes section and then include it in the container under volumeMounts. In this example, the document root of a Nginx is replaced.

deployment.yaml:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  selector:
    matchLabels:
      app: nginx
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx
        ports:
        - containerPort: 80
          protocol: TCP
        volumeMounts:
          - mountPath: /usr/share/nginx/html
            name: documentroot
      volumes:
      - name: documentroot
        persistentVolumeClaim:
          claimName: nginx-documentroot
          readOnly: false
service.yaml:

---
apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
spec:
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
    name: http
  selector:
    app: nginx

 

Kubernetes and the CSI Cinder Controller naturally ensure that your new volume and the associated pods are always started at the same worker node. With kubectl you can also quickly adjust the index.html and start the K8s proxy and you can already access your index.html in the persistent volume:

$ kubectl exec -it deployment/nginx — bash -c ‘echo “CSI FTW” > /usr/share/nginx/html/index.html’
$ kubectl proxy

 

Conclusion

With the CSI Cinder Controller, you can create and manage persistent volumes quickly and easily. Further features for creating snapshots or enlarging volumes are already included. And options such as Multinode Attachment re already being planned. So nothing stands in the way of your database cluster in Kubernetes and the next exciting topic in our Kubernetes Blog series has been decided!