OpenStack made easy – Creating and Using S3 Object Storage

OpenStack made easy – Creating and Using S3 Object Storage

In times of high availability and multiple web servers, the balancing act between central data storage, data security and fast access times must somehow be achieved. This is precisely why more and more users are now using technologies that entice them with buzzwords such as S3, buckets, object storage and Swift. We at NETWAYS Web Services have been offering this for some time, but we have noticed that it is apparently not documented enough for our customers. To shed some light on this, we have created this article. This explanation is therefore given piece by piece:

  • Why Object Storage
  • Creating S3-Credentials via Openstack-API
  • Configuration of s3cmd
  • Create a New Bucket
  • Place Files in the Bucket
  • List Contents of Bucket
  • Download Files from a Bucket
  • Further s3cmd Application Uses
  • Conclusion

Why Object Storage?

S3 Object storage offers many advantages over locally stored data or classic NFS storage because an NFS setup often consists of only one server and thus represents the single point of failure. The administrator must also always be concerned about the performance of the storage and the operating web servers. Another advantage of object storage is the size and number of data to be stored. This is because, in the case of object storage, the data is unlimited. The provider of the object storage (i.e. in the current case NWS) ensures that the space here never runs out. With your own NFS, the size must be taken into account in the planning from the beginning and the customer pays for the planned reserve in the long run. With object storage, only the volume that is used is paid for. Also worth mentioning is the relief of the web server, especially with regard to locally stored data. Imagine the following architecture: The application is run on several web servers, images and other objects are stored in the object storage and delivered directly to the website visitor through intelligent integration without burdening the own web servers or their bandwidth. In short, you tell your application not to deliver this image, but the application only passes on the path of the stored image to the visitor. The image is ultimately delivered directly from the Rados gateway behind it and was never delivered from the app server. This means that it reaches the website visitor quickly, saves bandwidth and reduces the load on one’s own web server, and is nevertheless kept centralised. By the way, the object storage uses our Ceph, where all the data is distributed over a large number of systems and naturally also has corresponding replica sets.

Creating S3-Credentials via Openstack-API

To be able to use an object storage at NWS, an OpenStack account is required. Once this is ready, the process can begin. In OpenStack, go to the menu item “API Access” and download the OpenStack RC file (top right). The script that has just been downloaded is now started on the local machine. All that is required is the OpenStack password. But beware, an OpenStack client is required.

source 9231-openstack-4707c-openrc.sh 
Please enter your OpenStack Password for project 9231-openstack-4707c as user 9231-openstack-4707c:

Now the EC2 credentials for the S3 access must be created, for this purpose the command

openstack ec2 credentials create

The result provides output similar to the following

+------------+-----------------------------------------------------------------------------------------------------------------------------------------+
| Field      | Value                                                                                                                                   |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------+
| access     | aceab8b6b85c51f8d3b305aec1af39c2                                                                                                        |
| links      | {'self': 'https://cloud.netways.de:5000/v3/users/24c527b4929f10bd6c6ee09af4b30be2/credentials/OS-EC2/aceab8b6b85c51f8d3b305aec1af39c2'} |
| project_id | a40c737aaf3027d87e98f309f9f671d4                                                                                                        |
| secret     | e5d312a7627cdf2a87ac4322a6d7716b                                                                                                        |
| trust_id   | None                                                                                                                                    |
| user_id    | 24c527b4929f10bd6c6ee09af4b30be2                                                                                                        |
+------------+-----------------------------------------------------------------------------------------------------------------------------------------+

Configuration of s3cmd

For further use, s3cmd is now configured. This requires an installed s3cmd. The setup starts by means of

s3cmd --configure

In the following wizard, the previously created data is entered, which then looks something like this (relevant entries in bold)

s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to the user manual for a detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: aceab8b6b85c51f8d3b305aec1af39c2
Secret Key: e5d312a7627cdf2a87ac4322a6d7716b
Default Region [US]:

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: rgw1.netways.de

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: rgw1.netways.de/%(bucket)s

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]:

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name:

New settings:
Access Key: aceab8b6b85c51f8d3b305aec1af39c2
Secret Key: e5d312a7627cdf2a87ac4322a6d7716b
Default Region: US
S3 Endpoint: rgw1.netways.de
DNS-style bucket+hostname:port template for accessing a bucket: rgw1.netways.de/%(bucket)s
Encryption password:
Path to GPG program: None
Use HTTPS protocol: True
HTTP Proxy server name:
HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] n

Save settings? [y/N] y
Configuration saved to '/Users/gmimietz/.s3cfg'

In order for the S3 commands to work as desired, a small adjustment has to be made to the config that was just created

vim ~/.s3cfg

There the line

signature_v2 = False

is adapted to

signature_v2 = True

After saving, the application can now begin.

Creating a new Bucket

A new bucket is created using the following command

s3cmd mb s3://myfirstbucket

To check the creation of the bucket, it is listed with the command

s3cmd ls

Place Files in the Bucket

Time to upload the first files. The put parameter is used for this

s3cmd put /Users/gmimietz/Downloads/water.liquid s3://myfirstbucket/

List Contents of Bucket

To see what is in the bucket that has just been filled, the ls command lists the contents. However, this is now expanded by the bucket name

s3cmd ls s3://myfirstbucket 
2021-02-02 11:04 402088862 s3://myfirstbucket/water.liquid

Download files from a Bucket

The file that has just been uploaded can now be downloaded using the get parameter.

s3cmd get s3://myfirstbucket/water.liquid

Further s3cmd Application Uses

Below are some helpful commands: Move an object from one bucket to another:

s3cmd mv s3://myfirstbucket/water.liquid s3://mysecondbucket

Store an object encrypted in the bucket (requires a predefined passphrase). (s3cmd –configure)

s3cmd put fire.element s3://myfirstbucket -e

A very valuable command is the sync to synchronise a local directory with a bucket (everything below /important-data is synced to s3://myfirstbucket/backup/)

s3cmd sync /important-data/ s3://myfirstbucket/backup/

To conserve bandwidth in certain use cases, the –limit-rate option helps, this defaults to bytes but the option also recognises a k or m suffix (kilo/megabyte) per second

s3cmd put myfile s3://myfirstbucket --limit-rate=1k

Conclusion

Of course, there are many more parameters for the s3cmd, all of which can be listed using –help. When using s3, however, modern and elegant procedures can also be included, e.g. the lifetime of a file with a predefined deletion date. In OpenStack you can also see what the object storage is doing by navigating to the corresponding menu via Project>Object Storage. By the way, our object storage also works with swift – so it is also addressed by OpenStack itself. So a mixed operation of S3 and swift is possible at your discretion. We have not yet documented swift further, but our friendly MyEngineers will be happy to provide information on request.

The prices for our object storage and everything else we offer at NWS can be found here at any time.

OpenStack made easy – Corosync Cluster with Failover IP

OpenStack made easy – Corosync Cluster with Failover IP

One of the first customer requirements you usually read is High availability. For a long time now, it has been more the norm that the project is still accessible without problems even in the event of partial failures and that “single points of failure” are avoided. A Corosync / Pacemaker cluster is often used for this purpose; the technology behind it has been tried and tested for more than a decade – the basic idea behind it is: you create virtual resources that can be started on every connected node.

The following describes how to create a Corosync / Pacemaker cluster using Ubuntu. If you are already familiar with these steps and would just like to know how to store the failover IP in OpenStack, you will find the relevant information further down in the article.

Create a Corosync / Pacemaker cluster

Installation

The first step is to install the necessary packages. crmsh offers a shell that can be used to control the cluster.

root@test-node-1:~# apt install corosync pacemaker crmsh

Configuration

Then the authkey is created, which is needed for the communication between the nodes. Without this, the service may not be started.

root@test-node-1:~# corosync-keygen

This can take some time, depending on how busy the server is. For newly created VMs, it would take too long and you can use the following snippet, for example, which writes random data to the hard disk – but please always keep in mind that you should not write all over the hard disk, so if necessary you should adapt the command to your own needs!

while /bin/true; do dd if=/dev/urandom of=/tmp/entropy bs=1024 count=10000; for i in {1..50}; 
do cp /tmp/entropy /tmp/tmp_$i_$RANDOM; done; rm -f /tmp/tmp_* /tmp/entropy; done

When this is done, you copy the authkey onto both servers at /etc/corosync/authkey .

root@test-node-1:~# cp authkey /etc/corosync/authkey
root@test-node-1:~# chown root.root /etc/corosync/authkey
root@test-node-1:~# chmod 0400 /etc/corosync/authkey
root@test-node-1:~# scp /etc/corosync/authkey root@test-node-2:/etc/corosync/authkey

The cluster is then configured in the file /etc/corosync/corosync.conf, in which, among other things, the private IPs of the cluster nodes are defined. This file is also identical on all nodes.

totem {
  version: 2
  cluster_name: test-cluster
  transport: udpu
  interface {
    ringnumber: 0
    bindnetaddr: 172.16.0.0
    broadcast: yes
    mcastport: 5407
  }
}

nodelist {
  node {
    ring0_addr: 172.16.0.10
  }
  node {
    ring0_addr: 172.16.0.20
  }
}

quorum {
  provider: corosync_votequorum
}

logging {
  to_logfile: yes
  logfile: /var/log/corosync/corosync.log
  to_syslog: yes
  timestamp: on
}

service {
  name: pacemaker
  ver: 1
}

The cluster then needs to be restarted, therefore all data is transferred. At this point, the cluster should already display its status and recognise all nodes. Initially, this may take a few moments.

root@test-node-1:~# systemctl restart corosync && systemctl restart pacemaker

root@test-node-1:~# crm status
Stack: corosync
Current DC: test-node-1 (version 1.1.18-2b07d5c5a9) - partition with quorum
Last updated: Mon Dec 7 15:40:20 2020
Last change: Mon Dec 7 15:40:20 2020 by hacluster via crmd on test-node-1

2 nodes configured
0 resource configured

Online: [ test-node-1 test-node-2 ]

By means of the crm, the cluster can be controlled and its current configuration accessed. The configuration should be very similar to the following:

root@test-node-1:~# crm configure show
node 2886729779: test-node-1
node 2886729826: test-node-2
property cib-bootstrap-options: \
  have-watchdog=false \
  dc-version=1.1.18-2b07d5c5a9 \
  cluster-infrastructure=corosync \
  cluster-name=test-cluster \
  stonith-action=reboot \
  no-quorum-policy=stop \
  stonith-enabled=false \
  last-lrm-refresh=1596896556 \
  maintenance-mode=false
rsc_defaults rsc-options: \
  resource-stickiness=1000

Now this configuration can be edited directly and resources can be defined. This can also be done using “crm configure“, but in this example the configuration is taken over directly.

root@test-node-1:~# crm configure edit

node 2886729779: test-node-1
node 2886729826: test-node-2
primitive ha-vip IPaddr2 \
  params ip=172.16.0.100 cidr_netmask=32 arp_count=10 arp_count_refresh=5 \
  op monitor interval=10s \
  meta target-role=Started
property cib-bootstrap-options: \
  have-watchdog=false \
  dc-version=1.1.18-2b07d5c5a9 \
  cluster-infrastructure=corosync \
  cluster-name=test-cluster \
  stonith-action=reboot \
  no-quorum-policy=ignore \
  stonith-enabled=false \
  last-lrm-refresh=1596896556 \
  maintenance-mode=false
rsc_defaults rsc-options: \
  resource-stickiness=1000

The attentive reader will notice that the “no-quorum-policy” has also been adapted. This is important for the operation of a cluster that consists of only two nodes, as no quorum could be formed if one node fails.

root@test-node-1:~# crm status
Stack: corosync
Current DC: test-node-1 (version 1.1.18-2b07d5c5a9) - partition with quorum
Last updated: Mon Dec 7 15:40:20 2020
Last change: Mon Dec 7 15:45:21 2020 by hacluster via crmd on test-node-1

2 nodes configured
1 resource configured

Online: [ test-node-1 test-node-2 ]

Full list of resources:

ha-vip (ocf::heartbeat:IPaddr2): Started test-node-1

 

Configure Failover IP in OpenStack

There are two possibilities to store the IP in OpenStack. Firstly, you can navigate to the VM’s port in the web interface via Network -> Networks -> said network -> Ports and add the desired IP to the tab “Allowed address pairs“. On the other hand, this is also possible via OpenStack CLI Tool:

openstack port list --server test-node-1
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+
| ID                                   | Name | MAC Address       | Fixed IP Addresses                                                         | Status |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+
| 0a7161f5-c2ff-402c-9bf4-976215a95cf3 |      | fa:16:3e:2a:f3:f2 | ip_address='172.16.0.10', subnet_id='9ede2a39-7f99-48c8-a542-85066e30a6f3' | ACTIVE |
+--------------------------------------+------+-------------------+----------------------------------------------------------------------------+--------+

The additionally permitted IP address is added to the port as follows. A complete network can also be defined here if several IP resources are to be created.

openstack port set 0a7161f5-c2ff-402c-9bf4-976215a95cf3 --allowed-address ip-address=172.16.0.100

This step must be repeated for both servers. After this, the IP is also accessible in the OpenStack project. If this is not the case, it may help to move the IP resource to the other node, therefore the IP is announced there. However, this should not be the case due to the ARP settings of the resource set above.
The additionally permitted IP address is added to the port as follows. A complete network can also be defined here if several IP resources are to be created.

crm resource migrate ha-vip test-node-2

The whole thing doesn’t work as planned, or there are still further questions? Our MyEngineers are sure to help!

OpenStack made easy – Scaling on Demand

OpenStack made easy – Scaling on Demand

Depending on the type of production, it may be worthwhile for some server operators to automatically create virtual machines for a certain period of time and – after the work is done – to delete them again just as automatically; for example, if a computing job with their own hardware would take longer than is acceptable. Our cloud is happy to take care of this for you – even if it involves resources other than processors.

In this example, I will go over the first steps regarding this scenario, how to speak against the API of our OpenStack platform using Linux CLI.
To do this, you need an OpenStack client on the host running the scaling. On Ubuntu, for example, this would be the package python-openstackclient .
Next, the project-specific “OpenStack RC File v3” from the OpenStack WebUI is necessary. This file can be downloaded after logging into the project via the drop-down menu with one’s own project ID at the top right.

You source the file, therefore the client also knows which project to talk to against the API – requires password entry:
source XXXX-openstack-XXXXX-openrc.sh

In order to be able to set the options to be passed on for the start of a new instance, it is now possible to search for values (UUID; except for the key pair) for these, decide on the correct ones and note them:

  • Source, the installation image to be used:
        openstack image list
  • Flavor, i.e. what dimensions the VM to be built should have:
        openstack flavor list
  • Networks, here I recommend the project’s own subnet, which is secured from the outside.:
        openstack network list
  • Security Groups, at least the default security group is recommended here, therefore all VMs can talk to each other fully, at least within the project.:
        openstack security group list
  • Key Pair, to connect via SSH:
        openstack keypair list

Then the instance can already be started – if there is more than one value to be passed per option, list the option several times with one value each, the instance or server name last:
    openstack server create --image $imID --flavor $flID --network $nID --security-group $sgID --key-name $Name $Servername

Ta-da, the VM is ready and willing to make its contribution to the day-to-day business.
If you would like to have more than one machine, z. B. three , additionally give these options in front of the server name:
    --min 3 --max 3

To save your wallet, the servers can be deleted after the work is done:
    openstack server list
    openstack server delete $deID

This could also be done automatically, i.e. without looking for the ID of the instance:
    deID=`openstack server list | grep Instanzname | cut -d ' ' -f 2` ; openstack server delete $deID

As I said, it is a good idea to include the Create, Compute and Delete commands in a script. If your own bash skills are not enough for you, you are welcome to contact our MyEngineers . The interposition of a load balancer is also not a problem here.

OpenStack made easy – Backup and Snapshot

OpenStack made easy – Backup and Snapshot

Sooner or later, everyone who has a server running will reach the point where the VM (or parts of it) irreversibly ” breaks” – by whatever means.
Those who have dedicated themselves to backing up their data in advance are now clearly at an advantage and can expect significantly lower adrenaline levels – especially if the last backup was less than 24 hours ago. In our NWS backups and snapshots are easily configured for your vms to take nightly backups automatically.
In case of vms with SSD system volume there is just the menu feature backup.

We set the retention period to a week for example and confirm:

The NWS server list will offer an summary like this for these three cases:
Vm Example: SSD: backup
Vm Example2: Ceph volume: backup
Vm Example3: Ceph volume: snapshot

After their automatical production each night by us, they will become visible under their
particular menu item.

To be able to use a taken backup, we’ll have to switch to  Horizon (OpenStack web UI) .
Five days after configuring the backups, you would find something like that for our three servers for example:

Backup (of SSD; findable under: Compute > Images

Volume Backup (of Ceph volume; findable under: Volumes > Backups)

Volume Snapshot (of Ceph volume; findable under: Volumes > Snapshots)

The last-mentioned Snapshots are increments of its underlying volume and are cheaper in comparison to backups; more shortly.

For those who wish to enter these backup settings manually via metadata:

what to modify:

Vm on SSD

Backup

Vm
(Compute > Instances)

nws_backup=true

rotation=7

Vm with
Ceph volume

Backup

Volume
(Volume > Volumes)

cinder-backup=true

cinder-backup-rotation=7

Vm with
Ceph volume

Snapshot

Volume
(Volume > Volumes)

nws_snapshot=true

rotation=7

Those who want to make their backups/snapshots available again, can e. g.

  • create a volume from them and attach it to an existing vm
  • (delete the crashed vm and) launch an new vm from it (vms with Ceph volume) or
  • rebuild the machine with a particular image/backup (vms on SSD)

Snapshot vs. Backup?

During the production of a volume snapshot it is created in the central OpenStack storage. It is quickly available if needed. This snapshot is replicated threefold at two locations. Due to this replication your data is secured against everyday incidents like hardware failure. Snapshots also save you from bigger desasters like fire or flood at one of the sites. Why is a backup reasonable anyway?

Protection against human error: Is a wrong volume deleted accidentially, all of its data and snapshots are being deleted accordingly. An available backup won’t be affected from that and the volume can be recovered.

Protection against bugs: Albeit current storage systems hold in high quality and run reliably for years due to experience, mistakes in software can lead to data loss. A second independent storage for your backup lessens this hazard considerably.

You got lost replicating/copying? In case of any questions we’d gladly like to help.

OpenStack made easy – Monitoring for machines with Icinga 2 Master

OpenStack made easy – Monitoring for machines with Icinga 2 Master

With our OpenStack Cloud it is very easy to build your own environment according to your own ideas. Quickly and easily start a few machines with Terraform, make the service available to the outside world with an attached floating IP and associated Security Group and the project is up and running.

But no environment runs without errors and monitoring is a big issue – you like to know in advance of your own users, or customers, when something doesn’t quite work as it should. I think every reader of this blog is aware of the importance of monitoring, but also the evaluation of performance data. How can I easily monitor my OpenStack environment, especially if my servers are not accessible from the outside world? We have prepared something!

In addition to our IaaS offer, we also provide various SaaS solutions. Among them is the app Icinga 2 Master, which provides a complete Icinga 2 Master, including Graphite and Grafana, within a few minutes.

Once this has been started, after logging in you will find various integration scripts for different operating systems under the tab “Add Agent” – depending on the browser language.

You simply download it to the server according to the instructions, execute it and the server is connected to the Icinga2 master.

Everything important is automated here. By default, some checks are created directly and with the help of the director it is also easy to distribute further checks for its hosts. The API of the director can also be addressed directly, so there are almost no limits. In addition, you will find some graphs on the performance data of the connected agent directly with the respective check. Therefore not only problems can be detected, but also trends are visualised directly. This data is kept for one year in our packages to ensure a long-term overview.

The first month of Icinga 2 Master is also free – it’s worth a try. Our MyEngineer is also happy to help with the setup!

OpenStack made easy – Manage and assign security groups

OpenStack made easy – Manage and assign security groups

After clicking on the first new instance in our OpenStack web interface and assigning an SSH public key with which to connect to this VM, the newly minted administrator is faced with the small problem that he/she cannot access the instance from the outside; we have the “default” security group to thank for this.

It contains these rules:


– Allow incoming connections with any protocol, on any port, but only from hosts on the internal network that also use the “default” security group (IPv4 and IPv6)
– Allow outgoing connections with any protocol, on any port and to anywhere (IPv4 and IPv6)

In this way, the protection of the new VM is ensured. Any connecting person from the outside can only really get through the access opening that has been provided and created for this purpose. There are two ways to create such an opening: A new security group can be created and provided with a rule, or a rule can be added to the default security group. The second has the disadvantage that the rule to be entered will be applied to all new instances with the default security group in the future, which will not always make sense on all VMs.

Create a new OpenStack security group

Click on: Network > Security groups > “+ create security group”.

A dialogue box appears in which, at will but obligatory, a name must be entered (and optionally a description can be entered). Here I call the new group “Example”, but any other name that follows its own grouping strategies, for example, will do. Then still Create Security Group.

Then the following appears in the list:

Add SSH reachability from external as a rule of a security group

Click on: Network > Security Groups > Manage Rules (on the security group that is to be edited).

In a new, still unprocessed security group, you will only find one rule each for leaving (IPv4 and IPv6). Continue with: “+ Add rule”. In the drop-down menu Rule, select the sub-item SSH and “Add”.

– If a security group (e.g. default) already assigned to the VM has been provided with this rule, the rule is applied immediately and the VM can be contacted via the CLI.

– If a rule has been created in a new security group that has not yet been assigned to the VM:

Assign a new security group to a VM

Navigate: Compute > Instances > Drop-down arrow (far right of the instance to be modified) > Edit security groups. Under “All security groups” you will find the new one, with the white-on-blue plus add the new security group to the “Instance security groups” and “Save”.

Create ICMP reachability from external as a rule

Network > Security Groups > Manage Rules (at the security group to be edited) > “+ Add Rule” > Rule = “All ICMP” > Add.

A rule for HTTP / HTTPS, for example, or the following also works in the same way

Rule example with more chicanery
External accessibility with TCP in port range 65530-65535 only from IP 200.135.41.125

Network > Security Groups > Manage Rules (at the security group to be edited) > “+ Add Rule” > Rule = “Custom TCP Rule > Open Port = Port Range >
“From-Port” = 65530 > “To-Port” = 65535 > CIDR = 200.135.41.125/32 > “Add”

For those who find setting up and configuring new VMs too extensive or difficult, MyEngineer will be happy to create any desired setup.

The first project can be started in our NWS Cloud.