Robin Bowes – YO61 « DevOps in the wilds of North Yorkshire

If you work with rpm-based systems you will probably have seen content like this in the repo config files:

[base] name=CentOS-$releasever - Base mirrorlist=http://mirrorlist.centos.org/?release=$releasever&arch=$basearch&repo=os&infra=$infra #baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/ gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

The items in bold are yum variables.

Today, I needed to install i386 packages on a system running an x86_64 kernel (don't ask!).

Here's how I did it:

echo i386 > /etc/yum/vars/basearch

Documentation here: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/sec-using_yum_variables

From DevOps, Docker, and Empathy by Jérôme Petazzoni (title shamelessly stolen from a talk by Bridget Kromhout):

The DevOps movement is more about a culture shift than embracing a new set of tools. One of the tenets of DevOps is to get people to talk together.

Implementing containers won’t give us DevOps.

You can’t buy DevOps by the pound, and it doesn’t come in a box, or even in intermodal containers.

It’s not just about merging “Dev” and “Ops,” but also getting these two to sit at the same table and talk to each other.

Docker doesn’t enforce these things (I pity the fool who preaches or believes it) but it gives us a table to sit at, and a common language to facilitate the conversation. It’s a tool, just a tool indeed, but it helps people share context and thus understanding.

Ever done a load of refactoring on some terraform code and wonder what variables are still required? This command gives you a sorted list of all occurences of the pattern "var.something":

ack -h 'var\.[\w]+' | perl -ne 'print "$1\n" while /var\.(\w+)/g' | sort -u

Updated 2017-11-30: match multiple variables on the same line

Updated 2017-12-06: complete re-write to support nested braces in variable description
Updated 2017-12-05: support minimal variable declaration, eg. variable "foo" {}

When developing terraform code, it is easy to end up with a bunch of variable definitions that are listed in no particular order.

Here's a bit of python code that will sort terraform variable definitions. Use it as a filter from inside vim, or as a standalone tool if you have all your variable definitions in one file.

eg:

tf_sort < variables.tf > variables.tf.sorted
mv variables.tf.sorted variables.tf

Here's the code:

#!/usr/bin/env python
# sort terraform variables

import sys

import regex

# this regex matches terraform variable definitions
# we capture the variable name so we can sort on it
pattern = regex.compile('''
(                   # capturing group #1
    variable        # literal text
    \s*             # white space (optional)
    "               # literal quote character
)
(                   # capturing group #2 (variable name)
    [^"]+           # anything except another quote
)
(                   # capturing group #3
    "               # literal quote character
    \s*             # white space (optional)
)
(?<rec>             # capturing group named "rec"
    {               # literal left brace
    (?:             # non-capturing group
        [^{}]++     # anything but braces one or more times with no backtracking
        |           # or
        (?&rec)     # recursive substitution of group "rec"
    )*              # this group can appear 0 or more times
    }               # literal right brace
)
''', regex.VERBOSE)


def process(content):
    # sort the content (a list of tuples) on the second item of the tuple
    # (which is the variable name)
    matches = sorted(regex.findall(pattern, content), key=lambda x: x[1])

    # iterate over the sorted list and output them
    for match in matches:
        print(''.join(map(str, match)))

        # don't print the newline on the last item
        if match != matches[-1]:
            print('')


# check if we're reading from stdin
if not sys.stdin.isatty():
    stdin = sys.stdin.read()
    if stdin:
        process(stdin)

# process any filenames on the command line
for filename in sys.argv[1:]:
    with open(filename) as f:
        process(f.read())

If you manually delete a resource that is being managed by terraform, it it not removed from the state file and becomes "orphaned".

You many see errors like this when running terraform:

1 error(s) occurred:
* aws_iam_role.s3_readonly (destroy): 1 error(s) occurred:
* aws_iam_role.s3_readonly (deposed #0): 1 error(s) occurred:
* aws_iam_role.s3_readonly (deposed #0): Error listing Profiles for IAM Role (s3_readonly) when trying to delete: NoSuchEntity: The role with name s3_readonly cannot be found.

This prevents terraform from running, even if you don't care about the missing resource such as when you're trying to delete everything, ie. running terraform destroy.

Fortunately, terraform has a command for exactly this situation, to remove a resource from the state file: terraform state rm <name of resource>

In the example above, the command would be terraform state rm aws_iam_role.s3_readonly

Update 2017-08-25: Additional characters excluded due to problems with password handling in PHP applications using the Laravel framework on AWS ElasticBeanstalk

I needed to generate random master passwords for several Amazon RDS MySQL instances.

The specification is as follows:

The password for the master database user can be any printable ASCII character except "/", """, or "@". Master password constraints differ for each database engine.

MySQL, Amazon Aurora, and MariaDB

Must contain 8 to 41 characters.

I came up with this:

head -n 1 < <(fold -w 41 < <(LC_ALL=C tr -d '@"$`!/'\'\\ < <(LC_ALL=C tr -dc '[:graph:]' < /dev/urandom)))

If you prefer to use pipes (rather than process substitution) the command would look like this:

cat /dev/urandom | LC_ALL=C tr -dc '[:graph:]' | tr -d '@"$`!/'\'\\ | fold -w 41 | head -n 1

Notes:

take a stream of random bytes
remove all chars not in the set specified by [:graph:], ie. get rid of everything that is not a printable ASCII character
remove the chars that are explicitly not permitted by the RDS password specification and others that can cause problems if not handled correctly
split the stream into lines 41 characters long, ie. the maximum password length
stop after the first line

I recently wanted to monitor my home NAS with prometheus, which essentially means running node_exporter on it.

The NAS is running FreeNAS, which is, at time of writing, based on FreeBSD 10.3-STABLE, so I need to somehow get hold of node_exporter for FreeBSD.

Rather than installing go and building node_exporter directly on the FreeNAS machine, I will build it on a Vagrant machine (running FreeBSD 10.3-STABLE) then copy it to the NAS. I will do this by creating a FreeBSD package which can be installed on the FreeNAS machine.

One final complication: I want to use an unreleased version of node_exporter that contains the fix for a bug I reported recently. So, I will need to build against an aribtrary github commit hash.

First, I created a simple Vagrant file as follows:

Vagrant.configure("2") do |config|
  config.vm.guest = :freebsd
  config.vm.synced_folder ".", "/vagrant", id: "vagrant-root", disabled: true
  config.vm.box = "freebsd/FreeBSD-10.3-STABLE"
  config.ssh.shell = "sh"
  config.vm.base_mac = "080027D14C66"

  config.vm.provider :virtualbox do |vb|
    vb.customize ["modifyvm", :id, "--memory", "1024"]
    vb.customize ["modifyvm", :id, "--cpus", "2"]
    vb.customize ["modifyvm", :id, "--hwvirtex", "on"]
    vb.customize ["modifyvm", :id, "--audio", "none"]
    vb.customize ["modifyvm", :id, "--nictype1", "virtio"]
    vb.customize ["modifyvm", :id, "--nictype2", "virtio"]
  end
end

I ran vagrant up and vagrant ssh to start the machine and jump onto it.

I then ran the following commands to initialize the FreeBSD ports:

sudo -i
portsnap fetch
portsnap extract

The port I want is in /usr/ports/sysutils/node_exporter, so normally I would simply change into that directory and build the package:

cd /usr/ports/sysutils/node_exporter
make package

This will create a .txz file which is a FreeBSD pkg and can be installed on FreeNAS.

However, to build node_exporter from a specific github commit hash I need to make a couple of changes to the Makefile.

First, I identify the short hash of the commit I want to build – 269ee7a, in this instance.

Then I modify the Makefile, adding GH_TAGNAME and modifying PORTVERSION, something like this:

PORTVERSION=    0.13.0.269ee7a
...
GH_TAGNAME=     269ee7a

Finally, I generate distinfo to match the modified version, and build the package:

make makesum
make package

I now copy node_exporter-0.13.0.269ee7a.txz to my FreeNAS box, where I can install and run it.

On the FreeNAS box, I use the following commands to install, enable, and start the node_exporter service:

pkg add ./node_exporter-0.13.0.269ee7a.txz
cat <<EOT >> /etc/rc.conf

# enable prometheus node_exporter
node_exporter_enable="YES"
EOT
service node_exporter start

I can now configure prometheus to scrape metrics from this machine.

This post documents my experiences with trying to run kubernetes on a five-node Raspberry Pip 2 cluster.

I started by setting up each of the five Raspberry Pis with Hypriot v1.1.1.

I have internal DNS & DHCP services on my Lab network so I used hardware addresses to ensure each node has a DNS entry and always gets the same IP, as follows:

node01.rpi.yo61.net 192.168.1.141 node02.rpi.yo61.net 192.168.1.142 node03.rpi.yo61.net 192.168.1.143 node04.rpi.yo61.net 192.168.1.144 node05.rpi.yo61.net 192.168.1.145

On each node, I ensured all packages were up-to-date:

apt-get update && apt-get -y upgrade

I used the kubeadm guide for the following steps.

Add a new repo and install various kubernetes commands on all nodes:

curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - cat <<EOF > /etc/apt/sources.list.d/kubernetes.list deb http://apt.kubernetes.io/ kubernetes-xenial main EOF
apt-get update apt-get install -y kubelet kubeadm kubectl kubernetes-cni

I designated node01 as the master and ran the following commands on the master only:

Initialise the master:

kubeadm init --pod-network-cidr=10.244.0.0/16

Install flannel networking:

export ARCH=arm curl -sSL "https://github.com/coreos/flannel/blob/master/Documentation/kube-flannel.yml?raw=true" | sed "s/amd64/${ARCH}/g" | kubectl create -f -

Check everything is running OK:

$ kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system dummy-2501624643-8gpcm 1/1 Running 0 24m kube-system etcd-node01.rpi.yo61.net 1/1 Running 0 23m kube-system kube-apiserver-node01.rpi.yo61.net 1/1 Running 0 23m kube-system kube-controller-manager-node01.rpi.yo61.net 1/1 Running 0 23m kube-system kube-discovery-2202902116-j9zjn 1/1 Running 0 24m kube-system kube-dns-2334855451-lu3qh 3/3 Running 0 22m kube-system kube-flannel-ds-p32x1 2/2 Running 0 15m kube-system kube-proxy-28edm 1/1 Running 0 22m kube-system kube-scheduler-node01.rpi.yo61.net 1/1 Running 0 23m

Join the other nodes:

kubeadm join --token=c2a0a6.7dd1d5b1c26795ef 192.168.1.141

Check they've all joined correctly:

$ kubectl get nodes

NAME STATUS AGE node01.rpi.yo61.net Ready 27m node02.rpi.yo61.net Ready 45s node03.rpi.yo61.net Ready 54s node04.rpi.yo61.net Ready 16s node05.rpi.yo61.net Ready 25s

Install the dashboard:

export ARCH=arm curl -sSL "https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml" | sed "s/amd64/${ARCH}/g" | kubectl create -f -

You should now have something like this:

$ kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system dummy-2501624643-8gpcm 1/1 Running 0 1h kube-system etcd-node01.rpi.yo61.net 1/1 Running 0 1h kube-system kube-apiserver-node01.rpi.yo61.net 1/1 Running 0 1h kube-system kube-controller-manager-node01.rpi.yo61.net 1/1 Running 0 1h kube-system kube-discovery-2202902116-j9zjn 1/1 Running 0 1h kube-system kube-dns-2334855451-lu3qh 3/3 Running 0 1h kube-system kube-flannel-ds-2g1tr 2/2 Running 0 1h kube-system kube-flannel-ds-a2v7q 2/2 Running 0 1h kube-system kube-flannel-ds-iqrs2 2/2 Running 0 1h kube-system kube-flannel-ds-p32x1 2/2 Running 0 1h kube-system kube-flannel-ds-qhfvc 2/2 Running 0 1h kube-system kube-proxy-0agjm 1/1 Running 0 1h kube-system kube-proxy-28edm 1/1 Running 0 1h kube-system kube-proxy-3w6e8 1/1 Running 0 1h kube-system kube-proxy-fgxxp 1/1 Running 0 1h kube-system kube-proxy-ypzyd 1/1 Running 0 1h kube-system kube-scheduler-node01.rpi.yo61.net 1/1 Running 0 1h kube-system kubernetes-dashboard-3507263287-so0mw 1/1 Running 0 1h

To be continued…

When using rsync to copy rsnapshot archives you should use the --hard-links option:

       -H, --hard-links
              This tells rsync to look for hard-linked files in the source and
              link together the corresponding files on the destination.  With-
              out  this option, hard-linked files in the source are treated as
              though they were separate files.

Thanks to this post: https://www.cyberciti.biz/faq/linux-unix-apple-osx-bsd-rsync-copy-hard-links/

As noted in my previous post, migrating a VM from one SmartOS node to another is pretty straight-forward, and is documented on the SmartOS wiki.

One thing that is not mentioned is that if your VM is based on an image, then that image must already be installed/imported on the target host. If it is not, you will see an error like this:

[root@source ~]# vmadm send 0983fe0f-677e-4f54-b5cf-40184d041b6d | ssh smart01 vmadm receive Invalid value(s) for: image_uuid

For example, on the source machine:

[root@source ~]# vmadm get 0983fe0f-677e-4f54-b5cf-40184d041b6d | grep image_uuid "image_uuid": "5e164fac-286d-11e4-9cf7-b3f73eefcd01",

[root@source ~]# imgadm list uuid=5e164fac-286d-11e4-9cf7-b3f73eefcd01 UUID NAME VERSION OS TYPE PUB 5e164fac-286d-11e4-9cf7-b3f73eefcd01 centos-7 20140820 linux zvol 2014-08-20

On the target machine:

[root@target ~]# imgadm list uuid=5e164fac-286d-11e4-9cf7-b3f73eefcd01

[root@target ~]# imgadm import 5e164fac-286d-11e4-9cf7-b3f73eefcd01 Importing 5e164fac-286d-11e4-9cf7-b3f73eefcd01 (centos-7@20140820) from "https://images.joyent.com" Gather image 5e164fac-286d-11e4-9cf7-b3f73eefcd01 ancestry Must download and install 1 image (336.9 MiB) Download 1 image [==============================================================================================>] 100% 337.00MB 4.72MB/s 1m11s Downloaded image 5e164fac-286d-11e4-9cf7-b3f73eefcd01 (336.9 MiB) zones/5e164fac-286d-11e4-9cf7-b3f73eefcd01 [==============================================================================================>] 100% 337.00MB 25.48MB/s 13s Imported image 5e164fac-286d-11e4-9cf7-b3f73eefcd01 (centos-7@20140820)

[root@target ~]# imgadm list uuid=5e164fac-286d-11e4-9cf7-b3f73eefcd01 UUID NAME VERSION OS TYPE PUB 5e164fac-286d-11e4-9cf7-b3f73eefcd01 centos-7 20140820 linux zvol 2014-08-20

You should now be able to successfully perform the migration.

Robin Bowes – YO61

DevOps in the wilds of North Yorkshire

Overriding yum variables

Containers will not fix your broken culture

Find all variables used in a terraform module

Sorting terraform variable blocks

Removing an orphaned resource from terraform state

Generate a random password for an RDS MySQL instance

Installing prometheus node_exporter on FreeNAS

kubernetes on a 5-node Raspberry Pi 2 cluster

Copying an rsnapshot archive

Migrating a VM from one SmartOS node to another

May 2018
M	T	W	T	F	S	S
« Mar
	1	2	3	4	5	6
7	8	9	10	11	12	13
14	15	16	17	18	19	20
21	22	23	24	25	26	27
28	29	30	31