Picture the scene…

There's a new release of Puppet Enterprise. You download it, run the upgrade in your test environment, run your regression tests, and all looks good. You then upgrade your production master – all looks good. All that remains to be done is to upgrade the puppet agent on all client nodes – all 750 of them.

Now, you could ssh to each node individually and run the PE installer via curl|bash. You could even automate that with pssh, or similar. But there's got to be a better way, right?


This was the position I found myself in earler this week.

I did some digging and found the puppet_agent module which, on the face of it, is written for just this situation. However, the module specifically doesn't automatically upgrade PE if the existing client is running v4.x.x but it *will* upgrade if a package version is passed to the module. Also, by default, it creates a new yum repo file pointing at the upstream Puppet repos which is not necessary on PE installs since the agent packages are already present on the master and available at https://<PUPPET_MASTER>:8140/packages/<PE_VERSION>/<OS+ARCH>/. In fact the PE install process creates a yum config pointing at this repo. This is not upgraded when the master is upgraded.

So, to summarise, I need to solve two issues:

  1. Create a yum config pointing at the new agent software on the master
  2. Pass the specific package version to the puppet_agent class.

I noticed on the puppet master under packages that, in addition to the versioned directories, there was a current link which points to the, er, "current" version of the agent. I also noticed that there was a top-level fact called platform_tag that defined the <OS+ARCH> combination. That gave me enough information to create a repo config that will always point to "current" agent software on the master.

Digging in the puppet_agent class, I found that it used a PE function pe_compiling_server_aio_build() to get the agent version available on the master. I now have all the information I need.

I wrote the following code in my profile::puppet_agent class, which is applied to all nodes:

  yumrepo { 'pe_repo':
    ensure    => present,
    baseurl   => "https://${::puppet_master_server}:8140/packages/current/${::platform_tag}",
    descr     => 'Puppet Labs PE Packages $releasever - $basearch',
    enabled   => 1,
    gpgcheck  => 1,
    gpgkey    => "https://${i::puppet_master_server}:8140/packages/GPG-KEY-puppetlabs",
    proxy     => '_none_',
    sslverify => false,
  }->
  class{ '::puppet_agent':
    manage_repo     => false,
    package_version => pe_compiling_server_aio_build(),
  }

As if by magic, all my client nodes were upgraded to the latest agent software.

In a previous post, I described how to describe puppet environments, roles, and profiles, as modules and how to use r10k and librarian-puppet to deploy them.

One possible problem with deploying to the puppet environment directory directly is that the librarian-puppet run can take some time and there is a possibility that puppet may attempt to compile a catalogue in an incomplete or inconsistent environment. One way to overcome this is to deploy the environments into a new directory, create a symlink, and move the symlink atomically into place.

This would look something like this:

cd /etc/puppet/envs
# create a new dir under /etc/puppet/envs - I use a timestamp in the name so I know when it was created
NEW_ENV_DIR=$(mktemp --directory envs.$(date -Isec).XXX")
cd /etc/puppet
# use r10k deploy the environments into the new dir
PUPPETFILE_DIR="envs/${NEW_ENV_DIR}" r10k puppetfile install
# loop over all the environments and use librarian-puppet to deploy all the roles/profiles/modules
while read env ; do
  pushd $env
  LIBRARIAN_PUPPET_PATH=modules librarian-puppet install --no-use-v1-api --strip-dot-git
  popd
done < <(find "/etc/puppet/envs/${NEW_ENV_DIR}" -maxdepth 1 -mindepth 1 -type d)
ln -s /etc/puppet/envs/${NEW_ENV_DIR} /etc/puppet/envs/environments
mv /etc/puppet/envs/environments /etc/puppet

I have written a script that does all of this in a more robust way and also uses parallel to speed up the deployment process.

The script and the role and profile modules references in my previous article are in this github repo.

By treating roles and profiles as puppet modules, we can use r10k and librarian-puppet to manage the deployment of our puppet code into our puppet environements.

I shall assume that puppet is configured to use to use directory environments and that the environment path is $confdir/environments (ie. the default location). I also assume that both r10k and librarian-puppet are installed and in the path.

You should also understand and embrace the role-profile-module pattern, first described by Craig Dunn and subsequently by Adrian Thebo and Gary Larizza. Quoting Gary:

  • Roles abstract profiles
  • Profiles abstract component modules
  • Hiera abstracts configuration data
  • Component modules abstract resources
  • Resources abstract the underlying OS implementation 

I find the following points useful to clarify the purpose of each of the layers in this model:

  • Roles, profiles, and component modules can all be implemented as puppet modules
  • Each node is assigned exactly one role (either in site.pp or, preferably, using some external node classifier)
  • Each role includes one or more profiles
  • Each profile loads configuration data and feeds it into the component modules – this is where your business logic should go
  • Each component module should be generic and contain no site-specific data. You should be able to publish all your component modules on PuppetForge without leaking any secrets.

We can further extend this model to include environments. An environment can be thought of as a group of roles and can also be implemented as a puppet module.

So, how do we set this up?

At the top-level, we put a Puppetfile in the puppet config dir containing a list of our environments. This will look something like this:

#!/usr/bin/env ruby
#^syntax detection

forge 'https://forgeapi.puppetlabs.com'

mod 'yo61-env_production',
  :git => 'git@github.com:yo61/puppet-demo_env_production.git'

mod 'yo61-env_staging',
  :git => 'git@github.com:yo61/puppet-demo_env_staging.git'

Each environment is defined as a puppet module. Any valid Puppetfile syntax may be used to specifiy the module location, including alternate branches or specific version tags. 

Each of the environment modules should contain all the usual things you would put in a puppet environment, eg. a manifests/site.pp, etc. as well as a Puppetfile containing a list of all the roles to be deployed to this environment. The Puppetfile for a simple environment would look something like this:

#!/usr/bin/env ruby
#^syntax detection

forge "https://forgeapi.puppetlabs.com"

# list all the roles that are included in this environment
mod 'yo61-role_default',
  :git => 'git@github.com:yo61/puppet-demo-roles-profiles.git',
  :path => 'modules/role_default'

mod 'yo61-role_foo',
  :git => 'git@github.com:yo61/puppet-demo-roles-profiles.git',
  :path => 'modules/role_foo'

mod 'yo61-role_bar',
  :git => 'git@github.com:yo61/puppet-demo-roles-profiles.git',
  :path => 'modules/role_bar'

Like the top-level Puppetfile used to defined environments, each role is defined as a puppet module.

Each of the role modules will contain a simple class that loads the profiles used by the role, and a Puppetfile containing a list of all profiles used by the role. The Puppetfile for a simple role would look something like this:

#!/usr/bin/env ruby
#^syntax detection

forge "https://forgeapi.puppetlabs.com"

# list all the profiles that are included in this role
mod 'yo61-profile_common',
  :git => 'git@github.com:yo61/puppet-demo-roles-profiles.git',
  :path => 'modules/profile_common'

Each of the profile modules will contain all the puppet code required to define the business logic, load configuration data, etc. and a Puppetfile containing a list of all the component modules used by the profile. The Puppetfile for a simple profile would look something like this:

#!/usr/bin/env ruby
#^syntax detection

forge "https://forgeapi.puppetlabs.com"

# include all the modules used by this profile
mod 'puppetlabs-stdlib',
mod 'stahnma-epel'
mod 'puppetlabs-ntp'

Again, any valid Puppetfile syntax may be used.

We've now defined all our environments, roles, and profiles and we're ready to deploy each environment.

First, we run r10k to deploy each of the environment modules into the environment dir (/etc/puppet/environments):

# switch to the location of the top-level Puppetfile
cd /etc/puppet
PUPPETFILE_DIR=/etc/puppet/environments r10k puppetfile install

This will create a directory in /etc/puppet/environments for each of the environments defined in the top-level Puppetfile.

Next, we change into each of the newly-created environment directories and run librarian-puppet to install all the roles required by that environment.

cd /etc/puppet/environments/production
LIBRARIAN_PUPPET_PATH=modules librarian-puppet install --no-use-v1-api --strip-dot-git

The best bit is that librarian-puppet supports recursive module dependency resolution so this one command installs not only the roles, but also all the profiles, and component modules required by each of the roles.

My next article will present a script that runs r10k and librarian-puppet as described in this article, and also updates puppet environments atomically

 

We're using puppet + puppetdb in an EC2 environment where nodes come and go quite regularly. We have a custom autosign script that uses ec2 security info to validate the nodes before allowing the autosigning. This is all good, but it can leave a lot of "dead" nodes in puppet, eg. if a bunch of nodes are created by an autoscale policy and then terminated.

To get rid of these zombie nodes from puppet/puppetdb we can just use:

puppet node deactivate <certname1> <certname2> ... <certnameN>

We can query puppetdb to get a list of nodes that have not sent puppet reports for, say, 24 hours. The puppetdb query we need is something like this:

'query=["<", "report-timestamp", "$cutoff_date"]'

where $cutoff_date is a date in ISO8601 format, eg. 2015-03-05T13:39:45+0000

We can use date to generate the cutoff date with something like this:

$cutoff_date=$(date -d '-1 day' -Isec)

We then plug this into the query string and send it with curl as follows:

curl --silent -G 'http://localhost:8080/v4/nodes' \
  --data-urlencode "query=[\"<\", \"report-timestamp\", \"$(date -d '-1 day' -Isec)\"]"

Finally, we filter through jq to get a list of certnames:

curl --silent -G 'http://localhost:8080/v4/nodes' \
  --data-urlencode "query=[\"<\", \"report-timestamp\", \"$(date -d '-1 day' -Isec)\"]" \
  | jq '.[].certname'

We can then pass the list of nodes to the "puppet node deactivate" command.

One of my very early frustrations with puppet was that it allows variables to be used when they were undefined. Primarily this bit me by not catching typos in variable names which were often very hard to track down. I was very pleased when Puppetlabs introduced a strict_variables mode which throws an error if a manifest attempts to use an undefined variable.

I recently need to check for the existence of a fact. Without strict_variables, this is straight-forward:

if $::some_fact {
  # do stuff here
}

If the fact "some_fact" exists, the variable is a non-empty string and evaluates as true in boolean context. If the fact doesn't exist, the variable is an empty string which evaluates as false in boolean context.

But, with strict_variables enforced, this throws an error:

Error: Undefined variable "::some_fact"; Undefined variable "some_fact" at line ...

The solution is to use the getvar function from stdlib:

if getvar('::some_fact') {
  # do stuff here
}

This works exactly the same as in the previous example, but doesn't throw an error if "some_fact" doesn't exist.

The team at bitly has written an http reverse proxy that provides authentication using Google's OAuth2 API. They write about it in a blog post.

The proxy is written in Go but builds to a single, statically-linked executable, ie. there are no complex run-time dependencies, which is great.

I've built an RPM for EL7 which also includes a sample systemd unit file, and sample configuration file. Both source and binary RPMs are available in my yum repo.

Additionally, I've create a puppet module that installs the RPM, creates a systemd service, and sets up an nginx front end to the proxy service. The module is available from the Puppetforge, and also on github.

I'd be interested in any feedback/comments/bug reports/pull requests.

I'm setting up a new puppet master running under passenger on CentOS 7 using packages from the puppetlabs and foreman repos. I used a fork of Stephen Johnson's puppet module to set everything up (with puppet apply). All went swimmingly, except I would see this error in the logs the first time the puppet master app loaded (ie. the first time it got a request):

[ 2014-11-07 23:22:13.2600 2603/7f1a0660e700 Pool2/Spawner.h:159 ]: [App 2643 stderr] *** Phusion Passenger: no passenger_native_support.so found for the current Ruby interpreter. Compiling one (set PASSENGER_COMPILE_NATIVE_SUPPORT_BINARY=0 to disable)...
[ 2014-11-07 23:22:13.2600 2603/7f1a0660e700 Pool2/Spawner.h:159 ]: [App 2643 stderr] # mkdir -p /usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/locations.ini/buildout/ruby/ruby-2.0.0-x86_64-linux
[ 2014-11-07 23:22:13.2600 2603/7f1a0660e700 Pool2/Spawner.h:159 ]: [App 2643 stderr] Not a valid directory. Trying a different one...
[ 2014-11-07 23:22:13.2600 2603/7f1a0660e700 Pool2/Spawner.h:159 ]: [App 2643 stderr] -------------------------------
[ 2014-11-07 23:22:13.2600 2603/7f1a0660e700 Pool2/Spawner.h:159 ]: [App 2643 stderr] # mkdir -p /var/lib/puppet/.passenger/native_support/4.0.18/ruby-2.0.0-x86_64-linux
[ 2014-11-07 23:22:13.2600 2603/7f1a0660e700 Pool2/Spawner.h:159 ]: [App 2643 stderr] # cd /var/lib/puppet/.passenger/native_support/4.0.18/ruby-2.0.0-x86_64-linux
[ 2014-11-07 23:22:13.2600 2603/7f1a0660e700 Pool2/Spawner.h:159 ]: [App 2643 stderr] # /usr/bin/ruby '/usr/share/gems/gems/passenger-4.0.18/ruby_extension_source/extconf.rb'
[ 2014-11-07 23:22:13.3048 2603/7f1a0660e700 Pool2/Spawner.h:159 ]: [App 2643 stderr] /usr/bin/ruby: No such file or directory -- /usr/share/gems/gems/passenger-4.0.18/ruby_extension_source/extconf.rb (LoadError)
[ 2014-11-07 23:22:13.3156 2603/7f1a0660e700 Pool2/Spawner.h:159 ]: [App 2643 stderr] Compilation failed.
[ 2014-11-07 23:22:13.3156 2603/7f1a0660e700 Pool2/Spawner.h:159 ]: [App 2643 stderr] -------------------------------
[ 2014-11-07 23:22:13.3157 2603/7f1a0660e700 Pool2/Spawner.h:159 ]: [App 2643 stderr] Ruby native_support extension not loaded. Continuing without native_support.

I double checked, and I do have the native libs installed – they're in the rubygem-passenger-native-libs rpm – the main library is in /usr/lib64/gems/ruby/passenger-4.0.18/native/passenger_native_support.so.

Digging in the passenger code, it tries to load the native libs by doing:

require 'native/passenger_native_support'

If I hacked this to:

require '/usr/lib64/gems/ruby/passenger-4.0.18/native/passenger_native_support'

then it loaded correctly.

It seems that /usr/lib64/gems/ruby/passenger-4.0.18 is not in the ruby load path.

Additional directories can be added to the ruby load path by setting an environment variable, RUBYLIB.

To set RUBYLIB for the apache process, I added the following line to /etc/sysconfig/httpd and restarted apache:

RUBYLIB=/usr/lib64/gems/ruby/passenger-4.0.18

The passenger native libraries now load correctly.

I was writing some basic RSpec tests for a puppet module this morning, methodically adding in fixtures and hiera data items to get the module to compile under the spec tests.

Then I hit this error:

Failures:

1) profile_puppet::master supported operating systems profile_puppet::master class without any parameters on redhat 6.4 should compile into a catalogue without dependency cycles
Failure/Error: it { should compile.with_all_deps }
NoMethodError:
undefined method `groups' for nil:NilClass
# ./spec/classes/init_spec.rb:36:in `block (5 levels) in '

Uh oh, that doesn't look good. I did what I always do in such circumstances and googled the error message: puppet NoMethodError: undefined method `groups' for nil:NilClass. The first hit was https://tickets.puppetlabs.com/browse/PUP-1547 which describes my situation completely (I am testing for RHEL 6.4 on OSX).

What's even better is that the ticket was updated 3 days ago with a pull request that fixes the issue. I applied the change locally, it worked perfectly, and I was able to complete my task.

Try doing that with proprietary software.

I'm a big fan of provisioning tools, particularly puppet.

Sometimes I just want to quickly throw a clean install on a new machine that I can then use to provision other machines (and even to re-configure the puppetmaster).

So, I wrote a script to do just that. The only requirement is a minimal install of your favourite CentOS/Red Hat/Fedora OS and the script will do the rest.

It's available from github: https://github.com/robinbowes/puppet-server-bootstrap

When I began using puppet, I quickly realised that configuration data was best kept separate from puppet manifests. Initially, I used extlookup and kept configuration data in CSV files. Then complex data structures came to puppet and I now use hirea/hiera-puppet with configuration data stored in hierarchical YAML files (other hiera backends are available). This article describes how to define in YAML the resources that should be applied to a node.

Continue reading