We're using puppet + puppetdb in an EC2 environment where nodes come and go quite regularly. We have a custom autosign script that uses ec2 security info to validate the nodes before allowing the autosigning. This is all good, but it can leave a lot of "dead" nodes in puppet, eg. if a bunch of nodes are created by an autoscale policy and then terminated.

To get rid of these zombie nodes from puppet/puppetdb we can just use:

puppet node deactivate <certname1> <certname2> ... <certnameN>

We can query puppetdb to get a list of nodes that have not sent puppet reports for, say, 24 hours. The puppetdb query we need is something like this:

'query=["<", "report-timestamp", "$cutoff_date"]'

where $cutoff_date is a date in ISO8601 format, eg. 2015-03-05T13:39:45+0000

We can use date to generate the cutoff date with something like this:

$cutoff_date=$(date -d '-1 day' -Isec)

We then plug this into the query string and send it with curl as follows:

curl --silent -G 'http://localhost:8080/v4/nodes' \
  --data-urlencode "query=[\"<\", \"report-timestamp\", \"$(date -d '-1 day' -Isec)\"]"

Finally, we filter through jq to get a list of certnames:

curl --silent -G 'http://localhost:8080/v4/nodes' \
  --data-urlencode "query=[\"<\", \"report-timestamp\", \"$(date -d '-1 day' -Isec)\"]" \
  | jq '.[].certname'

We can then pass the list of nodes to the "puppet node deactivate" command.

6 thoughts on “Finding “old” nodes in puppetdb

  1. How about if you wanted all the nodes where report-timestamp was null? I've been bangin' my head on that with no luck 🙁

  2. why do you have report-timestamp versus report_timestamp?

    this actually worked for me, with the dash not underscore, on puppetdb 2.3 but when I updated to puppetdb 3.2 it no longer works, returns nothing, even though I know for a fact it should return 3 nodes to be deleted.

    I totally stumped, any ideas???

    for node in `curl –silent -G 'http://localhost:8080/v4/nodes&#039; –data-urlencode "query=[\"<\", \"report_timestamp\", \"$(date -d '-3 day' -Isec)\"]" | jq '.[].certname'| tr -d '"'`
    do ……..

  3. @Robin – storing reports in puppetdb is optional so nodes can exist without a report. In puppet.conf [master] 'reports = puppetdb' turns this on. You can store reports on both filesystem and puppetdb, 'reports = store,puppetdb'

    @Jim,

    puppetdb 2.x to 3.x changed from – to _. I haven't messed with 3.2 yet but puppetexplorer forge module leads me to believe this should be working. 

    For null values, this works in 2.3:

    curl –silent -G 'http://localhost:8080/v4/nodes&#039; –data-urlode "query=[\"null?\", \"deactivated\", true]"

    I don't have any null report timestamps. You could also query the db directly:

    puppetdb# select * from reports where 'report-timestamp' is null;
    puppetdb=# select * from reports where 'report-timestamp' is not null;

Leave a reply

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> 

required