View source for Puppet setup

== Multiple repository setup ==

Our puppet environment is split into multiple repos (one repo per module), plus one "root" repository for multi-repository management.

First, clone the base repository, containing the configuration file for myrepos and a README file.

<pre>
$ git clone ssh://git@forge.softwareheritage.org/diffusion/SENV/puppet-environment.git
</pre>

Then, use that configuration to clone all the repositories :

<pre>
$ cd puppet-environment
$ readlink -f .mrconfig >> ~/.mrtrust
$ mr up
</pre>

(the mr command is in the myrepos Debian package).

All the swh-specific repositories are in ''swh-''-prefixed repositories. The other repositories come from other sources and have an ''upstream'' remote allowing updates (the ''origin'' remote is always on the swh git server).

Our puppet workflow is documented in [https://forge.softwareheritage.org/diffusion/SENV/browse/master/README.md the README.md file in the puppet-environment repository].

== Local puppet manifest diffing with octocatalog-diff ==

''puppet-environment'' contains the whole scaffolding to be able to use [https://github.com/github/octocatalog-diff <code>octocatalog-diff</code>] on our manifests. This allows for quick(er) local iterations while developing complex puppet manifests.

=== Dependencies ===
You need the following packages installed on your machine:
 r10k octocatalog-diff puppet

=== Running ===
The <code>bin/octocatalog-diff</code> script allows diffing the manifests between two environments (that is, between two branches of the ''swh-site'' repository. By default it diffs between <code>production</code> and <code>staging</code>.

Default usage:
 bin/octocatalog-diff pergamon

=== Limitations ===
Our setup for octocatalog-diff doesn't support exported resources, so you won't see your fancy icinga checks there.

== Integration of third party puppet modules ==

We mirror external repositories to our own forge, to avoid having external dependencies in our deployment.

In the ''swh-site'' <code>Puppetfile</code>, we pin the installation of those modules to the highest version (that works with our current puppet/facter version), by using the '':ref'' specifier.

=== Adding a new external puppet module ===

In the ''puppet-environment'' repository, the <code>bin/import-puppet-module</code> takes care of the following tasks:

* Getting metadata from the [https://forge.puppetlabs.com/ Puppet forge] for the module (description, upstream git URL)
* Cloning the repository
* Creating a mirror repository on the Software Heritage forge, with the proper permissions and metadata (notably the ''Sync to GitHub'' flag)
* Pushing the clone to the forge
* Updating the .mrconfig and .gitignore files to know the 

To be able to use the script, you need :
* Be a member of the [https://forge.softwareheritage.org/project/members/7/ System Administrators] Phabricator group
* Have the [[Arcanist]] API key setup
* A pair of python dependencies : <code>python3-phabricator</code> and <code>python3-requests</code> (pull them from testing if needed).

Example usage to pull the [https://forge.puppetlabs.com/elastic/elasticsearch elastic/elasticsearch] module
<pre>
bin/import-module elastic-elasticsearch
git diff # review changes
git add .mrconfig .gitignore
git commit -m "Add the elastic/elasticsearch module"
git push
</pre>

Once the module is added, you need to register it in the ''swh-site'' <code>Puppetfile</code>.

You should also check in the module metadata whether any dependencies need importing as well, which you should do using the same procedure.

=== Updating external puppet modules ===

There's two sides of this coin:

==== Updating our git clone of external puppet modules ====

The ''puppet-environment'' <code>.mrconfig</code> file has a <code>pullup</code> command which does the right thing.

To update all clones:
<pre>
mr -j4 pullup
</pre>

==== Upgrading external puppet modules ====

Upgrading external puppet modules happens manually.

In the ''puppet-environment'' repository, the <code>bin/check-module-updates</code> script compares the Puppetfile and the local clones and lists the available updates. (depends on <code>ruby r10k</code>).

On a staging branch of the ''swh-site'' repository, update the <code>:ref</code> value for the module in the <code>Puppetfile</code> to the latest tag. You can then run <code>octocatalog-diff</code> on a few relevant servers and look for changes.

== Deploy work-flow ==

=== Semi-automated ===

# you@localhost$ # ''hack on puppet Git repo''
# you@localhost$ <tt>rake validate</tt>
# you@localhost$ <tt>git commit</tt>
# you@localhost$ <tt>git push</tt>
# you@localhost$ <tt>cd [https://forge.softwareheritage.org/diffusion/SENV/ puppet-environment]</tt>
# you@localhost$ <tt>bin/deploy-on machine1 machine2...</tt>

Remember to pass <tt>--apt</tt> to <tt>bin/deploy-on</tt> if freshly uploaded Software Heritage packages are to be deployed.
Also, <tt>bin/deploy-on --help</tt> is your friend.

=== Manual ===

# you@localhost$ # ''hack on puppet Git repo''
# you@localhost$ <tt>rake validate</tt>
# you@localhost$ <tt>git commit</tt>
# you@localhost$ <tt>git push</tt>
# you@pergamon$ <tt>sudo swh-puppet-master-deploy</tt>
# you@machine$ <tt>sudo apt-get update</tt>  # ''if a new or updated version of a [[Debian packaging|Debian package]] needs deploying''
# you@machine$ <tt>sudo swh-puppet-test</tt>  # ''to test/review changes''
# you@machine$ <tt>sudo swh-puppet-apply</tt>  # ''to apply

[[Category:System administration]]