|
|
Line 1: |
Line 1: |
− | == Multiple repository setup ==
| + | #REDIRECT [[swhdocs:sysadm/puppet/reference-setup.html]] |
− | | |
− | Our puppet environment is split into multiple repos (one repo per module), plus one "root" repository for multi-repository management.
| |
− | | |
− | First, clone the base repository, containing the configuration file for myrepos and a README file.
| |
− | | |
− | <pre>
| |
− | $ git clone ssh://git@forge.softwareheritage.org/diffusion/SENV/puppet-environment.git
| |
− | </pre>
| |
− | | |
− | Then, use that configuration to clone all the repositories :
| |
− | | |
− | <pre>
| |
− | $ cd puppet-environment
| |
− | $ readlink -f .mrconfig >> ~/.mrtrust
| |
− | $ mr up
| |
− | </pre>
| |
− | | |
− | (the mr command is in the myrepos Debian package).
| |
− | | |
− | All the swh-specific repositories are in ''swh-''-prefixed repositories. The other repositories come from other sources and have an ''upstream'' remote allowing updates (the ''origin'' remote is always on the swh git server).
| |
− | | |
− | Our puppet workflow is documented in [https://forge.softwareheritage.org/diffusion/SENV/browse/master/README.md the README.md file in the puppet-environment repository].
| |
− | | |
− | == Local puppet manifest diffing with octocatalog-diff ==
| |
− | | |
− | ''puppet-environment'' contains the whole scaffolding to be able to use [https://github.com/github/octocatalog-diff <code>octocatalog-diff</code>] on our manifests. This allows for quick(er) local iterations while developing complex puppet manifests.
| |
− | | |
− | === Dependencies ===
| |
− | You need the following packages installed on your machine:
| |
− | r10k octocatalog-diff puppet
| |
− | | |
− | === Running ===
| |
− | The <code>bin/octocatalog-diff</code> script allows diffing the manifests between two environments (that is, between two branches of the ''swh-site'' repository. By default it diffs between <code>production</code> and <code>staging</code>.
| |
− | | |
− | Default usage:
| |
− | bin/octocatalog-diff pergamon
| |
− | | |
− | === Limitations ===
| |
− | Our setup for octocatalog-diff doesn't support exported resources, so you won't see your fancy icinga checks there.
| |
− | | |
− | == Integration of third party puppet modules ==
| |
− | | |
− | We mirror external repositories to our own forge, to avoid having external dependencies in our deployment.
| |
− | | |
− | In the ''swh-site'' <code>Puppetfile</code>, we pin the installation of those modules to the highest version (that works with our current puppet/facter version), by using the '':ref'' specifier.
| |
− | | |
− | === Adding a new external puppet module ===
| |
− | | |
− | In the ''puppet-environment'' repository, the <code>bin/import-puppet-module</code> takes care of the following tasks:
| |
− | | |
− | * Getting metadata from the [https://forge.puppetlabs.com/ Puppet forge] for the module (description, upstream git URL)
| |
− | * Cloning the repository
| |
− | * Creating a mirror repository on the Software Heritage forge, with the proper permissions and metadata (notably the ''Sync to GitHub'' flag)
| |
− | * Pushing the clone to the forge
| |
− | * Updating the .mrconfig and .gitignore files to know the
| |
− | | |
− | To be able to use the script, you need :
| |
− | * Be a member of the [https://forge.softwareheritage.org/project/members/7/ System Administrators] Phabricator group
| |
− | * Have the [[Arcanist]] API key setup
| |
− | * A pair of python dependencies : <code>python3-phabricator</code> and <code>python3-requests</code> (pull them from testing if needed).
| |
− | | |
− | Example usage to pull the [https://forge.puppetlabs.com/elastic/elasticsearch elastic/elasticsearch] module
| |
− | <pre>
| |
− | bin/import-module elastic-elasticsearch
| |
− | git diff # review changes
| |
− | git add .mrconfig .gitignore
| |
− | git commit -m "Add the elastic/elasticsearch module"
| |
− | git push
| |
− | </pre>
| |
− | | |
− | Once the module is added, you need to register it in the ''swh-site'' <code>Puppetfile</code>.
| |
− | | |
− | You should also check in the module metadata whether any dependencies need importing as well, which you should do using the same procedure.
| |
− | | |
− | === Updating external puppet modules ===
| |
− | | |
− | There's two sides of this coin:
| |
− | | |
− | ==== Updating our git clone of external puppet modules ====
| |
− | | |
− | The ''puppet-environment'' <code>.mrconfig</code> file has a <code>pullup</code> command which does the right thing.
| |
− | | |
− | To update all clones:
| |
− | <pre>
| |
− | mr -j4 pullup
| |
− | </pre>
| |
− | | |
− | ==== Upgrading external puppet modules ====
| |
− | | |
− | Upgrading external puppet modules happens manually.
| |
− | | |
− | In the ''puppet-environment'' repository, the <code>bin/check-module-updates</code> script compares the Puppetfile and the local clones and lists the available updates. (depends on <code>ruby r10k</code>).
| |
− | | |
− | On a staging branch of the ''swh-site'' repository, update the <code>:ref</code> value for the module in the <code>Puppetfile</code> to the latest tag. You can then run <code>octocatalog-diff</code> on a few relevant servers and look for changes.
| |
− | | |
− | == Deploy work-flow ==
| |
− | | |
− | === Semi-automated ===
| |
− | | |
− | # you@localhost$ # ''hack on puppet Git repo''
| |
− | # you@localhost$ <tt>rake validate</tt>
| |
− | # you@localhost$ <tt>git commit</tt>
| |
− | # you@localhost$ <tt>git push</tt>
| |
− | # you@localhost$ <tt>cd [https://forge.softwareheritage.org/diffusion/SENV/ puppet-environment]</tt>
| |
− | # you@localhost$ <tt>bin/deploy-on machine1 machine2...</tt>
| |
− | | |
− | Remember to pass <tt>--apt</tt> to <tt>bin/deploy-on</tt> if freshly uploaded Software Heritage packages are to be deployed.
| |
− | Also, <tt>bin/deploy-on --help</tt> is your friend.
| |
− | | |
− | === Manual ===
| |
− | | |
− | # you@localhost$ # ''hack on puppet Git repo''
| |
− | # you@localhost$ <tt>rake validate</tt>
| |
− | # you@localhost$ <tt>git commit</tt>
| |
− | # you@localhost$ <tt>git push</tt>
| |
− | # you@pergamon$ <tt>sudo swh-puppet-master-deploy</tt>
| |
− | # you@machine$ <tt>sudo apt-get update</tt> # ''if a new or updated version of a [[Debian packaging|Debian package]] needs deploying''
| |
− | # you@machine$ <tt>sudo swh-puppet-test</tt> # ''to test/review changes''
| |
− | # you@machine$ <tt>sudo swh-puppet-apply</tt> # ''to apply
| |
− | | |
− | [[Category:System administration]]
| |