Difference between revisions of "Puppet setup"

Latest revision as of 10:41, 27 October 2021

Redirect to:

swhdocs:sysadm/puppet/reference-setup.html

@@ Line 1: / Line 1: @@
-== Multiple repository setup ==
+#REDIRECT [[swhdocs:sysadm/puppet/reference-setup.html]]
-Our puppet environment is split into multiple repos (one repo per module), plus one "root" repository for multi-repository management.
-First, clone the base repository, containing the configuration file for myrepos and a README file.
-<pre>
-$ git clone ssh://git@forge.softwareheritage.org/diffusion/SENV/puppet-environment.git
-</pre>
-Then, use that configuration to clone all the repositories :
-<pre>
-$ cd puppet-environment
-$ readlink -f .mrconfig >> ~/.mrtrust
-$ mr up
-</pre>
-(the mr command is in the myrepos Debian package).
-All the swh-specific repositories are in ''swh-''-prefixed repositories. The other repositories come from other sources and have an ''upstream'' remote allowing updates (the ''origin'' remote is always on the swh git server).
-Our puppet workflow is documented in [https://forge.softwareheritage.org/diffusion/SENV/browse/master/README.md the README.md file in the puppet-environment repository].
-== Local puppet manifest diffing with octocatalog-diff ==
-''puppet-environment'' contains the whole scaffolding to be able to use [https://github.com/github/octocatalog-diff <code>octocatalog-diff</code>] on our manifests. This allows for quick(er) local iterations while developing complex puppet manifests.
-=== Dependencies ===
-You need the following packages installed on your machine:
- r10k octocatalog-diff puppet
-=== Running ===
-The <code>bin/octocatalog-diff</code> script allows diffing the manifests between two environments (that is, between two branches of the ''swh-site'' repository. By default it diffs between <code>production</code> and <code>staging</code>.
-Default usage:
- bin/octocatalog-diff pergamon
-=== Limitations ===
-Our setup for octocatalog-diff doesn't support exported resources, so you won't see your fancy icinga checks there.
-== Integration of third party puppet modules ==
-We mirror external repositories to our own forge, to avoid having external dependencies in our deployment.
-In the ''swh-site'' <code>Puppetfile</code>, we pin the installation of those modules to the highest version (that works with our current puppet/facter version), by using the '':ref'' specifier.
-=== Adding a new external puppet module ===
-In the ''puppet-environment'' repository, the <code>bin/import-puppet-module</code> takes care of the following tasks:
-* Getting metadata from the [https://forge.puppetlabs.com/ Puppet forge] for the module (description, upstream git URL)
-* Cloning the repository
-* Creating a mirror repository on the Software Heritage forge, with the proper permissions and metadata (notably the ''Sync to GitHub'' flag)
-* Pushing the clone to the forge
-* Updating the .mrconfig and .gitignore files to know the
-To be able to use the script, you need :
-* Be a member of the [https://forge.softwareheritage.org/project/members/7/ System Administrators] Phabricator group
-* Have the [[Arcanist]] API key setup
-* A pair of python dependencies : <code>python3-phabricator</code> and <code>python3-requests</code> (pull them from testing if needed).
-Example usage to pull the [https://forge.puppetlabs.com/elastic/elasticsearch elastic/elasticsearch] module
-<pre>
-bin/import-module elastic-elasticsearch
-git diff # review changes
-git add .mrconfig .gitignore
-git commit -m "Add the elastic/elasticsearch module"
-git push
-</pre>
-Once the module is added, you need to register it in the ''swh-site'' <code>Puppetfile</code>.
-You should also check in the module metadata whether any dependencies need importing as well, which you should do using the same procedure.
-=== Updating external puppet modules ===
-There's two sides of this coin:
-==== Updating our git clone of external puppet modules ====
-The ''puppet-environment'' <code>.mrconfig</code> file has a <code>pullup</code> command which does the right thing.
-To update all clones:
-<pre>
-mr -j4 pullup
-</pre>
-==== Upgrading external puppet modules ====
-Upgrading external puppet modules happens manually.
-In the ''puppet-environment'' repository, the <code>bin/check-module-updates</code> script compares the Puppetfile and the local clones and lists the available updates. (depends on <code>ruby r10k</code>).
-On a staging branch of the ''swh-site'' repository, update the <code>:ref</code> value for the module in the <code>Puppetfile</code> to the latest tag. You can then run <code>octocatalog-diff</code> on a few relevant servers and look for changes.
-== Deploy work-flow ==
-=== Semi-automated ===
-# you@localhost$ # ''hack on puppet Git repo''
-# you@localhost$ <tt>rake validate</tt>
-# you@localhost$ <tt>git commit</tt>
-# you@localhost$ <tt>git push</tt>
-# you@localhost$ <tt>cd [https://forge.softwareheritage.org/diffusion/SENV/ puppet-environment]</tt>
-# you@localhost$ <tt>bin/deploy-on machine1 machine2...</tt>
-Remember to pass <tt>--apt</tt> to <tt>bin/deploy-on</tt> if freshly uploaded Software Heritage packages are to be deployed.
-Also, <tt>bin/deploy-on --help</tt> is your friend.
-=== Manual ===
-# you@localhost$ # ''hack on puppet Git repo''
-# you@localhost$ <tt>rake validate</tt>
-# you@localhost$ <tt>git commit</tt>
-# you@localhost$ <tt>git push</tt>
-# you@pergamon$ <tt>sudo swh-puppet-master-deploy</tt>
-# you@machine$ <tt>sudo apt-get update</tt>  # ''if a new or updated version of a [[Debian packaging|Debian package]] needs deploying''
-# you@machine$ <tt>sudo swh-puppet-test</tt>  # ''to test/review changes''
-# you@machine$ <tt>sudo swh-puppet-apply</tt>  # ''to apply
-[[Category:System administration]]

Difference between revisions of "Puppet setup"

Latest revision as of 10:41, 27 October 2021

Navigation menu

Search