Puppet setup - Revision history

Ardumont: Install redirection to the puppet page in main docs site

2021-10-27T10:41:01Z

Install redirection to the puppet page in main docs site

← Older revision		Revision as of 10:41, 27 October 2021
Line 1:		Line 1:
	~~== Multiple repository setup ==~~		#REDIRECT [[swhdocs:sysadm/puppet/reference-setup.html]]

	~~Our puppet environment is split into multiple repos (one repo per module), plus one "root" repository for multi-repository management.~~

	~~First, clone the base repository, containing the configuration file for myrepos and a README file.~~

	~~<pre>~~
	~~$ git clone ssh://git@forge.softwareheritage.org/diffusion/SENV/puppet-environment.git~~
	~~</pre>~~

	~~Then, use that configuration to clone all the repositories :~~

	~~<pre>~~
	~~$ cd puppet-environment~~
	~~$ readlink -f .mrconfig >> ~/.mrtrust~~
	~~$ mr up~~
	~~</pre>~~

	~~(the mr command is in the myrepos Debian package).~~

	All the swh-specific repositories are in ''swh-''-prefixed repositories. The other repositories come from other sources and have an ''upstream'' remote allowing updates (the ''origin'' remote is always on the swh git server).

	~~Our puppet workflow is documented in~~ [~~https://forge.softwareheritage.org/diffusion/SENV/browse/master/README.md the README.md file in the puppet-environment repository].~~

	~~== Local puppet manifest diffing with octocatalog-diff ==~~

	~~''puppet-environment'' contains the whole scaffolding to be able to use~~ [~~https://github.com/github/octocatalog-diff <code>octocatalog-diff</code>] on our manifests. This allows for quick(er) local iterations while developing complex puppet manifests.~~

	~~=== Dependencies ===~~
	~~You need the following packages installed on your machine:~~
	~~r10k octocatalog-diff puppet~~

	~~=== Running ===~~
	The <code>bin/octocatalog-diff</code> script allows diffing the manifests between two environments (that is, between two branches of the ''swh-site'' repository. By default it diffs between <code>production</code> and <code>staging</code>.

	~~Default usage~~:
	~~bin/octocatalog-diff pergamon~~

	~~=== Limitations ===~~
	~~Our setup for octocatalog-diff doesn't support exported resources, so you won't see your fancy icinga checks there.~~

	~~== Integration of third party puppet modules ==~~

	~~We mirror external repositories to our own forge, to avoid having external dependencies in our deployment.~~

	~~In the ''swh-site'' <code>Puppetfile</code>, we pin the installation of those modules to the highest version (that works with our current puppet~~/~~facter version), by using the '':ref'' specifier.~~

	~~=== Adding a new external puppet module ===~~

	~~In the ''~~puppet~~-environment'' repository, the <code>bin~~/~~import-puppet~~-~~module</code> takes care of the following tasks:~~

	* Getting metadata from the [https://forge.puppetlabs.com/ Puppet forge] for the module (description, upstream git URL)
	* Cloning the repository
	* Creating a mirror repository on the Software Heritage forge, with the proper permissions and metadata (notably the ''Sync to GitHub'' flag)
	* Pushing the clone to the forge
	* Updating the .mrconfig and .gitignore files to know the

	~~To be able to use the script, you need :~~
	* Be a member of the [https://forge.softwareheritage.org/project/members/7/ System Administrators] Phabricator group
	* Have the [[Arcanist]] API key setup
	* A pair of python dependencies : <code>python3-phabricator</code> and <code>python3-requests</code> (pull them from testing if needed).

	~~Example usage to pull the [https://forge.puppetlabs~~.~~com/elastic/elasticsearch elastic/elasticsearch] module~~
	~~<pre>~~
	~~bin/import-module elastic-elasticsearch~~
	~~git diff # review changes~~
	~~git add .mrconfig .gitignore~~
	~~git commit -m "Add the elastic/elasticsearch module"~~
	~~git push~~
	~~</pre>~~

	~~Once the module is added, you need to register it in the ''swh-site'' <code>Puppetfile</code>.~~

	~~You should also check in the module metadata whether any dependencies need importing as well, which you should do using the same procedure.~~

	~~=== Updating external puppet modules ===~~

	~~There's two sides of this coin:~~

	~~==== Updating our git clone of external puppet modules ====~~

	~~The ''puppet-environment'' <code>.mrconfig</code> file has a <code>pullup</code> command which does the right thing.~~

	~~To update all clones:~~
	~~<pre>~~
	~~mr -j4 pullup~~
	~~</pre>~~

	~~==== Upgrading external puppet modules ====~~

	~~Upgrading external puppet modules happens manually.~~

	In the ''puppet-environment'' repository, the <code>bin/check-module-updates</code> script compares the Puppetfile and the local clones and lists the available updates. (depends on <code>ruby r10k</code>).

	On a staging branch of the ''swh-site'' repository, update the <code>:ref</code> value for the module in the <code>Puppetfile</code> to the latest tag. You can then run <code>octocatalog-diff</code> on a few relevant servers and look for changes.

	~~== Deploy work-flow ==~~

	~~=== Semi-automated ===~~

	~~# you@localhost$ # ''hack on puppet Git repo''~~
	~~# you@localhost$ <tt>rake validate</tt>~~
	~~# you@localhost$ <tt>git commit</tt>~~
	~~# you@localhost$ <tt>git push</tt>~~
	~~# you@localhost$ <tt>cd [https://forge.softwareheritage.org/diffusion/SENV/ puppet-environment]</tt>~~
	~~# you@localhost$ <tt>bin/deploy-on machine1 machine2...</tt>~~

	~~Remember to pass <tt>--apt</tt> to <tt>bin/deploy-on</tt> if freshly uploaded Software Heritage packages are to be deployed.~~
	~~Also, <tt>bin/deploy-on --help</tt> is your friend.~~

	~~=== Manual ===~~

	~~# you@localhost$ # ''hack on puppet Git repo''~~
	~~# you@localhost$ <tt>rake validate</tt>~~
	~~# you@localhost$ <tt>git commit</tt>~~
	~~# you@localhost$ <tt>git push</tt>~~
	~~# you@pergamon$ <tt>sudo swh-puppet-master-deploy</tt>~~
	~~# you@machine$ <tt>sudo apt-get update</tt> # ''if a new or updated version of a [[Debian packaging\|Debian package]] needs deploying''~~
	~~# you@machine$ <tt>sudo swh-puppet-test</tt> # ''to test/review changes''~~
	~~# you@machine$ <tt>sudo swh-puppet-apply</tt> # ''to apply~~

	~~[[Category:System administration~~]]

NicolasDandrimont: Expand puppet documentation

2018-06-15T13:57:54Z

Expand puppet documentation

@@ Line 22: / Line 22: @@
 Our puppet workflow is documented in [https://forge.softwareheritage.org/diffusion/SENV/browse/master/README.md the README.md file in the puppet-environment repository].
 == Deploy work-flow ==
@@ Line 47: / Line 119: @@
 # you@machine$ <tt>sudo swh-puppet-test</tt>  # ''to test/review changes''
 # you@machine$ <tt>sudo swh-puppet-apply</tt>  # ''to apply
 [[Category:System administration]]

NicolasDandrimont: /* Multiple repository setup */ link to readme.md

2017-10-03T14:16:02Z

Multiple repository setup: link to readme.md

← Older revision		Revision as of 14:16, 3 October 2017
Line 21:		Line 21:
	All the swh-specific repositories are in ''swh-''-prefixed repositories. The other repositories come from other sources and have an ''upstream'' remote allowing updates (the ''origin'' remote is always on the swh git server).		All the swh-specific repositories are in ''swh-''-prefixed repositories. The other repositories come from other sources and have an ''upstream'' remote allowing updates (the ''origin'' remote is always on the swh git server).

	Our puppet workflow is documented in the README.md file in the puppet-environment repository.		Our puppet workflow is documented in [https://forge.softwareheritage.org/diffusion/SENV/browse/master/README.md the README.md file in the puppet-environment repository].

	== Deploy work-flow ==		== Deploy work-flow ==

StefanoZacchiroli: 1 revision: import public pages from the intranet wiki

2016-07-20T13:02:35Z

1 revision: import public pages from the intranet wiki

← Older revision	Revision as of 13:02, 20 July 2016
(No difference)

StefanoZacchiroli: /* Semi-automated */

2016-05-04T22:36:50Z

Semi-automated

New page

== Multiple repository setup ==

Our puppet environment is split into multiple repos (one repo per module), plus one "root" repository for multi-repository management.

First, clone the base repository, containing the configuration file for myrepos and a README file.

<pre>
$ git clone ssh://git@forge.softwareheritage.org/diffusion/SENV/puppet-environment.git
</pre>

Then, use that configuration to clone all the repositories :

<pre>
$ cd puppet-environment
$ readlink -f .mrconfig >> ~/.mrtrust
$ mr up
</pre>

(the mr command is in the myrepos Debian package).

All the swh-specific repositories are in ''swh-''-prefixed repositories. The other repositories come from other sources and have an ''upstream'' remote allowing updates (the ''origin'' remote is always on the swh git server).

Our puppet workflow is documented in the README.md file in the puppet-environment repository.

== Deploy work-flow ==

=== Semi-automated ===

# you@localhost$ # ''hack on puppet Git repo''
# you@localhost$ <tt>rake validate</tt>
# you@localhost$ <tt>git commit</tt>
# you@localhost$ <tt>git push</tt>
# you@localhost$ <tt>cd [https://forge.softwareheritage.org/diffusion/SENV/ puppet-environment]</tt>
# you@localhost$ <tt>bin/deploy-on machine1 machine2...</tt>

Remember to pass <tt>--apt</tt> to <tt>bin/deploy-on</tt> if freshly uploaded Software Heritage packages are to be deployed.
Also, <tt>bin/deploy-on --help</tt> is your friend.

=== Manual ===

# you@localhost$ # ''hack on puppet Git repo''
# you@localhost$ <tt>rake validate</tt>
# you@localhost$ <tt>git commit</tt>
# you@localhost$ <tt>git push</tt>
# you@pergamon$ <tt>sudo swh-puppet-master-deploy</tt>
# you@machine$ <tt>sudo apt-get update</tt> # ''if a new or updated version of a [[Debian packaging|Debian package]] needs deploying''
# you@machine$ <tt>sudo swh-puppet-test</tt> # ''to test/review changes''
# you@machine$ <tt>sudo swh-puppet-apply</tt> # ''to apply

[[Category:System administration]]