https://wiki.softwareheritage.org/api.php?action=feedcontributions&user=Douardda&feedformat=atomSoftware Heritage Wiki - User contributions [en]2024-03-28T18:24:39ZUser contributionsMediaWiki 1.31.16https://wiki.softwareheritage.org/index.php?title=Software_Heritage_identifiers&diff=1436Software Heritage identifiers2021-02-01T15:10:40Z<p>Douardda: add jupyterhub/binderhub</p>
<hr />
<div>Software Heritage Identifiers, or SWHIDs for short, are persistent intrinsic identifiers used to reference objects stored in the Software Heritage Archive.<br />
<br />
The syntax and semantics of SWHIDs can be found in [https://docs.softwareheritage.org/devel/swh-model/persistent-identifiers.html SWHID specification].<br />
<br />
== Adoption ==<br />
<br />
SWHIDs are used in a number of different technologies and standards, such as:<br />
<br />
* the Software Heritage archive, via its [https://archive.softwareheritage.org/ Web user interface and API]<br />
* [https://docs.softwareheritage.org/devel/swh-model/cli.html <kbd>swh identify</kbd>] command line tool (by Software Heritage), available from PyPI package [https://pypi.org/project/swh.model/ swh.model]<br />
* [https://identifiers.org/ identifiers.org] resolved ([https://registry.identifiers.org/registry/swh prefix "swh"]<br />
* [https://n2t.net/ Name-to-Things] (N2T) resolver<br />
* [https://spdx.org/ Software Package Data Exchange (SPDX)] standard, starting from [https://www.linuxfoundation.org/en/blog/spdx-2-2-specification-released/ version 2.2]<br />
* [https://github.com/jupyterhub/binderhub Jupyter binderhub] will soon allow to (re)create a Jupyter notebook computing environment from a SWHID via [https://github.com/jupyterhub/repo2docker/pull/988] and [https://github.com/jupyterhub/binderhub/pull/1256]<br />
<br />
== References ==<br />
<br />
* [https://docs.softwareheritage.org/devel/swh-model/persistent-identifiers.html SWHID specification]</div>Douarddahttps://wiki.softwareheritage.org/index.php?title=Matrix&diff=1340Matrix2020-09-29T08:58:11Z<p>Douardda: fix urls</p>
<hr />
<div>== IRC channels ==<br />
<br />
The following channels have been registered on the [https://freenode.net/ Freenode] network for [[Software Heritage]] usage.<br />
<br />
* [https://app.element.io/#/room/#freenode_#swh-devel:matrix.org '''#swh-devel''']: public development discussions<br />
* [https://app.element.io/#/room/#freenode_#swh-team:matrix.org '''#swh-team''']: private discussions of the core team<br />
* [https://app.element.io/#/room/#freenode_#swh-sysadm:matrix.org '''#swh-sysadm''']: operations team discussions/bots<br />
* [https://app.element.io/#/room/#freenode_#softwareheritage:matrix.org '''#softwareheritage''']: general discussions about the project (currently unused)<br />
* [https://app.element.io/#/room/#freenode_#swh:matrix.org '''#swh''']: ditto, in case we end up preferring the short version<br />
<br />
If you use IRC, consider joining the channels.<br />
<br />
If you don't use IRC ''directly'', you can still join our chat channels from your web browser via a [https://matrix.org/ Matrix] bridge by clicking on the channel names in the list above. You will be asked to create a [https://element.io/ Element] account if you don't have one yet.<br />
<br />
== IRC authentication ==<br />
<br />
You should register their nick with NickServ using:<br />
<br />
/nick <USERNAME><br />
/msg nickserv register <PASSWORD> <EMAIL><br />
<br />
You will then receive an e-mail containing a link to activate you account. After doing so, you need to configure your client to auto-authenticate. The recommanded way of doing that is using [https://freenode.net/kb/answer/sasl SASL authentication].<br />
<br />
For Weechat:<br />
<br />
/set irc.server.freenode.sasl_username <USERNAME><br />
/set irc.server.freenode.sasl_password <PASSWORD><br />
<br />
Freenode also supports authentication via [https://freenode.net/kb/answer/certfp TLS client certificates].<br />
<br />
== IRC access list ==<br />
<br />
To auto-voice people with a registered nick (only doable by people with +fA access modes will be able to do it), add them to the channel access list:<br />
<br />
/msg chanserv access #swh-devel add zack +V<br />
<br />
[[Category:Infrastructure]]</div>Douarddahttps://wiki.softwareheritage.org/index.php?title=Debian_packaging&diff=1339Debian packaging2020-09-25T16:45:35Z<p>Douardda: /* Local package building */</p>
<hr />
<div>== Package repository ==<br />
<br />
A package repository is available on https://debian.softwareheritage.org/.<br />
<br />
Unstable / Testing :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ unstable main<br />
<br />
Stable / Buster :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main<br />
<br />
Oldstable / Stretch :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main<br />
<br />
This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).<br />
<br />
=== Uploading packages ===<br />
<br />
Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.<br />
<br />
For packages to be accepted, they need to be :<br />
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt><br />
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)<br />
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders<br />
<br />
== Git repositories for Debian packages ==<br />
<br />
Our git repository structure for Debian packages is compatible with <tt>git-buildpackage</tt>.<br />
<br />
We have two different ways of handling repositories for Debian packages:<br />
* Packages of python modules where *we* are upstream<br />
* Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)<br />
<br />
For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:<br />
* Our own modules are merged with the upstream repository<br />
* External dependencies ignore the upstream repository and only have packaging branches.<br />
<br />
=== Branch and tags structure ===<br />
<br />
Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:<br />
* debian/upstream (history of unpacked upstream releases)<br />
* debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, stretch-swh)<br />
* pristine-tar (data to regenerate upstream tarballs from a git export)<br />
<br />
The name of the debian/upstream branch doesn't matter ''as long as it's properly configured in the <tt>debian/gbp.conf</tt> file''. It's only really used by <tt>gbp import-orig</tt> when importing a new release.<br />
<br />
The tags marking upstream releases imported from tarballs for Debian packaging purposes are named <tt>debian/upstream/''<upstream version number>''</tt>.<br />
<br />
Our Jenkins jobs are triggered on incoming tags named <tt>debian/''<version>''</tt>. To generate the proper tags, use <tt>gbp buildpackage --git-tag-only</tt>.<br />
<br />
The git-buildpackage configuration, <tt>debian/gbp.conf</tt>, should be the following:<br />
[DEFAULT]<br />
upstream-branch=debian/upstream<br />
upstream-tag=debian/upstream/%(version)s<br />
debian-branch=debian/''<current suite>''<br />
pristine-tar=True<br />
<br />
==== Automatic packaging for swh python modules ====<br />
<br />
The <tt>swh.*</tt> python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs <tt>gbp import-orig</tt> with the tarball we release to PyPI, and the right options to merge the upstream history.<br />
<br />
To merge changes from the upstream history, we add the following option to <tt>gbp.conf</tt>:<br />
upstream-vcs-tag=v%(version)s<br />
<br />
=== Bootstrapping a dependency packaging repository ===<br />
<br />
Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:<br />
<br />
Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/<br />
wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz<br />
<br />
Create a new git repository<br />
git init pytest-postgresql<br />
cd pytest-postgresql<br />
<br />
Import the original upstream version<br />
git checkout -b debian/unstable-swh<br />
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz<br />
# What will be the source package name? [pytest-postgresql] <br />
# What is the upstream version? [1.3.4] <br />
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...<br />
# gbp:info: Source package is pytest-postgresql<br />
# gbp:info: Upstream version is 1.3.4<br />
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz<br />
<br />
Bootstrap the debian directory<br />
mkdir -p debian/source<br />
echo '3.0 (quilt)' > debian/source/format<br />
echo 9 > debian/compat<br />
cat > debian/gbp.conf << EOF<br />
[DEFAULT]<br />
upstream-branch=debian/upstream<br />
upstream-tag=debian/upstream/%(version)s<br />
debian-branch=debian/unstable-swh<br />
pristine-tar=True<br />
EOF<br />
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules<br />
vim debian/control<br />
# [...] adapt debian/control from another package<br />
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1 --distribution unstable-swh<br />
vim debian/copyright<br />
# [...] adapt debian/copyright from another package<br />
git add debian<br />
git commit -m "Initial packaging for pytest-postgresql"<br />
<br />
You can then go on to try building the package. Once the package builds, if you want to check your package's conformance to Debian policy, you can run <tt>lintian</tt> on the changes:<br />
lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes<br />
<br />
Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository<br />
<br />
We need to use a <tt>+swh1</tt> version suffix to avoid clashing with potential upstream Debian package versions.<br />
<br />
==== Bootstrapping the backport branches ====<br />
<br />
During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.<br />
<br />
The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.<br />
<br />
Checkout the new branch<br />
git checkout debian/<version number><br />
git checkout -b debian/stretch-swh<br />
<br />
Update the gbp config to match the branch<br />
sed -i s/unstable-swh/stretch-swh/ debian/gbp.conf<br />
<br />
Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)<br />
dch -l "~bpo9" -D stretch-swh --force-distribution 'Rebuild for stretch-swh'<br />
<br />
You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.<br />
<br />
==== Setting up the repository on Phabricator ====<br />
<br />
The repository on Phabricator needs the following settings:<br />
* Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)<br />
* Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)<br />
* Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)<br />
* Policy<br />
** View: Public (no login required)<br />
** Edit: Developers<br />
** Push: All users (actual restrictions are handled by Herald rules)<br />
* Activate the repository<br />
* Look up the path to the repository on the storage tab<br />
<br />
You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes.<br />
ssh -p 2222 -t tate.internal.softwareheritage.org phabricator-setup-hook /srv/phabricator/repos/<repo-id> post-receive-debian-deps<br />
<br />
Note: remember that access to tate is on port 2222.<br />
<br />
The repo ID can be found on the repo's "storage" property page on phabricator, typically <br />
https://forge.softwareheritage.org/source/swh-SHORTNAME/manage/storage/<br />
<br />
==== Setting up the Jenkins jobs ====<br />
<br />
The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/<br />
They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs<br />
<br />
Jobs for dependency packages are configured in <tt>jobs/dependency-packages.yaml</tt>. You can add a section as follows:<br />
<br />
- project:<br />
name: <Callsign><br />
display-name: <short-name><br />
pkg: <source-name><br />
jobs:<br />
- 'dependency-jobs-{name}'<br />
<br />
Use the regular review process to land your changes.<br />
Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.<br />
<br />
If your package needs extra repositories to build, you can add them as comma-separated values to the <tt>deb-extra-repositories</tt> setting, with the following notes:<br />
* When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.<br />
* When building packages for backports suites, the backports repository is automatically enabled.<br />
<br />
=== Updating a dependency packaging repository ===<br />
<br />
Place yourself on the debian/unstable-swh branch and "gbp import-origin" a more<br />
recent upstream release tarballs.<br />
<br />
For example (current version on 0.0.5, upstream bumped to 0.0.7):<br />
gbp import-origin https://files.pythonhosted.org/packages/7a/bb/cf8fec6009e7d0cec52dc179d09b28c4c70d158e79b565e8aab7606e1717/attrs-strict-0.0.7.tar.gz<br />
<br />
This will update the following branches:<br />
* debian/upstream<br />
* pristine-tar<br />
* debian/unstable-swh<br />
<br />
This also includes the necessary tags (`debian/upstream/0.0.7` here).<br />
<br />
You then need to push all branches/tags to the repository:<br />
git push origin --all --follow-tags<br />
<br />
Ensure the [https://wiki.softwareheritage.org/wiki/Debian_packaging#Local_package_building update builds fine] <br />
And [https://wiki.softwareheritage.org/wiki/Debian_packaging#Remote_package_building tags accordingly the debian/unstable-swh branch when ok]. <br />
Jenkins will then keep up on building the package.<br />
<br />
=== Local package building ===<br />
<br />
To locally test a package build, go on the appropriate debian packaging branch, and run<br />
gbp buildpackage --git-builder=sbuild -As --no-clean-source<br />
<br />
<tt>gbp buildpackage</tt> passes all options not starting with <tt>--git-</tt> to the builder. Some useful options are the following:<br />
<br />
* <tt>--git-ignore-new</tt> builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.<br />
* <tt>--no-clean-source</tt> doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies<br />
* <tt>--extra-repository="'''repository specification'''"</tt> adds the given repository in the chroot before building.<br />
* <tt>--extra-repository-key='''repository signing key'''</tt> adds the given key as a trusted gpg key for package sources<br />
* <tt>--extra-package='''<.deb file or directory>'''</tt> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.<br />
* <tt>--force-orig-source</tt> forces addition of the <tt>.orig.tar.gz</tt> file in the <tt>.changes</tt> file (useful when trying to upload a backport)<br />
<br />
See <tt>gbp help buildpackage</tt> and <tt>man sbuild</tt> for a full description of all options<br />
<br />
for example:<br />
gbp buildpackage --git-builder=sbuild -As --no-clean-source --force-orig-source \<br />
--extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main'<br />
<br />
or if you need some third-party repository, say for cassandra:<br />
gbp buildpackage --git-builder=sbuild -As --no-clean-source --force-orig-source \<br />
--extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main' \<br />
--extra-repository='deb [arch=amd64 trusted=yes] https://downloads.apache.org/cassandra/debian 40x main'<br />
<br />
<br />
(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping <tt>gbp buildpackage</tt> with the most common options)<br />
<br />
=== Remote package building ===<br />
<br />
Jenkins builds packages when the repository receives a tag.<br />
<br />
Once the local build succeeds, tag the package with:<br />
gbp buildpackage --git-tag-only --git-sign-tags<br />
<br />
Alternatively, you can add the <tt>--git-tag</tt> option to your <tt>gbp buildpackage</tt> command so the tag happens automatically on a successful build.<br />
<br />
Then, push your tag, and Jenkins jobs should get triggered<br />
git push --tags<br />
<br />
== Build Environment setup ==<br />
<br />
Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.<br />
<br />
=== sbuild setup ===<br />
<br />
# Install the package<br />
sudo apt-get install sbuild<br />
<br />
# Add your user to the sbuild group, to allow him to use the sbuild commands<br />
sudo sbuild-adduser $USER<br />
# You have to logout and log back in<br />
<br />
# Prepare chroots<br />
sudo mkdir /srv/chroots<br />
sudo mkdir /srv/chroots/var<br />
<br />
# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition<br />
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/<br />
sudo rm -r /var/lib/schroot<br />
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot<br />
<br />
sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/<br />
sudo rm -r /var/lib/sbuild<br />
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild<br />
# end optionally<br />
<br />
# Create unstable/sid chroot<br />
sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/<br />
<br />
# Create stretch chroot<br />
sudo sbuild-createchroot --include apt-transport-https,ca-certificates stretch /srv/chroots/stretch http://deb.debian.org/debian/<br />
<br />
<br />
# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts<br />
echo hosts >> /etc/schroot/sbuild/nssdatabases<br />
<br />
=== schroot setup ===<br />
<br />
Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.<br />
<br />
You need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:<br />
<br />
source-groups=root,sbuild<br />
source-root-groups=root,sbuild<br />
union-type=overlay<br />
<br />
This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.<br />
<br />
You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):<br />
<br />
For unstable:<br />
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild<br />
<br />
For stretch:<br />
aliases=stretch-swh-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild<br />
<br />
==== dependencies cache ====<br />
<br />
Add the following line to schroot's fstab /etc/schroot/sbuild/fstab<br />
to permit reuse of existing fetched dependencies:<br />
<br />
/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0<br />
<br />
You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in <tt>/etc/apt/apt.conf.d</tt> on each chroot<br />
<br />
=== schroot update ===<br />
<br />
You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):<br />
<br />
sudo sbuild-update -udcar sid; sudo sbuild-update -udcar stretch; sudo sbuild-update -ud jessie <br />
<br />
=== environment setup ===<br />
<br />
The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt><br />
<br />
export DEBFULLNAME="Debra Hacker"<br />
export DEBEMAIL=debra.hacker@example.com<br />
<br />
Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourfullkeyid></tt> variable.<br />
(Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).<br />
<br />
=== overlay in tmpfs for faster builds ===<br />
<br />
You can add this to your fstab to put the overlay hierarchy in RAM:<br />
<br />
tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0 0 0<br />
<br />
=== Base packages ===<br />
<br />
In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.<br />
<br />
'''If you do so, do not use these chroots to upload to Debian itself!'''<br />
<br />
<br />
[[Category:Software development]]<br />
[[Category:System administration]]</div>Douarddahttps://wiki.softwareheritage.org/index.php?title=Debian_packaging&diff=1337Debian packaging2020-09-23T13:23:19Z<p>Douardda: /* Setting up the repository on Phabricator */</p>
<hr />
<div>== Package repository ==<br />
<br />
A package repository is available on https://debian.softwareheritage.org/.<br />
<br />
Unstable / Testing :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ unstable main<br />
<br />
Stable / Buster :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main<br />
<br />
Oldstable / Stretch :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main<br />
<br />
This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).<br />
<br />
=== Uploading packages ===<br />
<br />
Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.<br />
<br />
For packages to be accepted, they need to be :<br />
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt><br />
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)<br />
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders<br />
<br />
== Git repositories for Debian packages ==<br />
<br />
Our git repository structure for Debian packages is compatible with <tt>git-buildpackage</tt>.<br />
<br />
We have two different ways of handling repositories for Debian packages:<br />
* Packages of python modules where *we* are upstream<br />
* Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)<br />
<br />
For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:<br />
* Our own modules are merged with the upstream repository<br />
* External dependencies ignore the upstream repository and only have packaging branches.<br />
<br />
=== Branch and tags structure ===<br />
<br />
Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:<br />
* debian/upstream (history of unpacked upstream releases)<br />
* debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, stretch-swh)<br />
* pristine-tar (data to regenerate upstream tarballs from a git export)<br />
<br />
The name of the debian/upstream branch doesn't matter ''as long as it's properly configured in the <tt>debian/gbp.conf</tt> file''. It's only really used by <tt>gbp import-orig</tt> when importing a new release.<br />
<br />
The tags marking upstream releases imported from tarballs for Debian packaging purposes are named <tt>debian/upstream/''<upstream version number>''</tt>.<br />
<br />
Our Jenkins jobs are triggered on incoming tags named <tt>debian/''<version>''</tt>. To generate the proper tags, use <tt>gbp buildpackage --git-tag-only</tt>.<br />
<br />
The git-buildpackage configuration, <tt>debian/gbp.conf</tt>, should be the following:<br />
[DEFAULT]<br />
upstream-branch=debian/upstream<br />
upstream-tag=debian/upstream/%(version)s<br />
debian-branch=debian/''<current suite>''<br />
pristine-tar=True<br />
<br />
==== Automatic packaging for swh python modules ====<br />
<br />
The <tt>swh.*</tt> python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs <tt>gbp import-orig</tt> with the tarball we release to PyPI, and the right options to merge the upstream history.<br />
<br />
To merge changes from the upstream history, we add the following option to <tt>gbp.conf</tt>:<br />
upstream-vcs-tag=v%(version)s<br />
<br />
=== Bootstrapping a dependency packaging repository ===<br />
<br />
Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:<br />
<br />
Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/<br />
wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz<br />
<br />
Create a new git repository<br />
git init pytest-postgresql<br />
cd pytest-postgresql<br />
<br />
Import the original upstream version<br />
git checkout -b debian/unstable-swh<br />
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz<br />
# What will be the source package name? [pytest-postgresql] <br />
# What is the upstream version? [1.3.4] <br />
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...<br />
# gbp:info: Source package is pytest-postgresql<br />
# gbp:info: Upstream version is 1.3.4<br />
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz<br />
<br />
Bootstrap the debian directory<br />
mkdir -p debian/source<br />
echo '3.0 (quilt)' > debian/source/format<br />
echo 9 > debian/compat<br />
cat > debian/gbp.conf << EOF<br />
[DEFAULT]<br />
upstream-branch=debian/upstream<br />
upstream-tag=debian/upstream/%(version)s<br />
debian-branch=debian/unstable-swh<br />
pristine-tar=True<br />
EOF<br />
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules<br />
vim debian/control<br />
# [...] adapt debian/control from another package<br />
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1 --distribution unstable-swh<br />
vim debian/copyright<br />
# [...] adapt debian/copyright from another package<br />
git add debian<br />
git commit -m "Initial packaging for pytest-postgresql"<br />
<br />
You can then go on to try building the package. Once the package builds, if you want to check your package's conformance to Debian policy, you can run <tt>lintian</tt> on the changes:<br />
lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes<br />
<br />
Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository<br />
<br />
We need to use a <tt>+swh1</tt> version suffix to avoid clashing with potential upstream Debian package versions.<br />
<br />
==== Bootstrapping the backport branches ====<br />
<br />
During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.<br />
<br />
The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.<br />
<br />
Checkout the new branch<br />
git checkout debian/<version number><br />
git checkout -b debian/stretch-swh<br />
<br />
Update the gbp config to match the branch<br />
sed -i s/unstable-swh/stretch-swh/ debian/gbp.conf<br />
<br />
Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)<br />
dch -l "~bpo9" -D stretch-swh --force-distribution 'Rebuild for stretch-swh'<br />
<br />
You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.<br />
<br />
==== Setting up the repository on Phabricator ====<br />
<br />
The repository on Phabricator needs the following settings:<br />
* Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)<br />
* Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)<br />
* Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)<br />
* Policy<br />
** View: Public (no login required)<br />
** Edit: Developers<br />
** Push: All users (actual restrictions are handled by Herald rules)<br />
* Activate the repository<br />
* Look up the path to the repository on the storage tab<br />
<br />
You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes.<br />
ssh -t tate.internal.softwareheritage.org phabricator-setup-hook /srv/phabricator/repos/<repo-id> post-receive-debian-deps<br />
<br />
Note: remember that access to tate is on port 2222.<br />
<br />
The repo ID can be found on the repo's "storage" property page on phabricator, typically <br />
https://forge.softwareheritage.org/source/swh-SHORTNAME/manage/storage/<br />
<br />
==== Setting up the Jenkins jobs ====<br />
<br />
The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/<br />
They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs<br />
<br />
Jobs for dependency packages are configured in <tt>jobs/dependency-packages.yaml</tt>. You can add a section as follows:<br />
<br />
- project:<br />
name: <Callsign><br />
display-name: <short-name><br />
pkg: <source-name><br />
jobs:<br />
- 'dependency-jobs-{name}'<br />
<br />
Use the regular review process to land your changes.<br />
Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.<br />
<br />
If your package needs extra repositories to build, you can add them as comma-separated values to the <tt>deb-extra-repositories</tt> setting, with the following notes:<br />
* When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.<br />
* When building packages for backports suites, the backports repository is automatically enabled.<br />
<br />
=== Updating a dependency packaging repository ===<br />
<br />
Place yourself on the debian/unstable-swh branch and "gbp import-origin" a more<br />
recent upstream release tarballs.<br />
<br />
For example (current version on 0.0.5, upstream bumped to 0.0.7):<br />
gbp import-origin https://files.pythonhosted.org/packages/7a/bb/cf8fec6009e7d0cec52dc179d09b28c4c70d158e79b565e8aab7606e1717/attrs-strict-0.0.7.tar.gz<br />
<br />
This will update the following branches:<br />
* debian/upstream<br />
* pristine-tar<br />
* debian/unstable-swh<br />
<br />
This also includes the necessary tags (`debian/upstream/0.0.7` here).<br />
<br />
You then need to push all branches/tags to the repository:<br />
git push origin --all --follow-tags<br />
<br />
Ensure the [https://wiki.softwareheritage.org/wiki/Debian_packaging#Local_package_building update builds fine] <br />
And [https://wiki.softwareheritage.org/wiki/Debian_packaging#Remote_package_building tags accordingly the debian/unstable-swh branch when ok]. <br />
Jenkins will then keep up on building the package.<br />
<br />
=== Local package building ===<br />
<br />
To locally test a package build, go on the appropriate debian packaging branch, and run<br />
gbp buildpackage --git-builder=sbuild -As --no-clean-source<br />
<br />
<tt>gbp buildpackage</tt> passes all options not starting with <tt>--git-</tt> to the builder. Some useful options are the following:<br />
<br />
* <tt>--git-ignore-new</tt> builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.<br />
* <tt>--no-clean-source</tt> doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies<br />
* <tt>--extra-repository="'''repository specification'''"</tt> adds the given repository in the chroot before building.<br />
* <tt>--extra-repository-key='''repository signing key'''</tt> adds the given key as a trusted gpg key for package sources<br />
* <tt>--extra-package='''<.deb file or directory>'''</tt> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.<br />
* <tt>--force-orig-source</tt> forces addition of the <tt>.orig.tar.gz</tt> file in the <tt>.changes</tt> file (useful when trying to upload a backport)<br />
<br />
See <tt>gbp help buildpackage</tt> and <tt>man sbuild</tt> for a full description of all options<br />
<br />
for example:<br />
gbp buildpackage --git-builder=sbuild -As --force-orig-source --extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main'<br />
<br />
or if you need some third-party repository, say for cassandra:<br />
gbp buildpackage --git-builder=sbuild -As --force-orig-source \<br />
--extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main' \<br />
--extra-repository='deb [arch=amd64 trusted=yes] https://downloads.apache.org/cassandra/debian 40x main'<br />
<br />
<br />
(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping <tt>gbp buildpackage</tt> with the most common options)<br />
<br />
=== Remote package building ===<br />
<br />
Jenkins builds packages when the repository receives a tag.<br />
<br />
Once the local build succeeds, tag the package with:<br />
gbp buildpackage --git-tag-only --git-sign-tags<br />
<br />
Alternatively, you can add the <tt>--git-tag</tt> option to your <tt>gbp buildpackage</tt> command so the tag happens automatically on a successful build.<br />
<br />
Then, push your tag, and Jenkins jobs should get triggered<br />
git push --tags<br />
<br />
== Build Environment setup ==<br />
<br />
Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.<br />
<br />
=== sbuild setup ===<br />
<br />
# Install the package<br />
sudo apt-get install sbuild<br />
<br />
# Add your user to the sbuild group, to allow him to use the sbuild commands<br />
sudo sbuild-adduser $USER<br />
# You have to logout and log back in<br />
<br />
# Prepare chroots<br />
sudo mkdir /srv/chroots<br />
sudo mkdir /srv/chroots/var<br />
<br />
# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition<br />
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/<br />
sudo rm -r /var/lib/schroot<br />
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot<br />
<br />
sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/<br />
sudo rm -r /var/lib/sbuild<br />
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild<br />
# end optionally<br />
<br />
# Create unstable/sid chroot<br />
sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/<br />
<br />
# Create stretch chroot<br />
sudo sbuild-createchroot --include apt-transport-https,ca-certificates stretch /srv/chroots/stretch http://deb.debian.org/debian/<br />
<br />
<br />
# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts<br />
echo hosts >> /etc/schroot/sbuild/nssdatabases<br />
<br />
=== schroot setup ===<br />
<br />
Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.<br />
<br />
You need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:<br />
<br />
source-groups=root,sbuild<br />
source-root-groups=root,sbuild<br />
union-type=overlay<br />
<br />
This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.<br />
<br />
You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):<br />
<br />
For unstable:<br />
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild<br />
<br />
For stretch:<br />
aliases=stretch-swh-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild<br />
<br />
==== dependencies cache ====<br />
<br />
Add the following line to schroot's fstab /etc/schroot/sbuild/fstab<br />
to permit reuse of existing fetched dependencies:<br />
<br />
/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0<br />
<br />
You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in <tt>/etc/apt/apt.conf.d</tt> on each chroot<br />
<br />
=== schroot update ===<br />
<br />
You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):<br />
<br />
sudo sbuild-update -udcar sid; sudo sbuild-update -udcar stretch; sudo sbuild-update -ud jessie <br />
<br />
=== environment setup ===<br />
<br />
The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt><br />
<br />
export DEBFULLNAME="Debra Hacker"<br />
export DEBEMAIL=debra.hacker@example.com<br />
<br />
Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourfullkeyid></tt> variable.<br />
(Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).<br />
<br />
=== overlay in tmpfs for faster builds ===<br />
<br />
You can add this to your fstab to put the overlay hierarchy in RAM:<br />
<br />
tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0 0 0<br />
<br />
=== Base packages ===<br />
<br />
In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.<br />
<br />
'''If you do so, do not use these chroots to upload to Debian itself!'''<br />
<br />
<br />
[[Category:Software development]]<br />
[[Category:System administration]]</div>Douarddahttps://wiki.softwareheritage.org/index.php?title=Debian_packaging&diff=1336Debian packaging2020-09-23T13:18:55Z<p>Douardda: /* Setting up the repository on Phabricator */</p>
<hr />
<div>== Package repository ==<br />
<br />
A package repository is available on https://debian.softwareheritage.org/.<br />
<br />
Unstable / Testing :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ unstable main<br />
<br />
Stable / Buster :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main<br />
<br />
Oldstable / Stretch :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main<br />
<br />
This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).<br />
<br />
=== Uploading packages ===<br />
<br />
Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.<br />
<br />
For packages to be accepted, they need to be :<br />
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt><br />
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)<br />
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders<br />
<br />
== Git repositories for Debian packages ==<br />
<br />
Our git repository structure for Debian packages is compatible with <tt>git-buildpackage</tt>.<br />
<br />
We have two different ways of handling repositories for Debian packages:<br />
* Packages of python modules where *we* are upstream<br />
* Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)<br />
<br />
For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:<br />
* Our own modules are merged with the upstream repository<br />
* External dependencies ignore the upstream repository and only have packaging branches.<br />
<br />
=== Branch and tags structure ===<br />
<br />
Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:<br />
* debian/upstream (history of unpacked upstream releases)<br />
* debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, stretch-swh)<br />
* pristine-tar (data to regenerate upstream tarballs from a git export)<br />
<br />
The name of the debian/upstream branch doesn't matter ''as long as it's properly configured in the <tt>debian/gbp.conf</tt> file''. It's only really used by <tt>gbp import-orig</tt> when importing a new release.<br />
<br />
The tags marking upstream releases imported from tarballs for Debian packaging purposes are named <tt>debian/upstream/''<upstream version number>''</tt>.<br />
<br />
Our Jenkins jobs are triggered on incoming tags named <tt>debian/''<version>''</tt>. To generate the proper tags, use <tt>gbp buildpackage --git-tag-only</tt>.<br />
<br />
The git-buildpackage configuration, <tt>debian/gbp.conf</tt>, should be the following:<br />
[DEFAULT]<br />
upstream-branch=debian/upstream<br />
upstream-tag=debian/upstream/%(version)s<br />
debian-branch=debian/''<current suite>''<br />
pristine-tar=True<br />
<br />
==== Automatic packaging for swh python modules ====<br />
<br />
The <tt>swh.*</tt> python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs <tt>gbp import-orig</tt> with the tarball we release to PyPI, and the right options to merge the upstream history.<br />
<br />
To merge changes from the upstream history, we add the following option to <tt>gbp.conf</tt>:<br />
upstream-vcs-tag=v%(version)s<br />
<br />
=== Bootstrapping a dependency packaging repository ===<br />
<br />
Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:<br />
<br />
Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/<br />
wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz<br />
<br />
Create a new git repository<br />
git init pytest-postgresql<br />
cd pytest-postgresql<br />
<br />
Import the original upstream version<br />
git checkout -b debian/unstable-swh<br />
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz<br />
# What will be the source package name? [pytest-postgresql] <br />
# What is the upstream version? [1.3.4] <br />
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...<br />
# gbp:info: Source package is pytest-postgresql<br />
# gbp:info: Upstream version is 1.3.4<br />
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz<br />
<br />
Bootstrap the debian directory<br />
mkdir -p debian/source<br />
echo '3.0 (quilt)' > debian/source/format<br />
echo 9 > debian/compat<br />
cat > debian/gbp.conf << EOF<br />
[DEFAULT]<br />
upstream-branch=debian/upstream<br />
upstream-tag=debian/upstream/%(version)s<br />
debian-branch=debian/unstable-swh<br />
pristine-tar=True<br />
EOF<br />
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules<br />
vim debian/control<br />
# [...] adapt debian/control from another package<br />
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1 --distribution unstable-swh<br />
vim debian/copyright<br />
# [...] adapt debian/copyright from another package<br />
git add debian<br />
git commit -m "Initial packaging for pytest-postgresql"<br />
<br />
You can then go on to try building the package. Once the package builds, if you want to check your package's conformance to Debian policy, you can run <tt>lintian</tt> on the changes:<br />
lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes<br />
<br />
Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository<br />
<br />
We need to use a <tt>+swh1</tt> version suffix to avoid clashing with potential upstream Debian package versions.<br />
<br />
==== Bootstrapping the backport branches ====<br />
<br />
During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.<br />
<br />
The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.<br />
<br />
Checkout the new branch<br />
git checkout debian/<version number><br />
git checkout -b debian/stretch-swh<br />
<br />
Update the gbp config to match the branch<br />
sed -i s/unstable-swh/stretch-swh/ debian/gbp.conf<br />
<br />
Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)<br />
dch -l "~bpo9" -D stretch-swh --force-distribution 'Rebuild for stretch-swh'<br />
<br />
You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.<br />
<br />
==== Setting up the repository on Phabricator ====<br />
<br />
The repository on Phabricator needs the following settings:<br />
* Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)<br />
* Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)<br />
* Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)<br />
* Policy<br />
** View: Public (no login required)<br />
** Edit: Developers<br />
** Push: All users (actual restrictions are handled by Herald rules)<br />
* Activate the repository<br />
* Look up the path to the repository on the storage tab<br />
<br />
You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes.<br />
ssh -t tate.internal.softwareheritage.org phabricator-setup-hook /srv/phabricator/repos/<repo-id> post-receive-debian-deps<br />
<br />
The repo ID can be found on the repo's "storage" property page on phabricator, typically <br />
https://forge.softwareheritage.org/source/swh-SHORTNAME/manage/storage/<br />
<br />
==== Setting up the Jenkins jobs ====<br />
<br />
The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/<br />
They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs<br />
<br />
Jobs for dependency packages are configured in <tt>jobs/dependency-packages.yaml</tt>. You can add a section as follows:<br />
<br />
- project:<br />
name: <Callsign><br />
display-name: <short-name><br />
pkg: <source-name><br />
jobs:<br />
- 'dependency-jobs-{name}'<br />
<br />
Use the regular review process to land your changes.<br />
Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.<br />
<br />
If your package needs extra repositories to build, you can add them as comma-separated values to the <tt>deb-extra-repositories</tt> setting, with the following notes:<br />
* When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.<br />
* When building packages for backports suites, the backports repository is automatically enabled.<br />
<br />
=== Updating a dependency packaging repository ===<br />
<br />
Place yourself on the debian/unstable-swh branch and "gbp import-origin" a more<br />
recent upstream release tarballs.<br />
<br />
For example (current version on 0.0.5, upstream bumped to 0.0.7):<br />
gbp import-origin https://files.pythonhosted.org/packages/7a/bb/cf8fec6009e7d0cec52dc179d09b28c4c70d158e79b565e8aab7606e1717/attrs-strict-0.0.7.tar.gz<br />
<br />
This will update the following branches:<br />
* debian/upstream<br />
* pristine-tar<br />
* debian/unstable-swh<br />
<br />
This also includes the necessary tags (`debian/upstream/0.0.7` here).<br />
<br />
You then need to push all branches/tags to the repository:<br />
git push origin --all --follow-tags<br />
<br />
Ensure the [https://wiki.softwareheritage.org/wiki/Debian_packaging#Local_package_building update builds fine] <br />
And [https://wiki.softwareheritage.org/wiki/Debian_packaging#Remote_package_building tags accordingly the debian/unstable-swh branch when ok]. <br />
Jenkins will then keep up on building the package.<br />
<br />
=== Local package building ===<br />
<br />
To locally test a package build, go on the appropriate debian packaging branch, and run<br />
gbp buildpackage --git-builder=sbuild -As --no-clean-source<br />
<br />
<tt>gbp buildpackage</tt> passes all options not starting with <tt>--git-</tt> to the builder. Some useful options are the following:<br />
<br />
* <tt>--git-ignore-new</tt> builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.<br />
* <tt>--no-clean-source</tt> doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies<br />
* <tt>--extra-repository="'''repository specification'''"</tt> adds the given repository in the chroot before building.<br />
* <tt>--extra-repository-key='''repository signing key'''</tt> adds the given key as a trusted gpg key for package sources<br />
* <tt>--extra-package='''<.deb file or directory>'''</tt> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.<br />
* <tt>--force-orig-source</tt> forces addition of the <tt>.orig.tar.gz</tt> file in the <tt>.changes</tt> file (useful when trying to upload a backport)<br />
<br />
See <tt>gbp help buildpackage</tt> and <tt>man sbuild</tt> for a full description of all options<br />
<br />
for example:<br />
gbp buildpackage --git-builder=sbuild -As --force-orig-source --extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main'<br />
<br />
or if you need some third-party repository, say for cassandra:<br />
gbp buildpackage --git-builder=sbuild -As --force-orig-source \<br />
--extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main' \<br />
--extra-repository='deb [arch=amd64 trusted=yes] https://downloads.apache.org/cassandra/debian 40x main'<br />
<br />
<br />
(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping <tt>gbp buildpackage</tt> with the most common options)<br />
<br />
=== Remote package building ===<br />
<br />
Jenkins builds packages when the repository receives a tag.<br />
<br />
Once the local build succeeds, tag the package with:<br />
gbp buildpackage --git-tag-only --git-sign-tags<br />
<br />
Alternatively, you can add the <tt>--git-tag</tt> option to your <tt>gbp buildpackage</tt> command so the tag happens automatically on a successful build.<br />
<br />
Then, push your tag, and Jenkins jobs should get triggered<br />
git push --tags<br />
<br />
== Build Environment setup ==<br />
<br />
Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.<br />
<br />
=== sbuild setup ===<br />
<br />
# Install the package<br />
sudo apt-get install sbuild<br />
<br />
# Add your user to the sbuild group, to allow him to use the sbuild commands<br />
sudo sbuild-adduser $USER<br />
# You have to logout and log back in<br />
<br />
# Prepare chroots<br />
sudo mkdir /srv/chroots<br />
sudo mkdir /srv/chroots/var<br />
<br />
# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition<br />
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/<br />
sudo rm -r /var/lib/schroot<br />
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot<br />
<br />
sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/<br />
sudo rm -r /var/lib/sbuild<br />
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild<br />
# end optionally<br />
<br />
# Create unstable/sid chroot<br />
sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/<br />
<br />
# Create stretch chroot<br />
sudo sbuild-createchroot --include apt-transport-https,ca-certificates stretch /srv/chroots/stretch http://deb.debian.org/debian/<br />
<br />
<br />
# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts<br />
echo hosts >> /etc/schroot/sbuild/nssdatabases<br />
<br />
=== schroot setup ===<br />
<br />
Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.<br />
<br />
You need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:<br />
<br />
source-groups=root,sbuild<br />
source-root-groups=root,sbuild<br />
union-type=overlay<br />
<br />
This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.<br />
<br />
You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):<br />
<br />
For unstable:<br />
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild<br />
<br />
For stretch:<br />
aliases=stretch-swh-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild<br />
<br />
==== dependencies cache ====<br />
<br />
Add the following line to schroot's fstab /etc/schroot/sbuild/fstab<br />
to permit reuse of existing fetched dependencies:<br />
<br />
/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0<br />
<br />
You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in <tt>/etc/apt/apt.conf.d</tt> on each chroot<br />
<br />
=== schroot update ===<br />
<br />
You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):<br />
<br />
sudo sbuild-update -udcar sid; sudo sbuild-update -udcar stretch; sudo sbuild-update -ud jessie <br />
<br />
=== environment setup ===<br />
<br />
The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt><br />
<br />
export DEBFULLNAME="Debra Hacker"<br />
export DEBEMAIL=debra.hacker@example.com<br />
<br />
Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourfullkeyid></tt> variable.<br />
(Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).<br />
<br />
=== overlay in tmpfs for faster builds ===<br />
<br />
You can add this to your fstab to put the overlay hierarchy in RAM:<br />
<br />
tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0 0 0<br />
<br />
=== Base packages ===<br />
<br />
In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.<br />
<br />
'''If you do so, do not use these chroots to upload to Debian itself!'''<br />
<br />
<br />
[[Category:Software development]]<br />
[[Category:System administration]]</div>Douarddahttps://wiki.softwareheritage.org/index.php?title=Matrix&diff=1332Matrix2020-08-26T14:02:53Z<p>Douardda: /* IRC channels */ riot->Element</p>
<hr />
<div>== IRC channels ==<br />
<br />
The following channels have been registered on the [https://freenode.net/ Freenode] network for [[Software Heritage]] usage.<br />
<br />
* [https://app.element.io/app/#/room/#freenode_#swh-devel:matrix.org '''#swh-devel''']: public development discussions<br />
* [https://app.element.io/app/#/room/#freenode_#swh-team:matrix.org '''#swh-team''']: private discussions of the core team<br />
* [https://app.element.io/app/#/room/#freenode_#swh-sysadm:matrix.org '''#swh-sysadm''']: operations team discussions/bots<br />
* [https://app.element.io/app/#/room/#freenode_#softwareheritage:matrix.org '''#softwareheritage''']: general discussions about the project (currently unused)<br />
* [https://app.element.io/app/#/room/#freenode_#swh:matrix.org '''#swh''']: ditto, in case we end up preferring the short version<br />
<br />
If you use IRC, consider joining the channels.<br />
<br />
If you don't use IRC ''directly'', you can still join our chat channels from your web browser via a [https://matrix.org/ Matrix] bridge by clicking on the channel names in the list above. You will be asked to create a [https://element.io/ Element] account if you don't have one yet.<br />
<br />
== IRC authentication ==<br />
<br />
You should register their nick with NickServ using:<br />
<br />
/nick <USERNAME><br />
/msg nickserv register <PASSWORD> <EMAIL><br />
<br />
You will then receive an e-mail containing a link to activate you account. After doing so, you need to configure your client to auto-authenticate. The recommanded way of doing that is using [https://freenode.net/kb/answer/sasl SASL authentication].<br />
<br />
For Weechat:<br />
<br />
/set irc.server.freenode.sasl_username <USERNAME><br />
/set irc.server.freenode.sasl_password <PASSWORD><br />
<br />
Freenode also supports authentication via [https://freenode.net/kb/answer/certfp TLS client certificates].<br />
<br />
== IRC access list ==<br />
<br />
To auto-voice people with a registered nick (only doable by people with +fA access modes will be able to do it), add them to the channel access list:<br />
<br />
/msg chanserv access #swh-devel add zack +V<br />
<br />
[[Category:Infrastructure]]</div>Douarddahttps://wiki.softwareheritage.org/index.php?title=Debian_packaging&diff=1286Debian packaging2020-06-04T11:40:01Z<p>Douardda: /* Local package building */ stretch->buster</p>
<hr />
<div>== Package repository ==<br />
<br />
A package repository is available on https://debian.softwareheritage.org/.<br />
<br />
Unstable / Testing :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ unstable main<br />
<br />
Stable / Buster :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main<br />
<br />
Oldstable / Stretch :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main<br />
<br />
This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).<br />
<br />
=== Uploading packages ===<br />
<br />
Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.<br />
<br />
For packages to be accepted, they need to be :<br />
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt><br />
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)<br />
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders<br />
<br />
== Git repositories for Debian packages ==<br />
<br />
Our git repository structure for Debian packages is compatible with <tt>git-buildpackage</tt>.<br />
<br />
We have two different ways of handling repositories for Debian packages:<br />
* Packages of python modules where *we* are upstream<br />
* Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)<br />
<br />
For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:<br />
* Our own modules are merged with the upstream repository<br />
* External dependencies ignore the upstream repository and only have packaging branches.<br />
<br />
=== Branch and tags structure ===<br />
<br />
Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:<br />
* debian/upstream (history of unpacked upstream releases)<br />
* debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, stretch-swh)<br />
* pristine-tar (data to regenerate upstream tarballs from a git export)<br />
<br />
The name of the debian/upstream branch doesn't matter ''as long as it's properly configured in the <tt>debian/gbp.conf</tt> file''. It's only really used by <tt>gbp import-orig</tt> when importing a new release.<br />
<br />
The tags marking upstream releases imported from tarballs for Debian packaging purposes are named <tt>debian/upstream/''<upstream version number>''</tt>.<br />
<br />
Our Jenkins jobs are triggered on incoming tags named <tt>debian/''<version>''</tt>. To generate the proper tags, use <tt>gbp buildpackage --git-tag-only</tt>.<br />
<br />
The git-buildpackage configuration, <tt>debian/gbp.conf</tt>, should be the following:<br />
[DEFAULT]<br />
upstream-branch=debian/upstream<br />
upstream-tag=debian/upstream/%(version)s<br />
debian-branch=debian/''<current suite>''<br />
pristine-tar=True<br />
<br />
==== Automatic packaging for swh python modules ====<br />
<br />
The <tt>swh.*</tt> python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs <tt>gbp import-orig</tt> with the tarball we release to PyPI, and the right options to merge the upstream history.<br />
<br />
To merge changes from the upstream history, we add the following option to <tt>gbp.conf</tt>:<br />
upstream-vcs-tag=v%(version)s<br />
<br />
=== Bootstrapping a dependency packaging repository ===<br />
<br />
Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:<br />
<br />
Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/<br />
wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz<br />
<br />
Create a new git repository<br />
git init pytest-postgresql<br />
cd pytest-postgresql<br />
<br />
Import the original upstream version<br />
git checkout -b debian/unstable-swh<br />
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz<br />
# What will be the source package name? [pytest-postgresql] <br />
# What is the upstream version? [1.3.4] <br />
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...<br />
# gbp:info: Source package is pytest-postgresql<br />
# gbp:info: Upstream version is 1.3.4<br />
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz<br />
<br />
Bootstrap the debian directory<br />
mkdir -p debian/source<br />
echo '3.0 (quilt)' > debian/source/format<br />
echo 9 > debian/compat<br />
cat > debian/gbp.conf << EOF<br />
[DEFAULT]<br />
upstream-branch=debian/upstream<br />
upstream-tag=debian/upstream/%(version)s<br />
debian-branch=debian/unstable-swh<br />
pristine-tar=True<br />
EOF<br />
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules<br />
vim debian/control<br />
# [...] adapt debian/control from another package<br />
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1 --distribution unstable-swh<br />
vim debian/copyright<br />
# [...] adapt debian/copyright from another package<br />
git add debian<br />
git commit -m "Initial packaging for pytest-postgresql"<br />
<br />
You can then go on to try building the package. Once the package builds, if you want to check your package's conformance to Debian policy, you can run <tt>lintian</tt> on the changes:<br />
lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes<br />
<br />
Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository<br />
<br />
We need to use a <tt>+swh1</tt> version suffix to avoid clashing with potential upstream Debian package versions.<br />
<br />
==== Bootstrapping the backport branches ====<br />
<br />
During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.<br />
<br />
The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.<br />
<br />
Checkout the new branch<br />
git checkout debian/<version number><br />
git checkout -b debian/stretch-swh<br />
<br />
Update the gbp config to match the branch<br />
sed -i s/unstable-swh/stretch-swh/ debian/gbp.conf<br />
<br />
Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)<br />
dch -l "~bpo9" -D stretch-swh --force-distribution 'Rebuild for stretch-swh'<br />
<br />
You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.<br />
<br />
==== Setting up the repository on Phabricator ====<br />
<br />
The repository on Phabricator needs the following settings:<br />
* Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)<br />
* Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)<br />
* Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)<br />
* Policy<br />
** View: Public (no login required)<br />
** Edit: Developers<br />
** Push: All users (actual restrictions are handled by Herald rules)<br />
* Activate the repository<br />
* Look up the path to the repository on the storage tab<br />
<br />
You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes.<br />
ssh -t tate.internal.softwareheritage.org phabricator-setup-hook <repository-path> post-receive-debian-deps<br />
<br />
==== Setting up the Jenkins jobs ====<br />
<br />
The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/<br />
They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs<br />
<br />
Jobs for dependency packages are configured in <tt>jobs/dependency-packages.yaml</tt>. You can add a section as follows:<br />
<br />
- project:<br />
name: <Callsign><br />
display-name: <short-name><br />
pkg: <source-name><br />
jobs:<br />
- 'dependency-jobs-{name}'<br />
<br />
Use the regular review process to land your changes.<br />
Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.<br />
<br />
If your package needs extra repositories to build, you can add them as comma-separated values to the <tt>deb-extra-repositories</tt> setting, with the following notes:<br />
* When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.<br />
* When building packages for backports suites, the backports repository is automatically enabled.<br />
<br />
=== Updating a dependency packaging repository ===<br />
<br />
Place yourself on the debian/unstable-swh branch and "gbp import-origin" a more<br />
recent upstream release tarballs.<br />
<br />
For example (current version on 0.0.5, upstream bumped to 0.0.7):<br />
gbp import-origin https://files.pythonhosted.org/packages/7a/bb/cf8fec6009e7d0cec52dc179d09b28c4c70d158e79b565e8aab7606e1717/attrs-strict-0.0.7.tar.gz<br />
<br />
This will update the following branches:<br />
* debian/upstream<br />
* pristine-tar<br />
* debian/unstable-swh<br />
<br />
This also includes the necessary tags (`debian/upstream/0.0.7` here).<br />
<br />
You then need to push all branches/tags to the repository:<br />
git push origin --all --follow-tags<br />
<br />
Ensure the [https://wiki.softwareheritage.org/wiki/Debian_packaging#Local_package_building update builds fine] <br />
And [https://wiki.softwareheritage.org/wiki/Debian_packaging#Remote_package_building tags accordingly the debian/unstable-swh branch when ok]. <br />
Jenkins will then keep up on building the package.<br />
<br />
=== Local package building ===<br />
<br />
To locally test a package build, go on the appropriate debian packaging branch, and run<br />
gbp buildpackage --git-builder=sbuild -As --no-clean-source<br />
<br />
<tt>gbp buildpackage</tt> passes all options not starting with <tt>--git-</tt> to the builder. Some useful options are the following:<br />
<br />
* <tt>--git-ignore-new</tt> builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.<br />
* <tt>--no-clean-source</tt> doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies<br />
* <tt>--extra-repository="'''repository specification'''"</tt> adds the given repository in the chroot before building.<br />
* <tt>--extra-repository-key='''repository signing key'''</tt> adds the given key as a trusted gpg key for package sources<br />
* <tt>--extra-package='''<.deb file or directory>'''</tt> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.<br />
* <tt>--force-orig-source</tt> forces addition of the <tt>.orig.tar.gz</tt> file in the <tt>.changes</tt> file (useful when trying to upload a backport)<br />
<br />
See <tt>gbp help buildpackage</tt> and <tt>man sbuild</tt> for a full description of all options<br />
<br />
for example:<br />
gbp buildpackage --git-builder=sbuild -As --force-orig-source --extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main'<br />
<br />
or if you need some third-party repository, say for cassandra:<br />
gbp buildpackage --git-builder=sbuild -As --force-orig-source \<br />
--extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main' \<br />
--extra-repository='deb [arch=amd64 trusted=yes] https://downloads.apache.org/cassandra/debian 40x main'<br />
<br />
<br />
(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping <tt>gbp buildpackage</tt> with the most common options)<br />
<br />
=== Remote package building ===<br />
<br />
Jenkins builds packages when the repository receives a tag.<br />
<br />
Once the local build succeeds, tag the package with:<br />
gbp buildpackage --git-tag-only --git-sign-tags<br />
<br />
Alternatively, you can add the <tt>--git-tag</tt> option to your <tt>gbp buildpackage</tt> command so the tag happens automatically on a successful build.<br />
<br />
Then, push your tag, and Jenkins jobs should get triggered<br />
git push --tags<br />
<br />
== Build Environment setup ==<br />
<br />
Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.<br />
<br />
=== sbuild setup ===<br />
<br />
# Install the package<br />
sudo apt-get install sbuild<br />
<br />
# Add your user to the sbuild group, to allow him to use the sbuild commands<br />
sudo sbuild-adduser $USER<br />
# You have to logout and log back in<br />
<br />
# Prepare chroots<br />
sudo mkdir /srv/chroots<br />
sudo mkdir /srv/chroots/var<br />
<br />
# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition<br />
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/<br />
sudo rm -r /var/lib/schroot<br />
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot<br />
<br />
sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/<br />
sudo rm -r /var/lib/sbuild<br />
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild<br />
# end optionally<br />
<br />
# Create unstable/sid chroot<br />
sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/<br />
<br />
# Create stretch chroot<br />
sudo sbuild-createchroot --include apt-transport-https,ca-certificates stretch /srv/chroots/stretch http://deb.debian.org/debian/<br />
<br />
<br />
# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts<br />
echo hosts >> /etc/schroot/sbuild/nssdatabases<br />
<br />
=== schroot setup ===<br />
<br />
Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.<br />
<br />
You need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:<br />
<br />
source-groups=root,sbuild<br />
source-root-groups=root,sbuild<br />
union-type=overlay<br />
<br />
This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.<br />
<br />
You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):<br />
<br />
For unstable:<br />
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild<br />
<br />
For stretch:<br />
aliases=stretch-swh-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild<br />
<br />
==== dependencies cache ====<br />
<br />
Add the following line to schroot's fstab /etc/schroot/sbuild/fstab<br />
to permit reuse of existing fetched dependencies:<br />
<br />
/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0<br />
<br />
You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in <tt>/etc/apt/apt.conf.d</tt> on each chroot<br />
<br />
=== schroot update ===<br />
<br />
You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):<br />
<br />
sudo sbuild-update -udcar sid; sudo sbuild-update -udcar stretch; sudo sbuild-update -ud jessie <br />
<br />
=== environment setup ===<br />
<br />
The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt><br />
<br />
export DEBFULLNAME="Debra Hacker"<br />
export DEBEMAIL=debra.hacker@example.com<br />
<br />
Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourfullkeyid></tt> variable.<br />
(Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).<br />
<br />
=== overlay in tmpfs for faster builds ===<br />
<br />
You can add this to your fstab to put the overlay hierarchy in RAM:<br />
<br />
tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0 0 0<br />
<br />
=== Base packages ===<br />
<br />
In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.<br />
<br />
'''If you do so, do not use these chroots to upload to Debian itself!'''<br />
<br />
<br />
[[Category:Software development]]<br />
[[Category:System administration]]</div>Douarddahttps://wiki.softwareheritage.org/index.php?title=Debian_packaging&diff=1285Debian packaging2020-06-04T11:39:07Z<p>Douardda: /* Local package building */</p>
<hr />
<div>== Package repository ==<br />
<br />
A package repository is available on https://debian.softwareheritage.org/.<br />
<br />
Unstable / Testing :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ unstable main<br />
<br />
Stable / Buster :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main<br />
<br />
Oldstable / Stretch :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main<br />
<br />
This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).<br />
<br />
=== Uploading packages ===<br />
<br />
Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.<br />
<br />
For packages to be accepted, they need to be :<br />
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt><br />
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)<br />
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders<br />
<br />
== Git repositories for Debian packages ==<br />
<br />
Our git repository structure for Debian packages is compatible with <tt>git-buildpackage</tt>.<br />
<br />
We have two different ways of handling repositories for Debian packages:<br />
* Packages of python modules where *we* are upstream<br />
* Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)<br />
<br />
For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:<br />
* Our own modules are merged with the upstream repository<br />
* External dependencies ignore the upstream repository and only have packaging branches.<br />
<br />
=== Branch and tags structure ===<br />
<br />
Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:<br />
* debian/upstream (history of unpacked upstream releases)<br />
* debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, stretch-swh)<br />
* pristine-tar (data to regenerate upstream tarballs from a git export)<br />
<br />
The name of the debian/upstream branch doesn't matter ''as long as it's properly configured in the <tt>debian/gbp.conf</tt> file''. It's only really used by <tt>gbp import-orig</tt> when importing a new release.<br />
<br />
The tags marking upstream releases imported from tarballs for Debian packaging purposes are named <tt>debian/upstream/''<upstream version number>''</tt>.<br />
<br />
Our Jenkins jobs are triggered on incoming tags named <tt>debian/''<version>''</tt>. To generate the proper tags, use <tt>gbp buildpackage --git-tag-only</tt>.<br />
<br />
The git-buildpackage configuration, <tt>debian/gbp.conf</tt>, should be the following:<br />
[DEFAULT]<br />
upstream-branch=debian/upstream<br />
upstream-tag=debian/upstream/%(version)s<br />
debian-branch=debian/''<current suite>''<br />
pristine-tar=True<br />
<br />
==== Automatic packaging for swh python modules ====<br />
<br />
The <tt>swh.*</tt> python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs <tt>gbp import-orig</tt> with the tarball we release to PyPI, and the right options to merge the upstream history.<br />
<br />
To merge changes from the upstream history, we add the following option to <tt>gbp.conf</tt>:<br />
upstream-vcs-tag=v%(version)s<br />
<br />
=== Bootstrapping a dependency packaging repository ===<br />
<br />
Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:<br />
<br />
Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/<br />
wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz<br />
<br />
Create a new git repository<br />
git init pytest-postgresql<br />
cd pytest-postgresql<br />
<br />
Import the original upstream version<br />
git checkout -b debian/unstable-swh<br />
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz<br />
# What will be the source package name? [pytest-postgresql] <br />
# What is the upstream version? [1.3.4] <br />
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...<br />
# gbp:info: Source package is pytest-postgresql<br />
# gbp:info: Upstream version is 1.3.4<br />
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz<br />
<br />
Bootstrap the debian directory<br />
mkdir -p debian/source<br />
echo '3.0 (quilt)' > debian/source/format<br />
echo 9 > debian/compat<br />
cat > debian/gbp.conf << EOF<br />
[DEFAULT]<br />
upstream-branch=debian/upstream<br />
upstream-tag=debian/upstream/%(version)s<br />
debian-branch=debian/unstable-swh<br />
pristine-tar=True<br />
EOF<br />
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules<br />
vim debian/control<br />
# [...] adapt debian/control from another package<br />
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1 --distribution unstable-swh<br />
vim debian/copyright<br />
# [...] adapt debian/copyright from another package<br />
git add debian<br />
git commit -m "Initial packaging for pytest-postgresql"<br />
<br />
You can then go on to try building the package. Once the package builds, if you want to check your package's conformance to Debian policy, you can run <tt>lintian</tt> on the changes:<br />
lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes<br />
<br />
Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository<br />
<br />
We need to use a <tt>+swh1</tt> version suffix to avoid clashing with potential upstream Debian package versions.<br />
<br />
==== Bootstrapping the backport branches ====<br />
<br />
During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.<br />
<br />
The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.<br />
<br />
Checkout the new branch<br />
git checkout debian/<version number><br />
git checkout -b debian/stretch-swh<br />
<br />
Update the gbp config to match the branch<br />
sed -i s/unstable-swh/stretch-swh/ debian/gbp.conf<br />
<br />
Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)<br />
dch -l "~bpo9" -D stretch-swh --force-distribution 'Rebuild for stretch-swh'<br />
<br />
You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.<br />
<br />
==== Setting up the repository on Phabricator ====<br />
<br />
The repository on Phabricator needs the following settings:<br />
* Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)<br />
* Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)<br />
* Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)<br />
* Policy<br />
** View: Public (no login required)<br />
** Edit: Developers<br />
** Push: All users (actual restrictions are handled by Herald rules)<br />
* Activate the repository<br />
* Look up the path to the repository on the storage tab<br />
<br />
You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes.<br />
ssh -t tate.internal.softwareheritage.org phabricator-setup-hook <repository-path> post-receive-debian-deps<br />
<br />
==== Setting up the Jenkins jobs ====<br />
<br />
The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/<br />
They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs<br />
<br />
Jobs for dependency packages are configured in <tt>jobs/dependency-packages.yaml</tt>. You can add a section as follows:<br />
<br />
- project:<br />
name: <Callsign><br />
display-name: <short-name><br />
pkg: <source-name><br />
jobs:<br />
- 'dependency-jobs-{name}'<br />
<br />
Use the regular review process to land your changes.<br />
Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.<br />
<br />
If your package needs extra repositories to build, you can add them as comma-separated values to the <tt>deb-extra-repositories</tt> setting, with the following notes:<br />
* When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.<br />
* When building packages for backports suites, the backports repository is automatically enabled.<br />
<br />
=== Updating a dependency packaging repository ===<br />
<br />
Place yourself on the debian/unstable-swh branch and "gbp import-origin" a more<br />
recent upstream release tarballs.<br />
<br />
For example (current version on 0.0.5, upstream bumped to 0.0.7):<br />
gbp import-origin https://files.pythonhosted.org/packages/7a/bb/cf8fec6009e7d0cec52dc179d09b28c4c70d158e79b565e8aab7606e1717/attrs-strict-0.0.7.tar.gz<br />
<br />
This will update the following branches:<br />
* debian/upstream<br />
* pristine-tar<br />
* debian/unstable-swh<br />
<br />
This also includes the necessary tags (`debian/upstream/0.0.7` here).<br />
<br />
You then need to push all branches/tags to the repository:<br />
git push origin --all --follow-tags<br />
<br />
Ensure the [https://wiki.softwareheritage.org/wiki/Debian_packaging#Local_package_building update builds fine] <br />
And [https://wiki.softwareheritage.org/wiki/Debian_packaging#Remote_package_building tags accordingly the debian/unstable-swh branch when ok]. <br />
Jenkins will then keep up on building the package.<br />
<br />
=== Local package building ===<br />
<br />
To locally test a package build, go on the appropriate debian packaging branch, and run<br />
gbp buildpackage --git-builder=sbuild -As --no-clean-source<br />
<br />
<tt>gbp buildpackage</tt> passes all options not starting with <tt>--git-</tt> to the builder. Some useful options are the following:<br />
<br />
* <tt>--git-ignore-new</tt> builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.<br />
* <tt>--no-clean-source</tt> doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies<br />
* <tt>--extra-repository="'''repository specification'''"</tt> adds the given repository in the chroot before building.<br />
* <tt>--extra-repository-key='''repository signing key'''</tt> adds the given key as a trusted gpg key for package sources<br />
* <tt>--extra-package='''<.deb file or directory>'''</tt> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.<br />
* <tt>--force-orig-source</tt> forces addition of the <tt>.orig.tar.gz</tt> file in the <tt>.changes</tt> file (useful when trying to upload a backport)<br />
<br />
See <tt>gbp help buildpackage</tt> and <tt>man sbuild</tt> for a full description of all options<br />
<br />
for example:<br />
gbp buildpackage --git-builder=sbuild -As --force-orig-source --extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main'<br />
<br />
or if you need some third-party repository, say for cassandra:<br />
gbp buildpackage --git-builder=sbuild -As --force-orig-source \<br />
--extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main' \<br />
--extra-repository='deb [arch=amd64 trusted=yes] https://downloads.apache.org/cassandra/debian 40x main'<br />
<br />
<br />
(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping <tt>gbp buildpackage</tt> with the most common options)<br />
<br />
=== Remote package building ===<br />
<br />
Jenkins builds packages when the repository receives a tag.<br />
<br />
Once the local build succeeds, tag the package with:<br />
gbp buildpackage --git-tag-only --git-sign-tags<br />
<br />
Alternatively, you can add the <tt>--git-tag</tt> option to your <tt>gbp buildpackage</tt> command so the tag happens automatically on a successful build.<br />
<br />
Then, push your tag, and Jenkins jobs should get triggered<br />
git push --tags<br />
<br />
== Build Environment setup ==<br />
<br />
Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.<br />
<br />
=== sbuild setup ===<br />
<br />
# Install the package<br />
sudo apt-get install sbuild<br />
<br />
# Add your user to the sbuild group, to allow him to use the sbuild commands<br />
sudo sbuild-adduser $USER<br />
# You have to logout and log back in<br />
<br />
# Prepare chroots<br />
sudo mkdir /srv/chroots<br />
sudo mkdir /srv/chroots/var<br />
<br />
# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition<br />
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/<br />
sudo rm -r /var/lib/schroot<br />
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot<br />
<br />
sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/<br />
sudo rm -r /var/lib/sbuild<br />
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild<br />
# end optionally<br />
<br />
# Create unstable/sid chroot<br />
sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/<br />
<br />
# Create stretch chroot<br />
sudo sbuild-createchroot --include apt-transport-https,ca-certificates stretch /srv/chroots/stretch http://deb.debian.org/debian/<br />
<br />
<br />
# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts<br />
echo hosts >> /etc/schroot/sbuild/nssdatabases<br />
<br />
=== schroot setup ===<br />
<br />
Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.<br />
<br />
You need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:<br />
<br />
source-groups=root,sbuild<br />
source-root-groups=root,sbuild<br />
union-type=overlay<br />
<br />
This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.<br />
<br />
You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):<br />
<br />
For unstable:<br />
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild<br />
<br />
For stretch:<br />
aliases=stretch-swh-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild<br />
<br />
==== dependencies cache ====<br />
<br />
Add the following line to schroot's fstab /etc/schroot/sbuild/fstab<br />
to permit reuse of existing fetched dependencies:<br />
<br />
/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0<br />
<br />
You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in <tt>/etc/apt/apt.conf.d</tt> on each chroot<br />
<br />
=== schroot update ===<br />
<br />
You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):<br />
<br />
sudo sbuild-update -udcar sid; sudo sbuild-update -udcar stretch; sudo sbuild-update -ud jessie <br />
<br />
=== environment setup ===<br />
<br />
The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt><br />
<br />
export DEBFULLNAME="Debra Hacker"<br />
export DEBEMAIL=debra.hacker@example.com<br />
<br />
Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourfullkeyid></tt> variable.<br />
(Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).<br />
<br />
=== overlay in tmpfs for faster builds ===<br />
<br />
You can add this to your fstab to put the overlay hierarchy in RAM:<br />
<br />
tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0 0 0<br />
<br />
=== Base packages ===<br />
<br />
In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.<br />
<br />
'''If you do so, do not use these chroots to upload to Debian itself!'''<br />
<br />
<br />
[[Category:Software development]]<br />
[[Category:System administration]]</div>Douarddahttps://wiki.softwareheritage.org/index.php?title=Debian_packaging&diff=1075Debian packaging2019-07-16T12:01:41Z<p>Douardda: /* Package repository */</p>
<hr />
<div>== Package repository ==<br />
<br />
A package repository is available on https://debian.softwareheritage.org/.<br />
<br />
Unstable / Testing :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ unstable main<br />
<br />
Stable / Buster :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main<br />
<br />
Oldstable / Stretch :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main<br />
<br />
This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).<br />
<br />
=== Uploading packages ===<br />
<br />
Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.<br />
<br />
For packages to be accepted, they need to be :<br />
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt><br />
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)<br />
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders<br />
<br />
== Git repositories for Debian packages ==<br />
<br />
Our git repository structure for Debian packages is compatible with <tt>git-buildpackage</tt>.<br />
<br />
We have two different ways of handling repositories for Debian packages:<br />
* Packages of python modules where *we* are upstream<br />
* Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)<br />
<br />
For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:<br />
* Our own modules are merged with the upstream repository<br />
* External dependencies ignore the upstream repository and only have packaging branches.<br />
<br />
=== Branch and tags structure ===<br />
<br />
Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:<br />
* debian/upstream (history of unpacked upstream releases)<br />
* debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, stretch-swh)<br />
* pristine-tar (data to regenerate upstream tarballs from a git export)<br />
<br />
The name of the debian/upstream branch doesn't matter ''as long as it's properly configured in the <tt>debian/gbp.conf</tt> file''. It's only really used by <tt>gbp import-orig</tt> when importing a new release.<br />
<br />
The tags marking upstream releases imported from tarballs for Debian packaging purposes are named <tt>debian/upstream/''<upstream version number>''</tt>.<br />
<br />
Our Jenkins jobs are triggered on incoming tags named <tt>debian/''<version>''</tt>. To generate the proper tags, use <tt>gbp buildpackage --git-tag-only</tt>.<br />
<br />
The git-buildpackage configuration, <tt>debian/gbp.conf</tt>, should be the following:<br />
[DEFAULT]<br />
upstream-branch=debian/upstream<br />
upstream-tag=debian/upstream/%(version)s<br />
debian-branch=debian/''<current suite>''<br />
pristine-tar=True<br />
<br />
==== Automatic packaging for swh python modules ====<br />
<br />
The <tt>swh.*</tt> python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs <tt>gbp import-orig</tt> with the tarball we release to PyPI, and the right options to merge the upstream history.<br />
<br />
To merge changes from the upstream history, we add the following option to <tt>gbp.conf</tt>:<br />
upstream-vcs-tag=v%(version)s<br />
<br />
=== Bootstrapping a dependency packaging repository ===<br />
<br />
Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:<br />
<br />
Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/<br />
wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz<br />
<br />
Create a new git repository<br />
git init pytest-postgresql<br />
cd pytest-postgresql<br />
<br />
Import the original upstream version<br />
git checkout -b debian/unstable-swh<br />
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz<br />
# What will be the source package name? [pytest-postgresql] <br />
# What is the upstream version? [1.3.4] <br />
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...<br />
# gbp:info: Source package is pytest-postgresql<br />
# gbp:info: Upstream version is 1.3.4<br />
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz<br />
<br />
Bootstrap the debian directory<br />
mkdir -p debian/source<br />
echo '3.0 (quilt)' > debian/source/format<br />
echo 9 > debian/compat<br />
cat > debian/gbp.conf << EOF<br />
[DEFAULT]<br />
upstream-branch=debian/upstream<br />
upstream-tag=debian/upstream/%(version)s<br />
debian-branch=debian/unstable-swh<br />
pristine-tar=True<br />
EOF<br />
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules<br />
vim debian/control<br />
# [...] adapt debian/control from another package<br />
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1 --distribution unstable-swh<br />
vim debian/copyright<br />
# [...] adapt debian/copyright from another package<br />
git add debian<br />
git commit -m "Initial packaging for pytest-postgresql"<br />
<br />
You can then go on to try building the package. Once the package builds, if you want to check your package's conformance to Debian policy, you can run <tt>lintian</tt> on the changes:<br />
lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes<br />
<br />
Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository<br />
<br />
We need to use a <tt>+swh1</tt> version suffix to avoid clashing with potential upstream Debian package versions.<br />
<br />
==== Bootstrapping the backport branches ====<br />
<br />
During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.<br />
<br />
The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.<br />
<br />
Checkout the new branch<br />
git checkout debian/<version number><br />
git checkout -b debian/stretch-swh<br />
<br />
Update the gbp config to match the branch<br />
sed -i s/unstable-swh/stretch-swh/ debian/gbp.conf<br />
<br />
Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)<br />
dch -l "~bpo9" -D stretch-swh --force-distribution 'Rebuild for stretch-swh'<br />
<br />
You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.<br />
<br />
==== Setting up the repository on Phabricator ====<br />
<br />
The repository on Phabricator needs the following settings:<br />
* Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)<br />
* Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)<br />
* Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)<br />
* Policy<br />
** View: Public (no login required)<br />
** Edit: Developers<br />
** Push: All users (actual restrictions are handled by Herald rules)<br />
* Activate the repository<br />
* Look up the path to the repository on the storage tab<br />
<br />
You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes.<br />
ssh -t tate.internal.softwareheritage.org phabricator-setup-hook <repository-path> post-receive-debian-deps<br />
<br />
==== Setting up the Jenkins jobs ====<br />
<br />
The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/<br />
They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs<br />
<br />
Jobs for dependency packages are configured in <tt>jobs/dependency-packages.yaml</tt>. You can add a section as follows:<br />
<br />
- project:<br />
name: <Callsign><br />
display-name: <short-name><br />
pkg: <source-name><br />
jobs:<br />
- 'dependency-jobs-{name}'<br />
<br />
Use the regular review process to land your changes.<br />
Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.<br />
<br />
If your package needs extra repositories to build, you can add them as comma-separated values to the <tt>deb-extra-repositories</tt> setting, with the following notes:<br />
* When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.<br />
* When building packages for backports suites, the backports repository is automatically enabled.<br />
<br />
=== Local package building ===<br />
<br />
To locally test a package build, go on the appropriate debian packaging branch, and run<br />
gbp buildpackage --git-builder=sbuild -As --no-clean-source<br />
<br />
<tt>gbp buildpackage</tt> passes all options not starting with <tt>--git-</tt> to the builder. Some useful options are the following:<br />
<br />
* <tt>--git-ignore-new</tt> builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.<br />
* <tt>--no-clean-source</tt> doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies<br />
* <tt>--extra-repository="'''repository specification'''"</tt> adds the given repository in the chroot before building.<br />
* <tt>--extra-repository-key='''repository signing key'''</tt> adds the given key as a trusted gpg key for package sources<br />
* <tt>--extra-package='''<.deb file or directory>'''</tt> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.<br />
* <tt>--force-orig-source</tt> forces addition of the <tt>.orig.tar.gz</tt> file in the <tt>.changes</tt> file (useful when trying to upload a backport)<br />
<br />
See <tt>gbp help buildpackage</tt> and <tt>man sbuild</tt> for a full description of all options<br />
<br />
for example:<br />
gbp buildpackage --git-builder=sbuild -As --force-orig-source --extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main'<br />
<br />
<br />
(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping <tt>gbp buildpackage</tt> with the most common options)<br />
<br />
=== Remote package building ===<br />
<br />
Jenkins builds packages when the repository receives a tag.<br />
<br />
Once the local build succeeds, tag the package with:<br />
gbp buildpackage --git-tag-only --git-sign-tags<br />
<br />
Alternatively, you can add the <tt>--git-tag</tt> option to your <tt>gbp buildpackage</tt> command so the tag happens automatically on a successful build.<br />
<br />
Then, push your tag, and Jenkins jobs should get triggered<br />
git push --tags<br />
<br />
== Build Environment setup ==<br />
<br />
Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.<br />
<br />
=== sbuild setup ===<br />
<br />
# Install the package<br />
sudo apt-get install sbuild<br />
<br />
# Add your user to the sbuild group, to allow him to use the sbuild commands<br />
sudo sbuild-adduser $USER<br />
# You have to logout and log back in<br />
<br />
# Prepare chroots<br />
sudo mkdir /srv/chroots<br />
sudo mkdir /srv/chroots/var<br />
<br />
# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition<br />
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/<br />
sudo rm -r /var/lib/schroot<br />
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot<br />
<br />
sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/<br />
sudo rm -r /var/lib/sbuild<br />
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild<br />
# end optionally<br />
<br />
# Create unstable/sid chroot<br />
sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/<br />
<br />
# Create stretch chroot<br />
sudo sbuild-createchroot --include apt-transport-https,ca-certificates stretch /srv/chroots/stretch http://deb.debian.org/debian/<br />
<br />
<br />
# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts<br />
echo hosts >> /etc/schroot/sbuild/nssdatabases<br />
<br />
=== schroot setup ===<br />
<br />
Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.<br />
<br />
You need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:<br />
<br />
source-groups=root,sbuild<br />
source-root-groups=root,sbuild<br />
union-type=overlay<br />
<br />
This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.<br />
<br />
You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):<br />
<br />
For unstable:<br />
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild<br />
<br />
For stretch:<br />
aliases=stable-amd64-sbuild,stable-backports-amd64-sbuild,stretch-swh-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild<br />
<br />
==== dependencies cache ====<br />
<br />
Add the following line to schroot's fstab /etc/schroot/sbuild/fstab<br />
to permit reuse of existing fetched dependencies:<br />
<br />
/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0<br />
<br />
You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in <tt>/etc/apt/apt.conf.d</tt> on each chroot<br />
<br />
=== schroot update ===<br />
<br />
You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):<br />
<br />
sudo sbuild-update -udcar sid; sudo sbuild-update -udcar stretch; sudo sbuild-update -ud jessie <br />
<br />
=== environment setup ===<br />
<br />
The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt><br />
<br />
export DEBFULLNAME="Debra Hacker"<br />
export DEBEMAIL=debra.hacker@example.com<br />
<br />
Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourfullkeyid></tt> variable.<br />
(Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).<br />
<br />
=== overlay in tmpfs for faster builds ===<br />
<br />
You can add this to your fstab to put the overlay hierarchy in RAM:<br />
<br />
tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0 0 0<br />
<br />
=== Base packages ===<br />
<br />
In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.<br />
<br />
'''If you do so, do not use these chroots to upload to Debian itself!'''<br />
<br />
<br />
[[Category:Software development]]<br />
[[Category:System administration]]</div>Douarddahttps://wiki.softwareheritage.org/index.php?title=Debian_packaging&diff=1071Debian packaging2019-07-11T10:09:02Z<p>Douardda: /* Bootstrapping a dependency packaging repository */</p>
<hr />
<div>== Package repository ==<br />
<br />
A package repository is available on https://debian.softwareheritage.org/.<br />
<br />
Unstable / Testing :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ unstable main<br />
<br />
Stable / Stretch :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main<br />
<br />
This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).<br />
<br />
=== Uploading packages ===<br />
<br />
Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.<br />
<br />
For packages to be accepted, they need to be :<br />
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt><br />
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)<br />
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders<br />
<br />
<br />
== Git repositories for Debian packages ==<br />
<br />
Our git repository structure for Debian packages is compatible with <tt>git-buildpackage</tt>.<br />
<br />
We have two different ways of handling repositories for Debian packages:<br />
* Packages of python modules where *we* are upstream<br />
* Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)<br />
<br />
For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:<br />
* Our own modules are merged with the upstream repository<br />
* External dependencies ignore the upstream repository and only have packaging branches.<br />
<br />
=== Branch and tags structure ===<br />
<br />
Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:<br />
* debian/upstream (history of unpacked upstream releases)<br />
* debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, stretch-swh)<br />
* pristine-tar (data to regenerate upstream tarballs from a git export)<br />
<br />
The name of the debian/upstream branch doesn't matter ''as long as it's properly configured in the <tt>debian/gbp.conf</tt> file''. It's only really used by <tt>gbp import-orig</tt> when importing a new release.<br />
<br />
The tags marking upstream releases imported from tarballs for Debian packaging purposes are named <tt>debian/upstream/''<upstream version number>''</tt>.<br />
<br />
Our Jenkins jobs are triggered on incoming tags named <tt>debian/''<version>''</tt>. To generate the proper tags, use <tt>gbp buildpackage --git-tag-only</tt>.<br />
<br />
The git-buildpackage configuration, <tt>debian/gbp.conf</tt>, should be the following:<br />
[DEFAULT]<br />
upstream-branch=debian/upstream<br />
upstream-tag=debian/upstream/%(version)s<br />
debian-branch=debian/''<current suite>''<br />
pristine-tar=True<br />
<br />
==== Automatic packaging for swh python modules ====<br />
<br />
The <tt>swh.*</tt> python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs <tt>gbp import-orig</tt> with the tarball we release to PyPI, and the right options to merge the upstream history.<br />
<br />
To merge changes from the upstream history, we add the following option to <tt>gbp.conf</tt>:<br />
upstream-vcs-tag=v%(version)s<br />
<br />
=== Bootstrapping a dependency packaging repository ===<br />
<br />
Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:<br />
<br />
Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/<br />
wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz<br />
<br />
Create a new git repository<br />
git init pytest-postgresql<br />
cd pytest-postgresql<br />
<br />
Import the original upstream version<br />
git checkout -b debian/unstable-swh<br />
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz<br />
# What will be the source package name? [pytest-postgresql] <br />
# What is the upstream version? [1.3.4] <br />
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...<br />
# gbp:info: Source package is pytest-postgresql<br />
# gbp:info: Upstream version is 1.3.4<br />
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz<br />
<br />
Bootstrap the debian directory<br />
mkdir -p debian/source<br />
echo '3.0 (quilt)' > debian/source/format<br />
echo 9 > debian/compat<br />
cat > debian/gbp.conf << EOF<br />
[DEFAULT]<br />
upstream-branch=debian/upstream<br />
upstream-tag=debian/upstream/%(version)s<br />
debian-branch=debian/unstable-swh<br />
pristine-tar=True<br />
EOF<br />
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules<br />
vim debian/control<br />
# [...] adapt debian/control from another package<br />
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1 --distribution unstable-swh<br />
vim debian/copyright<br />
# [...] adapt debian/copyright from another package<br />
git add debian<br />
git commit -m "Initial packaging for pytest-postgresql"<br />
<br />
You can then go on to try building the package. Once the package builds, if you want to check your package's conformance to Debian policy, you can run <tt>lintian</tt> on the changes:<br />
lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes<br />
<br />
Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository<br />
<br />
We need to use a <tt>+swh1</tt> version suffix to avoid clashing with potential upstream Debian package versions.<br />
<br />
==== Bootstrapping the backport branches ====<br />
<br />
During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.<br />
<br />
The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.<br />
<br />
Checkout the new branch<br />
git checkout debian/<version number><br />
git checkout -b debian/stretch-swh<br />
<br />
Update the gbp config to match the branch<br />
sed -i s/unstable-swh/stretch-swh/ debian/gbp.conf<br />
<br />
Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)<br />
dch -l "~bpo9" -D stretch-swh --force-distribution 'Rebuild for stretch-swh'<br />
<br />
You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.<br />
<br />
==== Setting up the repository on Phabricator ====<br />
<br />
The repository on Phabricator needs the following settings:<br />
* Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)<br />
* Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)<br />
* Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)<br />
* Policy<br />
** View: Public (no login required)<br />
** Edit: Developers<br />
** Push: All users (actual restrictions are handled by Herald rules)<br />
* Activate the repository<br />
* Look up the path to the repository on the storage tab<br />
<br />
You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes.<br />
ssh -t tate.internal.softwareheritage.org phabricator-setup-hook <repository-path> post-receive-debian-deps<br />
<br />
==== Setting up the Jenkins jobs ====<br />
<br />
The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/<br />
They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs<br />
<br />
Jobs for dependency packages are configured in <tt>jobs/dependency-packages.yaml</tt>. You can add a section as follows:<br />
<br />
- project:<br />
name: <Callsign><br />
display-name: <short-name><br />
pkg: <source-name><br />
jobs:<br />
- 'dependency-jobs-{name}'<br />
<br />
Use the regular review process to land your changes.<br />
Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.<br />
<br />
If your package needs extra repositories to build, you can add them as comma-separated values to the <tt>deb-extra-repositories</tt> setting, with the following notes:<br />
* When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.<br />
* When building packages for backports suites, the backports repository is automatically enabled.<br />
<br />
=== Local package building ===<br />
<br />
To locally test a package build, go on the appropriate debian packaging branch, and run<br />
gbp buildpackage --git-builder=sbuild -As --no-clean-source<br />
<br />
<tt>gbp buildpackage</tt> passes all options not starting with <tt>--git-</tt> to the builder. Some useful options are the following:<br />
<br />
* <tt>--git-ignore-new</tt> builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.<br />
* <tt>--no-clean-source</tt> doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies<br />
* <tt>--extra-repository="'''repository specification'''"</tt> adds the given repository in the chroot before building.<br />
* <tt>--extra-repository-key='''repository signing key'''</tt> adds the given key as a trusted gpg key for package sources<br />
* <tt>--extra-package='''<.deb file or directory>'''</tt> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.<br />
* <tt>--force-orig-source</tt> forces addition of the <tt>.orig.tar.gz</tt> file in the <tt>.changes</tt> file (useful when trying to upload a backport)<br />
<br />
See <tt>gbp help buildpackage</tt> and <tt>man sbuild</tt> for a full description of all options<br />
<br />
for example:<br />
gbp buildpackage --git-builder=sbuild -As --force-orig-source --extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main'<br />
<br />
<br />
(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping <tt>gbp buildpackage</tt> with the most common options)<br />
<br />
=== Remote package building ===<br />
<br />
Jenkins builds packages when the repository receives a tag.<br />
<br />
Once the local build succeeds, tag the package with:<br />
gbp buildpackage --git-tag-only --git-sign-tags<br />
<br />
Alternatively, you can add the <tt>--git-tag</tt> option to your <tt>gbp buildpackage</tt> command so the tag happens automatically on a successful build.<br />
<br />
Then, push your tag, and Jenkins jobs should get triggered<br />
git push --tags<br />
<br />
== Build Environment setup ==<br />
<br />
Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.<br />
<br />
=== sbuild setup ===<br />
<br />
# Install the package<br />
sudo apt-get install sbuild<br />
<br />
# Add your user to the sbuild group, to allow him to use the sbuild commands<br />
sudo sbuild-adduser $USER<br />
# You have to logout and log back in<br />
<br />
# Prepare chroots<br />
sudo mkdir /srv/chroots<br />
sudo mkdir /srv/chroots/var<br />
<br />
# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition<br />
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/<br />
sudo rm -r /var/lib/schroot<br />
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot<br />
<br />
sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/<br />
sudo rm -r /var/lib/sbuild<br />
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild<br />
# end optionally<br />
<br />
# Create unstable/sid chroot<br />
sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/<br />
<br />
# Create stretch chroot<br />
sudo sbuild-createchroot --include apt-transport-https,ca-certificates stretch /srv/chroots/stretch http://deb.debian.org/debian/<br />
<br />
<br />
# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts<br />
echo hosts >> /etc/schroot/sbuild/nssdatabases<br />
<br />
=== schroot setup ===<br />
<br />
Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.<br />
<br />
You need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:<br />
<br />
source-groups=root,sbuild<br />
source-root-groups=root,sbuild<br />
union-type=overlay<br />
<br />
This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.<br />
<br />
You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):<br />
<br />
For unstable:<br />
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild<br />
<br />
For stretch:<br />
aliases=stable-amd64-sbuild,stable-backports-amd64-sbuild,stretch-swh-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild<br />
<br />
==== dependencies cache ====<br />
<br />
Add the following line to schroot's fstab /etc/schroot/sbuild/fstab<br />
to permit reuse of existing fetched dependencies:<br />
<br />
/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0<br />
<br />
You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in <tt>/etc/apt/apt.conf.d</tt> on each chroot<br />
<br />
=== schroot update ===<br />
<br />
You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):<br />
<br />
sudo sbuild-update -udcar sid; sudo sbuild-update -udcar stretch; sudo sbuild-update -ud jessie <br />
<br />
=== environment setup ===<br />
<br />
The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt><br />
<br />
export DEBFULLNAME="Debra Hacker"<br />
export DEBEMAIL=debra.hacker@example.com<br />
<br />
Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourfullkeyid></tt> variable.<br />
(Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).<br />
<br />
=== overlay in tmpfs for faster builds ===<br />
<br />
You can add this to your fstab to put the overlay hierarchy in RAM:<br />
<br />
tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0 0 0<br />
<br />
=== Base packages ===<br />
<br />
In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.<br />
<br />
'''If you do so, do not use these chroots to upload to Debian itself!'''<br />
<br />
<br />
[[Category:Software development]]<br />
[[Category:System administration]]</div>Douarddahttps://wiki.softwareheritage.org/index.php?title=Debian_packaging&diff=1070Debian packaging2019-07-11T10:00:07Z<p>Douardda: /* Bootstrapping a dependency packaging repository */</p>
<hr />
<div>== Package repository ==<br />
<br />
A package repository is available on https://debian.softwareheritage.org/.<br />
<br />
Unstable / Testing :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ unstable main<br />
<br />
Stable / Stretch :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main<br />
<br />
This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).<br />
<br />
=== Uploading packages ===<br />
<br />
Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.<br />
<br />
For packages to be accepted, they need to be :<br />
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt><br />
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)<br />
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders<br />
<br />
<br />
== Git repositories for Debian packages ==<br />
<br />
Our git repository structure for Debian packages is compatible with <tt>git-buildpackage</tt>.<br />
<br />
We have two different ways of handling repositories for Debian packages:<br />
* Packages of python modules where *we* are upstream<br />
* Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)<br />
<br />
For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:<br />
* Our own modules are merged with the upstream repository<br />
* External dependencies ignore the upstream repository and only have packaging branches.<br />
<br />
=== Branch and tags structure ===<br />
<br />
Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:<br />
* debian/upstream (history of unpacked upstream releases)<br />
* debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, stretch-swh)<br />
* pristine-tar (data to regenerate upstream tarballs from a git export)<br />
<br />
The name of the debian/upstream branch doesn't matter ''as long as it's properly configured in the <tt>debian/gbp.conf</tt> file''. It's only really used by <tt>gbp import-orig</tt> when importing a new release.<br />
<br />
The tags marking upstream releases imported from tarballs for Debian packaging purposes are named <tt>debian/upstream/''<upstream version number>''</tt>.<br />
<br />
Our Jenkins jobs are triggered on incoming tags named <tt>debian/''<version>''</tt>. To generate the proper tags, use <tt>gbp buildpackage --git-tag-only</tt>.<br />
<br />
The git-buildpackage configuration, <tt>debian/gbp.conf</tt>, should be the following:<br />
[DEFAULT]<br />
upstream-branch=debian/upstream<br />
upstream-tag=debian/upstream/%(version)s<br />
debian-branch=debian/''<current suite>''<br />
pristine-tar=True<br />
<br />
==== Automatic packaging for swh python modules ====<br />
<br />
The <tt>swh.*</tt> python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs <tt>gbp import-orig</tt> with the tarball we release to PyPI, and the right options to merge the upstream history.<br />
<br />
To merge changes from the upstream history, we add the following option to <tt>gbp.conf</tt>:<br />
upstream-vcs-tag=v%(version)s<br />
<br />
=== Bootstrapping a dependency packaging repository ===<br />
<br />
Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:<br />
<br />
Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/<br />
wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz<br />
<br />
Create a new git repository<br />
git init pytest-postgresql<br />
cd pytest-postgresql<br />
<br />
Import the original upstream version<br />
git checkout -b debian/unstable-swh<br />
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz<br />
# What will be the source package name? [pytest-postgresql] <br />
# What is the upstream version? [1.3.4] <br />
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...<br />
# gbp:info: Source package is pytest-postgresql<br />
# gbp:info: Upstream version is 1.3.4<br />
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz<br />
<br />
Bootstrap the debian directory<br />
mkdir -p debian/source<br />
echo '3.0 (quilt)' > debian/source/format<br />
echo 9 > debian/compat<br />
cat > debian/gbp.conf << EOF<br />
[DEFAULT]<br />
upstream-branch=debian/upstream<br />
upstream-tag=debian/upstream/%(version)s<br />
debian-branch=debian/unstable-swh<br />
pristine-tar=True<br />
EOF<br />
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules<br />
vim debian/control<br />
# [...] adapt debian/control from another package<br />
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1 --distribution unstable<br />
vim debian/copyright<br />
# [...] adapt debian/copyright from another package<br />
git add debian<br />
git commit -m "Initial packaging for pytest-postgresql"<br />
<br />
You can then go on to try building the package. Once the package builds, if you want to check your package's conformance to Debian policy, you can run <tt>lintian</tt> on the changes:<br />
lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes<br />
<br />
Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository<br />
<br />
We need to use a <tt>+swh1</tt> version suffix to avoid clashing with potential upstream Debian package versions.<br />
<br />
==== Bootstrapping the backport branches ====<br />
<br />
During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.<br />
<br />
The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.<br />
<br />
Checkout the new branch<br />
git checkout debian/<version number><br />
git checkout -b debian/stretch-swh<br />
<br />
Update the gbp config to match the branch<br />
sed -i s/unstable-swh/stretch-swh/ debian/gbp.conf<br />
<br />
Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)<br />
dch -l "~bpo9" -D stretch-swh --force-distribution 'Rebuild for stretch-swh'<br />
<br />
You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.<br />
<br />
==== Setting up the repository on Phabricator ====<br />
<br />
The repository on Phabricator needs the following settings:<br />
* Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)<br />
* Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)<br />
* Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)<br />
* Policy<br />
** View: Public (no login required)<br />
** Edit: Developers<br />
** Push: All users (actual restrictions are handled by Herald rules)<br />
* Activate the repository<br />
* Look up the path to the repository on the storage tab<br />
<br />
You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes.<br />
ssh -t tate.internal.softwareheritage.org phabricator-setup-hook <repository-path> post-receive-debian-deps<br />
<br />
==== Setting up the Jenkins jobs ====<br />
<br />
The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/<br />
They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs<br />
<br />
Jobs for dependency packages are configured in <tt>jobs/dependency-packages.yaml</tt>. You can add a section as follows:<br />
<br />
- project:<br />
name: <Callsign><br />
display-name: <short-name><br />
pkg: <source-name><br />
jobs:<br />
- 'dependency-jobs-{name}'<br />
<br />
Use the regular review process to land your changes.<br />
Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.<br />
<br />
If your package needs extra repositories to build, you can add them as comma-separated values to the <tt>deb-extra-repositories</tt> setting, with the following notes:<br />
* When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.<br />
* When building packages for backports suites, the backports repository is automatically enabled.<br />
<br />
=== Local package building ===<br />
<br />
To locally test a package build, go on the appropriate debian packaging branch, and run<br />
gbp buildpackage --git-builder=sbuild -As --no-clean-source<br />
<br />
<tt>gbp buildpackage</tt> passes all options not starting with <tt>--git-</tt> to the builder. Some useful options are the following:<br />
<br />
* <tt>--git-ignore-new</tt> builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.<br />
* <tt>--no-clean-source</tt> doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies<br />
* <tt>--extra-repository="'''repository specification'''"</tt> adds the given repository in the chroot before building.<br />
* <tt>--extra-repository-key='''repository signing key'''</tt> adds the given key as a trusted gpg key for package sources<br />
* <tt>--extra-package='''<.deb file or directory>'''</tt> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.<br />
* <tt>--force-orig-source</tt> forces addition of the <tt>.orig.tar.gz</tt> file in the <tt>.changes</tt> file (useful when trying to upload a backport)<br />
<br />
See <tt>gbp help buildpackage</tt> and <tt>man sbuild</tt> for a full description of all options<br />
<br />
for example:<br />
gbp buildpackage --git-builder=sbuild -As --force-orig-source --extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main'<br />
<br />
<br />
(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping <tt>gbp buildpackage</tt> with the most common options)<br />
<br />
=== Remote package building ===<br />
<br />
Jenkins builds packages when the repository receives a tag.<br />
<br />
Once the local build succeeds, tag the package with:<br />
gbp buildpackage --git-tag-only --git-sign-tags<br />
<br />
Alternatively, you can add the <tt>--git-tag</tt> option to your <tt>gbp buildpackage</tt> command so the tag happens automatically on a successful build.<br />
<br />
Then, push your tag, and Jenkins jobs should get triggered<br />
git push --tags<br />
<br />
== Build Environment setup ==<br />
<br />
Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.<br />
<br />
=== sbuild setup ===<br />
<br />
# Install the package<br />
sudo apt-get install sbuild<br />
<br />
# Add your user to the sbuild group, to allow him to use the sbuild commands<br />
sudo sbuild-adduser $USER<br />
# You have to logout and log back in<br />
<br />
# Prepare chroots<br />
sudo mkdir /srv/chroots<br />
sudo mkdir /srv/chroots/var<br />
<br />
# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition<br />
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/<br />
sudo rm -r /var/lib/schroot<br />
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot<br />
<br />
sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/<br />
sudo rm -r /var/lib/sbuild<br />
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild<br />
# end optionally<br />
<br />
# Create unstable/sid chroot<br />
sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/<br />
<br />
# Create stretch chroot<br />
sudo sbuild-createchroot --include apt-transport-https,ca-certificates stretch /srv/chroots/stretch http://deb.debian.org/debian/<br />
<br />
<br />
# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts<br />
echo hosts >> /etc/schroot/sbuild/nssdatabases<br />
<br />
=== schroot setup ===<br />
<br />
Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.<br />
<br />
You need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:<br />
<br />
source-groups=root,sbuild<br />
source-root-groups=root,sbuild<br />
union-type=overlay<br />
<br />
This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.<br />
<br />
You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):<br />
<br />
For unstable:<br />
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild<br />
<br />
For stretch:<br />
aliases=stable-amd64-sbuild,stable-backports-amd64-sbuild,stretch-swh-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild<br />
<br />
==== dependencies cache ====<br />
<br />
Add the following line to schroot's fstab /etc/schroot/sbuild/fstab<br />
to permit reuse of existing fetched dependencies:<br />
<br />
/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0<br />
<br />
You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in <tt>/etc/apt/apt.conf.d</tt> on each chroot<br />
<br />
=== schroot update ===<br />
<br />
You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):<br />
<br />
sudo sbuild-update -udcar sid; sudo sbuild-update -udcar stretch; sudo sbuild-update -ud jessie <br />
<br />
=== environment setup ===<br />
<br />
The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt><br />
<br />
export DEBFULLNAME="Debra Hacker"<br />
export DEBEMAIL=debra.hacker@example.com<br />
<br />
Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourfullkeyid></tt> variable.<br />
(Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).<br />
<br />
=== overlay in tmpfs for faster builds ===<br />
<br />
You can add this to your fstab to put the overlay hierarchy in RAM:<br />
<br />
tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0 0 0<br />
<br />
=== Base packages ===<br />
<br />
In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.<br />
<br />
'''If you do so, do not use these chroots to upload to Debian itself!'''<br />
<br />
<br />
[[Category:Software development]]<br />
[[Category:System administration]]</div>Douarddahttps://wiki.softwareheritage.org/index.php?title=Debian_packaging&diff=1069Debian packaging2019-07-11T09:56:51Z<p>Douardda: /* Bootstrapping a dependency packaging repository */</p>
<hr />
<div>== Package repository ==<br />
<br />
A package repository is available on https://debian.softwareheritage.org/.<br />
<br />
Unstable / Testing :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ unstable main<br />
<br />
Stable / Stretch :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main<br />
<br />
This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).<br />
<br />
=== Uploading packages ===<br />
<br />
Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.<br />
<br />
For packages to be accepted, they need to be :<br />
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt><br />
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)<br />
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders<br />
<br />
<br />
== Git repositories for Debian packages ==<br />
<br />
Our git repository structure for Debian packages is compatible with <tt>git-buildpackage</tt>.<br />
<br />
We have two different ways of handling repositories for Debian packages:<br />
* Packages of python modules where *we* are upstream<br />
* Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)<br />
<br />
For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:<br />
* Our own modules are merged with the upstream repository<br />
* External dependencies ignore the upstream repository and only have packaging branches.<br />
<br />
=== Branch and tags structure ===<br />
<br />
Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:<br />
* debian/upstream (history of unpacked upstream releases)<br />
* debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, stretch-swh)<br />
* pristine-tar (data to regenerate upstream tarballs from a git export)<br />
<br />
The name of the debian/upstream branch doesn't matter ''as long as it's properly configured in the <tt>debian/gbp.conf</tt> file''. It's only really used by <tt>gbp import-orig</tt> when importing a new release.<br />
<br />
The tags marking upstream releases imported from tarballs for Debian packaging purposes are named <tt>debian/upstream/''<upstream version number>''</tt>.<br />
<br />
Our Jenkins jobs are triggered on incoming tags named <tt>debian/''<version>''</tt>. To generate the proper tags, use <tt>gbp buildpackage --git-tag-only</tt>.<br />
<br />
The git-buildpackage configuration, <tt>debian/gbp.conf</tt>, should be the following:<br />
[DEFAULT]<br />
upstream-branch=debian/upstream<br />
upstream-tag=debian/upstream/%(version)s<br />
debian-branch=debian/''<current suite>''<br />
pristine-tar=True<br />
<br />
==== Automatic packaging for swh python modules ====<br />
<br />
The <tt>swh.*</tt> python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs <tt>gbp import-orig</tt> with the tarball we release to PyPI, and the right options to merge the upstream history.<br />
<br />
To merge changes from the upstream history, we add the following option to <tt>gbp.conf</tt>:<br />
upstream-vcs-tag=v%(version)s<br />
<br />
=== Bootstrapping a dependency packaging repository ===<br />
<br />
Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:<br />
<br />
Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/<br />
wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz<br />
<br />
Create a new git repository<br />
git init pytest-postgresql<br />
cd pytest-postgresql<br />
<br />
Import the original upstream version<br />
git checkout -b debian/unstable-swh<br />
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz<br />
# What will be the source package name? [pytest-postgresql] <br />
# What is the upstream version? [1.3.4] <br />
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...<br />
# gbp:info: Source package is pytest-postgresql<br />
# gbp:info: Upstream version is 1.3.4<br />
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz<br />
<br />
Bootstrap the debian directory<br />
mkdir -p debian/source<br />
echo '3.0 (quilt)' > debian/source/format<br />
echo 9 > debian/compat<br />
cat > debian/gbp.conf << EOF<br />
[DEFAULT]<br />
upstream-branch=debian/upstream<br />
upstream-tag=debian/upstream/%(version)s<br />
debian-branch=debian/unstable-swh<br />
pristine-tar=True<br />
EOF<br />
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules<br />
vim debian/control<br />
# [...] adapt debian/control from another package<br />
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1 --distribution unstable-swh<br />
vim debian/copyright<br />
# [...] adapt debian/copyright from another package<br />
git add debian<br />
git commit -m "Initial packaging for pytest-postgresql"<br />
<br />
You can then go on to try building the package. Once the package builds, if you want to check your package's conformance to Debian policy, you can run <tt>lintian</tt> on the changes:<br />
lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes<br />
<br />
Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository<br />
<br />
We need to use a <tt>+swh1</tt> version suffix to avoid clashing with potential upstream Debian package versions.<br />
<br />
==== Bootstrapping the backport branches ====<br />
<br />
During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.<br />
<br />
The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.<br />
<br />
Checkout the new branch<br />
git checkout debian/<version number><br />
git checkout -b debian/stretch-swh<br />
<br />
Update the gbp config to match the branch<br />
sed -i s/unstable-swh/stretch-swh/ debian/gbp.conf<br />
<br />
Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)<br />
dch -l "~bpo9" -D stretch-swh --force-distribution 'Rebuild for stretch-swh'<br />
<br />
You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.<br />
<br />
==== Setting up the repository on Phabricator ====<br />
<br />
The repository on Phabricator needs the following settings:<br />
* Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)<br />
* Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)<br />
* Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)<br />
* Policy<br />
** View: Public (no login required)<br />
** Edit: Developers<br />
** Push: All users (actual restrictions are handled by Herald rules)<br />
* Activate the repository<br />
* Look up the path to the repository on the storage tab<br />
<br />
You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes.<br />
ssh -t tate.internal.softwareheritage.org phabricator-setup-hook <repository-path> post-receive-debian-deps<br />
<br />
==== Setting up the Jenkins jobs ====<br />
<br />
The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/<br />
They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs<br />
<br />
Jobs for dependency packages are configured in <tt>jobs/dependency-packages.yaml</tt>. You can add a section as follows:<br />
<br />
- project:<br />
name: <Callsign><br />
display-name: <short-name><br />
pkg: <source-name><br />
jobs:<br />
- 'dependency-jobs-{name}'<br />
<br />
Use the regular review process to land your changes.<br />
Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.<br />
<br />
If your package needs extra repositories to build, you can add them as comma-separated values to the <tt>deb-extra-repositories</tt> setting, with the following notes:<br />
* When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.<br />
* When building packages for backports suites, the backports repository is automatically enabled.<br />
<br />
=== Local package building ===<br />
<br />
To locally test a package build, go on the appropriate debian packaging branch, and run<br />
gbp buildpackage --git-builder=sbuild -As --no-clean-source<br />
<br />
<tt>gbp buildpackage</tt> passes all options not starting with <tt>--git-</tt> to the builder. Some useful options are the following:<br />
<br />
* <tt>--git-ignore-new</tt> builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.<br />
* <tt>--no-clean-source</tt> doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies<br />
* <tt>--extra-repository="'''repository specification'''"</tt> adds the given repository in the chroot before building.<br />
* <tt>--extra-repository-key='''repository signing key'''</tt> adds the given key as a trusted gpg key for package sources<br />
* <tt>--extra-package='''<.deb file or directory>'''</tt> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.<br />
* <tt>--force-orig-source</tt> forces addition of the <tt>.orig.tar.gz</tt> file in the <tt>.changes</tt> file (useful when trying to upload a backport)<br />
<br />
See <tt>gbp help buildpackage</tt> and <tt>man sbuild</tt> for a full description of all options<br />
<br />
for example:<br />
gbp buildpackage --git-builder=sbuild -As --force-orig-source --extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main'<br />
<br />
<br />
(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping <tt>gbp buildpackage</tt> with the most common options)<br />
<br />
=== Remote package building ===<br />
<br />
Jenkins builds packages when the repository receives a tag.<br />
<br />
Once the local build succeeds, tag the package with:<br />
gbp buildpackage --git-tag-only --git-sign-tags<br />
<br />
Alternatively, you can add the <tt>--git-tag</tt> option to your <tt>gbp buildpackage</tt> command so the tag happens automatically on a successful build.<br />
<br />
Then, push your tag, and Jenkins jobs should get triggered<br />
git push --tags<br />
<br />
== Build Environment setup ==<br />
<br />
Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.<br />
<br />
=== sbuild setup ===<br />
<br />
# Install the package<br />
sudo apt-get install sbuild<br />
<br />
# Add your user to the sbuild group, to allow him to use the sbuild commands<br />
sudo sbuild-adduser $USER<br />
# You have to logout and log back in<br />
<br />
# Prepare chroots<br />
sudo mkdir /srv/chroots<br />
sudo mkdir /srv/chroots/var<br />
<br />
# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition<br />
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/<br />
sudo rm -r /var/lib/schroot<br />
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot<br />
<br />
sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/<br />
sudo rm -r /var/lib/sbuild<br />
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild<br />
# end optionally<br />
<br />
# Create unstable/sid chroot<br />
sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/<br />
<br />
# Create stretch chroot<br />
sudo sbuild-createchroot --include apt-transport-https,ca-certificates stretch /srv/chroots/stretch http://deb.debian.org/debian/<br />
<br />
<br />
# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts<br />
echo hosts >> /etc/schroot/sbuild/nssdatabases<br />
<br />
=== schroot setup ===<br />
<br />
Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.<br />
<br />
You need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:<br />
<br />
source-groups=root,sbuild<br />
source-root-groups=root,sbuild<br />
union-type=overlay<br />
<br />
This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.<br />
<br />
You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):<br />
<br />
For unstable:<br />
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild<br />
<br />
For stretch:<br />
aliases=stable-amd64-sbuild,stable-backports-amd64-sbuild,stretch-swh-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild<br />
<br />
==== dependencies cache ====<br />
<br />
Add the following line to schroot's fstab /etc/schroot/sbuild/fstab<br />
to permit reuse of existing fetched dependencies:<br />
<br />
/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0<br />
<br />
You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in <tt>/etc/apt/apt.conf.d</tt> on each chroot<br />
<br />
=== schroot update ===<br />
<br />
You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):<br />
<br />
sudo sbuild-update -udcar sid; sudo sbuild-update -udcar stretch; sudo sbuild-update -ud jessie <br />
<br />
=== environment setup ===<br />
<br />
The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt><br />
<br />
export DEBFULLNAME="Debra Hacker"<br />
export DEBEMAIL=debra.hacker@example.com<br />
<br />
Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourfullkeyid></tt> variable.<br />
(Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).<br />
<br />
=== overlay in tmpfs for faster builds ===<br />
<br />
You can add this to your fstab to put the overlay hierarchy in RAM:<br />
<br />
tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0 0 0<br />
<br />
=== Base packages ===<br />
<br />
In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.<br />
<br />
'''If you do so, do not use these chroots to upload to Debian itself!'''<br />
<br />
<br />
[[Category:Software development]]<br />
[[Category:System administration]]</div>Douarddahttps://wiki.softwareheritage.org/index.php?title=Debian_packaging&diff=1068Debian packaging2019-07-11T08:13:45Z<p>Douardda: /* Bootstrapping a dependency packaging repository */</p>
<hr />
<div>== Package repository ==<br />
<br />
A package repository is available on https://debian.softwareheritage.org/.<br />
<br />
Unstable / Testing :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ unstable main<br />
<br />
Stable / Stretch :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main<br />
<br />
This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).<br />
<br />
=== Uploading packages ===<br />
<br />
Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.<br />
<br />
For packages to be accepted, they need to be :<br />
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt><br />
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)<br />
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders<br />
<br />
<br />
== Git repositories for Debian packages ==<br />
<br />
Our git repository structure for Debian packages is compatible with <tt>git-buildpackage</tt>.<br />
<br />
We have two different ways of handling repositories for Debian packages:<br />
* Packages of python modules where *we* are upstream<br />
* Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)<br />
<br />
For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:<br />
* Our own modules are merged with the upstream repository<br />
* External dependencies ignore the upstream repository and only have packaging branches.<br />
<br />
=== Branch and tags structure ===<br />
<br />
Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:<br />
* debian/upstream (history of unpacked upstream releases)<br />
* debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, stretch-swh)<br />
* pristine-tar (data to regenerate upstream tarballs from a git export)<br />
<br />
The name of the debian/upstream branch doesn't matter ''as long as it's properly configured in the <tt>debian/gbp.conf</tt> file''. It's only really used by <tt>gbp import-orig</tt> when importing a new release.<br />
<br />
The tags marking upstream releases imported from tarballs for Debian packaging purposes are named <tt>debian/upstream/''<upstream version number>''</tt>.<br />
<br />
Our Jenkins jobs are triggered on incoming tags named <tt>debian/''<version>''</tt>. To generate the proper tags, use <tt>gbp buildpackage --git-tag-only</tt>.<br />
<br />
The git-buildpackage configuration, <tt>debian/gbp.conf</tt>, should be the following:<br />
[DEFAULT]<br />
upstream-branch=debian/upstream<br />
upstream-tag=debian/upstream/%(version)s<br />
debian-branch=debian/''<current suite>''<br />
pristine-tar=True<br />
<br />
==== Automatic packaging for swh python modules ====<br />
<br />
The <tt>swh.*</tt> python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs <tt>gbp import-orig</tt> with the tarball we release to PyPI, and the right options to merge the upstream history.<br />
<br />
To merge changes from the upstream history, we add the following option to <tt>gbp.conf</tt>:<br />
upstream-vcs-tag=v%(version)s<br />
<br />
=== Bootstrapping a dependency packaging repository ===<br />
<br />
Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:<br />
<br />
Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/<br />
wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz<br />
<br />
Create a new git repository<br />
git init pytest-postgresql<br />
cd pytest-postgresql<br />
<br />
Import the original upstream version<br />
git checkout -b debian/unstable-swh<br />
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz<br />
# What will be the source package name? [pytest-postgresql] <br />
# What is the upstream version? [1.3.4] <br />
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...<br />
# gbp:info: Source package is pytest-postgresql<br />
# gbp:info: Upstream version is 1.3.4<br />
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz<br />
<br />
Bootstrap the debian directory<br />
mkdir -p debian/source<br />
echo '3.0 (quilt)' > debian/source/format<br />
echo 9 > debian/compat<br />
cat > debian/gbp.conf << EOF<br />
[DEFAULT]<br />
upstream-branch=debian/upstream<br />
upstream-tag=debian/upstream/%(version)s<br />
debian-branch=debian/unstable-swh<br />
pristine-tar=True<br />
EOF<br />
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules<br />
vim debian/control<br />
# [...] adapt debian/control from another package<br />
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1<br />
vim debian/copyright<br />
# [...] adapt debian/copyright from another package<br />
git add debian<br />
git commit -m "Initial packaging for pytest-postgresql"<br />
<br />
You can then go on to try building the package. Once the package builds, if you want to check your package's conformance to Debian policy, you can run <tt>lintian</tt> on the changes:<br />
lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes<br />
<br />
Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository<br />
<br />
We need to use a <tt>+swh1</tt> version suffix to avoid clashing with potential upstream Debian package versions.<br />
<br />
==== Bootstrapping the backport branches ====<br />
<br />
During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.<br />
<br />
The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.<br />
<br />
Checkout the new branch<br />
git checkout debian/<version number><br />
git checkout -b debian/stretch-swh<br />
<br />
Update the gbp config to match the branch<br />
sed -i s/unstable-swh/stretch-swh/ debian/gbp.conf<br />
<br />
Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)<br />
dch -l "~bpo9" -D stretch-swh --force-distribution 'Rebuild for stretch-swh'<br />
<br />
You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.<br />
<br />
==== Setting up the repository on Phabricator ====<br />
<br />
The repository on Phabricator needs the following settings:<br />
* Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)<br />
* Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)<br />
* Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)<br />
* Policy<br />
** View: Public (no login required)<br />
** Edit: Developers<br />
** Push: All users (actual restrictions are handled by Herald rules)<br />
* Activate the repository<br />
* Look up the path to the repository on the storage tab<br />
<br />
You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes.<br />
ssh -t tate.internal.softwareheritage.org phabricator-setup-hook <repository-path> post-receive-debian-deps<br />
<br />
==== Setting up the Jenkins jobs ====<br />
<br />
The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/<br />
They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs<br />
<br />
Jobs for dependency packages are configured in <tt>jobs/dependency-packages.yaml</tt>. You can add a section as follows:<br />
<br />
- project:<br />
name: <Callsign><br />
display-name: <short-name><br />
pkg: <source-name><br />
jobs:<br />
- 'dependency-jobs-{name}'<br />
<br />
Use the regular review process to land your changes.<br />
Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.<br />
<br />
If your package needs extra repositories to build, you can add them as comma-separated values to the <tt>deb-extra-repositories</tt> setting, with the following notes:<br />
* When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.<br />
* When building packages for backports suites, the backports repository is automatically enabled.<br />
<br />
=== Local package building ===<br />
<br />
To locally test a package build, go on the appropriate debian packaging branch, and run<br />
gbp buildpackage --git-builder=sbuild -As --no-clean-source<br />
<br />
<tt>gbp buildpackage</tt> passes all options not starting with <tt>--git-</tt> to the builder. Some useful options are the following:<br />
<br />
* <tt>--git-ignore-new</tt> builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.<br />
* <tt>--no-clean-source</tt> doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies<br />
* <tt>--extra-repository="'''repository specification'''"</tt> adds the given repository in the chroot before building.<br />
* <tt>--extra-repository-key='''repository signing key'''</tt> adds the given key as a trusted gpg key for package sources<br />
* <tt>--extra-package='''<.deb file or directory>'''</tt> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.<br />
* <tt>--force-orig-source</tt> forces addition of the <tt>.orig.tar.gz</tt> file in the <tt>.changes</tt> file (useful when trying to upload a backport)<br />
<br />
See <tt>gbp help buildpackage</tt> and <tt>man sbuild</tt> for a full description of all options<br />
<br />
for example:<br />
gbp buildpackage --git-builder=sbuild -As --force-orig-source --extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main'<br />
<br />
<br />
(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping <tt>gbp buildpackage</tt> with the most common options)<br />
<br />
=== Remote package building ===<br />
<br />
Jenkins builds packages when the repository receives a tag.<br />
<br />
Once the local build succeeds, tag the package with:<br />
gbp buildpackage --git-tag-only --git-sign-tags<br />
<br />
Alternatively, you can add the <tt>--git-tag</tt> option to your <tt>gbp buildpackage</tt> command so the tag happens automatically on a successful build.<br />
<br />
Then, push your tag, and Jenkins jobs should get triggered<br />
git push --tags<br />
<br />
== Build Environment setup ==<br />
<br />
Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.<br />
<br />
=== sbuild setup ===<br />
<br />
# Install the package<br />
sudo apt-get install sbuild<br />
<br />
# Add your user to the sbuild group, to allow him to use the sbuild commands<br />
sudo sbuild-adduser $USER<br />
# You have to logout and log back in<br />
<br />
# Prepare chroots<br />
sudo mkdir /srv/chroots<br />
sudo mkdir /srv/chroots/var<br />
<br />
# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition<br />
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/<br />
sudo rm -r /var/lib/schroot<br />
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot<br />
<br />
sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/<br />
sudo rm -r /var/lib/sbuild<br />
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild<br />
# end optionally<br />
<br />
# Create unstable/sid chroot<br />
sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/<br />
<br />
# Create stretch chroot<br />
sudo sbuild-createchroot --include apt-transport-https,ca-certificates stretch /srv/chroots/stretch http://deb.debian.org/debian/<br />
<br />
<br />
# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts<br />
echo hosts >> /etc/schroot/sbuild/nssdatabases<br />
<br />
=== schroot setup ===<br />
<br />
Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.<br />
<br />
You need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:<br />
<br />
source-groups=root,sbuild<br />
source-root-groups=root,sbuild<br />
union-type=overlay<br />
<br />
This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.<br />
<br />
You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):<br />
<br />
For unstable:<br />
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild<br />
<br />
For stretch:<br />
aliases=stable-amd64-sbuild,stable-backports-amd64-sbuild,stretch-swh-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild<br />
<br />
==== dependencies cache ====<br />
<br />
Add the following line to schroot's fstab /etc/schroot/sbuild/fstab<br />
to permit reuse of existing fetched dependencies:<br />
<br />
/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0<br />
<br />
You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in <tt>/etc/apt/apt.conf.d</tt> on each chroot<br />
<br />
=== schroot update ===<br />
<br />
You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):<br />
<br />
sudo sbuild-update -udcar sid; sudo sbuild-update -udcar stretch; sudo sbuild-update -ud jessie <br />
<br />
=== environment setup ===<br />
<br />
The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt><br />
<br />
export DEBFULLNAME="Debra Hacker"<br />
export DEBEMAIL=debra.hacker@example.com<br />
<br />
Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourfullkeyid></tt> variable.<br />
(Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).<br />
<br />
=== overlay in tmpfs for faster builds ===<br />
<br />
You can add this to your fstab to put the overlay hierarchy in RAM:<br />
<br />
tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0 0 0<br />
<br />
=== Base packages ===<br />
<br />
In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.<br />
<br />
'''If you do so, do not use these chroots to upload to Debian itself!'''<br />
<br />
<br />
[[Category:Software development]]<br />
[[Category:System administration]]</div>Douarddahttps://wiki.softwareheritage.org/index.php?title=Debian_packaging&diff=1067Debian packaging2019-07-11T07:53:16Z<p>Douardda: /* Bootstrapping a dependency packaging repository */</p>
<hr />
<div>== Package repository ==<br />
<br />
A package repository is available on https://debian.softwareheritage.org/.<br />
<br />
Unstable / Testing :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ unstable main<br />
<br />
Stable / Stretch :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main<br />
<br />
This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).<br />
<br />
=== Uploading packages ===<br />
<br />
Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.<br />
<br />
For packages to be accepted, they need to be :<br />
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt><br />
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)<br />
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders<br />
<br />
<br />
== Git repositories for Debian packages ==<br />
<br />
Our git repository structure for Debian packages is compatible with <tt>git-buildpackage</tt>.<br />
<br />
We have two different ways of handling repositories for Debian packages:<br />
* Packages of python modules where *we* are upstream<br />
* Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)<br />
<br />
For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:<br />
* Our own modules are merged with the upstream repository<br />
* External dependencies ignore the upstream repository and only have packaging branches.<br />
<br />
=== Branch and tags structure ===<br />
<br />
Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:<br />
* debian/upstream (history of unpacked upstream releases)<br />
* debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, stretch-swh)<br />
* pristine-tar (data to regenerate upstream tarballs from a git export)<br />
<br />
The name of the debian/upstream branch doesn't matter ''as long as it's properly configured in the <tt>debian/gbp.conf</tt> file''. It's only really used by <tt>gbp import-orig</tt> when importing a new release.<br />
<br />
The tags marking upstream releases imported from tarballs for Debian packaging purposes are named <tt>debian/upstream/''<upstream version number>''</tt>.<br />
<br />
Our Jenkins jobs are triggered on incoming tags named <tt>debian/''<version>''</tt>. To generate the proper tags, use <tt>gbp buildpackage --git-tag-only</tt>.<br />
<br />
The git-buildpackage configuration, <tt>debian/gbp.conf</tt>, should be the following:<br />
[DEFAULT]<br />
upstream-branch=debian/upstream<br />
upstream-tag=debian/upstream/%(version)s<br />
debian-branch=debian/''<current suite>''<br />
pristine-tar=True<br />
<br />
==== Automatic packaging for swh python modules ====<br />
<br />
The <tt>swh.*</tt> python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs <tt>gbp import-orig</tt> with the tarball we release to PyPI, and the right options to merge the upstream history.<br />
<br />
To merge changes from the upstream history, we add the following option to <tt>gbp.conf</tt>:<br />
upstream-vcs-tag=v%(version)s<br />
<br />
=== Bootstrapping a dependency packaging repository ===<br />
<br />
Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:<br />
<br />
Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/<br />
wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz<br />
<br />
Create a new git repository<br />
git init pytest-postgresql<br />
cd pytest-postgresql<br />
<br />
Import the original upstream version<br />
git checkout -b debian/unstable-swh<br />
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz<br />
# What will be the source package name? [pytest-postgresql] <br />
# What is the upstream version? [1.3.4] <br />
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...<br />
# gbp:info: Source package is pytest-postgresql<br />
# gbp:info: Upstream version is 1.3.4<br />
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz<br />
<br />
Bootstrap the debian directory<br />
mkdir -p debian/source<br />
echo '3.0 (quilt)' > debian/source/format<br />
cat > debian/gbp.conf << EOF<br />
[DEFAULT]<br />
upstream-branch=debian/upstream<br />
upstream-tag=debian/upstream/%(version)s<br />
debian-branch=debian/unstable-swh<br />
pristine-tar=True<br />
EOF<br />
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules<br />
vim debian/control<br />
# [...] adapt debian/control from another package<br />
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1<br />
vim debian/copyright<br />
# [...] adapt debian/copyright from another package<br />
git add debian<br />
git commit -m "Initial packaging for pytest-postgresql"<br />
<br />
You can then go on to try building the package. Once the package builds, if you want to check your package's conformance to Debian policy, you can run <tt>lintian</tt> on the changes:<br />
lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes<br />
<br />
Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository<br />
<br />
We need to use a <tt>+swh1</tt> version suffix to avoid clashing with potential upstream Debian package versions.<br />
<br />
==== Bootstrapping the backport branches ====<br />
<br />
During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.<br />
<br />
The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.<br />
<br />
Checkout the new branch<br />
git checkout debian/<version number><br />
git checkout -b debian/stretch-swh<br />
<br />
Update the gbp config to match the branch<br />
sed -i s/unstable-swh/stretch-swh/ debian/gbp.conf<br />
<br />
Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)<br />
dch -l "~bpo9" -D stretch-swh --force-distribution 'Rebuild for stretch-swh'<br />
<br />
You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.<br />
<br />
==== Setting up the repository on Phabricator ====<br />
<br />
The repository on Phabricator needs the following settings:<br />
* Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)<br />
* Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)<br />
* Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)<br />
* Policy<br />
** View: Public (no login required)<br />
** Edit: Developers<br />
** Push: All users (actual restrictions are handled by Herald rules)<br />
* Activate the repository<br />
* Look up the path to the repository on the storage tab<br />
<br />
You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes.<br />
ssh -t tate.internal.softwareheritage.org phabricator-setup-hook <repository-path> post-receive-debian-deps<br />
<br />
==== Setting up the Jenkins jobs ====<br />
<br />
The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/<br />
They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs<br />
<br />
Jobs for dependency packages are configured in <tt>jobs/dependency-packages.yaml</tt>. You can add a section as follows:<br />
<br />
- project:<br />
name: <Callsign><br />
display-name: <short-name><br />
pkg: <source-name><br />
jobs:<br />
- 'dependency-jobs-{name}'<br />
<br />
Use the regular review process to land your changes.<br />
Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.<br />
<br />
If your package needs extra repositories to build, you can add them as comma-separated values to the <tt>deb-extra-repositories</tt> setting, with the following notes:<br />
* When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.<br />
* When building packages for backports suites, the backports repository is automatically enabled.<br />
<br />
=== Local package building ===<br />
<br />
To locally test a package build, go on the appropriate debian packaging branch, and run<br />
gbp buildpackage --git-builder=sbuild -As --no-clean-source<br />
<br />
<tt>gbp buildpackage</tt> passes all options not starting with <tt>--git-</tt> to the builder. Some useful options are the following:<br />
<br />
* <tt>--git-ignore-new</tt> builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.<br />
* <tt>--no-clean-source</tt> doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies<br />
* <tt>--extra-repository="'''repository specification'''"</tt> adds the given repository in the chroot before building.<br />
* <tt>--extra-repository-key='''repository signing key'''</tt> adds the given key as a trusted gpg key for package sources<br />
* <tt>--extra-package='''<.deb file or directory>'''</tt> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.<br />
* <tt>--force-orig-source</tt> forces addition of the <tt>.orig.tar.gz</tt> file in the <tt>.changes</tt> file (useful when trying to upload a backport)<br />
<br />
See <tt>gbp help buildpackage</tt> and <tt>man sbuild</tt> for a full description of all options<br />
<br />
for example:<br />
gbp buildpackage --git-builder=sbuild -As --force-orig-source --extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main'<br />
<br />
<br />
(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping <tt>gbp buildpackage</tt> with the most common options)<br />
<br />
=== Remote package building ===<br />
<br />
Jenkins builds packages when the repository receives a tag.<br />
<br />
Once the local build succeeds, tag the package with:<br />
gbp buildpackage --git-tag-only --git-sign-tags<br />
<br />
Alternatively, you can add the <tt>--git-tag</tt> option to your <tt>gbp buildpackage</tt> command so the tag happens automatically on a successful build.<br />
<br />
Then, push your tag, and Jenkins jobs should get triggered<br />
git push --tags<br />
<br />
== Build Environment setup ==<br />
<br />
Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.<br />
<br />
=== sbuild setup ===<br />
<br />
# Install the package<br />
sudo apt-get install sbuild<br />
<br />
# Add your user to the sbuild group, to allow him to use the sbuild commands<br />
sudo sbuild-adduser $USER<br />
# You have to logout and log back in<br />
<br />
# Prepare chroots<br />
sudo mkdir /srv/chroots<br />
sudo mkdir /srv/chroots/var<br />
<br />
# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition<br />
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/<br />
sudo rm -r /var/lib/schroot<br />
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot<br />
<br />
sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/<br />
sudo rm -r /var/lib/sbuild<br />
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild<br />
# end optionally<br />
<br />
# Create unstable/sid chroot<br />
sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/<br />
<br />
# Create stretch chroot<br />
sudo sbuild-createchroot --include apt-transport-https,ca-certificates stretch /srv/chroots/stretch http://deb.debian.org/debian/<br />
<br />
<br />
# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts<br />
echo hosts >> /etc/schroot/sbuild/nssdatabases<br />
<br />
=== schroot setup ===<br />
<br />
Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.<br />
<br />
You need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:<br />
<br />
source-groups=root,sbuild<br />
source-root-groups=root,sbuild<br />
union-type=overlay<br />
<br />
This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.<br />
<br />
You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):<br />
<br />
For unstable:<br />
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild<br />
<br />
For stretch:<br />
aliases=stable-amd64-sbuild,stable-backports-amd64-sbuild,stretch-swh-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild<br />
<br />
==== dependencies cache ====<br />
<br />
Add the following line to schroot's fstab /etc/schroot/sbuild/fstab<br />
to permit reuse of existing fetched dependencies:<br />
<br />
/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0<br />
<br />
You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in <tt>/etc/apt/apt.conf.d</tt> on each chroot<br />
<br />
=== schroot update ===<br />
<br />
You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):<br />
<br />
sudo sbuild-update -udcar sid; sudo sbuild-update -udcar stretch; sudo sbuild-update -ud jessie <br />
<br />
=== environment setup ===<br />
<br />
The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt><br />
<br />
export DEBFULLNAME="Debra Hacker"<br />
export DEBEMAIL=debra.hacker@example.com<br />
<br />
Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourfullkeyid></tt> variable.<br />
(Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).<br />
<br />
=== overlay in tmpfs for faster builds ===<br />
<br />
You can add this to your fstab to put the overlay hierarchy in RAM:<br />
<br />
tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0 0 0<br />
<br />
=== Base packages ===<br />
<br />
In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.<br />
<br />
'''If you do so, do not use these chroots to upload to Debian itself!'''<br />
<br />
<br />
[[Category:Software development]]<br />
[[Category:System administration]]</div>Douarddahttps://wiki.softwareheritage.org/index.php?title=Debian_packaging&diff=1066Debian packaging2019-07-11T07:50:44Z<p>Douardda: /* Bootstrapping a dependency packaging repository */</p>
<hr />
<div>== Package repository ==<br />
<br />
A package repository is available on https://debian.softwareheritage.org/.<br />
<br />
Unstable / Testing :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ unstable main<br />
<br />
Stable / Stretch :<br />
deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main<br />
<br />
This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).<br />
<br />
=== Uploading packages ===<br />
<br />
Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.<br />
<br />
For packages to be accepted, they need to be :<br />
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt><br />
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)<br />
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders<br />
<br />
<br />
== Git repositories for Debian packages ==<br />
<br />
Our git repository structure for Debian packages is compatible with <tt>git-buildpackage</tt>.<br />
<br />
We have two different ways of handling repositories for Debian packages:<br />
* Packages of python modules where *we* are upstream<br />
* Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)<br />
<br />
For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:<br />
* Our own modules are merged with the upstream repository<br />
* External dependencies ignore the upstream repository and only have packaging branches.<br />
<br />
=== Branch and tags structure ===<br />
<br />
Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:<br />
* debian/upstream (history of unpacked upstream releases)<br />
* debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, stretch-swh)<br />
* pristine-tar (data to regenerate upstream tarballs from a git export)<br />
<br />
The name of the debian/upstream branch doesn't matter ''as long as it's properly configured in the <tt>debian/gbp.conf</tt> file''. It's only really used by <tt>gbp import-orig</tt> when importing a new release.<br />
<br />
The tags marking upstream releases imported from tarballs for Debian packaging purposes are named <tt>debian/upstream/''<upstream version number>''</tt>.<br />
<br />
Our Jenkins jobs are triggered on incoming tags named <tt>debian/''<version>''</tt>. To generate the proper tags, use <tt>gbp buildpackage --git-tag-only</tt>.<br />
<br />
The git-buildpackage configuration, <tt>debian/gbp.conf</tt>, should be the following:<br />
[DEFAULT]<br />
upstream-branch=debian/upstream<br />
upstream-tag=debian/upstream/%(version)s<br />
debian-branch=debian/''<current suite>''<br />
pristine-tar=True<br />
<br />
==== Automatic packaging for swh python modules ====<br />
<br />
The <tt>swh.*</tt> python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs <tt>gbp import-orig</tt> with the tarball we release to PyPI, and the right options to merge the upstream history.<br />
<br />
To merge changes from the upstream history, we add the following option to <tt>gbp.conf</tt>:<br />
upstream-vcs-tag=v%(version)s<br />
<br />
=== Bootstrapping a dependency packaging repository ===<br />
<br />
Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:<br />
<br />
Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/<br />
wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz<br />
<br />
Create a new git repository<br />
git init pytest-postgresql<br />
cd pytest-postgresql<br />
<br />
Import the original upstream version<br />
git checkout -b debian/unstable-swh<br />
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz<br />
# What will be the source package name? [pytest-postgresql] <br />
# What is the upstream version? [1.3.4] <br />
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...<br />
# gbp:info: Source package is pytest-postgresql<br />
# gbp:info: Upstream version is 1.3.4<br />
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz<br />
<br />
Bootstrap the debian directory<br />
mkdir debian<br />
mkdir debian/source<br />
echo '3.0 (quilt)' > debian/source/format<br />
cat > debian/gbp.conf << EOF<br />
[DEFAULT]<br />
upstream-branch=debian/upstream<br />
upstream-tag=debian/upstream/%(version)s<br />
debian-branch=debian/unstable-swh<br />
pristine-tar=True<br />
EOF<br />
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules<br />
vim debian/control<br />
# [...] adapt debian/control from another package<br />
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1<br />
vim debian/copyright<br />
# [...] adapt debian/copyright from another package<br />
git add debian<br />
git commit -m "Initial packaging for pytest-postgresql"<br />
<br />
You can then go on to try building the package. Once the package builds, if you want to check your package's conformance to Debian policy, you can run <tt>lintian</tt> on the changes:<br />
lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes<br />
<br />
Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository<br />
<br />
We need to use a <tt>+swh1</tt> version suffix to avoid clashing with potential upstream Debian package versions.<br />
<br />
==== Bootstrapping the backport branches ====<br />
<br />
During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.<br />
<br />
The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.<br />
<br />
Checkout the new branch<br />
git checkout debian/<version number><br />
git checkout -b debian/stretch-swh<br />
<br />
Update the gbp config to match the branch<br />
sed -i s/unstable-swh/stretch-swh/ debian/gbp.conf<br />
<br />
Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)<br />
dch -l "~bpo9" -D stretch-swh --force-distribution 'Rebuild for stretch-swh'<br />
<br />
You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.<br />
<br />
==== Setting up the repository on Phabricator ====<br />
<br />
The repository on Phabricator needs the following settings:<br />
* Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)<br />
* Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)<br />
* Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)<br />
* Policy<br />
** View: Public (no login required)<br />
** Edit: Developers<br />
** Push: All users (actual restrictions are handled by Herald rules)<br />
* Activate the repository<br />
* Look up the path to the repository on the storage tab<br />
<br />
You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes.<br />
ssh -t tate.internal.softwareheritage.org phabricator-setup-hook <repository-path> post-receive-debian-deps<br />
<br />
==== Setting up the Jenkins jobs ====<br />
<br />
The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/<br />
They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs<br />
<br />
Jobs for dependency packages are configured in <tt>jobs/dependency-packages.yaml</tt>. You can add a section as follows:<br />
<br />
- project:<br />
name: <Callsign><br />
display-name: <short-name><br />
pkg: <source-name><br />
jobs:<br />
- 'dependency-jobs-{name}'<br />
<br />
Use the regular review process to land your changes.<br />
Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.<br />
<br />
If your package needs extra repositories to build, you can add them as comma-separated values to the <tt>deb-extra-repositories</tt> setting, with the following notes:<br />
* When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.<br />
* When building packages for backports suites, the backports repository is automatically enabled.<br />
<br />
=== Local package building ===<br />
<br />
To locally test a package build, go on the appropriate debian packaging branch, and run<br />
gbp buildpackage --git-builder=sbuild -As --no-clean-source<br />
<br />
<tt>gbp buildpackage</tt> passes all options not starting with <tt>--git-</tt> to the builder. Some useful options are the following:<br />
<br />
* <tt>--git-ignore-new</tt> builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.<br />
* <tt>--no-clean-source</tt> doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies<br />
* <tt>--extra-repository="'''repository specification'''"</tt> adds the given repository in the chroot before building.<br />
* <tt>--extra-repository-key='''repository signing key'''</tt> adds the given key as a trusted gpg key for package sources<br />
* <tt>--extra-package='''<.deb file or directory>'''</tt> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.<br />
* <tt>--force-orig-source</tt> forces addition of the <tt>.orig.tar.gz</tt> file in the <tt>.changes</tt> file (useful when trying to upload a backport)<br />
<br />
See <tt>gbp help buildpackage</tt> and <tt>man sbuild</tt> for a full description of all options<br />
<br />
for example:<br />
gbp buildpackage --git-builder=sbuild -As --force-orig-source --extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main'<br />
<br />
<br />
(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping <tt>gbp buildpackage</tt> with the most common options)<br />
<br />
=== Remote package building ===<br />
<br />
Jenkins builds packages when the repository receives a tag.<br />
<br />
Once the local build succeeds, tag the package with:<br />
gbp buildpackage --git-tag-only --git-sign-tags<br />
<br />
Alternatively, you can add the <tt>--git-tag</tt> option to your <tt>gbp buildpackage</tt> command so the tag happens automatically on a successful build.<br />
<br />
Then, push your tag, and Jenkins jobs should get triggered<br />
git push --tags<br />
<br />
== Build Environment setup ==<br />
<br />
Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.<br />
<br />
=== sbuild setup ===<br />
<br />
# Install the package<br />
sudo apt-get install sbuild<br />
<br />
# Add your user to the sbuild group, to allow him to use the sbuild commands<br />
sudo sbuild-adduser $USER<br />
# You have to logout and log back in<br />
<br />
# Prepare chroots<br />
sudo mkdir /srv/chroots<br />
sudo mkdir /srv/chroots/var<br />
<br />
# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition<br />
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/<br />
sudo rm -r /var/lib/schroot<br />
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot<br />
<br />
sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/<br />
sudo rm -r /var/lib/sbuild<br />
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild<br />
# end optionally<br />
<br />
# Create unstable/sid chroot<br />
sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/<br />
<br />
# Create stretch chroot<br />
sudo sbuild-createchroot --include apt-transport-https,ca-certificates stretch /srv/chroots/stretch http://deb.debian.org/debian/<br />
<br />
<br />
# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts<br />
echo hosts >> /etc/schroot/sbuild/nssdatabases<br />
<br />
=== schroot setup ===<br />
<br />
Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.<br />
<br />
You need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:<br />
<br />
source-groups=root,sbuild<br />
source-root-groups=root,sbuild<br />
union-type=overlay<br />
<br />
This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.<br />
<br />
You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):<br />
<br />
For unstable:<br />
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild<br />
<br />
For stretch:<br />
aliases=stable-amd64-sbuild,stable-backports-amd64-sbuild,stretch-swh-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild<br />
<br />
==== dependencies cache ====<br />
<br />
Add the following line to schroot's fstab /etc/schroot/sbuild/fstab<br />
to permit reuse of existing fetched dependencies:<br />
<br />
/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0<br />
<br />
You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in <tt>/etc/apt/apt.conf.d</tt> on each chroot<br />
<br />
=== schroot update ===<br />
<br />
You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):<br />
<br />
sudo sbuild-update -udcar sid; sudo sbuild-update -udcar stretch; sudo sbuild-update -ud jessie <br />
<br />
=== environment setup ===<br />
<br />
The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt><br />
<br />
export DEBFULLNAME="Debra Hacker"<br />
export DEBEMAIL=debra.hacker@example.com<br />
<br />
Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourfullkeyid></tt> variable.<br />
(Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).<br />
<br />
=== overlay in tmpfs for faster builds ===<br />
<br />
You can add this to your fstab to put the overlay hierarchy in RAM:<br />
<br />
tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0 0 0<br />
<br />
=== Base packages ===<br />
<br />
In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.<br />
<br />
'''If you do so, do not use these chroots to upload to Debian itself!'''<br />
<br />
<br />
[[Category:Software development]]<br />
[[Category:System administration]]</div>Douarddahttps://wiki.softwareheritage.org/index.php?title=Jenkins&diff=919Jenkins2018-11-27T08:43:47Z<p>Douardda: </p>
<hr />
<div>= Jenkins =<br />
<br />
The CI is run by [https://jenkins.io/ Jenkins] on jenkins.softwareheritage.org<br />
<br />
== Authentication ==<br />
<br />
Software Heritage staffers can login via the Jenkins Web user interface using the same personal *nix credentials they use to login into other machines.<br />
<br />
== Jobs ==<br />
<br />
There are 2 categories of job:<br />
<br />
- jobs related to jenkins itself, like running Jenkins Job Builder, updating and managing docker images. These are found in the [https://jenkins.softwareheritage.org/job/jenkins-tools/ jenkins-tools] directory.<br />
<br />
- jobs dedicated to testing/building swh packages.<br />
<br />
=== Jobs definition ===<br />
<br />
Most of the jobs are created using [https://docs.openstack.org/infra/jenkins-job-builder/ Jenkins Job Builder] (aka JJB).<br />
<br />
The source repository is https://forge.softwareheritage.org/source/swh-jenkins-jobs/<br />
<br />
Each time a new revision is pushed on this repo, the [https://jenkins.softwareheritage.org/job/jenkins-tools/job/swh-jenkins-job-builder/ JJB job] is executed. This job updates already existing jobs or create new ones (if any). Beware not to break the JJB job!<br />
<br />
=== Job execution environments ===<br />
<br />
Jenkins job execution may occur either on the Jenkins master node directly (typically for [https://jenkins.softwareheritage.org/job/jenkins-tools/ jenkins-tools] jobs), or in a docker jenkins slave.<br />
<br />
Docker images used by Jenkins are created and updated using the Dockerfiles in <br />
https://forge.softwareheritage.org/source/swh-jenkins-dockerfiles/<br />
<br />
For now, there are 2 different images:<br />
<br />
* swh-jenkins/tox: a Debian stretch with python3 and tox installed; used to run unit tests; it's available in Jenkins under the label [https://jenkins.softwareheritage.org/label/swh-tox/ swh-tox]<br />
* swh-jenkins/sphinx: based on the former tox image, it adds everything needed to compile the documentation with sphinx; it's available in Jenkins under the label [https://jenkins.softwareheritage.org/label/swh-sphinx/ swh-sphinx].<br />
<br />
=== swh packages ===<br />
<br />
Each swh package has a Jenkins [https://plugins.jenkins.io/cloudbees-folder Folder] in which are all the jobs dedicated to this package.<br />
<br />
There are two jenkins jobs for each swh package:<br />
<br />
* [https://jenkins.softwareheritage.org/job/DCORE/job/tox/ Phab. Diff] (here for [https://forge.softwareheritage.org/source/swh-core/ swh-core]): these jobs are executed each time a Phabricator Diff is created or updated, <br />
* [https://jenkins.softwareheritage.org/job/DCORE/job/pipeline/ master branch] (here for [https://forge.softwareheritage.org/source/swh-core/ swh-core]): these jobs are executed when git revisions are pushed on the master branch.<br />
<br />
When these jobs are started by Phabricator, the job's end status is pushed back on the Phabricator object that triggered the build (ie. the Diff object or the Revision object). As a result, the build status for a given Diff or git revision will be displayed in the forge.<br />
<br />
The Diff in Phabricator will typically look like: <br />
[[File:JenkinsDiffReport.png]]<br />
<br />
The Diffusion view of a git repository will look like:<br />
<br />
[[File:JenkinsRevisionReport.png]]<br />
<br />
where the green "checks" in each revision means the CI passed OK on this revision.<br />
<br />
=== Dashboards and metrics ===<br />
<br />
The [https://jenkins.softwareheritage.org/view/swh%20dashboard/ swh dashboard] gives a global view on the CI status of swh packages. It shows all the "master branch" job status for all swh packages, as well as a few metrics for these builds (%success, code coverage, execution time, etc.)<br />
<br />
=== Starting a job manually ===<br />
<br />
[[File:JenkinsBuildButton.png|frameless|left]] <br />
You should be able to execute any job by hand. When logged in, on the job's description page (eg. [https://jenkins.softwareheritage.org/view/swh%20dashboard/job/DCORE/job/tests/ swh-core master]) you should be able to push the "Build with Parameters" button which opens a form where you can specify the job's input parameters. <br />
<br />
[[File:JenkinsRestartFromStage.png|frameless|left]]<br />
[[File:JenkinsReplay.png|frameless|left]]<br />
Note that you can also restart a (generally failed) job. From the job execution summary (eg. [https://jenkins.softwareheritage.org/view/swh%20dashboard/job/DCORE/job/tests/50/ this execution]) you may find a "Restart from Stage" button (on "master branch" jobs only) or a "Replay" button that will allow you to recreate a job with the same parameters as the original job execution.<br />
<br />
== Phabricator ==<br />
<br />
TODO<br />
<br />
<br />
[[Category:Software development]]</div>Douarddahttps://wiki.softwareheritage.org/index.php?title=File:JenkinsDiffReport.png&diff=918File:JenkinsDiffReport.png2018-11-27T08:40:35Z<p>Douardda: </p>
<hr />
<div></div>Douarddahttps://wiki.softwareheritage.org/index.php?title=File:JenkinsRevisionReport.png&diff=917File:JenkinsRevisionReport.png2018-11-27T08:40:27Z<p>Douardda: </p>
<hr />
<div></div>Douarddahttps://wiki.softwareheritage.org/index.php?title=File:JenkinsBuildButton.png&diff=916File:JenkinsBuildButton.png2018-11-27T08:40:18Z<p>Douardda: </p>
<hr />
<div></div>Douarddahttps://wiki.softwareheritage.org/index.php?title=File:JenkinsRestartFromStage.png&diff=915File:JenkinsRestartFromStage.png2018-11-27T08:40:10Z<p>Douardda: </p>
<hr />
<div></div>Douarddahttps://wiki.softwareheritage.org/index.php?title=File:JenkinsReplay.png&diff=914File:JenkinsReplay.png2018-11-27T08:40:03Z<p>Douardda: </p>
<hr />
<div></div>Douarddahttps://wiki.softwareheritage.org/index.php?title=Phabricator_callsign_naming_convention&diff=913Phabricator callsign naming convention2018-11-06T08:35:00Z<p>Douardda: </p>
<hr />
<div>Each repository in [[Phabricator]] can be associated to a short [https://secure.phabricator.com/book/phabricator/article/diffusion/#repository-callsigns-and callsign]. You can choose one at repository creation time, or configuring it later.<br />
<br />
Here is the ''callsign naming convention'' adopted by [[Software Heritage]].<br />
<br />
=== General ===<br />
<br />
* '''1st letter group''' (1 character) denotes the macro area the repository belongs to. It must best one of:<br />
** A: non-software Annexes (e.g., raw data), including Git annexes<br />
** C: CI related repositores ([https://docs.openstack.org/infra/jenkins-job-builder/ jjb], Dockerfiles, etc.)<br />
** D: Development repositories (e.g., Python modules)<br />
** M: project Management repositories<br />
** P: Packaging repositories (e.g., Debian packages)<br />
** S: Sysadm repositories (e.g., Puppet stuff)<br />
** T: Tools and utilities (misc)<br />
** X: eXternal projects, not strictly related to [[Software Heritage]], that might one day migrate elsewhere (e.g., Postgres extensions)<br />
<br />
=== Annexes (1st letter: A) ===<br />
<br />
* 2nd letter group<br />
** AG: Git annexes<br />
<br />
* notable repositories / exceptions<br />
** AGPUB: public Git annex<br />
** AGPRV: private Git annex (Software Heritage team only)<br />
<br />
=== Continuous Integration (1st letter: C)===<br />
<br />
=== Development (1st letter: D) ===<br />
<br />
* 2nd letter group<br />
** LD: loaders<br />
** CL: cloners<br />
** LS: listers<br />
** W: web stuff<br />
<br />
* 3rd letter group: format loaders/cloners/listers act on<br />
** DIR: directories<br />
** DEB: Debian packages<br />
** G: Git<br />
** CG: Cgit<br />
** ANT: Antepedia/Antelink<br />
<br />
* postfix modifiers<br />
** T: test stuff related to the corresponding repository without trailing T (e.g., DSTOT is test stuff for DSTO)<br />
<br />
* notable repositories / exceptions<br />
** DENV: development environment<br />
** DCORE: core foundations<br />
<br />
=== Management (1st letter: M) ===<br />
<br />
* notable repositories / exceptions<br />
** MGMT: catch-all management repository<br />
** MSLD: talk slides<br />
<br />
=== Packaging (1st letter: P) ===<br />
<br />
* 2nd letter group<br />
** FK: Flask-related packages<br />
<br />
=== Sysadm (1st letter: S) ===<br />
<br />
* 2nd letter group<br />
** P: puppet stuff<br />
<br />
* notable repositories / exceptions<br />
** SPENV: puppet environment (get all the repos)<br />
** SPSITE: puppet site<br />
** SPPROF: puppet profiles<br />
** SPROLE: puppet roles<br />
<br />
=== Tools and utilities (1st letter: T) ===<br />
<br />
=== External (1st letter: X) ===<br />
<br />
=== Notable top-level repositories / exceptions ===<br />
<br />
* PWD: passwords and credentials<br />
<br />
<br />
[[Category:Software development]]<br />
[[Category:Phabricator]]</div>Douardda