Difference between revisions of "Debian packaging"

From Software Heritage Wiki
Jump to: navigation, search
m (Replace stretch reference to buster where it makes sense)
(30 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 
== Creating a package for deployment ==
 
 
<tt>swh-environment</tt> contains a script, <tt>bin/make-package</tt>, which generates a Debian package from a given Git repository, and uploads it to our internal repository targeting suites unstable and jessie-backports.
 
 
This assumes that the '''current''' (''HEAD'') revision has been tagged with a version '''v<foo>''' (e.g. v0.0.4), and that the repository contains the debian metadata. (i.e. <tt>debian/{copyright,control,changelog.rules,source/format,compat}</tt>).
 
 
Example:
 
  $ git tag -as -m 'version 0.0.42' v0.0.42
 
  $ bin/make-package -b -u swh-core
 
 
 
== Package repository ==
 
== Package repository ==
  
A package repository is available on http://debian.internal.softwareheritage.org/.
+
A package repository is available on https://debian.softwareheritage.org/.
  
 
Unstable / Testing :
 
Unstable / Testing :
 
   deb [trusted=yes] https://debian.softwareheritage.org/ unstable main
 
   deb [trusted=yes] https://debian.softwareheritage.org/ unstable main
  
Stable / Stretch :
+
Stable / Buster :
 +
  deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main
 +
 
 +
Oldstable / Stretch :
 
   deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main
 
   deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main
 
Oldstable / Jessie :
 
  deb [trusted=yes] https://debian.softwareheritage.org/ jessie main
 
  
 
This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).
 
This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).
Line 33: Line 22:
 
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)
 
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)
 
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders
 
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders
 +
 +
== Git repositories for Debian packages ==
 +
 +
Our git repository structure for Debian packages is compatible with <tt>git-buildpackage</tt>.
 +
 +
We have two different ways of handling repositories for Debian packages:
 +
* Packages of python modules where *we* are upstream
 +
* Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)
 +
 +
For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:
 +
* Our own modules are merged with the upstream repository
 +
* External dependencies ignore the upstream repository and only have packaging branches.
 +
 +
=== Branch and tags structure ===
 +
 +
Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:
 +
* debian/upstream (history of unpacked upstream releases)
 +
* debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, buster-swh)
 +
* pristine-tar (data to regenerate upstream tarballs from a git export)
 +
 +
The name of the debian/upstream branch doesn't matter ''as long as it's properly configured in the <tt>debian/gbp.conf</tt> file''. It's only really used by <tt>gbp import-orig</tt> when importing a new release.
 +
 +
The tags marking upstream releases imported from tarballs for Debian packaging purposes are named <tt>debian/upstream/''<upstream version number>''</tt>.
 +
 +
Our Jenkins jobs are triggered on incoming tags named <tt>debian/''<version>''</tt>. To generate the proper tags, use <tt>gbp buildpackage --git-tag-only</tt>.
 +
 +
The git-buildpackage configuration, <tt>debian/gbp.conf</tt>, should be the following:
 +
[DEFAULT]
 +
upstream-branch=debian/upstream
 +
upstream-tag=debian/upstream/%(version)s
 +
debian-branch=debian/''<current suite>''
 +
pristine-tar=True
 +
 +
==== Automatic packaging for swh python modules ====
 +
 +
The <tt>swh.*</tt> python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs <tt>gbp import-orig</tt> with the tarball we release to PyPI, and the right options to merge the upstream history.
 +
 +
To merge changes from the upstream history, we add the following option to <tt>gbp.conf</tt>:
 +
upstream-vcs-tag=v%(version)s
 +
 +
=== Bootstrapping a dependency packaging repository ===
 +
 +
Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:
 +
 +
Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/
 +
wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz
 +
 +
Create a new git repository
 +
git init pytest-postgresql
 +
cd pytest-postgresql
 +
 +
Import the original upstream version
 +
git checkout -b debian/unstable-swh
 +
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz
 +
# What will be the source package name? [pytest-postgresql]
 +
# What is the upstream version? [1.3.4]
 +
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...
 +
# gbp:info: Source package is pytest-postgresql
 +
# gbp:info: Upstream version is 1.3.4
 +
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz
 +
 +
Bootstrap the debian directory
 +
mkdir -p debian/source
 +
echo '3.0 (quilt)' > debian/source/format
 +
echo 9 > debian/compat
 +
cat > debian/gbp.conf << EOF
 +
[DEFAULT]
 +
upstream-branch=debian/upstream
 +
upstream-tag=debian/upstream/%(version)s
 +
debian-branch=debian/unstable-swh
 +
pristine-tar=True
 +
EOF
 +
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules
 +
vim debian/control
 +
# [...] adapt debian/control from another package
 +
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1 --distribution unstable-swh
 +
vim debian/copyright
 +
# [...] adapt debian/copyright from another package
 +
git add debian
 +
git commit -m "Initial packaging for pytest-postgresql"
 +
 +
You can then go on to try building the package.
 +
gbp buildpackage --git-builder='sbuild -As'
 +
 +
Once the package builds, if you want to check your package's conformance to Debian policy, you can run <tt>lintian</tt> on the changes:
 +
lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes
 +
 +
Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository
 +
 +
We need to use a <tt>+swh1</tt> version suffix to avoid clashing with potential upstream Debian package versions.
 +
 +
==== Bootstrapping the backport branches ====
 +
 +
During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.
 +
 +
The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.
 +
 +
Checkout the new branch
 +
git checkout debian/<version number>
 +
git checkout -b debian/buster-swh
 +
 +
Update the gbp config to match the branch
 +
sed -i s/unstable-swh/buster-swh/ debian/gbp.conf
 +
 +
Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)
 +
dch -l "~bpo10" -D buster-swh --force-distribution 'Rebuild for buster-swh'
 +
 +
You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.
 +
 +
==== Setting up the repository on Phabricator ====
 +
 +
The repository on Phabricator needs the following settings:
 +
* Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)
 +
* Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)
 +
* Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)
 +
* Policy
 +
** View: Public (no login required)
 +
** Edit: Developers
 +
** Push: All users (actual restrictions are handled by Herald rules)
 +
* Activate the repository
 +
* Look up the path to the repository on the storage tab
 +
 +
You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes
 +
ssh -p 2222 -t tate.internal.softwareheritage.org phabricator-setup-hook /srv/phabricator/repos/<repo-id> <post-receive-hook>
 +
 +
Note:
 +
 +
* there exists 2 types of <post-receive-hook>:
 +
** ''post-receive-swh-modules'' for swh modules developed by the team
 +
** ''post-receive-debian-deps'' for external modules packaged by the team
 +
* remember that access to tate is on port 2222.
 +
 +
The repo ID can be found on the repo's "storage" property page on phabricator, typically
 +
https://forge.softwareheritage.org/source/swh-SHORTNAME/manage/storage/
 +
 +
==== Setting up the Jenkins jobs ====
 +
 +
The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/
 +
They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs
 +
 +
Jobs for dependency packages are configured in <tt>jobs/dependency-packages.yaml</tt>. You can add a section as follows:
 +
 +
- project:
 +
    name: <Callsign>
 +
    display-name: <short-name>
 +
    pkg: <source-name>
 +
    python_module: <python-module>
 +
    jobs:
 +
      - 'dependency-jobs-{name}'
 +
 +
For example:
 +
  - project:
 +
      name: DLDBASE
 +
      display-name: swh-loader-core
 +
      repo_name: swh-loader-core
 +
      pkg: loader.core
 +
      python_module: swh.loader.core
 +
      jobs:
 +
        - 'swh-jobs-{name}'
 +
 +
Other samples can be found in the dedicated repository.
 +
* usual swh package: [https://forge.softwareheritage.org/source/swh-jenkins-jobs/browse/master/jobs/swh-packages.yaml$15-22 swh.core]
 +
* peculiar swh package (with name divergences): [https://forge.softwareheritage.org/source/swh-jenkins-jobs/browse/master/jobs/swh-packages.yaml$51-58 swh.icinga_plugins]
 +
 +
Use the regular review process to land your changes.
 +
Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.
 +
 +
If your package needs extra repositories to build, you can add them as comma-separated values to the <tt>deb-extra-repositories</tt> setting, with the following notes:
 +
* When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.
 +
* When building packages for backports suites, the backports repository is automatically enabled.
 +
 +
=== Updating a dependency packaging repository ===
 +
 +
Place yourself on the debian/unstable-swh branch and "gbp import-origin" a more
 +
recent upstream release tarballs.
 +
 +
For example (current version on 0.0.5, upstream bumped to 0.0.7):
 +
gbp import-origin https://files.pythonhosted.org/packages/7a/bb/cf8fec6009e7d0cec52dc179d09b28c4c70d158e79b565e8aab7606e1717/attrs-strict-0.0.7.tar.gz
 +
 +
This will update the following branches:
 +
* debian/upstream
 +
* pristine-tar
 +
* debian/unstable-swh
 +
 +
This also includes the necessary tags (`debian/upstream/0.0.7` here).
 +
 +
You then need to push all branches/tags to the repository:
 +
git push origin --all --follow-tags
 +
 +
Ensure the [https://wiki.softwareheritage.org/wiki/Debian_packaging#Local_package_building update builds fine]
 +
And [https://wiki.softwareheritage.org/wiki/Debian_packaging#Remote_package_building tags accordingly the debian/unstable-swh branch when ok].
 +
Jenkins will then keep up on building the package.
 +
 +
=== Local package building ===
 +
 +
To locally test a package build, go on the appropriate debian packaging branch, and run
 +
gbp buildpackage --git-builder=sbuild -As --no-clean-source
 +
 +
<tt>gbp buildpackage</tt> passes all options not starting with <tt>--git-</tt> to the builder. Some useful options are the following:
 +
 +
* <tt>--git-ignore-new</tt> builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.
 +
* <tt>--no-clean-source</tt> doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies
 +
* <tt>--extra-repository="'''repository specification'''"</tt> adds the given repository in the chroot before building.
 +
* <tt>--extra-repository-key='''repository signing key'''</tt> adds the given key as a trusted gpg key for package sources
 +
* <tt>--extra-package='''<.deb file or directory>'''</tt> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.
 +
* <tt>--force-orig-source</tt> forces addition of the <tt>.orig.tar.gz</tt> file in the <tt>.changes</tt> file (useful when trying to upload a backport)
 +
 +
See <tt>gbp help buildpackage</tt> and <tt>man sbuild</tt> for a full description of all options
 +
 +
for example:
 +
gbp buildpackage --git-builder=sbuild -As --no-clean-source --force-orig-source \
 +
--extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main'
 +
 +
or if you need some third-party repository, say for cassandra:
 +
gbp buildpackage --git-builder=sbuild -As --no-clean-source --force-orig-source \
 +
--extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main' \
 +
--extra-repository='deb [arch=amd64 trusted=yes] https://downloads.apache.org/cassandra/debian 40x main'
 +
 +
 +
(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping <tt>gbp buildpackage</tt> with the most common options)
 +
 +
=== Remote package building ===
 +
 +
Jenkins builds packages when the repository receives a tag.
 +
 +
Once the local build succeeds, tag the package with:
 +
gbp buildpackage --git-tag-only --git-sign-tags
 +
 +
Alternatively, you can add the <tt>--git-tag</tt> option to your <tt>gbp buildpackage</tt> command so the tag happens automatically on a successful build.
 +
 +
Then, push your tag, and Jenkins jobs should get triggered
 +
git push --tags
  
 
== Build Environment setup ==
 
== Build Environment setup ==
  
<tt>bin/make-package.sh</tt> uses sbuild for package construction. You therefore need to setup sbuild before being able to use it.
+
Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.
  
 
=== sbuild setup ===
 
=== sbuild setup ===
Line 62: Line 283:
 
   
 
   
 
  # Create unstable/sid chroot
 
  # Create unstable/sid chroot
  sudo sbuild-createchroot sid /srv/chroots/sid http://deb.debian.org/debian/
+
  sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/
 
   
 
   
  # Create stretch chroot
+
  # Create buster chroot
  sudo sbuild-createchroot stretch /srv/chroots/stretch http://deb.debian.org/debian/
+
  sudo sbuild-createchroot --include apt-transport-https,ca-certificates buster /srv/chroots/buster http://deb.debian.org/debian/
 
 
# Create jessie chroot
 
sudo sbuild-createchroot jessie /srv/chroots/jessie http://deb.debian.org/debian/
 
  
 
   
 
   
Line 78: Line 296:
 
Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.
 
Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.
  
In recent (>= 1.6) versions of schroot, you need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:
+
You need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:
  
 
  source-groups=root,sbuild
 
  source-groups=root,sbuild
 
  source-root-groups=root,sbuild
 
  source-root-groups=root,sbuild
 
  union-type=overlay
 
  union-type=overlay
 +
 +
This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.
  
 
You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):
 
You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):
Line 89: Line 309:
 
  aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild
 
  aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild
  
For stretch:
+
For buster:
  aliases=stable-amd64-sbuild,stable-backports-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild
+
  aliases=buster-swh-amd64-sbuild,buster-backports-amd64-sbuild,buster-backports-swh-amd64-sbuild
 
 
For jessie:
 
aliases=oldstable-amd64-sbuild,oldstable-backports-amd64-sbuild,jessie-backports-amd64-sbuild,jessie-backports-swh-amd64-sbuild
 
  
 
==== dependencies cache ====
 
==== dependencies cache ====
Line 100: Line 317:
 
to permit reuse of existing fetched dependencies:
 
to permit reuse of existing fetched dependencies:
  
    /var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0
+
/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0
  
=== schroot migration ===
+
You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in <tt>/etc/apt/apt.conf.d</tt> on each chroot
 
 
From October the 19th 2017, we migrated our private repository from http://debian.internal.softwareheritage.org to https://debian.softwareheritage.org.
 
So, we need to update the chroots with the right certificate.
 
 
 
For unstable:
 
schroot -c source:sid-amd64-sbuild -u root --directory / -- apt-get -y install ca-certificates apt-transport-https
 
 
 
For stretch:
 
schroot -c source:stretch-amd64-sbuild -u root --directory / -- apt-get -y install ca-certificates apt-transport-https
 
 
 
For jessie:
 
schroot -c source:jessie-amd64-sbuild -u root --directory / -- apt-get -y install ca-certificates apt-transport-https
 
  
 
=== schroot update ===
 
=== schroot update ===
  
You can update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):
+
You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):
  
   sudo sbuild-update --update --dist-upgrade sid; sudo sbuild-update -ud stretch; sudo sbuild-update -ud jessie
+
   sudo sbuild-update -udcar sid; sudo sbuild-update -udcar buster
  
 
=== environment setup ===
 
=== environment setup ===
Line 129: Line 334:
 
  export DEBEMAIL=debra.hacker@example.com
 
  export DEBEMAIL=debra.hacker@example.com
  
Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourkeyid></tt> variable.
+
Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourfullkeyid></tt> variable.
 +
(Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).
 +
 
 +
=== overlay in tmpfs for faster builds ===
 +
 
 +
You can add this to your fstab to put the overlay hierarchy in RAM:
 +
 
 +
  tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0  0  0
 +
 
 +
=== Base packages ===
 +
 
 +
In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.
 +
 
 +
'''If you do so, do not use these chroots to upload to Debian itself!'''
  
  
 
[[Category:Software development]]
 
[[Category:Software development]]
 
[[Category:System administration]]
 
[[Category:System administration]]

Revision as of 14:11, 15 March 2021

Package repository

A package repository is available on https://debian.softwareheritage.org/.

Unstable / Testing :

 deb [trusted=yes] https://debian.softwareheritage.org/ unstable main

Stable / Buster :

 deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main

Oldstable / Stretch :

 deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main

This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).

Uploading packages

Packages are added to the repository using reprepro -vb /srv/softwareheritage/repository processincoming incoming.

For packages to be accepted, they need to be :

  1. A changes file uploaded to /srv/softwareheritage/repository/incoming
  2. Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)
  3. Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders

Git repositories for Debian packages

Our git repository structure for Debian packages is compatible with git-buildpackage.

We have two different ways of handling repositories for Debian packages:

  • Packages of python modules where *we* are upstream
  • Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)

For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:

  • Our own modules are merged with the upstream repository
  • External dependencies ignore the upstream repository and only have packaging branches.

Branch and tags structure

Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:

  • debian/upstream (history of unpacked upstream releases)
  • debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, buster-swh)
  • pristine-tar (data to regenerate upstream tarballs from a git export)

The name of the debian/upstream branch doesn't matter as long as it's properly configured in the debian/gbp.conf file. It's only really used by gbp import-orig when importing a new release.

The tags marking upstream releases imported from tarballs for Debian packaging purposes are named debian/upstream/<upstream version number>.

Our Jenkins jobs are triggered on incoming tags named debian/<version>. To generate the proper tags, use gbp buildpackage --git-tag-only.

The git-buildpackage configuration, debian/gbp.conf, should be the following:

[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/<current suite>
pristine-tar=True

Automatic packaging for swh python modules

The swh.* python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs gbp import-orig with the tarball we release to PyPI, and the right options to merge the upstream history.

To merge changes from the upstream history, we add the following option to gbp.conf:

upstream-vcs-tag=v%(version)s

Bootstrapping a dependency packaging repository

Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:

Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/

wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz

Create a new git repository

git init pytest-postgresql
cd pytest-postgresql

Import the original upstream version

git checkout -b debian/unstable-swh
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz
# What will be the source package name? [pytest-postgresql] 
# What is the upstream version? [1.3.4] 
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...
# gbp:info: Source package is pytest-postgresql
# gbp:info: Upstream version is 1.3.4
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz

Bootstrap the debian directory

mkdir -p debian/source
echo '3.0 (quilt)' > debian/source/format
echo 9 > debian/compat
cat > debian/gbp.conf << EOF
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/unstable-swh
pristine-tar=True
EOF
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules
vim debian/control
# [...] adapt debian/control from another package
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1 --distribution unstable-swh
vim debian/copyright
# [...] adapt debian/copyright from another package
git add debian
git commit -m "Initial packaging for pytest-postgresql"

You can then go on to try building the package.

gbp buildpackage --git-builder='sbuild -As'

Once the package builds, if you want to check your package's conformance to Debian policy, you can run lintian on the changes:

lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes

Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository

We need to use a +swh1 version suffix to avoid clashing with potential upstream Debian package versions.

Bootstrapping the backport branches

During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.

The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.

Checkout the new branch

git checkout debian/<version number>
git checkout -b debian/buster-swh

Update the gbp config to match the branch

sed -i s/unstable-swh/buster-swh/ debian/gbp.conf

Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)

dch -l "~bpo10" -D buster-swh --force-distribution 'Rebuild for buster-swh'

You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.

Setting up the repository on Phabricator

The repository on Phabricator needs the following settings:

  • Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)
  • Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)
  • Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)
  • Policy
    • View: Public (no login required)
    • Edit: Developers
    • Push: All users (actual restrictions are handled by Herald rules)
  • Activate the repository
  • Look up the path to the repository on the storage tab

You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes

ssh -p 2222 -t tate.internal.softwareheritage.org phabricator-setup-hook /srv/phabricator/repos/<repo-id> <post-receive-hook>

Note:

  • there exists 2 types of <post-receive-hook>:
    • post-receive-swh-modules for swh modules developed by the team
    • post-receive-debian-deps for external modules packaged by the team
  • remember that access to tate is on port 2222.

The repo ID can be found on the repo's "storage" property page on phabricator, typically

https://forge.softwareheritage.org/source/swh-SHORTNAME/manage/storage/

Setting up the Jenkins jobs

The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/ They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs

Jobs for dependency packages are configured in jobs/dependency-packages.yaml. You can add a section as follows:

- project:
    name: <Callsign>
    display-name: <short-name>
    pkg: <source-name>
    python_module: <python-module>
    jobs:
      - 'dependency-jobs-{name}'

For example:

 - project:
     name: DLDBASE
     display-name: swh-loader-core
     repo_name: swh-loader-core
     pkg: loader.core
     python_module: swh.loader.core
     jobs:
       - 'swh-jobs-{name}'

Other samples can be found in the dedicated repository.

Use the regular review process to land your changes. Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.

If your package needs extra repositories to build, you can add them as comma-separated values to the deb-extra-repositories setting, with the following notes:

  • When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.
  • When building packages for backports suites, the backports repository is automatically enabled.

Updating a dependency packaging repository

Place yourself on the debian/unstable-swh branch and "gbp import-origin" a more recent upstream release tarballs.

For example (current version on 0.0.5, upstream bumped to 0.0.7):

gbp import-origin https://files.pythonhosted.org/packages/7a/bb/cf8fec6009e7d0cec52dc179d09b28c4c70d158e79b565e8aab7606e1717/attrs-strict-0.0.7.tar.gz

This will update the following branches:

  • debian/upstream
  • pristine-tar
  • debian/unstable-swh

This also includes the necessary tags (`debian/upstream/0.0.7` here).

You then need to push all branches/tags to the repository:

git push origin --all --follow-tags

Ensure the update builds fine And tags accordingly the debian/unstable-swh branch when ok. Jenkins will then keep up on building the package.

Local package building

To locally test a package build, go on the appropriate debian packaging branch, and run

gbp buildpackage --git-builder=sbuild -As --no-clean-source

gbp buildpackage passes all options not starting with --git- to the builder. Some useful options are the following:

  • --git-ignore-new builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.
  • --no-clean-source doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies
  • --extra-repository="repository specification" adds the given repository in the chroot before building.
  • --extra-repository-key=repository signing key adds the given key as a trusted gpg key for package sources
  • --extra-package=<.deb file or directory> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.
  • --force-orig-source forces addition of the .orig.tar.gz file in the .changes file (useful when trying to upload a backport)

See gbp help buildpackage and man sbuild for a full description of all options

for example:

gbp buildpackage --git-builder=sbuild -As --no-clean-source --force-orig-source \
--extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main'

or if you need some third-party repository, say for cassandra:

gbp buildpackage --git-builder=sbuild -As --no-clean-source --force-orig-source \
--extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main' \
--extra-repository='deb [arch=amd64 trusted=yes] https://downloads.apache.org/cassandra/debian 40x main'


(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping gbp buildpackage with the most common options)

Remote package building

Jenkins builds packages when the repository receives a tag.

Once the local build succeeds, tag the package with:

gbp buildpackage --git-tag-only --git-sign-tags

Alternatively, you can add the --git-tag option to your gbp buildpackage command so the tag happens automatically on a successful build.

Then, push your tag, and Jenkins jobs should get triggered

git push --tags

Build Environment setup

Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.

sbuild setup

# Install the package
sudo apt-get install sbuild

# Add your user to the sbuild group, to allow him to use the sbuild commands
sudo sbuild-adduser $USER
# You have to logout and log back in

# Prepare chroots
sudo mkdir /srv/chroots
sudo mkdir /srv/chroots/var

# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/
sudo rm -r /var/lib/schroot
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot

sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/
sudo rm -r /var/lib/sbuild
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild
# end optionally

# Create unstable/sid chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/

# Create buster chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates buster /srv/chroots/buster http://deb.debian.org/debian/


# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts
echo hosts >> /etc/schroot/sbuild/nssdatabases

schroot setup

Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.

You need to update the configuration (in /etc/schroot/chroot.d/*-sbuild-*) with the following directives:

source-groups=root,sbuild
source-root-groups=root,sbuild
union-type=overlay

This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.

You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):

For unstable:

aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild

For buster:

aliases=buster-swh-amd64-sbuild,buster-backports-amd64-sbuild,buster-backports-swh-amd64-sbuild

dependencies cache

Add the following line to schroot's fstab /etc/schroot/sbuild/fstab to permit reuse of existing fetched dependencies:

/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0

You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in /etc/apt/apt.conf.d on each chroot

schroot update

You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):

 sudo sbuild-update -udcar sid; sudo sbuild-update -udcar buster

environment setup

The Debian tools use a few variables to preset your name and email. Add this to your .<shell>rc

export DEBFULLNAME="Debra Hacker"
export DEBEMAIL=debra.hacker@example.com

Make sure this data matches an uid for your GPG key. Else, you can use the DEBSIGN_KEYID=<yourfullkeyid> variable. (Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).

overlay in tmpfs for faster builds

You can add this to your fstab to put the overlay hierarchy in RAM:

 tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0  0  0

Base packages

In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.

If you do so, do not use these chroots to upload to Debian itself!