Debian packaging

Creating a package for deployment

swh-environment contains a script, bin/make-package, which generates a Debian package from a given Git repository, and uploads it to our internal repository targeting suites unstable and jessie-backports.

This assumes that the current (HEAD) revision has been tagged with a version v<foo> (e.g. v0.0.4), and that the repository contains the debian metadata. (i.e. debian/{copyright,control,changelog.rules,source/format,compat}).

Example:

 $ git tag -as -m 'version 0.0.42' v0.0.42
 $ bin/make-package -b -u swh-core

Package repository

A package repository is available on http://debian.internal.softwareheritage.org/.

Unstable / Testing :

 deb [trusted=yes] https://debian.softwareheritage.org/ unstable main

Stable / Stretch :

 deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main

Oldstable / Jessie :

 deb [trusted=yes] https://debian.softwareheritage.org/ jessie main

This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).

Uploading packages

Packages are added to the repository using reprepro -vb /srv/softwareheritage/repository processincoming incoming.

For packages to be accepted, they need to be :

1. A changes file uploaded to /srv/softwareheritage/repository/incoming
2. Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)
3. Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders

Build Environment setup

bin/make-package.sh uses sbuild for package construction. You therefore need to setup sbuild before being able to use it.

sbuild setup

# Install the package
sudo apt-get install sbuild

# Add your user to the sbuild group, to allow him to use the sbuild commands
sudo sbuild-adduser $USER
# You have to logout and log back in

# Prepare chroots
sudo mkdir /srv/chroots
sudo mkdir /srv/chroots/var

# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/
sudo rm -r /var/lib/schroot
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot

sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/
sudo rm -r /var/lib/sbuild
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild
# end optionally

# Create unstable/sid chroot
sudo sbuild-createchroot sid /srv/chroots/sid http://deb.debian.org/debian/

# Create stretch chroot
sudo sbuild-createchroot stretch /srv/chroots/stretch http://deb.debian.org/debian/

# Create jessie chroot
sudo sbuild-createchroot jessie /srv/chroots/jessie http://deb.debian.org/debian/

# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts
echo hosts >> /etc/schroot/sbuild/nssdatabases

schroot setup

Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.

In recent (>= 1.6) versions of schroot, you need to update the configuration (in /etc/schroot/chroot.d/*-sbuild-*) with the following directives:

source-groups=root,sbuild
source-root-groups=root,sbuild
union-type=overlay

You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):

For unstable:

aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild

For stretch:

aliases=stable-amd64-sbuild,stable-backports-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild

For jessie:

aliases=oldstable-amd64-sbuild,oldstable-backports-amd64-sbuild,jessie-backports-amd64-sbuild,jessie-backports-swh-amd64-sbuild

dependencies cache

Add the following line to schroot's fstab /etc/schroot/sbuild/fstab to permit reuse of existing fetched dependencies:

   /var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0

schroot migration

From October the 19th 2017, we migrated our private repository from http://debian.internal.softwareheritage.org to https://debian.softwareheritage.org. So, we need to update the chroots with the right certificate.

For unstable:

schroot -c source:sid-amd64-sbuild -u root --directory / -- apt-get -y install ca-certificates apt-transport-https

For stretch:

schroot -c source:stretch-amd64-sbuild -u root --directory / -- apt-get -y install ca-certificates apt-transport-https

For jessie:

schroot -c source:jessie-amd64-sbuild -u root --directory / -- apt-get -y install ca-certificates apt-transport-https

schroot update

You can update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):

 sudo sbuild-update --update --dist-upgrade sid; sudo sbuild-update -ud stretch; sudo sbuild-update -ud jessie 

environment setup

The Debian tools use a few variables to preset your name and email. Add this to your .<shell>rc

export DEBFULLNAME="Debra Hacker"
export DEBEMAIL=debra.hacker@example.com

Make sure this data matches an uid for your GPG key. Else, you can use the DEBSIGN_KEYID=<yourkeyid> variable.

overlay in tmpfs for faster builds

You can add this to your fstab to put the overlay hierarchy in RAM:

 tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0  0  0

Base packages

In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.