Software Heritage Wiki - User contributions [en]

Google Summer of Code 2022

2022-04-08T15:46:36Z

Ardumont: Suggest to avoid asking permission to propose a diff

[[File:GSoCLogo.png|512px]]

== General information ==

This page is the central point of information for [[Software Heritage]] participation into the [https://summerofcode.withgoogle.com/ Google Summer of Code] program in 2022.

Google Summer of Code is a program where Google pays students stipends to work over the (northern hemisphere) summer on free software projects such as Software Heritage. Each student works with mentors from the community to complete a software project.

== I want to participate as a student ==

Great!, we are very glad for your interest in contributing to Software Heritage and we are looking forward to work together.

=== Prerequisites ===

The following prerequisites apply to all Software Heritage GSoC projects:

* [https://www.python.org Python 3] is our language of choice, you should be fluent with that language to apply
* [https://git-scm.com Git] is our version control system of choice, you should be familiar with it to apply
* basic knowledge in using a CLI
* additional prerequisites depend on the project you will work on; check project descriptions for details

=== Before you apply ===

Here are the steps you should follow before applying, to make sure you have a good grasp of what we are doing at Software Heritage and how we do it:

# Follow our [https://docs.softwareheritage.org/devel/developer-setup.html developer setup tutorial]: it will make sure you have the source code of our software stack locally available and that you can run unit tests
# Create an account on our [https://forge.softwareheritage.org development forge]
# Familiarize yourself with our [[Code review in Phabricator|code review workflow]]
# Make at least one simple change to any one of our [https://docs.softwareheritage.org/devel/ software components] and submit it as a [https://forge.softwareheritage.org/differential/ diff] for code review, following the above workflow (no need to ask for permissions). [[Easy hacks]] and [https://forge.softwareheritage.org/project/view/20/ Web UI] issues are good options for what to fix, but feel free to submit any patch you think it might be useful.

=== What to include in your application ===

Make sure that your application includes the following information:

* Describe the '''specific project''' you want to work on. What do you want to achieve? Why is it important? Why is it useful for Software Heritage? The project might be one of the project ideas that we have prepared below, or something else entirely that you want to contribute to Software Heritage. Your source code archival pet peeve, surprise us!
* Detail your '''work plan''': a brief description of how you plan to go about your project, including a list of ''deliverables'' and a ''timeline'' of when do you expect them to be available.
* Include a reference to '''the diff''' you submitted before applying (see the "Before you apply" section above).

== Ideas list ==

Below you can find a list of project ideas that are good options for a reasonably-sized GSoC project (check individual idea pages for expected duration and difficulty of each task):
<DynamicPageList>
category = Available GSoC task
ordermethod = sortkey
order = ascending
</DynamicPageList>

We also have a list of internship topics below, which you can use for ideas when applying to GSoC with us.
Expect each internship topic to require 350 hours and to be on the harder side than GSoC-specific tasks.

<DynamicPageList>
category = Available internship
ordermethod = sortkey
order = ascending
</DynamicPageList>

All project ideas above are just suggestions, don't feel obliged to pick one of them if there is nothing that fits your taste and abilities.
Feel free to propose something else that you are excited about and that contributes to improve the Software Heritage archive: we will be happy to consider it!

== Contact ==

GSoC students are encouraged to get in touch with the Software Heritage community using the standard development communication channels, and in particular our [[IRC]] channel (#swh-devel on [https://libera.chat/ Libera Chat]) and mailing list (swh-devel). Prefer public channels over contacting mentors directly.

See our [https://www.softwareheritage.org/community/developers/ development information page] for details.

== Timeline ==

See the official [https://developers.google.com/open-source/gsoc/timeline Google Summer of Code timeline].

[[Category:Google Summer of Code]]
[[Category:Google Summer of Code 2022]]

VPN

2021-10-28T12:59:19Z

Ardumont: Install redirection to the docs page

#REDIRECT [[swhdocs:sysadm/user-management/openvpn/openvpn.html]]

Puppet setup

2021-10-27T10:41:01Z

Ardumont: Install redirection to the puppet page in main docs site

#REDIRECT [[swhdocs:sysadm/puppet/reference-setup.html]]

Jenkins

2021-10-27T10:03:38Z

Ardumont: Install redirect to docs page

#REDIRECT [[swhdocs:sysadm/deployment/jenkins.html]]

Debian packaging

2021-10-27T07:55:09Z

Ardumont: Install redirection to the docs page

#REDIRECT [[swhdocs:sysadm/deployment/howto-debian-packaging.html]]

Debian packaging

2021-03-15T14:11:39Z

Ardumont: Replace stretch reference to buster where it makes sense

== Package repository ==

A package repository is available on https://debian.softwareheritage.org/.

Unstable / Testing :
deb [trusted=yes] https://debian.softwareheritage.org/ unstable main

Stable / Buster :
deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main

Oldstable / Stretch :
deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main

This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).

=== Uploading packages ===

Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.

For packages to be accepted, they need to be :
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt>
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders

== Git repositories for Debian packages ==

Our git repository structure for Debian packages is compatible with <tt>git-buildpackage</tt>.

We have two different ways of handling repositories for Debian packages:
* Packages of python modules where *we* are upstream
* Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)

For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:
* Our own modules are merged with the upstream repository
* External dependencies ignore the upstream repository and only have packaging branches.

=== Branch and tags structure ===

Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:
* debian/upstream (history of unpacked upstream releases)
* debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, buster-swh)
* pristine-tar (data to regenerate upstream tarballs from a git export)

The name of the debian/upstream branch doesn't matter ''as long as it's properly configured in the <tt>debian/gbp.conf</tt> file''. It's only really used by <tt>gbp import-orig</tt> when importing a new release.

The tags marking upstream releases imported from tarballs for Debian packaging purposes are named <tt>debian/upstream/''<upstream version number>''</tt>.

Our Jenkins jobs are triggered on incoming tags named <tt>debian/''<version>''</tt>. To generate the proper tags, use <tt>gbp buildpackage --git-tag-only</tt>.

The git-buildpackage configuration, <tt>debian/gbp.conf</tt>, should be the following:
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/''<current suite>''
pristine-tar=True

==== Automatic packaging for swh python modules ====

The <tt>swh.*</tt> python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs <tt>gbp import-orig</tt> with the tarball we release to PyPI, and the right options to merge the upstream history.

To merge changes from the upstream history, we add the following option to <tt>gbp.conf</tt>:
upstream-vcs-tag=v%(version)s

=== Bootstrapping a dependency packaging repository ===

Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:

Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/
wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz

Create a new git repository
git init pytest-postgresql
cd pytest-postgresql

Import the original upstream version
git checkout -b debian/unstable-swh
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz
# What will be the source package name? [pytest-postgresql]
# What is the upstream version? [1.3.4]
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...
# gbp:info: Source package is pytest-postgresql
# gbp:info: Upstream version is 1.3.4
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz

Bootstrap the debian directory
mkdir -p debian/source
echo '3.0 (quilt)' > debian/source/format
echo 9 > debian/compat
cat > debian/gbp.conf << EOF
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/unstable-swh
pristine-tar=True
EOF
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules
vim debian/control
# [...] adapt debian/control from another package
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1 --distribution unstable-swh
vim debian/copyright
# [...] adapt debian/copyright from another package
git add debian
git commit -m "Initial packaging for pytest-postgresql"

You can then go on to try building the package.
gbp buildpackage --git-builder='sbuild -As'

Once the package builds, if you want to check your package's conformance to Debian policy, you can run <tt>lintian</tt> on the changes:
lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes

Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository

We need to use a <tt>+swh1</tt> version suffix to avoid clashing with potential upstream Debian package versions.

==== Bootstrapping the backport branches ====

During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.

The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.

Checkout the new branch
git checkout debian/<version number>
git checkout -b debian/buster-swh

Update the gbp config to match the branch
sed -i s/unstable-swh/buster-swh/ debian/gbp.conf

Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)
dch -l "~bpo10" -D buster-swh --force-distribution 'Rebuild for buster-swh'

You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.

==== Setting up the repository on Phabricator ====

The repository on Phabricator needs the following settings:
* Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)
* Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)
* Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)
* Policy
** View: Public (no login required)
** Edit: Developers
** Push: All users (actual restrictions are handled by Herald rules)
* Activate the repository
* Look up the path to the repository on the storage tab

You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes
ssh -p 2222 -t tate.internal.softwareheritage.org phabricator-setup-hook /srv/phabricator/repos/<repo-id> <post-receive-hook>

Note:

* there exists 2 types of <post-receive-hook>:
** ''post-receive-swh-modules'' for swh modules developed by the team
** ''post-receive-debian-deps'' for external modules packaged by the team
* remember that access to tate is on port 2222.

The repo ID can be found on the repo's "storage" property page on phabricator, typically
https://forge.softwareheritage.org/source/swh-SHORTNAME/manage/storage/

==== Setting up the Jenkins jobs ====

The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/
They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs

Jobs for dependency packages are configured in <tt>jobs/dependency-packages.yaml</tt>. You can add a section as follows:

- project:
name: <Callsign>
display-name: <short-name>
pkg: <source-name>
python_module: <python-module>
jobs:
- 'dependency-jobs-{name}'

For example:
- project:
name: DLDBASE
display-name: swh-loader-core
repo_name: swh-loader-core
pkg: loader.core
python_module: swh.loader.core
jobs:
- 'swh-jobs-{name}'

Other samples can be found in the dedicated repository.
* usual swh package: [https://forge.softwareheritage.org/source/swh-jenkins-jobs/browse/master/jobs/swh-packages.yaml$15-22 swh.core]
* peculiar swh package (with name divergences): [https://forge.softwareheritage.org/source/swh-jenkins-jobs/browse/master/jobs/swh-packages.yaml$51-58 swh.icinga_plugins]

Use the regular review process to land your changes.
Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.

If your package needs extra repositories to build, you can add them as comma-separated values to the <tt>deb-extra-repositories</tt> setting, with the following notes:
* When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.
* When building packages for backports suites, the backports repository is automatically enabled.

=== Updating a dependency packaging repository ===

Place yourself on the debian/unstable-swh branch and "gbp import-origin" a more
recent upstream release tarballs.

For example (current version on 0.0.5, upstream bumped to 0.0.7):
gbp import-origin https://files.pythonhosted.org/packages/7a/bb/cf8fec6009e7d0cec52dc179d09b28c4c70d158e79b565e8aab7606e1717/attrs-strict-0.0.7.tar.gz

This will update the following branches:
* debian/upstream
* pristine-tar
* debian/unstable-swh

This also includes the necessary tags (`debian/upstream/0.0.7` here).

You then need to push all branches/tags to the repository:
git push origin --all --follow-tags

Ensure the [https://wiki.softwareheritage.org/wiki/Debian_packaging#Local_package_building update builds fine]
And [https://wiki.softwareheritage.org/wiki/Debian_packaging#Remote_package_building tags accordingly the debian/unstable-swh branch when ok].
Jenkins will then keep up on building the package.

=== Local package building ===

To locally test a package build, go on the appropriate debian packaging branch, and run
gbp buildpackage --git-builder=sbuild -As --no-clean-source

<tt>gbp buildpackage</tt> passes all options not starting with <tt>--git-</tt> to the builder. Some useful options are the following:

* <tt>--git-ignore-new</tt> builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.
* <tt>--no-clean-source</tt> doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies
* <tt>--extra-repository="'''repository specification'''"</tt> adds the given repository in the chroot before building.
* <tt>--extra-repository-key='''repository signing key'''</tt> adds the given key as a trusted gpg key for package sources
* <tt>--extra-package='''<.deb file or directory>'''</tt> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.
* <tt>--force-orig-source</tt> forces addition of the <tt>.orig.tar.gz</tt> file in the <tt>.changes</tt> file (useful when trying to upload a backport)

See <tt>gbp help buildpackage</tt> and <tt>man sbuild</tt> for a full description of all options

for example:
gbp buildpackage --git-builder=sbuild -As --no-clean-source --force-orig-source \
--extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main'

or if you need some third-party repository, say for cassandra:
gbp buildpackage --git-builder=sbuild -As --no-clean-source --force-orig-source \
--extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main' \
--extra-repository='deb [arch=amd64 trusted=yes] https://downloads.apache.org/cassandra/debian 40x main'

(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping <tt>gbp buildpackage</tt> with the most common options)

=== Remote package building ===

Jenkins builds packages when the repository receives a tag.

Once the local build succeeds, tag the package with:
gbp buildpackage --git-tag-only --git-sign-tags

Alternatively, you can add the <tt>--git-tag</tt> option to your <tt>gbp buildpackage</tt> command so the tag happens automatically on a successful build.

Then, push your tag, and Jenkins jobs should get triggered
git push --tags

== Build Environment setup ==

Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.

=== sbuild setup ===

# Install the package
sudo apt-get install sbuild

# Add your user to the sbuild group, to allow him to use the sbuild commands
sudo sbuild-adduser $USER
# You have to logout and log back in

# Prepare chroots
sudo mkdir /srv/chroots
sudo mkdir /srv/chroots/var

# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/
sudo rm -r /var/lib/schroot
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot

sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/
sudo rm -r /var/lib/sbuild
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild
# end optionally

# Create unstable/sid chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/

# Create buster chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates buster /srv/chroots/buster http://deb.debian.org/debian/

# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts
echo hosts >> /etc/schroot/sbuild/nssdatabases

=== schroot setup ===

Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.

You need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:

source-groups=root,sbuild
source-root-groups=root,sbuild
union-type=overlay

This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.

You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):

For unstable:
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild

For buster:
aliases=buster-swh-amd64-sbuild,buster-backports-amd64-sbuild,buster-backports-swh-amd64-sbuild

==== dependencies cache ====

Add the following line to schroot's fstab /etc/schroot/sbuild/fstab
to permit reuse of existing fetched dependencies:

/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0

You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in <tt>/etc/apt/apt.conf.d</tt> on each chroot

=== schroot update ===

You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):

sudo sbuild-update -udcar sid; sudo sbuild-update -udcar buster

=== environment setup ===

The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt>

export DEBFULLNAME="Debra Hacker"
export DEBEMAIL=debra.hacker@example.com

Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourfullkeyid></tt> variable.
(Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).

=== overlay in tmpfs for faster builds ===

You can add this to your fstab to put the overlay hierarchy in RAM:

tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0 0 0

=== Base packages ===

In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.

'''If you do so, do not use these chroots to upload to Debian itself!'''

[[Category:Software development]]
[[Category:System administration]]

Git style guide

2021-03-11T15:57:53Z

Ardumont: /* Link commits to tasks */

Various information about how we use Git to develop [[Software Heritage]].

== Commits ==

Make your commits adhere to this [http://chris.beams.io/posts/git-commit/ ''How to Write a Git Commit Message''] tutorial.

== Link commits to tasks ==

You can reference [[Phabricator]] tasks from your commits, using a [https://secure.phabricator.com/T5132 dedicated syntax].
When you do so, please put the task action on a separate line, so that it is clearly visible.

Make sure commits that are enough to close a bug do so using a line like:
<pre>
Closes T123456
</pre>

If you just want to "ping" a task, updating it with the fact that a related commit has been pushed, use:
<pre>
Related to T123456
</pre>

== References ==

* [https://secure.phabricator.com/T5132 special syntax you can use in commit messages to cause effects]

[[Category:Guidelines]]
[[Category:Software development]]

Debian packaging

2021-03-10T11:22:51Z

Ardumont: /* Bootstrapping a dependency packaging repository */

== Package repository ==

A package repository is available on https://debian.softwareheritage.org/.

Unstable / Testing :
deb [trusted=yes] https://debian.softwareheritage.org/ unstable main

Stable / Buster :
deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main

Oldstable / Stretch :
deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main

This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).

=== Uploading packages ===

Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.

For packages to be accepted, they need to be :
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt>
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders

== Git repositories for Debian packages ==

Our git repository structure for Debian packages is compatible with <tt>git-buildpackage</tt>.

We have two different ways of handling repositories for Debian packages:
* Packages of python modules where *we* are upstream
* Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)

For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:
* Our own modules are merged with the upstream repository
* External dependencies ignore the upstream repository and only have packaging branches.

=== Branch and tags structure ===

Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:
* debian/upstream (history of unpacked upstream releases)
* debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, stretch-swh)
* pristine-tar (data to regenerate upstream tarballs from a git export)

The name of the debian/upstream branch doesn't matter ''as long as it's properly configured in the <tt>debian/gbp.conf</tt> file''. It's only really used by <tt>gbp import-orig</tt> when importing a new release.

The tags marking upstream releases imported from tarballs for Debian packaging purposes are named <tt>debian/upstream/''<upstream version number>''</tt>.

Our Jenkins jobs are triggered on incoming tags named <tt>debian/''<version>''</tt>. To generate the proper tags, use <tt>gbp buildpackage --git-tag-only</tt>.

The git-buildpackage configuration, <tt>debian/gbp.conf</tt>, should be the following:
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/''<current suite>''
pristine-tar=True

==== Automatic packaging for swh python modules ====

The <tt>swh.*</tt> python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs <tt>gbp import-orig</tt> with the tarball we release to PyPI, and the right options to merge the upstream history.

To merge changes from the upstream history, we add the following option to <tt>gbp.conf</tt>:
upstream-vcs-tag=v%(version)s

=== Bootstrapping a dependency packaging repository ===

Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:

Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/
wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz

Create a new git repository
git init pytest-postgresql
cd pytest-postgresql

Import the original upstream version
git checkout -b debian/unstable-swh
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz
# What will be the source package name? [pytest-postgresql]
# What is the upstream version? [1.3.4]
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...
# gbp:info: Source package is pytest-postgresql
# gbp:info: Upstream version is 1.3.4
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz

Bootstrap the debian directory
mkdir -p debian/source
echo '3.0 (quilt)' > debian/source/format
echo 9 > debian/compat
cat > debian/gbp.conf << EOF
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/unstable-swh
pristine-tar=True
EOF
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules
vim debian/control
# [...] adapt debian/control from another package
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1 --distribution unstable-swh
vim debian/copyright
# [...] adapt debian/copyright from another package
git add debian
git commit -m "Initial packaging for pytest-postgresql"

You can then go on to try building the package.
gbp buildpackage --git-builder='sbuild -As'

Once the package builds, if you want to check your package's conformance to Debian policy, you can run <tt>lintian</tt> on the changes:
lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes

Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository

We need to use a <tt>+swh1</tt> version suffix to avoid clashing with potential upstream Debian package versions.

==== Bootstrapping the backport branches ====

During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.

The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.

Checkout the new branch
git checkout debian/<version number>
git checkout -b debian/stretch-swh

Update the gbp config to match the branch
sed -i s/unstable-swh/stretch-swh/ debian/gbp.conf

Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)
dch -l "~bpo9" -D stretch-swh --force-distribution 'Rebuild for stretch-swh'

You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.

==== Setting up the repository on Phabricator ====

The repository on Phabricator needs the following settings:
* Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)
* Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)
* Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)
* Policy
** View: Public (no login required)
** Edit: Developers
** Push: All users (actual restrictions are handled by Herald rules)
* Activate the repository
* Look up the path to the repository on the storage tab

You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes
ssh -p 2222 -t tate.internal.softwareheritage.org phabricator-setup-hook /srv/phabricator/repos/<repo-id> <post-receive-hook>

Note:

* there exists 2 types of <post-receive-hook>:
** ''post-receive-swh-modules'' for swh modules developed by the team
** ''post-receive-debian-deps'' for external modules packaged by the team
* remember that access to tate is on port 2222.

The repo ID can be found on the repo's "storage" property page on phabricator, typically
https://forge.softwareheritage.org/source/swh-SHORTNAME/manage/storage/

==== Setting up the Jenkins jobs ====

The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/
They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs

Jobs for dependency packages are configured in <tt>jobs/dependency-packages.yaml</tt>. You can add a section as follows:

- project:
name: <Callsign>
display-name: <short-name>
pkg: <source-name>
python_module: <python-module>
jobs:
- 'dependency-jobs-{name}'

For example:
- project:
name: DLDBASE
display-name: swh-loader-core
repo_name: swh-loader-core
pkg: loader.core
python_module: swh.loader.core
jobs:
- 'swh-jobs-{name}'

Other samples can be found in the dedicated repository.
* usual swh package: [https://forge.softwareheritage.org/source/swh-jenkins-jobs/browse/master/jobs/swh-packages.yaml$15-22 swh.core]
* peculiar swh package (with name divergences): [https://forge.softwareheritage.org/source/swh-jenkins-jobs/browse/master/jobs/swh-packages.yaml$51-58 swh.icinga_plugins]

Use the regular review process to land your changes.
Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.

If your package needs extra repositories to build, you can add them as comma-separated values to the <tt>deb-extra-repositories</tt> setting, with the following notes:
* When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.
* When building packages for backports suites, the backports repository is automatically enabled.

=== Updating a dependency packaging repository ===

Place yourself on the debian/unstable-swh branch and "gbp import-origin" a more
recent upstream release tarballs.

For example (current version on 0.0.5, upstream bumped to 0.0.7):
gbp import-origin https://files.pythonhosted.org/packages/7a/bb/cf8fec6009e7d0cec52dc179d09b28c4c70d158e79b565e8aab7606e1717/attrs-strict-0.0.7.tar.gz

This will update the following branches:
* debian/upstream
* pristine-tar
* debian/unstable-swh

This also includes the necessary tags (`debian/upstream/0.0.7` here).

You then need to push all branches/tags to the repository:
git push origin --all --follow-tags

Ensure the [https://wiki.softwareheritage.org/wiki/Debian_packaging#Local_package_building update builds fine]
And [https://wiki.softwareheritage.org/wiki/Debian_packaging#Remote_package_building tags accordingly the debian/unstable-swh branch when ok].
Jenkins will then keep up on building the package.

=== Local package building ===

To locally test a package build, go on the appropriate debian packaging branch, and run
gbp buildpackage --git-builder=sbuild -As --no-clean-source

<tt>gbp buildpackage</tt> passes all options not starting with <tt>--git-</tt> to the builder. Some useful options are the following:

* <tt>--git-ignore-new</tt> builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.
* <tt>--no-clean-source</tt> doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies
* <tt>--extra-repository="'''repository specification'''"</tt> adds the given repository in the chroot before building.
* <tt>--extra-repository-key='''repository signing key'''</tt> adds the given key as a trusted gpg key for package sources
* <tt>--extra-package='''<.deb file or directory>'''</tt> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.
* <tt>--force-orig-source</tt> forces addition of the <tt>.orig.tar.gz</tt> file in the <tt>.changes</tt> file (useful when trying to upload a backport)

See <tt>gbp help buildpackage</tt> and <tt>man sbuild</tt> for a full description of all options

for example:
gbp buildpackage --git-builder=sbuild -As --no-clean-source --force-orig-source \
--extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main'

or if you need some third-party repository, say for cassandra:
gbp buildpackage --git-builder=sbuild -As --no-clean-source --force-orig-source \
--extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main' \
--extra-repository='deb [arch=amd64 trusted=yes] https://downloads.apache.org/cassandra/debian 40x main'

(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping <tt>gbp buildpackage</tt> with the most common options)

=== Remote package building ===

Jenkins builds packages when the repository receives a tag.

Once the local build succeeds, tag the package with:
gbp buildpackage --git-tag-only --git-sign-tags

Alternatively, you can add the <tt>--git-tag</tt> option to your <tt>gbp buildpackage</tt> command so the tag happens automatically on a successful build.

Then, push your tag, and Jenkins jobs should get triggered
git push --tags

== Build Environment setup ==

Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.

=== sbuild setup ===

# Install the package
sudo apt-get install sbuild

# Add your user to the sbuild group, to allow him to use the sbuild commands
sudo sbuild-adduser $USER
# You have to logout and log back in

# Prepare chroots
sudo mkdir /srv/chroots
sudo mkdir /srv/chroots/var

# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/
sudo rm -r /var/lib/schroot
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot

sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/
sudo rm -r /var/lib/sbuild
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild
# end optionally

# Create unstable/sid chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/

# Create stretch chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates stretch /srv/chroots/stretch http://deb.debian.org/debian/

# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts
echo hosts >> /etc/schroot/sbuild/nssdatabases

=== schroot setup ===

Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.

You need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:

source-groups=root,sbuild
source-root-groups=root,sbuild
union-type=overlay

This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.

You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):

For unstable:
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild

For stretch:
aliases=stretch-swh-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild

==== dependencies cache ====

Add the following line to schroot's fstab /etc/schroot/sbuild/fstab
to permit reuse of existing fetched dependencies:

/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0

You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in <tt>/etc/apt/apt.conf.d</tt> on each chroot

=== schroot update ===

You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):

sudo sbuild-update -udcar sid; sudo sbuild-update -udcar stretch; sudo sbuild-update -ud jessie

=== environment setup ===

The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt>

export DEBFULLNAME="Debra Hacker"
export DEBEMAIL=debra.hacker@example.com

Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourfullkeyid></tt> variable.
(Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).

=== overlay in tmpfs for faster builds ===

You can add this to your fstab to put the overlay hierarchy in RAM:

tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0 0 0

=== Base packages ===

In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.

'''If you do so, do not use these chroots to upload to Debian itself!'''

[[Category:Software development]]
[[Category:System administration]]

Debian packaging

2021-03-10T11:22:12Z

Ardumont: /* Bootstrapping a dependency packaging repository */

== Package repository ==

A package repository is available on https://debian.softwareheritage.org/.

Unstable / Testing :
deb [trusted=yes] https://debian.softwareheritage.org/ unstable main

Stable / Buster :
deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main

Oldstable / Stretch :
deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main

This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).

=== Uploading packages ===

Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.

For packages to be accepted, they need to be :
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt>
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders

== Git repositories for Debian packages ==

Our git repository structure for Debian packages is compatible with <tt>git-buildpackage</tt>.

We have two different ways of handling repositories for Debian packages:
* Packages of python modules where *we* are upstream
* Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)

For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:
* Our own modules are merged with the upstream repository
* External dependencies ignore the upstream repository and only have packaging branches.

=== Branch and tags structure ===

Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:
* debian/upstream (history of unpacked upstream releases)
* debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, stretch-swh)
* pristine-tar (data to regenerate upstream tarballs from a git export)

The name of the debian/upstream branch doesn't matter ''as long as it's properly configured in the <tt>debian/gbp.conf</tt> file''. It's only really used by <tt>gbp import-orig</tt> when importing a new release.

The tags marking upstream releases imported from tarballs for Debian packaging purposes are named <tt>debian/upstream/''<upstream version number>''</tt>.

Our Jenkins jobs are triggered on incoming tags named <tt>debian/''<version>''</tt>. To generate the proper tags, use <tt>gbp buildpackage --git-tag-only</tt>.

The git-buildpackage configuration, <tt>debian/gbp.conf</tt>, should be the following:
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/''<current suite>''
pristine-tar=True

==== Automatic packaging for swh python modules ====

The <tt>swh.*</tt> python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs <tt>gbp import-orig</tt> with the tarball we release to PyPI, and the right options to merge the upstream history.

To merge changes from the upstream history, we add the following option to <tt>gbp.conf</tt>:
upstream-vcs-tag=v%(version)s

=== Bootstrapping a dependency packaging repository ===

Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:

Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/
wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz

Create a new git repository
git init pytest-postgresql
cd pytest-postgresql

Import the original upstream version
git checkout -b debian/unstable-swh
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz
# What will be the source package name? [pytest-postgresql]
# What is the upstream version? [1.3.4]
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...
# gbp:info: Source package is pytest-postgresql
# gbp:info: Upstream version is 1.3.4
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz

Bootstrap the debian directory
mkdir -p debian/source
echo '3.0 (quilt)' > debian/source/format
echo 9 > debian/compat
cat > debian/gbp.conf << EOF
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/unstable-swh
pristine-tar=True
EOF
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules
vim debian/control
# [...] adapt debian/control from another package
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1 --distribution unstable-swh
vim debian/copyright
# [...] adapt debian/copyright from another package
git add debian
git commit -m "Initial packaging for pytest-postgresql"

You can then go on to try building the package.

gbp buildpackage --git-builder='sbuild -As'

Once the package builds, if you want to check your package's conformance to Debian policy, you can run <tt>lintian</tt> on the changes:
lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes

Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository

We need to use a <tt>+swh1</tt> version suffix to avoid clashing with potential upstream Debian package versions.

==== Bootstrapping the backport branches ====

During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.

The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.

Checkout the new branch
git checkout debian/<version number>
git checkout -b debian/stretch-swh

Update the gbp config to match the branch
sed -i s/unstable-swh/stretch-swh/ debian/gbp.conf

Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)
dch -l "~bpo9" -D stretch-swh --force-distribution 'Rebuild for stretch-swh'

You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.

==== Setting up the repository on Phabricator ====

The repository on Phabricator needs the following settings:
* Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)
* Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)
* Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)
* Policy
** View: Public (no login required)
** Edit: Developers
** Push: All users (actual restrictions are handled by Herald rules)
* Activate the repository
* Look up the path to the repository on the storage tab

You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes
ssh -p 2222 -t tate.internal.softwareheritage.org phabricator-setup-hook /srv/phabricator/repos/<repo-id> <post-receive-hook>

Note:

* there exists 2 types of <post-receive-hook>:
** ''post-receive-swh-modules'' for swh modules developed by the team
** ''post-receive-debian-deps'' for external modules packaged by the team
* remember that access to tate is on port 2222.

The repo ID can be found on the repo's "storage" property page on phabricator, typically
https://forge.softwareheritage.org/source/swh-SHORTNAME/manage/storage/

==== Setting up the Jenkins jobs ====

The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/
They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs

Jobs for dependency packages are configured in <tt>jobs/dependency-packages.yaml</tt>. You can add a section as follows:

- project:
name: <Callsign>
display-name: <short-name>
pkg: <source-name>
python_module: <python-module>
jobs:
- 'dependency-jobs-{name}'

For example:
- project:
name: DLDBASE
display-name: swh-loader-core
repo_name: swh-loader-core
pkg: loader.core
python_module: swh.loader.core
jobs:
- 'swh-jobs-{name}'

Other samples can be found in the dedicated repository.
* usual swh package: [https://forge.softwareheritage.org/source/swh-jenkins-jobs/browse/master/jobs/swh-packages.yaml$15-22 swh.core]
* peculiar swh package (with name divergences): [https://forge.softwareheritage.org/source/swh-jenkins-jobs/browse/master/jobs/swh-packages.yaml$51-58 swh.icinga_plugins]

Use the regular review process to land your changes.
Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.

If your package needs extra repositories to build, you can add them as comma-separated values to the <tt>deb-extra-repositories</tt> setting, with the following notes:
* When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.
* When building packages for backports suites, the backports repository is automatically enabled.

=== Updating a dependency packaging repository ===

Place yourself on the debian/unstable-swh branch and "gbp import-origin" a more
recent upstream release tarballs.

For example (current version on 0.0.5, upstream bumped to 0.0.7):
gbp import-origin https://files.pythonhosted.org/packages/7a/bb/cf8fec6009e7d0cec52dc179d09b28c4c70d158e79b565e8aab7606e1717/attrs-strict-0.0.7.tar.gz

This will update the following branches:
* debian/upstream
* pristine-tar
* debian/unstable-swh

This also includes the necessary tags (`debian/upstream/0.0.7` here).

You then need to push all branches/tags to the repository:
git push origin --all --follow-tags

Ensure the [https://wiki.softwareheritage.org/wiki/Debian_packaging#Local_package_building update builds fine]
And [https://wiki.softwareheritage.org/wiki/Debian_packaging#Remote_package_building tags accordingly the debian/unstable-swh branch when ok].
Jenkins will then keep up on building the package.

=== Local package building ===

To locally test a package build, go on the appropriate debian packaging branch, and run
gbp buildpackage --git-builder=sbuild -As --no-clean-source

<tt>gbp buildpackage</tt> passes all options not starting with <tt>--git-</tt> to the builder. Some useful options are the following:

* <tt>--git-ignore-new</tt> builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.
* <tt>--no-clean-source</tt> doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies
* <tt>--extra-repository="'''repository specification'''"</tt> adds the given repository in the chroot before building.
* <tt>--extra-repository-key='''repository signing key'''</tt> adds the given key as a trusted gpg key for package sources
* <tt>--extra-package='''<.deb file or directory>'''</tt> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.
* <tt>--force-orig-source</tt> forces addition of the <tt>.orig.tar.gz</tt> file in the <tt>.changes</tt> file (useful when trying to upload a backport)

See <tt>gbp help buildpackage</tt> and <tt>man sbuild</tt> for a full description of all options

for example:
gbp buildpackage --git-builder=sbuild -As --no-clean-source --force-orig-source \
--extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main'

or if you need some third-party repository, say for cassandra:
gbp buildpackage --git-builder=sbuild -As --no-clean-source --force-orig-source \
--extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main' \
--extra-repository='deb [arch=amd64 trusted=yes] https://downloads.apache.org/cassandra/debian 40x main'

(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping <tt>gbp buildpackage</tt> with the most common options)

=== Remote package building ===

Jenkins builds packages when the repository receives a tag.

Once the local build succeeds, tag the package with:
gbp buildpackage --git-tag-only --git-sign-tags

Alternatively, you can add the <tt>--git-tag</tt> option to your <tt>gbp buildpackage</tt> command so the tag happens automatically on a successful build.

Then, push your tag, and Jenkins jobs should get triggered
git push --tags

== Build Environment setup ==

Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.

=== sbuild setup ===

# Install the package
sudo apt-get install sbuild

# Add your user to the sbuild group, to allow him to use the sbuild commands
sudo sbuild-adduser $USER
# You have to logout and log back in

# Prepare chroots
sudo mkdir /srv/chroots
sudo mkdir /srv/chroots/var

# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/
sudo rm -r /var/lib/schroot
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot

sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/
sudo rm -r /var/lib/sbuild
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild
# end optionally

# Create unstable/sid chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/

# Create stretch chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates stretch /srv/chroots/stretch http://deb.debian.org/debian/

# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts
echo hosts >> /etc/schroot/sbuild/nssdatabases

=== schroot setup ===

Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.

You need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:

source-groups=root,sbuild
source-root-groups=root,sbuild
union-type=overlay

This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.

You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):

For unstable:
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild

For stretch:
aliases=stretch-swh-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild

==== dependencies cache ====

Add the following line to schroot's fstab /etc/schroot/sbuild/fstab
to permit reuse of existing fetched dependencies:

/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0

You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in <tt>/etc/apt/apt.conf.d</tt> on each chroot

=== schroot update ===

You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):

sudo sbuild-update -udcar sid; sudo sbuild-update -udcar stretch; sudo sbuild-update -ud jessie

=== environment setup ===

The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt>

export DEBFULLNAME="Debra Hacker"
export DEBEMAIL=debra.hacker@example.com

Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourfullkeyid></tt> variable.
(Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).

=== overlay in tmpfs for faster builds ===

You can add this to your fstab to put the overlay hierarchy in RAM:

tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0 0 0

=== Base packages ===

In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.

'''If you do so, do not use these chroots to upload to Debian itself!'''

[[Category:Software development]]
[[Category:System administration]]

Phabricator

2021-02-03T08:05:18Z

Ardumont: Add admin action on spammy tickets/users

[http://phabricator.org/ Phabricator] is the tool that [[Software Heritage]] uses as its coding/collaboration forge.

Software Heritage's Phabricator instance can be found at https://forge.softwareheritage.org/

== Client configuration ==

* [[Arcanist setup]]

== Conventions ==

* [[Phabricator callsign naming convention]]

== Admin ==

* Clean up spam task

phabricator@tate:~$ ~/phabricator/bin/remove destroy T1234

* Clean up spam user

phabricator@tate:~$ ~/phabricator/bin/remove destroy @shorrlouis701

== Links ==

* [https://forge.softwareheritage.org/ SWH Phabricator home]
* [https://www.mediawiki.org/wiki/Phabricator how Wikimedia uses Phabricator], with lots of good tips and mini-tutorials

[[Category:Infrastructure]]
[[Category:Phabricator]]

VPN

2020-12-17T13:58:18Z

Ardumont: /* Raw OpenVPN */

The [[Software Heritage]] server and the VMs running on it are severely firewalled.
To get onto their network unrestricted, a VPN based on [https://openvpn.net/ OpenVPN] is available.

The setup is client-server, with per-client certificates.

== OpenVPN client configuration ==

=== Raw OpenVPN ===

Sample configuration file, e.g., /etc/openvpn/swh.conf:

<pre>
remote louvre.softwareheritage.org
ns-cert-type server
comp-lzo
nobind
dev tun
proto udp
port 1194
log /var/log/openvpn.log
up-restart
persist-key
persist-tun
client
ca /etc/openvpn/keys/softwareheritage-ca.crt
cert /etc/openvpn/keys/softwareheritage.crt
key /etc/openvpn/keys/softwareheritage.key
user nobody
group nogroup

# If you are using resolvconf, add this:
# Make sure you add louvre to /etc/hosts to avoid issues in using the vpn-provided DNS server.
script-security 2
up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf

# If you want the connection to persist when your network fails, add this:
ping-restart 10
</pre>

In addition to the above configuration file, you will need to install the following 3 files under /etc/openvpn/keys (matching the paths within the sample above):

* '''[[softwareheritage-ca.crt]]''': ''public'' certificate for the Software Heritage certification authority (CA)
* '''[https://wiki.softwareheritage.org/index.php?title=VPN#For_admins softwareheritage.crt]''': ''public'', client-specific (certificate signed by the admin, see below)
* '''[https://wiki.softwareheritage.org/wiki/VPN#For_users softwareheritage.key]''': ''private'', client-specific key (generated by the user, see below)

Activate the openvpn server

as root, run

systemctl enable openvpn@swh.service
systemctl start openvpn@swh.service
systemctl status openvpn@swh.service

Note: Internally, the `swh` must match the /etc/openvpn/swh.conf filename.

Excerpt of a successful start:

root@machine:~# systemctl status openvpn@swh.service
openvpn@swh.service - OpenVPN connection to swh
Loaded: loaded (/lib/systemd/system/openvpn@.service; indirect; vendor preset: enabled)
Active: active (running) since Thu 2020-12-17 19:03:29 IST; 22min ago
Docs: man:openvpn(8)
https://community.openvpn.net/openvpn/wiki/Openvpn24ManPage
https://community.openvpn.net/openvpn/wiki/HOWTO
Main PID: 12302 (openvpn)
Status: "Initialization Sequence Completed"
Tasks: 1 (limit: 4915)
CGroup: /system.slice/system-openvpn.slice/openvpn@swh.service
└─12302 /usr/sbin/openvpn --daemon ovpn-swh --status /run/openvpn/swh.status 10 --cd /etc/openvpn --script-security 2 --config /etc/openvpn/swh.conf --writepid /run/openvpn/swh.pid

Dec 17 19:03:29 machine systemd[1]: Starting OpenVPN connection to swh...
Dec 17 19:03:29 machine systemd[1]: Started OpenVPN connection to swh.

=== Network Manager GUI ===

You need network-manager-openvpn and network-manager-openvpn-gnome for the configuration gui.

[[File:nm_openvpn_base.png]]
[[File:nm_openvpn_routes.png]]
[[File:nm_openvpn_advanced_general.png]]
[[File:nm_openvpn_advanced_security.png]]
[[File:nm_openvpn_advanced_tls_auth.png]]

== Obtaining a client certificate ==

=== For users ===

Generate a keypair (key + certificate signing request) using the following command:

<pre>
openssl req -new -newkey rsa:2048 -nodes -keyout openvpn.key -out openvpn.csr -subj "/CN=<your username>"
</pre>

Please replace <your username> with something that uniquely identifies the certificate.

Make sure openvpn.key is stored in a safe place (it's your private key, which will allow anyone to connect to the VPN).

Provide the CSR file to a sysadmin through a reasonably authenticated medium.

=== For admins ===

On louvre:

Fetch the CSR file provided by the user, for instance with <tt>scp USERNAME.csr louvre:</tt>

Then, as root on louvre:

<pre>
root@louvre:~# cd /etc/openvpn/keys
root@louvre:/etc/openvpn/keys# ./easyrsa import-req ~ADMIN/USERNAME.csr USERNAME
root@louvre:/etc/openvpn/keys# ./easyrsa sign-req client USERNAME
</pre>

The first command imports the csr into the EasyRSA PKI. The second command lets you review and sign it.

Send the signed certificate, <tt>/etc/openvpn/keys/pki/issued/USERNAME.crt</tt>, to the user. That file only contains public key material.

Add the DNS entry for the new host to hiera and do a puppet run on pergamon.

== Revoking a client certificate ==

On louvre:

<pre>
root@louvre:~# cd /etc/openvpn/keys
root@louvre:/etc/openvpn/keys# ./easyrsa revoke USERNAME
[ say yes ]
root@louvre:/etc/openvpn/keys# ./easyrsa gen-crl; chmod a+r pki/crl.pem
</pre>

OpenVPN re-reads the CRL at each connection (which is why we need the CRL to be world-readable), so once the cert is revoked, there's nothing more to do. If you want to make sure the client is disconnected, you need to restart OpenVPN (which will make all clients reconnect).

== /etc/hosts entries ==

Once the Vpn is setup on your machine, you can access Software Heritage hosts via their private IP addresses; see [[Network configuration]].

OpenVPN now pushes the address of our DNS server (192.168.100.29, pergamon).

You might want to add louvre.softwareheritage.org in your /etc/hosts to avoid a bootstrap problem if the "on-vpn" DNS server is in your resolv.conf.

[[Category:Infrastructure]]
[[Category:System administration]]

VPN

2020-12-17T13:23:48Z

Ardumont: Clarify raw openvpn steps to enable the service

The [[Software Heritage]] server and the VMs running on it are severely firewalled.
To get onto their network unrestricted, a VPN based on [https://openvpn.net/ OpenVPN] is available.

The setup is client-server, with per-client certificates.

== OpenVPN client configuration ==

=== Raw OpenVPN ===

Sample configuration file, e.g., /etc/openvpn/swh.conf:

<pre>
remote louvre.softwareheritage.org
ns-cert-type server
comp-lzo
nobind
dev tun
proto udp
port 1194
log /var/log/openvpn.log
up-restart
persist-key
persist-tun
client
ca /etc/openvpn/keys/softwareheritage-ca.crt
cert /etc/openvpn/keys/softwareheritage.crt
key /etc/openvpn/keys/softwareheritage.key
user nobody
group nogroup

# If you are using resolvconf, add this:
# Make sure you add louvre to /etc/hosts to avoid issues in using the vpn-provided DNS server.
script-security 2
up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf

# If you want the connection to persist when your network fails, add this:
ping-restart 10
</pre>

In addition to the above configuration file, you will need to install the following 3 files under /etc/openvpn/keys (matching the paths within the sample above):

* '''[[softwareheritage-ca.crt]]''': ''public'' certificate for the Software Heritage certification authority (CA)
* '''[https://wiki.softwareheritage.org/index.php?title=VPN#For_admins softwareheritage.crt]''': ''public'', client-specific (certificate signed by the admin, see below)
* '''[https://wiki.softwareheritage.org/wiki/VPN#For_users softwareheritage.key]''': ''private'', client-specific key (generated by the user, see below)

Activate the openvpn server

as root, run

systemctl enable openvpn@swh.service

Note: Internally, the `swh` must match the /etc/openvpn/swh.conf filename.

=== Network Manager GUI ===

You need network-manager-openvpn and network-manager-openvpn-gnome for the configuration gui.

[[File:nm_openvpn_base.png]]
[[File:nm_openvpn_routes.png]]
[[File:nm_openvpn_advanced_general.png]]
[[File:nm_openvpn_advanced_security.png]]
[[File:nm_openvpn_advanced_tls_auth.png]]

== Obtaining a client certificate ==

=== For users ===

Generate a keypair (key + certificate signing request) using the following command:

<pre>
openssl req -new -newkey rsa:2048 -nodes -keyout openvpn.key -out openvpn.csr -subj "/CN=<your username>"
</pre>

Please replace <your username> with something that uniquely identifies the certificate.

Make sure openvpn.key is stored in a safe place (it's your private key, which will allow anyone to connect to the VPN).

Provide the CSR file to a sysadmin through a reasonably authenticated medium.

=== For admins ===

On louvre:

Fetch the CSR file provided by the user, for instance with <tt>scp USERNAME.csr louvre:</tt>

Then, as root on louvre:

<pre>
root@louvre:~# cd /etc/openvpn/keys
root@louvre:/etc/openvpn/keys# ./easyrsa import-req ~ADMIN/USERNAME.csr USERNAME
root@louvre:/etc/openvpn/keys# ./easyrsa sign-req client USERNAME
</pre>

The first command imports the csr into the EasyRSA PKI. The second command lets you review and sign it.

Send the signed certificate, <tt>/etc/openvpn/keys/pki/issued/USERNAME.crt</tt>, to the user. That file only contains public key material.

Add the DNS entry for the new host to hiera and do a puppet run on pergamon.

== Revoking a client certificate ==

On louvre:

<pre>
root@louvre:~# cd /etc/openvpn/keys
root@louvre:/etc/openvpn/keys# ./easyrsa revoke USERNAME
[ say yes ]
root@louvre:/etc/openvpn/keys# ./easyrsa gen-crl; chmod a+r pki/crl.pem
</pre>

OpenVPN re-reads the CRL at each connection (which is why we need the CRL to be world-readable), so once the cert is revoked, there's nothing more to do. If you want to make sure the client is disconnected, you need to restart OpenVPN (which will make all clients reconnect).

== /etc/hosts entries ==

Once the Vpn is setup on your machine, you can access Software Heritage hosts via their private IP addresses; see [[Network configuration]].

OpenVPN now pushes the address of our DNS server (192.168.100.29, pergamon).

You might want to add louvre.softwareheritage.org in your /etc/hosts to avoid a bootstrap problem if the "on-vpn" DNS server is in your resolv.conf.

[[Category:Infrastructure]]
[[Category:System administration]]

VPN

2020-12-17T13:18:22Z

Ardumont: Internal links to clarify what's what

The [[Software Heritage]] server and the VMs running on it are severely firewalled.
To get onto their network unrestricted, a VPN based on [https://openvpn.net/ OpenVPN] is available.

The setup is client-server, with per-client certificates.

== OpenVPN client configuration ==

=== Raw OpenVPN ===

Sample configuration file, e.g., /etc/openvpn/softwareheritage.conf:

<pre>
remote louvre.softwareheritage.org
ns-cert-type server
comp-lzo
nobind
dev tun
proto udp
port 1194
log /var/log/openvpn.log
up-restart
persist-key
persist-tun
client
ca /etc/openvpn/keys/softwareheritage-ca.crt
cert /etc/openvpn/keys/softwareheritage.crt
key /etc/openvpn/keys/softwareheritage.key
user nobody
group nogroup

# If you are using resolvconf, add this:
# Make sure you add louvre to /etc/hosts to avoid issues in using the vpn-provided DNS server.
script-security 2
up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf

# If you want the connection to persist when your network fails, add this:
ping-restart 10
</pre>

In addition to the above configuration file, you will need to install the following 3 files under /etc/openvpn/keys:

* '''[[softwareheritage-ca.crt]]''': ''public'' certificate for the Software Heritage certification authority (CA)
* '''[https://wiki.softwareheritage.org/index.php?title=VPN#For_admins softwareheritage.crt]''': ''public'', client-specific (certificate signed by the admin, see below)
* '''[https://wiki.softwareheritage.org/wiki/VPN#For_users softwareheritage.key]''': ''private'', client-specific key (generated by the user, see below)

=== Network Manager GUI ===

You need network-manager-openvpn and network-manager-openvpn-gnome for the configuration gui.

[[File:nm_openvpn_base.png]]
[[File:nm_openvpn_routes.png]]
[[File:nm_openvpn_advanced_general.png]]
[[File:nm_openvpn_advanced_security.png]]
[[File:nm_openvpn_advanced_tls_auth.png]]

== Obtaining a client certificate ==

=== For users ===

Generate a keypair (key + certificate signing request) using the following command:

<pre>
openssl req -new -newkey rsa:2048 -nodes -keyout openvpn.key -out openvpn.csr -subj "/CN=<your username>"
</pre>

Please replace <your username> with something that uniquely identifies the certificate.

Make sure openvpn.key is stored in a safe place (it's your private key, which will allow anyone to connect to the VPN).

Provide the CSR file to a sysadmin through a reasonably authenticated medium.

=== For admins ===

On louvre:

Fetch the CSR file provided by the user, for instance with <tt>scp USERNAME.csr louvre:</tt>

Then, as root on louvre:

<pre>
root@louvre:~# cd /etc/openvpn/keys
root@louvre:/etc/openvpn/keys# ./easyrsa import-req ~ADMIN/USERNAME.csr USERNAME
root@louvre:/etc/openvpn/keys# ./easyrsa sign-req client USERNAME
</pre>

The first command imports the csr into the EasyRSA PKI. The second command lets you review and sign it.

Send the signed certificate, <tt>/etc/openvpn/keys/pki/issued/USERNAME.crt</tt>, to the user. That file only contains public key material.

Add the DNS entry for the new host to hiera and do a puppet run on pergamon.

== Revoking a client certificate ==

On louvre:

<pre>
root@louvre:~# cd /etc/openvpn/keys
root@louvre:/etc/openvpn/keys# ./easyrsa revoke USERNAME
[ say yes ]
root@louvre:/etc/openvpn/keys# ./easyrsa gen-crl; chmod a+r pki/crl.pem
</pre>

OpenVPN re-reads the CRL at each connection (which is why we need the CRL to be world-readable), so once the cert is revoked, there's nothing more to do. If you want to make sure the client is disconnected, you need to restart OpenVPN (which will make all clients reconnect).

== /etc/hosts entries ==

Once the Vpn is setup on your machine, you can access Software Heritage hosts via their private IP addresses; see [[Network configuration]].

OpenVPN now pushes the address of our DNS server (192.168.100.29, pergamon).

You might want to add louvre.softwareheritage.org in your /etc/hosts to avoid a bootstrap problem if the "on-vpn" DNS server is in your resolv.conf.

[[Category:Infrastructure]]
[[Category:System administration]]

Debian packaging

2020-10-22T09:37:38Z

Ardumont: Update jenkins job template to match the current reality and add samples

== Package repository ==

A package repository is available on https://debian.softwareheritage.org/.

Unstable / Testing :
deb [trusted=yes] https://debian.softwareheritage.org/ unstable main

Stable / Buster :
deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main

Oldstable / Stretch :
deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main

This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).

=== Uploading packages ===

Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.

For packages to be accepted, they need to be :
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt>
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders

== Git repositories for Debian packages ==

Our git repository structure for Debian packages is compatible with <tt>git-buildpackage</tt>.

We have two different ways of handling repositories for Debian packages:
* Packages of python modules where *we* are upstream
* Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)

For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:
* Our own modules are merged with the upstream repository
* External dependencies ignore the upstream repository and only have packaging branches.

=== Branch and tags structure ===

Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:
* debian/upstream (history of unpacked upstream releases)
* debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, stretch-swh)
* pristine-tar (data to regenerate upstream tarballs from a git export)

The name of the debian/upstream branch doesn't matter ''as long as it's properly configured in the <tt>debian/gbp.conf</tt> file''. It's only really used by <tt>gbp import-orig</tt> when importing a new release.

The tags marking upstream releases imported from tarballs for Debian packaging purposes are named <tt>debian/upstream/''<upstream version number>''</tt>.

Our Jenkins jobs are triggered on incoming tags named <tt>debian/''<version>''</tt>. To generate the proper tags, use <tt>gbp buildpackage --git-tag-only</tt>.

The git-buildpackage configuration, <tt>debian/gbp.conf</tt>, should be the following:
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/''<current suite>''
pristine-tar=True

==== Automatic packaging for swh python modules ====

The <tt>swh.*</tt> python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs <tt>gbp import-orig</tt> with the tarball we release to PyPI, and the right options to merge the upstream history.

To merge changes from the upstream history, we add the following option to <tt>gbp.conf</tt>:
upstream-vcs-tag=v%(version)s

=== Bootstrapping a dependency packaging repository ===

Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:

Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/
wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz

Create a new git repository
git init pytest-postgresql
cd pytest-postgresql

Import the original upstream version
git checkout -b debian/unstable-swh
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz
# What will be the source package name? [pytest-postgresql]
# What is the upstream version? [1.3.4]
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...
# gbp:info: Source package is pytest-postgresql
# gbp:info: Upstream version is 1.3.4
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz

Bootstrap the debian directory
mkdir -p debian/source
echo '3.0 (quilt)' > debian/source/format
echo 9 > debian/compat
cat > debian/gbp.conf << EOF
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/unstable-swh
pristine-tar=True
EOF
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules
vim debian/control
# [...] adapt debian/control from another package
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1 --distribution unstable-swh
vim debian/copyright
# [...] adapt debian/copyright from another package
git add debian
git commit -m "Initial packaging for pytest-postgresql"

You can then go on to try building the package. Once the package builds, if you want to check your package's conformance to Debian policy, you can run <tt>lintian</tt> on the changes:
lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes

Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository

We need to use a <tt>+swh1</tt> version suffix to avoid clashing with potential upstream Debian package versions.

==== Bootstrapping the backport branches ====

During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.

The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.

Checkout the new branch
git checkout debian/<version number>
git checkout -b debian/stretch-swh

Update the gbp config to match the branch
sed -i s/unstable-swh/stretch-swh/ debian/gbp.conf

Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)
dch -l "~bpo9" -D stretch-swh --force-distribution 'Rebuild for stretch-swh'

You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.

==== Setting up the repository on Phabricator ====

The repository on Phabricator needs the following settings:
* Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)
* Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)
* Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)
* Policy
** View: Public (no login required)
** Edit: Developers
** Push: All users (actual restrictions are handled by Herald rules)
* Activate the repository
* Look up the path to the repository on the storage tab

You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes
ssh -p 2222 -t tate.internal.softwareheritage.org phabricator-setup-hook /srv/phabricator/repos/<repo-id> <post-receive-hook>

Note:

* there exists 2 types of <post-receive-hook>:
** ''post-receive-swh-modules'' for swh modules developed by the team
** ''post-receive-debian-deps'' for external modules packaged by the team
* remember that access to tate is on port 2222.

The repo ID can be found on the repo's "storage" property page on phabricator, typically
https://forge.softwareheritage.org/source/swh-SHORTNAME/manage/storage/

==== Setting up the Jenkins jobs ====

The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/
They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs

Jobs for dependency packages are configured in <tt>jobs/dependency-packages.yaml</tt>. You can add a section as follows:

- project:
name: <Callsign>
display-name: <short-name>
pkg: <source-name>
python_module: <python-module>
jobs:
- 'dependency-jobs-{name}'

For example:
- project:
name: DLDBASE
display-name: swh-loader-core
repo_name: swh-loader-core
pkg: loader.core
python_module: swh.loader.core
jobs:
- 'swh-jobs-{name}'

Other samples can be found in the dedicated repository.
* usual swh package: [https://forge.softwareheritage.org/source/swh-jenkins-jobs/browse/master/jobs/swh-packages.yaml$15-22 swh.core]
* peculiar swh package (with name divergences): [https://forge.softwareheritage.org/source/swh-jenkins-jobs/browse/master/jobs/swh-packages.yaml$51-58 swh.icinga_plugins]

Use the regular review process to land your changes.
Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.

If your package needs extra repositories to build, you can add them as comma-separated values to the <tt>deb-extra-repositories</tt> setting, with the following notes:
* When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.
* When building packages for backports suites, the backports repository is automatically enabled.

=== Updating a dependency packaging repository ===

Place yourself on the debian/unstable-swh branch and "gbp import-origin" a more
recent upstream release tarballs.

For example (current version on 0.0.5, upstream bumped to 0.0.7):
gbp import-origin https://files.pythonhosted.org/packages/7a/bb/cf8fec6009e7d0cec52dc179d09b28c4c70d158e79b565e8aab7606e1717/attrs-strict-0.0.7.tar.gz

This will update the following branches:
* debian/upstream
* pristine-tar
* debian/unstable-swh

This also includes the necessary tags (`debian/upstream/0.0.7` here).

You then need to push all branches/tags to the repository:
git push origin --all --follow-tags

Ensure the [https://wiki.softwareheritage.org/wiki/Debian_packaging#Local_package_building update builds fine]
And [https://wiki.softwareheritage.org/wiki/Debian_packaging#Remote_package_building tags accordingly the debian/unstable-swh branch when ok].
Jenkins will then keep up on building the package.

=== Local package building ===

To locally test a package build, go on the appropriate debian packaging branch, and run
gbp buildpackage --git-builder=sbuild -As --no-clean-source

<tt>gbp buildpackage</tt> passes all options not starting with <tt>--git-</tt> to the builder. Some useful options are the following:

* <tt>--git-ignore-new</tt> builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.
* <tt>--no-clean-source</tt> doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies
* <tt>--extra-repository="'''repository specification'''"</tt> adds the given repository in the chroot before building.
* <tt>--extra-repository-key='''repository signing key'''</tt> adds the given key as a trusted gpg key for package sources
* <tt>--extra-package='''<.deb file or directory>'''</tt> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.
* <tt>--force-orig-source</tt> forces addition of the <tt>.orig.tar.gz</tt> file in the <tt>.changes</tt> file (useful when trying to upload a backport)

See <tt>gbp help buildpackage</tt> and <tt>man sbuild</tt> for a full description of all options

for example:
gbp buildpackage --git-builder=sbuild -As --no-clean-source --force-orig-source \
--extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main'

or if you need some third-party repository, say for cassandra:
gbp buildpackage --git-builder=sbuild -As --no-clean-source --force-orig-source \
--extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main' \
--extra-repository='deb [arch=amd64 trusted=yes] https://downloads.apache.org/cassandra/debian 40x main'

(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping <tt>gbp buildpackage</tt> with the most common options)

=== Remote package building ===

Jenkins builds packages when the repository receives a tag.

Once the local build succeeds, tag the package with:
gbp buildpackage --git-tag-only --git-sign-tags

Alternatively, you can add the <tt>--git-tag</tt> option to your <tt>gbp buildpackage</tt> command so the tag happens automatically on a successful build.

Then, push your tag, and Jenkins jobs should get triggered
git push --tags

== Build Environment setup ==

Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.

=== sbuild setup ===

# Install the package
sudo apt-get install sbuild

# Add your user to the sbuild group, to allow him to use the sbuild commands
sudo sbuild-adduser $USER
# You have to logout and log back in

# Prepare chroots
sudo mkdir /srv/chroots
sudo mkdir /srv/chroots/var

# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/
sudo rm -r /var/lib/schroot
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot

sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/
sudo rm -r /var/lib/sbuild
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild
# end optionally

# Create unstable/sid chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/

# Create stretch chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates stretch /srv/chroots/stretch http://deb.debian.org/debian/

# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts
echo hosts >> /etc/schroot/sbuild/nssdatabases

=== schroot setup ===

Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.

You need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:

source-groups=root,sbuild
source-root-groups=root,sbuild
union-type=overlay

This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.

You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):

For unstable:
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild

For stretch:
aliases=stretch-swh-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild

==== dependencies cache ====

Add the following line to schroot's fstab /etc/schroot/sbuild/fstab
to permit reuse of existing fetched dependencies:

/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0

You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in <tt>/etc/apt/apt.conf.d</tt> on each chroot

=== schroot update ===

You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):

sudo sbuild-update -udcar sid; sudo sbuild-update -udcar stretch; sudo sbuild-update -ud jessie

=== environment setup ===

The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt>

export DEBFULLNAME="Debra Hacker"
export DEBEMAIL=debra.hacker@example.com

Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourfullkeyid></tt> variable.
(Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).

=== overlay in tmpfs for faster builds ===

You can add this to your fstab to put the overlay hierarchy in RAM:

tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0 0 0

=== Base packages ===

In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.

'''If you do so, do not use these chroots to upload to Debian itself!'''

[[Category:Software development]]
[[Category:System administration]]

Debian packaging

2020-10-20T15:03:58Z

Ardumont: clarify the 2 types of post-receive hooks

== Package repository ==

A package repository is available on https://debian.softwareheritage.org/.

Unstable / Testing :
deb [trusted=yes] https://debian.softwareheritage.org/ unstable main

Stable / Buster :
deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main

Oldstable / Stretch :
deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main

This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).

=== Uploading packages ===

Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.

For packages to be accepted, they need to be :
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt>
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders

== Git repositories for Debian packages ==

Our git repository structure for Debian packages is compatible with <tt>git-buildpackage</tt>.

We have two different ways of handling repositories for Debian packages:
* Packages of python modules where *we* are upstream
* Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)

For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:
* Our own modules are merged with the upstream repository
* External dependencies ignore the upstream repository and only have packaging branches.

=== Branch and tags structure ===

Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:
* debian/upstream (history of unpacked upstream releases)
* debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, stretch-swh)
* pristine-tar (data to regenerate upstream tarballs from a git export)

The name of the debian/upstream branch doesn't matter ''as long as it's properly configured in the <tt>debian/gbp.conf</tt> file''. It's only really used by <tt>gbp import-orig</tt> when importing a new release.

The tags marking upstream releases imported from tarballs for Debian packaging purposes are named <tt>debian/upstream/''<upstream version number>''</tt>.

Our Jenkins jobs are triggered on incoming tags named <tt>debian/''<version>''</tt>. To generate the proper tags, use <tt>gbp buildpackage --git-tag-only</tt>.

The git-buildpackage configuration, <tt>debian/gbp.conf</tt>, should be the following:
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/''<current suite>''
pristine-tar=True

==== Automatic packaging for swh python modules ====

The <tt>swh.*</tt> python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs <tt>gbp import-orig</tt> with the tarball we release to PyPI, and the right options to merge the upstream history.

To merge changes from the upstream history, we add the following option to <tt>gbp.conf</tt>:
upstream-vcs-tag=v%(version)s

=== Bootstrapping a dependency packaging repository ===

Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:

Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/
wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz

Create a new git repository
git init pytest-postgresql
cd pytest-postgresql

Import the original upstream version
git checkout -b debian/unstable-swh
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz
# What will be the source package name? [pytest-postgresql]
# What is the upstream version? [1.3.4]
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...
# gbp:info: Source package is pytest-postgresql
# gbp:info: Upstream version is 1.3.4
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz

Bootstrap the debian directory
mkdir -p debian/source
echo '3.0 (quilt)' > debian/source/format
echo 9 > debian/compat
cat > debian/gbp.conf << EOF
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/unstable-swh
pristine-tar=True
EOF
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules
vim debian/control
# [...] adapt debian/control from another package
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1 --distribution unstable-swh
vim debian/copyright
# [...] adapt debian/copyright from another package
git add debian
git commit -m "Initial packaging for pytest-postgresql"

You can then go on to try building the package. Once the package builds, if you want to check your package's conformance to Debian policy, you can run <tt>lintian</tt> on the changes:
lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes

Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository

We need to use a <tt>+swh1</tt> version suffix to avoid clashing with potential upstream Debian package versions.

==== Bootstrapping the backport branches ====

During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.

The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.

Checkout the new branch
git checkout debian/<version number>
git checkout -b debian/stretch-swh

Update the gbp config to match the branch
sed -i s/unstable-swh/stretch-swh/ debian/gbp.conf

Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)
dch -l "~bpo9" -D stretch-swh --force-distribution 'Rebuild for stretch-swh'

You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.

==== Setting up the repository on Phabricator ====

The repository on Phabricator needs the following settings:
* Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)
* Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)
* Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)
* Policy
** View: Public (no login required)
** Edit: Developers
** Push: All users (actual restrictions are handled by Herald rules)
* Activate the repository
* Look up the path to the repository on the storage tab

You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes
ssh -p 2222 -t tate.internal.softwareheritage.org phabricator-setup-hook /srv/phabricator/repos/<repo-id> <post-receive-hook>

Note:

* there exists 2 types of <post-receive-hook>:
** ''post-receive-swh-modules'' for swh modules developed by the team
** ''post-receive-debian-deps'' for external modules packaged by the team
* remember that access to tate is on port 2222.

The repo ID can be found on the repo's "storage" property page on phabricator, typically
https://forge.softwareheritage.org/source/swh-SHORTNAME/manage/storage/

==== Setting up the Jenkins jobs ====

The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/
They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs

Jobs for dependency packages are configured in <tt>jobs/dependency-packages.yaml</tt>. You can add a section as follows:

- project:
name: <Callsign>
display-name: <short-name>
pkg: <source-name>
jobs:
- 'dependency-jobs-{name}'

Use the regular review process to land your changes.
Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.

If your package needs extra repositories to build, you can add them as comma-separated values to the <tt>deb-extra-repositories</tt> setting, with the following notes:
* When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.
* When building packages for backports suites, the backports repository is automatically enabled.

=== Updating a dependency packaging repository ===

Place yourself on the debian/unstable-swh branch and "gbp import-origin" a more
recent upstream release tarballs.

For example (current version on 0.0.5, upstream bumped to 0.0.7):
gbp import-origin https://files.pythonhosted.org/packages/7a/bb/cf8fec6009e7d0cec52dc179d09b28c4c70d158e79b565e8aab7606e1717/attrs-strict-0.0.7.tar.gz

This will update the following branches:
* debian/upstream
* pristine-tar
* debian/unstable-swh

This also includes the necessary tags (`debian/upstream/0.0.7` here).

You then need to push all branches/tags to the repository:
git push origin --all --follow-tags

Ensure the [https://wiki.softwareheritage.org/wiki/Debian_packaging#Local_package_building update builds fine]
And [https://wiki.softwareheritage.org/wiki/Debian_packaging#Remote_package_building tags accordingly the debian/unstable-swh branch when ok].
Jenkins will then keep up on building the package.

=== Local package building ===

To locally test a package build, go on the appropriate debian packaging branch, and run
gbp buildpackage --git-builder=sbuild -As --no-clean-source

<tt>gbp buildpackage</tt> passes all options not starting with <tt>--git-</tt> to the builder. Some useful options are the following:

* <tt>--git-ignore-new</tt> builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.
* <tt>--no-clean-source</tt> doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies
* <tt>--extra-repository="'''repository specification'''"</tt> adds the given repository in the chroot before building.
* <tt>--extra-repository-key='''repository signing key'''</tt> adds the given key as a trusted gpg key for package sources
* <tt>--extra-package='''<.deb file or directory>'''</tt> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.
* <tt>--force-orig-source</tt> forces addition of the <tt>.orig.tar.gz</tt> file in the <tt>.changes</tt> file (useful when trying to upload a backport)

See <tt>gbp help buildpackage</tt> and <tt>man sbuild</tt> for a full description of all options

for example:
gbp buildpackage --git-builder=sbuild -As --no-clean-source --force-orig-source \
--extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main'

or if you need some third-party repository, say for cassandra:
gbp buildpackage --git-builder=sbuild -As --no-clean-source --force-orig-source \
--extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main' \
--extra-repository='deb [arch=amd64 trusted=yes] https://downloads.apache.org/cassandra/debian 40x main'

(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping <tt>gbp buildpackage</tt> with the most common options)

=== Remote package building ===

Jenkins builds packages when the repository receives a tag.

Once the local build succeeds, tag the package with:
gbp buildpackage --git-tag-only --git-sign-tags

Alternatively, you can add the <tt>--git-tag</tt> option to your <tt>gbp buildpackage</tt> command so the tag happens automatically on a successful build.

Then, push your tag, and Jenkins jobs should get triggered
git push --tags

== Build Environment setup ==

Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.

=== sbuild setup ===

# Install the package
sudo apt-get install sbuild

# Add your user to the sbuild group, to allow him to use the sbuild commands
sudo sbuild-adduser $USER
# You have to logout and log back in

# Prepare chroots
sudo mkdir /srv/chroots
sudo mkdir /srv/chroots/var

# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/
sudo rm -r /var/lib/schroot
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot

sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/
sudo rm -r /var/lib/sbuild
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild
# end optionally

# Create unstable/sid chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/

# Create stretch chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates stretch /srv/chroots/stretch http://deb.debian.org/debian/

# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts
echo hosts >> /etc/schroot/sbuild/nssdatabases

=== schroot setup ===

Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.

You need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:

source-groups=root,sbuild
source-root-groups=root,sbuild
union-type=overlay

This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.

You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):

For unstable:
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild

For stretch:
aliases=stretch-swh-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild

==== dependencies cache ====

Add the following line to schroot's fstab /etc/schroot/sbuild/fstab
to permit reuse of existing fetched dependencies:

/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0

You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in <tt>/etc/apt/apt.conf.d</tt> on each chroot

=== schroot update ===

You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):

sudo sbuild-update -udcar sid; sudo sbuild-update -udcar stretch; sudo sbuild-update -ud jessie

=== environment setup ===

The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt>

export DEBFULLNAME="Debra Hacker"
export DEBEMAIL=debra.hacker@example.com

Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourfullkeyid></tt> variable.
(Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).

=== overlay in tmpfs for faster builds ===

You can add this to your fstab to put the overlay hierarchy in RAM:

tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0 0 0

=== Base packages ===

In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.

'''If you do so, do not use these chroots to upload to Debian itself!'''

[[Category:Software development]]
[[Category:System administration]]

Debian packaging

2020-09-23T13:23:29Z

Ardumont: mention the port tidbit for tate...

== Package repository ==

A package repository is available on https://debian.softwareheritage.org/.

Unstable / Testing :
deb [trusted=yes] https://debian.softwareheritage.org/ unstable main

Stable / Buster :
deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main

Oldstable / Stretch :
deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main

This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).

=== Uploading packages ===

Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.

For packages to be accepted, they need to be :
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt>
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders

== Git repositories for Debian packages ==

Our git repository structure for Debian packages is compatible with <tt>git-buildpackage</tt>.

We have two different ways of handling repositories for Debian packages:
* Packages of python modules where *we* are upstream
* Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)

For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:
* Our own modules are merged with the upstream repository
* External dependencies ignore the upstream repository and only have packaging branches.

=== Branch and tags structure ===

Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:
* debian/upstream (history of unpacked upstream releases)
* debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, stretch-swh)
* pristine-tar (data to regenerate upstream tarballs from a git export)

The name of the debian/upstream branch doesn't matter ''as long as it's properly configured in the <tt>debian/gbp.conf</tt> file''. It's only really used by <tt>gbp import-orig</tt> when importing a new release.

The tags marking upstream releases imported from tarballs for Debian packaging purposes are named <tt>debian/upstream/''<upstream version number>''</tt>.

Our Jenkins jobs are triggered on incoming tags named <tt>debian/''<version>''</tt>. To generate the proper tags, use <tt>gbp buildpackage --git-tag-only</tt>.

The git-buildpackage configuration, <tt>debian/gbp.conf</tt>, should be the following:
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/''<current suite>''
pristine-tar=True

==== Automatic packaging for swh python modules ====

The <tt>swh.*</tt> python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs <tt>gbp import-orig</tt> with the tarball we release to PyPI, and the right options to merge the upstream history.

To merge changes from the upstream history, we add the following option to <tt>gbp.conf</tt>:
upstream-vcs-tag=v%(version)s

=== Bootstrapping a dependency packaging repository ===

Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:

Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/
wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz

Create a new git repository
git init pytest-postgresql
cd pytest-postgresql

Import the original upstream version
git checkout -b debian/unstable-swh
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz
# What will be the source package name? [pytest-postgresql]
# What is the upstream version? [1.3.4]
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...
# gbp:info: Source package is pytest-postgresql
# gbp:info: Upstream version is 1.3.4
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz

Bootstrap the debian directory
mkdir -p debian/source
echo '3.0 (quilt)' > debian/source/format
echo 9 > debian/compat
cat > debian/gbp.conf << EOF
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/unstable-swh
pristine-tar=True
EOF
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules
vim debian/control
# [...] adapt debian/control from another package
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1 --distribution unstable-swh
vim debian/copyright
# [...] adapt debian/copyright from another package
git add debian
git commit -m "Initial packaging for pytest-postgresql"

You can then go on to try building the package. Once the package builds, if you want to check your package's conformance to Debian policy, you can run <tt>lintian</tt> on the changes:
lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes

Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository

We need to use a <tt>+swh1</tt> version suffix to avoid clashing with potential upstream Debian package versions.

==== Bootstrapping the backport branches ====

During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.

The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.

Checkout the new branch
git checkout debian/<version number>
git checkout -b debian/stretch-swh

Update the gbp config to match the branch
sed -i s/unstable-swh/stretch-swh/ debian/gbp.conf

Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)
dch -l "~bpo9" -D stretch-swh --force-distribution 'Rebuild for stretch-swh'

You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.

==== Setting up the repository on Phabricator ====

The repository on Phabricator needs the following settings:
* Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)
* Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)
* Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)
* Policy
** View: Public (no login required)
** Edit: Developers
** Push: All users (actual restrictions are handled by Herald rules)
* Activate the repository
* Look up the path to the repository on the storage tab

You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes.
ssh -p 2222 -t tate.internal.softwareheritage.org phabricator-setup-hook /srv/phabricator/repos/<repo-id> post-receive-debian-deps

Note: remember that access to tate is on port 2222.

The repo ID can be found on the repo's "storage" property page on phabricator, typically
https://forge.softwareheritage.org/source/swh-SHORTNAME/manage/storage/

==== Setting up the Jenkins jobs ====

The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/
They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs

Jobs for dependency packages are configured in <tt>jobs/dependency-packages.yaml</tt>. You can add a section as follows:

- project:
name: <Callsign>
display-name: <short-name>
pkg: <source-name>
jobs:
- 'dependency-jobs-{name}'

Use the regular review process to land your changes.
Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.

If your package needs extra repositories to build, you can add them as comma-separated values to the <tt>deb-extra-repositories</tt> setting, with the following notes:
* When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.
* When building packages for backports suites, the backports repository is automatically enabled.

=== Updating a dependency packaging repository ===

Place yourself on the debian/unstable-swh branch and "gbp import-origin" a more
recent upstream release tarballs.

For example (current version on 0.0.5, upstream bumped to 0.0.7):
gbp import-origin https://files.pythonhosted.org/packages/7a/bb/cf8fec6009e7d0cec52dc179d09b28c4c70d158e79b565e8aab7606e1717/attrs-strict-0.0.7.tar.gz

This will update the following branches:
* debian/upstream
* pristine-tar
* debian/unstable-swh

This also includes the necessary tags (`debian/upstream/0.0.7` here).

You then need to push all branches/tags to the repository:
git push origin --all --follow-tags

Ensure the [https://wiki.softwareheritage.org/wiki/Debian_packaging#Local_package_building update builds fine]
And [https://wiki.softwareheritage.org/wiki/Debian_packaging#Remote_package_building tags accordingly the debian/unstable-swh branch when ok].
Jenkins will then keep up on building the package.

=== Local package building ===

To locally test a package build, go on the appropriate debian packaging branch, and run
gbp buildpackage --git-builder=sbuild -As --no-clean-source

<tt>gbp buildpackage</tt> passes all options not starting with <tt>--git-</tt> to the builder. Some useful options are the following:

* <tt>--git-ignore-new</tt> builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.
* <tt>--no-clean-source</tt> doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies
* <tt>--extra-repository="'''repository specification'''"</tt> adds the given repository in the chroot before building.
* <tt>--extra-repository-key='''repository signing key'''</tt> adds the given key as a trusted gpg key for package sources
* <tt>--extra-package='''<.deb file or directory>'''</tt> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.
* <tt>--force-orig-source</tt> forces addition of the <tt>.orig.tar.gz</tt> file in the <tt>.changes</tt> file (useful when trying to upload a backport)

See <tt>gbp help buildpackage</tt> and <tt>man sbuild</tt> for a full description of all options

for example:
gbp buildpackage --git-builder=sbuild -As --force-orig-source --extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main'

or if you need some third-party repository, say for cassandra:
gbp buildpackage --git-builder=sbuild -As --force-orig-source \
--extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main' \
--extra-repository='deb [arch=amd64 trusted=yes] https://downloads.apache.org/cassandra/debian 40x main'

(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping <tt>gbp buildpackage</tt> with the most common options)

=== Remote package building ===

Jenkins builds packages when the repository receives a tag.

Once the local build succeeds, tag the package with:
gbp buildpackage --git-tag-only --git-sign-tags

Alternatively, you can add the <tt>--git-tag</tt> option to your <tt>gbp buildpackage</tt> command so the tag happens automatically on a successful build.

Then, push your tag, and Jenkins jobs should get triggered
git push --tags

== Build Environment setup ==

Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.

=== sbuild setup ===

# Install the package
sudo apt-get install sbuild

# Add your user to the sbuild group, to allow him to use the sbuild commands
sudo sbuild-adduser $USER
# You have to logout and log back in

# Prepare chroots
sudo mkdir /srv/chroots
sudo mkdir /srv/chroots/var

# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/
sudo rm -r /var/lib/schroot
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot

sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/
sudo rm -r /var/lib/sbuild
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild
# end optionally

# Create unstable/sid chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/

# Create stretch chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates stretch /srv/chroots/stretch http://deb.debian.org/debian/

# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts
echo hosts >> /etc/schroot/sbuild/nssdatabases

=== schroot setup ===

Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.

You need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:

source-groups=root,sbuild
source-root-groups=root,sbuild
union-type=overlay

This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.

You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):

For unstable:
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild

For stretch:
aliases=stretch-swh-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild

==== dependencies cache ====

Add the following line to schroot's fstab /etc/schroot/sbuild/fstab
to permit reuse of existing fetched dependencies:

/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0

You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in <tt>/etc/apt/apt.conf.d</tt> on each chroot

=== schroot update ===

You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):

sudo sbuild-update -udcar sid; sudo sbuild-update -udcar stretch; sudo sbuild-update -ud jessie

=== environment setup ===

The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt>

export DEBFULLNAME="Debra Hacker"
export DEBEMAIL=debra.hacker@example.com

Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourfullkeyid></tt> variable.
(Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).

=== overlay in tmpfs for faster builds ===

You can add this to your fstab to put the overlay hierarchy in RAM:

tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0 0 0

=== Base packages ===

In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.

'''If you do so, do not use these chroots to upload to Debian itself!'''

[[Category:Software development]]
[[Category:System administration]]

Jenkins

2020-09-23T13:11:48Z

Ardumont: Add links to the actual page declaring the steps to setup phabricator or new jenkins jobs

= Jenkins =

The CI is run by [https://jenkins.io/ Jenkins] on jenkins.softwareheritage.org

== Authentication ==

Software Heritage staffers can login via the Jenkins Web user interface using the same personal *nix credentials they use to login into other machines.

== Jobs ==

There are 2 categories of job:

* jobs related to jenkins itself, like running Jenkins Job Builder, updating and managing docker images. These are found in the [https://jenkins.softwareheritage.org/job/jenkins-tools/ jenkins-tools] directory.
* jobs dedicated to testing/building swh packages.

=== Jobs definition ===

Most of the jobs are created using [https://docs.openstack.org/infra/jenkins-job-builder/ Jenkins Job Builder] (aka JJB).

The source repository is https://forge.softwareheritage.org/source/swh-jenkins-jobs/

Each time a new revision is pushed on this repo, the [https://jenkins.softwareheritage.org/job/jenkins-tools/job/swh-jenkins-job-builder/ JJB job] is executed. This job updates already existing jobs or create new ones (if any). Beware not to break the JJB job!

=== Job execution environments ===

Jenkins job execution may occur either on the Jenkins master node directly (typically for [https://jenkins.softwareheritage.org/job/jenkins-tools/ jenkins-tools] jobs), or in a docker jenkins slave.

Docker images used by Jenkins are created and updated using the Dockerfiles in
https://forge.softwareheritage.org/source/swh-jenkins-dockerfiles/

For now, there are 2 different images:

* swh-jenkins/tox: a Debian stretch with python3 and tox installed; used to run unit tests; it's available in Jenkins under the label [https://jenkins.softwareheritage.org/label/swh-tox/ swh-tox]
* swh-jenkins/sphinx: based on the former tox image, it adds everything needed to compile the documentation with sphinx; it's available in Jenkins under the label [https://jenkins.softwareheritage.org/label/swh-sphinx/ swh-sphinx].

=== swh packages ===

Each swh package has a Jenkins [https://plugins.jenkins.io/cloudbees-folder Folder] in which are all the jobs dedicated to this package.

There are two jenkins jobs for each swh package:

* [https://jenkins.softwareheritage.org/job/DCORE/job/tox/ Phab. Diff] (here for [https://forge.softwareheritage.org/source/swh-core/ swh-core]): these jobs are executed each time a Phabricator Diff is created or updated,
* [https://jenkins.softwareheritage.org/job/DCORE/job/pipeline/ master branch] (here for [https://forge.softwareheritage.org/source/swh-core/ swh-core]): these jobs are executed when git revisions are pushed on the master branch.

When these jobs are started by Phabricator, the job's end status is pushed back on the Phabricator object that triggered the build (ie. the Diff object or the Revision object). As a result, the build status for a given Diff or git revision will be displayed in the forge.

The Diff in Phabricator will typically look like:

[[File:JenkinsDiffReport.png]]

The Diffusion view of a git repository will look like:

[[File:JenkinsRevisionReport.png]]

where the green "checks" in each revision means the CI passed OK on this revision.

=== Dashboards and metrics ===

The [https://jenkins.softwareheritage.org/view/swh%20dashboard/ swh dashboard] gives a global view on the CI status of swh packages. It shows all the "master branch" job status for all swh packages, as well as a few metrics for these builds (%success, code coverage, execution time, etc.)

=== Starting a job manually ===

[[File:JenkinsBuildButton.png|frameless|left]]
You should be able to execute any job by hand. When logged in, on the job's description page (eg. [https://jenkins.softwareheritage.org/view/swh%20dashboard/job/DCORE/job/tests/ swh-core master]) you should be able to push the "Build with Parameters" button which opens a form where you can specify the job's input parameters.

[[File:JenkinsRestartFromStage.png|frameless|left]]
[[File:JenkinsReplay.png|frameless|left]]
Note that you can also restart a (generally failed) job. From the job execution summary (eg. [https://jenkins.softwareheritage.org/view/swh%20dashboard/job/DCORE/job/tests/50/ this execution]) you may find a "Restart from Stage" button (on "master branch" jobs only) or a "Replay" button that will allow you to recreate a job with the same parameters as the original job execution.

== Phabricator ==

[https://wiki.softwareheritage.org/wiki/Debian_packaging#Setting_up_the_repository_on_Phabricator Setting up the repository to trigger debian package]

== New swh module jenkins job ==

[https://wiki.softwareheritage.org/wiki/Debian_packaging#Setting_up_the_Jenkins_jobs Setting up a new swh module jenkins job]

[[Category:Software development]]

Code review in Phabricator

2020-07-29T12:37:38Z

Ardumont: /* Dependencies between diffs */

We use the [[Differential]] application of [[Phabricator]] to perform [[code review|code reviews]] in the context of [[Software Heritage]].

* we use Git and history.immutable=true (but beware as that is partly a Phabricator misnomer, read on)
* when code reviews are required, developers will be allowed to push directly to master once an accepted Differential diff exists

== Configuration ==

=== Arcanist configuration ===

When using git, [[Arcanist]] by default mess with the local history, rewriting commits at the time of first submission. 
To avoid that we use so called [https://secure.phabricator.com/book/phabricator/article/arcanist_new_project/#history-mutability-git history immutability].

To that end, you shall configure your <tt>arc</tt> accordingly:
<pre>
arc set-config history.immutable true
</pre>

Note that this does ''not'' mean that you are forbidden to rewrite your local branches (e.g., with <tt>git rebase</tt>).
Quite the contrary: you are encouraged to locally rewrite branches before pushing to ensure that commits are logically separated and your commit history easy to bisect.
The above setting just means that ''arc'' will not rewrite commit history under your nose.

=== Enabling <tt>git push</tt> to our forge ===

The way we've configured our review setup for continuous integration needs you to configure git to allow pushes to our forge. There's two ways you can do this : setting a ssh key to push over ssh, or setting a specific password for git pushes over https.

==== SSH key for pushes ====

In your forge User settings page (On the top right, click on your avatar, then click ''Settings''), you have access to a ''Authentication'' > ''SSH Public Keys'' section (Direct link: <tt>hxxps://forge.softwareheritage.org/settings/user/'''<your username>'''/page/ssh/</tt>). You then have the option to upload a SSH public key, which will authenticate your pushes.

You then need to configure ssh/git to use that key pair, for instance by editing the <tt>~/.ssh/config</tt> file.

Finally, you should configure git to push over ssh when pushing to https://forge.softwareheritage.org, by running the following command:
git config --global url.git@forge.softwareheritage.org:.pushInsteadOf https://forge.softwareheritage.org

This lets git know that it should use <tt>git@forge.softwareheritage.org:</tt> as a base url when pushing repositories cloned from forge.softwareheritage.org over https.

==== VCS password for pushes ====

If you're not comfortable setting up SSH to upload your changes, you have the option of setting a VCS password. This password, ''separate from your account password'', allows Phabricator to authenticate your uploads over HTTPS.

In your forge User settings page (On the top right, click on your avatar, then click ''Settings''), you need to use the ''Authentication'' > ''VCS Password'' section to set your VCS password (Direct link: <tt>hxxps://forge.softwareheritage.org/settings/user/'''<your username>'''/page/vcspassword/</tt>).

If you still get a 403 error on push, this means you need a forge administrator to enable HTTPS pushes for the repository (which wasn't done by default in historical repositories). Please drop by on IRC and let us know!

== Workflow ==

* work in a feature branch: <tt>git checkout -b my-feat</tt>
* initial review request: hack/commit/hack/commit ; <tt>arc diff origin/master</tt>
* react to change requests: hack/commit/hack/commit ; <tt>arc diff --update Dxx origin/master</tt>
* landing change: <tt>git checkout master ; git merge my-feat ; git push</tt>

=== Starting a new feature and submit it for review ===

Use a '''one branch per feature''' workflow, with well-separated ''logical commits'' ([https://wiki.softwareheritage.org/wiki/Git_style_guide following those conventions]).
Please open one diff per logical commit to keep the diff size to a minimum.

<pre>
git checkout -b my-shiny-feature
... hack hack hack ...
git commit -m 'architecture skeleton for my-shiny-feature'
... hack hack hack ...
git commit -m 'my-shiny-feature: implement module foo'
... etc ...
</pre>

Please, follow the
To '''submit your code for review''' the first time:
<pre>
arc diff origin/master
</pre>
arc will prompt for a '''code review message'''. Provide the following information:
* first line: ''short description'' of the overall work (i.e., the feature you're working on). This will become the title of the review
* ''Summary'' field (optional): ''long description'' of the overall work; the field can continue in subsequent lines, up to the next field. This will become the "Summary" section of the review
* ''Test Plan'' field (optional): write here if something special is needed to test your change
* ''Reviewers'' field (optional): the (Phabricator) name(s) of desired reviewers. If you don't specify one (recommended) the default reviewers will be chosen
* ''Subscribers'' field (optional): the (Phabricator) name(s) of people that will be notified about changes to this review request. In most cases it should be left empty

For example:
<pre>
mercurial loader

Summary: first stab at a mercurial loader (T329)

The implementation follows the plan detailed in F2F discussion with @foo.

Performances seem decent enough for a first trial (XXX seconds for YYY repository
that contains ZZZ patches).

Test plan:

Reviewers:

Subscribers: foo
</pre>

After completing the message arc will submit the review request and tell you its number and URL:
<pre>
[...]
Created a new Differential revision:
Revision URI: https://forge.softwareheritage.org/Dxx
</pre>

=== Updating your branch to reflect requested changes ===

Your feature might get accepted as is, YAY!
Or, reviewers might request changes; no big deal!

Use the Differential web UI to follow-up to received comments, if needed.

To implement requested changes in the code, hack on your branch as usual by:

* adding new commits, and/or
* rewriting old commits with git rebase (to preserve a nice, easy to bisect history)

When you're ready to '''update your review request''':
<pre>
arc diff --update Dxx HEAD~
</pre>

Arc will prompt you for a message: describe what you've changed w.r.t. the previous review request, free form.
Your message will become the changelog entry in Differential for this new version of the diff.

Differential only care about the code diff, and not about the commits or their order.
Therefore each "update" can be a completely different series of commits, possibly rewritten from the previous submission.

=== Dependencies between diffs ===

Note that you can manage diff dependencies within the same module with the following keyword in the diff description:

<pre>
Depends on Dxx
</pre>

That allows to keep a logical view in your diff.
It's not strictly necessary (because the tooling now deals with it properly) but it might help reviewers or yourself to do so.

=== Landing your change onto master ===

Once your change has been approved in Differential, you will be able to land it onto the master branch.

Before doing so, you're encouraged to '''clean up your git commit history''', reordering/splitting/merging commits as needed to have separate logical commits and an easy to bisect history.
Update the diff [https://wiki.softwareheritage.org/wiki/Code_review_in_Phabricator#Updating_your_branch_to_reflect_requested_changes following the prior section].
(It'd be good to let the ci build finish to make sure everything is still green).

Once you're happy you can '''push to origin/master''' directly, e.g.:
<pre>
git checkout master
git merge --ff-only my-shiny-feature
git push
</pre>

<code>--ff-only</code> is optional, and makes sure you don't unintentionally create a merge commit.

Optionally you can then delete your local feature branch:
<pre>
git branch -d my-shiny-feature
</pre>

=== Reviewing locally / landing someone else's changes ===

You can do local reviews of code with arc patch:

<pre>
arc patch Dxyz
</pre>

This will create a branch '''arcpatch-Dxyz''' containing the changes on your local checkout.

You can then merge those changes upstream with

<pre>
git checkout master
git merge --ff arcpatch-Dxyz
git push origin master
</pre>

or, alternatively:

<pre>
arc land --squash
</pre>

== See also ==

* [[Code review]] for guidelines on how code is reviewed when developing for Software Heritage

[[Category:Software development]]

Code review in Phabricator

2020-07-29T12:36:58Z

Ardumont: About dependencies between diffs

We use the [[Differential]] application of [[Phabricator]] to perform [[code review|code reviews]] in the context of [[Software Heritage]].

* we use Git and history.immutable=true (but beware as that is partly a Phabricator misnomer, read on)
* when code reviews are required, developers will be allowed to push directly to master once an accepted Differential diff exists

== Configuration ==

=== Arcanist configuration ===

When using git, [[Arcanist]] by default mess with the local history, rewriting commits at the time of first submission. 
To avoid that we use so called [https://secure.phabricator.com/book/phabricator/article/arcanist_new_project/#history-mutability-git history immutability].

To that end, you shall configure your <tt>arc</tt> accordingly:
<pre>
arc set-config history.immutable true
</pre>

Note that this does ''not'' mean that you are forbidden to rewrite your local branches (e.g., with <tt>git rebase</tt>).
Quite the contrary: you are encouraged to locally rewrite branches before pushing to ensure that commits are logically separated and your commit history easy to bisect.
The above setting just means that ''arc'' will not rewrite commit history under your nose.

=== Enabling <tt>git push</tt> to our forge ===

The way we've configured our review setup for continuous integration needs you to configure git to allow pushes to our forge. There's two ways you can do this : setting a ssh key to push over ssh, or setting a specific password for git pushes over https.

==== SSH key for pushes ====

In your forge User settings page (On the top right, click on your avatar, then click ''Settings''), you have access to a ''Authentication'' > ''SSH Public Keys'' section (Direct link: <tt>hxxps://forge.softwareheritage.org/settings/user/'''<your username>'''/page/ssh/</tt>). You then have the option to upload a SSH public key, which will authenticate your pushes.

You then need to configure ssh/git to use that key pair, for instance by editing the <tt>~/.ssh/config</tt> file.

Finally, you should configure git to push over ssh when pushing to https://forge.softwareheritage.org, by running the following command:
git config --global url.git@forge.softwareheritage.org:.pushInsteadOf https://forge.softwareheritage.org

This lets git know that it should use <tt>git@forge.softwareheritage.org:</tt> as a base url when pushing repositories cloned from forge.softwareheritage.org over https.

==== VCS password for pushes ====

If you're not comfortable setting up SSH to upload your changes, you have the option of setting a VCS password. This password, ''separate from your account password'', allows Phabricator to authenticate your uploads over HTTPS.

In your forge User settings page (On the top right, click on your avatar, then click ''Settings''), you need to use the ''Authentication'' > ''VCS Password'' section to set your VCS password (Direct link: <tt>hxxps://forge.softwareheritage.org/settings/user/'''<your username>'''/page/vcspassword/</tt>).

If you still get a 403 error on push, this means you need a forge administrator to enable HTTPS pushes for the repository (which wasn't done by default in historical repositories). Please drop by on IRC and let us know!

== Workflow ==

* work in a feature branch: <tt>git checkout -b my-feat</tt>
* initial review request: hack/commit/hack/commit ; <tt>arc diff origin/master</tt>
* react to change requests: hack/commit/hack/commit ; <tt>arc diff --update Dxx origin/master</tt>
* landing change: <tt>git checkout master ; git merge my-feat ; git push</tt>

=== Starting a new feature and submit it for review ===

Use a '''one branch per feature''' workflow, with well-separated ''logical commits'' ([https://wiki.softwareheritage.org/wiki/Git_style_guide following those conventions]).
Please open one diff per logical commit to keep the diff size to a minimum.

<pre>
git checkout -b my-shiny-feature
... hack hack hack ...
git commit -m 'architecture skeleton for my-shiny-feature'
... hack hack hack ...
git commit -m 'my-shiny-feature: implement module foo'
... etc ...
</pre>

Please, follow the
To '''submit your code for review''' the first time:
<pre>
arc diff origin/master
</pre>
arc will prompt for a '''code review message'''. Provide the following information:
* first line: ''short description'' of the overall work (i.e., the feature you're working on). This will become the title of the review
* ''Summary'' field (optional): ''long description'' of the overall work; the field can continue in subsequent lines, up to the next field. This will become the "Summary" section of the review
* ''Test Plan'' field (optional): write here if something special is needed to test your change
* ''Reviewers'' field (optional): the (Phabricator) name(s) of desired reviewers. If you don't specify one (recommended) the default reviewers will be chosen
* ''Subscribers'' field (optional): the (Phabricator) name(s) of people that will be notified about changes to this review request. In most cases it should be left empty

For example:
<pre>
mercurial loader

Summary: first stab at a mercurial loader (T329)

The implementation follows the plan detailed in F2F discussion with @foo.

Performances seem decent enough for a first trial (XXX seconds for YYY repository
that contains ZZZ patches).

Test plan:

Reviewers:

Subscribers: foo
</pre>

After completing the message arc will submit the review request and tell you its number and URL:
<pre>
[...]
Created a new Differential revision:
Revision URI: https://forge.softwareheritage.org/Dxx
</pre>

=== Updating your branch to reflect requested changes ===

Your feature might get accepted as is, YAY!
Or, reviewers might request changes; no big deal!

Use the Differential web UI to follow-up to received comments, if needed.

To implement requested changes in the code, hack on your branch as usual by:

* adding new commits, and/or
* rewriting old commits with git rebase (to preserve a nice, easy to bisect history)

When you're ready to '''update your review request''':
<pre>
arc diff --update Dxx HEAD~
</pre>

Arc will prompt you for a message: describe what you've changed w.r.t. the previous review request, free form.
Your message will become the changelog entry in Differential for this new version of the diff.

Differential only care about the code diff, and not about the commits or their order.
Therefore each "update" can be a completely different series of commits, possibly rewritten from the previous submission.

=== Dependencies between diffs ===

Note that you can manage diff dependencies within the same module with the following keyword in the diff description:

```
Depends on Dxx
```

That allows to keep a logical view in your diff.
It's not strictly necessary (because the tooling now deals with it properly) but it might help reviewers or yourself to do so.

=== Landing your change onto master ===

Once your change has been approved in Differential, you will be able to land it onto the master branch.

Before doing so, you're encouraged to '''clean up your git commit history''', reordering/splitting/merging commits as needed to have separate logical commits and an easy to bisect history.
Update the diff [https://wiki.softwareheritage.org/wiki/Code_review_in_Phabricator#Updating_your_branch_to_reflect_requested_changes following the prior section].
(It'd be good to let the ci build finish to make sure everything is still green).

Once you're happy you can '''push to origin/master''' directly, e.g.:
<pre>
git checkout master
git merge --ff-only my-shiny-feature
git push
</pre>

<code>--ff-only</code> is optional, and makes sure you don't unintentionally create a merge commit.

Optionally you can then delete your local feature branch:
<pre>
git branch -d my-shiny-feature
</pre>

=== Reviewing locally / landing someone else's changes ===

You can do local reviews of code with arc patch:

<pre>
arc patch Dxyz
</pre>

This will create a branch '''arcpatch-Dxyz''' containing the changes on your local checkout.

You can then merge those changes upstream with

<pre>
git checkout master
git merge --ff arcpatch-Dxyz
git push origin master
</pre>

or, alternatively:

<pre>
arc land --squash
</pre>

== See also ==

* [[Code review]] for guidelines on how code is reviewed when developing for Software Heritage

[[Category:Software development]]

Code review in Phabricator

2020-07-29T12:33:00Z

Ardumont: /* Starting a new feature and submit it for review */

We use the [[Differential]] application of [[Phabricator]] to perform [[code review|code reviews]] in the context of [[Software Heritage]].

* we use Git and history.immutable=true (but beware as that is partly a Phabricator misnomer, read on)
* when code reviews are required, developers will be allowed to push directly to master once an accepted Differential diff exists

== Configuration ==

=== Arcanist configuration ===

When using git, [[Arcanist]] by default mess with the local history, rewriting commits at the time of first submission. 
To avoid that we use so called [https://secure.phabricator.com/book/phabricator/article/arcanist_new_project/#history-mutability-git history immutability].

To that end, you shall configure your <tt>arc</tt> accordingly:
<pre>
arc set-config history.immutable true
</pre>

Note that this does ''not'' mean that you are forbidden to rewrite your local branches (e.g., with <tt>git rebase</tt>).
Quite the contrary: you are encouraged to locally rewrite branches before pushing to ensure that commits are logically separated and your commit history easy to bisect.
The above setting just means that ''arc'' will not rewrite commit history under your nose.

=== Enabling <tt>git push</tt> to our forge ===

The way we've configured our review setup for continuous integration needs you to configure git to allow pushes to our forge. There's two ways you can do this : setting a ssh key to push over ssh, or setting a specific password for git pushes over https.

==== SSH key for pushes ====

In your forge User settings page (On the top right, click on your avatar, then click ''Settings''), you have access to a ''Authentication'' > ''SSH Public Keys'' section (Direct link: <tt>hxxps://forge.softwareheritage.org/settings/user/'''<your username>'''/page/ssh/</tt>). You then have the option to upload a SSH public key, which will authenticate your pushes.

You then need to configure ssh/git to use that key pair, for instance by editing the <tt>~/.ssh/config</tt> file.

Finally, you should configure git to push over ssh when pushing to https://forge.softwareheritage.org, by running the following command:
git config --global url.git@forge.softwareheritage.org:.pushInsteadOf https://forge.softwareheritage.org

This lets git know that it should use <tt>git@forge.softwareheritage.org:</tt> as a base url when pushing repositories cloned from forge.softwareheritage.org over https.

==== VCS password for pushes ====

If you're not comfortable setting up SSH to upload your changes, you have the option of setting a VCS password. This password, ''separate from your account password'', allows Phabricator to authenticate your uploads over HTTPS.

In your forge User settings page (On the top right, click on your avatar, then click ''Settings''), you need to use the ''Authentication'' > ''VCS Password'' section to set your VCS password (Direct link: <tt>hxxps://forge.softwareheritage.org/settings/user/'''<your username>'''/page/vcspassword/</tt>).

If you still get a 403 error on push, this means you need a forge administrator to enable HTTPS pushes for the repository (which wasn't done by default in historical repositories). Please drop by on IRC and let us know!

== Workflow ==

* work in a feature branch: <tt>git checkout -b my-feat</tt>
* initial review request: hack/commit/hack/commit ; <tt>arc diff origin/master</tt>
* react to change requests: hack/commit/hack/commit ; <tt>arc diff --update Dxx origin/master</tt>
* landing change: <tt>git checkout master ; git merge my-feat ; git push</tt>

=== Starting a new feature and submit it for review ===

Use a '''one branch per feature''' workflow, with well-separated ''logical commits'' ([https://wiki.softwareheritage.org/wiki/Git_style_guide following those conventions]).
Please open one diff per logical commit to keep the diff size to a minimum.

<pre>
git checkout -b my-shiny-feature
... hack hack hack ...
git commit -m 'architecture skeleton for my-shiny-feature'
... hack hack hack ...
git commit -m 'my-shiny-feature: implement module foo'
... etc ...
</pre>

Please, follow the
To '''submit your code for review''' the first time:
<pre>
arc diff origin/master
</pre>
arc will prompt for a '''code review message'''. Provide the following information:
* first line: ''short description'' of the overall work (i.e., the feature you're working on). This will become the title of the review
* ''Summary'' field (optional): ''long description'' of the overall work; the field can continue in subsequent lines, up to the next field. This will become the "Summary" section of the review
* ''Test Plan'' field (optional): write here if something special is needed to test your change
* ''Reviewers'' field (optional): the (Phabricator) name(s) of desired reviewers. If you don't specify one (recommended) the default reviewers will be chosen
* ''Subscribers'' field (optional): the (Phabricator) name(s) of people that will be notified about changes to this review request. In most cases it should be left empty

For example:
<pre>
mercurial loader

Summary: first stab at a mercurial loader (T329)

The implementation follows the plan detailed in F2F discussion with @foo.

Performances seem decent enough for a first trial (XXX seconds for YYY repository
that contains ZZZ patches).

Test plan:

Reviewers:

Subscribers: foo
</pre>

After completing the message arc will submit the review request and tell you its number and URL:
<pre>
[...]
Created a new Differential revision:
Revision URI: https://forge.softwareheritage.org/Dxx
</pre>

=== Updating your branch to reflect requested changes ===

Your feature might get accepted as is, YAY!
Or, reviewers might request changes; no big deal!

Use the Differential web UI to follow-up to received comments, if needed.

To implement requested changes in the code, hack on your branch as usual by:

* adding new commits, and/or
* rewriting old commits with git rebase (to preserve a nice, easy to bisect history)

When you're ready to '''update your review request''':
<pre>
arc diff --update Dxx origin/master
</pre>

Arc will prompt you for a message: describe what you've changed w.r.t. the previous review request, free form.
Your message will become the changelog entry in Differential for this new version of the diff.

Differential only care about the code diff, and not about the commits or their order.
Therefore each "update" can be a completely different series of commits, possibly rewritten from the previous submission.

=== Landing your change onto master ===

Once your change has been approved in Differential, you will be able to land it onto the master branch.

Before doing so, you're encouraged to '''clean up your git commit history''', reordering/splitting/merging commits as needed to have separate logical commits and an easy to bisect history.
Update the diff [https://wiki.softwareheritage.org/wiki/Code_review_in_Phabricator#Updating_your_branch_to_reflect_requested_changes following the prior section].
(It'd be good to let the ci build finish to make sure everything is still green).

Once you're happy you can '''push to origin/master''' directly, e.g.:
<pre>
git checkout master
git merge --ff-only my-shiny-feature
git push
</pre>

<code>--ff-only</code> is optional, and makes sure you don't unintentionally create a merge commit.

Optionally you can then delete your local feature branch:
<pre>
git branch -d my-shiny-feature
</pre>

=== Reviewing locally / landing someone else's changes ===

You can do local reviews of code with arc patch:

<pre>
arc patch Dxyz
</pre>

This will create a branch '''arcpatch-Dxyz''' containing the changes on your local checkout.

You can then merge those changes upstream with

<pre>
git checkout master
git merge --ff arcpatch-Dxyz
git push origin master
</pre>

or, alternatively:

<pre>
arc land --squash
</pre>

== See also ==

* [[Code review]] for guidelines on how code is reviewed when developing for Software Heritage

[[Category:Software development]]

Code review in Phabricator

2020-07-29T12:32:47Z

Ardumont: /* Starting a new feature and submit it for review */

We use the [[Differential]] application of [[Phabricator]] to perform [[code review|code reviews]] in the context of [[Software Heritage]].

* we use Git and history.immutable=true (but beware as that is partly a Phabricator misnomer, read on)
* when code reviews are required, developers will be allowed to push directly to master once an accepted Differential diff exists

== Configuration ==

=== Arcanist configuration ===

When using git, [[Arcanist]] by default mess with the local history, rewriting commits at the time of first submission. 
To avoid that we use so called [https://secure.phabricator.com/book/phabricator/article/arcanist_new_project/#history-mutability-git history immutability].

To that end, you shall configure your <tt>arc</tt> accordingly:
<pre>
arc set-config history.immutable true
</pre>

Note that this does ''not'' mean that you are forbidden to rewrite your local branches (e.g., with <tt>git rebase</tt>).
Quite the contrary: you are encouraged to locally rewrite branches before pushing to ensure that commits are logically separated and your commit history easy to bisect.
The above setting just means that ''arc'' will not rewrite commit history under your nose.

=== Enabling <tt>git push</tt> to our forge ===

The way we've configured our review setup for continuous integration needs you to configure git to allow pushes to our forge. There's two ways you can do this : setting a ssh key to push over ssh, or setting a specific password for git pushes over https.

==== SSH key for pushes ====

In your forge User settings page (On the top right, click on your avatar, then click ''Settings''), you have access to a ''Authentication'' > ''SSH Public Keys'' section (Direct link: <tt>hxxps://forge.softwareheritage.org/settings/user/'''<your username>'''/page/ssh/</tt>). You then have the option to upload a SSH public key, which will authenticate your pushes.

You then need to configure ssh/git to use that key pair, for instance by editing the <tt>~/.ssh/config</tt> file.

Finally, you should configure git to push over ssh when pushing to https://forge.softwareheritage.org, by running the following command:
git config --global url.git@forge.softwareheritage.org:.pushInsteadOf https://forge.softwareheritage.org

This lets git know that it should use <tt>git@forge.softwareheritage.org:</tt> as a base url when pushing repositories cloned from forge.softwareheritage.org over https.

==== VCS password for pushes ====

If you're not comfortable setting up SSH to upload your changes, you have the option of setting a VCS password. This password, ''separate from your account password'', allows Phabricator to authenticate your uploads over HTTPS.

In your forge User settings page (On the top right, click on your avatar, then click ''Settings''), you need to use the ''Authentication'' > ''VCS Password'' section to set your VCS password (Direct link: <tt>hxxps://forge.softwareheritage.org/settings/user/'''<your username>'''/page/vcspassword/</tt>).

If you still get a 403 error on push, this means you need a forge administrator to enable HTTPS pushes for the repository (which wasn't done by default in historical repositories). Please drop by on IRC and let us know!

== Workflow ==

* work in a feature branch: <tt>git checkout -b my-feat</tt>
* initial review request: hack/commit/hack/commit ; <tt>arc diff origin/master</tt>
* react to change requests: hack/commit/hack/commit ; <tt>arc diff --update Dxx origin/master</tt>
* landing change: <tt>git checkout master ; git merge my-feat ; git push</tt>

=== Starting a new feature and submit it for review ===

Use a '''one branch per feature''' workflow, with well-separated ''logical commits'' ([https://wiki.softwareheritage.org/wiki/Git_style_guide following those conventions])
Please open one diff per logical commit to keep the diff size to a minimum.

<pre>
git checkout -b my-shiny-feature
... hack hack hack ...
git commit -m 'architecture skeleton for my-shiny-feature'
... hack hack hack ...
git commit -m 'my-shiny-feature: implement module foo'
... etc ...
</pre>

Please, follow the
To '''submit your code for review''' the first time:
<pre>
arc diff origin/master
</pre>
arc will prompt for a '''code review message'''. Provide the following information:
* first line: ''short description'' of the overall work (i.e., the feature you're working on). This will become the title of the review
* ''Summary'' field (optional): ''long description'' of the overall work; the field can continue in subsequent lines, up to the next field. This will become the "Summary" section of the review
* ''Test Plan'' field (optional): write here if something special is needed to test your change
* ''Reviewers'' field (optional): the (Phabricator) name(s) of desired reviewers. If you don't specify one (recommended) the default reviewers will be chosen
* ''Subscribers'' field (optional): the (Phabricator) name(s) of people that will be notified about changes to this review request. In most cases it should be left empty

For example:
<pre>
mercurial loader

Summary: first stab at a mercurial loader (T329)

The implementation follows the plan detailed in F2F discussion with @foo.

Performances seem decent enough for a first trial (XXX seconds for YYY repository
that contains ZZZ patches).

Test plan:

Reviewers:

Subscribers: foo
</pre>

After completing the message arc will submit the review request and tell you its number and URL:
<pre>
[...]
Created a new Differential revision:
Revision URI: https://forge.softwareheritage.org/Dxx
</pre>

=== Updating your branch to reflect requested changes ===

Your feature might get accepted as is, YAY!
Or, reviewers might request changes; no big deal!

Use the Differential web UI to follow-up to received comments, if needed.

To implement requested changes in the code, hack on your branch as usual by:

* adding new commits, and/or
* rewriting old commits with git rebase (to preserve a nice, easy to bisect history)

When you're ready to '''update your review request''':
<pre>
arc diff --update Dxx origin/master
</pre>

Arc will prompt you for a message: describe what you've changed w.r.t. the previous review request, free form.
Your message will become the changelog entry in Differential for this new version of the diff.

Differential only care about the code diff, and not about the commits or their order.
Therefore each "update" can be a completely different series of commits, possibly rewritten from the previous submission.

=== Landing your change onto master ===

Once your change has been approved in Differential, you will be able to land it onto the master branch.

Before doing so, you're encouraged to '''clean up your git commit history''', reordering/splitting/merging commits as needed to have separate logical commits and an easy to bisect history.
Update the diff [https://wiki.softwareheritage.org/wiki/Code_review_in_Phabricator#Updating_your_branch_to_reflect_requested_changes following the prior section].
(It'd be good to let the ci build finish to make sure everything is still green).

Once you're happy you can '''push to origin/master''' directly, e.g.:
<pre>
git checkout master
git merge --ff-only my-shiny-feature
git push
</pre>

<code>--ff-only</code> is optional, and makes sure you don't unintentionally create a merge commit.

Optionally you can then delete your local feature branch:
<pre>
git branch -d my-shiny-feature
</pre>

=== Reviewing locally / landing someone else's changes ===

You can do local reviews of code with arc patch:

<pre>
arc patch Dxyz
</pre>

This will create a branch '''arcpatch-Dxyz''' containing the changes on your local checkout.

You can then merge those changes upstream with

<pre>
git checkout master
git merge --ff arcpatch-Dxyz
git push origin master
</pre>

or, alternatively:

<pre>
arc land --squash
</pre>

== See also ==

* [[Code review]] for guidelines on how code is reviewed when developing for Software Heritage

[[Category:Software development]]

Debian packaging

2020-05-27T15:22:35Z

Ardumont: Give more details

== Package repository ==

A package repository is available on https://debian.softwareheritage.org/.

Unstable / Testing :
deb [trusted=yes] https://debian.softwareheritage.org/ unstable main

Stable / Buster :
deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main

Oldstable / Stretch :
deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main

This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).

=== Uploading packages ===

Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.

For packages to be accepted, they need to be :
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt>
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders

== Git repositories for Debian packages ==

Our git repository structure for Debian packages is compatible with <tt>git-buildpackage</tt>.

We have two different ways of handling repositories for Debian packages:
* Packages of python modules where *we* are upstream
* Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)

For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:
* Our own modules are merged with the upstream repository
* External dependencies ignore the upstream repository and only have packaging branches.

=== Branch and tags structure ===

Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:
* debian/upstream (history of unpacked upstream releases)
* debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, stretch-swh)
* pristine-tar (data to regenerate upstream tarballs from a git export)

The name of the debian/upstream branch doesn't matter ''as long as it's properly configured in the <tt>debian/gbp.conf</tt> file''. It's only really used by <tt>gbp import-orig</tt> when importing a new release.

The tags marking upstream releases imported from tarballs for Debian packaging purposes are named <tt>debian/upstream/''<upstream version number>''</tt>.

Our Jenkins jobs are triggered on incoming tags named <tt>debian/''<version>''</tt>. To generate the proper tags, use <tt>gbp buildpackage --git-tag-only</tt>.

The git-buildpackage configuration, <tt>debian/gbp.conf</tt>, should be the following:
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/''<current suite>''
pristine-tar=True

==== Automatic packaging for swh python modules ====

The <tt>swh.*</tt> python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs <tt>gbp import-orig</tt> with the tarball we release to PyPI, and the right options to merge the upstream history.

To merge changes from the upstream history, we add the following option to <tt>gbp.conf</tt>:
upstream-vcs-tag=v%(version)s

=== Bootstrapping a dependency packaging repository ===

Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:

Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/
wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz

Create a new git repository
git init pytest-postgresql
cd pytest-postgresql

Import the original upstream version
git checkout -b debian/unstable-swh
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz
# What will be the source package name? [pytest-postgresql]
# What is the upstream version? [1.3.4]
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...
# gbp:info: Source package is pytest-postgresql
# gbp:info: Upstream version is 1.3.4
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz

Bootstrap the debian directory
mkdir -p debian/source
echo '3.0 (quilt)' > debian/source/format
echo 9 > debian/compat
cat > debian/gbp.conf << EOF
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/unstable-swh
pristine-tar=True
EOF
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules
vim debian/control
# [...] adapt debian/control from another package
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1 --distribution unstable-swh
vim debian/copyright
# [...] adapt debian/copyright from another package
git add debian
git commit -m "Initial packaging for pytest-postgresql"

You can then go on to try building the package. Once the package builds, if you want to check your package's conformance to Debian policy, you can run <tt>lintian</tt> on the changes:
lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes

Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository

We need to use a <tt>+swh1</tt> version suffix to avoid clashing with potential upstream Debian package versions.

==== Bootstrapping the backport branches ====

During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.

The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.

Checkout the new branch
git checkout debian/<version number>
git checkout -b debian/stretch-swh

Update the gbp config to match the branch
sed -i s/unstable-swh/stretch-swh/ debian/gbp.conf

Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)
dch -l "~bpo9" -D stretch-swh --force-distribution 'Rebuild for stretch-swh'

You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.

==== Setting up the repository on Phabricator ====

The repository on Phabricator needs the following settings:
* Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)
* Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)
* Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)
* Policy
** View: Public (no login required)
** Edit: Developers
** Push: All users (actual restrictions are handled by Herald rules)
* Activate the repository
* Look up the path to the repository on the storage tab

You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes.
ssh -t tate.internal.softwareheritage.org phabricator-setup-hook <repository-path> post-receive-debian-deps

==== Setting up the Jenkins jobs ====

The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/
They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs

Jobs for dependency packages are configured in <tt>jobs/dependency-packages.yaml</tt>. You can add a section as follows:

- project:
name: <Callsign>
display-name: <short-name>
pkg: <source-name>
jobs:
- 'dependency-jobs-{name}'

Use the regular review process to land your changes.
Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.

If your package needs extra repositories to build, you can add them as comma-separated values to the <tt>deb-extra-repositories</tt> setting, with the following notes:
* When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.
* When building packages for backports suites, the backports repository is automatically enabled.

=== Updating a dependency packaging repository ===

Place yourself on the debian/unstable-swh branch and "gbp import-origin" a more
recent upstream release tarballs.

For example (current version on 0.0.5, upstream bumped to 0.0.7):
gbp import-origin https://files.pythonhosted.org/packages/7a/bb/cf8fec6009e7d0cec52dc179d09b28c4c70d158e79b565e8aab7606e1717/attrs-strict-0.0.7.tar.gz

This will update the following branches:
* debian/upstream
* pristine-tar
* debian/unstable-swh

This also includes the necessary tags (`debian/upstream/0.0.7` here).

You then need to push all branches/tags to the repository:
git push origin --all --follow-tags

Ensure the [https://wiki.softwareheritage.org/wiki/Debian_packaging#Local_package_building update builds fine]
And [https://wiki.softwareheritage.org/wiki/Debian_packaging#Remote_package_building tags accordingly the debian/unstable-swh branch when ok].
Jenkins will then keep up on building the package.

=== Local package building ===

To locally test a package build, go on the appropriate debian packaging branch, and run
gbp buildpackage --git-builder=sbuild -As --no-clean-source

<tt>gbp buildpackage</tt> passes all options not starting with <tt>--git-</tt> to the builder. Some useful options are the following:

* <tt>--git-ignore-new</tt> builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.
* <tt>--no-clean-source</tt> doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies
* <tt>--extra-repository="'''repository specification'''"</tt> adds the given repository in the chroot before building.
* <tt>--extra-repository-key='''repository signing key'''</tt> adds the given key as a trusted gpg key for package sources
* <tt>--extra-package='''<.deb file or directory>'''</tt> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.
* <tt>--force-orig-source</tt> forces addition of the <tt>.orig.tar.gz</tt> file in the <tt>.changes</tt> file (useful when trying to upload a backport)

See <tt>gbp help buildpackage</tt> and <tt>man sbuild</tt> for a full description of all options

for example:
gbp buildpackage --git-builder=sbuild -As --force-orig-source --extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main'

(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping <tt>gbp buildpackage</tt> with the most common options)

=== Remote package building ===

Jenkins builds packages when the repository receives a tag.

Once the local build succeeds, tag the package with:
gbp buildpackage --git-tag-only --git-sign-tags

Alternatively, you can add the <tt>--git-tag</tt> option to your <tt>gbp buildpackage</tt> command so the tag happens automatically on a successful build.

Then, push your tag, and Jenkins jobs should get triggered
git push --tags

== Build Environment setup ==

Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.

=== sbuild setup ===

# Install the package
sudo apt-get install sbuild

# Add your user to the sbuild group, to allow him to use the sbuild commands
sudo sbuild-adduser $USER
# You have to logout and log back in

# Prepare chroots
sudo mkdir /srv/chroots
sudo mkdir /srv/chroots/var

# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/
sudo rm -r /var/lib/schroot
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot

sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/
sudo rm -r /var/lib/sbuild
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild
# end optionally

# Create unstable/sid chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/

# Create stretch chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates stretch /srv/chroots/stretch http://deb.debian.org/debian/

# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts
echo hosts >> /etc/schroot/sbuild/nssdatabases

=== schroot setup ===

Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.

You need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:

source-groups=root,sbuild
source-root-groups=root,sbuild
union-type=overlay

This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.

You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):

For unstable:
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild

For stretch:
aliases=stretch-swh-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild

==== dependencies cache ====

Add the following line to schroot's fstab /etc/schroot/sbuild/fstab
to permit reuse of existing fetched dependencies:

/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0

You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in <tt>/etc/apt/apt.conf.d</tt> on each chroot

=== schroot update ===

You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):

sudo sbuild-update -udcar sid; sudo sbuild-update -udcar stretch; sudo sbuild-update -ud jessie

=== environment setup ===

The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt>

export DEBFULLNAME="Debra Hacker"
export DEBEMAIL=debra.hacker@example.com

Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourfullkeyid></tt> variable.
(Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).

=== overlay in tmpfs for faster builds ===

You can add this to your fstab to put the overlay hierarchy in RAM:

tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0 0 0

=== Base packages ===

In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.

'''If you do so, do not use these chroots to upload to Debian itself!'''

[[Category:Software development]]
[[Category:System administration]]

Debian packaging

2020-05-27T15:10:03Z

Ardumont: Add chapter about updating a dependency packaging repository

== Package repository ==

A package repository is available on https://debian.softwareheritage.org/.

Unstable / Testing :
deb [trusted=yes] https://debian.softwareheritage.org/ unstable main

Stable / Buster :
deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main

Oldstable / Stretch :
deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main

This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).

=== Uploading packages ===

Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.

For packages to be accepted, they need to be :
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt>
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders

== Git repositories for Debian packages ==

Our git repository structure for Debian packages is compatible with <tt>git-buildpackage</tt>.

We have two different ways of handling repositories for Debian packages:
* Packages of python modules where *we* are upstream
* Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)

For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:
* Our own modules are merged with the upstream repository
* External dependencies ignore the upstream repository and only have packaging branches.

=== Branch and tags structure ===

Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:
* debian/upstream (history of unpacked upstream releases)
* debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, stretch-swh)
* pristine-tar (data to regenerate upstream tarballs from a git export)

The name of the debian/upstream branch doesn't matter ''as long as it's properly configured in the <tt>debian/gbp.conf</tt> file''. It's only really used by <tt>gbp import-orig</tt> when importing a new release.

The tags marking upstream releases imported from tarballs for Debian packaging purposes are named <tt>debian/upstream/''<upstream version number>''</tt>.

Our Jenkins jobs are triggered on incoming tags named <tt>debian/''<version>''</tt>. To generate the proper tags, use <tt>gbp buildpackage --git-tag-only</tt>.

The git-buildpackage configuration, <tt>debian/gbp.conf</tt>, should be the following:
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/''<current suite>''
pristine-tar=True

==== Automatic packaging for swh python modules ====

The <tt>swh.*</tt> python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs <tt>gbp import-orig</tt> with the tarball we release to PyPI, and the right options to merge the upstream history.

To merge changes from the upstream history, we add the following option to <tt>gbp.conf</tt>:
upstream-vcs-tag=v%(version)s

=== Bootstrapping a dependency packaging repository ===

Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:

Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/
wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz

Create a new git repository
git init pytest-postgresql
cd pytest-postgresql

Import the original upstream version
git checkout -b debian/unstable-swh
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz
# What will be the source package name? [pytest-postgresql]
# What is the upstream version? [1.3.4]
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...
# gbp:info: Source package is pytest-postgresql
# gbp:info: Upstream version is 1.3.4
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz

Bootstrap the debian directory
mkdir -p debian/source
echo '3.0 (quilt)' > debian/source/format
echo 9 > debian/compat
cat > debian/gbp.conf << EOF
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/unstable-swh
pristine-tar=True
EOF
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules
vim debian/control
# [...] adapt debian/control from another package
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1 --distribution unstable-swh
vim debian/copyright
# [...] adapt debian/copyright from another package
git add debian
git commit -m "Initial packaging for pytest-postgresql"

You can then go on to try building the package. Once the package builds, if you want to check your package's conformance to Debian policy, you can run <tt>lintian</tt> on the changes:
lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes

Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository

We need to use a <tt>+swh1</tt> version suffix to avoid clashing with potential upstream Debian package versions.

==== Bootstrapping the backport branches ====

During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.

The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.

Checkout the new branch
git checkout debian/<version number>
git checkout -b debian/stretch-swh

Update the gbp config to match the branch
sed -i s/unstable-swh/stretch-swh/ debian/gbp.conf

Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)
dch -l "~bpo9" -D stretch-swh --force-distribution 'Rebuild for stretch-swh'

You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.

==== Setting up the repository on Phabricator ====

The repository on Phabricator needs the following settings:
* Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)
* Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)
* Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)
* Policy
** View: Public (no login required)
** Edit: Developers
** Push: All users (actual restrictions are handled by Herald rules)
* Activate the repository
* Look up the path to the repository on the storage tab

You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes.
ssh -t tate.internal.softwareheritage.org phabricator-setup-hook <repository-path> post-receive-debian-deps

==== Setting up the Jenkins jobs ====

The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/
They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs

Jobs for dependency packages are configured in <tt>jobs/dependency-packages.yaml</tt>. You can add a section as follows:

- project:
name: <Callsign>
display-name: <short-name>
pkg: <source-name>
jobs:
- 'dependency-jobs-{name}'

Use the regular review process to land your changes.
Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.

If your package needs extra repositories to build, you can add them as comma-separated values to the <tt>deb-extra-repositories</tt> setting, with the following notes:
* When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.
* When building packages for backports suites, the backports repository is automatically enabled.

=== Updating a dependency packaging repository ===

Example (current version on 0.0.5, upstream bumped to 0.0.7):
gbp import-origin https://files.pythonhosted.org/packages/7a/bb/cf8fec6009e7d0cec52dc179d09b28c4c70d158e79b565e8aab7606e1717/attrs-strict-0.0.7.tar.gz

This will update the following branches:
* debian/upstream
* pristine-tar
* debian/unstable-swh
* debian/buster-swh

This also includes the necessary tags.

You then need to push all branches/tags to the repository for jenkins to do its
packaging work:
git push origin --all --follow-tags

=== Local package building ===

To locally test a package build, go on the appropriate debian packaging branch, and run
gbp buildpackage --git-builder=sbuild -As --no-clean-source

<tt>gbp buildpackage</tt> passes all options not starting with <tt>--git-</tt> to the builder. Some useful options are the following:

* <tt>--git-ignore-new</tt> builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.
* <tt>--no-clean-source</tt> doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies
* <tt>--extra-repository="'''repository specification'''"</tt> adds the given repository in the chroot before building.
* <tt>--extra-repository-key='''repository signing key'''</tt> adds the given key as a trusted gpg key for package sources
* <tt>--extra-package='''<.deb file or directory>'''</tt> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.
* <tt>--force-orig-source</tt> forces addition of the <tt>.orig.tar.gz</tt> file in the <tt>.changes</tt> file (useful when trying to upload a backport)

See <tt>gbp help buildpackage</tt> and <tt>man sbuild</tt> for a full description of all options

for example:
gbp buildpackage --git-builder=sbuild -As --force-orig-source --extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main'

(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping <tt>gbp buildpackage</tt> with the most common options)

=== Remote package building ===

Jenkins builds packages when the repository receives a tag.

Once the local build succeeds, tag the package with:
gbp buildpackage --git-tag-only --git-sign-tags

Alternatively, you can add the <tt>--git-tag</tt> option to your <tt>gbp buildpackage</tt> command so the tag happens automatically on a successful build.

Then, push your tag, and Jenkins jobs should get triggered
git push --tags

== Build Environment setup ==

Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.

=== sbuild setup ===

# Install the package
sudo apt-get install sbuild

# Add your user to the sbuild group, to allow him to use the sbuild commands
sudo sbuild-adduser $USER
# You have to logout and log back in

# Prepare chroots
sudo mkdir /srv/chroots
sudo mkdir /srv/chroots/var

# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/
sudo rm -r /var/lib/schroot
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot

sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/
sudo rm -r /var/lib/sbuild
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild
# end optionally

# Create unstable/sid chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/

# Create stretch chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates stretch /srv/chroots/stretch http://deb.debian.org/debian/

# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts
echo hosts >> /etc/schroot/sbuild/nssdatabases

=== schroot setup ===

Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.

You need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:

source-groups=root,sbuild
source-root-groups=root,sbuild
union-type=overlay

This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.

You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):

For unstable:
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild

For stretch:
aliases=stretch-swh-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild

==== dependencies cache ====

Add the following line to schroot's fstab /etc/schroot/sbuild/fstab
to permit reuse of existing fetched dependencies:

/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0

You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in <tt>/etc/apt/apt.conf.d</tt> on each chroot

=== schroot update ===

You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):

sudo sbuild-update -udcar sid; sudo sbuild-update -udcar stretch; sudo sbuild-update -ud jessie

=== environment setup ===

The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt>

export DEBFULLNAME="Debra Hacker"
export DEBEMAIL=debra.hacker@example.com

Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourfullkeyid></tt> variable.
(Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).

=== overlay in tmpfs for faster builds ===

You can add this to your fstab to put the overlay hierarchy in RAM:

tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0 0 0

=== Base packages ===

In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.

'''If you do so, do not use these chroots to upload to Debian itself!'''

[[Category:Software development]]
[[Category:System administration]]

Debian packaging

2019-09-04T07:51:27Z

Ardumont: /* schroot setup */

== Package repository ==

A package repository is available on https://debian.softwareheritage.org/.

Unstable / Testing :
deb [trusted=yes] https://debian.softwareheritage.org/ unstable main

Stable / Buster :
deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main

Oldstable / Stretch :
deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main

This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).

=== Uploading packages ===

Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.

For packages to be accepted, they need to be :
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt>
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders

== Git repositories for Debian packages ==

Our git repository structure for Debian packages is compatible with <tt>git-buildpackage</tt>.

We have two different ways of handling repositories for Debian packages:
* Packages of python modules where *we* are upstream
* Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)

For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:
* Our own modules are merged with the upstream repository
* External dependencies ignore the upstream repository and only have packaging branches.

=== Branch and tags structure ===

Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:
* debian/upstream (history of unpacked upstream releases)
* debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, stretch-swh)
* pristine-tar (data to regenerate upstream tarballs from a git export)

The name of the debian/upstream branch doesn't matter ''as long as it's properly configured in the <tt>debian/gbp.conf</tt> file''. It's only really used by <tt>gbp import-orig</tt> when importing a new release.

The tags marking upstream releases imported from tarballs for Debian packaging purposes are named <tt>debian/upstream/''<upstream version number>''</tt>.

Our Jenkins jobs are triggered on incoming tags named <tt>debian/''<version>''</tt>. To generate the proper tags, use <tt>gbp buildpackage --git-tag-only</tt>.

The git-buildpackage configuration, <tt>debian/gbp.conf</tt>, should be the following:
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/''<current suite>''
pristine-tar=True

==== Automatic packaging for swh python modules ====

The <tt>swh.*</tt> python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs <tt>gbp import-orig</tt> with the tarball we release to PyPI, and the right options to merge the upstream history.

To merge changes from the upstream history, we add the following option to <tt>gbp.conf</tt>:
upstream-vcs-tag=v%(version)s

=== Bootstrapping a dependency packaging repository ===

Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:

Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/
wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz

Create a new git repository
git init pytest-postgresql
cd pytest-postgresql

Import the original upstream version
git checkout -b debian/unstable-swh
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz
# What will be the source package name? [pytest-postgresql]
# What is the upstream version? [1.3.4]
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...
# gbp:info: Source package is pytest-postgresql
# gbp:info: Upstream version is 1.3.4
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz

Bootstrap the debian directory
mkdir -p debian/source
echo '3.0 (quilt)' > debian/source/format
echo 9 > debian/compat
cat > debian/gbp.conf << EOF
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/unstable-swh
pristine-tar=True
EOF
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules
vim debian/control
# [...] adapt debian/control from another package
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1 --distribution unstable-swh
vim debian/copyright
# [...] adapt debian/copyright from another package
git add debian
git commit -m "Initial packaging for pytest-postgresql"

You can then go on to try building the package. Once the package builds, if you want to check your package's conformance to Debian policy, you can run <tt>lintian</tt> on the changes:
lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes

Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository

We need to use a <tt>+swh1</tt> version suffix to avoid clashing with potential upstream Debian package versions.

==== Bootstrapping the backport branches ====

During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.

The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.

Checkout the new branch
git checkout debian/<version number>
git checkout -b debian/stretch-swh

Update the gbp config to match the branch
sed -i s/unstable-swh/stretch-swh/ debian/gbp.conf

Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)
dch -l "~bpo9" -D stretch-swh --force-distribution 'Rebuild for stretch-swh'

You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.

==== Setting up the repository on Phabricator ====

The repository on Phabricator needs the following settings:
* Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)
* Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)
* Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)
* Policy
** View: Public (no login required)
** Edit: Developers
** Push: All users (actual restrictions are handled by Herald rules)
* Activate the repository
* Look up the path to the repository on the storage tab

You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes.
ssh -t tate.internal.softwareheritage.org phabricator-setup-hook <repository-path> post-receive-debian-deps

==== Setting up the Jenkins jobs ====

The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/
They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs

Jobs for dependency packages are configured in <tt>jobs/dependency-packages.yaml</tt>. You can add a section as follows:

- project:
name: <Callsign>
display-name: <short-name>
pkg: <source-name>
jobs:
- 'dependency-jobs-{name}'

Use the regular review process to land your changes.
Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.

If your package needs extra repositories to build, you can add them as comma-separated values to the <tt>deb-extra-repositories</tt> setting, with the following notes:
* When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.
* When building packages for backports suites, the backports repository is automatically enabled.

=== Local package building ===

To locally test a package build, go on the appropriate debian packaging branch, and run
gbp buildpackage --git-builder=sbuild -As --no-clean-source

<tt>gbp buildpackage</tt> passes all options not starting with <tt>--git-</tt> to the builder. Some useful options are the following:

* <tt>--git-ignore-new</tt> builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.
* <tt>--no-clean-source</tt> doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies
* <tt>--extra-repository="'''repository specification'''"</tt> adds the given repository in the chroot before building.
* <tt>--extra-repository-key='''repository signing key'''</tt> adds the given key as a trusted gpg key for package sources
* <tt>--extra-package='''<.deb file or directory>'''</tt> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.
* <tt>--force-orig-source</tt> forces addition of the <tt>.orig.tar.gz</tt> file in the <tt>.changes</tt> file (useful when trying to upload a backport)

See <tt>gbp help buildpackage</tt> and <tt>man sbuild</tt> for a full description of all options

for example:
gbp buildpackage --git-builder=sbuild -As --force-orig-source --extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main'

(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping <tt>gbp buildpackage</tt> with the most common options)

=== Remote package building ===

Jenkins builds packages when the repository receives a tag.

Once the local build succeeds, tag the package with:
gbp buildpackage --git-tag-only --git-sign-tags

Alternatively, you can add the <tt>--git-tag</tt> option to your <tt>gbp buildpackage</tt> command so the tag happens automatically on a successful build.

Then, push your tag, and Jenkins jobs should get triggered
git push --tags

== Build Environment setup ==

Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.

=== sbuild setup ===

# Install the package
sudo apt-get install sbuild

# Add your user to the sbuild group, to allow him to use the sbuild commands
sudo sbuild-adduser $USER
# You have to logout and log back in

# Prepare chroots
sudo mkdir /srv/chroots
sudo mkdir /srv/chroots/var

# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/
sudo rm -r /var/lib/schroot
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot

sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/
sudo rm -r /var/lib/sbuild
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild
# end optionally

# Create unstable/sid chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/

# Create stretch chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates stretch /srv/chroots/stretch http://deb.debian.org/debian/

# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts
echo hosts >> /etc/schroot/sbuild/nssdatabases

=== schroot setup ===

Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.

You need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:

source-groups=root,sbuild
source-root-groups=root,sbuild
union-type=overlay

This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.

You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):

For unstable:
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild

For stretch:
aliases=stretch-swh-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild

==== dependencies cache ====

Add the following line to schroot's fstab /etc/schroot/sbuild/fstab
to permit reuse of existing fetched dependencies:

/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0

You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in <tt>/etc/apt/apt.conf.d</tt> on each chroot

=== schroot update ===

You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):

sudo sbuild-update -udcar sid; sudo sbuild-update -udcar stretch; sudo sbuild-update -ud jessie

=== environment setup ===

The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt>

export DEBFULLNAME="Debra Hacker"
export DEBEMAIL=debra.hacker@example.com

Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourfullkeyid></tt> variable.
(Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).

=== overlay in tmpfs for faster builds ===

You can add this to your fstab to put the overlay hierarchy in RAM:

tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0 0 0

=== Base packages ===

In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.

'''If you do so, do not use these chroots to upload to Debian itself!'''

[[Category:Software development]]
[[Category:System administration]]

Google Summer of Code 2019/Increase archive coverage

2019-08-26T13:14:28Z

Ardumont: /* Learnings: */

===Title:===
'''Increase archive coverage'''

=== Description:===

The goal of this project is to increase the archive coverage by making listers and loaders for different forges.
[https://docs.softwareheritage.org/devel/swh-lister/index.html#swh-lister Listers] are components that crawl the APIs of software forges (e.g., Bitbucket, GitHub, Sourceforge, ...) and return a list of the software available in it. Loaders take a bundle of software (tarball, Git repository ...) and load it into Software Heritage, by adapting it so that it matches the archive data model.

===Student: ===
Archit Agrawal
* [https://forge.softwareheritage.org/p/nahimilega/ Forge activity]

=== Mentors:===
* Nicolas Dandrimont
* Antoine R. Dumont

===Work Done:===
* '''Listers:'''
** Completed and merged
*** [https://forge.softwareheritage.org/rDLSfedfd73c8e4be8ce1d08b31c9a5cb99f9ca40fd6 Phabricator Lister]
*** [https://forge.softwareheritage.org/D1482 GNU Lister]
*** [https://forge.softwareheritage.org/rDLSa9a37a85bf9efac416cfdd152588bf01b7a063b2 CRAN Lister]
*** [https://forge.softwareheritage.org/D1584 Packagist Lister]
*** [https://forge.softwareheritage.org/D1610 CGit Lister]
** Did research on the methods that could be used to make following listers and made an implementation plan for the same
*** [https://forge.softwareheritage.org/T1734 Launchpad Lister]
*** [https://forge.softwareheritage.org/T1777 Rubygem Lister]
*** [https://forge.softwareheritage.org/T1718 NuGET(.NET) Lister]
*** [https://forge.softwareheritage.org/T1724 Maven Lister]
** [https://forge.softwareheritage.org/rDLS08ade29e6de0616a3964360454ab52b58c082b75 Add tests to PyPI Lister]
** [https://forge.softwareheritage.org/rDLSf424f07c7e628eb7a19d25f4fdb749682d97a21f Refactor base tests for listers]
** [https://forge.softwareheritage.org/D1441 Add documentation on *How to run a new lister*]
* '''Loaders:'''
** '''[https://forge.softwareheritage.org/T1389 Base Package Manager Loader]'''
*** Ingesting source code from package managers is a process somewhat similar for all of the package managers. This calls for a common base implementation for loading content from package managers into the archive. I worked on this idea, analysed the steps required to make a loader and the implementation of present package manager loader. Came up with the plan to implement the base loader and made the pass([https://forge.softwareheritage.org/D1694 D1694], [https://forge.softwareheritage.org/D1810 D1810], [https://forge.softwareheritage.org/D1811 D1811], [https://forge.softwareheritage.org/D1812 D1812], [https://forge.softwareheritage.org/D1813 D1813], [https://forge.softwareheritage.org/D1814 D1814], [https://forge.softwareheritage.org/D1744 D1744]). However, after the recommendation from my mentor, we changed the approach to make the base loader. Instead of making the whole base loader in one go, we decided to break it into multiple steps(3 steps) and follow the incremental approach.
**'''[https://forge.softwareheritage.org/D1824 GNU Loader]'''
*** As part of the first step towards the implementation of Base Loader, GNU Loader was implemented.

===TO-DO:===
* Implement the Listers using the research done and the implementation plan made for Launchpad, Rubygem.
* Find the workarounds to solve the challenges in making the Maven and NuGET(.NET) Lister.
* Work on the remaining steps in order to complete the Base Package Manager Loader.

=== Learnings: ===
Working in Software Heritage was a wholesome experience. I got to learn a new thing almost every day. Here is a few of the most prominent ones:
*Work on a huge codebase
*Plan and design before jumping to code
*Write clean and well-commented code
*Learn the difference between doing projects in college and in the industry(Spoiler Alert: '''A lot''')
*Multiple language integration in a python library (Used in CRAN Lister)
*Different programming methodologies explained to me by my mentors(eg [https://en.wikipedia.org/wiki/Test-driven_development TDD])
*Work with tools; DVCS (git), issue tracker (phabricator forge), containerization/virtualization (docker)

=== Activity reports:===
* May 2019
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-05/msg00003.html Week 20 Second Week (Community Bonding)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-05/msg00010.html Week 21 Third Week (Community Bonding)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-05/msg00017.html Week 22 First Week (Coding)]
* June 2019
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-06/msg00009.html Week 23 Second Week (Coding)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-06/msg00016.html Week 24 Third Week (Coding)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-06/msg00026.html Week 25 Fourth Week (Coding)(Work Summary)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-06/msg00033.html Week 26 Fifth Week (First Evaluation)]
* July 2019
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-07/msg00003.html Week 27 Sixth Week (Coding)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-07/msg00006.html Week 28 Seventh Week (Coding)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-07/msg00011.html Week 29 Eight Week (Coding)(Work Summary)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-07/msg00015.html Week 30 Nineth Week (Second Evaluation)]
* August 2019
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-08/msg00002.html Week 31 Tenth Week (Coding)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-08/msg00004.html Week 32 Eleventh Week (Coding)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-08/msg00008.html Week 33 Twelfth Week (Coding)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-08/msg00011.html Week 34 Thirteenth Week (Final Evaluation)]

== Links ==
* [https://forge.softwareheritage.org/source/swh-lister/ Lister source code repository]
* [https://forge.softwareheritage.org/source/swh-loader-core/ Loader source code repository]
* see project [https://summerofcode.withgoogle.com/projects/#5658995887439872 on the GSoC portal]

[[Category: Google Summer of Code]]
[[Category: Google Summer of Code 2019]]

Google Summer of Code 2019/Increase archive coverage

2019-08-26T12:38:50Z

Ardumont: /* Learnings: */

===Title:===
'''Increase archive coverage'''

=== Description:===
As Software Heritage works on archiving and sharing source code, one of the major tasks is to ingest the latest source code available in the database from time to time and from all the possible sources where you can fetch the source code using listers and ingest them using loaders. [https://docs.softwareheritage.org/devel/swh-lister/index.html#swh-lister Listers] are components that crawl the APIs of software forges (e.g., Bitbucket, GitHub, Sourceforge, ...) and return a list of the software available in it whereas [Loaders take a bundle of software (tarball, Git repository ...) and load it into Software Heritage, by adapting it so that it matches the archive data model. The goal of this project is to increase the archive coverage by making listers and loaders for different websites that which stores source code, so that Software Heritage can fetch as much source code as possible and store it in the database to preserve it for future generations.

===Student: ===
Archit Agrawal
* [https://forge.softwareheritage.org/p/nahimilega/ Forge activity]

=== Mentors:===
* Nicolas Dandrimont
* Antoine R. Dumont

===Work Done:===
* '''Listers:'''
** Completed and merged
*** [https://forge.softwareheritage.org/rDLSfedfd73c8e4be8ce1d08b31c9a5cb99f9ca40fd6 Phabricator Lister]
*** [https://forge.softwareheritage.org/D1482 GNU Lister]
*** [https://forge.softwareheritage.org/rDLSa9a37a85bf9efac416cfdd152588bf01b7a063b2 CRAN Lister]
*** [https://forge.softwareheritage.org/D1584 Packagist Lister]
*** [https://forge.softwareheritage.org/D1610 CGit Lister]
** Did research on the methods that could be used to make following listers and made an implementation plan for the same
*** [https://forge.softwareheritage.org/T1734 Launchpad Lister]
*** [https://forge.softwareheritage.org/T1777 Rubygem Lister]
*** [https://forge.softwareheritage.org/T1718 NuGET(.NET) Lister]
*** [https://forge.softwareheritage.org/T1724 Maven Lister]
** [https://forge.softwareheritage.org/rDLS08ade29e6de0616a3964360454ab52b58c082b75 Add tests to PyPI Lister]
** [https://forge.softwareheritage.org/rDLSf424f07c7e628eb7a19d25f4fdb749682d97a21f Refactor base tests for listers]
** [https://forge.softwareheritage.org/D1441 Add documentation on *How to run a new lister*]
* '''Loaders:'''
** '''[https://forge.softwareheritage.org/T1389 Base Package Manager Loader]'''
*** Ingesting source code from package managers is a process somewhat similar for all of the package managers. This calls for a common base implementation for loading content from package managers into the archive. I worked on this idea, analysed the steps required to make a loader and the implementation of present package manager loader. Came up with the plan to implement the base loader and made the pass([https://forge.softwareheritage.org/D1694 D1694], [https://forge.softwareheritage.org/D1810 D1810], [https://forge.softwareheritage.org/D1811 D1811], [https://forge.softwareheritage.org/D1812 D1812], [https://forge.softwareheritage.org/D1813 D1813], [https://forge.softwareheritage.org/D1814 D1814], [https://forge.softwareheritage.org/D1744 D1744]). However, after the recommendation from my mentor, we changed the approach to make the base loader. Instead of making the whole base loader in one go, we decided to break it into multiple steps(3 steps) and follow the incremental approach.
**'''[https://forge.softwareheritage.org/D1824 GNU Loader]'''
*** As part of the first step towards the implementation of Base Loader, GNU Loader was implemented.

===TO-DO:===
* Implement the Listers using the research done and the implementation plan made for Launchpad, Rubygem.
* Find the workarounds to solve the challenges in making the Maven and NuGET(.NET) Lister.
* Work on the remaining steps in order to complete the Base Package Manager Loader.

=== Learnings: ===
Working in Software Heritage was a wholesome experience. I got to learn a new thing almost every day. Here is a few of the most prominent ones:
*Work on a huge codebase
*Plan and design before jumping to code
*Write clean and well-commented code
*Learn the difference between doing projects in college and in the industry(Spoiler Alert: '''A lot''')
*Multiple language integration in a python library (Used in CRAN Lister)
*Different programming methodologies explained to me by my mentors(eg [https://en.wikipedia.org/wiki/Test-driven_development TDD])
*Work with tools; DVCS (git), issue tracker (phabricator forge), docker

=== Activity reports:===
* May 2019
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-05/msg00003.html Week 20 Second Week (Community Bonding)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-05/msg00010.html Week 21 Third Week (Community Bonding)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-05/msg00017.html Week 22 First Week (Coding)]
* June 2019
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-06/msg00009.html Week 23 Second Week (Coding)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-06/msg00016.html Week 24 Third Week (Coding)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-06/msg00026.html Week 25 Fourth Week (Coding)(Work Summary)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-06/msg00033.html Week 26 Fifth Week (First Evaluation)]
* July 2019
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-07/msg00003.html Week 27 Sixth Week (Coding)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-07/msg00006.html Week 28 Seventh Week (Coding)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-07/msg00011.html Week 29 Eight Week (Coding)(Work Summary)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-07/msg00015.html Week 30 Nineth Week (Second Evaluation)]
* August 2019
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-08/msg00002.html Week 31 Tenth Week (Coding)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-08/msg00004.html Week 32 Eleventh Week (Coding)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-08/msg00008.html Week 33 Twelfth Week (Coding)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-08/msg00011.html Week 34 Thirteenth Week (Final Evaluation)]

== Links ==
* [https://forge.softwareheritage.org/source/swh-lister/ Lister source code repository]
* [https://forge.softwareheritage.org/source/swh-loader-core/ Loader source code repository]
* see project [https://summerofcode.withgoogle.com/projects/#5658995887439872 on the GSoC portal]

[[Category: Google Summer of Code]]
[[Category: Google Summer of Code 2019]]

Google Summer of Code 2019/Increase archive coverage

2019-08-26T12:30:45Z

Ardumont: /* Work Done: */

===Title:===
'''Increase archive coverage'''

=== Description:===
As Software Heritage works on archiving and sharing source code, one of the major tasks is to ingest the latest source code available in the database from time to time and from all the possible sources where you can fetch the source code using listers and ingest them using loaders. [https://docs.softwareheritage.org/devel/swh-lister/index.html#swh-lister Listers] are components that crawl the APIs of software forges (e.g., Bitbucket, GitHub, Sourceforge, ...) and return a list of the software available in it whereas [Loaders take a bundle of software (tarball, Git repository ...) and load it into Software Heritage, by adapting it so that it matches the archive data model. The goal of this project is to increase the archive coverage by making listers and loaders for different websites that which stores source code, so that Software Heritage can fetch as much source code as possible and store it in the database to preserve it for future generations.

===Student: ===
Archit Agrawal
* [https://forge.softwareheritage.org/p/nahimilega/ Forge activity]

=== Mentors:===
* Nicolas Dandrimont
* Antoine R. Dumont

===Work Done:===
* '''Listers:'''
** Completed and merged
*** [https://forge.softwareheritage.org/rDLSfedfd73c8e4be8ce1d08b31c9a5cb99f9ca40fd6 Phabricator Lister]
*** [https://forge.softwareheritage.org/D1482 GNU Lister]
*** [https://forge.softwareheritage.org/rDLSa9a37a85bf9efac416cfdd152588bf01b7a063b2 CRAN Lister]
*** [https://forge.softwareheritage.org/D1584 Packagist Lister]
*** [https://forge.softwareheritage.org/D1610 CGit Lister]
** Did research on the methods that could be used to make following listers and made an implementation plan for the same
*** [https://forge.softwareheritage.org/T1734 Launchpad Lister]
*** [https://forge.softwareheritage.org/T1777 Rubygem Lister]
*** [https://forge.softwareheritage.org/T1718 NuGET(.NET) Lister]
*** [https://forge.softwareheritage.org/T1724 Maven Lister]
** [https://forge.softwareheritage.org/rDLS08ade29e6de0616a3964360454ab52b58c082b75 Add tests to PyPI Lister]
** [https://forge.softwareheritage.org/rDLSf424f07c7e628eb7a19d25f4fdb749682d97a21f Refactor base tests for listers]
** [https://forge.softwareheritage.org/D1441 Add documentation on *How to run a new lister*]
* '''Loaders:'''
** '''[https://forge.softwareheritage.org/T1389 Base Package Manager Loader]'''
*** Ingesting source code from package managers is a process somewhat similar for all of the package managers. This calls for a common base implementation for loading content from package managers into the archive. I worked on this idea, analysed the steps required to make a loader and the implementation of present package manager loader. Came up with the plan to implement the base loader and made the pass([https://forge.softwareheritage.org/D1694 D1694], [https://forge.softwareheritage.org/D1810 D1810], [https://forge.softwareheritage.org/D1811 D1811], [https://forge.softwareheritage.org/D1812 D1812], [https://forge.softwareheritage.org/D1813 D1813], [https://forge.softwareheritage.org/D1814 D1814], [https://forge.softwareheritage.org/D1744 D1744]). However, after the recommendation from my mentor, we changed the approach to make the base loader. Instead of making the whole base loader in one go, we decided to break it into multiple steps(3 steps) and follow the incremental approach.
**'''[https://forge.softwareheritage.org/D1824 GNU Loader]'''
*** As part of the first step towards the implementation of Base Loader, GNU Loader was implemented.

===TO-DO:===
* Implement the Listers using the research done and the implementation plan made for Launchpad, Rubygem.
* Find the workarounds to solve the challenges in making the Maven and NuGET(.NET) Lister.
* Work on the remaining steps in order to complete the Base Package Manager Loader.

=== Learnings: ===
Working in Software Heritage was a wholesome experience. I got to learn a new thing almost every day. It would me injustice id I say I can account all my learnings in a section of a blog, however here are a list of few of most prominent once:
*Working on a huge codebase
*Plan and design before jumping to code
*Writing clean and well-commented code
*Difference between doing projects in college and in the industry(Spoiler Alert: '''A lot''')
*Multiple language integration in a python library (Used in CRAN Lister)
*Different programming methodologies explained to me by my mentors(eg [https://en.wikipedia.org/wiki/Test-driven_development TDD])
*Working with git and forge
*Working with Docker

=== Activity reports:===
* May 2019
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-05/msg00003.html Week 20 Second Week (Community Bonding)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-05/msg00010.html Week 21 Third Week (Community Bonding)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-05/msg00017.html Week 22 First Week (Coding)]
* June 2019
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-06/msg00009.html Week 23 Second Week (Coding)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-06/msg00016.html Week 24 Third Week (Coding)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-06/msg00026.html Week 25 Fourth Week (Coding)(Work Summary)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-06/msg00033.html Week 26 Fifth Week (First Evaluation)]
* July 2019
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-07/msg00003.html Week 27 Sixth Week (Coding)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-07/msg00006.html Week 28 Seventh Week (Coding)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-07/msg00011.html Week 29 Eight Week (Coding)(Work Summary)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-07/msg00015.html Week 30 Nineth Week (Second Evaluation)]
* August 2019
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-08/msg00002.html Week 31 Tenth Week (Coding)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-08/msg00004.html Week 32 Eleventh Week (Coding)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-08/msg00008.html Week 33 Twelfth Week (Coding)]
** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-08/msg00011.html Week 34 Thirteenth Week (Final Evaluation)]

== Links ==
* [https://forge.softwareheritage.org/source/swh-lister/ Lister source code repository]
* [https://forge.softwareheritage.org/source/swh-loader-core/ Loader source code repository]
* see project [https://summerofcode.withgoogle.com/projects/#5658995887439872 on the GSoC portal]

[[Category: Google Summer of Code]]
[[Category: Google Summer of Code 2019]]

Report issues on phabricator

2019-03-18T23:30:39Z

Ardumont: Bootstrap report on phabricator

When you decide that there is a bug, it is important to
report it in a useful way in [https://forge.softwareheritage.org/task our forge].

What is most useful is an exact description of what commands you executed, until the problem happens.

If you feel the command and the stack trace are too long, you can use a paste
software. For example, [https://forge.softwareheritage.org/paste our forge] proposes one.
That's easier to discuss it in the #swh-devel irc channel by referencing it or even attach it to the task.

There is a bot in the irc channel which will automatically create a summary of the task/paste when mentioned.
In the forge, automatic linking happens when mentioning phabricator objects.

Note:
Inspired from [https://www.gnu.org/software/emacs/draft/manual/html_node/emacs/Understanding-Bug-Reporting.html the emacs manual]

Code review in Phabricator

2019-03-13T12:32:43Z

Ardumont: /* Landing your change onto master */

We use the [[Differential]] application of [[Phabricator]] to perform [[code review|code reviews]] in the context of [[Software Heritage]].

* we use Git and history.immutable=true (but beware as that is partly a Phabricator misnomer, read on)
* when code reviews are required, developers will be allowed to push directly to master once an accepted Differential diff exists

== Configuration ==

=== Arcanist configuration ===

When using git, [[Arcanist]] by default mess with the local history, rewriting commits at the time of first submission. 
To avoid that we use so called [https://secure.phabricator.com/book/phabricator/article/arcanist_new_project/#history-mutability-git history immutability].

To that end, you shall configure your <tt>arc</tt> accordingly:
<pre>
arc set-config history.immutable true
</pre>

Note that this does ''not'' mean that you are forbidden to rewrite your local branches (e.g., with <tt>git rebase</tt>).
Quite the contrary: you are encouraged to locally rewrite branches before pushing to ensure that commits are logically separated and your commit history easy to bisect.
The above setting just means that ''arc'' will not rewrite commit history under your nose.

=== Enabling <tt>git push</tt> to our forge ===

The way we've configured our review setup for continuous integration needs you to configure git to allow pushes to our forge. There's two ways you can do this : setting a ssh key to push over ssh, or setting a specific password for git pushes over https.

==== SSH key for pushes ====

In your forge User settings page (On the top right, click on your avatar, then click ''Settings''), you have access to a ''Authentication'' > ''SSH Public Keys'' section (Direct link: <tt>hxxps://forge.softwareheritage.org/settings/user/'''<your username>'''/page/ssh/</tt>). You then have the option to upload a SSH public key, which will authenticate your pushes.

You then need to configure ssh/git to use that key pair, for instance by editing the <tt>~/.ssh/config</tt> file.

Finally, you should configure git to push over ssh when pushing to https://forge.softwareheritage.org, by running the following command:
git config --global url.git@forge.softwareheritage.org:.pushInsteadOf https://forge.softwareheritage.org

This lets git know that it should use <tt>git@forge.softwareheritage.org:</tt> as a base url when pushing repositories cloned from forge.softwareheritage.org over https.

==== VCS password for pushes ====

If you're not comfortable setting up SSH to upload your changes, you have the option of setting a VCS password. This password, ''separate from your account password'', allows Phabricator to authenticate your uploads over HTTPS.

In your forge User settings page (On the top right, click on your avatar, then click ''Settings''), you need to use the ''Authentication'' > ''VCS Password'' section to set your VCS password (Direct link: <tt>hxxps://forge.softwareheritage.org/settings/user/'''<your username>'''/page/vcspassword/</tt>).

== Workflow ==

* work in a feature branch: <tt>git checkout -b my-feat</tt>
* initial review request: hack/commit/hack/commit ; <tt>arc diff origin/master</tt>
* react to change requests: hack/commit/hack/commit ; <tt>arc diff --update Dxx origin/master</tt>
* landing change: <tt>git checkout master ; git merge my-feat ; git push</tt>

=== Starting a new feature and submit it for review ===

Use a '''one branch per feature''' workflow, with well-separated ''logical commits'' ([https://wiki.softwareheritage.org/wiki/Git_style_guide following those conventions])
<pre>
git checkout -b my-shiny-feature
... hack hack hack ...
git commit -m 'architecture skeleton for my-shiny-feature'
... hack hack hack ...
git commit -m 'my-shiny-feature: implement module foo'
... etc ...
</pre>

Please, follow the
To '''submit your code for review''' the first time:
<pre>
arc diff origin/master
</pre>
arc will prompt for a '''code review message'''. Provide the following information:
* first line: ''short description'' of the overall work (i.e., the feature you're working on). This will become the title of the review
* ''Summary'' field (optional): ''long description'' of the overall work; the field can continue in subsequent lines, up to the next field. This will become the "Summary" section of the review
* ''Test Plan'' field (optional): write here if something special is needed to test your change
* ''Reviewers'' field (optional): the (Phabricator) name(s) of desired reviewers. If you don't specify one (recommended) the default reviewers will be chosen
* ''Subscribers'' field (optional): the (Phabricator) name(s) of people that will be notified about changes to this review request. In most cases it should be left empty

For example:
<pre>
mercurial loader

Summary: first stab at a mercurial loader (T329)

The implementation follows the plan detailed in F2F discussion with @foo.

Performances seem decent enough for a first trial (XXX seconds for YYY repository
that contains ZZZ patches).

Test plan:

Reviewers:

Subscribers: foo
</pre>

After completing the message arc will submit the review request and tell you its number and URL:
<pre>
[...]
Created a new Differential revision:
Revision URI: https://forge.softwareheritage.org/Dxx
</pre>

=== Updating your branch to reflect requested changes ===

Your feature might get accepted as is, YAY!
Or, reviewers might request changes; no big deal!

Use the Differential web UI to follow-up to received comments, if needed.

To implement requested changes in the code, hack on your branch as usual by:

* adding new commits, and/or
* rewriting old commits with git rebase (to preserve a nice, easy to bisect history)

When you're ready to '''update your review request''':
<pre>
arc diff --update Dxx origin/master
</pre>

Arc will prompt you for a message: describe what you've changed w.r.t. the previous review request, free form.
Your message will become the changelog entry in Differential for this new version of the diff.

Differential only care about the code diff, and not about the commits or their order.
Therefore each "update" can be a completely different series of commits, possibly rewritten from the previous submission.

=== Landing your change onto master ===

Once your change has been approved in Differential, you will be able to land it onto the master branch.

Before doing so, you're encouraged to '''clean up your git commit history''', reordering/splitting/merging commits as needed to have separate logical commits and an easy to bisect history.
Update the diff [https://wiki.softwareheritage.org/wiki/Code_review_in_Phabricator#Updating_your_branch_to_reflect_requested_changes following the prior section].
(It'd be good to let the ci build finish to make sure everything is still green).

Once you're happy you can '''push to origin/master''' directly, e.g.:
<pre>
git checkout master
git merge my-shiny-feature
git push
</pre>

Optionally you can then delete your local feature branch:
<pre>
git branch -d my-shiny-feature
</pre>

=== Reviewing locally / landing someone else's changes ===

You can do local reviews of code with arc patch:

<pre>
arc patch Dxyz
</pre>

This will create a branch '''arcpatch-Dxyz''' containing the changes on your local checkout.

You can then merge those changes upstream with

<pre>
git checkout master
git merge --ff arcpatch-Dxyz
git push origin master
</pre>

== See also ==

* [[Code review]] for guidelines on how code is reviewed when developing for Software Heritage

[[Category:Software development]]

Graph compression on the development history of software (internship)

2019-03-12T21:17:21Z

Ardumont: /* Graph compression on the development history of software */

== Graph compression on the development history of software ==

'''Context''': [https://www.softwareheritage.org/ Software Heritage] is an
ambitious research project whose goal is to collect, preserve in the very long
term, and share the whole publicly accessible Free/Open Source Software
(FOSS) in source code form.

'''Description''': The Software Heritage
[https://docs.softwareheritage.org/devel/swh-model/data-model.html data model]
is a big [https://en.wikipedia.org/wiki/Merkle_tree Merkle] DAG made of nodes
like revisions, releases, directories, etc. It is a very big graph, with ~10 B
nodes and ~100 B edges, which makes it hard to fit in memory using naive
approaches. Graph compression techniques have been successfully used to compress
the Web graph (which is slightly larger than the Software Heritage one) and make
it fit in memory. The goal of this internship is review existing graph
compression techniques and apply the most appropriate one to the Software
Heritage case, enabling in-memory processing of its Merkle DAG.

'''Desirable skills''' to obtain this internship:
* familiarity with system programming
* familiarity with basic graph algorithms
* C development
* working knowledge of basic compression techniques from information theory would be a plus

'''Workplace''': Inria Paris

'''Environment''': you will work shoulder to shoulder with all members of the
Software Heritage team, and you will have a chance to witness from within the
construction of the ultimate source code archive.

'''Internship mentors''':
* Francesca Bassi <francesca.bassi@l2s.centralesupelec.fr>
* Stefano Zacchiroli <zack@upsilon.cc>

'''References''':
* Paolo Boldi, Sebastiano Vigna, ''The webgraph framework I: compression techniques'', Proceedings of the 13th international conference on World Wide Web. ACM, 2004. [http://www.web.ethz.ch/CDstore/www2004/docs/1p595.pdf pdf]
* Paolo Boldi, Marco Rosa, Massimo Santini, Sebastiano Vigna, ''Layered label propagation: A multiresolution coordinate-free ordering for compressing social networks''. [https://arxiv.org/pdf/1011.5425 pdf]
* [http://webgraph.di.unimi.it WebGraph] (Java framework that implements the techniques above)

[[Category:Available internship]]
[[Category:Internship]]
[[Category:Lang:English]]

Debian packaging

2019-02-16T08:49:06Z

Ardumont: /* Bootstrapping the backport branches */

== Package repository ==

A package repository is available on https://debian.softwareheritage.org/.

Unstable / Testing :
deb [trusted=yes] https://debian.softwareheritage.org/ unstable main

Stable / Stretch :
deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main

This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).

=== Uploading packages ===

Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.

For packages to be accepted, they need to be :
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt>
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders

== Git repositories for Debian packages ==

Our git repository structure for Debian packages is compatible with <tt>git-buildpackage</tt>.

We have two different ways of handling repositories for Debian packages:
* Packages of python modules where *we* are upstream
* Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)

For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:
* Our own modules are merged with the upstream repository
* External dependencies ignore the upstream repository and only have packaging branches.

=== Branch and tags structure ===

Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:
* debian/upstream (history of unpacked upstream releases)
* debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, stretch-swh)
* pristine-tar (data to regenerate upstream tarballs from a git export)

The name of the debian/upstream branch doesn't matter ''as long as it's properly configured in the <tt>debian/gbp.conf</tt> file''. It's only really used by <tt>gbp import-orig</tt> when importing a new release.

The tags marking upstream releases imported from tarballs for Debian packaging purposes are named <tt>debian/upstream/''<upstream version number>''</tt>.

Our Jenkins jobs are triggered on incoming tags named <tt>debian/''<version>''</tt>. To generate the proper tags, use <tt>gbp buildpackage --git-tag-only</tt>.

The git-buildpackage configuration, <tt>debian/gbp.conf</tt>, should be the following:
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/''<current suite>''
pristine-tar=True

==== Automatic packaging for swh python modules ====

The <tt>swh.*</tt> python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs <tt>gbp import-orig</tt> with the tarball we release to PyPI, and the right options to merge the upstream history.

To merge changes from the upstream history, we add the following option to <tt>gbp.conf</tt>:
upstream-vcs-tag=v%(version)s

=== Bootstrapping a dependency packaging repository ===

Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:

Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/
wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz

Create a new git repository
mkdir pytest-postgresql
cd pytest-postgresql
git init

Import the original upstream version
git checkout -b debian/unstable-swh
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz
# What will be the source package name? [pytest-postgresql]
# What is the upstream version? [1.3.4]
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...
# gbp:info: Source package is pytest-postgresql
# gbp:info: Upstream version is 1.3.4
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz

Bootstrap the debian directory
mkdir debian
mkdir debian/source
echo '3.0 (quilt)' > debian/source/format
cat > debian/gbp.conf << EOF
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/unstable-swh
pristine-tar=True
EOF
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules
vim debian/control
# [...] adapt debian/control from another package
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1
vim debian/copyright
# [...] adapt debian/copyright from another package
git add debian
git commit -m "Initial packaging for pytest-postgresql"

You can then go on to try building the package. Once the package builds, if you want to check your package's conformance to Debian policy, you can run <tt>lintian</tt> on the changes:
lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes

Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository

We need to use a <tt>+swh1</tt> version suffix to avoid clashing with potential upstream Debian package versions.

==== Bootstrapping the backport branches ====

During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.

The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.

Checkout the new branch
git checkout debian/<version number>
git checkout -b debian/stretch-swh

Update the gbp config to match the branch
sed -i s/unstable-swh/stretch-swh/ debian/gbp.conf

Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)
dch -l "~bpo9" -D stretch-swh --force-distribution 'Rebuild for stretch-swh'

You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.

==== Setting up the repository on Phabricator ====

The repository on Phabricator needs the following settings:
* Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)
* Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)
* Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)
* Policy
** View: Public (no login required)
** Edit: Developers
** Push: All users (actual restrictions are handled by Herald rules)
* Activate the repository
* Look up the path to the repository on the storage tab

You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes.
ssh -t tate.internal.softwareheritage.org phabricator-setup-hook <repository-path> post-receive-debian-deps

==== Setting up the Jenkins jobs ====

The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/
They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs

Jobs for dependency packages are configured in <tt>jobs/dependency-packages.yaml</tt>. You can add a section as follows:

- project:
name: <Callsign>
display-name: <short-name>
pkg: <source-name>
jobs:
- 'dependency-jobs-{name}'

Use the regular review process to land your changes.
Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.

If your package needs extra repositories to build, you can add them as comma-separated values to the <tt>deb-extra-repositories</tt> setting, with the following notes:
* When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.
* When building packages for backports suites, the backports repository is automatically enabled.

=== Local package building ===

To locally test a package build, go on the appropriate debian packaging branch, and run
gbp buildpackage --git-builder=sbuild -As --no-clean-source

<tt>gbp buildpackage</tt> passes all options not starting with <tt>--git-</tt> to the builder. Some useful options are the following:

* <tt>--git-ignore-new</tt> builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.
* <tt>--no-clean-source</tt> doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies
* <tt>--extra-repository="'''repository specification'''"</tt> adds the given repository in the chroot before building.
* <tt>--extra-repository-key='''repository signing key'''</tt> adds the given key as a trusted gpg key for package sources
* <tt>--extra-package='''<.deb file or directory>'''</tt> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.
* <tt>--force-orig-source</tt> forces addition of the <tt>.orig.tar.gz</tt> file in the <tt>.changes</tt> file (useful when trying to upload a backport)

See <tt>gbp help buildpackage</tt> and <tt>man sbuild</tt> for a full description of all options

for example:
gbp buildpackage --git-builder=sbuild -As --force-orig-source --extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main'

(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping <tt>gbp buildpackage</tt> with the most common options)

=== Remote package building ===

Jenkins builds packages when the repository receives a tag.

Once the local build succeeds, tag the package with:
gbp buildpackage --git-tag-only --git-sign-tags

Alternatively, you can add the <tt>--git-tag</tt> option to your <tt>gbp buildpackage</tt> command so the tag happens automatically on a successful build.

Then, push your tag, and Jenkins jobs should get triggered
git push --tags

== Build Environment setup ==

Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.

=== sbuild setup ===

# Install the package
sudo apt-get install sbuild

# Add your user to the sbuild group, to allow him to use the sbuild commands
sudo sbuild-adduser $USER
# You have to logout and log back in

# Prepare chroots
sudo mkdir /srv/chroots
sudo mkdir /srv/chroots/var

# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/
sudo rm -r /var/lib/schroot
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot

sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/
sudo rm -r /var/lib/sbuild
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild
# end optionally

# Create unstable/sid chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/

# Create stretch chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates stretch /srv/chroots/stretch http://deb.debian.org/debian/

# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts
echo hosts >> /etc/schroot/sbuild/nssdatabases

=== schroot setup ===

Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.

You need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:

source-groups=root,sbuild
source-root-groups=root,sbuild
union-type=overlay

This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.

You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):

For unstable:
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild

For stretch:
aliases=stable-amd64-sbuild,stable-backports-amd64-sbuild,stretch-swh-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild

==== dependencies cache ====

Add the following line to schroot's fstab /etc/schroot/sbuild/fstab
to permit reuse of existing fetched dependencies:

/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0

You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in <tt>/etc/apt/apt.conf.d</tt> on each chroot

=== schroot update ===

You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):

sudo sbuild-update -udcar sid; sudo sbuild-update -udcar stretch; sudo sbuild-update -ud jessie

=== environment setup ===

The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt>

export DEBFULLNAME="Debra Hacker"
export DEBEMAIL=debra.hacker@example.com

Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourfullkeyid></tt> variable.
(Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).

=== overlay in tmpfs for faster builds ===

You can add this to your fstab to put the overlay hierarchy in RAM:

tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0 0 0

=== Base packages ===

In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.

'''If you do so, do not use these chroots to upload to Debian itself!'''

[[Category:Software development]]
[[Category:System administration]]

Code review in Phabricator

2019-02-14T09:59:05Z

Ardumont: /* Starting a new feature and submit it for review */

We use the [[Differential]] application of [[Phabricator]] to perform [[code review|code reviews]] in the context of [[Software Heritage]].

* we use Git and history.immutable=true (but beware as that is partly a Phabricator misnomer, read on)
* when code reviews are required, developers will be allowed to push directly to master once an accepted Differential diff exists

== Configuration ==

=== Arcanist configuration ===

When using git, [[Arcanist]] by default mess with the local history, rewriting commits at the time of first submission. 
To avoid that we use so called [https://secure.phabricator.com/book/phabricator/article/arcanist_new_project/#history-mutability-git history immutability].

To that end, you shall configure your <tt>arc</tt> accordingly:
<pre>
arc set-config history.immutable true
</pre>

Note that this does ''not'' mean that you are forbidden to rewrite your local branches (e.g., with <tt>git rebase</tt>).
Quite the contrary: you are encouraged to locally rewrite branches before pushing to ensure that commits are logically separated and your commit history easy to bisect.
The above setting just means that ''arc'' will not rewrite commit history under your nose.

=== Enabling <tt>git push</tt> to our forge ===

The way we've configured our review setup for continuous integration needs you to configure git to allow pushes to our forge. There's two ways you can do this : setting a ssh key to push over ssh, or setting a specific password for git pushes over https.

==== SSH key for pushes ====

In your forge User settings page (On the top right, click on your avatar, then click ''Settings''), you have access to a ''Authentication'' > ''SSH Public Keys'' section (Direct link: <tt>hxxps://forge.softwareheritage.org/settings/user/'''<your username>'''/page/ssh/</tt>). You then have the option to upload a SSH public key, which will authenticate your pushes.

You then need to configure ssh/git to use that key pair, for instance by editing the <tt>~/.ssh/config</tt> file.

Finally, you should configure git to push over ssh when pushing to https://forge.softwareheritage.org, by running the following command:
git config --global url.git@forge.softwareheritage.org:.pushInsteadOf https://forge.softwareheritage.org

This lets git know that it should use <tt>git@forge.softwareheritage.org:</tt> as a base url when pushing repositories cloned from forge.softwareheritage.org over https.

==== VCS password for pushes ====

If you're not comfortable setting up SSH to upload your changes, you have the option of setting a VCS password. This password, ''separate from your account password'', allows Phabricator to authenticate your uploads over HTTPS.

In your forge User settings page (On the top right, click on your avatar, then click ''Settings''), you need to use the ''Authentication'' > ''VCS Password'' section to set your VCS password (Direct link: <tt>hxxps://forge.softwareheritage.org/settings/user/'''<your username>'''/page/vcspassword/</tt>).

== Workflow ==

* work in a feature branch: <tt>git checkout -b my-feat</tt>
* initial review request: hack/commit/hack/commit ; <tt>arc diff origin/master</tt>
* react to change requests: hack/commit/hack/commit ; <tt>arc diff --update Dxx origin/master</tt>
* landing change: <tt>git checkout master ; git merge my-feat ; git push</tt>

=== Starting a new feature and submit it for review ===

Use a '''one branch per feature''' workflow, with well-separated ''logical commits'' ([https://wiki.softwareheritage.org/wiki/Git_style_guide following those conventions])
<pre>
git checkout -b my-shiny-feature
... hack hack hack ...
git commit -m 'architecture skeleton for my-shiny-feature'
... hack hack hack ...
git commit -m 'my-shiny-feature: implement module foo'
... etc ...
</pre>

Please, follow the
To '''submit your code for review''' the first time:
<pre>
arc diff origin/master
</pre>
arc will prompt for a '''code review message'''. Provide the following information:
* first line: ''short description'' of the overall work (i.e., the feature you're working on). This will become the title of the review
* ''Summary'' field (optional): ''long description'' of the overall work; the field can continue in subsequent lines, up to the next field. This will become the "Summary" section of the review
* ''Test Plan'' field (optional): write here if something special is needed to test your change
* ''Reviewers'' field (optional): the (Phabricator) name(s) of desired reviewers. If you don't specify one (recommended) the default reviewers will be chosen
* ''Subscribers'' field (optional): the (Phabricator) name(s) of people that will be notified about changes to this review request. In most cases it should be left empty

For example:
<pre>
mercurial loader

Summary: first stab at a mercurial loader (T329)

The implementation follows the plan detailed in F2F discussion with @foo.

Performances seem decent enough for a first trial (XXX seconds for YYY repository
that contains ZZZ patches).

Test plan:

Reviewers:

Subscribers: foo
</pre>

After completing the message arc will submit the review request and tell you its number and URL:
<pre>
[...]
Created a new Differential revision:
Revision URI: https://forge.softwareheritage.org/Dxx
</pre>

=== Updating your branch to reflect requested changes ===

Your feature might get accepted as is, YAY!
Or, reviewers might request changes; no big deal!

Use the Differential web UI to follow-up to received comments, if needed.

To implement requested changes in the code, hack on your branch as usual by:

* adding new commits, and/or
* rewriting old commits with git rebase (to preserve a nice, easy to bisect history)

When you're ready to '''update your review request''':
<pre>
arc diff --update Dxx origin/master
</pre>

Arc will prompt you for a message: describe what you've changed w.r.t. the previous review request, free form.
Your message will become the changelog entry in Differential for this new version of the diff.

Differential only care about the code diff, and not about the commits or their order.
Therefore each "update" can be a completely different series of commits, possibly rewritten from the previous submission.

=== Landing your change onto master ===

Once your change has been approved in Differential, you will be able to land it onto the master branch.

Before doing so, you're encouraged to '''clean up your git commit history''', reordering/splitting/merging commits as needed to have separate logical commits and an easy to bisect history.
The correspondence between the accepted review and pushed code is checked by looking only at the code diff, so commit fiddling will not impact your ability to push to master.

Once you're happy you can '''push to origin/master''' directly, e.g.:
<pre>
git checkout master
git merge my-shiny-feature
git push
</pre>

Optionally you can then delete your local feature branch:
<pre>
git branch -d my-shiny-feature
</pre>

=== Reviewing locally / landing someone else's changes ===

You can do local reviews of code with arc patch:

<pre>
arc patch Dxyz
</pre>

This will create a branch '''arcpatch-Dxyz''' containing the changes on your local checkout.

You can then merge those changes upstream with

<pre>
git checkout master
git merge --ff arcpatch-Dxyz
git push origin master
</pre>

== See also ==

* [[Code review]] for guidelines on how code is reviewed when developing for Software Heritage

[[Category:Software development]]

Debian packaging

2019-02-08T15:46:03Z

Ardumont: Keep the same visual order as the phabricator ui

== Package repository ==

A package repository is available on https://debian.softwareheritage.org/.

Unstable / Testing :
deb [trusted=yes] https://debian.softwareheritage.org/ unstable main

Stable / Stretch :
deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main

This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).

=== Uploading packages ===

Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.

For packages to be accepted, they need to be :
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt>
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders

== Git repositories for Debian packages ==

Our git repository structure for Debian packages is compatible with <tt>git-buildpackage</tt>.

We have two different ways of handling repositories for Debian packages:
* Packages of python modules where *we* are upstream
* Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)

For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:
* Our own modules are merged with the upstream repository
* External dependencies ignore the upstream repository and only have packaging branches.

=== Branch and tags structure ===

Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:
* debian/upstream (history of unpacked upstream releases)
* debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, stretch-swh)
* pristine-tar (data to regenerate upstream tarballs from a git export)

The name of the debian/upstream branch doesn't matter ''as long as it's properly configured in the <tt>debian/gbp.conf</tt> file''. It's only really used by <tt>gbp import-orig</tt> when importing a new release.

The tags marking upstream releases imported from tarballs for Debian packaging purposes are named <tt>debian/upstream/''<upstream version number>''</tt>.

Our Jenkins jobs are triggered on incoming tags named <tt>debian/''<version>''</tt>. To generate the proper tags, use <tt>gbp buildpackage --git-tag-only</tt>.

The git-buildpackage configuration, <tt>debian/gbp.conf</tt>, should be the following:
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/''<current suite>''
pristine-tar=True

==== Automatic packaging for swh python modules ====

The <tt>swh.*</tt> python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs <tt>gbp import-orig</tt> with the tarball we release to PyPI, and the right options to merge the upstream history.

To merge changes from the upstream history, we add the following option to <tt>gbp.conf</tt>:
upstream-vcs-tag=v%(version)s

=== Bootstrapping a dependency packaging repository ===

Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:

Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/
wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz

Create a new git repository
mkdir pytest-postgresql
cd pytest-postgresql
git init

Import the original upstream version
git checkout -b debian/unstable-swh
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz
# What will be the source package name? [pytest-postgresql]
# What is the upstream version? [1.3.4]
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...
# gbp:info: Source package is pytest-postgresql
# gbp:info: Upstream version is 1.3.4
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz

Bootstrap the debian directory
mkdir debian
mkdir debian/source
echo '3.0 (quilt)' > debian/source/format
cat > debian/gbp.conf << EOF
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/unstable-swh
pristine-tar=True
EOF
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules
vim debian/control
# [...] adapt debian/control from another package
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1
vim debian/copyright
# [...] adapt debian/copyright from another package
git add debian
git commit -m "Initial packaging for pytest-postgresql"

You can then go on to try building the package. Once the package builds, if you want to check your package's conformance to Debian policy, you can run <tt>lintian</tt> on the changes:
lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes

Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository

We need to use a <tt>+swh1</tt> version suffix to avoid clashing with potential upstream Debian package versions.

==== Bootstrapping the backport branches ====

During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.

The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.

Checkout the new branch
git checkout debian/<version number>
git checkout -b debian/stretch-swh

Update the gbp config to match the branch
sed -i s/unstable-swh/stretch-swh/ debian/gbp.conf

Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)
dch -l ~bpo9 -D stretch-swh --force-distribution 'Rebuild for stretch-swh'

You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.

==== Setting up the repository on Phabricator ====

The repository on Phabricator needs the following settings:
* Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)
* Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)
* Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)
* Policy
** View: Public (no login required)
** Edit: Developers
** Push: All users (actual restrictions are handled by Herald rules)
* Activate the repository
* Look up the path to the repository on the storage tab

You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes.
ssh -t tate.internal.softwareheritage.org phabricator-setup-hook <repository-path> post-receive-debian-deps

==== Setting up the Jenkins jobs ====

The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/
They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs

Jobs for dependency packages are configured in <tt>jobs/dependency-packages.yaml</tt>. You can add a section as follows:

- project:
name: <Callsign>
display-name: <short-name>
pkg: <source-name>
jobs:
- 'dependency-jobs-{name}'

Use the regular review process to land your changes.
Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.

If your package needs extra repositories to build, you can add them as comma-separated values to the <tt>deb-extra-repositories</tt> setting, with the following notes:
* When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.
* When building packages for backports suites, the backports repository is automatically enabled.

=== Local package building ===

To locally test a package build, go on the appropriate debian packaging branch, and run
gbp buildpackage --git-builder=sbuild -As --no-clean-source

<tt>gbp buildpackage</tt> passes all options not starting with <tt>--git-</tt> to the builder. Some useful options are the following:

* <tt>--git-ignore-new</tt> builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.
* <tt>--no-clean-source</tt> doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies
* <tt>--extra-repository="'''repository specification'''"</tt> adds the given repository in the chroot before building.
* <tt>--extra-repository-key='''repository signing key'''</tt> adds the given key as a trusted gpg key for package sources
* <tt>--extra-package='''<.deb file or directory>'''</tt> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.
* <tt>--force-orig-source</tt> forces addition of the <tt>.orig.tar.gz</tt> file in the <tt>.changes</tt> file (useful when trying to upload a backport)

See <tt>gbp help buildpackage</tt> and <tt>man sbuild</tt> for a full description of all options

for example:
gbp buildpackage --git-builder=sbuild -As --force-orig-source --extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main'

(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping <tt>gbp buildpackage</tt> with the most common options)

=== Remote package building ===

Jenkins builds packages when the repository receives a tag.

Once the local build succeeds, tag the package with:
gbp buildpackage --git-tag-only --git-sign-tags

Alternatively, you can add the <tt>--git-tag</tt> option to your <tt>gbp buildpackage</tt> command so the tag happens automatically on a successful build.

Then, push your tag, and Jenkins jobs should get triggered
git push --tags

== Build Environment setup ==

Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.

=== sbuild setup ===

# Install the package
sudo apt-get install sbuild

# Add your user to the sbuild group, to allow him to use the sbuild commands
sudo sbuild-adduser $USER
# You have to logout and log back in

# Prepare chroots
sudo mkdir /srv/chroots
sudo mkdir /srv/chroots/var

# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/
sudo rm -r /var/lib/schroot
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot

sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/
sudo rm -r /var/lib/sbuild
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild
# end optionally

# Create unstable/sid chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/

# Create stretch chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates stretch /srv/chroots/stretch http://deb.debian.org/debian/

# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts
echo hosts >> /etc/schroot/sbuild/nssdatabases

=== schroot setup ===

Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.

You need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:

source-groups=root,sbuild
source-root-groups=root,sbuild
union-type=overlay

This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.

You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):

For unstable:
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild

For stretch:
aliases=stable-amd64-sbuild,stable-backports-amd64-sbuild,stretch-swh-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild

==== dependencies cache ====

Add the following line to schroot's fstab /etc/schroot/sbuild/fstab
to permit reuse of existing fetched dependencies:

/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0

You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in <tt>/etc/apt/apt.conf.d</tt> on each chroot

=== schroot update ===

You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):

sudo sbuild-update -udcar sid; sudo sbuild-update -udcar stretch; sudo sbuild-update -ud jessie

=== environment setup ===

The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt>

export DEBFULLNAME="Debra Hacker"
export DEBEMAIL=debra.hacker@example.com

Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourfullkeyid></tt> variable.
(Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).

=== overlay in tmpfs for faster builds ===

You can add this to your fstab to put the overlay hierarchy in RAM:

tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0 0 0

=== Base packages ===

In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.

'''If you do so, do not use these chroots to upload to Debian itself!'''

[[Category:Software development]]
[[Category:System administration]]

Google Summer of Code 2019

2019-02-05T13:34:17Z

Ardumont: /* Improve and extend the archive Web UI */

[[File:GSoCLogo.png|1024px]]

== General information ==

This page is the central point of information for [[Software Heritage]] participation into the [https://summerofcode.withgoogle.com/ Google Summer of Code] program.

Google Summer of Code is a program where Google pays students stipends to work over the (northern hemisphere) summer on free software projects such as Software Heritage. Each student works with mentors from the community to complete a software project.

== I want to participate as a student ==

Great!, we are very glad for your interest in contributing to Software Heritage and we are looking forward to work together.

=== Prerequisites ===

The following prerequisites apply to Software Heritage GSoC projects:

* [https://www.python.org Python] 3 is our language of choice, you should be fluent with that language to apply
* [https://git-scm.com Git] is our version control system of choice, you should be familiar with it to apply
* additional prerequisites depend on the project you will work on; check project descriptions for details

=== Before you apply ===

Here are the steps you should follow before applying, to make sure you have a good grasp of what we are doing at Software Heritage and how we do it:

# Follow our [https://docs.softwareheritage.org/devel/getting-started.html getting started guide]: it will make sure you can locally run a (small) copy of the archive and ingest source code into it
# Create an account our [https://forge.softwareheritage.org development forge]
# Familiarize yourself with our [[Code review in Phabricator|code review workflow]]
# Make a simple change to any one of our [https://docs.softwareheritage.org/devel/ software components] and submit it as a [https://forge.softwareheritage.org/differential/ diff] for code review, following the above workflow. [[Easy hacks]] and [https://forge.softwareheritage.org/project/view/20/ Web UI] issues are good options for what to fix, but feel free to submit any patch you think it might be useful.

=== What to include in your application ===

Make sure that your application includes the following information:

* Describe the '''specific project''' you want to work on. What do you want to achieve? Why is it important? Why is it useful for Software Heritage? The project might be one of the project ideas that we have prepared, or something else entirely that you want to contribute to Software Heritage. Your source code archival pet peeve, surprise us!
* Detail your '''work plan''': a brief description of how you plan to go about your project, including a list of ''deliverables'' and a ''timeline'' of when do you expect them to be available.
* Include a reference to '''the diff''' you submitted before applying (see the "Before you apply" section above).

== Ideas list ==

Below you can find a list of project ideas that are good options for a reasonably sized GSoC project.
They are just suggestion though, don't feel obliged to pick one of them if there is nothing that fits your taste and abilities.
Feel free to propose something else that you are excited about and that contributes to improve the Software Heritage archive: we will be happy to consider it!

=== Increase archive coverage ===

Software Heritage aims to archive '''all the software'''. We naturally started with the place where most of the software is easily available today: git repositories on GitHub.

As Software Heritage grows, we're incrementally trying to increase the coverage of the archive by expanding the sources from which we archive software. We built ways of archiving things like mercurial repositories, Debian packages, pypi bundles, etc.

Expanding the coverage of the archive has two different components:

1. Create '''origin listers'''. Listers are pieces of code that crawl the APIs of Software Forges[https://en.wikipedia.org/wiki/Forge_(software)] (like Bitbucket, Gitorious, Sourceforge, NPM...) and return a list of the software available in it. The documentation on listers is here: https://docs.softwareheritage.org/devel/swh-lister/index.html

2. Create '''loaders'''. Loaders take a bundle of software (tarball, git repository, Python package, ...) and load it into Software Heritage, by adapting it so that it matches our uniform data model[https://docs.softwareheritage.org/devel/swh-model/data-model.html].

In a few words, a lister can be a way of asking "what are all the repositories available on npm.org?", while a loader would be "how do I load the NPM package I downloaded into Software Heritage?".

Writing a lister or a loader is a great way to contribute to Software Heritage by expanding its coverage! We have a list of software sources we would like to archive here[https://wiki.softwareheritage.org/wiki/Suggestion_box:_source_code_to_add], but you're free to suggest more.

=== Mine information from archived content ===

'''TODO'''

=== Improve and extend the archive Web UI ===

In order to easily navigate into the archive content, a [https://forge.softwareheritage.org/source/swh-web/ web application] is currently developed.
So far it offers the following main features:
* programmatic access to the content of the archive via the [https://archive.softwareheritage.org/api/ Software Heritage API]
* in-browser navigation of the content of the archive via the [https://archive.softwareheritage.org/browse/ Software Heritage browse UI]

There are still numerous improvements and new features to add to that web application, for instance:
* improve overall design
* improve navigation for mobile browsers
* add new source code search criteria and improve the search interface
* implement new developer oriented features: source file history, blame interface, ...
* improve web application [https://www.w3.org/WAI/ accessibility]

If you are interested in web development and want to contribute to the application
enabling users to navigate in the biggest public source code archive collected so far,
feel free to apply.

== Contact ==

GSoC students are encouraged to get in touch with the Software Heritage community using the standard development communication channels, i.e.:

* the #swh-devel IRC channel on [https://freenode.net Freenode]
* the [https://sympa.inria.fr/sympa/info/swh-devel swh-devel mailing list]

See our [https://www.softwareheritage.org/community/developers/ development information page] for more details.

== Timeline ==

See the official [https://developers.google.com/open-source/gsoc/timeline Google Summer of Code timeline].

Google Summer of Code 2019

2019-02-04T08:45:57Z

Ardumont: /* What to include in your application */

[[File:GSoCLogo.png|1024px]]

== General information ==

This page is the central point of information for [[Software Heritage]] participation into the [https://summerofcode.withgoogle.com/ Google Summer of Code] program.

Google Summer of Code is a program where Google pays students stipends to work over the (northern hemisphere) summer on free software projects such as Software Heritage. Each student works with mentors from the community to complete a software project.

== I want to participate as a student ==

Great!, we are very glad for your interest in contributing to Software Heritage and we are looking forward to work together.

=== Prerequisites ===

The following prerequisites apply to Software Heritage GSoC projects:

* [https://www.python.org Python] 3 is our language of choice, you should be fluent with that language to apply
* [https://git-scm.com Git] is our version control system of choice, you should be familiar with it to apply
* additional prerequisites depend on the project you will work on; check project descriptions for details

=== Before you apply ===

Here are the steps you should follow before applying, to make sure you have a good grasp of what we are doing at Software Heritage and how we do it:

# Follow our [https://docs.softwareheritage.org/devel/getting-started.html getting started guide]: it will make sure you can locally run a (small) copy of the archive and ingest source code into it
# Create an account our [https://forge.softwareheritage.org development forge]
# Familiarize yourself with our [[Code review in Phabricator|code review workflow]]
# Make a simple change to any one of our [https://docs.softwareheritage.org/devel/ software components] and submit it as a [https://forge.softwareheritage.org/differential/ diff] for code review, following the above workflow. [[Easy hacks]] and [https://forge.softwareheritage.org/project/view/20/ Web UI] issues are good options for what to fix, but feel free to submit any patch you think it might be useful.

=== What to include in your application ===

Make sure that your application includes the following information:

* Describe the '''specific project''' you want to work on. What do you want to achieve? Why is it important? Why is it useful for Software Heritage? The project might be one of the project ideas that we have prepared, or something else entirely that you want to contribute to Software Heritage. Your source code archival pet peeve, surprise us!
* Detail your '''work plan''': a brief description of how you plan to go about your project, including a list of ''deliverables'' and a ''timeline'' of when do you expect them to be available.
* Include a reference to '''the diff''' you submitted before applying (see the "Before you apply" section above).

== Ideas list ==

Below you can find a list of project ideas that are good options for a reasonably sized GSoC project.
They are just suggestion though, don't feel obliged to pick one of them if there is nothing that fits your taste and abilities.
Feel free to propose something else that you are excited about and that contributes to improve the Software Heritage archive: we will be happy to consider it!

=== Increase archive coverage ===

Software Heritage aims to archive '''all the software'''. We naturally started with the place where most of the software is easily available today: git repositories on GitHub.

As Software Heritage grows, we're incrementally trying to increase the coverage of the archive by expanding the sources from which we archive software. We built ways of archiving things like mercurial repositories, Debian packages, pypi bundles, etc.

Expanding the coverage of the archive has two different components:

1. Create '''origin listers'''. Listers are pieces of code that crawl the APIs of Software Forges[https://en.wikipedia.org/wiki/Forge_(software)] (like Bitbucket, Gitorious, Sourceforge, NPM...) and return a list of the software available in it. The documentation on listers is here: https://docs.softwareheritage.org/devel/swh-lister/index.html

2. Create '''loaders'''. Loaders take a bundle of software (tarball, git repository, Python package, ...) and load it into Software Heritage, by adapting it so that it matches our uniform data model[https://docs.softwareheritage.org/devel/swh-model/data-model.html].

In a few words, a lister can be a way of asking "what are all the repositories available on npm.org?", while a loader would be "how do I load the NPM package I downloaded into Software Heritage?".

Writing a lister or a loader is a great way to contribute to Software Heritage by expanding its coverage! We have a list of software sources we would like to archive here[https://wiki.softwareheritage.org/wiki/Suggestion_box:_source_code_to_add], but you're free to suggest more.

=== Mine information from archived content ===

'''TODO'''

=== Improve and extend the archive Web UI ===

'''TODO'''

== Contact ==

GSoC students are encouraged to get in touch with the Software Heritage community using the standard development communication channels, i.e.:

* the #swh-devel IRC channel on [https://freenode.net Freenode]
* the [https://sympa.inria.fr/sympa/info/swh-devel swh-devel mailing list]

See our [https://www.softwareheritage.org/community/developers/ development information page] for more details.

== Timeline ==

See the official [https://developers.google.com/open-source/gsoc/timeline Google Summer of Code timeline].

Streaming Replication

2018-06-15T09:38:22Z

Ardumont: Fix sentence phrasing

Migrated to [https://intranet.softwareheritage.org/index.php?title=Streaming_Replication intranet].

Streaming Replication

2018-06-15T09:38:05Z

Ardumont: Move page to the intranet (was wrongly created here, no credentials leaked or anything anyway)

Move away to [https://intranet.softwareheritage.org/index.php?title=Streaming_Replication intranet].

Streaming Replication

2018-06-15T09:30:40Z

Ardumont: Fix _, sub issues

Following steps are described how to setup the streaming replication
between a master and slave.

= Master =
== Master information ==
Example db:

* host: somerset.internal.softwareheritage.org

* port: 5434

* db: softwareheritage-indexer

== Replication user needed ==
Make sure a replication user exists on the master server

su - postgres
createuser -p 5434 replicator --replication
psql -p 5434 template1
alter user replicator with password '<precious-secret>';

Note: To retrieve the password.

swhpass ls infra/somerset/postgres/5434/replicator

== Postgres configuration adaptation ==
=== pg_hba.conf ===
File: /etc/postgresql/10/indexer/pg_hba.conf

# TYPE DATABASE USER CIDR-ADDRESS METHOD
host replication replicator 192.168.200.0/21 md5 # Azure

=== postgresql.conf ===
File: /etc/postgresql/10/indexer/postgresql.conf

wal_level = replica # or higher, logical is fine
wal_keep_segments = 500 # in logfile segments, 16MB each; 0 disables
wal_compression = on # enable compression of full-page writes

Note:

* wal_compression is an optimization which reduces I/O and network
traffic at the expense of CPU time. It is not required.

* wal_keep_segments needs to be set to a reasonable value in order to
keep enough wal data between the initial database backup and its
subsequent regular startup on the slave. Slave startup can fail at
first, so make sure to have at least a couple hours of slack in case
things go wrong.

== Restart db ==
Make the db aware of those changes:

# to determine the main pid of the concerned db
sudo systemctl status postgres@10-indexer.service
# Actually make the db aware
# kill -HUP <pid-of-db>
# or
sudo systemctl restart postgres@10-indexer.service

= Slave =
== Package Dependencies ==
Install client for connection part, and server for replication one.

# client part and server
sudo apt install postgresql-client-10 postgresl-10

== Postgres setup ==
Make sure the connection is ok.

As postgres user:

su - postgres
touch .pgpass
cat > .pgpass <<EOF
somerset.internal.softwareheritage.org:5434:replication:replicator:<precious-secret>
EOF
chmod 0600 .pgpass
# should permit connection

== Create replica cluster ==
This is a necessary and temporary step to permit to initialize
configuration files that would need editing down the line.

# drop installed main cluster as this won't be used
sudo pg_dropcluster --stop 10 main
# prepare directories in data volume
sudo mkdir /srv/softwareheritage/postgres
sudo chown postgres:postgres /srv/softwareheritage/postgres
# create the cluster configuration
sudo -u postgres env LC_ALL=C.UTF-8 \
pg_createcluster 10 replica \
-D /srv/softwareheritage/postgres/10/replica \
-p 5434

== Make sure the slave can connect to the master ==
Edit the local configuration according to the master. The slave must
have the at least the same boundaries for some configuration options:

file: /etc/postgresql/10/replica/postgresql.conf

max_connections = 128 # same as master somerset.internal.softwareheritage.org:5434:softwareheritage-indexer
max_worker_processes = 10 # same as master somerset.internal.softwareheritage.org:5434:softwareheritage-indexer

Note: You will be notified in logs when trying to start the db that
some setup are not correctly set. Some form of:

ardumont@dbreplica1 $ sudo journalctl -xef -u postgresql@10-replica.service
...
Jun 15 08:13:24 dbreplica1 postgresql@10-replica[654]: 2018-06-15 08:13:24.425 UTC [664] FATAL: hot standby is not possible because max_worker_processes = 8 is a lower setting than on the ma
ster server (its value was 10)
...

== Retrieve backup ==
From the replica node, as postgres user:

postgres@dbreplica1 $ pg_basebackup --progress --write-recovery-conf \
-h somerset.internal.softwareheritage.org -p 5434 -U replicator \
-D /srv/softwareheritage/postgres/10/somerset-swh-indexer.backup

Note that this step could fail if your .pgpass is not setup correctly.

== Install backup as db ==
Move the backup files to the regular pgdata location and start the
slave server:

postgres@dbreplica1 $ cd /srv/softwareheritage/postgres/10
# Remove 'temporary' cluster
postgres@dbreplica1 $ rm -rf replica
# Rename the backup as replica
postgres@dbreplica1 $ mv somerset-swh-indexer.backup replica
ardumont@dbreplica1 $ sudo systemctl status postgresql@10-replica.service

== Start cluser ==
Now that everything is ready, start the cluster. This will start by
redoing the necessary steps to synchronize the dbs.

sudo systemctl start postgresql@10-replica.service

Streaming Replication

2018-06-15T09:20:59Z

Ardumont: First iteration

Following steps are described how to setup the streaming replication
between a master and slave.

= Master =
== Master information ==
Example db:

* host: somerset.internal.softwareheritage.org

* port: 5434

* db: softwareheritage-indexer

== Replication user needed ==
Make sure a replication user exists on the master server

su - postgres
createuser -p 5434 replicator --replication
psql -p 5434 template1
alter user replicator with password '<precious-secret>';

Note: To retrieve the password.

swhpass ls infra/somerset/postgres/5434/replicator

== Postgres configuration adaptation ==
=== pghba.conf ===
File: /etc/postgresql/10/indexer/pghba.conf

# TYPE DATABASE USER CIDR-ADDRESS METHOD
host replication replicator 192.168.200.0/21 md5 # Azure

=== postgresql.conf ===
File: /etc/postgresql/10/indexer/postgresql.conf

wal_level = replica # or higher, logical is fine
wal_keep_segments = 500 # in logfile segments, 16MB each; 0 disables
wal_compression = on # enable compression of full-page writes

Note:

* walcompression is an optimization which reduces I/O and network
traffic at the expense of CPU time. It is not required.

* walkeepsegments needs to be set to a reasonable value in order to
keep enough wal data between the initial database backup and its
subsequent regular startup on the slave. Slave startup can fail at
first, so make sure to have at least a couple hours of slack in case
things go wrong.

== Restart db ==
Make the db aware of those changes:

# to determine the main pid of the concerned db
sudo systemctl status postgres@10-indexer.service
# Actually make the db aware
# kill -HUP <pid-of-db>
# or
sudo systemctl restart postgres@10-indexer.service

= Slave =
== Package Dependencies ==
Install client for connection part, and server for replication one.

# client part and server
sudo apt install postgresql-client-10 postgresl-10

== Postgres setup ==
Make sure the connection is ok.

As postgres user:

su - postgres
touch .pgpass
cat > .pgpass <<EOF
somerset.internal.softwareheritage.org:5434:replication:replicator:<precious-secret>
EOF
chmod 0600 .pgpass
# should permit connection

== Create replica cluster ==
This is a necessary and temporary step to permit to initialize
configuration files that would need editing down the line.

# drop installed main cluster as this won't be used
sudo pg_dropcluster --stop 10 main
# prepare directories in data volume
sudo mkdir /srv/softwareheritage/postgres
sudo chown postgres:postgres /srv/softwareheritage/postgres
# create the cluster configuration
sudo -u postgres env LC_ALL=C.UTF-8 \
pg_createcluster 10 replica \
-D /srv/softwareheritage/postgres/10/replica \
-p 5434

== Make sure the slave can connect to the master ==
Edit the local configuration according to the master. The slave must
have the at least the same boundaries for some configuration options:

file: /etc/postgresql/10/replica/postgresql.conf

max_connections = 128 # same as master somerset.internal.softwareheritage.org:5434:softwareheritage-indexer
max_worker_processes = 10 # same as master somerset.internal.softwareheritage.org:5434:softwareheritage-indexer

Note: You will be notified in logs when trying to start the db that
some setup are not correctly set. Some form of:

ardumont@dbreplica1 $ sudo journalctl -xef -u postgresql@10-replica.service
...
Jun 15 08:13:24 dbreplica1 postgresql@10-replica[654]: 2018-06-15 08:13:24.425 UTC [664] FATAL: hot standby is not possible because max_worker_processes = 8 is a lower setting than on the ma
ster server (its value was 10)
...

== Retrieve backup ==
From the replica node, as postgres user:

postgres@dbreplica1 $ pg_basebackup --progress --write-recovery-conf \
-h somerset.internal.softwareheritage.org -p 5434 -U replicator \
-D /srv/softwareheritage/postgres/10/somerset-swh-indexer.backup

Note that this step could fail if your .pgpass is not setup correctly.

== Install backup as db ==
Move the backup files to the regular pgdata location and start the
slave server:

postgres@dbreplica1 $ cd /srv/softwareheritage/postgres/10
# Remove 'temporary' cluster
postgres@dbreplica1 $ rm -rf replica
# Rename the backup as replica
postgres@dbreplica1 $ mv somerset-swh-indexer.backup replica
ardumont@dbreplica1 $ sudo systemctl status postgresql@10-replica.service

== Start cluser ==
Now that everything is ready, start the cluster. This will start by
redoing the necessary steps to synchronize the dbs.

sudo systemctl start postgresql@10-replica.service

Streaming Replication

2018-06-15T09:00:46Z

Ardumont: Bootstrap streaming replication documentation

[[Category:Postgres]]
[[Category:System administration]]

WIP

Related tasks
- https://forge.softwareheritage.org/T883
- https://forge.softwareheritage.org/T1094

External contribution integration

2018-06-14T12:06:29Z

Ardumont: Add category

= How to integrate external contribution =
New [https://forge.softwareheritage.org/D301 external contributions are starting]. Here is the current team
workflow defined on how to integrate them:

* Review the diff and explain what's need adapting

* Accept the diff if ready. We should only be able to accept it if the contributor has already signed the [https://wiki.softwareheritage.org/index.php?title=Contributor_License_Agreement CLA]

* Integrate the patch using [https://wiki.softwareheritage.org/index.php?title=Arcanist arcanist]:

cd /path/to/<repository-concerned-by-diff>
PATCH_NAME=<patch-name> # use the right patch name D301 for example
CONTRIBUTOR_NAME="<contributor-name>" # use the right contributor's full name
arc patch $PATCH_NAME
# this will create a local commit amend the commit message if
# necessary (added a `Close <concerned-task>` for example)
echo $CONTRIBUTOR_NAME >> CONTRIBUTORS
git add CONTRIBUTORS
git commit -m 'Update contributors file'
git checkout master
BRANCH_TO_MERGE="arcpatch-$PATCH_NAME"
git merge $BRANCH_TO_MERGE
git branch -d $BRANCH_TO_MERGE
git push

[[Category:Software_development]]

External contribution integration

2018-06-14T11:55:27Z

Ardumont: Add missing `

External contribution integration

2018-06-14T10:02:38Z

Ardumont: Simplify not using footnotes as i did not find out fast enough

= How to integrate external contribution =
New [https://forge.softwareheritage.org/D301 external contributions are starting]. Here is the current team
workflow defined on how to integrate them:

* Review the diff and explain what's need adapting

* Accept the diff if ready. We should only be able to accept it if the contributor has already

signed the [https://wiki.softwareheritage.org/index.php?title=Contributor_License_Agreement CLA]

* Integrate the patch using [https://wiki.softwareheritage.org/index.php?title=Arcanist arcanist]:

cd /path/to/<repository-concerned-by-diff>
PATCH_NAME=<patch-name> # use the right patch name D301 for example
CONTRIBUTOR_NAME="<contributor-name>" # use the right contributor's full name
arc patch $PATCH_NAME
# this will create a local commit amend the commit message if
# necessary (added a `Close <concerned-task> for example)
echo $CONTRIBUTOR_NAME >> CONTRIBUTORS
git add CONTRIBUTORS
git commit -m 'Update contributors file'
git checkout master
BRANCH_TO_MERGE="arcpatch-$PATCH_NAME"
git merge $BRANCH_TO_MERGE
git branch -d $BRANCH_TO_MERGE
git push

External contribution integration

2018-06-14T09:53:49Z

Ardumont:

= How to integrate external contribution =
New external contributions are starting [1]. Here is the current team
workflow defined.

We should:

* Review the diff and explain what's need adapting

* Accept the diff if ready [2]

* Integrate the patch using [https://wiki.softwareheritage.org/index.php?title=Arcanist arcanist]:

cd /path/to/<repository-concerned-by-diff>
PATCH_NAME=<patch-name> # use the right patch name D301 for example
CONTRIBUTOR_NAME="<contributor-name>" # use the right contributor's full name
arc patch $PATCH_NAME
# this will create a local commit amend the commit message if
# necessary (added a `Close <concerned-task> for example)
echo $CONTRIBUTOR_NAME >> CONTRIBUTORS
git add CONTRIBUTORS
git commit -m 'Update contributors file'
git checkout master
BRANCH_TO_MERGE="arcpatch-$PATCH_NAME"
git merge $BRANCH_TO_MERGE
git branch -d $BRANCH_TO_MERGE
git push

----
=== Footnotes ===

[1] https://forge.softwareheritage.org/D301
https://forge.softwareheritage.org/D302

[2] We should only be able to accept it if the contributor has already
signed the [https://wiki.softwareheritage.org/index.php?title=Contributor_License_Agreement CLA]

External contribution integration

2018-06-14T09:47:56Z

Ardumont: /* External contribution integration */

= External contribution integration =
New external contributions are starting [1]. Here is the current team
workflow defined.

We should:

* Review the diff and explain what's need adapting

* Accept the diff if ready [2]

* Integrate the patch using [https://wiki.softwareheritage.org/index.php?title=Arcanist arcanist]:

cd /path/to/<repository-concerned-by-diff>
PATCH_NAME=<patch-name> # use the right patch name D301 for example
CONTRIBUTOR_NAME="<contributor-name>" # use the right contributor's full name
arc patch $PATCH_NAME
# this will create a local commit amend the commit message if
# necessary (added a `Close <concerned-task> for example)
echo $CONTRIBUTOR_NAME >> CONTRIBUTORS
git add CONTRIBUTORS
git commit -m 'Update contributors file'
git checkout master
BRANCH_TO_MERGE="arcpatch-$PATCH_NAME"
git merge $BRANCH_TO_MERGE
git branch -d $BRANCH_TO_MERGE
git push

[1] https://forge.softwareheritage.org/D301
https://forge.softwareheritage.org/D302

[2] We should only be able to accept it if the contributor has already
signed the [https://wiki.softwareheritage.org/index.php?title=Contributor_License_Agreement CLA]

External contribution integration

2018-06-14T09:47:29Z

Ardumont: Improve readability

External contribution integration

2018-06-14T09:46:43Z

Ardumont: Bootstrap external contribution integration as a team workflow

= External contribution integration =
New external contributions are starting [1]. Here is the current team
workflow defined.

We should:

* Review the diff and explain what's need adapting

* Accept the diff if ready. We should only be able to accept it if the
contributor has already signed the [https://wiki.softwareheritage.org/index.php?title=Contributor_License_Agreement CLA]

* Integrate the patch using [https://wiki.softwareheritage.org/index.php?title=Arcanist arcanist]:

cd /path/to/<repository-concerned-by-diff>
PATCH_NAME=<patch-name> # use the right patch name D301 for example
CONTRIBUTOR_NAME="<contributor-name>" # use the right contributor's full name
arc patch $PATCH_NAME
# this will create a local commit amend the commit message if
# necessary (added a `Close <concerned-task> for example)
echo $CONTRIBUTOR_NAME >> CONTRIBUTORS
git add CONTRIBUTORS
git commit -m 'Update contributors file'
git checkout master
BRANCH_TO_MERGE="arcpatch-$PATCH_NAME"
git merge $BRANCH_TO_MERGE
git branch -d $BRANCH_TO_MERGE
git push

[1] https://forge.softwareheritage.org/D301
https://forge.softwareheritage.org/D302

New machine setup

2018-06-14T08:27:51Z

Ardumont:

= Setting up a new Software Heritage desktop machine =

== Debian install ==

* Stable
* root w/temporary password; no regular user (after setting up root password, cancel twice and jump forward to clock settings)
* full disk with LVM; reduce home LV to leave half of the disk free
* Standard system utilities, ssh server, no desktop environment (puppet will install that)

== Base system setup (from console) ==

* Login as root
* Enable password root access in ssh (/etc/ssh/sshd_config, PermitRootLogin yes)
* Write down IP configuration and add the machine to the Gandi DNS
* Test SSH login as root from your workstation
* Stay at your desk :)

== Full system setup (from your desk) ==

* SSH login as root
* Edit sources.list to add testing
* apt-get update, dist-upgrade, autoremove --purge
** While you wait, create [[Vpn]] certificates for the new machine
** add the machine to the puppet configuration, in the swh_desktop role
* apt-get install puppet openvpn
* configure openvpn per [[Vpn]]
** add pergamon IP address to /etc/resolv.conf
** add louvre.softwareheritage.org to /etc/hosts
* configure puppet
** systemctl disable puppet
** server=pergamon.internal.softwareheritage.org in /etc/puppet/puppet.conf
** puppet agent --enable
** puppet agent -t
** run puppet on pergamon to update munin server config
* set proper root password, add it to password store
* reboot

= Setting up a new Virtual Machine (semi-manual process) =
As a requisite step, clone the [https://forge.softwareheritage.org/diffusion/SPRE/repository/master/ sysadm-provisioning] repository.

== Naming scheme ==
<machine-name>.(<zone>.:<hoster>).internal.softwareheritage.org.

The parenthesis denotes optional scheme.

== Modus operandi ==
The modus operandi is as follows:

* Install infra's requisite dependencies (read per infra/cloud resources)

* provision a vm from the dedicated infrastructure cloud provider

* bootstrap packages puppet dependencies on that vm

* run puppet agent on that vm

* run puppet agent on the dns node

== Example with azure ==
First, Install [https://forge.softwareheritage.org/diffusion/SPRE/browse/master/azure/README.md$4 azure's requirements].

cd /path/to/sysadm-provisioning
# historic implementation detail (really use the following user)
# using this user (uid 1000) permits to simplify steps down the
# line
ADMIN_USER=zack
# Create the vm 'worker01' with type 'worker'
# (other possible types: db, replica, <whatever>)
./azure/create-vm.sh worker01 worker
# retrieve ip of the new vm
# then copy the provision-vm.sh script to run there
# (this does as entertained earlier puppet bootstrap + run)
scp ./azure/provision-vm.sh $ADMIN_USER@<ip>:/tmp
ssh $ADMIN_USER@<ip> chmod +x /tmp/provision-vm.sh
ssh $ADMIN_USER@<ip> /tmp/provision-vm.sh public

# note that you could also connect to the node, install tmux, run a
# tmux session, then trigger the script from within

After this, run the puppet agent on the dns server:

ssh <your-user>@pergamon.internal.softwareheritage.org sudo puppet agent --test

As always, the truth lies within the source code, details explained in
comments:

* [https://forge.softwareheritage.org/diffusion/SPRE/browse/master/azure/create-vm.sh create-vm.sh]

* [https://forge.softwareheritage.org/diffusion/SPRE/browse/master/azure/provision-vm.sh provision-vm.sh]

= Troubleshoot =

== Recreating machine with the same exact configuration ==

It so happens that we could scratch and recreate the same machine.
We then need to clean up on the puppet-master the old certificate (based on the machine's fqdn).

puppet cert clean <fqdn>

== Duplicate resource found error ==

For information, after a wrong manipulation (wrong hostname setup for example), you could end
up having stale data in the puppet master (in puppetdb).

You would end up with the puppet agent complaining about duplicate resources found, for example:
A duplicate resource was found while collecting exported resources

That means there exists some stale data in the master (puppetdb).
Here is the command to clean those up.

puppet node deactivate <wrong-fqdn>

[[Category:Infrastructure]]
[[Category:System administration]]

New machine setup

2018-06-14T08:23:22Z

Ardumont: /* Modus operandi */

= Setting up a new Software Heritage desktop machine =

== Debian install ==

* Stable
* root w/temporary password; no regular user (after setting up root password, cancel twice and jump forward to clock settings)
* full disk with LVM; reduce home LV to leave half of the disk free
* Standard system utilities, ssh server, no desktop environment (puppet will install that)

== Base system setup (from console) ==

* Login as root
* Enable password root access in ssh (/etc/ssh/sshd_config, PermitRootLogin yes)
* Write down IP configuration and add the machine to the Gandi DNS
* Test SSH login as root from your workstation
* Stay at your desk :)

== Full system setup (from your desk) ==

* SSH login as root
* Edit sources.list to add testing
* apt-get update, dist-upgrade, autoremove --purge
** While you wait, create [[Vpn]] certificates for the new machine
** add the machine to the puppet configuration, in the swh_desktop role
* apt-get install puppet openvpn
* configure openvpn per [[Vpn]]
** add pergamon IP address to /etc/resolv.conf
** add louvre.softwareheritage.org to /etc/hosts
* configure puppet
** systemctl disable puppet
** server=pergamon.internal.softwareheritage.org in /etc/puppet/puppet.conf
** puppet agent --enable
** puppet agent -t
** run puppet on pergamon to update munin server config
* set proper root password, add it to password store
* reboot

= Setting up a new Virtual Machine (semi-manual process) =
As a requisite step, clone the [https://forge.softwareheritage.org/diffusion/SPRE/repository/master/ sysadm-provisioning] repository.

== Naming scheme ==
<machine-name>.(<zone>.:<hoster>).internal.softwareheritage.org.

The parenthesis denotes optional scheme.

= Modus operandi =
The modus operandi is as follows:

* Install infra's requisite dependencies (read per infra/cloud resources)

* provision a vm from the dedicated infrastructure cloud provider

* bootstrap packages puppet dependencies on that vm

* run puppet agent on that vm

* run puppet agent on the dns node

= Example with azure =
First, Install [https://forge.softwareheritage.org/diffusion/SPRE/browse/master/azure/README.md$4 azure's requirements].

cd /path/to/sysadm-provisioning
# historic implementation detail (really use the following user)
# using this user (uid 1000) permits to simplify steps down the
# line
ADMIN_USER=zack
# Create the vm 'worker01' with type 'worker'
# (other possible types: db, replica, <whatever>)
./azure/create-vm.sh worker01 worker
# retrieve ip of the new vm
# then copy the provision-vm.sh script to run there
# (this does as entertained earlier puppet bootstrap + run)
scp ./azure/provision-vm.sh $ADMIN_USER@<ip>:/tmp
ssh $ADMIN_USER@<ip> chmod +x /tmp/provision-vm.sh
ssh $ADMIN_USER@<ip> /tmp/provision-vm.sh public

# note that you could also connect to the node, install tmux, run a
# tmux session, then trigger the script from within

After this, run the puppet agent on the dns server:

ssh <your-user>@pergamon.internal.softwareheritage.org sudo puppet agent --test

As always, the truth lies within the source code, details explained in
comments:

* [https://forge.softwareheritage.org/diffusion/SPRE/browse/master/azure/create-vm.sh create-vm.sh]

* [https://forge.softwareheritage.org/diffusion/SPRE/browse/master/azure/provision-vm.sh provision-vm.sh]

= Troubleshoot =

== Recreating machine with the same exact configuration ==

It so happens that we could scratch and recreate the same machine.
We then need to clean up on the puppet-master the old certificate (based on the machine's fqdn).

puppet cert clean <fqdn>

== Duplicate resource found error ==

For information, after a wrong manipulation (wrong hostname setup for example), you could end
up having stale data in the puppet master (in puppetdb).

You would end up with the puppet agent complaining about duplicate resources found, for example:
A duplicate resource was found while collecting exported resources

That means there exists some stale data in the master (puppetdb).
Here is the command to clean those up.

puppet node deactivate <wrong-fqdn>

[[Category:Infrastructure]]
[[Category:System administration]]

New machine setup

2018-06-14T08:22:42Z

Ardumont: /* Example with azure */

= Setting up a new Software Heritage desktop machine =

== Debian install ==

* Stable
* root w/temporary password; no regular user (after setting up root password, cancel twice and jump forward to clock settings)
* full disk with LVM; reduce home LV to leave half of the disk free
* Standard system utilities, ssh server, no desktop environment (puppet will install that)

== Base system setup (from console) ==

* Login as root
* Enable password root access in ssh (/etc/ssh/sshd_config, PermitRootLogin yes)
* Write down IP configuration and add the machine to the Gandi DNS
* Test SSH login as root from your workstation
* Stay at your desk :)

== Full system setup (from your desk) ==

* SSH login as root
* Edit sources.list to add testing
* apt-get update, dist-upgrade, autoremove --purge
** While you wait, create [[Vpn]] certificates for the new machine
** add the machine to the puppet configuration, in the swh_desktop role
* apt-get install puppet openvpn
* configure openvpn per [[Vpn]]
** add pergamon IP address to /etc/resolv.conf
** add louvre.softwareheritage.org to /etc/hosts
* configure puppet
** systemctl disable puppet
** server=pergamon.internal.softwareheritage.org in /etc/puppet/puppet.conf
** puppet agent --enable
** puppet agent -t
** run puppet on pergamon to update munin server config
* set proper root password, add it to password store
* reboot

= Setting up a new Virtual Machine (semi-manual process) =
As a requisite step, clone the [https://forge.softwareheritage.org/diffusion/SPRE/repository/master/ sysadm-provisioning] repository.

== Naming scheme ==
<machine-name>.(<zone>.:<hoster>).internal.softwareheritage.org.

The parenthesis denotes optional scheme.

== Modus operandi ==
The modus operandi is as follows:

* install pre-requisite as per infra (cf. dedicated infra/cloud resources)

* provision a vm from the dedicated infrastructure cloud provider

* bootstrap packages puppet dependencies on that vm

* run puppet agent on that vm

* run puppet agent on the dns node

= Example with azure =
First, Install [https://forge.softwareheritage.org/diffusion/SPRE/browse/master/azure/README.md$4 azure's requirements].

cd /path/to/sysadm-provisioning
# historic implementation detail (really use the following user)
# using this user (uid 1000) permits to simplify steps down the
# line
ADMIN_USER=zack
# Create the vm 'worker01' with type 'worker'
# (other possible types: db, replica, <whatever>)
./azure/create-vm.sh worker01 worker
# retrieve ip of the new vm
# then copy the provision-vm.sh script to run there
# (this does as entertained earlier puppet bootstrap + run)
scp ./azure/provision-vm.sh $ADMIN_USER@<ip>:/tmp
ssh $ADMIN_USER@<ip> chmod +x /tmp/provision-vm.sh
ssh $ADMIN_USER@<ip> /tmp/provision-vm.sh public

# note that you could also connect to the node, install tmux, run a
# tmux session, then trigger the script from within

After this, run the puppet agent on the dns server:

ssh <your-user>@pergamon.internal.softwareheritage.org sudo puppet agent --test

As always, the truth lies within the source code, details explained in
comments:

* [https://forge.softwareheritage.org/diffusion/SPRE/browse/master/azure/create-vm.sh create-vm.sh]

* [https://forge.softwareheritage.org/diffusion/SPRE/browse/master/azure/provision-vm.sh provision-vm.sh]

= Troubleshoot =

== Recreating machine with the same exact configuration ==

It so happens that we could scratch and recreate the same machine.
We then need to clean up on the puppet-master the old certificate (based on the machine's fqdn).

puppet cert clean <fqdn>

== Duplicate resource found error ==

For information, after a wrong manipulation (wrong hostname setup for example), you could end
up having stale data in the puppet master (in puppetdb).

You would end up with the puppet agent complaining about duplicate resources found, for example:
A duplicate resource was found while collecting exported resources

That means there exists some stale data in the master (puppetdb).
Here is the command to clean those up.

puppet node deactivate <wrong-fqdn>

[[Category:Infrastructure]]
[[Category:System administration]]