Software Heritage Wiki - User contributions [en]

Debian packaging

2021-09-01T12:21:11Z

NicolasDandrimont: Drop the dirty chroot suggestion

== Package repository ==

A package repository is available on https://debian.softwareheritage.org/.

Unstable / Testing :
deb [trusted=yes] https://debian.softwareheritage.org/ unstable main

Stable / Buster :
deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main

Oldstable / Stretch :
deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main

This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).

=== Uploading packages ===

Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.

For packages to be accepted, they need to be :
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt>
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders

== Git repositories for Debian packages ==

Our git repository structure for Debian packages is compatible with <tt>git-buildpackage</tt>.

We have two different ways of handling repositories for Debian packages:
* Packages of python modules where *we* are upstream
* Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)

For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:
* Our own modules are merged with the upstream repository
* External dependencies ignore the upstream repository and only have packaging branches.

=== Branch and tags structure ===

Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:
* debian/upstream (history of unpacked upstream releases)
* debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, buster-swh)
* pristine-tar (data to regenerate upstream tarballs from a git export)

The name of the debian/upstream branch doesn't matter ''as long as it's properly configured in the <tt>debian/gbp.conf</tt> file''. It's only really used by <tt>gbp import-orig</tt> when importing a new release.

The tags marking upstream releases imported from tarballs for Debian packaging purposes are named <tt>debian/upstream/''<upstream version number>''</tt>.

Our Jenkins jobs are triggered on incoming tags named <tt>debian/''<version>''</tt>. To generate the proper tags, use <tt>gbp buildpackage --git-tag-only</tt>.

The git-buildpackage configuration, <tt>debian/gbp.conf</tt>, should be the following:
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/''<current suite>''
pristine-tar=True

==== Automatic packaging for swh python modules ====

The <tt>swh.*</tt> python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs <tt>gbp import-orig</tt> with the tarball we release to PyPI, and the right options to merge the upstream history.

To merge changes from the upstream history, we add the following option to <tt>gbp.conf</tt>:
upstream-vcs-tag=v%(version)s

=== Bootstrapping a dependency packaging repository ===

Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:

Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/
wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz

Create a new git repository
git init pytest-postgresql
cd pytest-postgresql

Import the original upstream version
git checkout -b debian/unstable-swh
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz
# What will be the source package name? [pytest-postgresql]
# What is the upstream version? [1.3.4]
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...
# gbp:info: Source package is pytest-postgresql
# gbp:info: Upstream version is 1.3.4
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz

Bootstrap the debian directory
mkdir -p debian/source
echo '3.0 (quilt)' > debian/source/format
echo 9 > debian/compat
cat > debian/gbp.conf << EOF
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/unstable-swh
pristine-tar=True
EOF
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules
vim debian/control
# [...] adapt debian/control from another package
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1 --distribution unstable-swh
vim debian/copyright
# [...] adapt debian/copyright from another package
git add debian
git commit -m "Initial packaging for pytest-postgresql"

You can then go on to try building the package.
gbp buildpackage --git-builder='sbuild -As'

Once the package builds, if you want to check your package's conformance to Debian policy, you can run <tt>lintian</tt> on the changes:
lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes

Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository

We need to use a <tt>+swh1</tt> version suffix to avoid clashing with potential upstream Debian package versions.

==== Bootstrapping the backport branches ====

During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.

The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.

Checkout the new branch
git checkout debian/<version number>
git checkout -b debian/buster-swh

Update the gbp config to match the branch
sed -i s/unstable-swh/buster-swh/ debian/gbp.conf

Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)
dch -l "~bpo10" -D buster-swh --force-distribution 'Rebuild for buster-swh'

You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.

==== Setting up the repository on Phabricator ====

The repository on Phabricator needs the following settings:
* Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)
* Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)
* Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)
* Policy
** View: Public (no login required)
** Edit: Developers
** Push: All users (actual restrictions are handled by Herald rules)
* Activate the repository
* Look up the path to the repository on the storage tab

You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes
ssh -p 2222 -t tate.internal.softwareheritage.org phabricator-setup-hook /srv/phabricator/repos/<repo-id> <post-receive-hook>

Note:

* there exists 2 types of <post-receive-hook>:
** ''post-receive-swh-modules'' for swh modules developed by the team
** ''post-receive-debian-deps'' for external modules packaged by the team
* remember that access to tate is on port 2222.

The repo ID can be found on the repo's "storage" property page on phabricator, typically
https://forge.softwareheritage.org/source/swh-SHORTNAME/manage/storage/

==== Setting up the Jenkins jobs ====

The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/
They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs

Jobs for dependency packages are configured in <tt>jobs/dependency-packages.yaml</tt>. You can add a section as follows:

- project:
name: <Callsign>
display-name: <short-name>
pkg: <source-name>
python_module: <python-module>
jobs:
- 'dependency-jobs-{name}'

For example:
- project:
name: DLDBASE
display-name: swh-loader-core
repo_name: swh-loader-core
pkg: loader.core
python_module: swh.loader.core
jobs:
- 'swh-jobs-{name}'

Other samples can be found in the dedicated repository.
* usual swh package: [https://forge.softwareheritage.org/source/swh-jenkins-jobs/browse/master/jobs/swh-packages.yaml$15-22 swh.core]
* peculiar swh package (with name divergences): [https://forge.softwareheritage.org/source/swh-jenkins-jobs/browse/master/jobs/swh-packages.yaml$51-58 swh.icinga_plugins]

Use the regular review process to land your changes.
Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.

If your package needs extra repositories to build, you can add them as comma-separated values to the <tt>deb-extra-repositories</tt> setting, with the following notes:
* When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.
* When building packages for backports suites, the backports repository is automatically enabled.

=== Updating a dependency packaging repository ===

Place yourself on the debian/unstable-swh branch and "gbp import-origin" a more
recent upstream release tarballs.

For example (current version on 0.0.5, upstream bumped to 0.0.7):
gbp import-origin https://files.pythonhosted.org/packages/7a/bb/cf8fec6009e7d0cec52dc179d09b28c4c70d158e79b565e8aab7606e1717/attrs-strict-0.0.7.tar.gz

This will update the following branches:
* debian/upstream
* pristine-tar
* debian/unstable-swh

This also includes the necessary tags (`debian/upstream/0.0.7` here).

You then need to push all branches/tags to the repository:
git push origin --all --follow-tags

Ensure the [https://wiki.softwareheritage.org/wiki/Debian_packaging#Local_package_building update builds fine]
And [https://wiki.softwareheritage.org/wiki/Debian_packaging#Remote_package_building tags accordingly the debian/unstable-swh branch when ok].
Jenkins will then keep up on building the package.

=== Local package building ===

To locally test a package build, go on the appropriate debian packaging branch, and run
gbp buildpackage --git-builder=sbuild -As --no-clean-source

<tt>gbp buildpackage</tt> passes all options not starting with <tt>--git-</tt> to the builder. Some useful options are the following:

* <tt>--git-ignore-new</tt> builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.
* <tt>--no-clean-source</tt> doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies
* <tt>--extra-repository="'''repository specification'''"</tt> adds the given repository in the chroot before building.
* <tt>--extra-repository-key='''repository signing key'''</tt> adds the given key as a trusted gpg key for package sources
* <tt>--extra-package='''<.deb file or directory>'''</tt> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.
* <tt>--force-orig-source</tt> forces addition of the <tt>.orig.tar.gz</tt> file in the <tt>.changes</tt> file (useful when trying to upload a backport)

See <tt>gbp help buildpackage</tt> and <tt>man sbuild</tt> for a full description of all options

for example:
gbp buildpackage --git-builder=sbuild -As --no-clean-source --force-orig-source \
--extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main'

or if you need some third-party repository, say for cassandra:
gbp buildpackage --git-builder=sbuild -As --no-clean-source --force-orig-source \
--extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main' \
--extra-repository='deb [arch=amd64 trusted=yes] https://downloads.apache.org/cassandra/debian 40x main'

(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping <tt>gbp buildpackage</tt> with the most common options)

=== Remote package building ===

Jenkins builds packages when the repository receives a tag.

Once the local build succeeds, tag the package with:
gbp buildpackage --git-tag-only --git-sign-tags

Alternatively, you can add the <tt>--git-tag</tt> option to your <tt>gbp buildpackage</tt> command so the tag happens automatically on a successful build.

Then, push your tag, and Jenkins jobs should get triggered
git push --tags

== Build Environment setup ==

Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.

=== sbuild setup ===

# Install the package
sudo apt-get install sbuild

# Add your user to the sbuild group, to allow him to use the sbuild commands
sudo sbuild-adduser $USER
# You have to logout and log back in

# Prepare chroots
sudo mkdir /srv/chroots
sudo mkdir /srv/chroots/var

# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/
sudo rm -r /var/lib/schroot
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot

sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/
sudo rm -r /var/lib/sbuild
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild
# end optionally

# Create unstable/sid chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/

# Create buster chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates buster /srv/chroots/buster http://deb.debian.org/debian/

# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts
echo hosts >> /etc/schroot/sbuild/nssdatabases

=== schroot setup ===

Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.

You need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:

source-groups=root,sbuild
source-root-groups=root,sbuild
union-type=overlay

This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.

You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):

For unstable:
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild

For buster:
aliases=buster-swh-amd64-sbuild,buster-backports-amd64-sbuild,buster-backports-swh-amd64-sbuild

==== dependencies cache ====

Add the following line to schroot's fstab /etc/schroot/sbuild/fstab
to permit reuse of existing fetched dependencies:

/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0

You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in <tt>/etc/apt/apt.conf.d</tt> on each chroot

=== schroot update ===

You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):

sudo sbuild-update -udcar sid; sudo sbuild-update -udcar buster

=== environment setup ===

The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt>

export DEBFULLNAME="Debra Hacker"
export DEBEMAIL=debra.hacker@example.com

Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourfullkeyid></tt> variable.
(Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).

=== overlay in tmpfs for faster builds ===

You can add this to your fstab to put the overlay hierarchy in RAM:

tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0 0 0

[[Category:Software development]]
[[Category:System administration]]

A practical approach to efficiently store 100 billions small objects in Ceph

2021-07-06T14:23:35Z

NicolasDandrimont: /* Proposed object storage design */

The [https://en.wikipedia.org/wiki/Software_Heritage Software Heritage] project mission is to collect, preserve and share all software that is available in source code form, with the goal of building a common, shared infrastructure at the service of industry, research, culture and society as a whole. As of February 2021 it contains 10 billions unique source code files (or “objects”, in the following) totaling ~750TB of (uncompressed) data and grows by 50TB every month. 75% of these objects have a size smaller than 16KB and 50% have a size smaller than 4KB. But these small objects only account for ~5% of the 750TB: 25% of the objects have a size > 16KB and occupy ~700TB.

The desired performances for '''10PB''' and '''100 billions objects''' are as follows:

* The clients aggregated together can write at least 3,000 objects/s and at least 100MB/s.
* The clients aggregated together can read at least 3,000 objects/s and at least 100MB/s.
* There is no space amplification for small objects.
* Getting the first byte of any object never takes longer than 100ms.
* Objects can be enumerated in bulk, at least one million at a time.
* Mirroring the content of the Software Heritage archive can be done in bulk, at least one million objects at a time.

Using an off-the-shelf object storage such as the [https://docs.ceph.com/en/latest/radosgw/ Ceph Object Gateway] or [https://min.io/ MinIO] does not meet the requirements:

* There is a significant space amplification for small objects: at least 25%, depending on the object storage (see “How does packing Objects save space?” below for details)
* Mirroring the content of the archive can only be done one object at a time and not in bulk which takes at least 10 times longer (see “How does packing Objects help with enumeration?” for details)

A new solution must be implemented by re-using existing components and made available for system administrators to conveniently deploy and maintain in production. There are three ways to do that:

* Contribute packaging and stable releases to a codebase such as [https://github.com/linkedin/ambry Ambry].
* Modify an object storage such as MinIO to support object packing.
* Get inspiration from an object storage design such as [https://eos-web.web.cern.ch/eos-web/ EOS] and implement something from scratch.

For reasons explained below (see “Storage solutions and TCO”), it was decided to design a new object storage and implement it from scratch.

= Proposed object storage design =

In a nutshell, objects are written to databases running on a fixed number of machines (the Write Storage) that can vary to control the write throughput. When a threshold is reached (e.g. 100GB) all objects are put together in container (a Shard), and moved to a readonly storage that keeps expanding over time. After a successful write, a unique identifier (the Object ID) is returned to the client. It can be used to read the object back from the readonly storage. Reads scale out because the unique identifiers of the objects embed the name of the container (the Shard UUID). Writes also scales out because the Database is chosen randomly. This is the Layer 0.

Clients that cannot keep track of the name of the container can rely on an API that relies on an index mapping all known objects signatures (the Object HASH below) to the name of the container where they can be found. Although this index prevents scaling out writes, the readonly storage can still scale out by multiplying copies of the index as needed. This is the Layer 1.

<pre>
Layer 0 scales out

+--- write op ----+ +--- read op ----+
v ^ v ^
Object & | | |
Object HASH Object ID Object ID Object
| Object HASH | |
v Shard UUID v ^
| | | |
v ^ v ^
+---- Write Storage --------+ +---- Read Storage --------+
| | | |
| +----------+ | | +-------+ +-------+ |
| | Database |->--Packing->----> | Shard | | Shard | |
| +----------+ | | +-------+ +-------+ |
| +----------++----------+ | | +-------+ +-------+ |
| | Database || Database | | | | Shard | | Shard | |
| +----------++----------+ | | +-------+ +-------+ |
| | | +-------+ +-------+ |
+---------------------------+ | | Shard | | Shard | |
| +-------+ +-------+ |
| ... |
+--------------------------+

Layer 1 reads scale out

+---- Write Storage --------+ +---- Read Storage ---------+
| | | |
|+-------------------------+| |+-------------------------+|
||Object HASH to Shard UUID|| ||Object HASH to Shard UUID||
|| index |>>>>| index ||
|+-------------------------+| |+-------------------------+|
+---------------------------+ |+-------------------------+|
| | ||Object HASH to Shard UUID||
^ v || index ||
| | |+-------------------------+|
^ v | ... |
Object | +---------------------------+
Object HASH v | |
| | ^ v
^ v | |
| | Object HASH Object
^ v | |
| | ^ v
+--- write op ----+ +--- read op ----+
</pre>

[[File:Ceph-objstorage-sw-architecture.svg]]

== Glossary ==

* Object: an opaque sequence of bytes.
* Object HASH: the hash of an Object, e.g., the checksum part of a [https://docs.softwareheritage.org/devel/swh-model/persistent-identifiers.html#core-identifiers SWHID].
* Shard: a group of Objects, used to partition the full set of objects into manageable subsets.
* Shard UUID: the unique identifier of a Shard, as a [https://en.wikipedia.org/wiki/Universally_unique_identifier UUID].
* Object ID: a pair made of the Object HASH and the Shard UUID containing the object.
* Global Index: a table mapping the Object HASH to the Shard UUID that contains the Object.
* Read Storage: the unlimited size storage from which clients can only read Objects. It only contains Objects up to a given point in time.
* Write Storage: the fixed size storage from which clients can read or write. If an Object is not found in the Write storage, it must be retrieved from the Read Storage.
* Object Storage: the content of the Write Storage and the Read Storage combined.
* Database: [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL], [https://en.wikipedia.org/wiki/Apache_Cassandra Cassandra], etc.
* [https://en.wikipedia.org/wiki/Ceph_(software) Ceph]: a self-healing distributed storage.
* [https://docs.ceph.com/en/latest/rbd/ RBD] image: a Ceph block storage that can either be used via the librbd library or as a block device from /dev/rbd.
* [https://en.wikipedia.org/wiki/Total_cost_of_ownership TCO]: Total Cost of Ownership

The key concepts are:

* Packing millions of Objects together in Shards to:
** save space and,
** efficiently perform bulk actions such as mirroring or enumerations.
* Two different storage:
** Read Storage that takes advantage of the fact that Objects are immutable and never deleted and,
** Write Storage from which Shards are created and moved to the Read Storage.
* Identifying an object by its Object HASH and the Shard UUID that contains it so that its location can be determined from the Object ID.

While the architecture based on these concepts scales out for writing and reading, it cannot be used to address Objects with their Object HASH alone which is inconvenient for a number of use cases. An index mapping the Object HASH to the Shard UUID must be added to provide this feature, but it does not scale out writes.

The content of the Object Storage (i.e., the Write Storage and the Read Storage combined) is '''strongly/strictly consistent'''. As soon as an Object is written (i.e., the write operation returns to the client), a reader can get the Object content from the Object Storage (with the caveat that it may require looking up the object from both the Write Storage and Read Storage).

The Read Storage is '''eventually consistent'''. It does not contain the latest Objects inserted in the Write Storage but it will, eventually. It contains all objects inserted in the Object Storage, up to a given point in time.

== Layer 0 (Object lookup require a complete Object ID) ==

=== Architecture ===

* Write Storage:
** A fixed number of Databases
* Read Storage:
** Shards implemented as Ceph RBD images named after their Shard UUID
** The content of the Shard uses a format that allows retrieving an Object in O(1) given the Object HASH

=== Writing ===

The Object is stored in one of the Databases from the Write Storage. The Database is chosen at random. A database is associated with a unique Shard UUID, chosen at random. All Objects written to a Database will be stored in the same Shard.

A successful Object write returns the Object ID. Writing the same object twice may return different Object IDs. The Object HASH will be the same because it is based on the content of the Object. But the Shard in which the Object is stored may be different since it is chosen at random.

=== Packing ===

When a Database grows bigger than a threshold (for instance 100GB), it stops accepting writes. A Shard is created in the Read Storage and Objects in the Database are sorted and copied to it. When the Shard is complete, the Database is deleted. Another Database is created, a new Shard UUID is allocated and it starts accepting writes.

=== Reading ===

The Shard UUID is extracted from the Object ID. If a Shard exists in the Read Storage, the Object HASH is used to lookup the content of the Object. Otherwise the Database that owns the Shard UUID is looked up in the Write Storage and the Object HASH is used to lookup the content of the Object. If the reader is not interested in the most up to date content, it can limit its search to the Read Storage.

== Layer 1 (Objects can be looked up using the Object HASH alone) ==

A Global Index mapping the Object HASH of all known Objects to the Shard UUID is used to:

* allow clients to fetch Objects using their Object HASH only instead of their Object ID.
* deduplicate identical Objects based on their Object HASH

=== Architecture ===

* Write Storage:
** Read/write Global Index of all known Objects in the Write Storage and the Read Storage
* Read Storage:
** Read/write Global Index of all known Objects in the Read Storage
** Multiple readonly replicas of the Global Index of all known Objects in the Read Storage

=== Writing ===

If the Object HASH exists in the Read Storage Global Index, do nothing. Otherwise perform the write and add the Object ID to the Write Storage Global Index. There may be duplicate Objects in the Write Storage. It is expected that they race to be inserted in the Write Storage Global Index.

=== Packing ===

During packing, each Object HASH is looked up in the Read Storage Global Index. If it exists, the object is discarded. Otherwise its Object ID is added to the Read Storage Global Index. When packing is complete:

* Readonly replicas of the Read Storage Global Index are updated with the newly added Object IDs.
* Object HASH that were found to be duplicate are updated in the Write Storage Global Index. The Object HASH is mapped to the Shard UUID retrieved from the Read Storage Global Index.

=== Reading ===

If the Object HASH is found in the Read Storage Global Index, use the Shard UUID to read the Object content from the Shard found in the Read Storage. Otherwise lookup the Object HASH from the Write Storage Global Index and read the content of the Object from the Database that owns the Shard UUID.

= How does packing Objects save space? =

The short answer is: it does not when Objects are big enough, but it does when there are a lot of small Objects.

If there are billions of objects (i.e., less than one billion is not a lot) and 50% of them have a size smaller than 4KB and 75% of them have a size smaller than 16KB (i.e., bigger than 16KB is not small), then packing will save space.

In the simplest method of packing (i.e., appending each Object after another in a file) and since the Object HASH has a fixed size, the only overhead for each object is the size of the Object (8 bytes). Assuming the Shard containing the Objects is handled as a single 100GB Ceph RBD Image, it adds R bytes. If the underlying Ceph pool is erasure coded k=4,m=2 an additional 50% must be added.

Retrieving an Object from a Shard would be O(n) in this case because there is no index. It is more efficient to [https://en.wikipedia.org/wiki/Perfect_hash_function add a minimal hash table] to the Shard so that finding an object is O(1) instead. That optimization requires an additional 8 bytes per Object to store their offset, i.e. a total of 16 bytes per object.

If Objects are not packed together, each of them requires at least B bytes, which is the minimum space overhead imposed by the underlying storage system. And an additional 50% for durability. The space used by Objects that are smaller than a given threshold will be amplified, depending on the underlying storage. For instance all objects in Ceph have a minimum size of 4KB, therefore the size of a 1KB Object will be amplified to 4KB which translates to a [https://forge.softwareheritage.org/T3052#58864 35% space amplification]. Another example is MinIO with [https://github.com/minio/minio/issues/7395#issuecomment-475161144 over 200% space amplification] or [https://wiki.openstack.org/wiki/Swift/ideas/small_files#Challenges Swift] for which [https://www.ovh.com/blog/dealing-with-small-files-with-openstack-swift-part-2/ packing small files was recently proposed].

To summarize, the overhead of storing M Objects totaling S bytes with M=100 billions and S=10PB is:

* '''packed:''' ~15.5PB
** (S / 100GB) * R == (10PB / 100GB) * R bytes = 10,000 * R bytes
** (M * 24) = 100G Objects * 24 bytes = 2.4TB
** 50% for durability = 10PB * 0.5 = 5PB
* '''not packed:''' ~17.5PB based on the optimistic assumption that the storage system has a 25% space overhead for small files
** 25% for space amplification = 10PB * 0.25 = 2.5PB
** 50% for durability = 10PB * 0.5 = 5PB

= How does packing Objects help with enumeration? =

For mirroring or running an algorithm on all objects, they must be enumerated. If they are not packed together in any way, which is the case with MinIO or Swift, they must be looked up individually. When they are packed together (one million or more), the reader can download an entire Shard instead, saving the accumulated delay imposed by millions of individual lookup.

If looking up an individual Object takes 10 milliseconds and Shards can be read at 100MB/s:

* Getting 1 billion objects requires 10 millions seconds which is over 100 days.
* One billion objects is 1/10 of the current content of Software Heritage, i.e. ~75TB which can be transferred by reading the Shards in less than 10 days

= Storage solutions and TCO =

When looking for off-the-shelf solutions all options were considered, [https://forge.softwareheritage.org/T3107 including distributed file systems such as IPFs and more] and most of them were discarded because they had at least one blocker that could not be fixed (e.g. no feature to guarantee the durability of an object). In the end a few remained, either including the following features or with the possibility for a third party to contribute them back to the project:

* '''Scale''' to 100 billions objects
* Provide object '''packing'''
* Provide detailed '''documentation''' and community support for system administrators operating the storage
* Be thoroughly '''tested''' before a stable release is published
* Be '''packaged''' for at least one well known distribution
* Have '''stable releases''' maintained for at least two years
* A sound approach to address '''security''' problems (CVE etc.)

{|
! Name
! RGW
! EOS
! SeaweedFS
! MinIO
! Swift
! Ambry
|-
| Scaling
| Yes
| No
| Yes
| Yes
| Yes
| Yes
|-
| Packing
| No
| Yes
| Yes
| No
| No
| Yes
|-
| Documentation
| Good
| Average
| Terse
| Good
| Good
| Terse
|-
| Tests
| Good
| Few
| Few
| Average
| Good
| Few
|-
| Packages
| Yes
| No
| No
| No
| Yes
| No
|-
| Stable releases
| Yes
| No
| No
| Yes
| Yes
| No
|-
| Security
| Yes
| No
| No
| Yes
| Yes
| No
|}

== Does not have stable releases and testing ==

The performance goals, size distribution and the number of objects in Software Heritage are similar to what is described in the 2010 article “[https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf Finding a needle in Haystack: Facebook’s photo storage]” that motivated the implementation of [https://github.com/chrislusf/seaweedfs SeaweedFS] in 2013 or [https://github.com/linkedin/ambry Ambry], the object storage published in 2017 by LinkedIn to store and serve trillions of media objects in web companies.

Contributing to SeaweedFS or Ambry so they can be deployed and maintained would require:

* Creating packages for the target Operating System (e.g. Debian GNU/Linux), maintaining a repository to distribute them, upload them to the official distribution repository so that they are available in the next stable release (about two years from now)
* Creating Ansible roles or Puppet modules for deployment on multiple machines
* Improving the documentation with a configuration and architecture guide to deploy at scale
* Discuss with upstream to create stable releases, define their lifecycle and organize release management
* Establish a security team in charge of handling the CVE
* Setup and infrastructure and create the software for integration testing to be run before a stable release is published to reduce the risk of regressions or data loss. This is specially important because a significant part of the software is dedicated to data storage and replication: bugs can lead to data loss or corruption.

== Does not provide object packing ==

[https://min.io/ MinIO] and [https://docs.openstack.org/swift/latest/ Swift] suffer from a space amplification problem and they do not provide object packing. Although [https://docs.ceph.com/en/latest/radosgw/ Ceph Object Gateway] (also known as RGW) stores objects in RocksDB instead of files, it also suffers from a space amplification problem and does not provide object packing.

Contributing to RGW, MinIO or Swift to add object packing would require:

* Creating a blueprint to modify the internals to add object packing
* Discuss with upstream to validate the blueprint
* Implement the blueprint and the associated tests

== Does not scale ==

[https://eos-web.web.cern.ch/eos-web/ EOS] is based on Ceph and architectured for packing large objects in [https://docs.ceph.com/en/latest/rbd/ RBD]. However, it is not designed to scale over a few billion objects. Contrary to Ambry, Swift and other similar solutions, it delegates storage to Ceph which make it easier to modify and release without risking data loss or corruption. Instead of modifying EOS to scale to 100 billions objects, it is more practical to:

* Write an EOS alternative from scratch, using the same ideas and adding the desired scalability
* Package
* Document
* Test
* Publish stable releases
* Define a security policy

== Estimating the TCO ==

Since no solution can be used as is, some work must be done in each case and the effort it requires should be compared. It is however difficult because the nature of the effort is different. The following factors were considered and aggregated in a TCO estimate.

* '''Data loss risk:''' if a bug in the work done implies the risk of losing data, it makes the work significantly more complicated. It is the case if packing must be implemented in the internals of an existing object storage such as Swift. It is also the case if an object storage does not have integration testing to verify upgrading to a newer version won’t lead to a regression, which is the case with Ambry. It is likely that the Ambry upstream has extensive integration testing but they are not published.
* '''Large codebase:''' a large codebase means modifying it (to implement packing) or distributing it (packaging and documentation) is more difficult
* '''Language:''' if the language and its environment is familiar to the developers and the system administrators, the work is less difficult
* '''Skills:''' if the work requires highly specialized skills (such as an intimate understanding of how a distributed storage system guarantees a strict consistency of the data, or running integration tests that require a cluster of machines) it is more difficult

{|
!
! RGW
! EOS
! SeaweedFS
! MinIO
! Swift
! Ambry
|-
| Data loss risk
| Yes
| No
| Yes
| Yes
| Yes
| Yes
|-
| Large codebase
| Yes
| No
| Yes
| Yes
| Yes
| Yes
|-
| Language
| C++
| Python
| Go
| Go
| Python
| Java
|-
| Skills
| High
| Medium
| High
| High
| High
| High
|-
| TCO estimate
| High
| Medium
| High
| High
| High
| High
|}

In a nutshell, implementing an alternative to EOS from scratch has the lowest TCO estimate, primarily because it is independent of the underlying distributed storage.

[[Category:Software development]]

File:Ceph-objstorage-sw-architecture.svg

2021-07-06T14:22:32Z

NicolasDandrimont:

File:Nm openvpn base.png

2021-06-23T15:22:58Z

NicolasDandrimont: NicolasDandrimont uploaded a new version of File:Nm openvpn base.png

VPN

2021-06-23T15:19:27Z

NicolasDandrimont:

The [[Software Heritage]] server and the VMs running on it are severely firewalled.
To get onto their network unrestricted, a VPN based on [https://openvpn.net/ OpenVPN] is available.

The setup is client-server, with per-client certificates.

== OpenVPN client configuration ==

=== Raw OpenVPN ===

Sample configuration file, e.g., /etc/openvpn/swh.conf:

<pre>
remote vpn.softwareheritage.org
ns-cert-type server
comp-lzo
nobind
dev tun
proto udp
port 1194
log /var/log/openvpn.log
up-restart
persist-key
persist-tun
client
ca /etc/openvpn/keys/softwareheritage-ca.crt
cert /etc/openvpn/keys/softwareheritage.crt
key /etc/openvpn/keys/softwareheritage.key
user nobody
group nogroup

# If you are using resolvconf, add this:
# Make sure you add louvre to /etc/hosts to avoid issues in using the vpn-provided DNS server.
script-security 2
up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf

# If you want the connection to persist when your network fails, add this:
ping-restart 10
</pre>

In addition to the above configuration file, you will need to install the following 3 files under /etc/openvpn/keys (matching the paths within the sample above):

* '''[[softwareheritage-ca.crt]]''': ''public'' certificate for the Software Heritage certification authority (CA)
* '''[https://wiki.softwareheritage.org/index.php?title=VPN#For_admins softwareheritage.crt]''': ''public'', client-specific (certificate signed by the admin, see below)
* '''[https://wiki.softwareheritage.org/wiki/VPN#For_users softwareheritage.key]''': ''private'', client-specific key (generated by the user, see below)

Activate the openvpn server

as root, run

systemctl enable openvpn@swh.service
systemctl start openvpn@swh.service
systemctl status openvpn@swh.service

Note: Internally, the `swh` must match the /etc/openvpn/swh.conf filename.

Excerpt of a successful start:

root@machine:~# systemctl status openvpn@swh.service
openvpn@swh.service - OpenVPN connection to swh
Loaded: loaded (/lib/systemd/system/openvpn@.service; indirect; vendor preset: enabled)
Active: active (running) since Thu 2020-12-17 19:03:29 IST; 22min ago
Docs: man:openvpn(8)
https://community.openvpn.net/openvpn/wiki/Openvpn24ManPage
https://community.openvpn.net/openvpn/wiki/HOWTO
Main PID: 12302 (openvpn)
Status: "Initialization Sequence Completed"
Tasks: 1 (limit: 4915)
CGroup: /system.slice/system-openvpn.slice/openvpn@swh.service
└─12302 /usr/sbin/openvpn --daemon ovpn-swh --status /run/openvpn/swh.status 10 --cd /etc/openvpn --script-security 2 --config /etc/openvpn/swh.conf --writepid /run/openvpn/swh.pid

Dec 17 19:03:29 machine systemd[1]: Starting OpenVPN connection to swh...
Dec 17 19:03:29 machine systemd[1]: Started OpenVPN connection to swh.

=== Network Manager GUI ===

You need network-manager-openvpn and network-manager-openvpn-gnome for the configuration gui.

[[File:nm_openvpn_base.png]]
[[File:nm_openvpn_routes.png]]
[[File:nm_openvpn_advanced_general.png]]
[[File:nm_openvpn_advanced_security.png]]
[[File:nm_openvpn_advanced_tls_auth.png]]

== Obtaining a client certificate ==

=== For users ===

Generate a keypair (key + certificate signing request) using the following command:

<pre>
openssl req -new -newkey rsa:2048 -nodes -keyout openvpn.key -out openvpn.csr -subj "/CN=<your username>"
</pre>

Please replace <your username> with something that uniquely identifies the certificate.

Make sure openvpn.key is stored in a safe place (it's your private key, which will allow anyone to connect to the VPN).

Provide the CSR file to a sysadmin through a reasonably authenticated medium.

=== For admins ===

On louvre:

Fetch the CSR file provided by the user, for instance with <tt>scp USERNAME.csr louvre:</tt>

Then, as root on louvre:

<pre>
root@louvre:~# cd /etc/openvpn/keys
root@louvre:/etc/openvpn/keys# ./easyrsa import-req ~ADMIN/USERNAME.csr USERNAME
root@louvre:/etc/openvpn/keys# ./easyrsa sign-req client USERNAME
</pre>

The first command imports the csr into the EasyRSA PKI. The second command lets you review and sign it.

Send the signed certificate, <tt>/etc/openvpn/keys/pki/issued/USERNAME.crt</tt>, to the user. That file only contains public key material.

Add the DNS entry for the new host to hiera and do a puppet run on pergamon.

== Revoking a client certificate ==

On louvre:

<pre>
root@louvre:~# cd /etc/openvpn/keys
root@louvre:/etc/openvpn/keys# ./easyrsa revoke USERNAME
[ say yes ]
root@louvre:/etc/openvpn/keys# ./easyrsa gen-crl; chmod a+r pki/crl.pem
</pre>

OpenVPN re-reads the CRL at each connection (which is why we need the CRL to be world-readable), so once the cert is revoked, there's nothing more to do. If you want to make sure the client is disconnected, you need to restart OpenVPN (which will make all clients reconnect).

== /etc/hosts entries ==

Once the Vpn is setup on your machine, you can access Software Heritage hosts via their private IP addresses; see [[Network configuration]].

OpenVPN now pushes the address of our DNS server (192.168.100.29, pergamon).

You might want to add louvre.softwareheritage.org in your /etc/hosts to avoid a bootstrap problem if the "on-vpn" DNS server is in your resolv.conf.

[[Category:Infrastructure]]
[[Category:System administration]]

Matrix

2021-06-15T15:53:24Z

NicolasDandrimont: /* IRC authentication */

== IRC channels ==

The following channels have been registered on the [https://libera.chat/ libera.chat] IRC network for [[Software Heritage]] usage.

* [https://app.element.io/#/room/#swh-devel:matrix.org '''#swh-devel''']: public development discussions
* [https://app.element.io/#/room/#swh-sysadm:matrix.org '''#swh-sysadm''']: operations team discussions/bots
* [https://app.element.io/#/room/#swh-offtopic:matrix.org '''#swh-offtopic''']: Off-topic discussions
* [https://app.element.io/#/room/#swh-team:matrix.org '''#swh-team''']: private discussions of the core team

If you use IRC, consider joining the channels.

If you don't use IRC ''directly'', you can still join our chat channels from your web browser via a [https://matrix.org/ Matrix] bridge by clicking on the channel names in the list above. You will be asked to create a [https://element.io/ Element] account if you don't have one yet.

== IRC authentication ==

Libera.chat defaults to blocking private messages from unauthentified users! All users should register their nicknames to be able to message one another privately, by following the instructions below. If you're really unable to register, you should ask your correspondent to [https://libera.chat/guides/usermodes consider setting usermode <tt>-R</tt>, and <tt>+g</tt>]

To register an account with NickServ, please follow [https://libera.chat/guides/registration the registration instructions provided by libera.chat staff].

You will then receive an e-mail containing a link to activate you account. After doing so, you need to configure your client to auto-authenticate. The recommended way of doing that is using [https://libera.chat/guides/sasl SASL authentication].

For matrix, the relevant docs is here: https://github.com/matrix-org/matrix-appservice-irc/wiki/End-user-FAQ#how-do-i-registeridentify-to-nickserv

libera.chat also supports authentication via [https://libera.chat/guides/certfp TLS client certificates (using SASL EXTERNAL)].

=== Matrix bridge ===

For registering an account through the Matrix bridge ([https://github.com/matrix-org/matrix-appservice-irc/wiki/End-user-FAQ#how-do-i-registeridentify-to-nickserv relevant docs here]), please follow these instructions:

1. Choose a short nickname (the default nickname picked by the matrix bridge has a [m] and can be quite long, as it defaults to the full local part of your matrix handle)

/msg @appservice:libera.chat !nick <USERNAME>

2. Send this command to NickServ to register your account:

/msg @NickServ:libera.chat register <PASSWORD> <EMAIL>

3. Once you receive the confirmation email with a token, activate your account by using:

/msg @NickServ:libera.chat VERIFY REGISTER <USERNAME> <TOKEN RECEIVED BY EMAIL>

4. Give the Matrix bridge appservice your password so that you get identified automatically when matrix reconnects you to IRC:

/msg @appservice:libera.chat !username <USERNAME>
/msg @appservice:libera.chat !storepass <PASSWORD>

== IRC access list ==

To auto-voice people with a registered nick (only doable by people with +fA access modes will be able to do it), add them to the team channel access list:

/msg chanserv flags #swh-team add <nickname> Staff

Other channels pick their ACLs from that of the #swh-team channel.

If you already have the right (+o ChanServ flag), you can make yourself an operator, with:

/msg chanserv OP #swh-devel

[[Category:Infrastructure]]

Matrix

2021-06-15T15:52:47Z

NicolasDandrimont: /* IRC authentication */

== IRC channels ==

The following channels have been registered on the [https://libera.chat/ libera.chat] IRC network for [[Software Heritage]] usage.

* [https://app.element.io/#/room/#swh-devel:matrix.org '''#swh-devel''']: public development discussions
* [https://app.element.io/#/room/#swh-sysadm:matrix.org '''#swh-sysadm''']: operations team discussions/bots
* [https://app.element.io/#/room/#swh-offtopic:matrix.org '''#swh-offtopic''']: Off-topic discussions
* [https://app.element.io/#/room/#swh-team:matrix.org '''#swh-team''']: private discussions of the core team

If you use IRC, consider joining the channels.

If you don't use IRC ''directly'', you can still join our chat channels from your web browser via a [https://matrix.org/ Matrix] bridge by clicking on the channel names in the list above. You will be asked to create a [https://element.io/ Element] account if you don't have one yet.

== IRC authentication ==

Libera.chat defaults to blocking private messages from unauthentified users! All users should register their nicknames to be able to message one another privately, by following the instructions below. If you're really unable to register, you should ask your correspondent to [https://libera.chat/guides/usermodes consider setting usermode <tt>-R</tt>, and <tt>+g</tt>]

You should register your nickname with NickServ following [https://libera.chat/guides/registration the registration instructions provided by libera.chat staff].

You will then receive an e-mail containing a link to activate you account. After doing so, you need to configure your client to auto-authenticate. The recommended way of doing that is using [https://libera.chat/guides/sasl SASL authentication].

For matrix, the relevant docs is here: https://github.com/matrix-org/matrix-appservice-irc/wiki/End-user-FAQ#how-do-i-registeridentify-to-nickserv

libera.chat also supports authentication via [https://libera.chat/guides/certfp TLS client certificates (using SASL EXTERNAL)].

=== Matrix bridge ===

For registering an account through the Matrix bridge ([https://github.com/matrix-org/matrix-appservice-irc/wiki/End-user-FAQ#how-do-i-registeridentify-to-nickserv relevant docs here]), please follow these instructions:

1. Choose a short nickname (the default nickname picked by the matrix bridge has a [m] and can be quite long, as it defaults to the full local part of your matrix handle)

/msg @appservice:libera.chat !nick <USERNAME>

2. Send this command to NickServ to register your account:

/msg @NickServ:libera.chat register <PASSWORD> <EMAIL>

3. Once you receive the confirmation email with a token, activate your account by using:

/msg @NickServ:libera.chat VERIFY REGISTER <USERNAME> <TOKEN RECEIVED BY EMAIL>

4. Give the Matrix bridge appservice your password so that you get identified automatically when matrix reconnects you to IRC:

/msg @appservice:libera.chat !username <USERNAME>
/msg @appservice:libera.chat !storepass <PASSWORD>

== IRC access list ==

To auto-voice people with a registered nick (only doable by people with +fA access modes will be able to do it), add them to the team channel access list:

/msg chanserv flags #swh-team add <nickname> Staff

Other channels pick their ACLs from that of the #swh-team channel.

If you already have the right (+o ChanServ flag), you can make yourself an operator, with:

/msg chanserv OP #swh-devel

[[Category:Infrastructure]]

Matrix

2021-06-08T16:03:42Z

NicolasDandrimont: Update instructions for libera.chat and new matrix rooms.

== IRC channels ==

The following channels have been registered on the [https://libera.chat/ libera.chat] IRC network for [[Software Heritage]] usage.

* [https://app.element.io/#/room/#swh-devel:matrix.org '''#swh-devel''']: public development discussions
* [https://app.element.io/#/room/#swh-sysadm:matrix.org '''#swh-sysadm''']: operations team discussions/bots
* [https://app.element.io/#/room/#swh-offtopic:matrix.org '''#swh-offtopic''']: Off-topic discussions
* [https://app.element.io/#/room/#swh-team:matrix.org '''#swh-team''']: private discussions of the core team

If you use IRC, consider joining the channels.

If you don't use IRC ''directly'', you can still join our chat channels from your web browser via a [https://matrix.org/ Matrix] bridge by clicking on the channel names in the list above. You will be asked to create a [https://element.io/ Element] account if you don't have one yet.

== IRC authentication ==

You should register your nickname with NickServ following [https://libera.chat/guides/registration the registration instructions provided by libera.chat staff].

You will then receive an e-mail containing a link to activate you account. After doing so, you need to configure your client to auto-authenticate. The recommended way of doing that is using [https://libera.chat/guides/sasl SASL authentication].

For matrix, the relevant docs is here: https://github.com/matrix-org/matrix-appservice-irc/wiki/End-user-FAQ#how-do-i-registeridentify-to-nickserv

libera.chat also supports authentication via [https://libera.chat/guides/certfp TLS client certificates (using SASL EXTERNAL)].

=== Matrix bridge ===

For registering an account through the Matrix bridge ([https://github.com/matrix-org/matrix-appservice-irc/wiki/End-user-FAQ#how-do-i-registeridentify-to-nickserv relevant docs here]), please follow these instructions:

1. Choose a short nickname (the default nickname picked by the matrix bridge has a [m] and can be quite long, as it defaults to the full local part of your matrix handle)

/msg @appservice:libera.chat !nick <USERNAME>

2. Send this command to NickServ to register your account:

/msg @NickServ:libera.chat register <PASSWORD> <EMAIL>

3. Once you receive the confirmation email with a token, activate your account by using:

/msg @NickServ:libera.chat VERIFY REGISTER <USERNAME> <TOKEN RECEIVED BY EMAIL>

4. Give the Matrix bridge appservice your password so that you get identified automatically when matrix reconnects you to IRC:

/msg @appservice:libera.chat !username <USERNAME>
/msg @appservice:libera.chat !storepass <PASSWORD>

== IRC access list ==

To auto-voice people with a registered nick (only doable by people with +fA access modes will be able to do it), add them to the team channel access list:

/msg chanserv flags #swh-team add <nickname> Staff

Other channels pick their ACLs from that of the #swh-team channel.

If you already have the right (+o ChanServ flag), you can make yourself an operator, with:

/msg chanserv OP #swh-devel

[[Category:Infrastructure]]

Arcanist setup

2021-03-22T11:12:51Z

NicolasDandrimont: Redirect to swhdocs

#REDIRECT [[swhdocs:devel/contributing/phabricator.html#arcanist-configuration]]

[[Category:Software development]]

Code review

2021-03-22T11:12:15Z

NicolasDandrimont: Redirect to swhdocs

#REDIRECT [[swhdocs:devel/contributing/code-review.html#code-review]]

[[Category:Software development]]

Code review in Phabricator

2021-03-22T11:11:37Z

NicolasDandrimont: Redirect to swhdocs

#REDIRECT [[swhdocs:devel/contributing/code-review.html#code-review]]

[[Category:Software development]]

Python style guide

2021-03-22T11:10:56Z

NicolasDandrimont: Redirect to swhdocs

#REDIRECT [[swhdocs:devel/contributing/python-style-guide.html]]

[[Category:Guidelines]]
[[Category:Software development]]
[[Category:Python]]

Git style guide

2021-03-22T11:06:36Z

NicolasDandrimont: Use interwiki redirect

#REDIRECT [[swhdocs:devel/contributing/git-style-guide.html]]

HedgeDoc

2021-02-03T14:18:23Z

NicolasDandrimont: Create a basic hedgedoc documentation page

= HedgeDoc =

[https://hedgedoc.org/ HedgeDoc] (formerly known as CodiMD, formerly known as HackMD) is a collaborative, web-based markdown editor (a.k.a a "pad").

Software Heritage hosts an instance of HedgeDoc on https://hedgedoc.softwareheritage.org/

== Features ==

A list of the HedgeDoc features is available as a [https://hedgedoc.softwareheritage.org/features?both self-hosted document on the instance]. Most notably, compared to other pads, as the owner of a note (after you're logged in), you can configure whether guests can read or edit the note, or whether other logged in users can read or edit the note.

== Access ==

Hedgedoc is accessible to the public; Guests can (at least currently) create notes as well. Authentication is wired to the Software Heritage SSO service.

== Sysadmin information ==

Hedgedoc is hosted on [https://inventory.internal.softwareheritage.org/virtualization/virtual-machines/91/ bardo] (with its postgresql database backend as well as storage for uploaded files). Web access is going through a public access [https://inventory.internal.softwareheritage.org/virtualization/virtual-machines/98/ reverse proxy].

It is deployed using the <tt>profile::hedgedoc</tt> puppet manifest.

File:Nm openvpn routes.png

2020-07-17T08:50:52Z

NicolasDandrimont:

File:Nm openvpn base.png

2020-07-17T08:50:38Z

NicolasDandrimont:

File:Nm openvpn advanced tls auth.png

2020-07-17T08:48:31Z

NicolasDandrimont:

File:Nm openvpn advanced security.png

2020-07-17T08:48:15Z

NicolasDandrimont:

File:Nm openvpn advanced general.png

2020-07-17T08:48:00Z

NicolasDandrimont:

Softwareheritage-ca.crt

2020-07-17T08:44:12Z

NicolasDandrimont: 3 revisions imported

Public certificate for the [[Software Heritage]] certification authority (CA) installed on louvre.softwareheritage.org:

<pre>
-----BEGIN CERTIFICATE-----
MIIEqjCCA5KgAwIBAgIJAJjv4lm3AwjgMA0GCSqGSIb3DQEBCwUAMIGUMQswCQYD
VQQGEwJGUjEOMAwGA1UEBxMFUGFyaXMxGjAYBgNVBAoTEVNvZnR3YXJlIEhlcml0
YWdlMR0wGwYDVQQDExRTb2Z0d2FyZSBIZXJpdGFnZSBDQTEQMA4GA1UEKRMHRWFz
eVJTQTEoMCYGCSqGSIb3DQEJARYZaW5mb0Bzb2Z0d2FyZWhlcml0YWdlLm9yZzAe
Fw0xNTA3MTcwOTA0MTBaFw0yNTA3MTQwOTA0MTBaMIGUMQswCQYDVQQGEwJGUjEO
MAwGA1UEBxMFUGFyaXMxGjAYBgNVBAoTEVNvZnR3YXJlIEhlcml0YWdlMR0wGwYD
VQQDExRTb2Z0d2FyZSBIZXJpdGFnZSBDQTEQMA4GA1UEKRMHRWFzeVJTQTEoMCYG
CSqGSIb3DQEJARYZaW5mb0Bzb2Z0d2FyZWhlcml0YWdlLm9yZzCCASIwDQYJKoZI
hvcNAQEBBQADggEPADCCAQoCggEBAL9yJKQaRh3kqTXqzwtw6Kiugs1ZQV7UejUI
i9K4lXqLF1seIIq8beJPYMsBcHK6fi3oXJ51ejqCG1sBo4zI6i6hCrk4QOduF5x7
xK173La8q6K5S+lcevYqROLPRh96MqhMwQvWhy/Y/Q2VeuxfNZxNUs5BB7Y6DWFw
S36v0xntWzEH/0FQQ2phebY3tKMaA9ftPlSSVNkH39b10x1QMrRKE8T+RIuOnbjm
RbYm6WXZetMIDP9TNdBQpJ4VDTHNobFScAFW4qzeCW4q0gB94jK+lyOOKzRuRIn2
3oPUXIahhO4HAkmfA8++UzwX8Wigu1+8dGL84y14cy6YE3E38DkCAwEAAaOB/DCB
+TAdBgNVHQ4EFgQUkyOIziKlJuhVuVcu1VNpBaL9+mwwgckGA1UdIwSBwTCBvoAU
kyOIziKlJuhVuVcu1VNpBaL9+myhgZqkgZcwgZQxCzAJBgNVBAYTAkZSMQ4wDAYD
VQQHEwVQYXJpczEaMBgGA1UEChMRU29mdHdhcmUgSGVyaXRhZ2UxHTAbBgNVBAMT
FFNvZnR3YXJlIEhlcml0YWdlIENBMRAwDgYDVQQpEwdFYXN5UlNBMSgwJgYJKoZI
hvcNAQkBFhlpbmZvQHNvZnR3YXJlaGVyaXRhZ2Uub3JnggkAmO/iWbcDCOAwDAYD
VR0TBAUwAwEB/zANBgkqhkiG9w0BAQsFAAOCAQEAg8bd3vOwqtdJ423ON3CDAnlC
DUm3Lq+iyD8YLyeueXvvI7tvrl2uxLc0KXzOksEi1PvXimAqtFoy9CtpfQGywP7V
DMXc6hom6ddXD08vUknVfgRr6XRL0veDqGC3evTVKXjiDCQhUO2NuFYuaMGufSFx
eCjx1h3huxv7Z4g3n1zkkpBGVIHYO+9r09/8ufTImYMZ9BF5ZQaotstcGvCV6TgV
kLn4o2DOLhGeDpVQUDKNNxRpQCXC0bLBaDVBkP/2pp/QmHHgrJOydpK/TaLGYT2f
eEdfH3QKmPR/AyH2qQTuSxKPPnCzWBYyjLHcKAgjL4023ObDV1YUlSbf08QVcA==
-----END CERTIFICATE-----
</pre>

== Services that use the certificate ==

* [[Openvpn]]

[[Category:Infrastructure]]
[[Category:System administration]]

VPN

2020-07-17T08:44:12Z

NicolasDandrimont: 23 revisions imported

The [[Software Heritage]] server and the VMs running on it are severely firewalled.
To get onto their network unrestricted, a VPN based on [https://openvpn.net/ OpenVPN] is available.

The setup is client-server, with per-client certificates.

== OpenVPN client configuration ==

=== Raw OpenVPN ===

Sample configuration file, e.g., /etc/openvpn/softwareheritage.conf:

<pre>
remote louvre.softwareheritage.org
ns-cert-type server
comp-lzo
nobind
dev tun
proto udp
port 1194
log /var/log/openvpn.log
up-restart
persist-key
persist-tun
client
ca /etc/openvpn/keys/softwareheritage-ca.crt
cert /etc/openvpn/keys/softwareheritage.crt
key /etc/openvpn/keys/softwareheritage.key
user nobody
group nogroup

# If you are using resolvconf, add this:
# Make sure you add louvre to /etc/hosts to avoid issues in using the vpn-provided DNS server.
script-security 2
up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf

# If you want the connection to persist when your network fails, add this:
ping-restart 10
</pre>

In addition to the above configuration file, you will need to install the following 3 files under /etc/openvpn/keys:

* '''[[softwareheritage-ca.crt]]''': ''public'' certificate for the Software Heritage certification authority (CA)
* '''softwareheritage.crt''': ''public'', client-specific certificate (see below)
* '''softwareheritage.key''': ''private'', client-specific key (see below)

=== Network Manager GUI ===

You need network-manager-openvpn and network-manager-openvpn-gnome for the configuration gui.

[[File:nm_openvpn_base.png]]
[[File:nm_openvpn_routes.png]]
[[File:nm_openvpn_advanced_general.png]]
[[File:nm_openvpn_advanced_security.png]]
[[File:nm_openvpn_advanced_tls_auth.png]]

== Obtaining a client certificate ==

=== For users ===

Generate a keypair (key + certificate signing request) using the following command:

<pre>
openssl req -new -newkey rsa:2048 -nodes -keyout openvpn.key -out openvpn.csr -subj "/CN=<your username>"
</pre>

Please replace <your username> with something that uniquely identifies the certificate.

Make sure openvpn.key is stored in a safe place (it's your private key, which will allow anyone to connect to the VPN).

Provide the CSR file to a sysadmin through a reasonably authenticated medium.

=== For admins ===

On louvre:

Fetch the CSR file provided by the user, for instance with <tt>scp USERNAME.csr louvre:</tt>

Then, as root on louvre:

<pre>
root@louvre:~# cd /etc/openvpn/keys
root@louvre:/etc/openvpn/keys# ./easyrsa import-req ~ADMIN/USERNAME.csr USERNAME
root@louvre:/etc/openvpn/keys# ./easyrsa sign-req client USERNAME
</pre>

The first command imports the csr into the EasyRSA PKI. The second command lets you review and sign it.

Send the signed certificate, <tt>/etc/openvpn/keys/pki/issued/USERNAME.crt</tt>, to the user. That file only contains public key material.

Add the DNS entry for the new host to hiera and do a puppet run on pergamon.

== Revoking a client certificate ==

On louvre:

<pre>
root@louvre:~# cd /etc/openvpn/keys
root@louvre:/etc/openvpn/keys# ./easyrsa revoke USERNAME
[ say yes ]
root@louvre:/etc/openvpn/keys# ./easyrsa gen-crl; chmod a+r pki/crl.pem
</pre>

OpenVPN re-reads the CRL at each connection (which is why we need the CRL to be world-readable), so once the cert is revoked, there's nothing more to do. If you want to make sure the client is disconnected, you need to restart OpenVPN (which will make all clients reconnect).

== /etc/hosts entries ==

Once the Vpn is setup on your machine, you can access Software Heritage hosts via their private IP addresses; see [[Network configuration]].

OpenVPN now pushes the address of our DNS server (192.168.100.29, pergamon).

You might want to add louvre.softwareheritage.org in your /etc/hosts to avoid a bootstrap problem if the "on-vpn" DNS server is in your resolv.conf.

[[Category:Infrastructure]]
[[Category:System administration]]

VPN

2019-12-04T17:41:51Z

NicolasDandrimont: /* Obtaining a client certificate */ Improve instructions to avoid key material transfers.

The [[Software Heritage]] server and the VMs running on it are severely firewalled.
To get onto their network unrestricted, a VPN based on [https://openvpn.net/ OpenVPN] is available.

The setup is client-server, with per-client certificates.

== OpenVPN client configuration ==

=== Raw OpenVPN ===

Sample configuration file, e.g., /etc/openvpn/softwareheritage.conf:

<pre>
remote louvre.softwareheritage.org
ns-cert-type server
comp-lzo
nobind
dev tun
proto udp
port 1194
log /var/log/openvpn.log
up-restart
persist-key
persist-tun
client
ca /etc/openvpn/keys/softwareheritage-ca.crt
cert /etc/openvpn/keys/softwareheritage.crt
key /etc/openvpn/keys/softwareheritage.key
user nobody
group nogroup

# If you are using resolvconf, add this:
# Make sure you add louvre to /etc/hosts to avoid issues in using the vpn-provided DNS server.
script-security 2
up /etc/openvpn/update-resolv-conf
down /etc/openvpn/update-resolv-conf

# If you want the connection to persist when your network fails, add this:
ping-restart 10
</pre>

In addition to the above configuration file, you will need to install the following 3 files under /etc/openvpn/keys:

* '''[[softwareheritage-ca.crt]]''': ''public'' certificate for the Software Heritage certification authority (CA)
* '''softwareheritage.crt''': ''public'', client-specific certificate (see below)
* '''softwareheritage.key''': ''private'', client-specific key (see below)

=== Network Manager GUI ===

You need network-manager-openvpn and network-manager-openvpn-gnome for the configuration gui.

[[File:nm_openvpn_base.png]]
[[File:nm_openvpn_routes.png]]
[[File:nm_openvpn_advanced_general.png]]
[[File:nm_openvpn_advanced_security.png]]
[[File:nm_openvpn_advanced_tls_auth.png]]

== Obtaining a client certificate ==

=== For users ===

Generate a keypair (key + certificate signing request) using the following command:

<pre>
openssl req -new -newkey rsa:2048 -nodes -keyout openvpn.key -out openvpn.csr -subj "/CN=<your username>"
</pre>

Please replace <your username> with something that uniquely identifies the certificate.

Make sure openvpn.key is stored in a safe place (it's your private key, which will allow anyone to connect to the VPN).

Provide the CSR file to a sysadmin through a reasonably authenticated medium.

=== For admins ===

On louvre:

Fetch the CSR file provided by the user, for instance with <tt>scp USERNAME.csr louvre:</tt>

Then, as root on louvre:

<pre>
root@louvre:~# cd /etc/openvpn/keys
root@louvre:/etc/openvpn/keys# ./easyrsa import-req ~ADMIN/USERNAME.csr USERNAME
root@louvre:/etc/openvpn/keys# ./easyrsa sign-req client USERNAME
</pre>

The first command imports the csr into the EasyRSA PKI. The second command lets you review and sign it.

Send the signed certificate, <tt>/etc/openvpn/keys/pki/issued/USERNAME.crt</tt>, to the user. That file only contains public key material.

Add the DNS entry for the new host to hiera and do a puppet run on pergamon.

== Revoking a client certificate ==

On louvre:

<pre>
root@louvre:~# cd /etc/openvpn/keys
root@louvre:/etc/openvpn/keys# ./easyrsa revoke USERNAME
[ say yes ]
root@louvre:/etc/openvpn/keys# ./easyrsa gen-crl; chmod a+r pki/crl.pem
</pre>

OpenVPN re-reads the CRL at each connection (which is why we need the CRL to be world-readable), so once the cert is revoked, there's nothing more to do. If you want to make sure the client is disconnected, you need to restart OpenVPN (which will make all clients reconnect).

== /etc/hosts entries ==

Once the Vpn is setup on your machine, you can access Software Heritage hosts via their private IP addresses; see [[Network configuration]].

OpenVPN now pushes the address of our DNS server (192.168.100.29, pergamon).

You might want to add louvre.softwareheritage.org in your /etc/hosts to avoid a bootstrap problem if the "on-vpn" DNS server is in your resolv.conf.

[[Category:Infrastructure]]
[[Category:System administration]]

VPN

2019-12-04T17:33:48Z

NicolasDandrimont: /* For users */

VPN

2019-12-04T17:32:00Z

NicolasDandrimont: /* Revoking a client certificate */ update CRL instructions

VPN

2019-12-04T17:03:53Z

NicolasDandrimont: /* Revoking a client certificate */ Update to easy-rsa 3.x

VPN

2019-12-04T17:03:07Z

NicolasDandrimont: /* For admins */ Migrate to easyrsa 3.x

Code review in Phabricator

2019-06-18T13:55:07Z

NicolasDandrimont: /* VCS password for pushes */

We use the [[Differential]] application of [[Phabricator]] to perform [[code review|code reviews]] in the context of [[Software Heritage]].

* we use Git and history.immutable=true (but beware as that is partly a Phabricator misnomer, read on)
* when code reviews are required, developers will be allowed to push directly to master once an accepted Differential diff exists

== Configuration ==

=== Arcanist configuration ===

When using git, [[Arcanist]] by default mess with the local history, rewriting commits at the time of first submission. 
To avoid that we use so called [https://secure.phabricator.com/book/phabricator/article/arcanist_new_project/#history-mutability-git history immutability].

To that end, you shall configure your <tt>arc</tt> accordingly:
<pre>
arc set-config history.immutable true
</pre>

Note that this does ''not'' mean that you are forbidden to rewrite your local branches (e.g., with <tt>git rebase</tt>).
Quite the contrary: you are encouraged to locally rewrite branches before pushing to ensure that commits are logically separated and your commit history easy to bisect.
The above setting just means that ''arc'' will not rewrite commit history under your nose.

=== Enabling <tt>git push</tt> to our forge ===

The way we've configured our review setup for continuous integration needs you to configure git to allow pushes to our forge. There's two ways you can do this : setting a ssh key to push over ssh, or setting a specific password for git pushes over https.

==== SSH key for pushes ====

In your forge User settings page (On the top right, click on your avatar, then click ''Settings''), you have access to a ''Authentication'' > ''SSH Public Keys'' section (Direct link: <tt>hxxps://forge.softwareheritage.org/settings/user/'''<your username>'''/page/ssh/</tt>). You then have the option to upload a SSH public key, which will authenticate your pushes.

You then need to configure ssh/git to use that key pair, for instance by editing the <tt>~/.ssh/config</tt> file.

Finally, you should configure git to push over ssh when pushing to https://forge.softwareheritage.org, by running the following command:
git config --global url.git@forge.softwareheritage.org:.pushInsteadOf https://forge.softwareheritage.org

This lets git know that it should use <tt>git@forge.softwareheritage.org:</tt> as a base url when pushing repositories cloned from forge.softwareheritage.org over https.

==== VCS password for pushes ====

If you're not comfortable setting up SSH to upload your changes, you have the option of setting a VCS password. This password, ''separate from your account password'', allows Phabricator to authenticate your uploads over HTTPS.

In your forge User settings page (On the top right, click on your avatar, then click ''Settings''), you need to use the ''Authentication'' > ''VCS Password'' section to set your VCS password (Direct link: <tt>hxxps://forge.softwareheritage.org/settings/user/'''<your username>'''/page/vcspassword/</tt>).

If you still get a 403 error on push, this means you need a forge administrator to enable HTTPS pushes for the repository (which wasn't done by default in historical repositories). Please drop by on IRC and let us know!

== Workflow ==

* work in a feature branch: <tt>git checkout -b my-feat</tt>
* initial review request: hack/commit/hack/commit ; <tt>arc diff origin/master</tt>
* react to change requests: hack/commit/hack/commit ; <tt>arc diff --update Dxx origin/master</tt>
* landing change: <tt>git checkout master ; git merge my-feat ; git push</tt>

=== Starting a new feature and submit it for review ===

Use a '''one branch per feature''' workflow, with well-separated ''logical commits'' ([https://wiki.softwareheritage.org/wiki/Git_style_guide following those conventions])
<pre>
git checkout -b my-shiny-feature
... hack hack hack ...
git commit -m 'architecture skeleton for my-shiny-feature'
... hack hack hack ...
git commit -m 'my-shiny-feature: implement module foo'
... etc ...
</pre>

Please, follow the
To '''submit your code for review''' the first time:
<pre>
arc diff origin/master
</pre>
arc will prompt for a '''code review message'''. Provide the following information:
* first line: ''short description'' of the overall work (i.e., the feature you're working on). This will become the title of the review
* ''Summary'' field (optional): ''long description'' of the overall work; the field can continue in subsequent lines, up to the next field. This will become the "Summary" section of the review
* ''Test Plan'' field (optional): write here if something special is needed to test your change
* ''Reviewers'' field (optional): the (Phabricator) name(s) of desired reviewers. If you don't specify one (recommended) the default reviewers will be chosen
* ''Subscribers'' field (optional): the (Phabricator) name(s) of people that will be notified about changes to this review request. In most cases it should be left empty

For example:
<pre>
mercurial loader

Summary: first stab at a mercurial loader (T329)

The implementation follows the plan detailed in F2F discussion with @foo.

Performances seem decent enough for a first trial (XXX seconds for YYY repository
that contains ZZZ patches).

Test plan:

Reviewers:

Subscribers: foo
</pre>

After completing the message arc will submit the review request and tell you its number and URL:
<pre>
[...]
Created a new Differential revision:
Revision URI: https://forge.softwareheritage.org/Dxx
</pre>

=== Updating your branch to reflect requested changes ===

Your feature might get accepted as is, YAY!
Or, reviewers might request changes; no big deal!

Use the Differential web UI to follow-up to received comments, if needed.

To implement requested changes in the code, hack on your branch as usual by:

* adding new commits, and/or
* rewriting old commits with git rebase (to preserve a nice, easy to bisect history)

When you're ready to '''update your review request''':
<pre>
arc diff --update Dxx origin/master
</pre>

Arc will prompt you for a message: describe what you've changed w.r.t. the previous review request, free form.
Your message will become the changelog entry in Differential for this new version of the diff.

Differential only care about the code diff, and not about the commits or their order.
Therefore each "update" can be a completely different series of commits, possibly rewritten from the previous submission.

=== Landing your change onto master ===

Once your change has been approved in Differential, you will be able to land it onto the master branch.

Before doing so, you're encouraged to '''clean up your git commit history''', reordering/splitting/merging commits as needed to have separate logical commits and an easy to bisect history.
Update the diff [https://wiki.softwareheritage.org/wiki/Code_review_in_Phabricator#Updating_your_branch_to_reflect_requested_changes following the prior section].
(It'd be good to let the ci build finish to make sure everything is still green).

Once you're happy you can '''push to origin/master''' directly, e.g.:
<pre>
git checkout master
git merge my-shiny-feature
git push
</pre>

Optionally you can then delete your local feature branch:
<pre>
git branch -d my-shiny-feature
</pre>

=== Reviewing locally / landing someone else's changes ===

You can do local reviews of code with arc patch:

<pre>
arc patch Dxyz
</pre>

This will create a branch '''arcpatch-Dxyz''' containing the changes on your local checkout.

You can then merge those changes upstream with

<pre>
git checkout master
git merge --ff arcpatch-Dxyz
git push origin master
</pre>

or, alternatively:

<pre>
arc land --squash
</pre>

== See also ==

* [[Code review]] for guidelines on how code is reviewed when developing for Software Heritage

[[Category:Software development]]

Google Season of Docs 2019

2019-05-13T12:23:59Z

NicolasDandrimont: fix Season of Docs timeline

[[File:GSoDLogo.png|800px]]

== General information ==

This page is the central point of information for [[Software Heritage]] participation into the [https://developers.google.com/season-of-docs/ Google Season of Docs] program.

Google Summer of Code is a program where Google pays technical writers stipends to work on free software projects such as Software Heritage. Each writer works with mentors from the community to complete a documentation project.

== I want to participate as a technical writer ==

Great!, we are very glad for your interest in contributing to Software Heritage and we are looking forward to work together.

=== Prerequisites ===

The following prerequisites apply to Software Heritage GSoD projects:

* [http://www.sphinx-doc.org/ Sphinx] is our documentation system of choice, you should be familiar with it to apply. In particular, we generally use reStructuredText markup and (for API references) Python docstrings with the [https://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html Napoleon style]
* [https://git-scm.com Git] is our version control system of choice, you should be familiar with it to apply
* additional prerequisites depend on the project you will work on; check project descriptions for details

=== Before you apply ===

Here are the steps you should follow before applying, to make sure you have a general idea of the current state of Software Heritage technical documentation:

* Learn about our project via our [https://www.softwareheritage.org/ main website] and the actual [https://archive.softwareheritage.org/ source code archive]
* Check out the index of our [https://www.softwareheritage.org/community/developers/ resources for developers]
* In particular make sure to have a look at:
** our [https://wiki.softwareheritage.org/ public wiki]
** our [https://docs.softwareheritage.org/devel/ documentation index]
* Technical setup:
** Create an account on our [https://forge.softwareheritage.org development forge]
** Familiarize yourself with our [[Code review in Phabricator|code review workflow]]
** Make a simple change to the documentation of any one of our [https://docs.softwareheritage.org/devel/ software components] and submit it as a [https://forge.softwareheritage.org/differential/ diff] for code review, following the above workflow. Feel free to submit any patch you think it might be useful.

=== What to include in your application ===

Make sure that your application includes the following information:

* Describe the '''specific project''' you want to work on. What do you want to achieve? Why is it important? Why is it useful for Software Heritage? The project might be one of the project ideas that we have prepared below, or something else entirely that you want to contribute to Software Heritage. Your pet peeve, surprise us!
* Detail your '''work plan''': a brief description of how you plan to go about your project, including a list of ''deliverables'' and a ''timeline'' of when do you expect them to be available.
* Include a reference to '''the diff''' you submitted before applying (see the "Before you apply" section above).

== Ideas list ==

Below you can find a list of project ideas that are good options for a
reasonably sized GSoD project. They are just suggestion though, don't feel
obliged to pick one of them if there is nothing that fits your taste and
abilities. Feel free to propose something else that you are excited about and
that contributes to improve the Software Heritage documentation: we will be
happy to consider it!

=== Reorganize developer documentation as tutorials/how-to/discussions/references ===

Our [https://docs.softwareheritage.org/devel/ developer documentation] is not particularly structured, making it hard to use it properly.
We would like to reorganize it following the [https://www.youtube.com/watch?v=t4vKPhjcMZg tutorials/how-to/discussions/references taxonomy], or something equally sensible.

This project will require discussing the intended use cases of this page, conceptually structure the future version of it, implement it, and (re)write suitable language to introduce the various parts.

=== Write a high-level technical overview of the project, data model, and data flow ===

We have described the data model and archival data flow of Software Heritage in [https://upsilon.cc/~zack/research/publications/ipres-2018-doi.pdf various] [https://upsilon.cc/~zack/research/publications/cacm-2018-software-heritage.pdf scientific] [https://upsilon.cc/~zack/research/publications/msr-2019-swh.pdf papers], but haven't really worked on a general, high-level, technical presentation of it that targets developers.
The current [https://docs.softwareheritage.org/devel/swh-model/data-model.html#data-model data model documentation] has been copy-pasted from scientific papers.
We want to revisit it, to make sure the language description is suitable for a more general technical public, possibly complementing it with more abstract data-structure descriptions and cross-references to the code implementing the model and the workflow in the actual implementation.

=== Design and document consistent writing conventions for Python docstrings ===

The docstrings of the [https://docs.softwareheritage.org/devel/#components various software components] in the Software Heritage stack are not very consistent.
They have been written by a number of different developers, with varying degrees of English proficiency.

We would like to document, as a set of writing guidelines, how to write them to the benefits of current and future developers.
The guidelines should include what to write in docstrings and whatnot, writing styles and guidance, examples of good and bad content for them, etc.

To the extent it is possible, this project will also explore automating checks for guidelines conformance, in the form of basic structural (e.g., are all function parameters documented? is the function name correct? etc.) and writing checks (e.g., spellchecking, grammar checking, etc.).

=== Revamp new (code) contributor tutorial ===

The current [https://docs.softwareheritage.org/devel/#getting-started getting started documentation] include two overlapping documents: one about how to ''run'' a local instance of the full Software Heritage stack, another about how to create a local setup for current and future project ''developers''.
We would like to review those documents for consistency and, more generally, revamp documentation that will help wannabe code contributors to quickly get started and submit patches.

As part of this project we might also want to revamp glossary, data model and other cross-cutting documents that help newcomers understand where-is-what in the code base and how to practically prepare and send us patches.

=== Restructure wiki landing page as main community entry point ===

Our [[Main_Page|public wiki landing page]] is, well, a mess.
It has grown inorganically by accumulating links to pages used more as working drafts than reasonably structured documents.
We would like to restructure it to cater for various use cases (users, developers, students, curators, scientists, etc.) in the attempt of making it the primary entry point into Software Heritage for all community members.

This project will require discussing the various uses cases, design a suitable page layout, implement it in the wiki, write the introductory parts for each path into the wiki.
A related task, if time permits, will be to reorganize the wiki [[Special:Categories|category taxonomy]] and document when/how to use the available categories in the future.

Working on this project will require familiarity with [https://www.mediawiki.org/wiki/MediaWiki Mediawiki] and its markup language.

== Contact ==

GSoD applicants are encouraged to get in touch with the Software Heritage community using the standard development communication channels, i.e.:

* the #swh-devel IRC channel on [https://freenode.net Freenode]
* the [https://sympa.inria.fr/sympa/info/swh-devel swh-devel mailing list]

See our [https://www.softwareheritage.org/community/developers/ development information page] for more details.

== Timeline ==

See the official [https://developers.google.com/season-of-docs/docs/timeline Google Season of Docs timeline].

Debian packaging

2019-02-08T15:28:08Z

NicolasDandrimont:

== Package repository ==

A package repository is available on https://debian.softwareheritage.org/.

Unstable / Testing :
deb [trusted=yes] https://debian.softwareheritage.org/ unstable main

Stable / Stretch :
deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main

This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).

=== Uploading packages ===

Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.

For packages to be accepted, they need to be :
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt>
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders

== Git repositories for Debian packages ==

Our git repository structure for Debian packages is compatible with <tt>git-buildpackage</tt>.

We have two different ways of handling repositories for Debian packages:
* Packages of python modules where *we* are upstream
* Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)

For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:
* Our own modules are merged with the upstream repository
* External dependencies ignore the upstream repository and only have packaging branches.

=== Branch and tags structure ===

Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:
* debian/upstream (history of unpacked upstream releases)
* debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, stretch-swh)
* pristine-tar (data to regenerate upstream tarballs from a git export)

The name of the debian/upstream branch doesn't matter ''as long as it's properly configured in the <tt>debian/gbp.conf</tt> file''. It's only really used by <tt>gbp import-orig</tt> when importing a new release.

The tags marking upstream releases imported from tarballs for Debian packaging purposes are named <tt>debian/upstream/''<upstream version number>''</tt>.

Our Jenkins jobs are triggered on incoming tags named <tt>debian/''<version>''</tt>. To generate the proper tags, use <tt>gbp buildpackage --git-tag-only</tt>.

The git-buildpackage configuration, <tt>debian/gbp.conf</tt>, should be the following:
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/''<current suite>''
pristine-tar=True

==== Automatic packaging for swh python modules ====

The <tt>swh.*</tt> python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs <tt>gbp import-orig</tt> with the tarball we release to PyPI, and the right options to merge the upstream history.

To merge changes from the upstream history, we add the following option to <tt>gbp.conf</tt>:
upstream-vcs-tag=v%(version)s

=== Bootstrapping a dependency packaging repository ===

Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:

Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/
wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz

Create a new git repository
mkdir pytest-postgresql
cd pytest-postgresql
git init

Import the original upstream version
git checkout -b debian/unstable-swh
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz
# What will be the source package name? [pytest-postgresql]
# What is the upstream version? [1.3.4]
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...
# gbp:info: Source package is pytest-postgresql
# gbp:info: Upstream version is 1.3.4
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz

Bootstrap the debian directory
mkdir debian
mkdir debian/source
echo '3.0 (quilt)' > debian/source/format
cat > debian/gbp.conf << EOF
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/unstable-swh
pristine-tar=True
EOF
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules
vim debian/control
# [...] adapt debian/control from another package
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1
vim debian/copyright
# [...] adapt debian/copyright from another package
git add debian
git commit -m "Initial packaging for pytest-postgresql"

You can then go on to try building the package. Once the package builds, if you want to check your package's conformance to Debian policy, you can run <tt>lintian</tt> on the changes:
lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes

Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository

We need to use a <tt>+swh1</tt> version suffix to avoid clashing with potential upstream Debian package versions.

==== Bootstrapping the backport branches ====

During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.

The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.

Checkout the new branch
git checkout debian/<version number>
git checkout -b debian/stretch-swh

Update the gbp config to match the branch
sed -i s/unstable-swh/stretch-swh/ debian/gbp.conf

Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)
dch -l ~bpo9 -D stretch-swh --force-distribution 'Rebuild for stretch-swh'

You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.

==== Setting up the repository on Phabricator ====

The repository on Phabricator needs the following settings:
* Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)
* Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)
* Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)
* Policy
** View: Public (no login required)
** Push: All users (actual restrictions are handled by Herald rules)
** Edit: Developers
* Activate the repository
* Look up the path to the repository on the storage tab

You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes.
ssh -t tate.internal.softwareheritage.org phabricator-setup-hook <repository-path> post-receive-debian-deps

==== Setting up the Jenkins jobs ====

The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/
They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs

Jobs for dependency packages are configured in <tt>jobs/dependency-packages.yaml</tt>. You can add a section as follows:

- project:
name: <Callsign>
display-name: <short-name>
pkg: <source-name>
jobs:
- 'dependency-jobs-{name}'

Use the regular review process to land your changes.
Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.

If your package needs extra repositories to build, you can add them as comma-separated values to the <tt>deb-extra-repositories</tt> setting, with the following notes:
* When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.
* When building packages for backports suites, the backports repository is automatically enabled.

=== Local package building ===

To locally test a package build, go on the appropriate debian packaging branch, and run
gbp buildpackage --git-builder=sbuild -As --no-clean-source

<tt>gbp buildpackage</tt> passes all options not starting with <tt>--git-</tt> to the builder. Some useful options are the following:

* <tt>--git-ignore-new</tt> builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.
* <tt>--no-clean-source</tt> doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies
* <tt>--extra-repository="'''repository specification'''"</tt> adds the given repository in the chroot before building.
* <tt>--extra-repository-key='''repository signing key'''</tt> adds the given key as a trusted gpg key for package sources
* <tt>--extra-package='''<.deb file or directory>'''</tt> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.
* <tt>--force-orig-source</tt> forces addition of the <tt>.orig.tar.gz</tt> file in the <tt>.changes</tt> file (useful when trying to upload a backport)

See <tt>gbp help buildpackage</tt> and <tt>man sbuild</tt> for a full description of all options

for example:
gbp buildpackage --git-builder=sbuild -As --force-orig-source --extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main'

(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping <tt>gbp buildpackage</tt> with the most common options)

=== Remote package building ===

Jenkins builds packages when the repository receives a tag.

Once the local build succeeds, tag the package with:
gbp buildpackage --git-tag-only --git-sign-tags

Alternatively, you can add the <tt>--git-tag</tt> option to your <tt>gbp buildpackage</tt> command so the tag happens automatically on a successful build.

Then, push your tag, and Jenkins jobs should get triggered
git push --tags

== Build Environment setup ==

Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.

=== sbuild setup ===

# Install the package
sudo apt-get install sbuild

# Add your user to the sbuild group, to allow him to use the sbuild commands
sudo sbuild-adduser $USER
# You have to logout and log back in

# Prepare chroots
sudo mkdir /srv/chroots
sudo mkdir /srv/chroots/var

# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/
sudo rm -r /var/lib/schroot
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot

sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/
sudo rm -r /var/lib/sbuild
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild
# end optionally

# Create unstable/sid chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/

# Create stretch chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates stretch /srv/chroots/stretch http://deb.debian.org/debian/

# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts
echo hosts >> /etc/schroot/sbuild/nssdatabases

=== schroot setup ===

Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.

You need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:

source-groups=root,sbuild
source-root-groups=root,sbuild
union-type=overlay

This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.

You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):

For unstable:
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild

For stretch:
aliases=stable-amd64-sbuild,stable-backports-amd64-sbuild,stretch-swh-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild

==== dependencies cache ====

Add the following line to schroot's fstab /etc/schroot/sbuild/fstab
to permit reuse of existing fetched dependencies:

/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0

You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in <tt>/etc/apt/apt.conf.d</tt> on each chroot

=== schroot update ===

You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):

sudo sbuild-update -udcar sid; sudo sbuild-update -udcar stretch; sudo sbuild-update -ud jessie

=== environment setup ===

The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt>

export DEBFULLNAME="Debra Hacker"
export DEBEMAIL=debra.hacker@example.com

Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourfullkeyid></tt> variable.
(Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).

=== overlay in tmpfs for faster builds ===

You can add this to your fstab to put the overlay hierarchy in RAM:

tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0 0 0

=== Base packages ===

In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.

'''If you do so, do not use these chroots to upload to Debian itself!'''

[[Category:Software development]]
[[Category:System administration]]

Debian packaging

2019-02-08T15:27:59Z

NicolasDandrimont: Overhaul packaging docs

Wiki page https://wiki.softwareheritage.org/index.php?title=Debian_packaging&action=edit

== Package repository ==

A package repository is available on https://debian.softwareheritage.org/.

Unstable / Testing :
deb [trusted=yes] https://debian.softwareheritage.org/ unstable main

Stable / Stretch :
deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main

This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).

=== Uploading packages ===

Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.

For packages to be accepted, they need to be :
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt>
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders

== Git repositories for Debian packages ==

Our git repository structure for Debian packages is compatible with <tt>git-buildpackage</tt>.

We have two different ways of handling repositories for Debian packages:
* Packages of python modules where *we* are upstream
* Packages of dependencies from another upstream (this also encompasses upstream Debian packages that we wish to backport for deployment)

For these classes of packages, we have two sets of (identical) Jenkins jobs to handle building and uploading these packages to our package repository. The structure of the packaging branches for both classes is pretty much the same, the repositories only differ on how we handle upstream commits:
* Our own modules are merged with the upstream repository
* External dependencies ignore the upstream repository and only have packaging branches.

=== Branch and tags structure ===

Our debian packaging Jenkins jobs expect the following branches, which are pretty close to what https://dep-team.pages.debian.net/deps/dep14/ mandates:
* debian/upstream (history of unpacked upstream releases)
* debian/<suite> (history of the packaging of the given suite, e.g. unstable-swh, stretch-swh)
* pristine-tar (data to regenerate upstream tarballs from a git export)

The name of the debian/upstream branch doesn't matter ''as long as it's properly configured in the <tt>debian/gbp.conf</tt> file''. It's only really used by <tt>gbp import-orig</tt> when importing a new release.

The tags marking upstream releases imported from tarballs for Debian packaging purposes are named <tt>debian/upstream/''<upstream version number>''</tt>.

Our Jenkins jobs are triggered on incoming tags named <tt>debian/''<version>''</tt>. To generate the proper tags, use <tt>gbp buildpackage --git-tag-only</tt>.

The git-buildpackage configuration, <tt>debian/gbp.conf</tt>, should be the following:
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/''<current suite>''
pristine-tar=True

==== Automatic packaging for swh python modules ====

The <tt>swh.*</tt> python modules have an extra jenkins job that updates the packaging automatically when we do an upstream release. This job only runs <tt>gbp import-orig</tt> with the tarball we release to PyPI, and the right options to merge the upstream history.

To merge changes from the upstream history, we add the following option to <tt>gbp.conf</tt>:
upstream-vcs-tag=v%(version)s

=== Bootstrapping a dependency packaging repository ===

Bootstrapping the packaging repository for a dependency is analoguous to regular Debian practices:

Download the upstream tarball. For PyPI, use the redirector at http://pypi.debian.net/<pkgname>/
wget http://pypi.debian.net/pytest-postgresql/pytest-postgresql-1.3.4.tar.gz

Create a new git repository
mkdir pytest-postgresql
cd pytest-postgresql
git init

Import the original upstream version
git checkout -b debian/unstable-swh
gbp import-orig --pristine-tar --upstream-branch=debian/upstream --upstream-tag=debian/upstream/%(version)s --debian-branch=debian/unstable-swh ../pytest-postgresql-1.3.4.tar.gz
# What will be the source package name? [pytest-postgresql]
# What is the upstream version? [1.3.4]
# gbp:info: Importing '../pytest-postgresql-1.3.4.tar.gz' to branch 'debian/upstream'...
# gbp:info: Source package is pytest-postgresql
# gbp:info: Upstream version is 1.3.4
# gbp:info: Successfully imported version 1.3.4 of ../pytest-postgresql-1.3.4.tar.gz

Bootstrap the debian directory
mkdir debian
mkdir debian/source
echo '3.0 (quilt)' > debian/source/format
cat > debian/gbp.conf << EOF
[DEFAULT]
upstream-branch=debian/upstream
upstream-tag=debian/upstream/%(version)s
debian-branch=debian/unstable-swh
pristine-tar=True
EOF
cp /usr/share/doc/debhelper/examples/rules.tiny debian/rules
vim debian/control
# [...] adapt debian/control from another package
dch --create --package pytest-postgresql --newversion 1.3.4-1+swh1
vim debian/copyright
# [...] adapt debian/copyright from another package
git add debian
git commit -m "Initial packaging for pytest-postgresql"

You can then go on to try building the package. Once the package builds, if you want to check your package's conformance to Debian policy, you can run <tt>lintian</tt> on the changes:
lintian -EI ../pytest-postgresql_1.3.4-1+swh1_amd64.changes

Note that you have to ignore warnings about unknown distributions, as we're building specifically for our repository

We need to use a <tt>+swh1</tt> version suffix to avoid clashing with potential upstream Debian package versions.

==== Bootstrapping the backport branches ====

During most of the operation, backports should happen automatically as we have a Jenkins job that generates backports on successful builds. However, when creating a packaging repository, we need to bootstrap the branches once, before Jenkins is able to do the work automatically.

The backport branches should (ideally) be bootstrapped from a debian tag that has successfully built on Jenkins.

Checkout the new branch
git checkout debian/<version number>
git checkout -b debian/stretch-swh

Update the gbp config to match the branch
sed -i s/unstable-swh/stretch-swh/ debian/gbp.conf

Generate the initial backports entry. Use the current Debian version number (9 for stretch, 10 for buster, ...)
dch -l ~bpo9 -D stretch-swh --force-distribution 'Rebuild for stretch-swh'

You should then be able to try a local package build, and if that succeeds, to push the tag for Jenkins to autobuild.

==== Setting up the repository on Phabricator ====

The repository on Phabricator needs the following settings:
* Callsign: non-empty (prefix should be P according to https://wiki.softwareheritage.org/wiki/Phabricator_callsign_naming_convention)
* Short name: non-empty (used to make pretty git clone URLs; ideally matching the source package name)
* Repository tags: "Has debian packaging branches" (allows Jenkins to push on the debian/* branches)
* Policy
** View: Public (no login required)
** Push: All users (actual restrictions are handled by Herald rules)
** Edit: Developers
* Activate the repository
* Look up the path to the repository on the storage tab

You need to setup the post-receive hook for Jenkins to be able to trigger on tag pushes.
ssh -t tate.internal.softwareheritage.org phabricator-setup-hook <repository-path> post-receive-debian-deps

==== Setting up the Jenkins jobs ====

The Jenkins jobs are accessible through the ui: https://jenkins.softwareheritage.org/view/Debian%20dependency%20packages/
They are declared in the repository: https://forge.softwareheritage.org/source/swh-jenkins-jobs

Jobs for dependency packages are configured in <tt>jobs/dependency-packages.yaml</tt>. You can add a section as follows:

- project:
name: <Callsign>
display-name: <short-name>
pkg: <source-name>
jobs:
- 'dependency-jobs-{name}'

Use the regular review process to land your changes.
Once your changes are pushed, a dedicated Jenkins job will generate the jobs from the configuration.

If your package needs extra repositories to build, you can add them as comma-separated values to the <tt>deb-extra-repositories</tt> setting, with the following notes:
* When building packages for the "*-swh" suites, the Software Heritage Debian repository is automatically enabled.
* When building packages for backports suites, the backports repository is automatically enabled.

=== Local package building ===

To locally test a package build, go on the appropriate debian packaging branch, and run
gbp buildpackage --git-builder=sbuild -As --no-clean-source

<tt>gbp buildpackage</tt> passes all options not starting with <tt>--git-</tt> to the builder. Some useful options are the following:

* <tt>--git-ignore-new</tt> builds from the working tree, with all the uncommitted changes. Useful for quick iteration when something *just* *doesn't* *work*.
* <tt>--no-clean-source</tt> doesn't run debian/rules clean outside of the chroot, so you don't have to clutter your dev machine with all build dependencies
* <tt>--extra-repository="'''repository specification'''"</tt> adds the given repository in the chroot before building.
* <tt>--extra-repository-key='''repository signing key'''</tt> adds the given key as a trusted gpg key for package sources
* <tt>--extra-package='''<.deb file or directory>'''</tt> makes the given package (or all .deb packages in the given directory) available for dependency resolution. Useful when testing builds with a dependency chain.
* <tt>--force-orig-source</tt> forces addition of the <tt>.orig.tar.gz</tt> file in the <tt>.changes</tt> file (useful when trying to upload a backport)

See <tt>gbp help buildpackage</tt> and <tt>man sbuild</tt> for a full description of all options

for example:
gbp buildpackage --git-builder=sbuild -As --force-orig-source --extra-repository='deb [trusted=yes] https://debian.softwareheritage.org/ stretch-swh main'

(TODO: rewrite bin/make-package as bin/swh-gbp-buildpackage wrapping <tt>gbp buildpackage</tt> with the most common options)

=== Remote package building ===

Jenkins builds packages when the repository receives a tag.

Once the local build succeeds, tag the package with:
gbp buildpackage --git-tag-only --git-sign-tags

Alternatively, you can add the <tt>--git-tag</tt> option to your <tt>gbp buildpackage</tt> command so the tag happens automatically on a successful build.

Then, push your tag, and Jenkins jobs should get triggered
git push --tags

== Build Environment setup ==

Our automated packaging setup uses sbuild, which is also used by the Debian build daemons themselves. This section shows how to set it up for local use.

=== sbuild setup ===

# Install the package
sudo apt-get install sbuild

# Add your user to the sbuild group, to allow him to use the sbuild commands
sudo sbuild-adduser $USER
# You have to logout and log back in

# Prepare chroots
sudo mkdir /srv/chroots
sudo mkdir /srv/chroots/var

# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/
sudo rm -r /var/lib/schroot
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot

sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/
sudo rm -r /var/lib/sbuild
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild
# end optionally

# Create unstable/sid chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates sid /srv/chroots/sid http://deb.debian.org/debian/

# Create stretch chroot
sudo sbuild-createchroot --include apt-transport-https,ca-certificates stretch /srv/chroots/stretch http://deb.debian.org/debian/

# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts
echo hosts >> /etc/schroot/sbuild/nssdatabases

=== schroot setup ===

Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.

You need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:

source-groups=root,sbuild
source-root-groups=root,sbuild
union-type=overlay

This allows the sbuild group to edit the contents of the source chroot (for instance to update it) and sets up the overlay.

You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):

For unstable:
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild

For stretch:
aliases=stable-amd64-sbuild,stable-backports-amd64-sbuild,stretch-swh-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild

==== dependencies cache ====

Add the following line to schroot's fstab /etc/schroot/sbuild/fstab
to permit reuse of existing fetched dependencies:

/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0

You can also run apt-cacher-ng, which will avoid locking issues when several chroots try to access the package cache at once. You then need to add the proxy configuration to apt by adding a file in <tt>/etc/apt/apt.conf.d</tt> on each chroot

=== schroot update ===

You should update your chroot environments once in a while (to avoid repeating over and over the same step during your package build):

sudo sbuild-update -udcar sid; sudo sbuild-update -udcar stretch; sudo sbuild-update -ud jessie

=== environment setup ===

The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt>

export DEBFULLNAME="Debra Hacker"
export DEBEMAIL=debra.hacker@example.com

Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourfullkeyid></tt> variable.
(Future version of gpg2, e.g. 2.2.5 can refuse to sign with the short key id).

=== overlay in tmpfs for faster builds ===

You can add this to your fstab to put the overlay hierarchy in RAM:

tmpfs /var/lib/schroot/union/overlay tmpfs uid=root,gid=root,mode=0750,nr_inodes=0 0 0

=== Base packages ===

In order not to reinstall the same packages every time, it is also reasonable to install debhelper, python3 and python3-all in the chroot.

'''If you do so, do not use these chroots to upload to Debian itself!'''

[[Category:Software development]]
[[Category:System administration]]

Code review in Phabricator

2019-02-08T12:14:10Z

NicolasDandrimont: Add documentation to set up git push

We use the [[Differential]] application of [[Phabricator]] to perform [[code review|code reviews]] in the context of [[Software Heritage]].

* we use Git and history.immutable=true (but beware as that is partly a Phabricator misnomer, read on)
* when code reviews are required, developers will be allowed to push directly to master once an accepted Differential diff exists

== Configuration ==

=== Arcanist configuration ===

When using git, [[Arcanist]] by default mess with the local history, rewriting commits at the time of first submission. 
To avoid that we use so called [https://secure.phabricator.com/book/phabricator/article/arcanist_new_project/#history-mutability-git history immutability].

To that end, you shall configure your <tt>arc</tt> accordingly:
<pre>
arc set-config history.immutable true
</pre>

Note that this does ''not'' mean that you are forbidden to rewrite your local branches (e.g., with <tt>git rebase</tt>).
Quite the contrary: you are encouraged to locally rewrite branches before pushing to ensure that commits are logically separated and your commit history easy to bisect.
The above setting just means that ''arc'' will not rewrite commit history under your nose.

=== Enabling <tt>git push</tt> to our forge ===

The way we've configured our review setup for continuous integration needs you to configure git to allow pushes to our forge. There's two ways you can do this : setting a ssh key to push over ssh, or setting a specific password for git pushes over https.

==== SSH key for pushes ====

In your forge User settings page (On the top right, click on your avatar, then click ''Settings''), you have access to a ''Authentication'' > ''SSH Public Keys'' section (Direct link: <tt>hxxps://forge.softwareheritage.org/settings/user/'''<your username>'''/page/ssh/</tt>). You then have the option to upload a SSH public key, which will authenticate your pushes.

You then need to configure ssh/git to use that key pair, for instance by editing the <tt>~/.ssh/config</tt> file.

Finally, you should configure git to push over ssh when pushing to https://forge.softwareheritage.org, by running the following command:
git config --global url.git@forge.softwareheritage.org:.pushInsteadOf https://forge.softwareheritage.org

This lets git know that it should use <tt>git@forge.softwareheritage.org:</tt> as a base url when pushing repositories cloned from forge.softwareheritage.org over https.

==== VCS password for pushes ====

If you're not comfortable setting up SSH to upload your changes, you have the option of setting a VCS password. This password, ''separate from your account password'', allows Phabricator to authenticate your uploads over HTTPS.

In your forge User settings page (On the top right, click on your avatar, then click ''Settings''), you need to use the ''Authentication'' > ''VCS Password'' section to set your VCS password (Direct link: <tt>hxxps://forge.softwareheritage.org/settings/user/'''<your username>'''/page/vcspassword/</tt>).

== Workflow ==

'''TL;DR:'''
* work in a feature branch: <tt>git checkout -b my-feat</tt>
* initial review request: hack/commit/hack/commit ; <tt>arc diff origin/master</tt>
* react to change requests: hack/commit/hack/commit ; <tt>arc diff --update Dxx origin/master</tt>
* landing change: <tt>git checkout master ; git merge my-feat ; git push</tt>

=== Starting a new feature and submit it for review ===

Use a '''one branch per feature''' workflow, with well-separated ''logical commits'':
<pre>
git checkout -b my-shiny-feature
... hack hack hack ...
git commit -m 'architecture skeleton for my-shiny-feature'
... hack hack hack ...
git commit -m 'my-shiny-feature: implement module foo'
... etc ...
</pre>

To '''submit your code for review''' the first time:
<pre>
arc diff origin/master
</pre>
arc will prompt for a '''code review message'''. Provide the following information:
* first line: ''short description'' of the overall work (i.e., the feature you're working on). This will become the title of the review
* ''Summary'' field (optional): ''long description'' of the overall work; the field can continue in subsequent lines, up to the next field. This will become the "Summary" section of the review
* ''Test Plan'' field (optional): write here if something special is needed to test your change
* ''Reviewers'' field (optional): the (Phabricator) name(s) of desired reviewers. If you don't specify one (recommended) the default reviewers will be chosen
* ''Subscribers'' field (optional): the (Phabricator) name(s) of people that will be notified about changes to this review request. In most cases it should be left empty

For example:
<pre>
mercurial loader

Summary: first stab at a mercurial loader (T329)

The implementation follows the plan detailed in F2F discussion with @foo.

Performances seem decent enough for a first trial (XXX seconds for YYY repository
that contains ZZZ patches).

Test plan:

Reviewers:

Subscribers: foo
</pre>

After completing the message arc will submit the review request and tell you its number and URL:
<pre>
[...]
Created a new Differential revision:
Revision URI: https://forge.softwareheritage.org/Dxx
</pre>

=== Updating your branch to reflect requested changes ===

Your feature might get accepted as is, YAY!
Or, reviewers might request changes; no big deal!

Use the Differential web UI to follow-up to received comments, if needed.

To implement requested changes in the code, hack on your branch as usual by:

* adding new commits, and/or
* rewriting old commits with git rebase (to preserve a nice, easy to bisect history)

When you're ready to '''update your review request''':
<pre>
arc diff --update Dxx origin/master
</pre>

Arc will prompt you for a message: describe what you've changed w.r.t. the previous review request, free form.
Your message will become the changelog entry in Differential for this new version of the diff.

Differential only care about the code diff, and not about the commits or their order.
Therefore each "update" can be a completely different series of commits, possibly rewritten from the previous submission.

=== Landing your change onto master ===

Once your change has been approved in Differential, you will be able to land it onto the master branch.

Before doing so, you're encouraged to '''clean up your git commit history''', reordering/splitting/merging commits as needed to have separate logical commits and an easy to bisect history.
The correspondence between the accepted review and pushed code is checked by looking only at the code diff, so commit fiddling will not impact your ability to push to master.

Once you're happy you can '''push to origin/master''' directly, e.g.:
<pre>
git checkout master
git merge my-shiny-feature
git push
</pre>

Optionally you can then delete your local feature branch:
<pre>
git branch -d my-shiny-feature
</pre>

=== Reviewing locally / landing someone else's changes ===

You can do local reviews of code with arc patch:

<pre>
arc patch Dxyz
</pre>

This will create a branch '''arcpatch-Dxyz''' containing the changes on your local checkout.

You can then merge those changes upstream with

<pre>
git checkout master
git merge --ff arcpatch-Dxyz
git push origin master
</pre>

== See also ==

* [[Code review]] for guidelines on how code is reviewed when developing for Software Heritage

[[Category:Software development]]

Puppet setup

2018-06-15T13:57:54Z

NicolasDandrimont: Expand puppet documentation

== Multiple repository setup ==

Our puppet environment is split into multiple repos (one repo per module), plus one "root" repository for multi-repository management.

First, clone the base repository, containing the configuration file for myrepos and a README file.

<pre>
$ git clone ssh://git@forge.softwareheritage.org/diffusion/SENV/puppet-environment.git
</pre>

Then, use that configuration to clone all the repositories :

<pre>
$ cd puppet-environment
$ readlink -f .mrconfig >> ~/.mrtrust
$ mr up
</pre>

(the mr command is in the myrepos Debian package).

All the swh-specific repositories are in ''swh-''-prefixed repositories. The other repositories come from other sources and have an ''upstream'' remote allowing updates (the ''origin'' remote is always on the swh git server).

Our puppet workflow is documented in [https://forge.softwareheritage.org/diffusion/SENV/browse/master/README.md the README.md file in the puppet-environment repository].

== Local puppet manifest diffing with octocatalog-diff ==

''puppet-environment'' contains the whole scaffolding to be able to use [https://github.com/github/octocatalog-diff <code>octocatalog-diff</code>] on our manifests. This allows for quick(er) local iterations while developing complex puppet manifests.

=== Dependencies ===
You need the following packages installed on your machine:
r10k octocatalog-diff puppet

=== Running ===
The <code>bin/octocatalog-diff</code> script allows diffing the manifests between two environments (that is, between two branches of the ''swh-site'' repository. By default it diffs between <code>production</code> and <code>staging</code>.

Default usage:
bin/octocatalog-diff pergamon

=== Limitations ===
Our setup for octocatalog-diff doesn't support exported resources, so you won't see your fancy icinga checks there.

== Integration of third party puppet modules ==

We mirror external repositories to our own forge, to avoid having external dependencies in our deployment.

In the ''swh-site'' <code>Puppetfile</code>, we pin the installation of those modules to the highest version (that works with our current puppet/facter version), by using the '':ref'' specifier.

=== Adding a new external puppet module ===

In the ''puppet-environment'' repository, the <code>bin/import-puppet-module</code> takes care of the following tasks:

* Getting metadata from the [https://forge.puppetlabs.com/ Puppet forge] for the module (description, upstream git URL)
* Cloning the repository
* Creating a mirror repository on the Software Heritage forge, with the proper permissions and metadata (notably the ''Sync to GitHub'' flag)
* Pushing the clone to the forge
* Updating the .mrconfig and .gitignore files to know the

To be able to use the script, you need :
* Be a member of the [https://forge.softwareheritage.org/project/members/7/ System Administrators] Phabricator group
* Have the [[Arcanist]] API key setup
* A pair of python dependencies : <code>python3-phabricator</code> and <code>python3-requests</code> (pull them from testing if needed).

Example usage to pull the [https://forge.puppetlabs.com/elastic/elasticsearch elastic/elasticsearch] module
<pre>
bin/import-module elastic-elasticsearch
git diff # review changes
git add .mrconfig .gitignore
git commit -m "Add the elastic/elasticsearch module"
git push
</pre>

Once the module is added, you need to register it in the ''swh-site'' <code>Puppetfile</code>.

You should also check in the module metadata whether any dependencies need importing as well, which you should do using the same procedure.

=== Updating external puppet modules ===

There's two sides of this coin:

==== Updating our git clone of external puppet modules ====

The ''puppet-environment'' <code>.mrconfig</code> file has a <code>pullup</code> command which does the right thing.

To update all clones:
<pre>
mr -j4 pullup
</pre>

==== Upgrading external puppet modules ====

Upgrading external puppet modules happens manually.

In the ''puppet-environment'' repository, the <code>bin/check-module-updates</code> script compares the Puppetfile and the local clones and lists the available updates. (depends on <code>ruby r10k</code>).

On a staging branch of the ''swh-site'' repository, update the <code>:ref</code> value for the module in the <code>Puppetfile</code> to the latest tag. You can then run <code>octocatalog-diff</code> on a few relevant servers and look for changes.

== Deploy work-flow ==

=== Semi-automated ===

# you@localhost$ # ''hack on puppet Git repo''
# you@localhost$ <tt>rake validate</tt>
# you@localhost$ <tt>git commit</tt>
# you@localhost$ <tt>git push</tt>
# you@localhost$ <tt>cd [https://forge.softwareheritage.org/diffusion/SENV/ puppet-environment]</tt>
# you@localhost$ <tt>bin/deploy-on machine1 machine2...</tt>

Remember to pass <tt>--apt</tt> to <tt>bin/deploy-on</tt> if freshly uploaded Software Heritage packages are to be deployed.
Also, <tt>bin/deploy-on --help</tt> is your friend.

=== Manual ===

# you@localhost$ # ''hack on puppet Git repo''
# you@localhost$ <tt>rake validate</tt>
# you@localhost$ <tt>git commit</tt>
# you@localhost$ <tt>git push</tt>
# you@pergamon$ <tt>sudo swh-puppet-master-deploy</tt>
# you@machine$ <tt>sudo apt-get update</tt> # ''if a new or updated version of a [[Debian packaging|Debian package]] needs deploying''
# you@machine$ <tt>sudo swh-puppet-test</tt> # ''to test/review changes''
# you@machine$ <tt>sudo swh-puppet-apply</tt> # ''to apply

[[Category:System administration]]

Arcanist setup

2018-06-15T12:43:40Z

NicolasDandrimont: /* Configuration */

[https://secure.phabricator.com/book/phabricator/article/arcanist/ Arcanist] is a command line interface to [[Phabricator]].
This page details how to configure it for use with the [[Software Heritage]] forge.

== Installation ==

<pre>
# apt-get install arcanist
</pre>

== Configuration ==

<pre>
$ arc set-config default https://forge.softwareheritage.org/
$ arc install-certificate
</pre>

arc will prompt you to login into Phabricator via web (which will ask your personal Phabricator credentials).
You will then have to copy past the API token from the web page to arc, and hit Enter to complete the certificate installation.

'''All done!''' Now have a look at ''arc help'' and start hacking.

=== Configuration file ===

Arcanist configuration is stored in ''~/.arcrc''.

== Links ==

* [https://www.mediawiki.org/wiki/Phabricator/Arcanist#Setup Wikimedia guide to Arcanist setup]

[[Category:Software development]]

Contributor License Agreement

2018-06-06T11:26:58Z

NicolasDandrimont: Add <hr> to separate references from rest of article.

= Software Heritage Contributor License Agreement =
version 1.0, 2018-06-01

'''Origin and beneficiary of the present contributor license agreement:'''

'''INSTITUT NATIONAL DE RECHERCHE EN INFORMATIQUE ET EN AUTOMATIQUE ''[National Institute for Research in Computing and Automation]'',''' Public Institution of a scientific and technological nature under decree 85-831 of August 2, 1982, Headquartered at Domaine de Voluceau – Rocquencourt – BP 105 – 78153 Le Chesnay cedex represented by its Chairman and CEO, Mr. François Sillion, founder of the Software Heritage project.

== ARTICLE 1 — DEFINITIONS ==

* '''“Software”:''' means any software component collaboratively developed on the Software Heritage forge ([https://forge.softwareheritage.org/ ''https://forge.softwareheritage.org/'']).
* '''“Contribution:”''' means any original contribution protected by copyright, in particular modifications of existing functionalities or the development of new ones, of which You are the author, and that You intentionally submit for integration into the Software. A Contribution includes its source code, its object code, as well as any specifications and documentation related thereto.

* '''“You”:''' means you, the Contributor, as a physical and individual person, owning copyrights on the Contribution and free from any obligation (in particular regarding to your employer) which may prevent you from developing and submitting the Contribution for integration into the Software.

== ARTICLE 2 — COPYRIGHTS LICENSED ==

# You grant to Inria a worldwide, non-exclusive, transferable, royalty-free, irrevocable license, including the right to sublicense, on the Contribution, for the duration of the corresponding copyright. In particular, you grant for any mean and purpose, the right to use, to reproduce, to modify, to display, to perform with any media or technical mean, and to distribute through any form of communication the Contribution.
# As a condition to the grant of rights mentioned in article 2.1 above, Inria shall license the Contribution ''only'' under the terms of a license complying with both the free software criteria defined by the Free Software Foundation (FSF)<ref>see [https://www.gnu.org/licenses/license-list.en.html ''https://www.gnu.org/licenses/license-list.en.html''] for a list of free software licenses</ref> and the Open Source Definition by the Open Source Initiative (OSI).<ref>see [https://opensource.org/licenses ''https://opensource.org/licenses''] for a list of open source licenses</ref>
# You will promptly notify Inria via email at ''[mailto:legal@softwareheritage.org legal@softwareheritage.org]'' if You become aware of any facts or circumstances that would make these commitments inaccurate in any way.

== ARTICLE 3 — GUARANTEES ==

# You guarantee that You own copyrights over the Contribution and that You hold, to the best of your knowledge, all of the rights necessary to grant a license on said Contribution to Inria and that You do not infringe third party’s rights.
# You acknowledge that You hold no patent which could be enforced against any use by Inria of the copyright over the Contribution. In the opposite case, You promise not to enforce the rights granted by these patents against Inria, or any of its licensees, sub-licensees or assignees, for using, exploiting, modifying, distributing your Contribution accordingly with article 2 here above.
# You declare that You are licensing your Contribution “as is”, without guarantee as to its commercial value, and without guaranteeing that the Contribution is free of errors.

== ARTICLE 4 – MISCELLANEOUS ==

# This contract is subject to French law.
# Any dispute concerning the interpretation, validity or execution of this contract will be submitted, failing an out-of-court resolution, to the competent French court.
# The present agreement may be assigned by Inria to any non-profit entity which would become entitled to carry out Software Heritage’s mission.
# If one or several stipulations of this contract are held to be invalid or declared such in application of a law or regulation, or by reason of a final ruling by a competent court, the others will retain all of their force and scope.
# You shall be asked to confirm the present agreement periodically, every three (3) years after signing it. Should our contribution policy change before, you will be notified and asked to sign the new contributor license agreement as a condition to submit further Contributions.

----

[[Category:Legal]]
[[Category:Software development]]

Statistics

2017-10-19T15:19:53Z

NicolasDandrimont: Add object counts

Collection of descriptive statistics about the [[Software Heritage archive]].

== Across objects ==

* Object counts: [https://archive.softwareheritage.org/api/1/stat/counters/ live from the API], [https://stats.export.softwareheritage.org/history_counters.json historical data]

== Contents (i.e., blobs) ==

* [[Statistics/Content_size|content size]]

Puppet setup

2017-10-03T14:16:02Z

NicolasDandrimont: /* Multiple repository setup */ link to readme.md

Talks

2017-09-29T12:16:39Z

NicolasDandrimont: Add Kernel Recipes

In this page we keep track of the past and upcoming talks about [[Software Heritage]], in various venues.

Please keep the table ''sorted by reverse date'' (most recent talk first).

== 2017 ==

{| class="wikitable"
!Date
!Venue
!Title
!Speaker
!Slides
!Video
|-
|27 September 2017
|[https://kernel-recipes.org/en/2017/ Kernel Recipes 2017]
|Software Heritage: Our software commons, forever - Challenges in storing the biggest VCS DAG in history
|[[User:NicolasDandrimont|Nicolas Dandrimont]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-09-27-kernel-recipes.pdf PDF]
|
|-
|10 August 2017
|[https://debconf17.debconf.org/ DebConf 2017]
|Software Heritage: Our software commons, forever.
|[[User:NicolasDandrimont|Nicolas Dandrimont]], [[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-08-10-debconf.pdf PDF]
|[https://annex.softwareheritage.org/public/talks/2017/2017-08-10-debconf.vp9.webm VP9]
|-
|5 July 2017
|[http://iticse.acm.org/ ITiCSE 2017]
|Software Heritage: scholarly and educational synergies with preserving our software commons (keynote)
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|
|
|-
|6 Jun 2017
|[https://reseau-loops.github.io/ Café LoOPS]
|[https://reseau-loops.github.io/2017/06/01/cafe-loops Software Heritage: Construire la bibliothèque d'Alexandrie du logiciel]
|[[User:NicolasDandrimont|Nicolas Dandrimont]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-06-06-loops.pdf PDF]
|
|-
|25 May 2017
|[http://www.dauin.polito.it/ DAUIN, Politecnico di Torino]
|Preserving Source Code: Challenges and Opportunities for the Reproductibility of Science
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-05-25-torino-polito.pdf PDF]
|
|-
|24 May 2017
|[https://nexa.polito.it/lunch-seminars Nexa Lunch Seminar]
|[https://nexa.polito.it/lunch-51 Software Heritage: Archiving the Software Commons for Fun and Social Benefit]
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-05-24-torino-nexa.pdf PDF]
|
|-
|10 Mar 2017
|[http://www.incontrodevops.it/events/idi2017/ Incontro DevOps 2017]
|[http://www.incontrodevops.it/sessions/keynote-idi2017/ Software Heritage: DevOps Challenges to Preserve our Software Commons] (keynote)
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-03-10-devops-italia.pdf PDF]
|
|-
|08 Feb 2017
|[https://project.inria.fr/epfl-Inria/workshops/workshop-2017/ INRIA-EPFL workshop]
|[https://project.inria.fr/epfl-Inria/files/2017/01/RobertoDiCosmo-abstract-workshop2017.pdf What would you do with billions of source code files? Challenges and opportunities in software archival]
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-02-07-epfl-billion-files.pdf PDF]
|
|-
|04 Feb 2017
|[https://fosdem.org/2017/ FOSDEM'17]
|[https://fosdem.org/2017/schedule/event/software_heritage/ Software Heritage: Preserving the Free Software Commons] (keynote)
|[[User:RobertoDiCosmo|Roberto Di Cosmo]], [[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-02-04-fosdem-software-heritage-foss-commons.pdf PDF]
|[https://video.fosdem.org/2017/Janson/software_heritage.vp8.webm VP8]
|-
|10 Jan 2017
|[http://www.congresodelfuturo.cl/ Congreso del Futuro]
|Software [is our] Heritage: Collecting, preserving and sharing the software source code of Mankind
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-01-10-congresofuturo.pdf PDF]
|
|}

== 2016 ==

{| class="wikitable"
!Date
!Venue
!Title
!Speaker
!Slides
!Video
|-
|26 Nov 2016
|[http://milan2016.codemotionworld.com/ Codemotion Milan 2016]
|[http://milan2016.codemotionworld.com/talk-detail/?detail=4654 Software Heritage: let's build together the universal archive of our software commons]
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-11-26-codemotion-milan-zack-software-commons-archive.pdf PDF]
|[https://www.youtube.com/watch?v=gi_HydCumRE YouTube]
|-
|9 Nov 2016
|[http://www.dpconline.org/ Digital Preservation Coalition]
|Building the Universal Software Archive
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-11-09-webex-rdicosmo-digital-preservation-coalition.pdf PDF]
|
|-
|8 Nov 2016
|[https://github.com/alegrand/RR_webinars Series of Webinars on Reproducible Research]
|[https://github.com/alegrand/RR_webinars/blob/master/5_archiving_software_and_data/index.org Preserving Software and Data: Ensuring Availability and Traceability]
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-11-08-grenoble-rdicosmo-reproductibility-preserving-software.pdf PDF]
|[https://mi2s.imag.fr/preserving-software-ensuring-availability-and-traceability-0 Flowplayer]
|-
|18 Oct 2016
|[http://conferences.oreilly.com/oscon/open-source-eu OSCON Europe 2016]
|[http://conferences.oreilly.com/oscon/open-source-eu/public/schedule/detail/55989 Why and how Software Heritage is building the universal software archive] (keynote)
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-10-18-oscon-london-rdicosmo-keynote-software-heritage-building-the-universal-software-archive.pdf PDF]
|[https://www.oreilly.com/ideas/why-and-how-software-heritage-is-building-the-universal-software-archive Excerpt]
|-
|14 Oct 2016
|[http://www.upmc.fr/fr/formations/diplomes/sciences_et_technologies2/masters2/master_informatique_m1/master_science_et_technologie_du_logiciel_m2.html UPMC Master 2 STL]
|[https://www-apr.lip6.fr/~chaillou/Public/enseignement/2016-2017/conf-STL/ Software Heritage: Preserving the Free Software Commons]
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-10-14-upmc-software-heritage-preserving-the-free-software-commons.pdf PDF]
|
|-
|21 Sep 2016
|[https://www.ow2con.org/bin/view/2016/ OW2con'16]
|[https://ow2con16.sched.org/event/80KP Beyond OW2: Software Heritage, Building the Universal Software Archive]
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-09-21-ow2con-zack-software-heritage-building-the-universal-software-archive.pdf PDF]
|
|-
|16 Sep 2016
|[https://www.irif.univ-paris-diderot.fr/ IRIF] [https://www.irif.univ-paris-diderot.fr/seminaires/irif/index seminar series]
|[https://www.irif.univ-paris-diderot.fr/seminaires/irif/index Preserving Software: challenges and opportunities for the reproductibility of Science ]
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-09-16-irif-science-crisis-software-preservation.pdf PDF]
|
|-
|07 Sep 2016
|[https://www.meetup.com/git-Paris/events/233368705/?eventId=233368705 Meetup git Paris #3]
|Software Heritage: une archive mondiale du logiciel libre, inspirée de Git
|[[User:NicolasDandrimont|Nicolas Dandrimont]], [[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-09-07-paris-olasd+zack-git-meetup.pdf PDF]
|
|-
|04 Sep 2016
|[https://fsfe.org/community/events/2016/summit/frontpage.en.html FSFE Summit 2016]
|[https://conf.qtcon.org/en/qtcon/public/events/466 Software Heritage - the Universal Archive of Free Software]
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-09-04-qtcon-fsfe-rdicosmo-software-heritage-building-the-universal-software-archive.pdf PDF]
|
|-
|05 Aug 2016
|[https://www.emfcamp.org Electromagnetic Field 2016]
|Software Heritage (lightning talk)
|[[User:NicolasDandrimont|Nicolas Dandrimont]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-08-07-emfcamp-olasd-lightning-talk/2016-08-05_emf.html HTML]
|
|-
|04 Jul 2016
|[https://portail.umons.ac.be/FR/universite/admin/aff_etudiant/CultureEtSport/UMONS-culture/Documents_agendas_culture/2015/Research%20Seminar%20on%20Open%20Source%20Software_4-07-2016.pdf Open Source Seminar at UMONS]
|Software Heritage: Building the Universal Software Archive
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-07-04-umons-rdicosmo-software-heritage-building-the-universal-software-archive.pdf PDF]
|
|-
|04 Jul 2016
|[https://debconf16.debconf.org DebConf16]
|[https://debconf16.debconf.org/talks/42/ Software Heritage: Building the Universal Software Archive]
|[[User:NicolasDandrimont|Nicolas Dandrimont]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-07-04-debconf16-olasd-software-heritage-building-the-universal-software-archive.pdf PDF]
|[http://gensho.acc.umu.se/pub/debian-meetings/2016/debconf16/Software_Heritage_Building_the_Universal_Software_Archive.webm WebM]
|-
|21 Jun 2016
|[http://journees-scientifiques2016.inria.fr/francais-programme/ Journées Scientifiques Inria, Rennes]
|What would you do with ''billions'' of source code files? Challenges and opportunities in software archival
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-06-21-inriarennes-rdicosmo-what-would-you-do-with-billions-of-source-code-files.pdf PDF]
|
|}

== 2015 ==

{| class="wikitable"
!Date
!Venue
!Title
!Speaker
!Slides
!Video
|-
|16 Dec 2015
|[http://codesource.hypotheses.org/ Seminar "Codes Sources", UPMC]
|Large-scale source code archival, publishing, and indexing with Debsources [and Software Heritage]
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2015/2015-12-16-semcodesources-zack-debsources-and-software-heritage.pdf PDF]
|
|-
|04 Dec 2015
|[http://cristal.univ-lille.fr/evolille2015/ EvoLille 2015]
|Ten years analysing large code bases: a perspective
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2015/2015-12-04-evolille2015-rdicosmo-ten-years-analysing-large-code-bases-a-perspective.pdf PDF]
|
|-
|21 May 2015
|[http://www.scilabtec.com/ International Scilab Users Conference]
|[https://www.scilab.org/fr/community/scilabtec/2015/Keynote-Preserving-Software-challenges-and-opportunities-for-reproducibility-of-Science-and-Technology Preserving Software: Challenges and Opportunities for Reproducibility of Science and Technology]
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2015/2015-05-21-scilabtec-rdicosmo-preserving-software-challenges-and-opportunities-for-reproducibility-of-science-and-technology.pdf PDF]
|[https://vimeo.com/132074333 Vimeo]
|}

== 2014 ==

{| class="wikitable"
!Date
!Venue
!Title
!Speaker
!Slides
!Video
|-
|09 Dec 2014
|Reproductibility Working Group, Inria
|Preserving Software: Challenges and opportunities for reproductibility
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2014/2014-12-04-inriareprowg-rdicosmo-preserving-software-challenges-and-opportunities-for-reproducibility.pdf PDF]
|
|-
|}

[[Category:Communication]]
[[Category:Talks]]

Code review in Phabricator

2017-09-25T12:55:41Z

NicolasDandrimont: add local review/merge instructions

We use the [[Differential]] application of [[Phabricator]] to do code reviews in the context of [[Software Heritage]].

* we use Git and history.immutable=true (but beware as that is partly a Phabricator misnomer, read on)
* when code reviews are required, developers will be allowed to push directly to master once an accepted Differential diff exists

= Configuration =

When using git, [[Arcanist]] by default mess with the local history, rewriting commits at the time of first submission. 
To avoid that we use so called [https://secure.phabricator.com/book/phabricator/article/arcanist_new_project/#history-mutability-git history immutability].

To that end, you shall configure your <tt>arc</tt> accordingly:
<pre>
arc set-config history.immutable true
</pre>

Note that this does ''not'' mean that you are forbidden to rewrite your local branches (e.g., with <tt>git rebase</tt>).
Quite the contrary: you are encouraged to locally rewrite branches before pushing to ensure that commits are logically separated and your commit history easy to bisect.
The above setting just means that ''arc'' will not rewrite commit history under your nose.

= Workflow =

'''TL;DR:'''
* work in a feature branch: <tt>git checkout -b my-feat</tt>
* initial review request: hack/commit/hack/commit ; <tt>arc diff origin/master</tt>
* react to change requests: hack/commit/hack/commit ; <tt>arc diff --update Dxx origin/master</tt>
* landing change: <tt>git checkout master ; git merge my-feat ; git push</tt>

== Starting a new feature and submit it for review ==

Use a '''one branch per feature''' workflow, with well-separated ''logical commits'':
<pre>
git checkout -b my-shiny-feature
... hack hack hack ...
git commit -m 'architecture skeleton for my-shiny-feature'
... hack hack hack ...
git commit -m 'my-shiny-feature: implement module foo'
... etc ...
</pre>

To '''submit your code for review''' the first time:
<pre>
arc diff origin/master
</pre>
arc will prompt for a '''code review message'''. Provide the following information:
* first line: ''short description'' of the overall work (i.e., the feature you're working on). This will become the title of the review
* ''Summary'' field (optional): ''long description'' of the overall work; the field can continue in subsequent lines, up to the next field. This will become the "Summary" section of the review
* ''Test Plan'' field (optional): write here if something special is needed to test your change
* ''Reviewers'' field (optional): the (Phabricator) name(s) of desired reviewers. If you don't specify one (recommended) the default reviewers will be chosen
* ''Subscribers'' field (optional): the (Phabricator) name(s) of people that will be notified about changes to this review request. In most cases it should be left empty

For example:
<pre>
mercurial loader

Summary: first stab at a mercurial loader (T329)

The implementation follows the plan detailed in F2F discussion with @foo.

Performances seem decent enough for a first trial (XXX seconds for YYY repository
that contains ZZZ patches).

Test plan:

Reviewers:

Subscribers: foo
</pre>

After completing the message arc will submit the review request and tell you its number and URL:
<pre>
[...]
Created a new Differential revision:
Revision URI: https://forge.softwareheritage.org/Dxx
</pre>

== Updating your branch to reflect requested changes ==

Your feature might get accepted as is, YAY!
Or, reviewers might request changes; no big deal!

Use the Differential web UI to follow-up to received comments, if needed.

To implement requested changes in the code, hack on your branch as usual by:

* adding new commits, and/or
* rewriting old commits with git rebase (to preserve a nice, easy to bisect history)

When you're ready to '''update your review request''':
<pre>
arc diff --update Dxx origin/master
</pre>

Arc will prompt you for a message: describe what you've changed w.r.t. the previous review request, free form.
Your message will become the changelog entry in Differential for this new version of the diff.

Differential only care about the code diff, and not about the commits or their order.
Therefore each "update" can be a completely different series of commits, possibly rewritten from the previous submission.

== Landing your change onto master ==

Once your change has been approved in Differential, you will be able to land it onto the master branch.

Before doing so, you're encouraged to '''clean up your git commit history''', reordering/splitting/merging commits as needed to have separate logical commits and an easy to bisect history.
The correspondence between the accepted review and pushed code is checked by looking only at the code diff, so commit fiddling will not impact your ability to push to master.

Once you're happy you can '''push to origin/master''' directly, e.g.:
<pre>
git checkout master
git merge my-shiny-feature
git push
</pre>

Optionally you can then delete your local feature branch:
<pre>
git branch -d my-shiny-feature
</pre>

== Reviewing locally / landing someone else's changes ==

You can do local reviews of code with arc patch:

<pre>
arc patch Dxyz
</pre>

This will create a branch '''arcpatch-Dxyz''' containing the changes on your local checkout.

You can then merge those changes upstream with

<pre>
git checkout master
git merge --ff arcpatch-Dxyz
git push origin master
</pre>

[[Category:Software development]]

Talks

2017-09-07T18:25:55Z

NicolasDandrimont: /* 2017 */ uniformatting

In this page we keep track of the past and upcoming talks about [[Software Heritage]], in various venues.

Please keep the table ''sorted by reverse date'' (most recent talk first).

== 2017 ==

{| class="wikitable"
!Date
!Venue
!Title
!Speaker
!Slides
!Video
|-
|10 August 2017
|[https://debconf17.debconf.org/ DebConf 2017]
|Software Heritage: Our software commons, forever.
|[[User:NicolasDandrimont|Nicolas Dandrimont]], [[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-08-10-debconf.pdf PDF]
|[https://annex.softwareheritage.org/public/talks/2017/2017-08-10-debconf.vp9.webm VP9]
|-
|5 July 2017
|[http://iticse.acm.org/ ITiCSE 2017]
|Software Heritage: scholarly and educational synergies with preserving our software commons (keynote)
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|
|
|-
|6 Jun 2017
|[https://reseau-loops.github.io/ Café LoOPS]
|[https://reseau-loops.github.io/2017/06/01/cafe-loops Software Heritage: Construire la bibliothèque d'Alexandrie du logiciel]
|[[User:NicolasDandrimont|Nicolas Dandrimont]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-06-06-loops.pdf PDF]
|
|-
|25 May 2017
|[http://www.dauin.polito.it/ DAUIN, Politecnico di Torino]
|Preserving Source Code: Challenges and Opportunities for the Reproductibility of Science
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-05-25-torino-polito.pdf PDF]
|
|-
|24 May 2017
|[https://nexa.polito.it/lunch-seminars Nexa Lunch Seminar]
|[https://nexa.polito.it/lunch-51 Software Heritage: Archiving the Software Commons for Fun and Social Benefit]
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-05-24-torino-nexa.pdf PDF]
|
|-
|10 Mar 2017
|[http://www.incontrodevops.it/events/idi2017/ Incontro DevOps 2017]
|[http://www.incontrodevops.it/sessions/keynote-idi2017/ Software Heritage: DevOps Challenges to Preserve our Software Commons] (keynote)
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-03-10-devops-italia.pdf PDF]
|
|-
|08 Feb 2017
|[https://project.inria.fr/epfl-Inria/workshops/workshop-2017/ INRIA-EPFL workshop]
|[https://project.inria.fr/epfl-Inria/files/2017/01/RobertoDiCosmo-abstract-workshop2017.pdf What would you do with billions of source code files? Challenges and opportunities in software archival]
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-02-07-epfl-billion-files.pdf PDF]
|
|-
|04 Feb 2017
|[https://fosdem.org/2017/ FOSDEM'17]
|[https://fosdem.org/2017/schedule/event/software_heritage/ Software Heritage: Preserving the Free Software Commons] (keynote)
|[[User:RobertoDiCosmo|Roberto Di Cosmo]], [[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-02-04-fosdem-software-heritage-foss-commons.pdf PDF]
|[https://video.fosdem.org/2017/Janson/software_heritage.vp8.webm VP8]
|-
|10 Jan 2017
|[http://www.congresodelfuturo.cl/ Congreso del Futuro]
|Software [is our] Heritage: Collecting, preserving and sharing the software source code of Mankind
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-01-10-congresofuturo.pdf PDF]
|
|}

== 2016 ==

{| class="wikitable"
!Date
!Venue
!Title
!Speaker
!Slides
!Video
|-
|26 Nov 2016
|[http://milan2016.codemotionworld.com/ Codemotion Milan 2016]
|[http://milan2016.codemotionworld.com/talk-detail/?detail=4654 Software Heritage: let's build together the universal archive of our software commons]
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-11-26-codemotion-milan-zack-software-commons-archive.pdf PDF]
|[https://www.youtube.com/watch?v=gi_HydCumRE YouTube]
|-
|9 Nov 2016
|[http://www.dpconline.org/ Digital Preservation Coalition]
|Building the Universal Software Archive
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-11-09-webex-rdicosmo-digital-preservation-coalition.pdf PDF]
|
|-
|8 Nov 2016
|[https://github.com/alegrand/RR_webinars Series of Webinars on Reproducible Research]
|[https://github.com/alegrand/RR_webinars/blob/master/5_archiving_software_and_data/index.org Preserving Software and Data: Ensuring Availability and Traceability]
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-11-08-grenoble-rdicosmo-reproductibility-preserving-software.pdf PDF]
|[https://mi2s.imag.fr/preserving-software-ensuring-availability-and-traceability-0 Flowplayer]
|-
|18 Oct 2016
|[http://conferences.oreilly.com/oscon/open-source-eu OSCON Europe 2016]
|[http://conferences.oreilly.com/oscon/open-source-eu/public/schedule/detail/55989 Why and how Software Heritage is building the universal software archive] (keynote)
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-10-18-oscon-london-rdicosmo-keynote-software-heritage-building-the-universal-software-archive.pdf PDF]
|[https://www.oreilly.com/ideas/why-and-how-software-heritage-is-building-the-universal-software-archive Excerpt]
|-
|14 Oct 2016
|[http://www.upmc.fr/fr/formations/diplomes/sciences_et_technologies2/masters2/master_informatique_m1/master_science_et_technologie_du_logiciel_m2.html UPMC Master 2 STL]
|[https://www-apr.lip6.fr/~chaillou/Public/enseignement/2016-2017/conf-STL/ Software Heritage: Preserving the Free Software Commons]
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-10-14-upmc-software-heritage-preserving-the-free-software-commons.pdf PDF]
|
|-
|21 Sep 2016
|[https://www.ow2con.org/bin/view/2016/ OW2con'16]
|[https://ow2con16.sched.org/event/80KP Beyond OW2: Software Heritage, Building the Universal Software Archive]
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-09-21-ow2con-zack-software-heritage-building-the-universal-software-archive.pdf PDF]
|
|-
|16 Sep 2016
|[https://www.irif.univ-paris-diderot.fr/ IRIF] [https://www.irif.univ-paris-diderot.fr/seminaires/irif/index seminar series]
|[https://www.irif.univ-paris-diderot.fr/seminaires/irif/index Preserving Software: challenges and opportunities for the reproductibility of Science ]
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-09-16-irif-science-crisis-software-preservation.pdf PDF]
|
|-
|07 Sep 2016
|[https://www.meetup.com/git-Paris/events/233368705/?eventId=233368705 Meetup git Paris #3]
|Software Heritage: une archive mondiale du logiciel libre, inspirée de Git
|[[User:NicolasDandrimont|Nicolas Dandrimont]], [[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-09-07-paris-olasd+zack-git-meetup.pdf PDF]
|
|-
|04 Sep 2016
|[https://fsfe.org/community/events/2016/summit/frontpage.en.html FSFE Summit 2016]
|[https://conf.qtcon.org/en/qtcon/public/events/466 Software Heritage - the Universal Archive of Free Software]
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-09-04-qtcon-fsfe-rdicosmo-software-heritage-building-the-universal-software-archive.pdf PDF]
|
|-
|05 Aug 2016
|[https://www.emfcamp.org Electromagnetic Field 2016]
|Software Heritage (lightning talk)
|[[User:NicolasDandrimont|Nicolas Dandrimont]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-08-07-emfcamp-olasd-lightning-talk/2016-08-05_emf.html HTML]
|
|-
|04 Jul 2016
|[https://portail.umons.ac.be/FR/universite/admin/aff_etudiant/CultureEtSport/UMONS-culture/Documents_agendas_culture/2015/Research%20Seminar%20on%20Open%20Source%20Software_4-07-2016.pdf Open Source Seminar at UMONS]
|Software Heritage: Building the Universal Software Archive
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-07-04-umons-rdicosmo-software-heritage-building-the-universal-software-archive.pdf PDF]
|
|-
|04 Jul 2016
|[https://debconf16.debconf.org DebConf16]
|[https://debconf16.debconf.org/talks/42/ Software Heritage: Building the Universal Software Archive]
|[[User:NicolasDandrimont|Nicolas Dandrimont]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-07-04-debconf16-olasd-software-heritage-building-the-universal-software-archive.pdf PDF]
|[http://gensho.acc.umu.se/pub/debian-meetings/2016/debconf16/Software_Heritage_Building_the_Universal_Software_Archive.webm WebM]
|-
|21 Jun 2016
|[http://journees-scientifiques2016.inria.fr/francais-programme/ Journées Scientifiques Inria, Rennes]
|What would you do with ''billions'' of source code files? Challenges and opportunities in software archival
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-06-21-inriarennes-rdicosmo-what-would-you-do-with-billions-of-source-code-files.pdf PDF]
|
|}

== 2015 ==

{| class="wikitable"
!Date
!Venue
!Title
!Speaker
!Slides
!Video
|-
|16 Dec 2015
|[http://codesource.hypotheses.org/ Seminar "Codes Sources", UPMC]
|Large-scale source code archival, publishing, and indexing with Debsources [and Software Heritage]
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2015/2015-12-16-semcodesources-zack-debsources-and-software-heritage.pdf PDF]
|
|-
|04 Dec 2015
|[http://cristal.univ-lille.fr/evolille2015/ EvoLille 2015]
|Ten years analysing large code bases: a perspective
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2015/2015-12-04-evolille2015-rdicosmo-ten-years-analysing-large-code-bases-a-perspective.pdf PDF]
|
|-
|21 May 2015
|[http://www.scilabtec.com/ International Scilab Users Conference]
|[https://www.scilab.org/fr/community/scilabtec/2015/Keynote-Preserving-Software-challenges-and-opportunities-for-reproducibility-of-Science-and-Technology Preserving Software: Challenges and Opportunities for Reproducibility of Science and Technology]
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2015/2015-05-21-scilabtec-rdicosmo-preserving-software-challenges-and-opportunities-for-reproducibility-of-science-and-technology.pdf PDF]
|[https://vimeo.com/132074333 Vimeo]
|}

== 2014 ==

{| class="wikitable"
!Date
!Venue
!Title
!Speaker
!Slides
!Video
|-
|09 Dec 2014
|Reproductibility Working Group, Inria
|Preserving Software: Challenges and opportunities for reproductibility
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2014/2014-12-04-inriareprowg-rdicosmo-preserving-software-challenges-and-opportunities-for-reproducibility.pdf PDF]
|
|-
|}

[[Category:Communication]]
[[Category:Talks]]

Talks

2017-09-07T18:24:51Z

NicolasDandrimont: /* 2017 */ add DebConf

In this page we keep track of the past and upcoming talks about [[Software Heritage]], in various venues.

Please keep the table ''sorted by reverse date'' (most recent talk first).

== 2017 ==

{| class="wikitable"
!Date
!Venue
!Title
!Speaker
!Slides
!Video
|-
|10 August 2017
|[https://debconf17.debconf.org/ DebConf 2017]
|Software Heritage: Our software commons, forever.
|[[User:NicolasDandrimont|Nicolas Dandrimont]] and [[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-08-10-debconf.pdf PDF]
|[https://annex.softwareheritage.org/public/talks/2017/2017-08-10-debconf.vp9.webm VP9]
|-
|5 July 2017
|[http://iticse.acm.org/ ITiCSE 2017]
|Software Heritage: scholarly and educational synergies with preserving our software commons (keynote)
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|
|
|-
|6 Jun 2017
|[https://reseau-loops.github.io/ Café LoOPS]
|[https://reseau-loops.github.io/2017/06/01/cafe-loops Software Heritage: Construire la bibliothèque d'Alexandrie du logiciel]
|[[User:NicolasDandrimont|Nicolas Dandrimont]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-06-06-loops.pdf PDF]
|
|-
|25 May 2017
|[http://www.dauin.polito.it/ DAUIN, Politecnico di Torino]
|Preserving Source Code: Challenges and Opportunities for the Reproductibility of Science
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-05-25-torino-polito.pdf PDF]
|
|-
|24 May 2017
|[https://nexa.polito.it/lunch-seminars Nexa Lunch Seminar]
|[https://nexa.polito.it/lunch-51 Software Heritage: Archiving the Software Commons for Fun and Social Benefit]
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-05-24-torino-nexa.pdf PDF]
|
|-
|10 Mar 2017
|[http://www.incontrodevops.it/events/idi2017/ Incontro DevOps 2017]
|[http://www.incontrodevops.it/sessions/keynote-idi2017/ Software Heritage: DevOps Challenges to Preserve our Software Commons] (keynote)
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-03-10-devops-italia.pdf PDF]
|
|-
|08 Feb 2017
|[https://project.inria.fr/epfl-Inria/workshops/workshop-2017/ INRIA-EPFL workshop]
|[https://project.inria.fr/epfl-Inria/files/2017/01/RobertoDiCosmo-abstract-workshop2017.pdf What would you do with billions of source code files? Challenges and opportunities in software archival]
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-02-07-epfl-billion-files.pdf PDF]
|
|-
|04 Feb 2017
|[https://fosdem.org/2017/ FOSDEM'17]
|[https://fosdem.org/2017/schedule/event/software_heritage/ Software Heritage: Preserving the Free Software Commons] (keynote)
|[[User:RobertoDiCosmo|Roberto Di Cosmo]], [[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-02-04-fosdem-software-heritage-foss-commons.pdf PDF]
|[https://video.fosdem.org/2017/Janson/software_heritage.vp8.webm VP8]
|-
|10 Jan 2017
|[http://www.congresodelfuturo.cl/ Congreso del Futuro]
|Software [is our] Heritage: Collecting, preserving and sharing the software source code of Mankind
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-01-10-congresofuturo.pdf PDF]
|
|}

== 2016 ==

{| class="wikitable"
!Date
!Venue
!Title
!Speaker
!Slides
!Video
|-
|26 Nov 2016
|[http://milan2016.codemotionworld.com/ Codemotion Milan 2016]
|[http://milan2016.codemotionworld.com/talk-detail/?detail=4654 Software Heritage: let's build together the universal archive of our software commons]
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-11-26-codemotion-milan-zack-software-commons-archive.pdf PDF]
|[https://www.youtube.com/watch?v=gi_HydCumRE YouTube]
|-
|9 Nov 2016
|[http://www.dpconline.org/ Digital Preservation Coalition]
|Building the Universal Software Archive
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-11-09-webex-rdicosmo-digital-preservation-coalition.pdf PDF]
|
|-
|8 Nov 2016
|[https://github.com/alegrand/RR_webinars Series of Webinars on Reproducible Research]
|[https://github.com/alegrand/RR_webinars/blob/master/5_archiving_software_and_data/index.org Preserving Software and Data: Ensuring Availability and Traceability]
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-11-08-grenoble-rdicosmo-reproductibility-preserving-software.pdf PDF]
|[https://mi2s.imag.fr/preserving-software-ensuring-availability-and-traceability-0 Flowplayer]
|-
|18 Oct 2016
|[http://conferences.oreilly.com/oscon/open-source-eu OSCON Europe 2016]
|[http://conferences.oreilly.com/oscon/open-source-eu/public/schedule/detail/55989 Why and how Software Heritage is building the universal software archive] (keynote)
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-10-18-oscon-london-rdicosmo-keynote-software-heritage-building-the-universal-software-archive.pdf PDF]
|[https://www.oreilly.com/ideas/why-and-how-software-heritage-is-building-the-universal-software-archive Excerpt]
|-
|14 Oct 2016
|[http://www.upmc.fr/fr/formations/diplomes/sciences_et_technologies2/masters2/master_informatique_m1/master_science_et_technologie_du_logiciel_m2.html UPMC Master 2 STL]
|[https://www-apr.lip6.fr/~chaillou/Public/enseignement/2016-2017/conf-STL/ Software Heritage: Preserving the Free Software Commons]
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-10-14-upmc-software-heritage-preserving-the-free-software-commons.pdf PDF]
|
|-
|21 Sep 2016
|[https://www.ow2con.org/bin/view/2016/ OW2con'16]
|[https://ow2con16.sched.org/event/80KP Beyond OW2: Software Heritage, Building the Universal Software Archive]
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-09-21-ow2con-zack-software-heritage-building-the-universal-software-archive.pdf PDF]
|
|-
|16 Sep 2016
|[https://www.irif.univ-paris-diderot.fr/ IRIF] [https://www.irif.univ-paris-diderot.fr/seminaires/irif/index seminar series]
|[https://www.irif.univ-paris-diderot.fr/seminaires/irif/index Preserving Software: challenges and opportunities for the reproductibility of Science ]
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-09-16-irif-science-crisis-software-preservation.pdf PDF]
|
|-
|07 Sep 2016
|[https://www.meetup.com/git-Paris/events/233368705/?eventId=233368705 Meetup git Paris #3]
|Software Heritage: une archive mondiale du logiciel libre, inspirée de Git
|[[User:NicolasDandrimont|Nicolas Dandrimont]], [[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-09-07-paris-olasd+zack-git-meetup.pdf PDF]
|
|-
|04 Sep 2016
|[https://fsfe.org/community/events/2016/summit/frontpage.en.html FSFE Summit 2016]
|[https://conf.qtcon.org/en/qtcon/public/events/466 Software Heritage - the Universal Archive of Free Software]
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-09-04-qtcon-fsfe-rdicosmo-software-heritage-building-the-universal-software-archive.pdf PDF]
|
|-
|05 Aug 2016
|[https://www.emfcamp.org Electromagnetic Field 2016]
|Software Heritage (lightning talk)
|[[User:NicolasDandrimont|Nicolas Dandrimont]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-08-07-emfcamp-olasd-lightning-talk/2016-08-05_emf.html HTML]
|
|-
|04 Jul 2016
|[https://portail.umons.ac.be/FR/universite/admin/aff_etudiant/CultureEtSport/UMONS-culture/Documents_agendas_culture/2015/Research%20Seminar%20on%20Open%20Source%20Software_4-07-2016.pdf Open Source Seminar at UMONS]
|Software Heritage: Building the Universal Software Archive
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-07-04-umons-rdicosmo-software-heritage-building-the-universal-software-archive.pdf PDF]
|
|-
|04 Jul 2016
|[https://debconf16.debconf.org DebConf16]
|[https://debconf16.debconf.org/talks/42/ Software Heritage: Building the Universal Software Archive]
|[[User:NicolasDandrimont|Nicolas Dandrimont]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-07-04-debconf16-olasd-software-heritage-building-the-universal-software-archive.pdf PDF]
|[http://gensho.acc.umu.se/pub/debian-meetings/2016/debconf16/Software_Heritage_Building_the_Universal_Software_Archive.webm WebM]
|-
|21 Jun 2016
|[http://journees-scientifiques2016.inria.fr/francais-programme/ Journées Scientifiques Inria, Rennes]
|What would you do with ''billions'' of source code files? Challenges and opportunities in software archival
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-06-21-inriarennes-rdicosmo-what-would-you-do-with-billions-of-source-code-files.pdf PDF]
|
|}

== 2015 ==

{| class="wikitable"
!Date
!Venue
!Title
!Speaker
!Slides
!Video
|-
|16 Dec 2015
|[http://codesource.hypotheses.org/ Seminar "Codes Sources", UPMC]
|Large-scale source code archival, publishing, and indexing with Debsources [and Software Heritage]
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2015/2015-12-16-semcodesources-zack-debsources-and-software-heritage.pdf PDF]
|
|-
|04 Dec 2015
|[http://cristal.univ-lille.fr/evolille2015/ EvoLille 2015]
|Ten years analysing large code bases: a perspective
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2015/2015-12-04-evolille2015-rdicosmo-ten-years-analysing-large-code-bases-a-perspective.pdf PDF]
|
|-
|21 May 2015
|[http://www.scilabtec.com/ International Scilab Users Conference]
|[https://www.scilab.org/fr/community/scilabtec/2015/Keynote-Preserving-Software-challenges-and-opportunities-for-reproducibility-of-Science-and-Technology Preserving Software: Challenges and Opportunities for Reproducibility of Science and Technology]
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2015/2015-05-21-scilabtec-rdicosmo-preserving-software-challenges-and-opportunities-for-reproducibility-of-science-and-technology.pdf PDF]
|[https://vimeo.com/132074333 Vimeo]
|}

== 2014 ==

{| class="wikitable"
!Date
!Venue
!Title
!Speaker
!Slides
!Video
|-
|09 Dec 2014
|Reproductibility Working Group, Inria
|Preserving Software: Challenges and opportunities for reproductibility
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2014/2014-12-04-inriareprowg-rdicosmo-preserving-software-challenges-and-opportunities-for-reproducibility.pdf PDF]
|
|-
|}

[[Category:Communication]]
[[Category:Talks]]

Matrix

2017-09-05T13:45:07Z

NicolasDandrimont: add CertFP and clarify #swh-sysadm

== IRC channels ==

The following channels have been registered on the [https://freenode.net/ Freenode] network for [[Software Heritage]] usage.

* '''#softwareheritage''': general discussions about the project (currently unused)
* '''#swh''': ditto, in case we end up preferring the short version
* '''#swh-devel''': public development discussions
* '''#swh-team''': private discussions of the core team
* '''#swh-sysadm''': operations team discussions/bots

If you do IRC, consider joining the channels.

== IRC authentication ==

You should register their nick with NickServ using:

/nick <USERNAME>
/msg nickserv register <PASSWORD> <EMAIL>

You will then receive an e-mail containing a link to activate you account. After doing so, you need to configure your client to auto-authenticate. The recommanded way of doing that is using [https://freenode.net/kb/answer/sasl SASL authentication].

For Weechat:

/set irc.server.freenode.sasl_username <USERNAME>
/set irc.server.freenode.sasl_password <PASSWORD>

Freenode also supports authentication via [https://freenode.net/kb/answer/certfp TLS client certificates].

== IRC access list ==

To auto-voice people with a registered nick (only doable by people with +fA access modes will be able to do it), add them to the channel access list:

/msg chanserv access #swh-devel add zack +V

[[Category:Infrastructure]]

Debian packaging

2017-06-19T17:21:36Z

NicolasDandrimont: /* Package repository */ add stretch

== Creating a package for deployment ==

<tt>swh-environment</tt> contains a script, <tt>bin/make-package</tt>, which generates a Debian package from a given Git repository, and uploads it to our internal repository targeting suites unstable and jessie-backports.

This assumes that the '''current''' (''HEAD'') revision has been tagged with a version '''v<foo>''' (e.g. v0.0.4), and that the repository contains the debian metadata. (i.e. <tt>debian/{copyright,control,changelog.rules,source/format,compat}</tt>).

Example:
$ git tag -as -m 'version 0.0.42' v0.0.42
$ bin/make-package -b -u swh-core

== Package repository ==

A package repository is available on http://debian.internal.softwareheritage.org/.

Unstable / Testing :
deb [trusted=yes] http://debian.internal.softwareheritage.org/ unstable main

Stable / Stretch :
deb [trusted=yes] http://debian.internal.softwareheritage.org/ stretch-swh main

Oldstable / Jessie :
deb [trusted=yes] http://debian.internal.softwareheritage.org/ jessie main

This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).

=== Uploading packages ===

Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.

For packages to be accepted, they need to be :
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt>
# Targetted at one of the supported distributions (unstable, unstable-swh, stretch, stretch-backports, stretch-backports-swh), jessie, jessie-backports, jessie-backports-swh)
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders

== Build Environment setup ==

<tt>bin/make-package.sh</tt> uses sbuild for package construction. You therefore need to setup sbuild before being able to use it.

=== sbuild setup ===

# Install the package
sudo apt-get install sbuild

# Add your user to the sbuild group, to allow him to use the sbuild commands
sudo sbuild-adduser $USER
# You have to logout and log back in

# Prepare chroots
sudo mkdir /srv/chroots
sudo mkdir /srv/chroots/var

# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/
sudo rm -r /var/lib/schroot
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot

sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/
sudo rm -r /var/lib/sbuild
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild
# end optionally

# Create unstable/sid chroot
sudo sbuild-createchroot sid /srv/chroots/sid http://deb.debian.org/debian/

# Create stretch chroot
sudo sbuild-createchroot stretch /srv/chroots/stretch http://deb.debian.org/debian/

# Create jessie chroot
sudo sbuild-createchroot jessie /srv/chroots/jessie http://deb.debian.org/debian/

# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts
echo hosts >> /etc/schroot/sbuild/nssdatabases

=== schroot setup ===

Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.

In recent (>= 1.6) versions of schroot, you need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:

source-groups=root,sbuild
source-root-groups=root,sbuild
union-type=overlay

You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):

For unstable:
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild

For stretch:
aliases=stable-amd64-sbuild,stable-backports-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild

For jessie:
aliases=oldstable-amd64-sbuild,oldstable-backports-amd64-sbuild,jessie-backports-amd64-sbuild,jessie-backports-swh-amd64-sbuild

==== dependencies cache ====

Add the following line to schroot's fstab /etc/schroot/sbuild/fstab
to permit reuse of existing fetched dependencies:

/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0

=== environment setup ===

The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt>

export DEBFULLNAME="Debra Hacker"
export DEBEMAIL=debra.hacker@example.com

Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourkeyid></tt> variable.

[[Category:Software development]]
[[Category:System administration]]

Debian packaging

2017-06-19T17:20:09Z

NicolasDandrimont: Update packaging for stretch

== Creating a package for deployment ==

<tt>swh-environment</tt> contains a script, <tt>bin/make-package</tt>, which generates a Debian package from a given Git repository, and uploads it to our internal repository targeting suites unstable and jessie-backports.

This assumes that the '''current''' (''HEAD'') revision has been tagged with a version '''v<foo>''' (e.g. v0.0.4), and that the repository contains the debian metadata. (i.e. <tt>debian/{copyright,control,changelog.rules,source/format,compat}</tt>).

Example:
$ git tag -as -m 'version 0.0.42' v0.0.42
$ bin/make-package -b -u swh-core

== Package repository ==

A package repository is available on http://debian.internal.softwareheritage.org/.

Unstable / Testing :
deb [trusted=yes] http://debian.internal.softwareheritage.org/ unstable main

Stable / Jessie :
deb [trusted=yes] http://debian.internal.softwareheritage.org/ jessie main

This package repository is handled via reprepro on pergamon.internal.softwareheritage.org (base directory : /srv/softwareheritage/repository).

=== Uploading packages ===

Packages are added to the repository using <tt>reprepro -vb /srv/softwareheritage/repository processincoming incoming</tt>.

For packages to be accepted, they need to be :
# A changes file uploaded to <tt>/srv/softwareheritage/repository/incoming</tt>
# Targetted at one of the supported distributions (unstable, unstable-swh, jessie, jessie-backports, jessie-backports-swh)
# Signed by one of the keys listed in /srv/softwareheritage/repository/conf/uploaders

== Build Environment setup ==

<tt>bin/make-package.sh</tt> uses sbuild for package construction. You therefore need to setup sbuild before being able to use it.

=== sbuild setup ===

# Install the package
sudo apt-get install sbuild

# Add your user to the sbuild group, to allow him to use the sbuild commands
sudo sbuild-adduser $USER
# You have to logout and log back in

# Prepare chroots
sudo mkdir /srv/chroots
sudo mkdir /srv/chroots/var

# Optionally create a separate filesystem for /srv/chroots and move the sbuild/schroot data to that partition
sudo rsync -avz --delete /var/lib/schroot/ /srv/chroots/var/schroot/
sudo rm -r /var/lib/schroot
sudo ln -sf /srv/chroots/var/schroot /var/lib/schroot

sudo rsync -avz --delete /var/lib/sbuild/ /srv/chroots/var/sbuild/
sudo rm -r /var/lib/sbuild
sudo ln -sf /srv/chroots/var/sbuild /var/lib/sbuild
# end optionally

# Create unstable/sid chroot
sudo sbuild-createchroot sid /srv/chroots/sid http://deb.debian.org/debian/

# Create stretch chroot
sudo sbuild-createchroot stretch /srv/chroots/stretch http://deb.debian.org/debian/

# Create jessie chroot
sudo sbuild-createchroot jessie /srv/chroots/jessie http://deb.debian.org/debian/

# If you use /etc/hosts to resolve *.internal.softwareheritage.org hosts
echo hosts >> /etc/schroot/sbuild/nssdatabases

=== schroot setup ===

Now that the sbuild base setup is done. You now need to configure schroot to use an overlay filesystem, which will avoid copying the chroots at each build.

In recent (>= 1.6) versions of schroot, you need to update the configuration (in <tt>/etc/schroot/chroot.d/*-sbuild-*</tt>) with the following directives:

source-groups=root,sbuild
source-root-groups=root,sbuild
union-type=overlay

You should also use this opportunity to add "aliases" to your chroot, so that sbuild will directly support the distributions we're using (unstable-swh, jessie-backports-swh):

For unstable:
aliases=unstable-amd64-sbuild,UNRELEASED-amd64-sbuild,unstable-swh-amd64-sbuild

For stretch:
aliases=stable-amd64-sbuild,stable-backports-amd64-sbuild,stretch-backports-amd64-sbuild,stretch-backports-swh-amd64-sbuild

For jessie:
aliases=oldstable-amd64-sbuild,oldstable-backports-amd64-sbuild,jessie-backports-amd64-sbuild,jessie-backports-swh-amd64-sbuild

==== dependencies cache ====

Add the following line to schroot's fstab /etc/schroot/sbuild/fstab
to permit reuse of existing fetched dependencies:

/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0

=== environment setup ===

The Debian tools use a few variables to preset your name and email. Add this to your <tt>.<shell>rc</tt>

export DEBFULLNAME="Debra Hacker"
export DEBEMAIL=debra.hacker@example.com

Make sure this data matches an uid for your GPG key. Else, you can use the <tt>DEBSIGN_KEYID=<yourkeyid></tt> variable.

[[Category:Software development]]
[[Category:System administration]]

Talks

2017-06-06T13:28:55Z

NicolasDandrimont: /* 2017 */ Add LoOPS talk

In this page we keep track of the past and upcoming talks about [[Software Heritage]], in various venues.

Please keep the table ''sorted by reverse date'' (most recent talk first).

== 2017 ==

{| class="wikitable"
!Date
!Venue
!Title
!Speaker
!Slides
!Video
|-
|6 Jun 2017
|[https://reseau-loops.github.io/ Café LoOPS]
|[https://reseau-loops.github.io/2017/06/01/cafe-loops Software Heritage: Construire la bibliothèque d'Alexandrie du logiciel]
|[[User:NicolasDandrimont|Nicolas Dandrimont]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-06-06-loops.pdf PDF]
|
|-
|10 Mar 2017
|[http://www.incontrodevops.it/events/idi2017/ Incontro DevOps 2017]
|[http://www.incontrodevops.it/sessions/keynote-idi2017/ Software Heritage: DevOps Challenges to Preserve our Software Commons]
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-03-10-devops-italia.pdf PDF]
|
|-
|08 Feb 2017
|[https://project.inria.fr/epfl-Inria/workshops/workshop-2017/ INRIA-EPFL workshop]
|[https://project.inria.fr/epfl-Inria/files/2017/01/RobertoDiCosmo-abstract-workshop2017.pdf What would you do with billions of source code files? Challenges and opportunities in software archival]
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-02-07-epfl-billion-files.pdf PDF]
|
|-
|04 Feb 2017
|[https://fosdem.org/2017/ FOSDEM'17]
|[https://fosdem.org/2017/schedule/event/software_heritage/ Software Heritage: Preserving the Free Software Commons] (keynote)
|[[User:RobertoDiCosmo|Roberto Di Cosmo]], [[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-02-04-fosdem-software-heritage-foss-commons.pdf PDF]
|[https://video.fosdem.org/2017/Janson/software_heritage.vp8.webm VP8]
|-
|10 Jan 2017
|[http://www.congresodelfuturo.cl/ Congreso del Futuro]
|Software [is our] Heritage: Collecting, preserving and sharing the software source code of Mankind
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2017/2017-01-10-congresofuturo.pdf PDF]
|
|}

== 2016 ==

{| class="wikitable"
!Date
!Venue
!Title
!Speaker
!Slides
!Video
|-
|26 Nov 2016
|[http://milan2016.codemotionworld.com/ Codemotion Milan 2016]
|[http://milan2016.codemotionworld.com/talk-detail/?detail=4654 Software Heritage: let's build together the universal archive of our software commons]
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-11-26-codemotion-milan-zack-software-commons-archive.pdf PDF]
|[https://www.youtube.com/watch?v=gi_HydCumRE YouTube]
|-
|9 Nov 2016
|[http://www.dpconline.org/ Digital Preservation Coalition]
|Building the Universal Software Archive
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-11-09-webex-rdicosmo-digital-preservation-coalition.pdf PDF]
|
|-
|8 Nov 2016
|[https://github.com/alegrand/RR_webinars Series of Webinars on Reproducible Research]
|[https://github.com/alegrand/RR_webinars/blob/master/5_archiving_software_and_data/index.org Preserving Software and Data: Ensuring Availability and Traceability]
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-11-08-grenoble-rdicosmo-reproductibility-preserving-software.pdf PDF]
|[https://mi2s.imag.fr/preserving-software-ensuring-availability-and-traceability-0 Flowplayer]
|-
|18 Oct 2016
|[http://conferences.oreilly.com/oscon/open-source-eu OSCON Europe 2016]
|[http://conferences.oreilly.com/oscon/open-source-eu/public/schedule/detail/55989 Why and how Software Heritage is building the universal software archive] (keynote)
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-10-18-oscon-london-rdicosmo-keynote-software-heritage-building-the-universal-software-archive.pdf PDF]
|[https://www.oreilly.com/ideas/why-and-how-software-heritage-is-building-the-universal-software-archive Excerpt]
|-
|14 Oct 2016
|[http://www.upmc.fr/fr/formations/diplomes/sciences_et_technologies2/masters2/master_informatique_m1/master_science_et_technologie_du_logiciel_m2.html UPMC Master 2 STL]
|[https://www-apr.lip6.fr/~chaillou/Public/enseignement/2016-2017/conf-STL/ Software Heritage: Preserving the Free Software Commons]
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-10-14-upmc-software-heritage-preserving-the-free-software-commons.pdf PDF]
|
|-
|21 Sep 2016
|[https://www.ow2con.org/bin/view/2016/ OW2con'16]
|[https://ow2con16.sched.org/event/80KP Beyond OW2: Software Heritage, Building the Universal Software Archive]
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-09-21-ow2con-zack-software-heritage-building-the-universal-software-archive.pdf PDF]
|
|-
|16 Sep 2016
|[https://www.irif.univ-paris-diderot.fr/ IRIF] [https://www.irif.univ-paris-diderot.fr/seminaires/irif/index seminar series]
|[https://www.irif.univ-paris-diderot.fr/seminaires/irif/index Preserving Software: challenges and opportunities for the reproductibility of Science ]
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-09-16-irif-science-crisis-software-preservation.pdf PDF]
|
|-
|07 Sep 2016
|[https://www.meetup.com/git-Paris/events/233368705/?eventId=233368705 Meetup git Paris #3]
|Software Heritage: une archive mondiale du logiciel libre, inspirée de Git
|[[User:NicolasDandrimont|Nicolas Dandrimont]], [[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-09-07-paris-olasd+zack-git-meetup.pdf PDF]
|
|-
|04 Sep 2016
|[https://fsfe.org/community/events/2016/summit/frontpage.en.html FSFE Summit 2016]
|[https://conf.qtcon.org/en/qtcon/public/events/466 Software Heritage - the Universal Archive of Free Software]
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-09-04-qtcon-fsfe-rdicosmo-software-heritage-building-the-universal-software-archive.pdf PDF]
|
|-
|05 Aug 2016
|[https://www.emfcamp.org Electromagnetic Field 2016]
|Software Heritage (lightning talk)
|[[User:NicolasDandrimont|Nicolas Dandrimont]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-08-07-emfcamp-olasd-lightning-talk/2016-08-05_emf.html HTML]
|
|-
|04 Jul 2016
|[https://portail.umons.ac.be/FR/universite/admin/aff_etudiant/CultureEtSport/UMONS-culture/Documents_agendas_culture/2015/Research%20Seminar%20on%20Open%20Source%20Software_4-07-2016.pdf Open Source Seminar at UMONS]
|Software Heritage: Building the Universal Software Archive
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-07-04-umons-rdicosmo-software-heritage-building-the-universal-software-archive.pdf PDF]
|
|-
|04 Jul 2016
|[https://debconf16.debconf.org DebConf16]
|[https://debconf16.debconf.org/talks/42/ Software Heritage: Building the Universal Software Archive]
|[[User:NicolasDandrimont|Nicolas Dandrimont]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-07-04-debconf16-olasd-software-heritage-building-the-universal-software-archive.pdf PDF]
|[http://gensho.acc.umu.se/pub/debian-meetings/2016/debconf16/Software_Heritage_Building_the_Universal_Software_Archive.webm WebM]
|-
|21 Jun 2016
|[http://journees-scientifiques2016.inria.fr/francais-programme/ Journées Scientifiques Inria, Rennes]
|What would you do with ''billions'' of source code files? Challenges and opportunities in software archival
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2016/2016-06-21-inriarennes-rdicosmo-what-would-you-do-with-billions-of-source-code-files.pdf PDF]
|
|}

== 2015 ==

{| class="wikitable"
!Date
!Venue
!Title
!Speaker
!Slides
!Video
|-
|16 Dec 2015
|[http://codesource.hypotheses.org/ Seminar "Codes Sources", UPMC]
|Large-scale source code archival, publishing, and indexing with Debsources [and Software Heritage]
|[[User:StefanoZacchiroli|Stefano Zacchiroli]]
|[https://annex.softwareheritage.org/public/talks/2015/2015-12-16-semcodesources-zack-debsources-and-software-heritage.pdf PDF]
|
|-
|04 Dec 2015
|[http://cristal.univ-lille.fr/evolille2015/ EvoLille 2015]
|Ten years analysing large code bases: a perspective
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2015/2015-12-04-evolille2015-rdicosmo-ten-years-analysing-large-code-bases-a-perspective.pdf PDF]
|
|-
|21 May 2015
|[http://www.scilabtec.com/ International Scilab Users Conference]
|[https://www.scilab.org/fr/community/scilabtec/2015/Keynote-Preserving-Software-challenges-and-opportunities-for-reproducibility-of-Science-and-Technology Preserving Software: Challenges and Opportunities for Reproducibility of Science and Technology]
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2015/2015-05-21-scilabtec-rdicosmo-preserving-software-challenges-and-opportunities-for-reproducibility-of-science-and-technology.pdf PDF]
|[https://vimeo.com/132074333 Vimeo]
|}

== 2014 ==

{| class="wikitable"
!Date
!Venue
!Title
!Speaker
!Slides
!Video
|-
|09 Dec 2014
|Reproductibility Working Group, Inria
|Preserving Software: Challenges and opportunities for reproductibility
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[https://annex.softwareheritage.org/public/talks/2014/2014-12-04-inriareprowg-rdicosmo-preserving-software-challenges-and-opportunities-for-reproducibility.pdf PDF]
|
|-
|}

[[Category:Communication]]
[[Category:Talks]]

VPN

2016-10-10T14:03:58Z

NicolasDandrimont: Add revocation instructions

New machine setup

2016-09-23T14:11:12Z

NicolasDandrimont: add new VM setup

= Setting up a new Software Heritage desktop machine =

== Debian install ==

* Stable
* root w/temporary password; no regular user (after setting up root password, cancel twice and jump forward to clock settings)
* full disk with LVM; reduce home LV to leave half of the disk free
* Standard system utilities, ssh server, no desktop environment (puppet will install that)

== Base system setup (from console) ==

* Login as root
* Enable password root access in ssh (/etc/ssh/sshd_config, PermitRootLogin yes)
* Write down IP configuration and add the machine to the Gandi DNS
* Test SSH login as root from your workstation
* Stay at your desk :)

== Full system setup (from your desk) ==

* SSH login as root
* Edit sources.list to add testing
* apt-get update, dist-upgrade, autoremove --purge
** While you wait, create [[Vpn]] certificates for the new machine
** add the machine to the puppet configuration, in the swh_desktop role
* apt-get install puppet openvpn
* configure openvpn per [[Vpn]]
** add pergamon IP address to /etc/resolv.conf
** add louvre.softwareheritage.org to /etc/hosts
* configure puppet
** systemctl disable puppet
** server=pergamon.internal.softwareheritage.org in /etc/puppet/puppet.conf
** puppet agent --enable
** puppet agent -t
** run puppet on pergamon to update munin server config
* set proper root password, add it to password store
* reboot

= Setting up a new Virtual Machine (manual process) =

Naming scheme: machine_name.<zone>.<hoster>.internal.softwareheritage.org.

* Provision the virtual machine from a Debian image
** Set the admin user to something temporary with an ssh key
** Avoid public IPs if you don't need them
** Add the machine to the internal dns (swh-site + puppet run on pergamon)
* Connect to the machine with the temp admin user
* apt-get update, dist-upgrade, autoremove --purge
* set a root password (xckdpass, add to password store)
* allow root ssh password login
* connect as root
* remove temporary user
** deluser foo
** rm -rf /home/foo
* set the hostname
** /etc/hostname: machine.zone.hoster (e.g. worker01.euwest.azure)
** /etc/hosts: add {{<ip> machine.zone.hoster.internal.softwareheritage.org machine.zone.hoster}}} line
* reboot to get new hostname
* install and setup puppet
** apt-get install puppet
** systemctl disable puppet
** server=pergamon.internal.softwareheritage.org in /etc/puppet/puppet.conf
** puppet agent --enable
** puppet agent -t
** run puppet on pergamon to update munin server config
* reboot to check new services
* update clustershell configuration on louvre

[[Category:Infrastructure]]
[[Category:System administration]]

Talks

2016-09-08T10:25:50Z

NicolasDandrimont: Add EMF slides

Licensing

2016-07-26T12:42:57Z

NicolasDandrimont: /* Licenses of external dependencies */ add kafka-related stuff

Software developed by [[Software Heritage]] is released under the following licenses:

;default license
: [http://www.gnu.org/licenses/gpl-3.0.en.html GNU General Public License, version 3] or above ('''GPL3+''') is our default license
;web applications
: [http://www.gnu.org/licenses/agpl-3.0.en.html GNU Affero General Public License, version 3] or above ('''AGPL3+''') for Web applications and other software whose main purpose is offer a service via remote network interaction
;puppet
: [http://www.apache.org/licenses/LICENSE-2.0 Apache License, version 2] ('''Apache2''') for Puppet recipes

See the license of specific software modules for details.

Below we list our external software dependencies, their licenses, and give pointers to how and why they are compatible with the above license choices.

An external dependency whose license is incompatible with the above license choices '''cannot be used''' in [[Software Heritage]], so please carefully review the relevant licenses before adopting a new external dependency.

== Licenses of external dependencies ==

Our external software dependencies are:

* boto: [https://raw.githubusercontent.com/boto/boto/develop/LICENSE MIT]
* celery: [https://sources.debian.net/src/celery/latest/debian/copyright/ BSD2/PSF]
* confluent-kafka-python: [https://github.com/confluentinc/confluent-kafka-python/blob/master/LICENSE Apache2]
* crcmod: [http://crcmod.sourceforge.net/intro.html#license MIT]
* cryptojs: [https://code.google.com/archive/p/crypto-js/wikis/License.wiki BSD3]
* dulwich: [https://sources.debian.net/src/dulwich/latest/debian/copyright/ GPL2+] in progress of being relicensed to [https://github.com/jelmer/dulwich/issues/153 GPL2+/Apache2]
* flask: [https://sources.debian.net/src/flask/latest/debian/copyright/ BSD3]
* flask-api: [http://www.flaskapi.org/about/license BSD2]
* flask-testing: [https://github.com/jarus/flask-testing/blob/master/LICENSE BSD3]
* flot : [https://raw.githubusercontent.com/flot/flot/master/LICENSE.txt MIT]
* kafka: [https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=blob;f=LICENSE;hb=HEAD Apache2]
* kafka-manager [https://github.com/yahoo/kafka-manager/blob/master/LICENCE Apache2]
* libgit2: [https://github.com/libgit2/libgit2/blob/master/COPYING GPL2 with linking exception]
* msgpack: [https://sources.debian.net/src/msgpack-python/latest/debian/copyright/ Apache2]
* psycopg2: [https://sources.debian.net/src/psycopg2/latest/debian/copyright/ LGPL3]
* pygit2: [https://sources.debian.net/src/python-pygit2/latest/debian/copyright/ GPL2 with linking exception]
* python (runtime): [https://sources.debian.net/src/python3.4/latest/debian/copyright/ PSF]
* python-arrow: [https://sources.debian.net/src/python-arrow/latest/debian/copyright/ Apache2 (doc themes GPL3+)]
* python-click [https://sources.debian.net/src/python-click/latest/debian/copyright/ BSD3]
* python-dateutil: [https://sources.debian.net/src/python-dateutil/latest/LICENSE/ BSD3]
* python-requests: [http://docs.python-requests.org/en/latest/user/intro/#apache2 Apache2]
* python-retrying: [https://sources.debian.net/src/python-retrying/latest/debian/copyright/ Apache2]
* qless-py: [https://github.com/seomoz/qless-py/blob/master/LICENSE MIT]
* redis-py: [https://github.com/andymccurdy/redis-py/blob/master/LICENSE MIT]
* subvertpy: [https://raw.githubusercontent.com/jelmer/subvertpy/python3-branch/COPYING LGPL2.1+]
* vcversioner: [https://sources.debian.net/src/vcversioner/latest/debian/copyright/ ISC]

When adopting a new external dependency, please add it to the above list, in alphabetical order and with a pointer to evidence of its license.

== License compatibility ==

(non-GPL variants only)

* Apache2: [http://www.gnu.org/licenses/license-list.en.html#apache2 GPL3-compatible]
* BSD2: [http://www.gnu.org/licenses/license-list.en.html#FreeBSD GPL-compatible]
* BSD3: [http://www.gnu.org/licenses/license-list.en.html#ModifiedBSD GPL-compatible]
* ISC: [http://www.gnu.org/licenses/license-list.en.html#ISC GPL-compatible]
* PSF: [https://www.gnu.org/licenses/license-list.html#Python GPL-compatible]

[[Category:Software development]]
[[Category:Legal]]

Talks

2016-07-05T21:40:09Z

NicolasDandrimont: Add video of DebConf talk

In this page we keep track of the past and upcoming talks about [[Software Heritage]], in various venues.

{| class="wikitable"
!Date
!Venue
!Title
!Speaker
!Slides URL
!Video URL
|-
|4 July 2016
|[https://debconf16.debconf.org DebConf16]
|[https://debconf16.debconf.org/talks/42/ Software Heritage: Building the Universal Software Archive]
|[[User:NicolasDandrimont|Nicolas Dandrimont]]
|[https://annex.debconf.org/debconf-share/debconf16/slides/42-software-heritage-building-the-universal-software-archive.pdf Slides PDF]
|[http://gensho.acc.umu.se/pub/debian-meetings/2016/debconf16/Software_Heritage_Building_the_Universal_Software_Archive.webm WebM video]
|-
|4 July 2016
|[https://portail.umons.ac.be/FR/universite/admin/aff_etudiant/CultureEtSport/UMONS-culture/Documents_agendas_culture/2015/Research%20Seminar%20on%20Open%20Source%20Software_4-07-2016.pdf Open Source Seminar at UMONS]
|Software Heritage: Building the Universal Software Archive
|[[User:RobertoDiCosmo|Roberto Di Cosmo]]
|[http://www.dicosmo.org/share/mons-07-2016.pdf Slides PDF]
|''n/a''
|}

[[Category:Communication]]
[[Category:Talks]]

Talks

2016-07-05T08:41:54Z

NicolasDandrimont: Add slides url for DebConf talk