Software Heritage Wiki - User contributions [en]

Matrix

2025-09-07T08:43:17Z

Vlorentz: simplify phrasing

== Matrix rooms ==

The following rooms have been registered on the [https://matrix.org/ Matrix] network for [[Software Heritage]] usage.

* [https://matrix.to/#/#swh-devel:matrix.org '''#swh-devel:matrix.org''']: public development discussions
* [https://matrix.to/#/#swh-sysadm:matrix.org '''#swh-sysadm:matrix.org''']: operations team discussions/bots
* [https://matrix.to/#/#swh-offtopic:matrix.org '''#swh-offtopic:matrix.org''']: Off-topic discussions
* [https://matrix.to/#/#swh-team:matrix.org '''#swh-team:matrix.org''']: private discussions of the core team
* [https://matrix.to/#/#swh:matrix.org '''#swh:matrix.org''']: general discussions around Software Heritage

They are all part of the [https://matrix.to/#/#swh-space:matrix.org '''#swh-space:matrix.org'''] Matrix space.

You will be asked to create a Matrix account if you don't have one yet.

Software Heritage closed their former [https://libera.chat libera.chat] IRC channels on 2023-12-08

Matrix

2025-09-07T07:26:38Z

Vlorentz: Use complete room alias instead of just the local part

== Matrix rooms ==

The following rooms have been registered on the [https://matrix.org/ Matrix] network for [[Software Heritage]] usage.

* [https://matrix.to/#/#swh-devel:matrix.org '''#swh-devel:matrix.org''']: public development discussions
* [https://matrix.to/#/#swh-sysadm:matrix.org '''#swh-sysadm:matrix.org''']: operations team discussions/bots
* [https://matrix.to/#/#swh-offtopic:matrix.org '''#swh-offtopic:matrix.org''']: Off-topic discussions
* [https://matrix.to/#/#swh-team:matrix.org '''#swh-team:matrix.org''']: private discussions of the core team
* [https://matrix.to/#/#swh:matrix.org '''#swh:matrix.org''']: general discussions around Software Heritage

They are all part of the [https://matrix.to/#/#swh-space:matrix.org '''#swh-space:matrix.org'''] Matrix space.

You will be asked to create a Matrix account if you don't have one yet.

Software Heritage has made the decision to close their former [https://libera.chat libera.chat] IRC channels on 2023-12-08

Matrix

2025-09-07T07:26:08Z

Vlorentz: /* Matrix rooms */ Matrix is the network, matrix.org is a homeserver (though also the front page of the network)

== Matrix rooms ==

The following rooms have been registered on the [https://matrix.org/ Matrix] network for [[Software Heritage]] usage.

* [https://matrix.to/#/#swh-devel:matrix.org '''#swh-devel''']: public development discussions
* [https://matrix.to/#/#swh-sysadm:matrix.org '''#swh-sysadm''']: operations team discussions/bots
* [https://matrix.to/#/#swh-offtopic:matrix.org '''#swh-offtopic''']: Off-topic discussions
* [https://matrix.to/#/#swh-team:matrix.org '''#swh-team''']: private discussions of the core team
* [https://matrix.to/#/#swh:matrix.org '''#swh''']: general discussions around Software Heritage

They are all part of the [https://matrix.to/#/#swh-space:matrix.org '''#swh-space'''] Matrix space.

You will be asked to create a Matrix account if you don't have one yet.

Software Heritage has made the decision to close their former [https://libera.chat libera.chat] IRC channels on 2023-12-08

Matrix

2025-09-07T07:24:36Z

Vlorentz:

== Matrix rooms ==

The following rooms have been registered on the [https://matrix.org/ matrix.org] network for [[Software Heritage]] usage.

* [https://matrix.to/#/#swh-devel:matrix.org '''#swh-devel''']: public development discussions
* [https://matrix.to/#/#swh-sysadm:matrix.org '''#swh-sysadm''']: operations team discussions/bots
* [https://matrix.to/#/#swh-offtopic:matrix.org '''#swh-offtopic''']: Off-topic discussions
* [https://matrix.to/#/#swh-team:matrix.org '''#swh-team''']: private discussions of the core team
* [https://matrix.to/#/#swh:matrix.org '''#swh''']: general discussions around Software Heritage

They are all part of the [https://matrix.to/#/#swh-space:matrix.org '''#swh-space'''] Matrix space.

You will be asked to create a Matrix account if you don't have one yet.

Software Heritage has made the decision to close their former [https://libera.chat libera.chat] IRC channels on 2023-12-08

Matrix

2025-09-07T07:23:23Z

Vlorentz: /* Matrix rooms */ Use matrix.to links instead of hardcoding the app.element.io client

== Matrix rooms ==

The following rooms have been registered on the [https://matrix.org/ matrix.org] network for [[Software Heritage]] usage.

* [https://matrix.to/#/#swh-devel:matrix.org '''#swh-devel''']: public development discussions
* [https://matrix.to/#/#swh-sysadm:matrix.org '''#swh-sysadm''']: operations team discussions/bots
* [https://matrix.to/#/#swh-offtopic:matrix.org '''#swh-offtopic''']: Off-topic discussions
* [https://matrix.to/#/#swh-team:matrix.org '''#swh-team''']: private discussions of the core team
* [https://matrix.to/#/#swh:matrix.org '''#swh''']: general discussions around Software Heritage

You will be asked to create a Matrix account if you don't have one yet.

Software Heritage has made the decision to close their former [https://libera.chat libera.chat] IRC channels on 2023-12-08

Matrix

2024-09-12T15:19:24Z

Vlorentz: s/Element account/Matrix account/

== Matrix rooms ==

The following rooms have been registered on the [https://matrix.org/ matrix.org] network for [[Software Heritage]] usage.

* [https://app.element.io/#/room/#swh-devel:matrix.org '''#swh-devel''']: public development discussions
* [https://app.element.io/#/room/#swh-sysadm:matrix.org '''#swh-sysadm''']: operations team discussions/bots
* [https://app.element.io/#/room/#swh-offtopic:matrix.org '''#swh-offtopic''']: Off-topic discussions
* [https://app.element.io/#/room/#swh-team:matrix.org '''#swh-team''']: private discussions of the core team
* [https://app.element.io/#/room/#swh:matrix.org '''#swh''']: general discussions around Software Heritage

You will be asked to create a Matrix account if you don't have one yet.

Software Heritage has made the decision to close their former [https://libera.chat libera.chat] IRC channels on 2023-12-08

Easy hacks

2024-02-20T20:39:43Z

Vlorentz: fix link

A '''easy hack''' (or a "easy bug", or a "newcomer bug", [http://wiki.openhatch.org/Easy_bugs_for_newcomers depending on the project]) is a pending task in the development of a collaborative Free/Open Source project that is considered to be well-suited for new contributors.
It might indeed be a trivial bug to fix/task to accomplish, or it might be something less trivial but that is well adapted to learn how a specific part/process of the project works.

[[Software Heritage]] curates a [https://forge.softwareheritage.org/tag/easy_hack/ list of easy hacks] that are suitable for newcomers to the project.
Any in the [[Phabricator|forge]] tagged with <code>Easy hack</code> will appear on the list.

* if '''you are a newcomer''' to Software Heritage wondering how you can help, have a look at [https://forge.softwareheritage.org/tag/easy_hack/ the list] and see if anything there inspires you
* if '''you are a contributor''' to Software Heritage already, keep an eye on tasks that are easy and/or suitable for newcomers and tag them with <code>Easy hack</code> as needed

== Links ==

* [https://gitlab.softwareheritage.org/dashboard/issues?sort=created_date&state=opened&label_name%5B%5D=Easy+hack list of easy hacks]
* [https://gitlab.softwareheritage.org/dashboard/issues?sort=created_date&state=opened&assignee_id=None bug tracker], with all unassigned bugs/tasks

[[Category:Software development]]

Easy hacks

2024-02-20T20:38:57Z

Vlorentz: /* Links */ Link to Gitlab instead of Phabricator

A '''easy hack''' (or a "easy bug", or a "newcomer bug", [http://wiki.openhatch.org/Easy_bugs_for_newcomers depending on the project]) is a pending task in the development of a collaborative Free/Open Source project that is considered to be well-suited for new contributors.
It might indeed be a trivial bug to fix/task to accomplish, or it might be something less trivial but that is well adapted to learn how a specific part/process of the project works.

[[Software Heritage]] curates a [https://forge.softwareheritage.org/tag/easy_hack/ list of easy hacks] that are suitable for newcomers to the project.
Any in the [[Phabricator|forge]] tagged with <code>Easy hack</code> will appear on the list.

* if '''you are a newcomer''' to Software Heritage wondering how you can help, have a look at [https://forge.softwareheritage.org/tag/easy_hack/ the list] and see if anything there inspires you
* if '''you are a contributor''' to Software Heritage already, keep an eye on tasks that are easy and/or suitable for newcomers and tag them with <code>Easy hack</code> as needed

== Links ==

* [https://gitlab.softwareheritage.org/dashboard/issues?sort=created_date&state=opened&label_name[]=Easy+hack list of easy hacks]
* [https://gitlab.softwareheritage.org/dashboard/issues?sort=created_date&state=opened&assignee_id=None bug tracker], with all unassigned bugs/tasks

[[Category:Software development]]

Matrix

2023-11-30T10:05:42Z

Vlorentz:

== Matrix rooms ==

The following rooms have been registered on the [https://matrix.org/ matrix.org] network for [[Software Heritage]] usage.

* [https://app.element.io/#/room/#swh-devel:matrix.org '''#swh-devel''']: public development discussions
* [https://app.element.io/#/room/#swh-sysadm:matrix.org '''#swh-sysadm''']: operations team discussions/bots
* [https://app.element.io/#/room/#swh-offtopic:matrix.org '''#swh-offtopic''']: Off-topic discussions
* [https://app.element.io/#/room/#swh-team:matrix.org '''#swh-team''']: private discussions of the core team
* [https://app.element.io/#/room/#swh:matrix.org '''#swh''']: general discussions around Software Heritage

You will be asked to create a [https://element.io/ Element] account if you don't have one yet.

Software Heritage has made the decision to close their former [https://libera.chat libera.chat] IRC channels on 2023-12-08

Matrix

2023-06-10T16:26:40Z

Vlorentz: /* Matrix bridge */

== IRC channels ==

The following channels have been registered on the [https://libera.chat/ libera.chat] IRC network for [[Software Heritage]] usage.

* [https://app.element.io/#/room/#swh-devel:matrix.org '''#swh-devel''']: public development discussions
* [https://app.element.io/#/room/#swh-sysadm:matrix.org '''#swh-sysadm''']: operations team discussions/bots
* [https://app.element.io/#/room/#swh-offtopic:matrix.org '''#swh-offtopic''']: Off-topic discussions
* [https://app.element.io/#/room/#swh-team:matrix.org '''#swh-team''']: private discussions of the core team
* [https://app.element.io/#/room/#swh:matrix.org '''#swh''']: general discussions around Software Heritage

If you use IRC, consider joining the channels.

If you don't use IRC ''directly'', you can still join our chat channels from your web browser via a [https://matrix.org/ Matrix] bridge by clicking on the channel names in the list above. You will be asked to create a [https://element.io/ Element] account if you don't have one yet.

== IRC authentication ==

<b>Libera.chat defaults to blocking private messages from unauthentified users! All users should register their nicknames to be able to message one another privately, by following the instructions below.</b> If you're really unable to register, you should ask your correspondent to [https://libera.chat/guides/usermodes consider setting usermode <tt>-R</tt>, and <tt>+g</tt>]

To register an account with NickServ, please follow [https://libera.chat/guides/registration the registration instructions provided by libera.chat staff].

You will then receive an e-mail containing a link to activate you account. After doing so, you need to configure your client to auto-authenticate. The recommended way of doing that is using [https://libera.chat/guides/sasl SASL authentication].

For matrix, the relevant docs is here: https://github.com/matrix-org/matrix-appservice-irc/wiki/End-user-FAQ#how-do-i-registeridentify-to-nickserv

libera.chat also supports authentication via [https://libera.chat/guides/certfp TLS client certificates (using SASL EXTERNAL)].

=== Matrix bridge ===

For registering an account through the Matrix bridge ([https://github.com/matrix-org/matrix-appservice-irc/wiki/End-user-FAQ#how-do-i-registeridentify-to-nickserv relevant docs here]), please follow these instructions:

1. Choose a short nickname (the default nickname picked by the matrix bridge has a [m] and can be quite long, as it defaults to your Matrix display name (minus non-ASCII non-alphanumerical characters)

/msg @appservice:libera.chat !nick <USERNAME>

2. Send this command to NickServ to register your account:

/msg @NickServ:libera.chat register <PASSWORD> <EMAIL>

3. Once you receive the confirmation email with a token, activate your account by using:

/msg @NickServ:libera.chat VERIFY REGISTER <USERNAME> <TOKEN RECEIVED BY EMAIL>

4. Give the Matrix bridge appservice your password so that you get identified automatically when matrix reconnects you to IRC:

/msg @appservice:libera.chat !username <USERNAME>
/msg @appservice:libera.chat !storepass <PASSWORD>

== IRC access list ==

To auto-voice people with a registered nick (only doable by people with +fA access modes will be able to do it), add them to the team channel access list:

/msg chanserv flags #swh-team add <nickname> Staff

Other channels pick their ACLs from that of the #swh-team channel.

If you already have the right (+o ChanServ flag), you can make yourself an operator, with:

/msg chanserv OP #swh-devel

[[Category:Infrastructure]]

A practical approach to efficiently store 100 billions small objects in Ceph

2023-04-30T13:35:32Z

Vlorentz: Finish removing the EOS column (only partially removed by "EOS is not based on Ceph" causing cells to be mismatched)

The [https://en.wikipedia.org/wiki/Software_Heritage Software Heritage] project mission is to collect, preserve and share all software that is available in source code form, with the goal of building a common, shared infrastructure at the service of industry, research, culture and society as a whole. As of February 2021 it contains 10 billions unique source code files (or “objects”, in the following) totaling ~750TB of (uncompressed) data and grows by 50TB every month. 75% of these objects have a size smaller than 16KB and 50% have a size smaller than 4KB. But these small objects only account for ~5% of the 750TB: 25% of the objects have a size > 16KB and occupy ~700TB.

The desired performances for '''10PB''' and '''100 billions objects''' are as follows:

* The clients aggregated together can write at least 3,000 objects/s and at least 100MB/s.
* The clients aggregated together can read at least 3,000 objects/s and at least 100MB/s.
* There is no space amplification for small objects.
* Getting the first byte of any object never takes longer than 100ms.
* Objects can be enumerated in bulk, at least one million at a time.
* Mirroring the content of the Software Heritage archive can be done in bulk, at least one million objects at a time.

Using an off-the-shelf object storage such as the [https://docs.ceph.com/en/latest/radosgw/ Ceph Object Gateway] or [https://min.io/ MinIO] does not meet the requirements:

* There is a significant space amplification for small objects: at least 25%, depending on the object storage (see “How does packing Objects save space?” below for details)
* Mirroring the content of the archive can only be done one object at a time and not in bulk which takes at least 10 times longer (see “How does packing Objects help with enumeration?” for details)

A new solution must be implemented by re-using existing components and made available for system administrators to conveniently deploy and maintain in production. There are three ways to do that:

* Contribute packaging and stable releases to a codebase such as [https://github.com/linkedin/ambry Ambry].
* Modify an object storage such as MinIO to support object packing.
* Get inspiration from an object storage design.

For reasons explained below (see “Storage solutions and TCO”), it was decided to design a new object storage and implement it from scratch.

= Proposed object storage design =

In a nutshell, objects are written to databases running on a fixed number of machines (the Write Storage) that can vary to control the write throughput. When a threshold is reached (e.g. 100GB) all objects are put together in container (a Shard), and moved to a readonly storage that keeps expanding over time. After a successful write, a unique identifier (the Object ID) is returned to the client. It can be used to read the object back from the readonly storage. Reads scale out because the unique identifiers of the objects embed the name of the container (the Shard UUID). Writes also scales out because the Database is chosen randomly. This is the Layer 0.

Clients that cannot keep track of the name of the container can rely on an API that relies on an index mapping all known objects signatures (the Object HASH below) to the name of the container where they can be found. Although this index prevents scaling out writes, the readonly storage can still scale out by multiplying copies of the index as needed. This is the Layer 1.

<pre>
Layer 0 scales out

+--- write op ----+ +--- read op ----+
v ^ v ^
Object & | | |
Object HASH Object ID Object ID Object
| Object HASH | |
v Shard UUID v ^
| | | |
v ^ v ^
+---- Write Storage --------+ +---- Read Storage --------+
| | | |
| +----------+ | | +-------+ +-------+ |
| | Database |->--Packing->----> | Shard | | Shard | |
| +----------+ | | +-------+ +-------+ |
| +----------++----------+ | | +-------+ +-------+ |
| | Database || Database | | | | Shard | | Shard | |
| +----------++----------+ | | +-------+ +-------+ |
| | | +-------+ +-------+ |
+---------------------------+ | | Shard | | Shard | |
| +-------+ +-------+ |
| ... |
+--------------------------+

Layer 1 reads scale out

+---- Write Storage --------+ +---- Read Storage ---------+
| | | |
|+-------------------------+| |+-------------------------+|
||Object HASH to Shard UUID|| ||Object HASH to Shard UUID||
|| index |>>>>| index ||
|+-------------------------+| |+-------------------------+|
+---------------------------+ |+-------------------------+|
| | ||Object HASH to Shard UUID||
^ v || index ||
| | |+-------------------------+|
^ v | ... |
Object | +---------------------------+
Object HASH v | |
| | ^ v
^ v | |
| | Object HASH Object
^ v | |
| | ^ v
+--- write op ----+ +--- read op ----+
</pre>

[[File:Ceph-objstorage-sw-architecture.svg]]

== Glossary ==

* Object: an opaque sequence of bytes.
* Object HASH: the hash of an Object, e.g., the checksum part of a [https://docs.softwareheritage.org/devel/swh-model/persistent-identifiers.html#core-identifiers SWHID].
* Shard: a group of Objects, used to partition the full set of objects into manageable subsets.
* Shard UUID: the unique identifier of a Shard, as a [https://en.wikipedia.org/wiki/Universally_unique_identifier UUID].
* Object ID: a pair made of the Object HASH and the Shard UUID containing the object.
* Global Index: a table mapping the Object HASH to the Shard UUID that contains the Object.
* Read Storage: the unlimited size storage from which clients can only read Objects. It only contains Objects up to a given point in time.
* Write Storage: the fixed size storage from which clients can read or write. If an Object is not found in the Write storage, it must be retrieved from the Read Storage.
* Object Storage: the content of the Write Storage and the Read Storage combined.
* Database: [https://en.wikipedia.org/wiki/PostgreSQL PostgreSQL], [https://en.wikipedia.org/wiki/Apache_Cassandra Cassandra], etc.
* [https://en.wikipedia.org/wiki/Ceph_(software) Ceph]: a self-healing distributed storage.
* [https://docs.ceph.com/en/latest/rbd/ RBD] image: a Ceph block storage that can either be used via the librbd library or as a block device from /dev/rbd.
* [https://en.wikipedia.org/wiki/Total_cost_of_ownership TCO]: Total Cost of Ownership

The key concepts are:

* Packing millions of Objects together in Shards to:
** save space and,
** efficiently perform bulk actions such as mirroring or enumerations.
* Two different storage:
** Read Storage that takes advantage of the fact that Objects are immutable and never deleted and,
** Write Storage from which Shards are created and moved to the Read Storage.
* Identifying an object by its Object HASH and the Shard UUID that contains it so that its location can be determined from the Object ID.

While the architecture based on these concepts scales out for writing and reading, it cannot be used to address Objects with their Object HASH alone which is inconvenient for a number of use cases. An index mapping the Object HASH to the Shard UUID must be added to provide this feature, but it does not scale out writes.

The content of the Object Storage (i.e., the Write Storage and the Read Storage combined) is '''strongly/strictly consistent'''. As soon as an Object is written (i.e., the write operation returns to the client), a reader can get the Object content from the Object Storage (with the caveat that it may require looking up the object from both the Write Storage and Read Storage).

The Read Storage is '''eventually consistent'''. It does not contain the latest Objects inserted in the Write Storage but it will, eventually. It contains all objects inserted in the Object Storage, up to a given point in time.

== Layer 0 (Object lookup require a complete Object ID) ==

=== Architecture ===

* Write Storage:
** A fixed number of Databases
* Read Storage:
** Shards implemented as Ceph RBD images named after their Shard UUID
** The content of the Shard uses a format that allows retrieving an Object in O(1) given the Object HASH

=== Writing ===

The Object is stored in one of the Databases from the Write Storage. The Database is chosen at random. A database is associated with a unique Shard UUID, chosen at random. All Objects written to a Database will be stored in the same Shard.

A successful Object write returns the Object ID. Writing the same object twice may return different Object IDs. The Object HASH will be the same because it is based on the content of the Object. But the Shard in which the Object is stored may be different since it is chosen at random.

=== Packing ===

When a Database grows bigger than a threshold (for instance 100GB), it stops accepting writes. A Shard is created in the Read Storage and Objects in the Database are sorted and copied to it. When the Shard is complete, the Database is deleted. Another Database is created, a new Shard UUID is allocated and it starts accepting writes.

=== Reading ===

The Shard UUID is extracted from the Object ID. If a Shard exists in the Read Storage, the Object HASH is used to lookup the content of the Object. Otherwise the Database that owns the Shard UUID is looked up in the Write Storage and the Object HASH is used to lookup the content of the Object. If the reader is not interested in the most up to date content, it can limit its search to the Read Storage.

== Layer 1 (Objects can be looked up using the Object HASH alone) ==

A Global Index mapping the Object HASH of all known Objects to the Shard UUID is used to:

* allow clients to fetch Objects using their Object HASH only instead of their Object ID.
* deduplicate identical Objects based on their Object HASH

=== Architecture ===

* Write Storage:
** Read/write Global Index of all known Objects in the Write Storage and the Read Storage
* Read Storage:
** Read/write Global Index of all known Objects in the Read Storage
** Multiple readonly replicas of the Global Index of all known Objects in the Read Storage

=== Writing ===

If the Object HASH exists in the Read Storage Global Index, do nothing. Otherwise perform the write and add the Object ID to the Write Storage Global Index. There may be duplicate Objects in the Write Storage. It is expected that they race to be inserted in the Write Storage Global Index.

=== Packing ===

During packing, each Object HASH is looked up in the Read Storage Global Index. If it exists, the object is discarded. Otherwise its Object ID is added to the Read Storage Global Index. When packing is complete:

* Readonly replicas of the Read Storage Global Index are updated with the newly added Object IDs.
* Object HASH that were found to be duplicate are updated in the Write Storage Global Index. The Object HASH is mapped to the Shard UUID retrieved from the Read Storage Global Index.

=== Reading ===

If the Object HASH is found in the Read Storage Global Index, use the Shard UUID to read the Object content from the Shard found in the Read Storage. Otherwise lookup the Object HASH from the Write Storage Global Index and read the content of the Object from the Database that owns the Shard UUID.

= How does packing Objects save space? =

The short answer is: it does not when Objects are big enough, but it does when there are a lot of small Objects.

If there are billions of objects (i.e., less than one billion is not a lot) and 50% of them have a size smaller than 4KB and 75% of them have a size smaller than 16KB (i.e., bigger than 16KB is not small), then packing will save space.

In the simplest method of packing (i.e., appending each Object after another in a file) and since the Object HASH has a fixed size, the only overhead for each object is the size of the Object (8 bytes). Assuming the Shard containing the Objects is handled as a single 100GB Ceph RBD Image, it adds R bytes. If the underlying Ceph pool is erasure coded k=4,m=2 an additional 50% must be added.

Retrieving an Object from a Shard would be O(n) in this case because there is no index. It is more efficient to [https://en.wikipedia.org/wiki/Perfect_hash_function add a minimal hash table] to the Shard so that finding an object is O(1) instead. That optimization requires an additional 8 bytes per Object to store their offset, i.e. a total of 16 bytes per object.

If Objects are not packed together, each of them requires at least B bytes, which is the minimum space overhead imposed by the underlying storage system. And an additional 50% for durability. The space used by Objects that are smaller than a given threshold will be amplified, depending on the underlying storage. For instance all objects in Ceph have a minimum size of 4KB, therefore the size of a 1KB Object will be amplified to 4KB which translates to a [https://forge.softwareheritage.org/T3052#58864 35% space amplification]. Another example is MinIO with [https://github.com/minio/minio/issues/7395#issuecomment-475161144 over 200% space amplification] or [https://wiki.openstack.org/wiki/Swift/ideas/small_files#Challenges Swift] for which [https://www.ovh.com/blog/dealing-with-small-files-with-openstack-swift-part-2/ packing small files was recently proposed].

To summarize, the overhead of storing M Objects totaling S bytes with M=100 billions and S=10PB is:

* '''packed:''' ~15.5PB
** (S / 100GB) * R == (10PB / 100GB) * R bytes = 10,000 * R bytes
** (M * 24) = 100G Objects * 24 bytes = 2.4TB
** 50% for durability = 10PB * 0.5 = 5PB
* '''not packed:''' ~17.5PB based on the optimistic assumption that the storage system has a 25% space overhead for small files
** 25% for space amplification = 10PB * 0.25 = 2.5PB
** 50% for durability = 10PB * 0.5 = 5PB

= How does packing Objects help with enumeration? =

For mirroring or running an algorithm on all objects, they must be enumerated. If they are not packed together in any way, which is the case with MinIO or Swift, they must be looked up individually. When they are packed together (one million or more), the reader can download an entire Shard instead, saving the accumulated delay imposed by millions of individual lookup.

If looking up an individual Object takes 10 milliseconds and Shards can be read at 100MB/s:

* Getting 1 billion objects requires 10 millions seconds which is over 100 days.
* One billion objects is 1/10 of the current content of Software Heritage, i.e. ~75TB which can be transferred by reading the Shards in less than 10 days

= Storage solutions and TCO =

When looking for off-the-shelf solutions all options were considered, [https://forge.softwareheritage.org/T3107 including distributed file systems such as IPFs and more] and most of them were discarded because they had at least one blocker that could not be fixed (e.g. no feature to guarantee the durability of an object). In the end a few remained, either including the following features or with the possibility for a third party to contribute them back to the project:

* '''Scale''' to 100 billions objects
* Provide object '''packing'''
* Provide detailed '''documentation''' and community support for system administrators operating the storage
* Be thoroughly '''tested''' before a stable release is published
* Be '''packaged''' for at least one well known distribution
* Have '''stable releases''' maintained for at least two years
* A sound approach to address '''security''' problems (CVE etc.)

{|
! Name
! RGW
! SeaweedFS
! MinIO
! Swift
! Ambry
|-
| Scaling
| Yes
| Yes
| Yes
| Yes
| Yes
|-
| Packing
| No
| Yes
| No
| No
| Yes
|-
| Documentation
| Good
| Terse
| Good
| Good
| Terse
|-
| Tests
| Good
| Few
| Average
| Good
| Few
|-
| Packages
| Yes
| No
| No
| Yes
| No
|-
| Stable releases
| Yes
| No
| Yes
| Yes
| No
|-
| Security
| Yes
| No
| Yes
| Yes
| No
|}

== Does not have stable releases and testing ==

The performance goals, size distribution and the number of objects in Software Heritage are similar to what is described in the 2010 article “[https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf Finding a needle in Haystack: Facebook’s photo storage]” that motivated the implementation of [https://github.com/chrislusf/seaweedfs SeaweedFS] in 2013 or [https://github.com/linkedin/ambry Ambry], the object storage published in 2017 by LinkedIn to store and serve trillions of media objects in web companies.

Contributing to SeaweedFS or Ambry so they can be deployed and maintained would require:

* Creating packages for the target Operating System (e.g. Debian GNU/Linux), maintaining a repository to distribute them, upload them to the official distribution repository so that they are available in the next stable release (about two years from now)
* Creating Ansible roles or Puppet modules for deployment on multiple machines
* Improving the documentation with a configuration and architecture guide to deploy at scale
* Discuss with upstream to create stable releases, define their lifecycle and organize release management
* Establish a security team in charge of handling the CVE
* Setup and infrastructure and create the software for integration testing to be run before a stable release is published to reduce the risk of regressions or data loss. This is specially important because a significant part of the software is dedicated to data storage and replication: bugs can lead to data loss or corruption.

== Does not provide object packing ==

[https://min.io/ MinIO] and [https://docs.openstack.org/swift/latest/ Swift] suffer from a space amplification problem and they do not provide object packing. Although [https://docs.ceph.com/en/latest/radosgw/ Ceph Object Gateway] (also known as RGW) stores objects in RocksDB instead of files, it also suffers from a space amplification problem and does not provide object packing.

Contributing to RGW, MinIO or Swift to add object packing would require:

* Creating a blueprint to modify the internals to add object packing
* Discuss with upstream to validate the blueprint
* Implement the blueprint and the associated tests

== Estimating the TCO ==

Since no solution can be used as is, some work must be done in each case and the effort it requires should be compared. It is however difficult because the nature of the effort is different. The following factors were considered and aggregated in a TCO estimate.

* '''Data loss risk:''' if a bug in the work done implies the risk of losing data, it makes the work significantly more complicated. It is the case if packing must be implemented in the internals of an existing object storage such as Swift. It is also the case if an object storage does not have integration testing to verify upgrading to a newer version won’t lead to a regression, which is the case with Ambry. It is likely that the Ambry upstream has extensive integration testing but they are not published.
* '''Large codebase:''' a large codebase means modifying it (to implement packing) or distributing it (packaging and documentation) is more difficult
* '''Language:''' if the language and its environment is familiar to the developers and the system administrators, the work is less difficult
* '''Skills:''' if the work requires highly specialized skills (such as an intimate understanding of how a distributed storage system guarantees a strict consistency of the data, or running integration tests that require a cluster of machines) it is more difficult

{|
!
! RGW
! SeaweedFS
! MinIO
! Swift
! Ambry
|-
| Data loss risk
| Yes
| Yes
| Yes
| Yes
| Yes
|-
| Large codebase
| Yes
| Yes
| Yes
| Yes
| Yes
|-
| Language
| C++
| Go
| Go
| Python
| Java
|-
| Skills
| Medium
| High
| High
| High
| High
|-
| TCO estimate
| High
| High
| High
| High
| High
|}

In a nutshell, implementing a system from scratch has the lowest TCO estimate, primarily because it is independent of the underlying distributed storage.

[[Category:Software development]]

Suggestion box: source code to add

2022-07-19T21:15:32Z

Vlorentz: /* Suggestions */

The [[Archive]] is growing organically. We started "small", tracking 3 '''software origins''' (GitHub + Debian + GNU), and we will be adding new origins bit by bit, depending on the urgency of archiving them and available development energies to integrate them into Software Heritage.

Using this page you can add suggestions of software origins that we aren't following yet, but we should. You can include information about who to contact for technical collaboration, the urgency of archival, and other useful information. To that end, just add a row to the table below. Here some information about the meaning of the various columns.

Entries are currently listed simply in order of addition to this page; we will add more structure when the list will start growing.

=== Legend ===

;Software origin
: any (public accessible) "place" on the Internet that host software in source code form. Please provide a title for it and hyperlink it to the relevant URL
;Type of origin
: information about the kind of hosting, e.g., whether it is a forge, a collection of repositories, an homepage publishing tarball, or a one shot source code repository. For all kind of repositories please specify which VCS system is in use (Git, SVN, CVS, etc.)
;Contact
: who to contact for technical collaboration on how to best archive source code hosted on the software origin. You can list yourself if you're the relevant person, or provide the most relevant contact point if you know it
;Conservation status
: information about how likely it is that the software origin will disappear; high likelihood will make it more urgent for us to archive software hosted there. We suggest to use the [https://en.wikipedia.org/wiki/Conservation_status species conservation status], i.e., one of: Critically endangered (CR), Endangered (EN), Vulnerable (VU), Near threatened (NT), Least concern (LC).
;How to mirror
: (pointers to) technical information on how to do a full mirror of ''all'' the source code available at the software origin, ideally one shot and in batch
;How to keep up
: (pointers to) technical information on how to incrementally retrieve new source code accumulated since the last visit; usually this should be based on some kind of incremental change feed or event API
;Notes
: anything else you think we should know about this software origin
;Forge task
: pointer to the task on our [[forge]] tracking the work needed to ingest the software origin

== Suggestions ==

* https://notabug.org/ (customized gogs)
* https://gitgud.io/ (GitLab, run by Sapphire, a user-supported open source company)
* https://octo.sh/ (GitLab)
* https://chaos.expert/explore (GitLab by Chaos Computer Club)
* https://gitlab.coko.foundation/public (GitLab)
* https://git.teknik.io/explore/repos (Gitea)
* https://gitlab.gnome.org/explore/groups (GNOME software)
* https://launchpad.net/ (used by Ubuntu and others)
* https://archive.codeplex.com/ (was Microsoft's free, open source project hosting). It is now dead, but there seems to be a dump here: https://archive.org/details/sylirana_ms_codeplex_zips
* https://wiki.p2pfoundation.net/List_of_Community-Hosted_GitLab_Instances
* https://0xacab.org/explore (Gitlab)
* https://git.fosscommunity.in/explore/projects (GitLab by Free Software Community of India)
* https://git.sdf.org/humanacollaborator/humanacollabora/src/branch/master/forge_comparison.md <= list of forges to eventually sync with the table below. Note that there may be some [https://framablog.org/2019/09/26/lets-de-frama-tify-the-internet urgency] to harvest framagit.org.

{| class="wikitable"
!Software origin
!Type of origin
!Contact
!Conservation status
!How to mirror
!How to keep up
!Notes
![https://forge.softwareheritage.org/ Forge] task
|-
|'''(sample entry)''' ''GitHubBub forge''
|''Git hosting''
|''John Doe <john@example.com>''
|''LC''
|''retrieve full repo list at /api/list, then git clone on each entry''
|''poll RSS feed at /api/updates?since=YYYY-MM-DD''
|''nothing special to add''
|''[https://forge.softwareheritage.org/T123456 T123456]''
|-
|[https://bitbucket.org/ Bitbucket]
|Git and hg/Mercurial hosting
|
|style="background-color: lightgreen"| LC
|
|
|
|[https://forge.softwareheritage.org/T561 T561]
|-
|[https://sourceforge.net/ SourceForge]
|CVS, SVN, Mercurial, Git
|
|style="background-color: orange"|VU
|
|
|
|
|-
|[https://wiki.debian.org/Derivatives/Census all Debian derivatives]
|Debian-based distros
|Paul Wise <pabs@debian.org>
|varying, depending on the distro
|see [[Suggestion_box:_source_code_to_add/Debian_derivatives|details]]
|see [[Suggestion_box:_source_code_to_add/Debian_derivatives|details]]
|
|
|-
|[https://www.gentoo.org/ Gentoo]
|
|Johannes Kellner <gentoo@johannes-kellner.eu>
|style="background-color: lightgreen"|LC
|
|
|
|
|-
|[http://pauillac.inria.fr/~huet/cea.html Gérard Huet's seminal work on 3D]
|Scanned source code
|Gérard Huet <gerard.huet@inria.fr>
|style="background-color: red"|EN
|retrieve listing images from the web pages
|N/A
|links are half broken, yquem should be replaced with pauillac everywhere it appears
|
|-
|[http://www.softwarepreservation.org/projects Software Preservation Project]
|Website with a collection of archives
|Paul McJones <paul@mcjones.org>
|style="background-color: lightgreen"|LC
|
|
|
|
|-
|[https://code.nasa.gov/ 253 NASA open source software projects]
|
|
|style="background-color: lightgreen"|LC
|
|
|
|
|-
|[http://smaky.ch/ Smaky], the swiss micro-computer series
|[http://infini.smaky.ch/sources.html Partial code dump]
|[mailto:arnaud@epsitec.ch Pierre Arnaud] (current CEO of Epsitec) and/or [mailto:jean-daniel.nicoud@epfl.ch Jean-Daniel Nicoud] (founder of the computer series]
|style="background-color: red"|EN
|Probably manually
|No new updates
|Some references to this history: [http://www.memoires-informatiques.org/ Fondation Mémoires Informatiques], [http://smaky.ch/ Smaky.ch] (in particular, [http://smaky.ch/theme.php?id=lami the short history]
|
|-
|[https://www.wikidata.org/wiki/Wikidata:WikiProject_Informatics/FLOSS#Conservation_status_2 wikidata endangered software]
|depends on the "source code repository" property
|Loic Dachary <loic@dachary.org>
|style="background-color: yellow"|The risk is higher than [https://www.wikidata.org/wiki/Property_talk:P141 LC]
|A script should obtain the "source code repository" property for the software and mirror it depending on the [https://www.wikidata.org/wiki/Wikidata:WikiProject_Informatics/FLOSS#source_code_repository protocol] qualifier. If the "source code repository" is "no value", the [https://www.wikidata.org/wiki/Wikidata:WikiProject_Informatics/Software#streaming_media_URL streaming media URL] of the "preferred" [https://www.wikidata.org/wiki/Wikidata:WikiProject_Informatics/Software#software_version_.28P348.29 software version] should be downloaded instead.
|Once a copy is secured by software heritage, a URL to the software heritage repository should be added to the "source code repository" property and the "conservation status" property should be removed, meaning it is "least concerned" by default. The software will no longer show in the list of endangered software.
|This is work in progress, part of the [https://www.wikidata.org/wiki/Wikidata:WikiProject_Informatics/FLOSS wikidata FLOSS project] and the scripts do not exist yet.
|
|-
|historical KDE repos
|CVS/SVN/Git
|KDE sysadmin team <sysadmin@kde.org>
|style="background-color: yellow"|NT
|See [[Suggestion box: source code to add/KDE|details]]
|See [[Suggestion box: source code to add/KDE|details]]
|
|
|-
|[https://java.net/projects Java.net] & [https://kenai.com/ Kenai.com]
|hg, git, svn
|communitymanager@java.net
|style="background-color: black; color: white"|CR
|
|
|[https://community.oracle.com/community/java/javanet-forge-sunset Shutting down on April 28, 2017]
|
|-
|[https://fedorahosted.org/ fedorahosted.org]
|git, svn, hg, bzr
|[https://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org infrastructure@lists.fedoraproject.org]
|style="background-color: black; color: white"|CR
|
|
|[https://communityblog.fedoraproject.org/fedorahosted-sunset-2017-02-28/ Shutting down on Feb. 28, 2017]
|
|-
|[http://www.societe-informatique-de-france.fr/wp-content/uploads/2015/12/1024-no7-Baude.pdf Langage symbolique d'Enseignement (LSE)]
|archives
|Association Enseignement Public et Informatique (EPI) <bureau@epi.asso.fr>
|style="background-color: black; color: white"|CR
|
|
| + educational software (INRP-CNDP)
|
|-
|[http://www.netlib.org The Netlib collection of numerical software]
|structured website with links to archives
|
|style="background-color: lightgreen"|LC
|
|
|many of these libraries are mirrored in sources already collected in Software Heritage; there is sure value in the curation information.
|
|-
|[https://codebender.cc/ codebender]
|IOT and educationnal ressources
|
|style="background-color: black; color: white"|CR
|
|
|closing expected at Dec31st, 2016 upon this announce by founders https://codebender.cc/next-chapter
|
|-
|[https://git.oschina.net/ OS China]
|Chinese GitHub equivalent
|
|style="background-color: lightgreen"|LC
|
|
|interesting test case for all the unicode tooling in Software Heritage
|
|-
|Usenet source code archives
|NNTP
|
|style="background-color: yellow"|NT
|crawl relevant newsgroup archives (e.g., at Google Groups), retrieve archives (possibly chunked), ingest
|one shot might be enough?
|suggestion by John Gilmore
|
|-
|[https://www.x.org/releases/ X11/XOrg archives]
|http
|
|style="background-color: lightgreen"|LC
|
|one shot might be enough
|
||''[https://forge.softwareheritage.org/T1774 T1774]''
|-
|[https://www.curseforge.com/ Curse mods]
|Code distributed as versioned tarballs
|
|style="background-color: lightgreen"|LC
|The API should be sufficient, maybe some scraping will be required
|
|
|
|-
|Symbian source code
|HG
|carlo.daffara@nodeweaver.eu
|style="background-color: orange"|VU
|
|
|Cloned from Nokia's Symbian Mercurial repository, a few days before the closure of the repo and the change to a proprietary license. The mercurial HG files are the only copy publicly available; I have made a snapshot of the code and placed on sourceforge- but missing is the entire project history and commit log.
|
|-
|[https://gist.github.com/ GitHub's gists]
|git
|
|style="background-color: lightgreen"|LC
|No proper listing API, but there's [https://developer.github.com/v3/gists/#list-all-public-gists an endpoint] to get gists created after a given date
|use the same endpoint
|
|
|-
|[https://codeberg.org/ Codeberg]
|git
|contact@codeberg.org
|style="background-color: lightgreen"|LC
|[https://github.com/go-gitea Gitea] API
|[https://github.com/go-gitea Gitea] API
|Codeberg e.V. is a Non-Profit Collaboration Community for Free and Open Source Projects
|
|-
|[https://puszcza.gnu.org.ua/ Puszcza]
|cvs/git/hg/svn, VCS snapshot tarballs, tarballs
|https://puszcza.gnu.org.ua/contact.php
|style="background-color: lightgreen"|LC
|Savane instance, https://download.gnu.org.ua/ (also has ftp), http://git.gnu.org.ua/
|Savane instance
|
|
|-
|[https://directory.fsf.org/ Free Software Directory]
|solely contains links (to VCS and tarballs)
|https://lists.gnu.org/archive/html/directory-discuss/
|style="background-color: lightgreen"|LC
|MediaWiki instance, use API to clone (or git-remote-mediawiki)
|MediaWiki instance, use API for updates clone (or git-remote-mediawiki)
|
|
|-
|[https://kb.netgear.com/2649/NETGEAR-Open-Source-Code-for-Programmers-GPL NetGear GPL tarballs]
|Tarballs
|
|style="background-color: yellow"|NT
|
|
|
|
|-
|[https://www.ui.com/download/edgemax/edgerouter/default/edgerouter-er-8erpro-8ep-r8-firmware-v11011 Ubiquity firmware GPL tarballs]
|Tarballs
|
|style="background-color: yellow"|NT
|
|
|
|
|-
[[Category:Archive]]
[[Category:Suggestions]]

Google Summer of Code 2022

2022-04-18T17:23:16Z

Vlorentz: /* I want to participate as a student */ Update link

[[File:GSoCLogo.png|512px]]

== General information ==

This page is the central point of information for [[Software Heritage]] participation into the [https://summerofcode.withgoogle.com/ Google Summer of Code] program in 2022.

Google Summer of Code is a program where Google pays students stipends to work over the (northern hemisphere) summer on free software projects such as Software Heritage. Each student works with mentors from the community to complete a software project.

== I want to participate as a student ==

Great!, we are very glad for your interest in contributing to Software Heritage and we are looking forward to work together.

=== Prerequisites ===

The following prerequisites apply to all Software Heritage GSoC projects:

* [https://www.python.org Python 3] is our language of choice, you should be fluent with that language to apply
* [https://git-scm.com Git] is our version control system of choice, you should be familiar with it to apply
* basic knowledge in using a CLI
* additional prerequisites depend on the project you will work on; check project descriptions for details

=== Before you apply ===

Here are the steps you should follow before applying, to make sure you have a good grasp of what we are doing at Software Heritage and how we do it:

# Follow our [https://docs.softwareheritage.org/devel/developer-setup.html developer setup tutorial]: it will make sure you have the source code of our software stack locally available and that you can run unit tests
# Create an account on our [https://forge.softwareheritage.org development forge]
# Familiarize yourself with our [https://docs.softwareheritage.org/devel/contributing/phabricator.html? code review workflow]
# Make at least one simple change to any one of our [https://docs.softwareheritage.org/devel/ software components] and submit it as a [https://forge.softwareheritage.org/differential/ diff] for code review, following the above workflow (no need to ask for permissions). [[Easy hacks]] and [https://forge.softwareheritage.org/project/view/20/ Web UI] issues are good options for what to fix, but feel free to submit any patch you think it might be useful.

=== What to include in your application ===

Make sure that your application includes the following information:

* Describe the '''specific project''' you want to work on. What do you want to achieve? Why is it important? Why is it useful for Software Heritage? The project might be one of the project ideas that we have prepared below, or something else entirely that you want to contribute to Software Heritage. Your source code archival pet peeve, surprise us!
* Detail your '''work plan''': a brief description of how you plan to go about your project, including a list of ''deliverables'' and a ''timeline'' of when do you expect them to be available.
* Include a reference to '''the diff''' you submitted before applying (see the "Before you apply" section above).

== Ideas list ==

Below you can find a list of project ideas that are good options for a reasonably-sized GSoC project (check individual idea pages for expected duration and difficulty of each task):
<DynamicPageList>
category = Available GSoC task
ordermethod = sortkey
order = ascending
</DynamicPageList>

We also have a list of internship topics below, which you can use for ideas when applying to GSoC with us.
Expect each internship topic to require 350 hours and to be on the harder side than GSoC-specific tasks.

<DynamicPageList>
category = Available internship
ordermethod = sortkey
order = ascending
</DynamicPageList>

All project ideas above are just suggestions, don't feel obliged to pick one of them if there is nothing that fits your taste and abilities.
Feel free to propose something else that you are excited about and that contributes to improve the Software Heritage archive: we will be happy to consider it!

== Contact ==

GSoC students are encouraged to get in touch with the Software Heritage community using the standard development communication channels, and in particular our [[IRC]] channel (#swh-devel on [https://libera.chat/ Libera Chat]) and mailing list (swh-devel). Prefer public channels over contacting mentors directly.

See our [https://www.softwareheritage.org/community/developers/ development information page] for details.

== Timeline ==

See the official [https://developers.google.com/open-source/gsoc/timeline Google Summer of Code timeline].

[[Category:Google Summer of Code]]
[[Category:Google Summer of Code 2022]]

Add sources to the project search engine (GSoC task)

2022-04-02T06:33:48Z

Vlorentz: /* Task description */ Add example of data sources

== Introduction ==

The [https://archive.softwareheritage.org/ homepage of the Software Heritage archive]
features a small search engine, that searched in project URLs and project metadata.
Project metadata includes name, description, authors, etc.

This is implemented by a Python service backed by an ElasticSearch database,
which contains one document for each project; each document containing metadata
mined from the project itself

== Task description ==

We would like to add more data sources to the ElasticSearch database;
typically sources that are not authoritative, but provide metadata of usually
good quality.
These sources include data received from [https://docs.softwareheritage.org/devel/swh-deposit/index.html deposit clients] like HAL and data we will archive from [https://swmath.org/ swMATH], [https://github.com/ GitHub.com], ...

This comes with the following challenges:

# there are multiple sources, and their contents must work together
# sources have different reliability, that should be taken into account when ranking search results

Therefore, this task will require making a plan to address these,
define a data model, and finally implement it in a backend.
It may involve some frontend work if necessary, to provide an interface for
these.

== Expected duration ==
175 or 350 hours, at your option (longer duration means you can handle more data sources). Difficulty: medium

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* ElasticSearch
* Experience with cross-referenced data mining would be appreciated

== Potential mentors ==

* Kumar Shivendu (KShivendu on [[IRC]])
* Valentin Lorentz (vlorentz on [[IRC]])
* Vincent Sellier (vsellier on [[IRC]])

== Other relevant (but independent) tasks ==

This task is only about adding data we already collected the existing Elasticsearch database;
you may also be interested in [[Mine information from archived content (GSoC task)]]
and [[Mine information from external sources (GSoC task)]] to fill this
database; but those are completely independent tasks.

This database only contains project URLs and metadata, not source code.
Source code search is more complex, but is available as an
[[Source code search engine prototype (internship)|internship topic]]

[[Category:GSoC task]]
[[Category:Available GSoC task]]

Source code search engine prototype (internship)

2022-03-16T11:12:37Z

Vlorentz: update numbers

{{Internship
|description=The current [https://archive.softwareheritage.org/browse/search/ archive search engine] supports searching archived projects (or "origins") via their URL or metadata.
We would like to extend search to also support searching within archived source code files, based on their textual content.
Indexing all source code files archived by Software Heritage (~500-600 TB) is a major undertaking in terms of indexing time and storage.
The goal of this internship is to design and implement a medium-scale prototype of such an index (covering, e.g., 0.1 to 1% of the archive) that will allow to evaluate the best indexing approach (e.g., which kind of index, tokenizer, etc.) as well as the time and resources that doing so will require (e.g., via extrapolation).
The first technology that will be tried is [https://www.elastic.co/ ElasticSearch], as it's already deployed for other Software Heritage needs, but depending on the candidate other search engines can also be tested in the context of the internship.

|skills=
* Python development
* database administration experience (SQL and/or NoSQL and/or document-oriented)

Will be considered a plus:
* experience with full-text or code search

|mentors=
* Valentin Lorentz (vlorentz on [[IRC]])
* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])
}}

[[Category:Available internship]]

Google Summer of Code 2022

2022-03-11T14:35:18Z

Vlorentz: /* Contact */

[[File:GSoCLogo.png|512px]]

== General information ==

This page is the central point of information for [[Software Heritage]] participation into the [https://summerofcode.withgoogle.com/ Google Summer of Code] program in 2022.

Google Summer of Code is a program where Google pays students stipends to work over the (northern hemisphere) summer on free software projects such as Software Heritage. Each student works with mentors from the community to complete a software project.

== I want to participate as a student ==

Great!, we are very glad for your interest in contributing to Software Heritage and we are looking forward to work together.

=== Prerequisites ===

The following prerequisites apply to all Software Heritage GSoC projects:

* [https://www.python.org Python 3] is our language of choice, you should be fluent with that language to apply
* [https://git-scm.com Git] is our version control system of choice, you should be familiar with it to apply
* basic knowledge in using a CLI
* additional prerequisites depend on the project you will work on; check project descriptions for details

=== Before you apply ===

Here are the steps you should follow before applying, to make sure you have a good grasp of what we are doing at Software Heritage and how we do it:

# Follow our [https://docs.softwareheritage.org/devel/developer-setup.html developer setup tutorial]: it will make sure you have the source code of our software stack locally available and that you can run unit tests
# Create an account on our [https://forge.softwareheritage.org development forge]
# Familiarize yourself with our [[Code review in Phabricator|code review workflow]]
# Make at least one simple change to any one of our [https://docs.softwareheritage.org/devel/ software components] and submit it as a [https://forge.softwareheritage.org/differential/ diff] for code review, following the above workflow. [[Easy hacks]] and [https://forge.softwareheritage.org/project/view/20/ Web UI] issues are good options for what to fix, but feel free to submit any patch you think it might be useful.

=== What to include in your application ===

Make sure that your application includes the following information:

* Describe the '''specific project''' you want to work on. What do you want to achieve? Why is it important? Why is it useful for Software Heritage? The project might be one of the project ideas that we have prepared below, or something else entirely that you want to contribute to Software Heritage. Your source code archival pet peeve, surprise us!
* Detail your '''work plan''': a brief description of how you plan to go about your project, including a list of ''deliverables'' and a ''timeline'' of when do you expect them to be available.
* Include a reference to '''the diff''' you submitted before applying (see the "Before you apply" section above).

== Ideas list ==

Below you can find a list of project ideas that are good options for a reasonably-sized GSoC project (check individual idea pages for expected duration and difficulty of each task):
<DynamicPageList>
category = Available GSoC task
ordermethod = sortkey
order = ascending
</DynamicPageList>

We also have a list of internship topics below, which you can use for ideas when applying to GSoC with us.
Expect each internship topic to require 350 hours and to be on the harder side than GSoC-specific tasks.

<DynamicPageList>
category = Available internship
ordermethod = sortkey
order = ascending
</DynamicPageList>

All project ideas above are just suggestions, don't feel obliged to pick one of them if there is nothing that fits your taste and abilities.
Feel free to propose something else that you are excited about and that contributes to improve the Software Heritage archive: we will be happy to consider it!

== Contact ==

GSoC students are encouraged to get in touch with the Software Heritage community using the standard development communication channels, and in particular our [[IRC]] channel (#swh-devel on [https://libera.chat/ Libera Chat]) and mailing list (swh-devel). Prefer public channels over contacting mentors directly.

See our [https://www.softwareheritage.org/community/developers/ development information page] for details.

== Timeline ==

See the official [https://developers.google.com/open-source/gsoc/timeline Google Summer of Code timeline].

[[Category:Google Summer of Code]]
[[Category:Google Summer of Code 2022]]

Add sources to the project search engine (GSoC task)

2022-03-10T14:45:44Z

Vlorentz: add vsellier as mentor

== Introduction ==

The [https://archive.softwareheritage.org/ homepage of the Software Heritage archive]
features a small search engine, that searched in project URLs and project metadata.
Project metadata includes name, description, authors, etc.

This is implemented by a Python service backed by an ElasticSearch database,
which contains one document for each project; each document containing metadata
mined from the project itself

== Task description ==

We would like to add more data sources to the ElasticSearch database;
typically sources that are not authoritative, but provide metadata of usually
good quality.

This comes with the following challenges:

1. there are multiple sources, and their contents must work together
2. sources have different reliability, that should be taken into account when ranking search results

Therefore, this task will require making a plan to address these,
define a data model, and finally implement it in a backend.
It may involve some frontend work if necessary, to provide an interface for
these.

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* ElasticSearch
* Experience with cross-referenced data mining would be appreciated

== Potential mentors ==

* Kumar Shivendu (KShivendu on [[IRC]])
* Valentin Lorentz (vlorentz on [[IRC]])
* Vincent Sellier (vsellier on [[IRC]])

== Other relevant (but independent) tasks ==

This task is only about adding data we already collected the existing Elasticsearch database;
you may also be interested in [[Mine information from archived content (GSoC task)]]
and [[Mine information from external sources (GSoC task)]] to fill this
database; but those are completely independent tasks.

This database only contains project URLs and metadata, not source code.
Source code search is more complex, but is available as an
[[Source code search engine prototype (internship)|internship topic]]

[[Category:GSoC task]]
[[Category:Available GSoC task]]

Add sources to the project search engine (GSoC task)

2022-03-10T13:55:44Z

Vlorentz:

== Introduction ==

The [https://archive.softwareheritage.org/ homepage of the Software Heritage archive]
features a small search engine, that searched in project URLs and project metadata.
Project metadata includes name, description, authors, etc.

This is implemented by a Python service backed by an ElasticSearch database,
which contains one document for each project; each document containing metadata
mined from the project itself

== Task description ==

We would like to add more data sources to the ElasticSearch database;
typically sources that are not authoritative, but provide metadata of usually
good quality.

This comes with the following challenges:

1. there are multiple sources, and their contents must work together
2. sources have different reliability, that should be taken into account when ranking search results

Therefore, this task will require making a plan to address these,
define a data model, and finally implement it in a backend.
It may involve some frontend work if necessary, to provide an interface for
these.

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* ElasticSearch
* Experience with cross-referenced data mining would be appreciated

== Potential mentors ==

* Valentin Lorentz (vlorentz on [[IRC]])
* Kumar Shivendu (KShivendu on [[IRC]])

== Other relevant (but independent) tasks ==

This task is only about adding data we already collected the existing Elasticsearch database;
you may also be interested in [[Mine information from archived content (GSoC task)]]
and [[Mine information from external sources (GSoC task)]] to fill this
database; but those are completely independent tasks.

This database only contains project URLs and metadata, not source code.
Source code search is more complex, but is available as an
[[Source code search engine prototype (internship)|internship topic]]

[[Category:GSoC task]]
[[Category:Available GSoC task]]

Add sources to the project search engine (GSoC task)

2022-03-10T13:54:57Z

Vlorentz: /* Task description */

== Introduction ==

The [https://archive.softwareheritage.org/ homepage of the Software Heritage archive]
features a small search engine, that searched in project URLs and project metadata.
Project metadata includes name, description, authors, etc.

This is implemented by a Python service backed by an ElasticSearch database,
which contains one document for each project; each document containing metadata
mined from the project itself

== Task description ==

We would like to add more data sources to the ElasticSearch database;
typically sources that are not authoritative, but provide metadata of usually
good quality.

This comes with the following challenges:

1. there are multiple sources, and their contents must work together
2. sources have different reliability, that should be taken into account when ranking search results

Therefore, this task will require making a plan to address these,
define a data model, and finally implement it in a backend.
It may involve some frontend work if necessary, to provide an interface for
these.

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* ElasticSearch
* Experience with cross-referenced data mining would be appreciated

== Potential mentors ==

* Valentin Lorentz (vlorentz on [[IRC]])
* Kumar Shivendu (KShivendu on [[IRC]])

== Other relevant (but independent) tasks ==

This task is only about adding data we already collected the existing Elasticsearch database;
you may also be interested in [[Mine information from archived content (GSoC task)]]
and [[Mine information from external sources (GSoC task)]] to fill this
database; but those are completely independent tasks.

This database only contains project URLs and metadata, not source code.
Source code search is more complex, but is available as an
[[Source code search engine prototype (internship)|internship topic]]

[[Category:GSoC task]]

Add sources to the project search engine (GSoC task)

2022-03-10T13:54:50Z

Vlorentz: Created page with "== Introduction == The [https://archive.softwareheritage.org/ homepage of the Software Heritage archive] features a small search engine, that searched in project URLs and pro..."

== Introduction ==

The [https://archive.softwareheritage.org/ homepage of the Software Heritage archive]
features a small search engine, that searched in project URLs and project metadata.
Project metadata includes name, description, authors, etc.

This is implemented by a Python service backed by an ElasticSearch database,
which contains one document for each project; each document containing metadata
mined from the project itself

== Task description ==

We would like to add more data sources to the ElasticSearch database;
typically sources that are not authoritative, but provide metadata of usually
good quality.

This comes with the following challenges:

1. there are multiple sources, and their contents must work together
2. sources have different reliability, that should be taken into account
when ranking search results

Therefore, this task will require making a plan to address these,
define a data model, and finally implement it in a backend.
It may involve some frontend work if necessary, to provide an interface for
these.

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* ElasticSearch
* Experience with cross-referenced data mining would be appreciated

== Potential mentors ==

* Valentin Lorentz (vlorentz on [[IRC]])
* Kumar Shivendu (KShivendu on [[IRC]])

== Other relevant (but independent) tasks ==

This task is only about adding data we already collected the existing Elasticsearch database;
you may also be interested in [[Mine information from archived content (GSoC task)]]
and [[Mine information from external sources (GSoC task)]] to fill this
database; but those are completely independent tasks.

This database only contains project URLs and metadata, not source code.
Source code search is more complex, but is available as an
[[Source code search engine prototype (internship)|internship topic]]

[[Category:GSoC task]]

Google Summer of Code 2022

2022-03-09T19:10:27Z

Vlorentz: /* Ideas list */ re-add the list of internship

[[File:GSoCLogo.png|512px]]

== General information ==

This page is the central point of information for [[Software Heritage]] participation into the [https://summerofcode.withgoogle.com/ Google Summer of Code] program in 2022.

Google Summer of Code is a program where Google pays students stipends to work over the (northern hemisphere) summer on free software projects such as Software Heritage. Each student works with mentors from the community to complete a software project.

== I want to participate as a student ==

Great!, we are very glad for your interest in contributing to Software Heritage and we are looking forward to work together.

=== Prerequisites ===

The following prerequisites apply to all Software Heritage GSoC projects:

* [https://www.python.org Python 3] is our language of choice, you should be fluent with that language to apply
* [https://git-scm.com Git] is our version control system of choice, you should be familiar with it to apply
* basic knowledge in using a CLI
* additional prerequisites depend on the project you will work on; check project descriptions for details

=== Before you apply ===

Here are the steps you should follow before applying, to make sure you have a good grasp of what we are doing at Software Heritage and how we do it:

# Follow our [https://docs.softwareheritage.org/devel/developer-setup.html developer setup tutorial]: it will make sure you have the source code of our software stack locally available and that you can run unit tests
# Create an account on our [https://forge.softwareheritage.org development forge]
# Familiarize yourself with our [[Code review in Phabricator|code review workflow]]
# Make at least one simple change to any one of our [https://docs.softwareheritage.org/devel/ software components] and submit it as a [https://forge.softwareheritage.org/differential/ diff] for code review, following the above workflow. [[Easy hacks]] and [https://forge.softwareheritage.org/project/view/20/ Web UI] issues are good options for what to fix, but feel free to submit any patch you think it might be useful.

=== What to include in your application ===

Make sure that your application includes the following information:

* Describe the '''specific project''' you want to work on. What do you want to achieve? Why is it important? Why is it useful for Software Heritage? The project might be one of the project ideas that we have prepared below, or something else entirely that you want to contribute to Software Heritage. Your source code archival pet peeve, surprise us!
* Detail your '''work plan''': a brief description of how you plan to go about your project, including a list of ''deliverables'' and a ''timeline'' of when do you expect them to be available.
* Include a reference to '''the diff''' you submitted before applying (see the "Before you apply" section above).

== Ideas list ==

Below you can find a list of project ideas that are good options for a reasonably-sized GSoC project (check individual idea pages for expected duration and difficulty of each task):
<DynamicPageList>
category = Available GSoC task
ordermethod = sortkey
order = ascending
</DynamicPageList>

We also have a list of internship topics below, which you can use for ideas when applying to GSoC with us.
Expect each internship topic to require 350 hours and to be on the harder side than GSoC-specific tasks.

<DynamicPageList>
category = Available internship
ordermethod = sortkey
order = ascending
</DynamicPageList>

All project ideas above are just suggestions, don't feel obliged to pick one of them if there is nothing that fits your taste and abilities.
Feel free to propose something else that you are excited about and that contributes to improve the Software Heritage archive: we will be happy to consider it!

== Contact ==

GSoC students are encouraged to get in touch with the Software Heritage community using the standard development communication channels, and in particular our [[IRC]] channel (#swh-devel on [https://libera.chat/ Libera Chat]) and mailing list (swh-devel).

See our [https://www.softwareheritage.org/community/developers/ development information page] for details.

== Timeline ==

See the official [https://developers.google.com/open-source/gsoc/timeline Google Summer of Code timeline].

[[Category:Google Summer of Code]]
[[Category:Google Summer of Code 2022]]

Create a browser extension (GSoC task)

2022-03-07T20:01:22Z

Vlorentz: /* Expected duration */

== Introduction ==

As you probably know already, The Software Heritage archive can be
[https://archive.softwareheritage.org browsed on the Web]. The
[https://forge.softwareheritage.org/source/swh-web/ code] powering that
interface is a Django application that also implements a
[https://archive.softwareheritage.org/api/ Web API].

== Task description ==

When browsing a repository (on GitHub, Gitlab, ...) or a package description (on NPM, Debian),
people may want to check when (and if) this repository or package was last archived in Software Heritage.
Currently, this means opening the archive in a new tab, and searching for the URL, and looking at the status.
Then, they can trigger a new archival with another few clicks (via the "Save Code Now" feature).

This workflow may be streamlined by the [https://forge.softwareheritage.org/T3756 creation of a browser extension or bookmarklet].
This extension/bookmarklet would, for example, show an icon next to the URL bar to show the status of the currently visited repository;
and clicking it would show details (like the date of last visit) and run "Save Code Now" in just two clicks.

=== Expected duration ===

175 hours. Difficulty: medium

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Javascript experience is also needed for this project
* Prior experience in working with browser extensions is a plus

== Potential mentors ==

* Antoine Lambert (anlambert on [[IRC]])
* Valentin Lorentz (vlorentz on [[IRC]])
* Jayesh Velayudhan (jayeshv on [[IRC]])

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Google Summer of Code 2022

2022-02-25T10:06:41Z

Vlorentz: /* Ideas list */ Replace the internship list with a link, as we can't provide the duration+difficulty information in GSoC terms directly on internship pages

<div style="text-align: center; font-size: 1.2em; border: solid 1px black; padding: 1em;">
This page is a work in progress; the list of retained organizations for Google Summer of Code 2022 has not been determined yet.
</div>

[[File:GSoCLogo.png|512px]]

== General information ==

This page is the central point of information for [[Software Heritage]] participation into the [https://summerofcode.withgoogle.com/ Google Summer of Code] program in 2022.

Google Summer of Code is a program where Google pays students stipends to work over the (northern hemisphere) summer on free software projects such as Software Heritage. Each student works with mentors from the community to complete a software project.

== I want to participate as a student ==

Great!, we are very glad for your interest in contributing to Software Heritage and we are looking forward to work together.

=== Prerequisites ===

The following prerequisites apply to all Software Heritage GSoC projects:

* [https://www.python.org Python 3] is our language of choice, you should be fluent with that language to apply
* [https://git-scm.com Git] is our version control system of choice, you should be familiar with it to apply
* basic knowledge in using a CLI
* additional prerequisites depend on the project you will work on; check project descriptions for details

=== Before you apply ===

Here are the steps you should follow before applying, to make sure you have a good grasp of what we are doing at Software Heritage and how we do it:

# Follow our [https://docs.softwareheritage.org/devel/developer-setup.html developer setup tutorial]: it will make sure you have the source code of our software stack locally available and that you can run unit tests
# Create an account on our [https://forge.softwareheritage.org development forge]
# Familiarize yourself with our [[Code review in Phabricator|code review workflow]]
# Make at least one simple change to any one of our [https://docs.softwareheritage.org/devel/ software components] and submit it as a [https://forge.softwareheritage.org/differential/ diff] for code review, following the above workflow. [[Easy hacks]] and [https://forge.softwareheritage.org/project/view/20/ Web UI] issues are good options for what to fix, but feel free to submit any patch you think it might be useful.

=== What to include in your application ===

Make sure that your application includes the following information:

* Describe the '''specific project''' you want to work on. What do you want to achieve? Why is it important? Why is it useful for Software Heritage? The project might be one of the project ideas that we have prepared below, or something else entirely that you want to contribute to Software Heritage. Your source code archival pet peeve, surprise us!
* Detail your '''work plan''': a brief description of how you plan to go about your project, including a list of ''deliverables'' and a ''timeline'' of when do you expect them to be available.
* Include a reference to '''the diff''' you submitted before applying (see the "Before you apply" section above).

== Ideas list ==

Below you can find a list of project ideas that are good options for a reasonably-sized GSoC project.
They include both GSoC-specific tasks (that are only available as part of our GSoC participation)

<DynamicPageList>
category = Available GSoC task
ordermethod = sortkey
order = ascending
</DynamicPageList>

We also have a [[Internships|list of internship topics]], which you can use for ideas when applying to GSoC with us. Expect them to be on the harder than usual GSoC tasks, and they will require 350 hours.

All project ideas above are just suggestions, don't feel obliged to pick one of them if there is nothing that fits your taste and abilities.
Feel free to propose something else that you are excited about and that contributes to improve the Software Heritage archive: we will be happy to consider it!

== Contact ==

GSoC students are encouraged to get in touch with the Software Heritage community using the standard development communication channels, and in particular our [[IRC]] channel (#swh-devel on [https://libera.chat/ Libera Chat]) and mailing list (swh-devel).

See our [https://www.softwareheritage.org/community/developers/ development information page] for details.

== Timeline ==

See the official [https://developers.google.com/open-source/gsoc/timeline Google Summer of Code timeline].

[[Category:Google Summer of Code]]
[[Category:Google Summer of Code 2022]]

Mine information from external sources (GSoC task)

2022-02-25T07:18:23Z

Vlorentz: Add missing required info

== Introduction ==

In addition to archiving source code artifacts, Software Heritage is interested in
archive metadata from external sources and correlate it to source code artifacts.
This is also to enable semantic searches on the archive and scientific research.

Collecting this extrinsic metadata is a
[https://forge.softwareheritage.org/T1739 work in progress], and you are welcome
to contribute to its implementation.

== Task description ==

You would contribute to the design of our metadata-fetching architecture.
This includes:

* Review what metadata we want to fetch
* How to efficiently fetch it at regular intervals and store it
* Implement metadata fetching from at least one source, in a way that can be generalized to other sources

Expected duration: 175 or 350 hours, at your option (longer duration means you can tackle more formats). Difficulty: hard

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Prior experience in working with software metadata is a plus, but not required

== Potential mentors ==

* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])
* Valentin Lorentz (vlorentz on [[IRC]])

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Mine information from archived content (GSoC task)

2022-02-25T07:17:56Z

Vlorentz: Add missing required info

== Introduction ==

In addition to archival, Software Heritage indexes the retrieved source code
artifacts, to enable semantic searches on the archive and scientific research.

Indexing can happen at the individual file-level (e.g., detect the programming
language the file is written in or the license declared in its header), or at
more coarse grained granularity (e.g., what metadata are declared for the most
recently archived version of a given project).

A number of indexes are [https://forge.softwareheritage.org/source/swh-indexer/ currently supported],
such as:

* file level mining:
** MIME type detection (using libmagic)
** license detection (using FOSSology/nomossa)
** language detection (using Pygments)
** ctags extraction (using universal-ctags)
* project level mining:
** Ruby gemspec metadata
** Python PKG-INFO metadata
** Maven pom.xml metadata
** NPM package.json metadata

== Task description ==

Writing additional indexers that extract more information from archived source
code is welcome and would constitute a suitable GSoC project.

Name the kind of data mining you want to do!

For inspiration you can have a look at [https://libraries.io Libraries.io], as
most package formats/package managers support dedicated ways of expressing
metadata and we only support a small number of them up-to-now. But do not
restrict your ambition to those, any kind of data extraction/mining you want to
do on the archive could work.

You may also add support for multiple formats at once, using an external tool,
such as [https://github.com/datacite/bolognese Bolognese] or
[https://github.com/librariesio/bibliothecary/ bibliothecary].

Expected duration: 175 or 350 hours, at your option (longer duration means you can tackle more formats). Difficulty: medium

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Prior experience in working with (source code) metadata is a plus, but not required

== Potential mentors ==

* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])
* Valentin Lorentz (vlorentz on [[IRC]])

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Make the software deposit service (swh-deposit) modular (GSoC task)

2022-02-25T07:17:04Z

Vlorentz: /* Task description */

== Introduction ==

The Software Heritage archive is the most comprehensive open data knowledge base about source code that has been published openly.

In addition to fetching source code from public repositories, it offers to Deposit service, to allow platforms to send code for Software Heritage to archive.

This service is currently written as a monolith, that grew over years to include a complete [http://swordapp.github.io/SWORDv2-Profile/SWORDProfile.html SWORDv2] server, a partial SWORDv2 client with extensions, and business logic specific to Software Heritage in both. This makes the current code hard to maintain and impossible to reuse.

== Task description ==

[https://forge.softwareheritage.org/source/swh-deposit/ swh-deposit] would need to be split into the following parts:

* a generic SWORDv2 server (based on Django)
* a generic SWORDv2 client
* server-side business logic (currently implemented mostly in swh/deposit/api/common.py, but is tightly coupled with the views)
* client-side business logic

The generic server and client will need to be extensively documented, so they can be reused by other software projects.

Possible extensions include:

* The code should also be designed to allow extensions to support SWORDv3, if we ever need to support it
* A new administration front-end and/or addition of administrative tools in [https://forge.softwareheritage.org/source/swh-web/ swh-web]

Expected duration: 175 hours. Difficulty: medium

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Basic understanding of the Software Heritage [https://docs.softwareheritage.org/devel/swh-model/data-model.html data model] and of [https://docs.softwareheritage.org/devel/swh-model/persistent-identifiers.html SWHID identifiers]
* Experience with Django

== Potential mentors ==

* Antoine Dumont <ardumont@softwareheritage.org> (ardumont on [[IRC]])
* Valentin Lorentz <vlorentz@softwareheritage.org> (vlorentz on [[IRC]])

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Improve and extend the archive Web UI (GSoC task)

2022-02-25T07:16:32Z

Vlorentz: Add missing required info

== Introduction ==

As you probably know already, The Software Heritage archive can be
[https://archive.softwareheritage.org browsed on the Web]. The
[https://forge.softwareheritage.org/source/swh-web/ code] powering that
interface is a Django application that also implements a
[https://archive.softwareheritage.org/api/ Web API].

== Task description ==

Several improvements are possible on the archive Web interface and would make
great GSoC projects, some ideas to whet your appetite:

* add developer-oriented features, e.g., source file history, blame/praise interface, in-browser edit (with patch download), ... (note that this will also require backend design and implementation)
* improve [https://www.w3.org/WAI/ accessibility]
* display metadata we already mined from the archive

Expected duration:

* if you choose the developer-oriented features: 350 hours. Difficulty: hard
* for others mentioned above: 175 hours. Difficulty: easy

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Django
* web development and/or design
* Javascript knowledge is useful but not required

== Potential mentors ==

* Antoine Lambert (anlambert on [[IRC]])
* Valentin Lorentz (vlorentz on [[IRC]])
* Jayesh Velayudhan (jayeshv on [[IRC]])

== Other relevant (but independent) tasks ==

[[Improve project search engine (GSoC task)]] is an independent task,
which may or may not involve improvements to the Web UI, depending on
you tastes.

While this task can be about displaying metadata we already mined,
you may also be interested in [[Mine information from archived content (GSoC task)]]
and [[Mine information from external sources (GSoC task)]] to mine more
of this metadata; but those are completely independent tasks.

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Dashboard UI for the Code Scanner (GSoC task)

2022-02-25T07:14:37Z

Vlorentz: /* Task description */

== Introduction ==

The Software Heritage archive is the most comprehensive open data knowledge base about source code that has been published openly.

As such, it can be used to scan local source code bases to detect which parts of it come from public code, including Free and Open Source Software.

The Software Heritage scanner (<code>swh-scanner</code>) ([https://docs.softwareheritage.org/devel/swh-scanner/ documentation], [https://forge.softwareheritage.org/source/swh-scanner/ code], [https://upsilon.cc/~zack/talks/2021/2021-04-07-llw.pdf slides of a 2021 presentation about swh-scanner]) is a command line tool that enables doing that.

== Task description ==

<code>swh-scanner</code> is currently an experimental tool, which works well in practice, but need a real '''dashboard user interface''' to be useful.
Several output options are currently available when invoking the <code>swh scanner scan</code> command, in particular batch output in textual and JSON format, and an interactive
dashboard (with the <code>-i/--interactive</code>) option.

The interactive view currently works by producing a local HTML file and opening it using the local browser.
The goal of this project is to '''improve the interactive view''', making it a serious dashboard-style UI to peruse scanning results.

The following improvements are suggested, although more can be proposed (and even more could be discovered during the project work):

* Technology: generating a local HTML file is not necessarily the best way to render results, alternative solutions should be explored, including a self-hosted web app, rendering results with state-of-the-art frontend web frameworks (css/html/javascript)
* Scalability: currently rendering doesn't work when scanning large code bases such as the Linux kernel, rendering should be made lazy, by only loading data to show when needed
* Functionality: dashboard rendering should be integrated with the possibility of opening the local source code files that have been scanned, e.g., users will want to be able to open in-browser files that have been detected as known/unknown, in order to figure why
* Functionality: in the future additional information will be added to scanning results, including license and provenance information. While not yet available right now due to backend limitations, the proposed UI should plan ahead about how/where to display such information
* Paper cuts: [https://forge.softwareheritage.org/tag/code_scanner/ various issues] affect the usability of swh-scanner, improving them would be welcome as part of this project

Expected duration: 350 hours. Difficulty: medium

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Basic understanding of the Software Heritage [https://docs.softwareheritage.org/devel/swh-model/data-model.html data model] and of [https://docs.softwareheritage.org/devel/swh-model/persistent-identifiers.html SWHID identifiers]
* HTML/CSS/JavaScript and web development in general
* Working knowledge of UI/UX design principles

== Potential mentors ==

* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Create embeddable widgets (GSoC task)

2022-02-25T07:14:22Z

Vlorentz: Add missing required info

== Introduction ==

Create embeddable JS widgets to make SWH features easily available.

== Task description ==

The idea is to have a set of JS widgets that can be embedded in any web page with JS enabled.
Widgets will be smart enough to make their own api calls and render the results.

Some widgets could be
* SWH search box
* SWH Search results
* SWH Browse (for Project/revision or file)
* Save code now.

Expected duration: 350 hours. Difficulty: hard

== Desirable skills ==
* Python 3 and Git are a must to work on any Software Heritage project
* web development and/or design
* Javascript

== Potential mentors ==
* Jayesh Velayudhan (jayeshv on [[IRC]])

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Create a browser extension (GSoC task)

2022-02-25T07:13:55Z

Vlorentz: Add missing required info

== Introduction ==

As you probably know already, The Software Heritage archive can be
[https://archive.softwareheritage.org browsed on the Web]. The
[https://forge.softwareheritage.org/source/swh-web/ code] powering that
interface is a Django application that also implements a
[https://archive.softwareheritage.org/api/ Web API].

== Task description ==

When browsing a repository (on GitHub, Gitlab, ...) or a package description (on NPM, Debian),
people may want to check when (and if) this repository or package was last archived in Software Heritage.
Currently, this means opening the archive in a new tab, and searching for the URL, and looking at the status.
Then, they can trigger a new archival with another few clicks (via the "Save Code Now" feature).

This workflow may be streamlined by the [https://forge.softwareheritage.org/T3756 creation of a browser extension or bookmarklet].
This extension/bookmarklet would, for example, show an icon next to the URL bar to show the status of the currently visited repository;
and clicking it would show details (like the date of last visit) and run "Save Code Now" in just two clicks.

Expected duration: 175 hours. Difficulty: easy

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Javascript experience is also needed for this project
* Prior experience in working with browser extensions is a plus

== Potential mentors ==

* Antoine Lambert (anlambert on [[IRC]])
* Valentin Lorentz (vlorentz on [[IRC]])
* Jayesh Velayudhan (jayeshv on [[IRC]])

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Create a browser extension (GSoC task)

2022-02-11T10:02:37Z

Vlorentz: link to the task

== Introduction ==

As you probably know already, The Software Heritage archive can be
[https://archive.softwareheritage.org browsed on the Web]. The
[https://forge.softwareheritage.org/source/swh-web/ code] powering that
interface is a Django application that also implements a
[https://archive.softwareheritage.org/api/ Web API].

== Task description ==

When browsing a repository (on GitHub, Gitlab, ...) or a package description (on NPM, Debian),
people may want to check when (and if) this repository or package was last archived in Software Heritage.
Currently, this means opening the archive in a new tab, and searching for the URL, and looking at the status.
Then, they can trigger a new archival with another few clicks (via the "Save Code Now" feature).

This workflow may be streamlined by the [https://forge.softwareheritage.org/T3756 creation of a browser extension or bookmarklet].
This extension/bookmarklet would, for example, show an icon next to the URL bar to show the status of the currently visited repository;
and clicking it would show details (like the date of last visit) and run "Save Code Now" in just two clicks.

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Javascript experience is also needed for this project
* Prior experience in working with browser extensions is a plus

== Potential mentors ==

* Antoine Lambert (anlambert on [[IRC]])
* Valentin Lorentz (vlorentz on [[IRC]])

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Make the software deposit service (swh-deposit) modular (GSoC task)

2022-02-10T16:12:45Z

Vlorentz:

== Introduction ==

The Software Heritage archive is the most comprehensive open data knowledge base about source code that has been published openly.

In addition to fetching source code from public repositories, it offers to Deposit service, to allow platforms to send code for Software Heritage to archive.

This service is currently written as a monolith, that grew over years to include a complete [http://swordapp.github.io/SWORDv2-Profile/SWORDProfile.html SWORDv2] server, a partial SWORDv2 client with extensions, and business logic specific to Software Heritage in both. This makes the current code hard to maintain and impossible to reuse.

== Task description ==

[https://forge.softwareheritage.org/source/swh-deposit/ swh-deposit] would need to be split into the following parts:

* a generic SWORDv2 server (based on Django)
* a generic SWORDv2 client
* server-side business logic (currently implemented mostly in swh/deposit/api/common.py, but is tightly coupled with the views)
* client-side business logic

The generic server and client will need to be extensively documented, so they can be reused by other software projects.

Possible extensions include:

* The code should also be designed to allow extensions to support SWORDv3, if we ever need to support it
* A new administration front-end and/or addition of administrative tools in [https://forge.softwareheritage.org/source/swh-web/ swh-web]

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Basic understanding of the Software Heritage [https://docs.softwareheritage.org/devel/swh-model/data-model.html data model] and of [https://docs.softwareheritage.org/devel/swh-model/persistent-identifiers.html SWHID identifiers]
* Experience with Django

== Potential mentors ==

* Antoine Dumont <ardumont@softwareheritage.org> (ardumont on [[IRC]])
* Valentin Lorentz <vlorentz@softwareheritage.org> (vlorentz on [[IRC]])

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Create a browser extension (GSoC task)

2022-02-10T16:07:25Z

Vlorentz: Created page with "== Introduction == As you probably know already, The Software Heritage archive can be [https://archive.softwareheritage.org browsed on the Web]. The [https://forge.softwarehe..."

== Introduction ==

As you probably know already, The Software Heritage archive can be
[https://archive.softwareheritage.org browsed on the Web]. The
[https://forge.softwareheritage.org/source/swh-web/ code] powering that
interface is a Django application that also implements a
[https://archive.softwareheritage.org/api/ Web API].

== Task description ==

When browsing a repository (on GitHub, Gitlab, ...) or a package description (on NPM, Debian),
people may want to check when (and if) this repository or package was last archived in Software Heritage.
Currently, this means opening the archive in a new tab, and searching for the URL, and looking at the status.
Then, they can trigger a new archival with another few clicks (via the "Save Code Now" feature).

This workflow may be streamlined by the creation of a browser extension or bookmarklet.
This extension/bookmarklet would, for example, show an icon next to the URL bar to show the status of the currently visited repository;
and clicking it would show details (like the date of last visit) and run "Save Code Now" in just two clicks.

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Javascript experience is also needed for this project
* Prior experience in working with browser extensions is a plus

== Potential mentors ==

* Antoine Lambert (anlambert on [[IRC]])
* Valentin Lorentz (vlorentz on [[IRC]])

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Make the software deposit service (swh-deposit) modular (GSoC task)

2022-02-10T15:58:18Z

Vlorentz: /* Task description */

== Introduction ==

The Software Heritage archive is the most comprehensive open data knowledge base about source code that has been published openly.

In addition to fetching source code from public repositories, it offers to Deposit service, to allow platforms to send code for Software Heritage to archive.

This service is currently written as a monolith, that grew over years to include a complete [http://swordapp.github.io/SWORDv2-Profile/SWORDProfile.html SWORDv2] server, a partial SWORDv2 client with extensions, and business logic specific to Software Heritage in both. This makes the current code hard to maintain and impossible to reuse.

== Task description ==

[https://forge.softwareheritage.org/source/swh-deposit/ swh-deposit] would need to be split into the following parts:

* a generic SWORDv2 server (based on Django)
* a generic SWORDv2 client
* server-side business logic
* client-side business logic

The generic server and client will need to be extensively documented, so they can be reused by other software projects.

Possible extensions include:

* The code should also be designed to allow extensions to support SWORDv3, if we ever need to support it
* A new administration front-end and/or addition of administrative tools in [https://forge.softwareheritage.org/source/swh-web/ swh-web]

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Basic understanding of the Software Heritage [https://docs.softwareheritage.org/devel/swh-model/data-model.html data model] and of [https://docs.softwareheritage.org/devel/swh-model/persistent-identifiers.html SWHID identifiers]
* Experience with Django

== Potential mentors ==

* Antoine Dumont <ardumont@softwareheritage.org> (ardumont on [[IRC]])
* Valentin Lorentz <vlorentz@softwareheritage.org> (vlorentz on [[IRC]])

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Make the software deposit service (swh-deposit) modular (GSoC task)

2022-02-10T15:48:44Z

Vlorentz: replace zack with ardumont as mentor

== Introduction ==

The Software Heritage archive is the most comprehensive open data knowledge base about source code that has been published openly.

In addition to fetching source code from public repositories, it offers to Deposit service, to allow platforms to send code for Software Heritage to archive.

This service is currently written as a monolith, that grew over years to include a complete [http://swordapp.github.io/SWORDv2-Profile/SWORDProfile.html SWORDv2] server, a partial SWORDv2 client with extensions, and business logic specific to Software Heritage in both. This makes the current code hard to maintain and impossible to reuse.

== Task description ==

[https://forge.softwareheritage.org/source/swh-deposit/ swh-deposit] would need to be split into the following parts:

* a generic SWORDv2 server (based on Django)
* a generic SWORDv2 client
* server-side business logic
* client-side business logic

The generic server and client will need to be extensively documented, so they can be reused by other software projects.

Stretch goals include:

* The code should also be designed to allow extensions to support SWORDv3, if we ever need to support it
* A new administration front-end and/or addition of administrative tools in [https://forge.softwareheritage.org/source/swh-web/ swh-web]

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Basic understanding of the Software Heritage [https://docs.softwareheritage.org/devel/swh-model/data-model.html data model] and of [https://docs.softwareheritage.org/devel/swh-model/persistent-identifiers.html SWHID identifiers]
* Experience with Django

== Potential mentors ==

* Antoine Dumont <ardumont@softwareheritage.org> (ardumont on [[IRC]])
* Valentin Lorentz <vlorentz@softwareheritage.org> (vlorentz on [[IRC]])

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Make the software deposit service (swh-deposit) modular (GSoC task)

2022-02-10T15:48:08Z

Vlorentz: Created page with "== Introduction == The Software Heritage archive is the most comprehensive open data knowledge base about source code that has been published openly. In addition to fetching..."

== Introduction ==

The Software Heritage archive is the most comprehensive open data knowledge base about source code that has been published openly.

In addition to fetching source code from public repositories, it offers to Deposit service, to allow platforms to send code for Software Heritage to archive.

This service is currently written as a monolith, that grew over years to include a complete [http://swordapp.github.io/SWORDv2-Profile/SWORDProfile.html SWORDv2] server, a partial SWORDv2 client with extensions, and business logic specific to Software Heritage in both. This makes the current code hard to maintain and impossible to reuse.

== Task description ==

[https://forge.softwareheritage.org/source/swh-deposit/ swh-deposit] would need to be split into the following parts:

* a generic SWORDv2 server (based on Django)
* a generic SWORDv2 client
* server-side business logic
* client-side business logic

The generic server and client will need to be extensively documented, so they can be reused by other software projects.

Stretch goals include:

* The code should also be designed to allow extensions to support SWORDv3, if we ever need to support it
* A new administration front-end and/or addition of administrative tools in [https://forge.softwareheritage.org/source/swh-web/ swh-web]

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Basic understanding of the Software Heritage [https://docs.softwareheritage.org/devel/swh-model/data-model.html data model] and of [https://docs.softwareheritage.org/devel/swh-model/persistent-identifiers.html SWHID identifiers]
* Experience with Django

== Potential mentors ==

* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])
* Valentin Lorentz <vlorentz@softwareheritage.org> (vlorentz on [[IRC]])

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Google Summer of Code 2022

2022-02-08T12:02:33Z

Vlorentz:

<div style="text-align: center; font-size: 1.2em; border: solid 1px black; padding: 1em;">
This page is a work in progress; Software Heritage did not apply to Google Summer of Code 2022 yet.
</div>

[[File:GSoCLogo.png|512px]]

== General information ==

This page is the central point of information for [[Software Heritage]] participation into the [https://summerofcode.withgoogle.com/ Google Summer of Code] program in 2022.

Google Summer of Code is a program where Google pays students stipends to work over the (northern hemisphere) summer on free software projects such as Software Heritage. Each student works with mentors from the community to complete a software project.

== I want to participate as a student ==

Great!, we are very glad for your interest in contributing to Software Heritage and we are looking forward to work together.

=== Prerequisites ===

The following prerequisites apply to all Software Heritage GSoC projects:

* [https://www.python.org Python 3] is our language of choice, you should be fluent with that language to apply
* [https://git-scm.com Git] is our version control system of choice, you should be familiar with it to apply
* basic knowledge in using a CLI
* additional prerequisites depend on the project you will work on; check project descriptions for details

=== Before you apply ===

Here are the steps you should follow before applying, to make sure you have a good grasp of what we are doing at Software Heritage and how we do it:

# Follow our [https://docs.softwareheritage.org/devel/developer-setup.html developer setup tutorial]: it will make sure you have the source code of our software stack locally available and that you can run unit tests
# Create an account on our [https://forge.softwareheritage.org development forge]
# Familiarize yourself with our [[Code review in Phabricator|code review workflow]]
# Make at least one simple change to any one of our [https://docs.softwareheritage.org/devel/ software components] and submit it as a [https://forge.softwareheritage.org/differential/ diff] for code review, following the above workflow. [[Easy hacks]] and [https://forge.softwareheritage.org/project/view/20/ Web UI] issues are good options for what to fix, but feel free to submit any patch you think it might be useful.

=== What to include in your application ===

Make sure that your application includes the following information:

* Describe the '''specific project''' you want to work on. What do you want to achieve? Why is it important? Why is it useful for Software Heritage? The project might be one of the project ideas that we have prepared below, or something else entirely that you want to contribute to Software Heritage. Your source code archival pet peeve, surprise us!
* Detail your '''work plan''': a brief description of how you plan to go about your project, including a list of ''deliverables'' and a ''timeline'' of when do you expect them to be available.
* Include a reference to '''the diff''' you submitted before applying (see the "Before you apply" section above).

== Ideas list ==

Below you can find a list of project ideas that are good options for a
reasonably-sized GSoC project:
<DynamicPageList>
category = Available GSoC task
ordermethod = sortkey
order = ascending
</DynamicPageList>

We also maintain the following list of [[Internships]].
They are usually reserved to on-site university students, but during GSoC they are also available as GSoC project ideas:
<DynamicPageList>
category = Available internship
ordermethod = sortkey
order = ascending
</DynamicPageList>

Both GSoC tasks and internship topics are just suggestion though, don't feel
obliged to pick one of them if there is nothing that fits your taste and
abilities. Feel free to propose something else that you are excited about and
that contributes to improve the Software Heritage archive: we will be happy to
consider it!

== Contact ==

GSoC students are encouraged to get in touch with the Software Heritage community using the standard development communication channels, and in particular our [[IRC]] channel (#swh-devel on [https://libera.chat/ Libera Chat]) and mailing list (swh-devel).

See our [https://www.softwareheritage.org/community/developers/ development information page] for details.

== Timeline ==

See the official [https://developers.google.com/open-source/gsoc/timeline Google Summer of Code timeline].

[[Category:Google Summer of Code]]
[[Category:Google Summer of Code 2022]]

Category:GSoC task

2022-02-08T12:01:15Z

Vlorentz:

Google Summer of Code tasks, past, present, and future.

See the [[Google_Summer_of_Code_2022|main page for GSoC 2022]]

See also: the [[:Category:Available_GSoC_task|list of available tasks]]

Category:GSoC task

2022-02-08T12:00:46Z

Vlorentz:

Google Summer of Code tasks, past, present, and future.

See the [[Google_Summer_of_Code_2022|main page for GSoC 2022]]

See also: the [[Category:Available_GSoC_task|list of available tasks]]

Category:Available GSoC task

2022-02-08T11:59:51Z

Vlorentz: Created page with "Google Summer of Code tasks See the main page for GSoC 2022"

Google Summer of Code tasks

See the [[Google_Summer_of_Code_2022|main page for GSoC 2022]]

Category:GSoC task

2022-02-08T11:59:29Z

Vlorentz:

Google Summer of Code tasks

See the [[Google_Summer_of_Code_2022|main page for GSoC 2022]]

Mine information from external sources (GSoC task)

2022-02-08T11:59:15Z

Vlorentz:

== Introduction ==

In addition to archiving source code artifacts, Software Heritage is interested in
archive metadata from external sources and correlate it to source code artifacts.
This is also to enable semantic searches on the archive and scientific research.

Collecting this extrinsic metadata is a
[https://forge.softwareheritage.org/T1739 work in progress], and you are welcome
to contribute to its implementation.

== Task description ==

You would contribute to the design of our metadata-fetching architecture.
This includes:

* Review what metadata we want to fetch
* How to efficiently fetch it at regular intervals and store it
* Implement metadata fetching from at least one source, in a way that can be generalized to other sources

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Prior experience in working with software metadata is a plus, but not required

== Potential mentors ==

* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])
* Valentin Lorentz (vlorentz on [[IRC]])

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Mine information from archived content (GSoC task)

2022-02-08T11:59:11Z

Vlorentz:

== Introduction ==

In addition to archival, Software Heritage indexes the retrieved source code
artifacts, to enable semantic searches on the archive and scientific research.

Indexing can happen at the individual file-level (e.g., detect the programming
language the file is written in or the license declared in its header), or at
more coarse grained granularity (e.g., what metadata are declared for the most
recently archived version of a given project).

A number of indexes are [https://forge.softwareheritage.org/source/swh-indexer/ currently supported],
such as:

* file level mining:
** MIME type detection (using libmagic)
** license detection (using FOSSology/nomossa)
** language detection (using Pygments)
** ctags extraction (using universal-ctags)
* project level mining:
** Ruby gemspec metadata
** Python PKG-INFO metadata
** Maven pom.xml metadata
** NPM package.json metadata

== Task description ==

Writing additional indexers that extract more information from archived source
code is welcome and would constitute a suitable GSoC project.

Name the kind of data mining you want to do!

For inspiration you can have a look at [https://libraries.io Libraries.io], as
most package formats/package managers support dedicated ways of expressing
metadata and we only support a small number of them up-to-now. But do not
restrict your ambition to those, any kind of data extraction/mining you want to
do on the archive could work.

You may also add support for multiple formats at once, using an external tool,
such as [https://github.com/datacite/bolognese Bolognese] or
[https://github.com/librariesio/bibliothecary/ bibliothecary].

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Prior experience in working with (source code) metadata is a plus, but not required

== Potential mentors ==

* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])
* Valentin Lorentz (vlorentz on [[IRC]])

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Improve the Code Scanner (GSoC task)

2022-02-08T11:59:06Z

Vlorentz:

== Introduction ==

The Software Heritage archive is the most comprehensive open data knowledge base about source code that has been published openly.

As such, it can be used to scan local source code bases to detect which parts of it come from public code, including Free and Open Source Software.

The Software Heritage scanner (<code>swh-scanner</code>) ([https://docs.softwareheritage.org/devel/swh-scanner/ documentation], [https://forge.softwareheritage.org/source/swh-scanner/ code]) is a command line tool that enables doing that.

== Task description ==

<code>swh-scanner</code> is currently an experimental tool, which works well in practice, but need some polishing to make it usable in production in real use cases.

Several improvements are possible:

* integrate provenance information in results (using [https://docs.softwareheritage.org/devel/swh-graph/ swh-graph] and/or [https://forge.softwareheritage.org/source/swh-provenance/ swh-provenance])
* make the different algorithms (see [https://forge.softwareheritage.org/source/swh-scanner/history/benchmark/ benchmark branch] in Git) used to query the backend user-selectable
* minimize the number of queries to the [https://archive.softwareheritage.org/api/1/known/doc/ /known API endpoint], in order to consume API rate limit less and be generally more efficient
* be adaptive in how the backend is queried, e.g., for code trees that contain less than 1000 files it is more efficient to just query all of them at once, without following the DAG structure (even if it is in theory a faster approach)
* improve the web-based dashboard view (<code>--interactive</code>), making it more user friendly
* add support for scanning from within repositories (e.g., if the code is in git, we can lookup the commit ID directly)
* add progress reporting during scanning, in particular for large code bases
* add on-disk caching, in particular for large code bases
* integrate into the generated output other information available from the archive, e.g., license information, metadata, provenance information, etc.
* general code improvements, including refactoring and deduplication w.r.t. the rest of the Software Heritage code base (see [https://forge.softwareheritage.org/tag/code_scanner/ open tasks])

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Basic understanding of the Software Heritage [https://docs.softwareheritage.org/devel/swh-model/data-model.html data model] and of [https://docs.softwareheritage.org/devel/swh-model/persistent-identifiers.html SWHID identifiers]
* JavaScript and front-end web development, if you want to work on the interactive dashboard

== Potential mentors ==

* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Improve and extend the archive Web UI (GSoC task)

2022-02-08T11:58:54Z

Vlorentz:

== Introduction ==

As you probably know already, The Software Heritage archive can be
[https://archive.softwareheritage.org browsed on the Web]. The
[https://forge.softwareheritage.org/source/swh-web/ code] powering that
interface is a Django application that also implements a
[https://archive.softwareheritage.org/api/ Web API].

== Task description ==

Several improvements are possible on the archive Web interface and would make
great GSoC projects, some ideas to whet your appetite:

* add developer-oriented features, e.g., source file history, blame/praise interface, in-browser edit (with patch download), ... (note that this will also require backend design and implementation)
* improve [https://www.w3.org/WAI/ accessibility]
* display metadata we already mined from the archive
* help us design and implement our [https://forge.softwareheritage.org/T1805 next API version]

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Django
* web development and/or design
* Javascript knowledge is useful but not required

== Potential mentors ==

* Antoine Lambert (anlambert on [[IRC]])
* Valentin Lorentz (vlorentz on [[IRC]])

== Other relevant (but independent) tasks ==

[[Improve project search engine (GSoC task)]] is an independent task,
which may or may not involve improvements to the Web UI, depending on
you tastes.

While this task can be about displaying metadata we already mined,
you may also be interested in [[Mine information from archived content (GSoC task)]]
and [[Mine information from external sources (GSoC task)]] to mine more
of this metadata; but those are completely independent tasks.

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Google Summer of Code 2022

2022-02-08T11:58:29Z

Vlorentz: Created page with "512px == General information == This page is the central point of information for Software Heritage participation into the [https://summerofcode.wi..."

[[File:GSoCLogo.png|512px]]

== General information ==

This page is the central point of information for [[Software Heritage]] participation into the [https://summerofcode.withgoogle.com/ Google Summer of Code] program in 2022.

Google Summer of Code is a program where Google pays students stipends to work over the (northern hemisphere) summer on free software projects such as Software Heritage. Each student works with mentors from the community to complete a software project.

== I want to participate as a student ==

Great!, we are very glad for your interest in contributing to Software Heritage and we are looking forward to work together.

=== Prerequisites ===

The following prerequisites apply to all Software Heritage GSoC projects:

* [https://www.python.org Python 3] is our language of choice, you should be fluent with that language to apply
* [https://git-scm.com Git] is our version control system of choice, you should be familiar with it to apply
* basic knowledge in using a CLI
* additional prerequisites depend on the project you will work on; check project descriptions for details

=== Before you apply ===

Here are the steps you should follow before applying, to make sure you have a good grasp of what we are doing at Software Heritage and how we do it:

# Follow our [https://docs.softwareheritage.org/devel/developer-setup.html developer setup tutorial]: it will make sure you have the source code of our software stack locally available and that you can run unit tests
# Create an account on our [https://forge.softwareheritage.org development forge]
# Familiarize yourself with our [[Code review in Phabricator|code review workflow]]
# Make at least one simple change to any one of our [https://docs.softwareheritage.org/devel/ software components] and submit it as a [https://forge.softwareheritage.org/differential/ diff] for code review, following the above workflow. [[Easy hacks]] and [https://forge.softwareheritage.org/project/view/20/ Web UI] issues are good options for what to fix, but feel free to submit any patch you think it might be useful.

=== What to include in your application ===

Make sure that your application includes the following information:

* Describe the '''specific project''' you want to work on. What do you want to achieve? Why is it important? Why is it useful for Software Heritage? The project might be one of the project ideas that we have prepared below, or something else entirely that you want to contribute to Software Heritage. Your source code archival pet peeve, surprise us!
* Detail your '''work plan''': a brief description of how you plan to go about your project, including a list of ''deliverables'' and a ''timeline'' of when do you expect them to be available.
* Include a reference to '''the diff''' you submitted before applying (see the "Before you apply" section above).

== Ideas list ==

Below you can find a list of project ideas that are good options for a
reasonably-sized GSoC project:
<DynamicPageList>
category = Available GSoC task
ordermethod = sortkey
order = ascending
</DynamicPageList>

We also maintain the following list of [[Internships]].
They are usually reserved to on-site university students, but during GSoC they are also available as GSoC project ideas:
<DynamicPageList>
category = Available internship
ordermethod = sortkey
order = ascending
</DynamicPageList>

Both GSoC tasks and internship topics are just suggestion though, don't feel
obliged to pick one of them if there is nothing that fits your taste and
abilities. Feel free to propose something else that you are excited about and
that contributes to improve the Software Heritage archive: we will be happy to
consider it!

== Contact ==

GSoC students are encouraged to get in touch with the Software Heritage community using the standard development communication channels, and in particular our [[IRC]] channel (#swh-devel on [https://libera.chat/ Libera Chat]) and mailing list (swh-devel).

See our [https://www.softwareheritage.org/community/developers/ development information page] for details.

== Timeline ==

See the official [https://developers.google.com/open-source/gsoc/timeline Google Summer of Code timeline].

[[Category:Google Summer of Code]]
[[Category:Google Summer of Code 2022]]

Language and infrastructure for analyzing the archive (internship)

2021-05-07T11:42:18Z

Vlorentz: Fix broken link to Boa

(voir aussi: [[Language_and_infrastructure_for_analyzing_the_archive_(internship)/fr|version française]] du sujet)
----
{{Internship
|description=

The Software Heritage archive is structured as a graph (specifically, a
[https://en.wikipedia.org/wiki/Merkle_tree Merkle DAG]) and is huge: tens of
billion nodes, hundreds of billion edges. The graph exhibits a lot of sharing:
the same source code files and directories can be reached starting from many
different commits (e.g., different commits in the same repository), and the
same commits can be reached starting from many different repositories (e.g.,
repositories that are "forks" of one another).
When analyzing source code at a very large-scale (e.g, all the commits of the
same large repository, or even all projects hosted on GitHub) it is pointless,
and a waste of resources, to re-analyze source code artifacts already analyzed
in the past, and encountered again in the future due to sharing in the
graph.

The goal of this internship is to design and implement a prototype platform
(similar in spirit to [http://design.cs.iastate.edu/papers/ICSE-13/icse13.pdf Boa])
that allows to describe empirical experiments to be run on the Software
Heritage archive, exploiting artifact sharing as a way to speed up the
analysis. The platform will constitute of a simple language to describe
experiments (e.g., "start from these repositories and run this script on all
files in each commit") and of a runtime implementing the language that
transparently handles caching of previous results.
As a stretch goal: the runtime will delegate actual compute to multiple
workers, running either on a single machine or distributed over a cluster.

If successfully implemented, the internship will conclude with a demonstration
(e.g., in the form of a paper) benchmarking in practice the performance
advantages of the proposed approach over a naive implementation.

|skills=
* Python development
* experience with functional programming

Will be considered a plus:
* experience with programming language theory and implementation
* experience with the [https://en.wikipedia.org/wiki/MapReduce MapReduce] programming model

|mentors=
* Antoine Pietri (seirl on [[IRC]])
* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])
}}

[[Category:Ongoing internship]]

Mine information from external sources (GSoC task)

2021-04-12T09:34:30Z

Vlorentz: Removing "Possible metadata sources"; we want to review the list after the internship starts; and the wishlist should be in subtasks of https://forge.softwareheritage.org/T2202

== Introduction ==

In addition to archiving source code artifacts, Software Heritage is interested in
archive metadata from external sources and correlate it to source code artifacts.
This is also to enable semantic searches on the archive and scientific research.

Collecting this extrinsic metadata is a
[https://forge.softwareheritage.org/T1739 work in progress], and you are welcome
to contribute to its implementation.

== Task description ==

You would contribute to the design of our metadata-fetching architecture.
This includes:

* Review what metadata we want to fetch
* How to efficiently fetch it at regular intervals and store it
* Implement metadata fetching from at least one source, in a way that can be generalized to other sources

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Prior experience in working with software metadata is a plus, but not required

== Potential mentors ==

* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])
* Valentin Lorentz (vlorentz on [[IRC]])

[[Category:GSoC task]]

Matrix

2021-04-12T08:50:22Z

Vlorentz: /* IRC access list */ add command to op yourself

== IRC channels ==

The following channels have been registered on the [https://freenode.net/ Freenode] network for [[Software Heritage]] usage.

* [https://app.element.io/#/room/#freenode_#swh-devel:matrix.org '''#swh-devel''']: public development discussions
* [https://app.element.io/#/room/#freenode_#swh-team:matrix.org '''#swh-team''']: private discussions of the core team
* [https://app.element.io/#/room/#freenode_#swh-sysadm:matrix.org '''#swh-sysadm''']: operations team discussions/bots
* [https://app.element.io/#/room/#freenode_#softwareheritage:matrix.org '''#softwareheritage''']: general discussions about the project (currently unused)
* [https://app.element.io/#/room/#freenode_#swh:matrix.org '''#swh''']: ditto, in case we end up preferring the short version

If you use IRC, consider joining the channels.

If you don't use IRC ''directly'', you can still join our chat channels from your web browser via a [https://matrix.org/ Matrix] bridge by clicking on the channel names in the list above. You will be asked to create a [https://element.io/ Element] account if you don't have one yet.

== IRC authentication ==

You should register their nick with NickServ using:

/nick <USERNAME>
/msg nickserv register <PASSWORD> <EMAIL>

You will then receive an e-mail containing a link to activate you account. After doing so, you need to configure your client to auto-authenticate. The recommended way of doing that is using [https://freenode.net/kb/answer/sasl SASL authentication].

For Weechat:

/set irc.server.freenode.sasl_username <USERNAME>
/set irc.server.freenode.sasl_password <PASSWORD>

For matrix, the relevant docs is here: https://github.com/matrix-org/matrix-appservice-irc/wiki/End-user-FAQ#how-do-i-registeridentify-to-nickserv

Freenode also supports authentication via [https://freenode.net/kb/answer/certfp TLS client certificates].

=== Matrix bridge ===

For the Matrix bridge ([https://github.com/matrix-org/matrix-appservice-irc/wiki/End-user-FAQ#how-do-i-registeridentify-to-nickserv relevant docs here]), you first need to choose you IRC nick by sending this to the bridge appservice:

/msg @appservice-irc:matrix.org !nick <USERNAME>

Then send this command to NickServ to register your account:

/msg @freenode_NickServ:matrix.org register <PASSWORD> <EMAIL>

You will then receive an e-mail containing a link to activate you account. You can then identify using:

/msg @freenode_NickServ:matrix.org identify <PASSWORD>

Then, ask the IRC bridge appservice to remember your password so that you get identified automatically:

/msg @appservice-irc:matrix.org !storepass <USERNAME>:<PASSWORD>

== IRC access list ==

To auto-voice people with a registered nick (only doable by people with +fA access modes will be able to do it), add them to the channel access list:

/msg chanserv access #swh-devel add zack +V

If you already have the right (+o ChanServ flag), you can make yourself an operator, with:

/msg chanserv OP #swh-devel

[[Category:Infrastructure]]

Code review

2021-03-31T12:44:49Z

Vlorentz: Undo revision 1550 by Vlorentz (talk)

#REDIRECT [[swhdocs:devel/contributing/code-review.html#code-review]]

[[Category:Software development]]