Software Heritage Wiki - User contributions [en]

User:StefanoZacchiroli

2024-09-18T11:15:21Z

StefanoZacchiroli:

Heya, I'm Stefano Zacchiroli, or simply "Zack", and I'm [[Software Heritage]] [https://www.softwareheritage.org/people/ co-founder and CSO].

* Nickname: Zack
* Full name: Stefano Zacchiroli
* Homepage: http://upsilon.cc/~zack
* Mastodon: [https://mastodon.xyz/@zacchiro @zacchiro@mastodon.xyz]
* Email: [mailto:zack@upsilon.cc zack@upsilon.cc]
* GPG fingerprint: [http://pgp.cs.uu.nl/stats/6D866396.html 4900 707D DC5C 07F2 DECB 0283 9C31 503C 6D86 6396]

== Scratch ==

* [[/Content deduplication]]
* [[/Repository snapshot objects]]
* [[/Scratch]]

Software Heritage in a bottle - local repository mining toolchain (internship)

2024-09-13T13:41:56Z

StefanoZacchiroli:

{{Internship
|description=The goal of this internship is to develop a fully-automated "mini [https://www.softwareheritage.org Software Heritage]" pipeline, capable of crawling a (potentially large) set of Version Control System repositories and store all their information into a local deployment of Software Heritage software components.

This will allow analyzing all crawled data locally in an efficient manner. Some sample intended use cases for this are:

# mining of private (or hybrid) development forges, e.g., for [https://en.wikipedia.org/wiki/Inner_source inner source] industry scenarios
# testing and validation of mining software repository (MSR) analyses, e.g., for the research needs of the [https://swhsec.github.io/ SWHSec project]

'''Objectives:'''
The primary goal of this internship is to develop scripts and tools to generate a local, fully functional mini Software Heritage environment. This environment will mimic the larger SWH infrastructure and support complete export and import functionalities. It will be used to test algorithms and validate scenarios by applying various manipulations to software repositories.

'''Specific objectives:'''
# Scenarization: automate the creation and management of local Git repositories (for testing purposes).
# Deploy a local SWH ingestion pipeline
# Automate the initial, periodic, and on-demand (re)crawling of Git repositories
# Periodically and on-demand export all indexed data in the [https://docs.softwareheritage.org/devel/swh-dataset/graph/ same formats] exported by the SWH archive (compressed graph and ORC formats).
# Develop a local [https://docs.softwareheritage.org/devel/swh-objstorage/ SWH object storage] instance to allow accessing individual file contents.
# Automate all relevant workflows in a Continuous Integration (CI) environment.

'''Expected outcomes:'''
By the end of the internship, we expect the following deliverables:
# A set of scripts and tools to create local Git repositories, apply manipulations, index them, and export the data.
# A minimal local server setup that replicates the API functionalities of the SWH project.
# Automated test cases integrated into a CI system to ensure ongoing validation of different manipulation scenarios.
# Documentation and guidelines on using the developed tools and reproducing the test scenarios.

|skills=
* Strong proficiency in Python.
* Knowledge of Rust is a plus.
* Familiarity with Git and related tools.
* Basic understanding of software version control and indexing.
* Experience with API development and server setup.
* Ability to work independently and collaboratively in a team environment.

|workplace=on site at [https://www.telecom-paris.fr/en/home Télécom Paris] (contact mentors for alternative hosting options)

|environment=The intern will be supervised by members of the SWH and SWHSec project teams, which includes experts in software preservation, security, and data management.

|mentors=
* Samuel Tardieu <samuel.tardieu@telecom-paris.fr> (Sam on [[Matrix]])
* Stefano Zacchiroli <stefano.zacchiroli@telecom-paris.fr> (Zack on [[Matrix]])
}}

[[Category:Available internship]]

Software Heritage in a bottle - local repository mining toolchain (internship)

2024-09-13T13:39:13Z

StefanoZacchiroli:

{{Internship
|description=The goal of this internship is to develop a fully-automated "mini [https://www.softwareheritage.org Software Heritage]" pipeline, capable of crawling a (potentially large) set of Version Control System repositories and store all their information into a local deployment of Software Heritage software components.

This will allow analyzing all crawled data locally in an efficient manner. Some sample intended use cases for this are:

# mining of private (or hybrid) development forges, e.g., for [https://en.wikipedia.org/wiki/Inner_source inner source] industry scenarios
# testing and validation of mining software repository (MSR) analyses, e.g., for the research needs of the [https://swhsec.github.io/ SWHSec project]

'''Objectives:'''
The primary goal of this internship is to develop scripts and tools to generate a local, fully functional mini Software Heritage environment. This environment will mimic the larger SWH infrastructure and support complete export and import functionalities. It will be used to test algorithms and validate scenarios by applying various manipulations to software repositories.

'''Specific objectives:'''
# Scenarization: automate the creation and management of local Git repositories (for testing purposes).
# Deploy a local SWH ingestion pipeline
# Automate the initial, periodic, and on-demand (re)crawling of Git repositories
# Periodically and on-demand export all indexed data in the [https://docs.softwareheritage.org/devel/swh-dataset/graph/ same formats] exported by the SWH archive (compressed graph and ORC formats).
# Develop a local [https://docs.softwareheritage.org/devel/swh-objstorage/ SWH object storage] instance to allow accessing individual file contents.
# Automate all relevant workflows in a Continuous Integration (CI) environment.

'''Expected outcomes:'''
By the end of the internship, we expect the following deliverables:
# A set of scripts and tools to create local Git repositories, apply manipulations, index them, and export the data.
# A minimal local server setup that replicates the API functionalities of the SWH project.
# Automated test cases integrated into a CI system to ensure ongoing validation of different manipulation scenarios.
# Documentation and guidelines on using the developed tools and reproducing the test scenarios.

|skills=
* Strong proficiency in Python.
* Knowledge of Rust is a plus.
* Familiarity with Git and related tools.
* Basic understanding of software version control and indexing.
* Experience with API development and server setup.
* Ability to work independently and collaboratively in a team environment.

|workplace=on site at [https://www.telecom-paris.fr/en/home Télécom Paris] (contact mentors for remote opportunities)

|environment=The intern will be supervised by members of the SWH and SWHSec project teams, which includes experts in software preservation, security, and data management.

|mentors=
* Samuel Tardieu <samuel.tardieu@telecom-paris.fr> (Sam on [[Matrix]])
* Stefano Zacchiroli <stefano.zacchiroli@telecom-paris.fr> (Zack on [[Matrix]])
}}

[[Category:Available internship]]

Software Heritage in a bottle - local repository mining toolchain (internship)

2024-09-13T13:37:29Z

StefanoZacchiroli: StefanoZacchiroli moved page Software Heritage in a bottle (internship) to Software Heritage in a bottle - local repository mining toolchain (internship) without leaving a redirect

{{Internship
|description=The goal of this internship is to develop a fully-automated "mini Software Heritage" pipeline, capable of crawling a (potentially large) set of Version Control System repositories and store all their information into a local deployment of Software Heritage software components.

This will allow analyzing all crawled data locally in an efficient manner. Some sample intended use cases for this are:

# mining of private (or hybrid) development forges, e.g., for [https://en.wikipedia.org/wiki/Inner_source inner source] industry scenarios
# testing and validation of mining software repository (MSR) analyses, e.g., for the research needs of the [https://swhsec.github.io/ SWHSec project]

'''Objectives:'''
The primary goal of this internship is to develop scripts and tools to generate a local, fully functional mini Software Heritage environment. This environment will mimic the larger SWH infrastructure and support complete export and import functionalities. It will be used to test algorithms and validate scenarios by applying various manipulations to software repositories.

'''Specific objectives:'''
# Scenarization: automate the creation and management of local Git repositories (for testing purposes).
# Deploy a local SWH ingestion pipeline
# Automate the initial, periodic, and on-demand (re)crawling of Git repositories
# Periodically and on-demand export all indexed data in the same formats exported by the SWH archive (compressed graph and ORC formats).
# Develop a local SWH object storage instance to allow accessing individual file contents.
# Automate all relevant workflows in a Continuous Integration (CI) environment.

'''Expected outcomes:'''
By the end of the internship, we expect the following deliverables:
# A set of scripts and tools to create local Git repositories, apply manipulations, index them, and export the data.
# A minimal local server setup that replicates the API functionalities of the SWH project.
# Automated test cases integrated into a CI system to ensure ongoing validation of different manipulation scenarios.
# Documentation and guidelines on using the developed tools and reproducing the test scenarios.

|skills=
* Strong proficiency in Python.
* Knowledge of Rust is a plus.
* Familiarity with Git and related tools.
* Basic understanding of software version control and indexing.
* Experience with API development and server setup.
* Ability to work independently and collaboratively in a team environment.

|workplace=on site at [https://www.telecom-paris.fr/en/home Télécom Paris] (contact mentors for remote opportunities)

|environment=The intern will be supervised by members of the SWH and SWHSec project teams, which includes experts in software preservation, security, and data management.

|mentors=
* Samuel Tardieu <samuel.tardieu@telecom-paris.fr> (Sam on [[Matrix]])
* Stefano Zacchiroli <stefano.zacchiroli@telecom-paris.fr> (Zack on [[Matrix]])
}}

[[Category:Available internship]]

Software Heritage in a bottle - local repository mining toolchain (internship)

2024-09-13T13:36:37Z

StefanoZacchiroli:

Software Heritage in a bottle - local repository mining toolchain (internship)

2024-09-13T13:33:51Z

StefanoZacchiroli: Created page with "{{Internship |description=The goal of this internship is to develop a fully-automated "mini Software Heritage" pipeline, capable of crawling a (potentially large) set of Versi..."

{{Internship
|description=The goal of this internship is to develop a fully-automated "mini Software Heritage" pipeline, capable of crawling a (potentially large) set of Version Control System repositories and store all their information into a local deployment of Software Heritage software components.

This will allow analyzing all crawled data locally in an efficient manner. Some sample intended use cases for this are:

# mining of private (or hybrid) development forges, e.g., for [https://en.wikipedia.org/wiki/Inner_source inner source] industry scenarios
# testing and validation of mining software repository (MSR) analyses, e.g., for the research needs of the [https://swhsec.github.io/ SWHSec project]

'''Objectives:'''
The primary goal of this internship is to develop scripts and tools to generate a local, fully functional mini Software Heritage environment. This environment will mimic the larger SWH infrastructure and support complete export and import functionalities. It will be used to test algorithms and validate scenarios by applying various manipulations to software repositories.

'''Specific objectives:'''
# Scenarization: automate the creation and management of local Git repositories (for testing purposes).
# Deploy a local SWH ingestion pipeline
# Automate the initial, periodic, and on-demand (re)crawling of Git repositories
# Periodically and on-demand export all indexed data in the same formats exported by the SWH archive (compressed graph and ORC formats).
# Develop a local SWH object storage instance to allow accessing individual file contents.
# Automate all relevant workflows in a Continuous Integration (CI) environment.

'''Expected outcomes:'''
By the end of the internship, we expect the following deliverables:
# A set of scripts and tools to create local Git repositories, apply manipulations, index them, and export the data.
# A minimal local server setup that replicates the API functionalities of the SWH project.
# Automated test cases integrated into a CI system to ensure ongoing validation of different manipulation scenarios.
# Documentation and guidelines on using the developed tools and reproducing the test scenarios.

|skills=
* Strong proficiency in Python.
* Knowledge of Rust is a plus.
* Familiarity with Git and related tools.
* Basic understanding of software version control and indexing.
* Experience with API development and server setup.
* Ability to work independently and collaboratively in a team environment.

|mentors=
* Samuel Tardieu <samuel.tardieu@telecom-paris.fr> (Sam on [[Matrix]])
* Stefano Zacchiroli <stefano.zacchiroli@telecom-paris.fr> (Zack on [[Matrix]])
}}

[[Category:Available internship]]

Template:Internship context

2024-09-13T10:05:45Z

StefanoZacchiroli:

[https://www.softwareheritage.org/ Software Heritage] is an ambitious initiative whose goal is to collect, preserve forever, and make publicly available the entire body of software, in the preferred form for making modifications to it.

Source code search engine prototype (internship)

2024-02-04T15:11:59Z

StefanoZacchiroli:

{{Internship
|description=The current [https://archive.softwareheritage.org/browse/search/ archive search engine] supports searching archived projects (or "origins") via their URL or metadata.
We would like to extend search to also support searching within archived source code files, based on their textual content.
Indexing all source code files archived by Software Heritage (~500-600 TB) is a major undertaking in terms of indexing time and storage.
The goal of this internship is to design and implement a medium-scale prototype of such an index (covering, e.g., 0.1 to 1% of the archive) that will allow to evaluate the best indexing approach (e.g., which kind of index, tokenizer, etc.) as well as the time and resources that doing so will require (e.g., via extrapolation).
The first technology that will be tried is [https://www.elastic.co/ ElasticSearch], as it's already deployed for other Software Heritage needs, but depending on the candidate other search engines can also be tested in the context of the internship.

|skills=
* Python development
* database administration experience (SQL and/or NoSQL and/or document-oriented)

Will be considered a plus:
* experience with full-text or code search

|mentors=
* Valentin Lorentz (vlorentz on [[Matrix]])
* Stefano Zacchiroli <zack@upsilon.cc> (Zack on [[Matrix]])
}}

[[Category:Available internship]]

Reverse project phylogenesis (internship)

2024-02-04T15:11:37Z

StefanoZacchiroli:

{{Internship
|description=The [https://archive.softwareheritage.org/ Software Heritage Archive] contains a large number of projects (over one million!) salvaged from code hosting platforms that have been closed down, ranging from large ones like Google Code or Gitorious.org, to small institutional ones, like the old Inria gForge, and from platforms that have phased out support for some version control systems, like Bitbucket.
Some of these projects migrated to other platforms, where they continued their development.

This means that there are documents, research articles, blog posts, documentation, and many other sources out there that contain broken links: the Software Heritage archive provides a way to find easily the archived version of the project, but does not help identifying the new repository where the development may have migrated.

The goal of this internship is to explore heuristics that exploit the special feature of the Software Heritage merkle graph to identify repositories that may be the new development strand of an old repository saved from a discontinued platform, and show these links in the relevant repositories: this corresponds to developing the [https://www.vocabulary.com/dictionary/phylogenesis phylogenesis] of a software project.

One of the challenges will be to compare various heuristics and scale the approach up to the millions of repositories involved.

|skills=
* Python development
* understanding of version control systems (git in particular) and familiarity with code hosting platforms

|mentors=
* Roberto Di Cosmo <roberto@dicosmo.org> (rdicosmo on [[Matrix]])
}}

[[Category:Available internship]]

Large-scale license text recognition (internship)/fr

2024-02-04T15:11:03Z

StefanoZacchiroli:

(see: [[Large-scale_license_text_recognition_(internship)|english version]] of this internship description)
----
{{Internship
|description=Plusieurs solutions open source existent pour reconnaître la licence logicielle déclarée dans un fichier source, p.ex.: [https://www.fossology.org/ Fossology], [https://github.com/nexB/scancode-toolkit ScanCode], [http://ninka.turingmachine.org/ Ninka].
La pluspart des ces solutions sont basées sur des heuristiques qui ont été maintenues au fil des années sur la base des [https://spdx.org/licenses/ licences] (open source ou pas) existantes.
Seulement récemment les techniques d'apprentissage automatiques ont été appliquées au problème de reconnaissance des licences, avec des prototypes comme [https://github.com/fossology/FOSSologyML FOSSologyML].
Le but de ce stage est d'appliquer des techniques d'apprentissage automatique à un sous-problème spécifique de la reconnaissance des licences: classifier un texte complet de licence (et pas seulement une déclaration courte de licence, comme on peut le trouver dans les entêtes des fichiers source), comme on peut le trouver dans des fichiers comme <code>LICENSE</code>, <code>COPYING</code>, etc., à la racine des dépôts logiciels, et de le faire à l'échelle de Software Heritage.
Un jeu de données de ces fichiers sera extrait depuis l'archive, et on expérimentera avec plusieurs techniques d'apprentissage automatique pour identifier la méthode la plus efficace et performante.

|skills=
* savoir développer en Python
* une expérience avec l'apprentissage automatique
* connaissance de un ou plusieurs framework d'apprentissage automatique (p.ex., [https://keras.io/ Keras], [https://www.tensorflow.org/ TensorFlow], [https://scikit-learn.org/stable/ scikit-learn])

Est considéré comme un plus:
* une expérience avec la [https://fr.wikipedia.org/wiki/Traitement_automatique_des_langues traitement automatique des langues (TAL)]

|mentors=
* Stefano Zacchiroli <zack@upsilon.cc> (Zack on [[Matrix]])
}}

Large-scale license text recognition (internship)

2024-02-04T15:10:42Z

StefanoZacchiroli:

(voir aussi: [[Large-scale license text recognition (internship)/fr|version française]] du sujet)
----
{{Internship
|description=A number of free/open source software (FOSS) tools are available to automatically detect software licenses declared in source code files, e.g., [https://www.fossology.org/ Fossology], [https://github.com/nexB/scancode-toolkit ScanCode], [http://ninka.turingmachine.org/ Ninka].
Most of them rely on carefully maintained heuristics that have been tuned over many years to detect [https://spdx.org/licenses/ licenses] (FOSS or otherwise) that can be found in the wild.
Only relatively recently machine-learning techniques have been applied to the license-detection problem, in prototypes like [https://github.com/fossology/FOSSologyML FOSSologyML].
The goal of this internship is to apply machine-learning techniques to a limited sub-problem of license-detection, i.e., recognizing full license texts as they are commonly found in top-level files such as <code>LICENSE</code>, <code>COPYING</code>, etc., at the scale of Software Heritage.
''All'' such files will be extracted from the archive, and suitable machine learning models will be designed and tested on the obtained corpus.

|skills=
* Python development
* machine learning training and experience
* working knowledge of one or more machine learning frameworks (e.g., [https://keras.io/ Keras], [https://www.tensorflow.org/ TensorFlow], [https://scikit-learn.org/stable/ scikit-learn])

Will be considered a plus:
* [https://en.wikipedia.org/wiki/Natural_language_processing natural language processing (NLP)] training and experience

|mentors=
* Stefano Zacchiroli <zack@upsilon.cc> (Zack on [[Matrix]])
}}

[[Category:Available internship]]

Language and infrastructure for analyzing the archive (internship)/fr

2024-02-04T15:10:21Z

StefanoZacchiroli:

(see: [[Language_and_infrastructure_for_analyzing_the_archive_(internship)|english version]] of this internship description)
----
{{Internship
|description=

L'archive de Software Heritage est structurée comme un graphe (plus précisément, un arbre de Merkle), et ce
graphe est énorme :
des dizaines de milliards de noeuds, des centaines de milliards d'arêtes. Le graphe nous montre que de nombreux
commits différents nous mènent vers les mêmes fichiers et repertoires de fichiers. Et des mêmes commits peuvent
être atteins a partir de plusieurs dépôts (des dépôts qui sont des forks d'autres dépôts).

Lors de l'analyse de code source a très grande échelle (par exemple, tout les projets hébergés sur GitHub), le fait
qu'il existe une abondance des données partagées entraîne un gaspillage de ressource et de temps.

L'objectif de ce stage est de concevoir et de mettre en oeuvre un prototype de plateforme (inspiré de Boa) qui
nous permettrais de décrire des experiences empirique a lancer sur Software heritage, en exploitant les données
partagées dans le graphe pour accélérer les analyse.
La plateforme sera constituée d'un langage simple pour décrire les experiences, et d'un runtime implémentant le
langage qui gère de manière transparente la mise en cache des résultats précédents.
L'objectif est ambitieux : le runtime distribueras le calcul sur une seule machine ou un cluster.

Si l'implémentation est un succès, le stage se termineras avec une démonstration (sous la forme d'un article) des
performances, en pratique, de notre solution par rapport a une implémentation naive.

|skills=
* savoir développer en Python
* une experience avec la programmation fonctionnelle

Est considéré comme un plus:
* une experience avec la théorie/implémentation d'un langage de programmation
* une experience avec le modèle de calcul MapReduce

|mentors=
* [https://upsilon.cc/~zack Stefano Zacchiroli] (zack@upsilon.cc, Zack on [[Matrix]])
}}

Language and infrastructure for analyzing the archive (internship)

2024-02-04T15:09:57Z

StefanoZacchiroli:

(voir aussi: [[Language_and_infrastructure_for_analyzing_the_archive_(internship)/fr|version française]] du sujet)
----
{{Internship
|description=

The Software Heritage archive is structured as a graph (specifically, a
[https://en.wikipedia.org/wiki/Merkle_tree Merkle DAG]) and is huge: tens of
billion nodes, hundreds of billion edges. The graph exhibits a lot of sharing:
the same source code files and directories can be reached starting from many
different commits (e.g., different commits in the same repository), and the
same commits can be reached starting from many different repositories (e.g.,
repositories that are "forks" of one another).
When analyzing source code at a very large-scale (e.g, all the commits of the
same large repository, or even all projects hosted on GitHub) it is pointless,
and a waste of resources, to re-analyze source code artifacts already analyzed
in the past, and encountered again in the future due to sharing in the
graph.

The goal of this internship is to design and implement a prototype platform
(similar in spirit to [http://design.cs.iastate.edu/papers/ICSE-13/icse13.pdf Boa])
that allows to describe empirical experiments to be run on the Software
Heritage archive, exploiting artifact sharing as a way to speed up the
analysis. The platform will constitute of a simple language to describe
experiments (e.g., "start from these repositories and run this script on all
files in each commit") and of a runtime implementing the language that
transparently handles caching of previous results.
As a stretch goal: the runtime will delegate actual compute to multiple
workers, running either on a single machine or distributed over a cluster.

If successfully implemented, the internship will conclude with a demonstration
(e.g., in the form of a paper) benchmarking in practice the performance
advantages of the proposed approach over a naive implementation.

|skills=
* Python development
* experience with functional programming

Will be considered a plus:
* experience with programming language theory and implementation
* experience with the [https://en.wikipedia.org/wiki/MapReduce MapReduce] programming model

|mentors=
* Stefano Zacchiroli <zack@upsilon.cc> (Zack on [[Matrix]])
}}

[[Category:Available internship]]

Integrate Software Heritage and ClearlyDefined (internship)

2024-02-04T15:09:31Z

StefanoZacchiroli:

{{Internship
|description=[https://clearlydefined.io/ ClearlyDefined] is a project whose goal is to collaboratively and semi-automatically curate information about Free/Open Source Software (FOSS) projects, including licensing and vulnerability information.
As one of its main output, ClearyDefined maintains an open data knowledge-base that cross references FOSS source code artifacts found in version control systems, package repositories, etc. to curated information about their licenses and vulnerabilities. The same source code artifacts are archived by Software Heritage for long-term preservation purposes.
The goal of this internship is to integrate ClearlyDefined and Software Heritage, for mutual benefit.
Software Heritage will benefit from mirroring ClearlyDefined data, allowing to query them while navigating the archive and at scale; ClearlyDefined will benefit from learning about the existing of FOSS projects that have not been analyzed for "clarity" yet.

|skills=
* JavaScript / NodeJS
* Python
* experience with database management systems (of any kind)

|mentors=
* Valentin Lorentz
* Stefano Zacchiroli <zack@upsilon.cc> (Zack on [[Matrix]])

|environment=
you will work shoulder to shoulder with members of the [https://www.softwareheritage.org/people/ Software Heritage] and [https://clearlydefined.io/about ClearlyDefined project], with mentors from both projects
}}

== Clearlydefined ==

* [[ClearlyDefinedObject]]

[[Category:Available internship]]

Ingest Wikidata software origins (internship)

2024-02-04T15:09:09Z

StefanoZacchiroli:

{{Internship
|description=The Software Heritage archive currently contains source code
coming mostly from major development forges and distributions.
[https://www.wikidata.org/ Wikidata] is a free and open knowledge base about
''everything'', including software development projects. The goal of this
internship is to list ''software origins'' described in Wikidata (in particular,
but not only, version control system) and make sure they get periodically crawled
and ingested into the Software Heritage archive.

|skills=
* familiarity with the Version Control Systems
* familiarity with Wikipedia and/or Wikidata
* Python development

|mentors=
* TBD (ask on [[Matrix]])
}}

[[Category:Available internship]]

Ingest all Debian derivatives (internship)

2024-02-04T15:08:39Z

StefanoZacchiroli:

{{Internship
|description=The Software Heritage archive currently archives
[https://www.debian.org Debian] as the only supported GNU/Linux distribution.
The goal of this internship is to extend archive coverage to all Debian
derivatives, as listed in the
[https://wiki.debian.org/Derivatives/Census Debian Derivatives Census].
As part of this internship Software Heritage will be integrated with the
Derivatives Census so that distributions listed there are automatically and
periodically archived.

|skills=
* familiarity with the Debian distribution
* Python development

|mentors=
* Nicolas Dandrimont (olasd on [[IRC]])
}}

[[Category:Available internship]]

Graph query language for the archive (internship)

2024-02-04T15:08:26Z

StefanoZacchiroli:

{{Internship
|description=The Software Heritage archive is structured as a graph (specifically, a [https://en.wikipedia.org/wiki/Merkle_tree Merkle DAG]) and is huge: 20 billion nodes, 200 billion edges.
It has recently been verified that a compressed representation of the graph structure can fit in memory, whereas node/edge properties can be memory-mapped to secondary storage (see: [https://docs.softwareheritage.org/devel/swh-graph/ documentation] and in particular the [https://arxiv.org/abs/2001.08647 SANER 2020 paper referenced there]).
An [https://docs.softwareheritage.org/devel/swh-graph/api.html ad hoc API] is available to traverse the graph, with very limited querying capabilities.
The goal of this internship is to experiment with the possibility of querying the archive graph via state-of-the-art [https://en.wikipedia.org/wiki/Graph_database#Graph_query-programming_languages graph query languages].

The ideal outcome of the internship will be a prototype of query engine that answers queries on top of the compressed graph representation plus associated property maps.
A prototype implementation will be developed for a target platform to be chosen during the internship.
Tentative target platforms include: [https://neo4j.com/ Neo4j] and [http://tinkerpop.apache.org/ Apache TinkerPop].
In either case a backend/compatibility layer for WebGraph [http://webgraph.di.unimi.it/docs/it/unimi/dsi/webgraph/ImmutableGraph.html ImmutableGraph] will be developed.

|skills=
* Java development
* Query languages for structured, semi-structured, or graph data (e.g., one or more among: SQL, Xquery, GraphQL, SPARQL, GQL, etc.)

Will be considered a plus:
* experience with graph traversal languages (e.g., [https://en.wikipedia.org/wiki/Gremlin_(query_language) Gremlin])
* experience with graph databases (e.g., [https://neo4j.com/ Neo4j])

|workplace=[https://liris.cnrs.fr/en LIRIS (Univ. Lyon 1, Lyon)] or [https://www.inria.fr/en/centre-inria-de-paris Inria Paris] or contact mentors for remote opportunities

|mentors=
* [https://perso.liris.cnrs.fr/angela.bonifati/ Angela Bonifati] <angela.bonifati@univ-lyon1.fr>
* Stefano Zacchiroli <zack@upsilon.cc> (Zack on [[Matrix]])
}}

[[Category:Available internship]]

Fine-grained tracking of source code provenance (internship)

2024-02-04T15:07:44Z

StefanoZacchiroli: archive internship

{{Internship
|description=
Software Heritage is the largest existing public archive of software source code, which also keeps track of where and when source code files have been observed in the wild.
Given the checksum of a source code file and the current state of the archive, one can produce a list of all the places where said file has been (publicly) published in the past.
The goal of this internship is to experiment with increasing the granularity at which source code can be tracked, from entire files (current solution) to code snippets and/or individual lines of code.
Different techniques will be explored, implemented, and benchmarked on archive subsets to estimate their viability.

|skills=
* Python development

Will be considered a plus:
* experience with source code indexing and/or search
* experience with software audit solutions (for license compliance issues, security vulnerabilities, etc.)

|mentors=
* Guillaume Rousseau <guillaume.rousseau@univ-paris-diderot.fr>
* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])
}}

[[Category:Archived internship]]

Expand package metadata coverage (internship)

2024-02-04T15:07:27Z

StefanoZacchiroli:

{{Internship
|description=[https://archive.softwareheritage.org/browse/search/ searching] projects in the Software Heritage archive is currently possible by either (parts of) URL or by [https://www.softwareheritage.org/2019/05/28/mining-software-metadata-for-80-m-projects-and-even-more/ package metadata].
Currently, only a limited number of package metadata are [https://docs.softwareheritage.org/devel/swh-indexer/metadata-workflow.html#supported-intrinsic-metadata supported], including Maven, NPM, PyPI, and Gems.
The goal of this internship is to extend the coverage of supported metadata to additional package managers, the long-term goal being supporting all [https://libraries.io/ Libraries.io]-indexed package managers.

For more information of the existing tools, you can read our [https://www.softwareheritage.org/2019/05/28/mining-software-metadata-for-80-m-projects-and-even-more/ metadata blog post] or dive into the [https://docs.softwareheritage.org/devel/swh-indexer/metadata-workflow.html#adding-support-for-additional-ecosystem-specific-metadata technical tutorial]

|skills=
* Python development

Will be considered a plus:
* knowledge of linked data technologies and ontologies (e.g., RDFa, JSON-LD, OWL, etc.)

|mentors=
* Morane Gruenpeter (moranegg on [[IRC]])
* Valentin Lorentz (vlorentz on [[IRC]])
}}

[[Category:Available internship]]

Graph query language for the archive (internship)

2024-02-04T15:06:20Z

StefanoZacchiroli: re-enable

{{Internship
|description=The Software Heritage archive is structured as a graph (specifically, a [https://en.wikipedia.org/wiki/Merkle_tree Merkle DAG]) and is huge: 20 billion nodes, 200 billion edges.
It has recently been verified that a compressed representation of the graph structure can fit in memory, whereas node/edge properties can be memory-mapped to secondary storage (see: [https://docs.softwareheritage.org/devel/swh-graph/ documentation] and in particular the [https://arxiv.org/abs/2001.08647 SANER 2020 paper referenced there]).
An [https://docs.softwareheritage.org/devel/swh-graph/api.html ad hoc API] is available to traverse the graph, with very limited querying capabilities.
The goal of this internship is to experiment with the possibility of querying the archive graph via state-of-the-art [https://en.wikipedia.org/wiki/Graph_database#Graph_query-programming_languages graph query languages].

The ideal outcome of the internship will be a prototype of query engine that answers queries on top of the compressed graph representation plus associated property maps.
A prototype implementation will be developed for a target platform to be chosen during the internship.
Tentative target platforms include: [https://neo4j.com/ Neo4j] and [http://tinkerpop.apache.org/ Apache TinkerPop].
In either case a backend/compatibility layer for WebGraph [http://webgraph.di.unimi.it/docs/it/unimi/dsi/webgraph/ImmutableGraph.html ImmutableGraph] will be developed.

|skills=
* Java development
* Query languages for structured, semi-structured, or graph data (e.g., one or more among: SQL, Xquery, GraphQL, SPARQL, GQL, etc.)

Will be considered a plus:
* experience with graph traversal languages (e.g., [https://en.wikipedia.org/wiki/Gremlin_(query_language) Gremlin])
* experience with graph databases (e.g., [https://neo4j.com/ Neo4j])

|workplace=[https://liris.cnrs.fr/en LIRIS (Univ. Lyon 1, Lyon)] or [https://www.inria.fr/en/centre-inria-de-paris Inria Paris] or contact mentors for remote opportunities

|mentors=
* [https://perso.liris.cnrs.fr/angela.bonifati/ Angela Bonifati] <angela.bonifati@univ-lyon1.fr>
* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])
}}

[[Category:Available internship]]

Python bindings for WebGraph (internship)

2024-02-04T15:05:50Z

StefanoZacchiroli:

{{Internship
|description=Software Heritage uses the [http://webgraph.di.unimi.it/ WebGraph] framework for graph compression. This allows to manipulate the huge archive Merkle DAG in RAM efficiently, via the [https://docs.softwareheritage.org/devel/swh-graph/ swh-graph component]. WebGraph being written in Java, we would like to have Python bindings for both Software Heritage needs and to allow researchers used to the [https://www.scipy.org/ Python scientific ecosystem] to analyze and exploit the [https://annex.softwareheritage.org/public/dataset/graph/latest/compressed/ compressed representation of the Software Heritage archive].
The goal of this internship is to design and implement efficient Python bindings to the WebGraph framework, using the most appropriate [https://wiki.python.org/moin/IntegratingPythonWithOtherLanguages#Java bridge technology] between the two languages.

|skills=
* Java development
* Python development

Will be considered a plus:
* experience with system programming in the C language

|mentors=
* Antoine Pietri (seirl on [[IRC]])
* [http://vigna.di.unimi.it/ Sebastiano Vigna]
* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])
}}

[[Category:Archived internship]]

Python bindings for WebGraph (internship)

2024-02-04T15:04:49Z

StefanoZacchiroli: disable internship (obsolete now with webgraph-rs)

Git remote support for Software Heritage (internship)

2024-02-04T15:04:13Z

StefanoZacchiroli:

{{Internship
|description=Content from the Software Heritage [https://archive.softwareheritage.org/ archive] can be accessed object-by-object (e.g., one file, one commit, etc.) or reconstructed asynchronously in batch using [https://docs.softwareheritage.org/devel/swh-vault/index.html the Vault service] to download a tarball or a [https://git-scm.com/docs/git-bundle git bundle].
For local use it would be more convenient to have direct support in [https://git-scm.com/ git], so that developers can directly <code>git clone</code> or <code>git pull</code> from the Software Heritage archive, using just git.
The goal of this internship is to explore the possibility, design, and implement prototype support in Git and Software Heritage to enable that.
The intended building blocks that we plan to leverage to this end are git [https://git-scm.com/docs/partial-clone partial clones] and [https://git-scm.com/docs/gitremote-helpers remote helpers].

|skills=
* C development
* Python development
* Git

Will be considered a plus:
* experience with [https://git-scm.com/book/en/v2/Git-Internals-Plumbing-and-Porcelain Git internals and plumbing]

|mentors=
* Stefano Zacchiroli <zack@upsilon.cc> (Zack on [[Matrix]])
* Christian Couder
}}

[[Category:Available internship]]

Main Page

2024-02-04T15:03:21Z

StefanoZacchiroli: /* Students */

Welcome to the wiki of [[Software Heritage]].

We are just getting started, so please bear with us while we organize the content of the wiki.<br >
In the meantime you can find below entry points for various communities we're collaborating with.

== General ==

* [[Suggestion box]] for software we should add to the [[Archive]] ← add your entry here!
* [[Talks]] about Software Heritage

== Developers ==

* Read the [https://docs.softwareheritage.org/devel/ docs]
* Dive into the [https://forge.softwareheritage.org/ code]
* Subscribe to the [[Mailing lists]]
* Chat with us on [[Matrix]]

== Scientists ==

* [[:Category:Related work|Related work]]
* [[Working groups]]

== Students ==

* [[Internships|Internship opportunities]]

== Ambassadors ==
* [[Ambassadors onboarding]]

Integrate Software Heritage and ClearlyDefined (internship)

2023-03-26T09:23:08Z

StefanoZacchiroli:

{{Internship
|description=[https://clearlydefined.io/ ClearlyDefined] is a project whose goal is to collaboratively and semi-automatically curate information about Free/Open Source Software (FOSS) projects, including licensing and vulnerability information.
As one of its main output, ClearyDefined maintains an open data knowledge-base that cross references FOSS source code artifacts found in version control systems, package repositories, etc. to curated information about their licenses and vulnerabilities. The same source code artifacts are archived by Software Heritage for long-term preservation purposes.
The goal of this internship is to integrate ClearlyDefined and Software Heritage, for mutual benefit.
Software Heritage will benefit from mirroring ClearlyDefined data, allowing to query them while navigating the archive and at scale; ClearlyDefined will benefit from learning about the existing of FOSS projects that have not been analyzed for "clarity" yet.

|skills=
* JavaScript / NodeJS
* Python
* experience with database management systems (of any kind)

|mentors=
* Philippe Ombredanne (nexB)
* Valentin Lorentz
* Stefano Zacchiroli <zack@upsilon.cc> (Software Heritage)

|environment=
you will work shoulder to shoulder with members of the [https://www.softwareheritage.org/people/ Software Heritage] and [https://clearlydefined.io/about ClearlyDefined project], with mentors from both projects
}}

== Clearlydefined ==

* [[ClearlyDefinedObject]]

[[Category:Available internship]]

Graph query language for the archive (internship)

2022-11-10T13:17:12Z

StefanoZacchiroli:

{{Internship
|description=The Software Heritage archive is structured as a graph (specifically, a [https://en.wikipedia.org/wiki/Merkle_tree Merkle DAG]) and is huge: 20 billion nodes, 200 billion edges.
It has recently been verified that a compressed representation of the graph structure can fit in memory, whereas node/edge properties can be memory-mapped to secondary storage (see: [https://docs.softwareheritage.org/devel/swh-graph/ documentation] and in particular the [https://arxiv.org/abs/2001.08647 SANER 2020 paper referenced there]).
An [https://docs.softwareheritage.org/devel/swh-graph/api.html ad hoc API] is available to traverse the graph, with very limited querying capabilities.
The goal of this internship is to experiment with the possibility of querying the archive graph via state-of-the-art [https://en.wikipedia.org/wiki/Graph_database#Graph_query-programming_languages graph query languages].

The ideal outcome of the internship will be a prototype of query engine that answers queries on top of the compressed graph representation plus associated property maps.
A prototype implementation will be developed for a target platform to be chosen during the internship.
Tentative target platforms include: [https://neo4j.com/ Neo4j] and [http://tinkerpop.apache.org/ Apache TinkerPop].
In either case a backend/compatibility layer for WebGraph [http://webgraph.di.unimi.it/docs/it/unimi/dsi/webgraph/ImmutableGraph.html ImmutableGraph] will be developed.

|skills=
* Java development
* Query languages for structured, semi-structured, or graph data (e.g., one or more among: SQL, Xquery, GraphQL, SPARQL, GQL, etc.)

Will be considered a plus:
* experience with graph traversal languages (e.g., [https://en.wikipedia.org/wiki/Gremlin_(query_language) Gremlin])
* experience with graph databases (e.g., [https://neo4j.com/ Neo4j])

|workplace=[https://liris.cnrs.fr/en LIRIS (Univ. Lyon 1, Lyon)] or [https://www.inria.fr/en/centre-inria-de-paris Inria Paris] or contact mentors for remote opportunities

|mentors=
* [https://perso.liris.cnrs.fr/angela.bonifati/ Angela Bonifati] <angela.bonifati@univ-lyon1.fr>
* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])
}}

[[Category:Ongoing internship]]

TinkerPop Gremlin backend for WebGraph (internship)

2022-07-15T09:58:19Z

StefanoZacchiroli: mark internship as completed

{{Internship
|description=Software Heritage uses the [http://webgraph.di.unimi.it/ WebGraph] framework for graph compression. This allows to manipulate the huge archive Merkle DAG in RAM efficiently, via the [https://docs.softwareheritage.org/devel/swh-graph/ swh-graph component]. The [https://docs.softwareheritage.org/devel/swh-graph/api.html current RPC API] to navigate the graph is however very limited and ad hoc. We would like to exploit the current compressed graph representation using a standard graph traversal language such as the [https://en.wikipedia.org/wiki/Gremlin_(query_language) Gremlin graph traversal language].
The goal of this internship is to design, implement, and experiment with a backend for [http://tinkerpop.apache.org/ Apache TinkerPop] (a popular open source implementation of Gremlin) that sits on top of WebGraph. If successful it will allow to traverse the huge Software Heritage graph with both the current efficiency and the convenience of a high-level and expressive graph traversal language.

|skills=
* Java development
* basic knowledge of graph theory

Will be considered a plus:
* experience with the implementation of graph-based applications

|mentors=
* Antoine Pietri (seirl on [[IRC]])
* [http://vigna.di.unimi.it/ Sebastiano Vigna]
* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])
}}

[[Category:Completed internship]]

Add sources to the project search engine (GSoC task)

2022-03-18T09:08:03Z

StefanoZacchiroli: /* Task description */ fix markup for ordered list

== Introduction ==

The [https://archive.softwareheritage.org/ homepage of the Software Heritage archive]
features a small search engine, that searched in project URLs and project metadata.
Project metadata includes name, description, authors, etc.

This is implemented by a Python service backed by an ElasticSearch database,
which contains one document for each project; each document containing metadata
mined from the project itself

== Task description ==

We would like to add more data sources to the ElasticSearch database;
typically sources that are not authoritative, but provide metadata of usually
good quality.

This comes with the following challenges:

# there are multiple sources, and their contents must work together
# sources have different reliability, that should be taken into account when ranking search results

Therefore, this task will require making a plan to address these,
define a data model, and finally implement it in a backend.
It may involve some frontend work if necessary, to provide an interface for
these.

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* ElasticSearch
* Experience with cross-referenced data mining would be appreciated

== Potential mentors ==

* Kumar Shivendu (KShivendu on [[IRC]])
* Valentin Lorentz (vlorentz on [[IRC]])
* Vincent Sellier (vsellier on [[IRC]])

== Other relevant (but independent) tasks ==

This task is only about adding data we already collected the existing Elasticsearch database;
you may also be interested in [[Mine information from archived content (GSoC task)]]
and [[Mine information from external sources (GSoC task)]] to fill this
database; but those are completely independent tasks.

This database only contains project URLs and metadata, not source code.
Source code search is more complex, but is available as an
[[Source code search engine prototype (internship)|internship topic]]

[[Category:GSoC task]]
[[Category:Available GSoC task]]

Google Summer of Code 2022

2022-02-25T10:12:55Z

StefanoZacchiroli: /* Ideas list */

<div style="text-align: center; font-size: 1.2em; border: solid 1px black; padding: 1em;">
This page is a work in progress; the list of retained organizations for Google Summer of Code 2022 has not been determined yet.
</div>

[[File:GSoCLogo.png|512px]]

== General information ==

This page is the central point of information for [[Software Heritage]] participation into the [https://summerofcode.withgoogle.com/ Google Summer of Code] program in 2022.

Google Summer of Code is a program where Google pays students stipends to work over the (northern hemisphere) summer on free software projects such as Software Heritage. Each student works with mentors from the community to complete a software project.

== I want to participate as a student ==

Great!, we are very glad for your interest in contributing to Software Heritage and we are looking forward to work together.

=== Prerequisites ===

The following prerequisites apply to all Software Heritage GSoC projects:

* [https://www.python.org Python 3] is our language of choice, you should be fluent with that language to apply
* [https://git-scm.com Git] is our version control system of choice, you should be familiar with it to apply
* basic knowledge in using a CLI
* additional prerequisites depend on the project you will work on; check project descriptions for details

=== Before you apply ===

Here are the steps you should follow before applying, to make sure you have a good grasp of what we are doing at Software Heritage and how we do it:

# Follow our [https://docs.softwareheritage.org/devel/developer-setup.html developer setup tutorial]: it will make sure you have the source code of our software stack locally available and that you can run unit tests
# Create an account on our [https://forge.softwareheritage.org development forge]
# Familiarize yourself with our [[Code review in Phabricator|code review workflow]]
# Make at least one simple change to any one of our [https://docs.softwareheritage.org/devel/ software components] and submit it as a [https://forge.softwareheritage.org/differential/ diff] for code review, following the above workflow. [[Easy hacks]] and [https://forge.softwareheritage.org/project/view/20/ Web UI] issues are good options for what to fix, but feel free to submit any patch you think it might be useful.

=== What to include in your application ===

Make sure that your application includes the following information:

* Describe the '''specific project''' you want to work on. What do you want to achieve? Why is it important? Why is it useful for Software Heritage? The project might be one of the project ideas that we have prepared below, or something else entirely that you want to contribute to Software Heritage. Your source code archival pet peeve, surprise us!
* Detail your '''work plan''': a brief description of how you plan to go about your project, including a list of ''deliverables'' and a ''timeline'' of when do you expect them to be available.
* Include a reference to '''the diff''' you submitted before applying (see the "Before you apply" section above).

== Ideas list ==

Below you can find a list of project ideas that are good options for a reasonably-sized GSoC project (check individual idea pages for expected duration and difficulty of each task):
<DynamicPageList>
category = Available GSoC task
ordermethod = sortkey
order = ascending
</DynamicPageList>

We also have a [[Internships|list of internship topics]], which you can use for ideas when applying to GSoC with us.
Expect each internship topic to require 350 hours and to be on the harder side than GSoC-specific tasks.

All project ideas above are just suggestions, don't feel obliged to pick one of them if there is nothing that fits your taste and abilities.
Feel free to propose something else that you are excited about and that contributes to improve the Software Heritage archive: we will be happy to consider it!

== Contact ==

GSoC students are encouraged to get in touch with the Software Heritage community using the standard development communication channels, and in particular our [[IRC]] channel (#swh-devel on [https://libera.chat/ Libera Chat]) and mailing list (swh-devel).

See our [https://www.softwareheritage.org/community/developers/ development information page] for details.

== Timeline ==

See the official [https://developers.google.com/open-source/gsoc/timeline Google Summer of Code timeline].

[[Category:Google Summer of Code]]
[[Category:Google Summer of Code 2022]]

Google Summer of Code 2022

2022-02-25T10:11:19Z

StefanoZacchiroli: /* Ideas list */

<div style="text-align: center; font-size: 1.2em; border: solid 1px black; padding: 1em;">
This page is a work in progress; the list of retained organizations for Google Summer of Code 2022 has not been determined yet.
</div>

[[File:GSoCLogo.png|512px]]

== General information ==

This page is the central point of information for [[Software Heritage]] participation into the [https://summerofcode.withgoogle.com/ Google Summer of Code] program in 2022.

Google Summer of Code is a program where Google pays students stipends to work over the (northern hemisphere) summer on free software projects such as Software Heritage. Each student works with mentors from the community to complete a software project.

== I want to participate as a student ==

Great!, we are very glad for your interest in contributing to Software Heritage and we are looking forward to work together.

=== Prerequisites ===

The following prerequisites apply to all Software Heritage GSoC projects:

* [https://www.python.org Python 3] is our language of choice, you should be fluent with that language to apply
* [https://git-scm.com Git] is our version control system of choice, you should be familiar with it to apply
* basic knowledge in using a CLI
* additional prerequisites depend on the project you will work on; check project descriptions for details

=== Before you apply ===

Here are the steps you should follow before applying, to make sure you have a good grasp of what we are doing at Software Heritage and how we do it:

# Follow our [https://docs.softwareheritage.org/devel/developer-setup.html developer setup tutorial]: it will make sure you have the source code of our software stack locally available and that you can run unit tests
# Create an account on our [https://forge.softwareheritage.org development forge]
# Familiarize yourself with our [[Code review in Phabricator|code review workflow]]
# Make at least one simple change to any one of our [https://docs.softwareheritage.org/devel/ software components] and submit it as a [https://forge.softwareheritage.org/differential/ diff] for code review, following the above workflow. [[Easy hacks]] and [https://forge.softwareheritage.org/project/view/20/ Web UI] issues are good options for what to fix, but feel free to submit any patch you think it might be useful.

=== What to include in your application ===

Make sure that your application includes the following information:

* Describe the '''specific project''' you want to work on. What do you want to achieve? Why is it important? Why is it useful for Software Heritage? The project might be one of the project ideas that we have prepared below, or something else entirely that you want to contribute to Software Heritage. Your source code archival pet peeve, surprise us!
* Detail your '''work plan''': a brief description of how you plan to go about your project, including a list of ''deliverables'' and a ''timeline'' of when do you expect them to be available.
* Include a reference to '''the diff''' you submitted before applying (see the "Before you apply" section above).

== Ideas list ==

Below you can find a list of project ideas that are good options for a reasonably-sized GSoC project:
<DynamicPageList>
category = Available GSoC task
ordermethod = sortkey
order = ascending
</DynamicPageList>

We also have a [[Internships|list of internship topics]], which you can use for ideas when applying to GSoC with us.
Expect each internship topic to require 350 hours and to be on the harder side than GSoC-specific tasks.

All project ideas above are just suggestions, don't feel obliged to pick one of them if there is nothing that fits your taste and abilities.
Feel free to propose something else that you are excited about and that contributes to improve the Software Heritage archive: we will be happy to consider it!

== Contact ==

GSoC students are encouraged to get in touch with the Software Heritage community using the standard development communication channels, and in particular our [[IRC]] channel (#swh-devel on [https://libera.chat/ Libera Chat]) and mailing list (swh-devel).

See our [https://www.softwareheritage.org/community/developers/ development information page] for details.

== Timeline ==

See the official [https://developers.google.com/open-source/gsoc/timeline Google Summer of Code timeline].

[[Category:Google Summer of Code]]
[[Category:Google Summer of Code 2022]]

Google Summer of Code 2022

2022-02-25T10:01:51Z

StefanoZacchiroli:

<div style="text-align: center; font-size: 1.2em; border: solid 1px black; padding: 1em;">
This page is a work in progress; the list of retained organizations for Google Summer of Code 2022 has not been determined yet.
</div>

[[File:GSoCLogo.png|512px]]

== General information ==

This page is the central point of information for [[Software Heritage]] participation into the [https://summerofcode.withgoogle.com/ Google Summer of Code] program in 2022.

Google Summer of Code is a program where Google pays students stipends to work over the (northern hemisphere) summer on free software projects such as Software Heritage. Each student works with mentors from the community to complete a software project.

== I want to participate as a student ==

Great!, we are very glad for your interest in contributing to Software Heritage and we are looking forward to work together.

=== Prerequisites ===

The following prerequisites apply to all Software Heritage GSoC projects:

* [https://www.python.org Python 3] is our language of choice, you should be fluent with that language to apply
* [https://git-scm.com Git] is our version control system of choice, you should be familiar with it to apply
* basic knowledge in using a CLI
* additional prerequisites depend on the project you will work on; check project descriptions for details

=== Before you apply ===

Here are the steps you should follow before applying, to make sure you have a good grasp of what we are doing at Software Heritage and how we do it:

# Follow our [https://docs.softwareheritage.org/devel/developer-setup.html developer setup tutorial]: it will make sure you have the source code of our software stack locally available and that you can run unit tests
# Create an account on our [https://forge.softwareheritage.org development forge]
# Familiarize yourself with our [[Code review in Phabricator|code review workflow]]
# Make at least one simple change to any one of our [https://docs.softwareheritage.org/devel/ software components] and submit it as a [https://forge.softwareheritage.org/differential/ diff] for code review, following the above workflow. [[Easy hacks]] and [https://forge.softwareheritage.org/project/view/20/ Web UI] issues are good options for what to fix, but feel free to submit any patch you think it might be useful.

=== What to include in your application ===

Make sure that your application includes the following information:

* Describe the '''specific project''' you want to work on. What do you want to achieve? Why is it important? Why is it useful for Software Heritage? The project might be one of the project ideas that we have prepared below, or something else entirely that you want to contribute to Software Heritage. Your source code archival pet peeve, surprise us!
* Detail your '''work plan''': a brief description of how you plan to go about your project, including a list of ''deliverables'' and a ''timeline'' of when do you expect them to be available.
* Include a reference to '''the diff''' you submitted before applying (see the "Before you apply" section above).

== Ideas list ==

Below you can find a list of project ideas that are good options for a reasonably-sized GSoC project.
They include both GSoC-specific tasks (that are only available as part of our GSoC participation) and project ideas that are also available as [[Internships|internship]] topics outside of GSoC.

<DynamicPageList>
category = Available GSoC task
ordermethod = sortkey
order = ascending
</DynamicPageList>
<DynamicPageList>
category = Available internship
ordermethod = sortkey
order = ascending
</DynamicPageList>

All project ideas above are just suggestions, don't feel obliged to pick one of them if there is nothing that fits your taste and abilities.
Feel free to propose something else that you are excited about and that contributes to improve the Software Heritage archive: we will be happy toconsider it!

== Contact ==

GSoC students are encouraged to get in touch with the Software Heritage community using the standard development communication channels, and in particular our [[IRC]] channel (#swh-devel on [https://libera.chat/ Libera Chat]) and mailing list (swh-devel).

See our [https://www.softwareheritage.org/community/developers/ development information page] for details.

== Timeline ==

See the official [https://developers.google.com/open-source/gsoc/timeline Google Summer of Code timeline].

[[Category:Google Summer of Code]]
[[Category:Google Summer of Code 2022]]

Category:Google Summer of Code 2022

2022-02-25T10:00:59Z

StefanoZacchiroli: Created page with "Pages related to the 2022 edition of the [https://summerofcode.withgoogle.com/ Google summer of Code program]. Category:Google Summer of Code"

Pages related to the 2022 edition of the [https://summerofcode.withgoogle.com/ Google summer of Code program].

[[Category:Google Summer of Code]]

Category:Google Summer of Code 2021

2022-02-25T10:00:37Z

StefanoZacchiroli:

Pages related to the 2021 edition of the [https://summerofcode.withgoogle.com/ Google summer of Code program].

[[Category:Google Summer of Code]]

Category:Google Summer of Code 2021

2022-02-25T10:00:23Z

StefanoZacchiroli:

Pages related to the 2022 edition of the [https://summerofcode.withgoogle.com/ Google summer of Code program].

[[Category:Google Summer of Code]]

Mine information from external sources (GSoC task)

2022-02-25T09:57:46Z

StefanoZacchiroli: /* Task description */

== Introduction ==

In addition to archiving source code artifacts, Software Heritage is interested in
archive metadata from external sources and correlate it to source code artifacts.
This is also to enable semantic searches on the archive and scientific research.

Collecting this extrinsic metadata is a
[https://forge.softwareheritage.org/T1739 work in progress], and you are welcome
to contribute to its implementation.

== Task description ==

You would contribute to the design of our metadata-fetching architecture.
This includes:

* Review what metadata we want to fetch
* How to efficiently fetch it at regular intervals and store it
* Implement metadata fetching from at least one source, in a way that can be generalized to other sources

=== Expected duration ===

175 or 350 hours, at your option (longer duration means you can tackle more formats).
Difficulty: hard

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Prior experience in working with software metadata is a plus, but not required

== Potential mentors ==

* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])
* Valentin Lorentz (vlorentz on [[IRC]])

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Mine information from archived content (GSoC task)

2022-02-25T09:57:30Z

StefanoZacchiroli:

== Introduction ==

In addition to archival, Software Heritage indexes the retrieved source code
artifacts, to enable semantic searches on the archive and scientific research.

Indexing can happen at the individual file-level (e.g., detect the programming
language the file is written in or the license declared in its header), or at
more coarse grained granularity (e.g., what metadata are declared for the most
recently archived version of a given project).

A number of indexes are [https://forge.softwareheritage.org/source/swh-indexer/ currently supported],
such as:

* file level mining:
** MIME type detection (using libmagic)
** license detection (using FOSSology/nomossa)
** language detection (using Pygments)
** ctags extraction (using universal-ctags)
* project level mining:
** Ruby gemspec metadata
** Python PKG-INFO metadata
** Maven pom.xml metadata
** NPM package.json metadata

== Task description ==

Writing additional indexers that extract more information from archived source
code is welcome and would constitute a suitable GSoC project.

Name the kind of data mining you want to do!

For inspiration you can have a look at [https://libraries.io Libraries.io], as
most package formats/package managers support dedicated ways of expressing
metadata and we only support a small number of them up-to-now. But do not
restrict your ambition to those, any kind of data extraction/mining you want to
do on the archive could work.

You may also add support for multiple formats at once, using an external tool,
such as [https://github.com/datacite/bolognese Bolognese] or
[https://github.com/librariesio/bibliothecary/ bibliothecary].

=== Expected duration ===

175 or 350 hours, at your option (longer duration means you can tackle more formats). Difficulty: medium

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Prior experience in working with (source code) metadata is a plus, but not required

== Potential mentors ==

* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])
* Valentin Lorentz (vlorentz on [[IRC]])

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Make the Deposit modular (GSoC task)

2022-02-25T09:57:14Z

StefanoZacchiroli: StefanoZacchiroli moved page Make the Deposit modular (GSoC task) to Make the software deposit service (swh-deposit) modular (GSoC task)

#REDIRECT [[Make the software deposit service (swh-deposit) modular (GSoC task)]]

Make the software deposit service (swh-deposit) modular (GSoC task)

2022-02-25T09:57:14Z

StefanoZacchiroli: StefanoZacchiroli moved page Make the Deposit modular (GSoC task) to Make the software deposit service (swh-deposit) modular (GSoC task)

== Introduction ==

The Software Heritage archive is the most comprehensive open data knowledge base about source code that has been published openly.

In addition to fetching source code from public repositories, it offers to Deposit service, to allow platforms to send code for Software Heritage to archive.

This service is currently written as a monolith, that grew over years to include a complete [http://swordapp.github.io/SWORDv2-Profile/SWORDProfile.html SWORDv2] server, a partial SWORDv2 client with extensions, and business logic specific to Software Heritage in both. This makes the current code hard to maintain and impossible to reuse.

== Task description ==

[https://forge.softwareheritage.org/source/swh-deposit/ swh-deposit] would need to be split into the following parts:

* a generic SWORDv2 server (based on Django)
* a generic SWORDv2 client
* server-side business logic (currently implemented mostly in swh/deposit/api/common.py, but is tightly coupled with the views)
* client-side business logic

The generic server and client will need to be extensively documented, so they can be reused by other software projects.

Possible extensions include:

* The code should also be designed to allow extensions to support SWORDv3, if we ever need to support it
* A new administration front-end and/or addition of administrative tools in [https://forge.softwareheritage.org/source/swh-web/ swh-web]

=== Expected duration ===

175 hours. Difficulty: medium

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Basic understanding of the Software Heritage [https://docs.softwareheritage.org/devel/swh-model/data-model.html data model] and of [https://docs.softwareheritage.org/devel/swh-model/persistent-identifiers.html SWHID identifiers]
* Experience with Django

== Potential mentors ==

* Antoine Dumont <ardumont@softwareheritage.org> (ardumont on [[IRC]])
* Valentin Lorentz <vlorentz@softwareheritage.org> (vlorentz on [[IRC]])

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Make the software deposit service (swh-deposit) modular (GSoC task)

2022-02-25T09:56:48Z

StefanoZacchiroli: /* Task description */

Improve and extend the archive Web UI (GSoC task)

2022-02-25T09:56:20Z

StefanoZacchiroli: /* Task description */

== Introduction ==

As you probably know already, The Software Heritage archive can be
[https://archive.softwareheritage.org browsed on the Web]. The
[https://forge.softwareheritage.org/source/swh-web/ code] powering that
interface is a Django application that also implements a
[https://archive.softwareheritage.org/api/ Web API].

== Task description ==

Several improvements are possible on the archive Web interface and would make
great GSoC projects, some ideas to whet your appetite:

* add developer-oriented features, e.g., source file history, blame/praise interface, in-browser edit (with patch download), ... (note that this will also require backend design and implementation)
* improve [https://www.w3.org/WAI/ accessibility]
* display metadata we already mined from the archive

=== Expected duration ===

* if you choose the developer-oriented features: 350 hours. Difficulty: hard
* for others mentioned above: 175 hours. Difficulty: easy

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Django
* web development and/or design
* Javascript knowledge is useful but not required

== Potential mentors ==

* Antoine Lambert (anlambert on [[IRC]])
* Valentin Lorentz (vlorentz on [[IRC]])
* Jayesh Velayudhan (jayeshv on [[IRC]])

== Other relevant (but independent) tasks ==

[[Improve project search engine (GSoC task)]] is an independent task,
which may or may not involve improvements to the Web UI, depending on
you tastes.

While this task can be about displaying metadata we already mined,
you may also be interested in [[Mine information from archived content (GSoC task)]]
and [[Mine information from external sources (GSoC task)]] to mine more
of this metadata; but those are completely independent tasks.

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Dashboard UI for the Code Scanner (GSoC task)

2022-02-25T09:56:00Z

StefanoZacchiroli: /* Task description */

== Introduction ==

The Software Heritage archive is the most comprehensive open data knowledge base about source code that has been published openly.

As such, it can be used to scan local source code bases to detect which parts of it come from public code, including Free and Open Source Software.

The Software Heritage scanner (<code>swh-scanner</code>) ([https://docs.softwareheritage.org/devel/swh-scanner/ documentation], [https://forge.softwareheritage.org/source/swh-scanner/ code], [https://upsilon.cc/~zack/talks/2021/2021-04-07-llw.pdf slides of a 2021 presentation about swh-scanner]) is a command line tool that enables doing that.

== Task description ==

<code>swh-scanner</code> is currently an experimental tool, which works well in practice, but need a real '''dashboard user interface''' to be useful.
Several output options are currently available when invoking the <code>swh scanner scan</code> command, in particular batch output in textual and JSON format, and an interactive
dashboard (with the <code>-i/--interactive</code>) option.

The interactive view currently works by producing a local HTML file and opening it using the local browser.
The goal of this project is to '''improve the interactive view''', making it a serious dashboard-style UI to peruse scanning results.

The following improvements are suggested, although more can be proposed (and even more could be discovered during the project work):

* Technology: generating a local HTML file is not necessarily the best way to render results, alternative solutions should be explored, including a self-hosted web app, rendering results with state-of-the-art frontend web frameworks (css/html/javascript)
* Scalability: currently rendering doesn't work when scanning large code bases such as the Linux kernel, rendering should be made lazy, by only loading data to show when needed
* Functionality: dashboard rendering should be integrated with the possibility of opening the local source code files that have been scanned, e.g., users will want to be able to open in-browser files that have been detected as known/unknown, in order to figure why
* Functionality: in the future additional information will be added to scanning results, including license and provenance information. While not yet available right now due to backend limitations, the proposed UI should plan ahead about how/where to display such information
* Paper cuts: [https://forge.softwareheritage.org/tag/code_scanner/ various issues] affect the usability of swh-scanner, improving them would be welcome as part of this project

=== Expected duration ===

350 hours. Difficulty: medium

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Basic understanding of the Software Heritage [https://docs.softwareheritage.org/devel/swh-model/data-model.html data model] and of [https://docs.softwareheritage.org/devel/swh-model/persistent-identifiers.html SWHID identifiers]
* HTML/CSS/JavaScript and web development in general
* Working knowledge of UI/UX design principles

== Potential mentors ==

* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Create embeddable widgets (GSoC task)

2022-02-25T09:55:47Z

StefanoZacchiroli: /* Task description */

== Introduction ==

Create embeddable JS widgets to make SWH features easily available.

== Task description ==

The idea is to have a set of JS widgets that can be embedded in any web page with JS enabled.
Widgets will be smart enough to make their own api calls and render the results.

Some widgets could be
* SWH search box
* SWH Search results
* SWH Browse (for Project/revision or file)
* Save code now.

=== Expected duration ===

350 hours. Difficulty: hard

== Desirable skills ==
* Python 3 and Git are a must to work on any Software Heritage project
* web development and/or design
* Javascript

== Potential mentors ==
* Jayesh Velayudhan (jayeshv on [[IRC]])

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Create a browser extension (GSoC task)

2022-02-25T09:55:20Z

StefanoZacchiroli:

== Introduction ==

As you probably know already, The Software Heritage archive can be
[https://archive.softwareheritage.org browsed on the Web]. The
[https://forge.softwareheritage.org/source/swh-web/ code] powering that
interface is a Django application that also implements a
[https://archive.softwareheritage.org/api/ Web API].

== Task description ==

When browsing a repository (on GitHub, Gitlab, ...) or a package description (on NPM, Debian),
people may want to check when (and if) this repository or package was last archived in Software Heritage.
Currently, this means opening the archive in a new tab, and searching for the URL, and looking at the status.
Then, they can trigger a new archival with another few clicks (via the "Save Code Now" feature).

This workflow may be streamlined by the [https://forge.softwareheritage.org/T3756 creation of a browser extension or bookmarklet].
This extension/bookmarklet would, for example, show an icon next to the URL bar to show the status of the currently visited repository;
and clicking it would show details (like the date of last visit) and run "Save Code Now" in just two clicks.

=== Expected duration ===

175 hours. Difficulty: easy

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Javascript experience is also needed for this project
* Prior experience in working with browser extensions is a plus

== Potential mentors ==

* Antoine Lambert (anlambert on [[IRC]])
* Valentin Lorentz (vlorentz on [[IRC]])
* Jayesh Velayudhan (jayeshv on [[IRC]])

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Main Page

2022-02-16T12:44:55Z

StefanoZacchiroli: /* Students */

Welcome to the wiki of [[Software Heritage]].

We are just getting started, so please bear with us while we organize the content of the wiki.<br >
In the meantime you can find below entry points for various communities we're collaborating with.

== General ==

* [[Suggestion box]] for software we should add to the [[Archive]] ← add your entry here!
* [[Talks]] about Software Heritage

== Developers ==

* Read the [https://docs.softwareheritage.org/devel/ docs]
* Dive into the [https://forge.softwareheritage.org/ code]
* Subscribe to the [[Mailing lists]]
* Chat with us on [[IRC]]

== Scientists ==

* [[:Category:Related work|Related work]]
* [[Working groups]]

== Students ==

* [[Internships|Internship opportunities]]
* [[Google Summer of Code 2022]]

== Ambassadors ==
* [[Ambassadors onboarding]]

Archive search query language (internship)

2022-02-16T12:43:31Z

StefanoZacchiroli:

{{Internship
|description=The current [https://archive.softwareheritage.org/browse/search/ archive search engine] accepts a single list of tokens that are searched either in across origin URLs or [https://www.softwareheritage.org/2019/05/28/mining-software-metadata-for-80-m-projects-and-even-more/ extracted metadata].
The goal of this internship is to design and implement an archive query language that allows to mix terms, boolean connectors, and operators (as [https://support.google.com/websearch/answer/2466433 other] [https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html search] [https://help.qwant.com/help/qwant-junior/refine-search-with-operators/ engines] do).
Part of this internship is language design, part language parsing, and part language evaluation (by linking together query results returned by the already existing search backends for the archive).

|skills=
* Python development

Will be considered a plus:
* experience with data mining and/or information retrieval and/or web searches

|mentors=
* Antoine Lambert (anlambert on [[IRC]])
* Valentin Lorentz (vlorentz on [[IRC]])
* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])
}}

[[Category:Completed internship]]

Expand metadata search coverage (internship)

2022-02-16T12:42:48Z

StefanoZacchiroli: StefanoZacchiroli moved page Expand metadata search coverage (internship) to Expand package metadata coverage (internship)

#REDIRECT [[Expand package metadata coverage (internship)]]

Expand package metadata coverage (internship)

2022-02-16T12:42:48Z

StefanoZacchiroli: StefanoZacchiroli moved page Expand metadata search coverage (internship) to Expand package metadata coverage (internship)

TinkerPop Gremlin backend for WebGraph (internship)

2022-02-16T12:40:40Z

StefanoZacchiroli:

{{Internship
|description=Software Heritage uses the [http://webgraph.di.unimi.it/ WebGraph] framework for graph compression. This allows to manipulate the huge archive Merkle DAG in RAM efficiently, via the [https://docs.softwareheritage.org/devel/swh-graph/ swh-graph component]. The [https://docs.softwareheritage.org/devel/swh-graph/api.html current RPC API] to navigate the graph is however very limited and ad hoc. We would like to exploit the current compressed graph representation using a standard graph traversal language such as the [https://en.wikipedia.org/wiki/Gremlin_(query_language) Gremlin graph traversal language].
The goal of this internship is to design, implement, and experiment with a backend for [http://tinkerpop.apache.org/ Apache TinkerPop] (a popular open source implementation of Gremlin) that sits on top of WebGraph. If successful it will allow to traverse the huge Software Heritage graph with both the current efficiency and the convenience of a high-level and expressive graph traversal language.

|skills=
* Java development
* basic knowledge of graph theory

Will be considered a plus:
* experience with the implementation of graph-based applications

|mentors=
* Antoine Pietri (seirl on [[IRC]])
* [http://vigna.di.unimi.it/ Sebastiano Vigna]
* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])
}}

[[Category:Ongoing internship]]

Integrate Software Heritage and GHTorrent (internship)

2022-02-16T12:40:17Z

StefanoZacchiroli:

{{Internship
|description=Software Heritage is building the largest source code repository in existence,
initially populated with all projects from GitHub.
The [http://ghtorrent.org/ GHTorrent] project collects
and archives data from the GitHub API, including issues, teams, pull requests and
commits. The purpose of this internship is to integrate the construction processes
of the respective datasets.
The goal is to allow the two projects to be updated independently but also create
a fusion point where updates from either project's database are integrated into a
centralized, query-able archive in a streaming fashion.

|skills=
* knowledge of streaming data technologies
* familiarity of the internals of Git
* familiarity with the GitHub API
* working knowledge of any/more of Python, Kafka, Postgres, MySQL, and MongoDB would be a plus

|mentors=
* Georgios Gousios <g.gousios@tudelft.nl>
* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])
}}

[[Category:Internship]]

Gsoc

2022-02-16T12:39:06Z

StefanoZacchiroli: Changed redirect target from Google Summer of Code 2021 to Google Summer of Code 2022

#REDIRECT [[Google Summer of Code 2022]]