Software Heritage Wiki - User contributions [en]

Add sources to the project search engine (GSoC task)

2022-04-02T06:22:47Z

KShivendu: Add expected duration section

== Introduction ==

The [https://archive.softwareheritage.org/ homepage of the Software Heritage archive]
features a small search engine, that searched in project URLs and project metadata.
Project metadata includes name, description, authors, etc.

This is implemented by a Python service backed by an ElasticSearch database,
which contains one document for each project; each document containing metadata
mined from the project itself

== Task description ==

We would like to add more data sources to the ElasticSearch database;
typically sources that are not authoritative, but provide metadata of usually
good quality.

This comes with the following challenges:

# there are multiple sources, and their contents must work together
# sources have different reliability, that should be taken into account when ranking search results

Therefore, this task will require making a plan to address these,
define a data model, and finally implement it in a backend.
It may involve some frontend work if necessary, to provide an interface for
these.

== Expected duration ==
175 or 350 hours, at your option (longer duration means you can handle more data sources). Difficulty: medium

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* ElasticSearch
* Experience with cross-referenced data mining would be appreciated

== Potential mentors ==

* Kumar Shivendu (KShivendu on [[IRC]])
* Valentin Lorentz (vlorentz on [[IRC]])
* Vincent Sellier (vsellier on [[IRC]])

== Other relevant (but independent) tasks ==

This task is only about adding data we already collected the existing Elasticsearch database;
you may also be interested in [[Mine information from archived content (GSoC task)]]
and [[Mine information from external sources (GSoC task)]] to fill this
database; but those are completely independent tasks.

This database only contains project URLs and metadata, not source code.
Source code search is more complex, but is available as an
[[Source code search engine prototype (internship)|internship topic]]

[[Category:GSoC task]]
[[Category:Available GSoC task]]

Mine information from external sources (GSoC task)

2022-03-11T06:42:33Z

KShivendu: Add KShivendu

== Introduction ==

In addition to archiving source code artifacts, Software Heritage is interested in
archive metadata from external sources and correlate it to source code artifacts.
This is also to enable semantic searches on the archive and scientific research.

Collecting this extrinsic metadata is a
[https://forge.softwareheritage.org/T1739 work in progress], and you are welcome
to contribute to its implementation.

== Task description ==

You would contribute to the design of our metadata-fetching architecture.
This includes:

* Review what metadata we want to fetch
* How to efficiently fetch it at regular intervals and store it
* Implement metadata fetching from at least one source, in a way that can be generalized to other sources

=== Expected duration ===

175 or 350 hours, at your option (longer duration means you can tackle more formats).
Difficulty: hard

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Prior experience in working with software metadata is a plus, but not required

== Potential mentors ==

* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])
* Valentin Lorentz (vlorentz on [[IRC]])
* Kumar Shivendu (KShivendu on [[IRC]])

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Mine information from archived content (GSoC task)

2022-03-11T06:42:08Z

KShivendu: Add KShivendu

== Introduction ==

In addition to archival, Software Heritage indexes the retrieved source code
artifacts, to enable semantic searches on the archive and scientific research.

Indexing can happen at the individual file-level (e.g., detect the programming
language the file is written in or the license declared in its header), or at
more coarse grained granularity (e.g., what metadata are declared for the most
recently archived version of a given project).

A number of indexes are [https://forge.softwareheritage.org/source/swh-indexer/ currently supported],
such as:

* file level mining:
** MIME type detection (using libmagic)
** license detection (using FOSSology/nomossa)
** language detection (using Pygments)
** ctags extraction (using universal-ctags)
* project level mining:
** Ruby gemspec metadata
** Python PKG-INFO metadata
** Maven pom.xml metadata
** NPM package.json metadata

== Task description ==

Writing additional indexers that extract more information from archived source
code is welcome and would constitute a suitable GSoC project.

Name the kind of data mining you want to do!

For inspiration you can have a look at [https://libraries.io Libraries.io], as
most package formats/package managers support dedicated ways of expressing
metadata and we only support a small number of them up-to-now. But do not
restrict your ambition to those, any kind of data extraction/mining you want to
do on the archive could work.

You may also add support for multiple formats at once, using an external tool,
such as [https://github.com/datacite/bolognese Bolognese] or
[https://github.com/librariesio/bibliothecary/ bibliothecary].

=== Expected duration ===

175 or 350 hours, at your option (longer duration means you can tackle more formats). Difficulty: medium

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Prior experience in working with (source code) metadata is a plus, but not required

== Potential mentors ==

* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])
* Valentin Lorentz (vlorentz on [[IRC]])
* Kumar Shivendu (KShivendu on [[IRC]])

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Dashboard UI for the Code Scanner (GSoC task)

2022-03-11T06:41:55Z

KShivendu: Add KShivendu

== Introduction ==

The Software Heritage archive is the most comprehensive open data knowledge base about source code that has been published openly.

As such, it can be used to scan local source code bases to detect which parts of it come from public code, including Free and Open Source Software.

The Software Heritage scanner (<code>swh-scanner</code>) ([https://docs.softwareheritage.org/devel/swh-scanner/ documentation], [https://forge.softwareheritage.org/source/swh-scanner/ code], [https://upsilon.cc/~zack/talks/2021/2021-04-07-llw.pdf slides of a 2021 presentation about swh-scanner]) is a command line tool that enables doing that.

== Task description ==

<code>swh-scanner</code> is currently an experimental tool, which works well in practice, but need a real '''dashboard user interface''' to be useful.
Several output options are currently available when invoking the <code>swh scanner scan</code> command, in particular batch output in textual and JSON format, and an interactive
dashboard (with the <code>-i/--interactive</code>) option.

The interactive view currently works by producing a local HTML file and opening it using the local browser.
The goal of this project is to '''improve the interactive view''', making it a serious dashboard-style UI to peruse scanning results.

The following improvements are suggested, although more can be proposed (and even more could be discovered during the project work):

* Technology: generating a local HTML file is not necessarily the best way to render results, alternative solutions should be explored, including a self-hosted web app, rendering results with state-of-the-art frontend web frameworks (css/html/javascript)
* Scalability: currently rendering doesn't work when scanning large code bases such as the Linux kernel, rendering should be made lazy, by only loading data to show when needed
* Functionality: dashboard rendering should be integrated with the possibility of opening the local source code files that have been scanned, e.g., users will want to be able to open in-browser files that have been detected as known/unknown, in order to figure why
* Functionality: in the future additional information will be added to scanning results, including license and provenance information. While not yet available right now due to backend limitations, the proposed UI should plan ahead about how/where to display such information
* Paper cuts: [https://forge.softwareheritage.org/tag/code_scanner/ various issues] affect the usability of swh-scanner, improving them would be welcome as part of this project

=== Expected duration ===

350 hours. Difficulty: medium

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Basic understanding of the Software Heritage [https://docs.softwareheritage.org/devel/swh-model/data-model.html data model] and of [https://docs.softwareheritage.org/devel/swh-model/persistent-identifiers.html SWHID identifiers]
* HTML/CSS/JavaScript and web development in general
* Working knowledge of UI/UX design principles

== Potential mentors ==

* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])
* Kumar Shivendu (KShivendu on [[IRC]])

[[Category:Available GSoC task]]
[[Category:GSoC task]]

Google Summer of Code 2022

2022-03-09T06:20:02Z

KShivendu: Make it clear that we are selected for GSoC 2022

[[File:GSoCLogo.png|512px]]

== General information ==

This page is the central point of information for [[Software Heritage]] participation into the [https://summerofcode.withgoogle.com/ Google Summer of Code] program in 2022.

Google Summer of Code is a program where Google pays students stipends to work over the (northern hemisphere) summer on free software projects such as Software Heritage. Each student works with mentors from the community to complete a software project.

== I want to participate as a student ==

Great!, we are very glad for your interest in contributing to Software Heritage and we are looking forward to work together.

=== Prerequisites ===

The following prerequisites apply to all Software Heritage GSoC projects:

* [https://www.python.org Python 3] is our language of choice, you should be fluent with that language to apply
* [https://git-scm.com Git] is our version control system of choice, you should be familiar with it to apply
* basic knowledge in using a CLI
* additional prerequisites depend on the project you will work on; check project descriptions for details

=== Before you apply ===

Here are the steps you should follow before applying, to make sure you have a good grasp of what we are doing at Software Heritage and how we do it:

# Follow our [https://docs.softwareheritage.org/devel/developer-setup.html developer setup tutorial]: it will make sure you have the source code of our software stack locally available and that you can run unit tests
# Create an account on our [https://forge.softwareheritage.org development forge]
# Familiarize yourself with our [[Code review in Phabricator|code review workflow]]
# Make at least one simple change to any one of our [https://docs.softwareheritage.org/devel/ software components] and submit it as a [https://forge.softwareheritage.org/differential/ diff] for code review, following the above workflow. [[Easy hacks]] and [https://forge.softwareheritage.org/project/view/20/ Web UI] issues are good options for what to fix, but feel free to submit any patch you think it might be useful.

=== What to include in your application ===

Make sure that your application includes the following information:

* Describe the '''specific project''' you want to work on. What do you want to achieve? Why is it important? Why is it useful for Software Heritage? The project might be one of the project ideas that we have prepared below, or something else entirely that you want to contribute to Software Heritage. Your source code archival pet peeve, surprise us!
* Detail your '''work plan''': a brief description of how you plan to go about your project, including a list of ''deliverables'' and a ''timeline'' of when do you expect them to be available.
* Include a reference to '''the diff''' you submitted before applying (see the "Before you apply" section above).

== Ideas list ==

Below you can find a list of project ideas that are good options for a reasonably-sized GSoC project (check individual idea pages for expected duration and difficulty of each task):
<DynamicPageList>
category = Available GSoC task
ordermethod = sortkey
order = ascending
</DynamicPageList>

We also have a [[Internships|list of internship topics]], which you can use for ideas when applying to GSoC with us.
Expect each internship topic to require 350 hours and to be on the harder side than GSoC-specific tasks.

All project ideas above are just suggestions, don't feel obliged to pick one of them if there is nothing that fits your taste and abilities.
Feel free to propose something else that you are excited about and that contributes to improve the Software Heritage archive: we will be happy to consider it!

== Contact ==

GSoC students are encouraged to get in touch with the Software Heritage community using the standard development communication channels, and in particular our [[IRC]] channel (#swh-devel on [https://libera.chat/ Libera Chat]) and mailing list (swh-devel).

See our [https://www.softwareheritage.org/community/developers/ development information page] for details.

== Timeline ==

See the official [https://developers.google.com/open-source/gsoc/timeline Google Summer of Code timeline].

[[Category:Google Summer of Code]]
[[Category:Google Summer of Code 2022]]