Software Heritage Wiki - User contributions [en]

Matrix

2021-03-23T13:33:06Z

Seirl: /* IRC authentication */

== IRC channels ==

The following channels have been registered on the [https://freenode.net/ Freenode] network for [[Software Heritage]] usage.

* [https://app.element.io/#/room/#freenode_#swh-devel:matrix.org '''#swh-devel''']: public development discussions
* [https://app.element.io/#/room/#freenode_#swh-team:matrix.org '''#swh-team''']: private discussions of the core team
* [https://app.element.io/#/room/#freenode_#swh-sysadm:matrix.org '''#swh-sysadm''']: operations team discussions/bots
* [https://app.element.io/#/room/#freenode_#softwareheritage:matrix.org '''#softwareheritage''']: general discussions about the project (currently unused)
* [https://app.element.io/#/room/#freenode_#swh:matrix.org '''#swh''']: ditto, in case we end up preferring the short version

If you use IRC, consider joining the channels.

If you don't use IRC ''directly'', you can still join our chat channels from your web browser via a [https://matrix.org/ Matrix] bridge by clicking on the channel names in the list above. You will be asked to create a [https://element.io/ Element] account if you don't have one yet.

== IRC authentication ==

You should register their nick with NickServ using:

/nick <USERNAME>
/msg nickserv register <PASSWORD> <EMAIL>

You will then receive an e-mail containing a link to activate you account. After doing so, you need to configure your client to auto-authenticate. The recommended way of doing that is using [https://freenode.net/kb/answer/sasl SASL authentication].

For Weechat:

/set irc.server.freenode.sasl_username <USERNAME>
/set irc.server.freenode.sasl_password <PASSWORD>

For matrix, the relevant docs is here: https://github.com/matrix-org/matrix-appservice-irc/wiki/End-user-FAQ#how-do-i-registeridentify-to-nickserv

Freenode also supports authentication via [https://freenode.net/kb/answer/certfp TLS client certificates].

=== Matrix bridge ===

For the Matrix bridge ([https://github.com/matrix-org/matrix-appservice-irc/wiki/End-user-FAQ#how-do-i-registeridentify-to-nickserv relevant docs here]), you first need to choose you IRC nick by sending this to the bridge appservice:

/msg @appservice-irc:matrix.org !nick <USERNAME>

Then send this command to NickServ to register your account:

/msg @freenode_NickServ:matrix.org register <PASSWORD> <EMAIL>

You will then receive an e-mail containing a link to activate you account. You can then identify using:

/msg @freenode_NickServ:matrix.org identify <PASSWORD>

Then, ask the IRC bridge appservice to remember your password so that you get identified automatically:

/msg @appservice-irc:matrix.org !storepass <USERNAME>:<PASSWORD>

== IRC access list ==

To auto-voice people with a registered nick (only doable by people with +fA access modes will be able to do it), add them to the channel access list:

/msg chanserv access #swh-devel add zack +V

[[Category:Infrastructure]]

Suggestion box: source code to add

2020-05-27T11:06:35Z

Seirl:

The [[Archive]] is growing organically. We started "small", tracking 3 '''software origins''' (GitHub + Debian + GNU), and we will be adding new origins bit by bit, depending on the urgency of archiving them and available development energies to integrate them into Software Heritage.

Using this page you can add suggestions of software origins that we aren't following yet, but we should. You can include information about who to contact for technical collaboration, the urgency of archival, and other useful information. To that end, just add a row to the table below. Here some information about the meaning of the various columns.

Entries are currently listed simply in order of addition to this page; we will add more structure when the list will start growing.

=== Legend ===

;Software origin
: any (public accessible) "place" on the Internet that host software in source code form. Please provide a title for it and hyperlink it to the relevant URL
;Type of origin
: information about the kind of hosting, e.g., whether it is a forge, a collection of repositories, an homepage publishing tarball, or a one shot source code repository. For all kind of repositories please specify which VCS system is in use (Git, SVN, CVS, etc.)
;Contact
: who to contact for technical collaboration on how to best archive source code hosted on the software origin. You can list yourself if you're the relevant person, or provide the most relevant contact point if you know it
;Conservation status
: information about how likely it is that the software origin will disappear; high likelihood will make it more urgent for us to archive software hosted there. We suggest to use the [https://en.wikipedia.org/wiki/Conservation_status species conservation status], i.e., one of: Critically endangered (CR), Endangered (EN), Vulnerable (VU), Near threatened (NT), Least concern (LC).
;How to mirror
: (pointers to) technical information on how to do a full mirror of ''all'' the source code available at the software origin, ideally one shot and in batch
;How to keep up
: (pointers to) technical information on how to incrementally retrieve new source code accumulated since the last visit; usually this should be based on some kind of incremental change feed or event API
;Notes
: anything else you think we should know about this software origin
;Forge task
: pointer to the task on our [[forge]] tracking the work needed to ingest the software origin

== Suggestions ==

* https://notabug.org/ (customized gogs)
* https://gitgud.io/ (GitLab, run by Sapphire, a user-supported open source company)
* https://octo.sh/ (GitLab)
* https://chaos.expert/explore (GitLab by Chaos Computer Club)
* https://gitlab.coko.foundation/public (GitLab)
* https://git.teknik.io/explore/repos (Gitea)
* https://gitlab.gnome.org/explore/groups (GNOME software)
* https://launchpad.net/ (used by Ubuntu and others)
* https://archive.codeplex.com/ (was Microsoft's free, open source project hosting)
* https://wiki.p2pfoundation.net/List_of_Community-Hosted_GitLab_Instances
* https://0xacab.org/explore (Gitlab)
* https://git.fosscommunity.in/explore/projects (GitLab by Free Software Community of India)

{| class="wikitable"
!Software origin
!Type of origin
!Contact
!Conservation status
!How to mirror
!How to keep up
!Notes
![https://forge.softwareheritage.org/ Forge] task
|-
|'''(sample entry)''' ''GitHubBub forge''
|''Git hosting''
|''John Doe <john@example.com>''
|''LC''
|''retrieve full repo list at /api/list, then git clone on each entry''
|''poll RSS feed at /api/updates?since=YYYY-MM-DD''
|''nothing special to add''
|''[https://forge.softwareheritage.org/T123456 T123456]''
|-
|[https://bitbucket.org/ Bitbucket]
|Git and hg/Mercurial hosting
|
|style="background-color: lightgreen"| LC
|
|
|
|[https://forge.softwareheritage.org/T561 T561]
|-
|[https://sourceforge.net/ SourceForge]
|CVS, SVN, Mercurial, Git
|
|style="background-color: orange"|VU
|
|
|
|
|-
|[https://wiki.debian.org/Derivatives/Census all Debian derivatives]
|Debian-based distros
|Paul Wise <pabs@debian.org>
|varying, depending on the distro
|see [[Suggestion_box:_source_code_to_add/Debian_derivatives|details]]
|see [[Suggestion_box:_source_code_to_add/Debian_derivatives|details]]
|
|
|-
|[https://www.gentoo.org/ Gentoo]
|
|Johannes Kellner <gentoo@johannes-kellner.eu>
|style="background-color: lightgreen"|LC
|
|
|
|
|-
|[http://pauillac.inria.fr/~huet/cea.html Gérard Huet's seminal work on 3D]
|Scanned source code
|Gérard Huet <gerard.huet@inria.fr>
|style="background-color: red"|EN
|retrieve listing images from the web pages
|N/A
|links are half broken, yquem should be replaced with pauillac everywhere it appears
|
|-
|[http://www.softwarepreservation.org/projects Software Preservation Project]
|Website with a collection of archives
|Paul McJones <paul@mcjones.org>
|style="background-color: lightgreen"|LC
|
|
|
|
|-
|[https://code.nasa.gov/ 253 NASA open source software projects]
|
|
|style="background-color: lightgreen"|LC
|
|
|
|
|-
|[http://smaky.ch/ Smaky], the swiss micro-computer series
|[http://infini.smaky.ch/sources.html Partial code dump]
|[mailto:arnaud@epsitec.ch Pierre Arnaud] (current CEO of Epsitec) and/or [mailto:jean-daniel.nicoud@epfl.ch Jean-Daniel Nicoud] (founder of the computer series]
|style="background-color: red"|EN
|Probably manually
|No new updates
|Some references to this history: [http://www.memoires-informatiques.org/ Fondation Mémoires Informatiques], [http://smaky.ch/ Smaky.ch] (in particular, [http://smaky.ch/theme.php?id=lami the short history]
|
|-
|[https://www.wikidata.org/wiki/Wikidata:WikiProject_Informatics/FLOSS#Conservation_status_2 wikidata endangered software]
|depends on the "source code repository" property
|Loic Dachary <loic@dachary.org>
|style="background-color: yellow"|The risk is higher than [https://www.wikidata.org/wiki/Property_talk:P141 LC]
|A script should obtain the "source code repository" property for the software and mirror it depending on the [https://www.wikidata.org/wiki/Wikidata:WikiProject_Informatics/FLOSS#source_code_repository protocol] qualifier. If the "source code repository" is "no value", the [https://www.wikidata.org/wiki/Wikidata:WikiProject_Informatics/Software#streaming_media_URL streaming media URL] of the "preferred" [https://www.wikidata.org/wiki/Wikidata:WikiProject_Informatics/Software#software_version_.28P348.29 software version] should be downloaded instead.
|Once a copy is secured by software heritage, a URL to the software heritage repository should be added to the "source code repository" property and the "conservation status" property should be removed, meaning it is "least concerned" by default. The software will no longer show in the list of endangered software.
|This is work in progress, part of the [https://www.wikidata.org/wiki/Wikidata:WikiProject_Informatics/FLOSS wikidata FLOSS project] and the scripts do not exist yet.
|
|-
|historical KDE repos
|CVS/SVN/Git
|KDE sysadmin team <sysadmin@kde.org>
|style="background-color: yellow"|NT
|See [[Suggestion box: source code to add/KDE|details]]
|See [[Suggestion box: source code to add/KDE|details]]
|
|
|-
|[https://java.net/projects Java.net] & [https://kenai.com/ Kenai.com]
|hg, git, svn
|communitymanager@java.net
|style="background-color: black; color: white"|CR
|
|
|[https://community.oracle.com/community/java/javanet-forge-sunset Shutting down on April 28, 2017]
|
|-
|[https://fedorahosted.org/ fedorahosted.org]
|git, svn, hg, bzr
|[https://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org infrastructure@lists.fedoraproject.org]
|style="background-color: black; color: white"|CR
|
|
|[https://communityblog.fedoraproject.org/fedorahosted-sunset-2017-02-28/ Shutting down on Feb. 28, 2017]
|
|-
|[http://www.societe-informatique-de-france.fr/wp-content/uploads/2015/12/1024-no7-Baude.pdf Langage symbolique d'Enseignement (LSE)]
|archives
|Association Enseignement Public et Informatique (EPI) <bureau@epi.asso.fr>
|style="background-color: black; color: white"|CR
|
|
| + educational software (INRP-CNDP)
|
|-
|[http://www.netlib.org The Netlib collection of numerical software]
|structured website with links to archives
|
|style="background-color: lightgreen"|LC
|
|
|many of these libraries are mirrored in sources already collected in Software Heritage; there is sure value in the curation information.
|
|-
|[https://codebender.cc/ codebender]
|IOT and educationnal ressources
|
|style="background-color: black; color: white"|CR
|
|
|closing expected at Dec31st, 2016 upon this announce by founders https://codebender.cc/next-chapter
|
|-
|[https://git.oschina.net/ OS China]
|Chinese GitHub equivalent
|
|style="background-color: lightgreen"|LC
|
|
|interesting test case for all the unicode tooling in Software Heritage
|
|-
|Usenet source code archives
|NNTP
|
|style="background-color: yellow"|NT
|crawl relevant newsgroup archives (e.g., at Google Groups), retrieve archives (possibly chunked), ingest
|one shot might be enough?
|suggestion by John Gilmore
|
|-
|[https://www.x.org/releases/ X11/XOrg archives]
|http
|
|style="background-color: lightgreen"|LC
|
|one shot might be enough
|
||''[https://forge.softwareheritage.org/T1774 T1774]''
|-
|[https://www.curseforge.com/ Curse mods]
|Code distributed as versioned tarballs
|
|style="background-color: lightgreen"|LC
|The API should be sufficient, maybe some scraping will be required
|
|
|
|-
|Symbian source code
|HG
|carlo.daffara@nodeweaver.eu
|style="background-color: orange"|VU
|
|
|Cloned from Nokia's Symbian Mercurial repository, a few days before the closure of the repo and the change to a proprietary license. The mercurial HG files are the only copy publicly available; I have made a snapshot of the code and placed on sourceforge- but missing is the entire project history and commit log.
|
|-
|[https://gist.github.com/ GitHub's gists]
|git
|
|style="background-color: lightgreen"|LC
|No proper listing API, but there's [https://developer.github.com/v3/gists/#list-all-public-gists an endpoint] to get gists created after a given date
|use the same endpoint
|
|
|-
|[https://codeberg.org/ Codeberg]
|git
|contact@codeberg.org
|style="background-color: lightgreen"|LC
|[https://github.com/go-gitea Gitea] API
|[https://github.com/go-gitea Gitea] API
|Codeberg e.V. is a Non-Profit Collaboration Community for Free and Open Source Projects
|
|-
|[https://puszcza.gnu.org.ua/ Puszcza]
|cvs/git/hg/svn, VCS snapshot tarballs, tarballs
|https://puszcza.gnu.org.ua/contact.php
|style="background-color: lightgreen"|LC
|Savane instance, https://download.gnu.org.ua/ (also has ftp), http://git.gnu.org.ua/
|Savane instance
|
|
|-
|[https://directory.fsf.org/ Free Software Directory]
|solely contains links (to VCS and tarballs)
|https://lists.gnu.org/archive/html/directory-discuss/
|style="background-color: lightgreen"|LC
|MediaWiki instance, use API to clone (or git-remote-mediawiki)
|MediaWiki instance, use API for updates clone (or git-remote-mediawiki)
|
|
|-
|[https://kb.netgear.com/2649/NETGEAR-Open-Source-Code-for-Programmers-GPL NetGear GPL tarballs]
|Tarballs
|
|style="background-color: yellow"|NT
|
|
|
|
|-
|[https://www.ui.com/download/edgemax/edgerouter/default/edgerouter-er-8erpro-8ep-r8-firmware-v11011 Ubiquity firmware GPL tarballs]
|Tarballs
|
|style="background-color: yellow"|NT
|
|
|
|
|-
[[Category:Archive]]
[[Category:Suggestions]]

Google Summer of Code 2020

2020-01-29T14:53:02Z

Seirl: /* Ideas list */

[[File:GSoCLogo.png|1024px]]

== General information ==

This page is the central point of information for [[Software Heritage]] participation into the [https://summerofcode.withgoogle.com/ Google Summer of Code] program in 2020.

Google Summer of Code is a program where Google pays students stipends to work over the (northern hemisphere) summer on free software projects such as Software Heritage. Each student works with mentors from the community to complete a software project.

== Accepted projects ==

== I want to participate as a student ==

Great!, we are very glad for your interest in contributing to Software Heritage and we are looking forward to work together.

=== Prerequisites ===

The following prerequisites apply to Software Heritage GSoC projects:

* [https://www.python.org Python 3] is our language of choice, you should be fluent with that language to apply
* [https://git-scm.com Git] is our version control system of choice, you should be familiar with it to apply
* additional prerequisites depend on the project you will work on; check project descriptions for details

=== Before you apply ===

Here are the steps you should follow before applying, to make sure you have a good grasp of what we are doing at Software Heritage and how we do it:

# Follow our [https://docs.softwareheritage.org/devel/getting-started.html getting started guide]: it will make sure you can locally run a (small) copy of the archive and ingest source code into it
# Create an account on our [https://forge.softwareheritage.org development forge]
# Familiarize yourself with our [[Code review in Phabricator|code review workflow]]
# Make at least one simple change to any one of our [https://docs.softwareheritage.org/devel/ software components] and submit it as a [https://forge.softwareheritage.org/differential/ diff] for code review, following the above workflow. [[Easy hacks]] and [https://forge.softwareheritage.org/project/view/20/ Web UI] issues are good options for what to fix, but feel free to submit any patch you think it might be useful.

=== What to include in your application ===

Make sure that your application includes the following information:

* Describe the '''specific project''' you want to work on. What do you want to achieve? Why is it important? Why is it useful for Software Heritage? The project might be one of the project ideas that we have prepared below, or something else entirely that you want to contribute to Software Heritage. Your source code archival pet peeve, surprise us!
* Detail your '''work plan''': a brief description of how you plan to go about your project, including a list of ''deliverables'' and a ''timeline'' of when do you expect them to be available.
* Include a reference to '''the diff''' you submitted before applying (see the "Before you apply" section above).

== Ideas list ==

Below you can find a list of project ideas that are good options for a
reasonably sized GSoC project. They are just suggestion though, don't feel
obliged to pick one of them if there is nothing that fits your taste and
abilities. Feel free to propose something else that you are excited about and
that contributes to improve the Software Heritage archive: we will be happy to
consider it!

=== Mine information from archived content ===

In addition to archival, Software Heritage indexes the retrieved source code
artifacts, to enable semantic searches on the archive and scientific research.

Indexing can happen at the individual file-level (e.g., detect the programming
language the file is written in or the license declared in its header), or at
more coarse grained granularity (e.g., what metadata are declared for the most
recently archived version of a given project).

A number of indexes are [https://forge.softwareheritage.org/source/swh-indexer/ currently supported],
such as:

* file level mining:
** MIME type detection (using libmagic)
** license detection (using FOSSology/nomossa)
** language detection (using Pygments)
** ctags extraction (using universal-ctags)
* project level mining:
** Ruby gemspec metadata
** Python PKG-INFO metadata
** Maven pom.xml metadata
** NPM package.json metadata

Writing additional indexers that extract more information from archived source
code is welcome and would constitute a suitable GSoC project.

Name the kind of data mining you want to do!

For inspiration you can have a look at [https://libraries.io Libraries.io], as
most package formats/package managers support dedicated ways of expressing
metadata and we only support a small number of them up-to-now. But do not
restrict your ambition to those, any kind of data extraction/mining you want to
do on the archive could work.

You may also add support for multiple formats at once, using an external tool,
such as [https://github.com/datacite/bolognese Bolognese] or
[https://github.com/librariesio/bibliothecary/ bibliothecary].

=== Mine information from external sources ===

In addition to arching source code artifacts, Software Heritage is interested in
archive metadata from external sources and correlate it to source code artifacts.
This is also to enable semantic searches on the archive and scientific research.

Collecting this extrinsic metadata is a
[https://forge.softwareheritage.org/T1739 work in progress], and you are welcome
to contribute to its implementation.

=== Improve and extend the archive Web UI ===

As you probably know already, The Software Heritage archive can be
[https://archive.softwareheritage.org browsed on the Web]. The
[https://forge.softwareheritage.org/source/swh-web/ code] powering that
interface is a Django application that also implements a
[https://archive.softwareheritage.org/api/ Web API].

Several improvements are possible on the archive Web interface and would make
great GSoC projects, some ideas to whet your appetite:

* add new source code search criteria and improve the search interface
* add developer-oriented features, e.g., source file history, blame/praise interface, in-browser edit (with patch download), ... (note that this will also require backend design and implementation)
* improve [https://www.w3.org/WAI/ accessibility]
* help us design and implement our [https://forge.softwareheritage.org/T1805 next API version]

=== Improve the Vault ===

The Software Heritage archive allows retrieval of archived objects of different formats.
Once an object has been chosen for retrieval, it can be "cooked" using the [https://docs.softwareheritage.org/devel/swh-vault/index.html Software Heritage Vault].

Right now the Vault has several limitations: it only handles two kinds of objects (revisions and directories), it requires recursively requesting the database to get the full subgraph of an object, and it generates revisions in an unpractical format (git fast-import).

Several improvements are possible:

* add coverage for new kinds of objects (releases, snapshots and even origins?)
* use our in-memory graph database [https://docs.softwareheritage.org/devel/swh-graph/index.html swh-graph] to speed up fetching the necessary subgraphs.
* write cookers to output new formats (e.g git tarballs/git bundles or even other VCS?)
* improve end-to-end testing
* other general code improvements (better progress/error reporting in the frontend, etc.)

=== Research internships ===

For the more research-inclined students, we also maintain a separate list of [[Internships]].
Any topic below is also a viable GSoC project.

<DynamicPageList>
category=Available internship
</DynamicPageList>

== Contact ==

GSoC students are encouraged to get in touch with the Software Heritage community using the standard development communication channels, i.e.:

* the #swh-devel IRC channel on [https://freenode.net Freenode]
* the [https://sympa.inria.fr/sympa/info/swh-devel swh-devel mailing list]

See our [https://www.softwareheritage.org/community/developers/ development information page] for more details.

== Timeline ==

See the official [https://developers.google.com/open-source/gsoc/timeline Google Summer of Code timeline].

[[Category:Google Summer of Code]]
[[Category:Google Summer of Code 2020]]

Main Page

2019-03-25T14:45:25Z

Seirl:

Welcome to the wiki of [[Software Heritage]].

We are just getting started, so please bear with us while we organize the content of the wiki.<br >
In the meantime you can find below entry points for various communities we're collaborating with.

== General ==

* [[Suggestion box]] for software we should add to the [[Archive]] ← add your entry here!
* [[Talks]] about Software Heritage

== Developers ==

* see pages in [[:Category:Software development]]

== Scientists ==

* [[:Category:Related work|Related work]]
* [[Working groups]]

== Students ==

* [[Internships|Internship opportunities]]
* [[Google Summer of Code 2019]]

Suggestion box: source code to add

2019-03-22T17:36:05Z

Seirl:

The [[Archive]] is growing organically. We started "small", tracking 3 '''software origins''' (GitHub + Debian + GNU), and we will be adding new origins bit by bit, depending on the urgency of archiving them and available development energies to integrate them into Software Heritage.

Using this page you can add suggestions of software origins that we aren't following yet, but we should. You can include information about who to contact for technical collaboration, the urgency of archival, and other useful information. To that end, just add a row to the table below. Here some information about the meaning of the various columns.

Entries are currently listed simply in order of addition to this page; we will add more structure when the list will start growing.

=== Legend ===

;Software origin
: any (public accessible) "place" on the Internet that host software in source code form. Please provide a title for it and hyperlink it to the relevant URL
;Type of origin
: information about the kind of hosting, e.g., whether it is a forge, a collection of repositories, an homepage publishing tarball, or a one shot source code repository. For all kind of repositories please specify which VCS system is in use (Git, SVN, CVS, etc.)
;Contact
: who to contact for technical collaboration on how to best archive source code hosted on the software origin. You can list yourself if you're the relevant person, or provide the most relevant contact point if you know it
;Conservation status
: information about how likely it is that the software origin will disappear; high likelihood will make it more urgent for us to archive software hosted there. We suggest to use the [https://en.wikipedia.org/wiki/Conservation_status species conservation status], i.e., one of: Critically endangered (CR), Endangered (EN), Vulnerable (VU), Near threatened (NT), Least concern (LC).
;How to mirror
: (pointers to) technical information on how to do a full mirror of ''all'' the source code available at the software origin, ideally one shot and in batch
;How to keep up
: (pointers to) technical information on how to incrementally retrieve new source code accumulated since the last visit; usually this should be based on some kind of incremental change feed or event API
;Notes
: anything else you think we should know about this software origin
;Forge task
: pointer to the task on our [[forge]] tracking the work needed to ingest the software origin

== Suggestions ==

* https://notabug.org/ (customized gogs)
* https://gitgud.io/ (GitLab, run by Sapphire, a user-supported open source company)
* https://octo.sh/ (GitLab)
* https://chaos.expert/explore (GitLab by Chaos Computer Club)
* https://gitlab.coko.foundation/public (GitLab)
* https://git.teknik.io/explore/repos (Gitea)
* https://gitlab.gnome.org/explore/groups (GNOME software)
* https://launchpad.net/ (used by Ubuntu and others)
* https://archive.codeplex.com/ (was Microsoft's free, open source project hosting)
* https://wiki.p2pfoundation.net/List_of_Community-Hosted_GitLab_Instances
* https://0xacab.org/explore (Gitlab)
* https://git.fosscommunity.in/explore/projects (GitLab by Free Software Community of India)

{| class="wikitable"
!Software origin
!Type of origin
!Contact
!Conservation status
!How to mirror
!How to keep up
!Notes
![https://forge.softwareheritage.org/ Forge] task
|-
|'''(sample entry)''' ''GitHubBub forge''
|''Git hosting''
|''John Doe <john@example.com>''
|''LC''
|''retrieve full repo list at /api/list, then git clone on each entry''
|''poll RSS feed at /api/updates?since=YYYY-MM-DD''
|''nothing special to add''
|''[https://forge.softwareheritage.org/T123456 T123456]''
|-
|[https://bitbucket.org/ Bitbucket]
|Git and hg/Mercurial hosting
|
|style="background-color: lightgreen"| LC
|
|
|
|[https://forge.softwareheritage.org/T561 T561]
|-
|[https://sourceforge.net/ SourceForge]
|CVS, SVN, Mercurial, Git
|
|style="background-color: orange"|VU
|
|
|
|
|-
|[https://wiki.debian.org/Derivatives/Census all Debian derivatives]
|Debian-based distros
|Paul Wise <pabs@debian.org>
|varying, depending on the distro
|see [[Suggestion_box:_source_code_to_add/Debian_derivatives|details]]
|see [[Suggestion_box:_source_code_to_add/Debian_derivatives|details]]
|
|
|-
|[https://www.gentoo.org/ Gentoo]
|
|Johannes Kellner <gentoo@johannes-kellner.eu>
|style="background-color: lightgreen"|LC
|
|
|
|
|-
|[http://pauillac.inria.fr/~huet/cea.html Gérard Huet's seminal work on 3D]
|Scanned source code
|Gérard Huet <gerard.huet@inria.fr>
|style="background-color: red"|EN
|retrieve listing images from the web pages
|N/A
|links are half broken, yquem should be replaced with pauillac everywhere it appears
|
|-
|[http://www.softwarepreservation.org/projects Software Preservation Project]
|Website with a collection of archives
|Paul McJones <paul@mcjones.org>
|style="background-color: lightgreen"|LC
|
|
|
|
|-
|[https://code.nasa.gov/ 253 NASA open source software projects]
|
|
|style="background-color: lightgreen"|LC
|
|
|
|
|-
|[http://smaky.ch/ Smaky], the swiss micro-computer series
|[http://infini.smaky.ch/sources.html Partial code dump]
|[mailto:arnaud@epsitec.ch Pierre Arnaud] (current CEO of Epsitec) and/or [mailto:jean-daniel.nicoud@epfl.ch Jean-Daniel Nicoud] (founder of the computer series]
|style="background-color: red"|EN
|Probably manually
|No new updates
|Some references to this history: [http://www.memoires-informatiques.org/ Fondation Mémoires Informatiques], [http://smaky.ch/ Smaky.ch] (in particular, [http://smaky.ch/theme.php?id=lami the short history]
|
|-
|[https://www.wikidata.org/wiki/Wikidata:WikiProject_Informatics/FLOSS#Conservation_status_2 wikidata endangered software]
|depends on the "source code repository" property
|Loic Dachary <loic@dachary.org>
|style="background-color: yellow"|The risk is higher than [https://www.wikidata.org/wiki/Property_talk:P141 LC]
|A script should obtain the "source code repository" property for the software and mirror it depending on the [https://www.wikidata.org/wiki/Wikidata:WikiProject_Informatics/FLOSS#source_code_repository protocol] qualifier. If the "source code repository" is "no value", the [https://www.wikidata.org/wiki/Wikidata:WikiProject_Informatics/Software#streaming_media_URL streaming media URL] of the "preferred" [https://www.wikidata.org/wiki/Wikidata:WikiProject_Informatics/Software#software_version_.28P348.29 software version] should be downloaded instead.
|Once a copy is secured by software heritage, a URL to the software heritage repository should be added to the "source code repository" property and the "conservation status" property should be removed, meaning it is "least concerned" by default. The software will no longer show in the list of endangered software.
|This is work in progress, part of the [https://www.wikidata.org/wiki/Wikidata:WikiProject_Informatics/FLOSS wikidata FLOSS project] and the scripts do not exist yet.
|
|-
|historical KDE repos
|CVS/SVN/Git
|KDE sysadmin team <sysadmin@kde.org>
|style="background-color: yellow"|NT
|See [[Suggestion box: source code to add/KDE|details]]
|See [[Suggestion box: source code to add/KDE|details]]
|
|
|-
|[https://java.net/projects Java.net] & [https://kenai.com/ Kenai.com]
|hg, git, svn
|communitymanager@java.net
|style="background-color: black; color: white"|CR
|
|
|[https://community.oracle.com/community/java/javanet-forge-sunset Shutting down on April 28, 2017]
|
|-
|[https://fedorahosted.org/ fedorahosted.org]
|git, svn, hg, bzr
|[https://lists.fedoraproject.org/admin/lists/infrastructure@lists.fedoraproject.org infrastructure@lists.fedoraproject.org]
|style="background-color: black; color: white"|CR
|
|
|[https://communityblog.fedoraproject.org/fedorahosted-sunset-2017-02-28/ Shutting down on Feb. 28, 2017]
|
|-
|[http://www.societe-informatique-de-france.fr/wp-content/uploads/2015/12/1024-no7-Baude.pdf Langage symbolique d'Enseignement (LSE)]
|archives
|Association Enseignement Public et Informatique (EPI) <bureau@epi.asso.fr>
|style="background-color: black; color: white"|CR
|
|
| + educational software (INRP-CNDP)
|
|-
|[http://www.netlib.org The Netlib collection of numerical software]
|structured website with links to archives
|
|style="background-color: lightgreen"|LC
|
|
|many of these libraries are mirrored in sources already collected in Software Heritage; there is sure value in the curation information.
|
|-
|[https://codebender.cc/ codebender]
|IOT and educationnal ressources
|
|style="background-color: black; color: white"|CR
|
|
|closing expected at Dec31st, 2016 upon this announce by founders https://codebender.cc/next-chapter
|
|-
|[https://git.oschina.net/ OS China]
|Chinese GitHub equivalent
|
|style="background-color: lightgreen"|LC
|
|
|interesting test case for all the unicode tooling in Software Heritage
|
|-
|Usenet source code archives
|NNTP
|
|style="background-color: yellow"|NT
|crawl relevant newsgroup archives (e.g., at Google Groups), retrieve archives (possibly chunked), ingest
|one shot might be enough?
|suggestion by John Gilmore
|
|-
|[https://www.x.org/releases/ X11/XOrg archives]
|http
|
|style="background-color: lightgreen"|LC
|
|one shot might be enough
|
|
|-
|[https://www.curseforge.com/ Curse mods]
|Code distributed as versioned tarballs
|
|style="background-color: lightgreen"|LC
|The API should be sufficient, maybe some scraping will be required
|
|
|
|-
|Symbian source code
|HG
|carlo.daffara@nodeweaver.eu
|style="background-color: orange"|VU
|
|
|Cloned from Nokia's Symbian Mercurial repository, a few days before the closure of the repo and the change to a proprietary license. The mercurial HG files are the only copy publicly available; I have made a snapshot of the code and placed on sourceforge- but missing is the entire project history and commit log.
|
|-
|[https://gist.github.com/ GitHub's gists]
|git
|
|style="background-color: lightgreen"|LC
|No proper listing API, but there's [https://developer.github.com/v3/gists/#list-all-public-gists an endpoint] to get gists created after a given date
|use the same endpoint
|
|
|-

[[Category:Archive]]
[[Category:Suggestions]]

Graph Dataset on Amazon Athena

2019-03-11T18:09:57Z

Seirl: Created page with "== Using Amazon Athena == The ''Software Heritage graph dataset'' is available as a public dataset in [https://aws.amazon.com/athena/ Amazon Athena]. === Setup === In order..."

== Using Amazon Athena ==

The ''Software Heritage graph dataset'' is available as a public dataset in [https://aws.amazon.com/athena/ Amazon Athena].

=== Setup ===

In order to query the dataset using Athena, you will first need to [https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/ create an AWS account and setup billing].

Once your AWS account is ready, you will need to install a few dependencies on your machine:

* Python 3
* The [https://docs.aws.amazon.com/cli/index.html aws cli]
* The [https://boto3.amazonaws.com/v1/documentation/api/latest/index.html boto3 Python package]

On Debian, the dependencies can be installed with the following commands:

<pre>sudo apt install python3 python3-boto3 awscli</pre>
Once the dependencies are installed, run:

<pre>aws configure</pre>

and add your AWS Access Key ID and your AWS Secret Access Key, to give Python access to your AWS account.

=== Create the tables ===

To import the schema of the dataset into your account, download the scripts from the [https://annex.softwareheritage.org/public/dataset/swh-graph-2019-01-28/athena/ athena/] folder, then run the following command:

<pre>./gen_schema.py</pre>

This will create the required tables in your AWS account. You can check that the tables were successfuly created by going to the [https://console.aws.amazon.com/athena/home Amazon Athena console] and selecting the "swh" database.

=== Run queries ===

From the console, once you have selected the "swh" database, you can directly run queries from the Query Editor.

Here is an example query that computes the most frequent file names in the archive:

<pre>SELECT FROM_UTF8(name, '?') AS name,
COUNT(DISTINCT target) AS cnt
FROM directory_entry_file
GROUP BY name
ORDER BY cnt DESC
LIMIT 1;</pre>
More documentation on Amazon Athena is available [https://docs.aws.amazon.com/athena/index.html here].

Google Summer of Code 2019

2019-02-03T20:48:09Z