Mine information from archived content (GSoC task) - Revision history

KShivendu: Add KShivendu

2022-03-11T06:42:08Z

Add KShivendu

← Older revision		Revision as of 06:42, 11 March 2022
Line 53:		Line 53:
	* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])		* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])
	* Valentin Lorentz (vlorentz on [[IRC]])		* Valentin Lorentz (vlorentz on [[IRC]])
			* Kumar Shivendu (KShivendu on [[IRC]])

	[[Category:Available GSoC task]]		[[Category:Available GSoC task]]
	[[Category:GSoC task]]		[[Category:GSoC task]]

StefanoZacchiroli at 09:57, 25 February 2022

2022-02-25T09:57:30Z

← Older revision		Revision as of 09:57, 25 February 2022
Line 40:		Line 40:
	[https://github.com/librariesio/bibliothecary/ bibliothecary].		[https://github.com/librariesio/bibliothecary/ bibliothecary].

	Expected duration: 175 or 350 hours, at your option (longer duration means you can tackle more formats). Difficulty: medium		=== Expected duration ===

			175 or 350 hours, at your option (longer duration means you can tackle more formats). Difficulty: medium

	== Desirable skills ==		== Desirable skills ==

Vlorentz: Add missing required info

2022-02-25T07:17:56Z

Add missing required info

← Older revision		Revision as of 07:17, 25 February 2022
Line 39:		Line 39:
	such as [https://github.com/datacite/bolognese Bolognese] or		such as [https://github.com/datacite/bolognese Bolognese] or
	[https://github.com/librariesio/bibliothecary/ bibliothecary].		[https://github.com/librariesio/bibliothecary/ bibliothecary].

			Expected duration: 175 or 350 hours, at your option (longer duration means you can tackle more formats). Difficulty: medium

	== Desirable skills ==		== Desirable skills ==

Vlorentz at 11:59, 8 February 2022

2022-02-08T11:59:11Z

← Older revision		Revision as of 11:59, 8 February 2022
Line 50:		Line 50:
	* Valentin Lorentz (vlorentz on [[IRC]])		* Valentin Lorentz (vlorentz on [[IRC]])

			[[Category:Available GSoC task]]
	[[Category:GSoC task]]		[[Category:GSoC task]]

Vlorentz: /* Potential mentors */

2021-03-02T10:33:44Z

Potential mentors

← Older revision		Revision as of 10:33, 2 March 2021
Line 47:		Line 47:
	== Potential mentors ==		== Potential mentors ==

	* Stefano Zacchiroli <zack@upsilon.cc>		* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])
	* Valentin Lorentz		* Valentin Lorentz (vlorentz on [[IRC]])

	[[Category:GSoC task]]		[[Category:GSoC task]]

Vlorentz: /* Desirable skills */

2021-03-02T10:08:47Z

Desirable skills

← Older revision		Revision as of 10:08, 2 March 2021
Line 43:		Line 43:

	* Python 3 and Git are a must to work on any Software Heritage project		* Python 3 and Git are a must to work on any Software Heritage project
	* Prior experience in working with (source code) metadata is a plus,		* Prior experience in working with (source code) metadata is a plus, but not required
	but not required

	== Potential mentors ==		== Potential mentors ==

Vlorentz: add skills & mentors

2021-03-02T10:08:40Z

add skills & mentors

← Older revision		Revision as of 10:08, 2 March 2021
Line 1:		Line 1:
	== ~~Introduct~~ ==		== Introduction ==

	In addition to archival, Software Heritage indexes the retrieved source code		In addition to archival, Software Heritage indexes the retrieved source code
Line 39:		Line 39:
	such as [https://github.com/datacite/bolognese Bolognese] or		such as [https://github.com/datacite/bolognese Bolognese] or
	[https://github.com/librariesio/bibliothecary/ bibliothecary].		[https://github.com/librariesio/bibliothecary/ bibliothecary].

			== Desirable skills ==

			* Python 3 and Git are a must to work on any Software Heritage project
			* Prior experience in working with (source code) metadata is a plus,
			but not required

			== Potential mentors ==

			* Stefano Zacchiroli <zack@upsilon.cc>
			* Valentin Lorentz

	[[Category:GSoC task]]		[[Category:GSoC task]]

Vlorentz: split into sections

2021-03-02T10:04:18Z

split into sections

← Older revision		Revision as of 10:04, 2 March 2021
Line 1:		Line 1:
			== Introduct ==

	In addition to archival, Software Heritage indexes the retrieved source code		In addition to archival, Software Heritage indexes the retrieved source code
	artifacts, to enable semantic searches on the archive and scientific research.		artifacts, to enable semantic searches on the archive and scientific research.
Line 20:		Line 22:
	** Maven pom.xml metadata		** Maven pom.xml metadata
	** NPM package.json metadata		** NPM package.json metadata

			== Task description ==

	Writing additional indexers that extract more information from archived source		Writing additional indexers that extract more information from archived source

StefanoZacchiroli: factor out from main GSoC page

2020-02-10T09:06:21Z

factor out from main GSoC page

New page

In addition to archival, Software Heritage indexes the retrieved source code
artifacts, to enable semantic searches on the archive and scientific research.

Indexing can happen at the individual file-level (e.g., detect the programming
language the file is written in or the license declared in its header), or at
more coarse grained granularity (e.g., what metadata are declared for the most
recently archived version of a given project).

A number of indexes are [https://forge.softwareheritage.org/source/swh-indexer/ currently supported],
such as:

* file level mining:
** MIME type detection (using libmagic)
** license detection (using FOSSology/nomossa)
** language detection (using Pygments)
** ctags extraction (using universal-ctags)
* project level mining:
** Ruby gemspec metadata
** Python PKG-INFO metadata
** Maven pom.xml metadata
** NPM package.json metadata

Writing additional indexers that extract more information from archived source
code is welcome and would constitute a suitable GSoC project.

Name the kind of data mining you want to do!

For inspiration you can have a look at [https://libraries.io Libraries.io], as
most package formats/package managers support dedicated ways of expressing
metadata and we only support a small number of them up-to-now. But do not
restrict your ambition to those, any kind of data extraction/mining you want to
do on the archive could work.

You may also add support for multiple formats at once, using an external tool,
such as [https://github.com/datacite/bolognese Bolognese] or
[https://github.com/librariesio/bibliothecary/ bibliothecary].

[[Category:GSoC task]]