Difference between revisions of "Mine information from archived content (GSoC task)"

From Software Heritage Wiki
Jump to: navigation, search
(factor out from main GSoC page)
 
(split into sections)
Line 1: Line 1:
 +
== Introduct ==
 +
 
In addition to archival, Software Heritage indexes the retrieved source code
 
In addition to archival, Software Heritage indexes the retrieved source code
 
artifacts, to enable semantic searches on the archive and scientific research.
 
artifacts, to enable semantic searches on the archive and scientific research.
Line 20: Line 22:
 
** Maven pom.xml metadata
 
** Maven pom.xml metadata
 
** NPM package.json metadata
 
** NPM package.json metadata
 +
 +
== Task description ==
  
 
Writing additional indexers that extract more information from archived source
 
Writing additional indexers that extract more information from archived source

Revision as of 10:04, 2 March 2021

Introduct

In addition to archival, Software Heritage indexes the retrieved source code artifacts, to enable semantic searches on the archive and scientific research.

Indexing can happen at the individual file-level (e.g., detect the programming language the file is written in or the license declared in its header), or at more coarse grained granularity (e.g., what metadata are declared for the most recently archived version of a given project).

A number of indexes are currently supported, such as:

  • file level mining:
    • MIME type detection (using libmagic)
    • license detection (using FOSSology/nomossa)
    • language detection (using Pygments)
    • ctags extraction (using universal-ctags)
  • project level mining:
    • Ruby gemspec metadata
    • Python PKG-INFO metadata
    • Maven pom.xml metadata
    • NPM package.json metadata

Task description

Writing additional indexers that extract more information from archived source code is welcome and would constitute a suitable GSoC project.

Name the kind of data mining you want to do!

For inspiration you can have a look at Libraries.io, as most package formats/package managers support dedicated ways of expressing metadata and we only support a small number of them up-to-now. But do not restrict your ambition to those, any kind of data extraction/mining you want to do on the archive could work.

You may also add support for multiple formats at once, using an external tool, such as Bolognese or bibliothecary.