Improve the Code Scanner (GSoC task) - Revision history

StefanoZacchiroli: mark as no longer available as gsoc task

2022-02-13T16:30:28Z

mark as no longer available as gsoc task

← Older revision		Revision as of 16:30, 13 February 2022
Line 34:		Line 34:
	* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])		* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])

	~~[[Category:Available GSoC task]]~~
	[[Category:GSoC task]]		[[Category:GSoC task]]

Vlorentz at 11:59, 8 February 2022

2022-02-08T11:59:06Z

← Older revision		Revision as of 11:59, 8 February 2022
Line 34:		Line 34:
	* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])		* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])

			[[Category:Available GSoC task]]
	[[Category:GSoC task]]		[[Category:GSoC task]]

StefanoZacchiroli: /* Task description */ add provenance integration

2021-03-14T11:06:03Z

Task description: add provenance integration

← Older revision		Revision as of 11:06, 14 March 2021
Line 13:		Line 13:
	Several improvements are possible:		Several improvements are possible:

			* integrate provenance information in results (using [https://docs.softwareheritage.org/devel/swh-graph/ swh-graph] and/or [https://forge.softwareheritage.org/source/swh-provenance/ swh-provenance])
	* make the different algorithms (see [https://forge.softwareheritage.org/source/swh-scanner/history/benchmark/ benchmark branch] in Git) used to query the backend user-selectable		* make the different algorithms (see [https://forge.softwareheritage.org/source/swh-scanner/history/benchmark/ benchmark branch] in Git) used to query the backend user-selectable
	* minimize the number of queries to the [https://archive.softwareheritage.org/api/1/known/doc/ /known API endpoint], in order to consume API rate limit less and be generally more efficient		* minimize the number of queries to the [https://archive.softwareheritage.org/api/1/known/doc/ /known API endpoint], in order to consume API rate limit less and be generally more efficient

StefanoZacchiroli: /* Task description */ mention git scanning

2021-03-09T21:36:35Z

Task description: mention git scanning

← Older revision		Revision as of 21:36, 9 March 2021
Line 17:		Line 17:
	* be adaptive in how the backend is queried, e.g., for code trees that contain less than 1000 files it is more efficient to just query all of them at once, without following the DAG structure (even if it is in theory a faster approach)		* be adaptive in how the backend is queried, e.g., for code trees that contain less than 1000 files it is more efficient to just query all of them at once, without following the DAG structure (even if it is in theory a faster approach)
	* improve the web-based dashboard view (<code>--interactive</code>), making it more user friendly		* improve the web-based dashboard view (<code>--interactive</code>), making it more user friendly
			* add support for scanning from within repositories (e.g., if the code is in git, we can lookup the commit ID directly)
	* add progress reporting during scanning, in particular for large code bases		* add progress reporting during scanning, in particular for large code bases
	* add on-disk caching, in particular for large code bases		* add on-disk caching, in particular for large code bases

StefanoZacchiroli at 21:31, 9 March 2021

2021-03-09T21:31:22Z

← Older revision		Revision as of 21:31, 9 March 2021
Line 5:		Line 5:
	As such, it can be used to scan local source code bases to detect which parts of it come from public code, including Free and Open Source Software.		As such, it can be used to scan local source code bases to detect which parts of it come from public code, including Free and Open Source Software.

	The Software Heritage scanner (swh-scanner) ([https://docs.softwareheritage.org/devel/swh-scanner/ documentation], [https://forge.softwareheritage.org/source/swh-scanner/ code]) is a command line tool that enables doing that.		The Software Heritage scanner (<code>swh-scanner</code>) ([https://docs.softwareheritage.org/devel/swh-scanner/ documentation], [https://forge.softwareheritage.org/source/swh-scanner/ code]) is a command line tool that enables doing that.

	== Task description ==		== Task description ==

	swh-scanner is currently an experimental tool, which works well in practice, but need some polishing to make it usable in production in real use cases.		<code>swh-scanner</code> is currently an experimental tool, which works well in practice, but need some polishing to make it usable in production in real use cases.

	Several improvements are possible:		Several improvements are possible:

StefanoZacchiroli: remove WIP marker

2021-03-09T21:30:42Z

remove WIP marker

← Older revision		Revision as of 21:30, 9 March 2021
Line 1:		Line 1:
	~~'''WORK IN PROGRESS'''~~

	== Introduction ==		== Introduction ==

StefanoZacchiroli: StefanoZacchiroli moved page Improve the Source Code Scanner (GSoC task) to Improve the Code Scanner (GSoC task) without leaving a redirect

2021-03-09T21:30:22Z

StefanoZacchiroli moved page Improve the Source Code Scanner (GSoC task) to Improve the Code Scanner (GSoC task) without leaving a redirect

← Older revision	Revision as of 21:30, 9 March 2021
(No difference)

StefanoZacchiroli: /* Task description */

2021-03-09T21:30:08Z

Task description

← Older revision		Revision as of 21:30, 9 March 2021
Line 15:		Line 15:
	Several improvements are possible:		Several improvements are possible:

	* make		* make the different algorithms (see [https://forge.softwareheritage.org/source/swh-scanner/history/benchmark/ benchmark branch] in Git) used to query the backend user-selectable
	* use our in-memory graph database [https://~~docs~~.softwareheritage.org/~~devel~~/swh-~~graph~~/~~index.html swh~~-~~graph]~~ to ~~speed up fetching~~ the ~~necessary subgraphs~~.		* minimize the number of queries to the [https://archive.softwareheritage.org/api/1/known/doc/ /known API endpoint], in order to consume API rate limit less and be generally more efficient
	* ~~write cookers to output new formats (~~e.g ~~git tarballs/git bundles or~~ even ~~other VCS?~~)		* be adaptive in how the backend is queried, e.g., for code trees that contain less than 1000 files it is more efficient to just query all of them at once, without following the DAG structure (even if it is in theory a faster approach)
	* improve ~~unit and end~~-to-~~end testing~~		* improve the web-based dashboard view (<code>--interactive</code>), making it more user friendly
	* other general code improvements (~~better progress~~/~~error reporting in the frontend, etc~~.)		* add progress reporting during scanning, in particular for large code bases
			* add on-disk caching, in particular for large code bases
			* integrate into the generated output other information available from the archive, e.g., license information, metadata, provenance information, etc.
			* general code improvements, including refactoring and deduplication w.r.t. the rest of the Software Heritage code base (see [https://forge.softwareheritage.org/tag/code_scanner/ open tasks])

	== Desirable skills ==		== Desirable skills ==

StefanoZacchiroli: initial page skeleton, still WIP

2021-03-09T21:21:31Z

initial page skeleton, still WIP

New page

'''WORK IN PROGRESS'''

== Introduction ==

The Software Heritage archive is the most comprehensive open data knowledge base about source code that has been published openly.

As such, it can be used to scan local source code bases to detect which parts of it come from public code, including Free and Open Source Software.

The Software Heritage scanner (swh-scanner) ([https://docs.softwareheritage.org/devel/swh-scanner/ documentation], [https://forge.softwareheritage.org/source/swh-scanner/ code]) is a command line tool that enables doing that.

== Task description ==

swh-scanner is currently an experimental tool, which works well in practice, but need some polishing to make it usable in production in real use cases.

Several improvements are possible:

* make
* use our in-memory graph database [https://docs.softwareheritage.org/devel/swh-graph/index.html swh-graph] to speed up fetching the necessary subgraphs.
* write cookers to output new formats (e.g git tarballs/git bundles or even other VCS?)
* improve unit and end-to-end testing
* other general code improvements (better progress/error reporting in the frontend, etc.)

== Desirable skills ==

* Python 3 and Git are a must to work on any Software Heritage project
* Basic understanding of the Software Heritage [https://docs.softwareheritage.org/devel/swh-model/data-model.html data model] and of [https://docs.softwareheritage.org/devel/swh-model/persistent-identifiers.html SWHID identifiers]
* JavaScript and front-end web development, if you want to work on the interactive dashboard

== Potential mentors ==

* Stefano Zacchiroli <zack@upsilon.cc> (zack on [[IRC]])

[[Category:GSoC task]]