WG/Source Discovery and Ingestion - Revision history

StefanoZacchiroli at 13:43, 31 July 2016

2016-07-31T13:43:10Z

← Older revision		Revision as of 13:43, 31 July 2016
Line 58:		Line 58:


	[[Category: ~~technical]]~~		[[Category:Working group]]
	~~[[Category: ingestion~~]]

StefanoZacchiroli at 07:49, 4 May 2016

2016-05-04T07:49:06Z

← Older revision		Revision as of 07:49, 4 May 2016
Line 1:		Line 1:
	= ~~Working group on Source Discovery and Ingestion~~ =		= Charter =
	~~== Charter ==~~
	Collecting the source code that is publicly available is an essential part of Software Heritage's mission.		Collecting the source code that is publicly available is an essential part of Software Heritage's mission.
	In order to fulfill this mission, we will need to discover, harvest, and keep up to date, content coming from a very diverse		In order to fulfill this mission, we will need to discover, harvest, and keep up to date, content coming from a very diverse
Line 13:		Line 13:
	This is a challenging task, and in order to succeed, the involvement of a large community is needed.		This is a challenging task, and in order to succeed, the involvement of a large community is needed.

	=== Mission ===		== Mission ==
	The SODI working group's mission is to foster ''the development'' and		The SODI working group's mission is to foster ''the development'' and
	''the adoption'' of software components that can, for a given origin, make it		''the adoption'' of software components that can, for a given origin, make it
Line 21:		Line 21:
	content.		content.

	=== Duration ===		== Duration ==
	This working group is open ended.		This working group is open ended.

	=== Expected outcomes ===		== Expected outcomes ==
	The main expected outcomes of the SODI working group are listed below:		The main expected outcomes of the SODI working group are listed below:

Line 36:		Line 36:
	The SODI working group will establish the relevant connections in order to raise awareness among all the interested parties.		The SODI working group will establish the relevant connections in order to raise awareness among all the interested parties.

	=== Milestones ===		== Milestones ==
	A first set of requirements for the API, and a preliminary draft of the API is		A first set of requirements for the API, and a preliminary draft of the API is
	expected to emerge from the work performed to ensure that Inria's own forge(s)		expected to emerge from the work performed to ensure that Inria's own forge(s)
	are properly tracked in the Software Heritage architecture.		are properly tracked in the Software Heritage architecture.

	=== Related working groups ===		== Related working groups ==
	This working group is related to: [[Modeling_and_Ingesting_Version_control_systems \| Modeling and Ingesting Version control systems (MIV)]]		This working group is related to: [[Modeling_and_Ingesting_Version_control_systems \| Modeling and Ingesting Version control systems (MIV)]]

	== Team contact(s) ==		= Team contact(s) =
	* [http://upsilon.cc/~zack/ Stefano Zacchiroli]		* [http://upsilon.cc/~zack/ Stefano Zacchiroli]

	== Documents ==		= Documents =
	Documents produced by the working group will be listed in this section.		Documents produced by the working group will be listed in this section.

	== Connections ==		= Connections =
	Active or planned connections to other initiatives, and activities, will be listed in this section.		Active or planned connections to other initiatives, and activities, will be listed in this section.

	== Infrastructure ==		= Infrastructure =
	=== Mailing list ===		== Mailing list ==
	* https://sympa.inria.fr/sympa/info/sodi-wg-swh		* https://sympa.inria.fr/sympa/info/sodi-wg-swh

StefanoZacchiroli at 16:00, 3 May 2016

2016-05-03T16:00:43Z

← Older revision		Revision as of 16:00, 3 May 2016
Line 1:		Line 1:
	= Working group on Source Discovery and Ingestion ~~(SODI)~~ =		= Working group on Source Discovery and Ingestion =
	== Charter ==		== Charter ==
	Collecting the source code that is publicly available is an essential part of Software Heritage's mission.		Collecting the source code that is publicly available is an essential part of Software Heritage's mission.

StefanoZacchiroli at 15:59, 3 May 2016

2016-05-03T15:59:56Z

← Older revision		Revision as of 15:59, 3 May 2016
Line 1:		Line 1:
	= Working group on Source Discovery and Ingestion =		= Working group on Source Discovery and Ingestion (SODI) =
	== Charter ==		== Charter ==
	Collecting the source code that is publicly available is an essential part of Software Heritage's mission.		Collecting the source code that is publicly available is an essential part of Software Heritage's mission.

StefanoZacchiroli: StefanoZacchiroli moved page Source Discovery and Ingestion to WG/Source Discovery and Ingestion

2016-05-03T15:57:23Z

StefanoZacchiroli moved page Source Discovery and Ingestion to WG/Source Discovery and Ingestion

← Older revision	Revision as of 15:57, 3 May 2016
(No difference)

Rdicosmo at 19:16, 3 April 2016

2016-04-03T19:16:10Z

← Older revision		Revision as of 19:16, 3 April 2016
Line 11:		Line 11:
	** a variety of different source code archives, ranging from the [ftp://ftp.gnu.org Gnu ftp server] to individual web pages		** a variety of different source code archives, ranging from the [ftp://ftp.gnu.org Gnu ftp server] to individual web pages

	This is a challenging task, and in order to ~~suceed~~, the involvement of a large community is needed.		This is a challenging task, and in order to succeed, the involvement of a large community is needed.

	=== Mission ===		=== Mission ===

RobertoDiCosmo: /* Team contact(s) */

2016-03-28T15:08:38Z

Team contact(s)

← Older revision		Revision as of 15:08, 28 March 2016
Line 45:		Line 45:

	== Team contact(s) ==		== Team contact(s) ==
	* [http://www.dicosmo.org/ Roberto Di Cosmo]
	* [http://upsilon.cc/~zack/ Stefano Zacchiroli]		* [http://upsilon.cc/~zack/ Stefano Zacchiroli]

RobertoDiCosmo: /* Team contact(s) */

2016-03-25T15:55:15Z

Team contact(s)

← Older revision		Revision as of 15:55, 25 March 2016
Line 45:		Line 45:

	== Team contact(s) ==		== Team contact(s) ==
	* Roberto Di Cosmo		* [http://www.dicosmo.org/ Roberto Di Cosmo]
			* [http://upsilon.cc/~zack/ Stefano Zacchiroli]
	* Stefano Zacchiroli

	== Documents ==		== Documents ==

RobertoDiCosmo: First version of the complete description of the SODI working group, including charter and expected outcomes

2016-03-25T15:53:42Z

First version of the complete description of the SODI working group, including charter and expected outcomes

← Older revision		Revision as of 15:53, 25 March 2016
Line 1:		Line 1:
			= Working group on Source Discovery and Ingestion =
	== Charter ==		== Charter ==
			Collecting the source code that is publicly available is an essential part of Software Heritage's mission.
			In order to fulfill this mission, we will need to discover, harvest, and keep up to date, content coming from a very diverse
			set of possible origins like:

	~~This working group~~, ~~codename SODI~~, ~~is in charge of~~ the ~~issues related to adding and updating external sources~~ of ~~content to~~ the ~~archive~~.		* several kinds of [https://en.wikipedia.org/wiki/Comparison_of_source_code_hosting_facilities source code repositories], including:
	~~It is expected~~ to ~~work both on technical and non technical aspects~~		** mainstream development platforms, like GitHub, BitBucket, Sourceforge or Codeplex;
			** institutional forges, like Inria's, Cenatic's or Adullact's FusionForge;
			** community repositories like Debian's FusionForge or Joomla's gForge, Gnu's [http://savannah.gnu.org/ Savane], and the [http://www.apache.org/dyn/closer.cgi/ Apache] or [https://git.eclipse.org/c/ Eclipse] custom repositories;
			** a variety of different source code archives, ranging from the [ftp://ftp.gnu.org Gnu ftp server] to individual web pages

	~~=== Technical aspects ===~~		This is a challenging task, and in order to suceed, the involvement of a large community is needed.
	* an API for the archive to interact with an external source (forge, repository, web page, ~~etc.)~~ in order to ~~perform at least~~ the ~~following tasks~~
	** list the full content of a ~~source~~
	** get the list of updates in a source since a given moment in time
	* mechanisms for discovering new sources

	=== ~~Non technical aspects~~ ===		=== Mission ===
			The SODI working group's mission is to foster ''the development'' and
			''the adoption'' of software components that can, for a given origin, make it
			''discoverable'', and ''list'' its contents, both ''in its entirety'', which is
			useful for a newly discovered origin, and ''incrementally'', which is useful
			to keep the Software Heritage archive up to date with the origin's evolving
			content.

	~~The~~ working group ~~members will contribute to raise awareness, and foster broad adoption of the APIs and mechanisms defined.~~		=== Duration ===
	~~The main objective~~ is ~~to make sure that there will be numerous external contributors willing to write software components adapted to the different kinds of existing sources of content~~.		This working group is open ended.

	== ~~Group animation~~ ==		=== Expected outcomes ===
			The main expected outcomes of the SODI working group are listed below:

	~~== Resources ==~~		''APIs for discovering and tracking origins''
			The SODI working group will define and evolve, in collaboration with the Software Heritage core team, standard APIs for software components that can be plugged in the Software Heritage infrastructure to track a (class of) origin(s). Whenever possible, proactive mechanisms for informing the Software Heritage infrastructure of content updates, like event feeds, will be preferred to approaches that require periodic polling of an origin.

	~~Mailing list: [https://sympa~~.~~inria.fr/sympa/info/sodi-wg-swh sodi-wg-swh]~~		''Adoption''
			The SODI working group will strive, whenever possible, to have such components incorporated in the upstream code of the forges, and adopted widely.

			''Awareness''
			The SODI working group will establish the relevant connections in order to raise awareness among all the interested parties.

			=== Milestones ===
			A first set of requirements for the API, and a preliminary draft of the API is
			expected to emerge from the work performed to ensure that Inria's own forge(s)
			are properly tracked in the Software Heritage architecture.

			=== Related working groups ===
			This working group is related to: [[Modeling_and_Ingesting_Version_control_systems \| Modeling and Ingesting Version control systems (MIV)]]

			== Team contact(s) ==
			* Roberto Di Cosmo

			* Stefano Zacchiroli

			== Documents ==
			Documents produced by the working group will be listed in this section.

			== Connections ==
			Active or planned connections to other initiatives, and activities, will be listed in this section.

			== Infrastructure ==
			=== Mailing list ===
			* https://sympa.inria.fr/sympa/info/sodi-wg-swh


	[[Category: technical]]		[[Category: technical]]
	[[Category: ingestion]]		[[Category: ingestion]]

RobertoDiCosmo: /* Charter */

2016-03-20T17:06:06Z

Charter

← Older revision		Revision as of 17:06, 20 March 2016
Line 1:		Line 1:
	== Charter ==		== Charter ==

	This working group is in charge of the issues related to adding and updating external sources of content to the archive.		This working group, codename SODI, is in charge of the issues related to adding and updating external sources of content to the archive.
	It is expected to work both on technical and non technical aspects		It is expected to work both on technical and non technical aspects