Google Summer of Code 2019/Increase archive coverage - Revision history

Archit agrawal: /* Student: */

2019-08-28T21:46:54Z

Student:

← Older revision		Revision as of 21:46, 28 August 2019
Line 11:		Line 11:
	Archit Agrawal		Archit Agrawal
	* [https://forge.softwareheritage.org/p/nahimilega/ Forge activity]		* [https://forge.softwareheritage.org/p/nahimilega/ Forge activity]
			* [https://wiki.softwareheritage.org/wiki/Google_Summer_of_Code_2019/Increase_archive_coverage/Commit_list List Of Commits]

	=== Mentors:===		=== Mentors:===

Ardumont: /* Learnings: */

2019-08-26T13:14:28Z

Learnings:

← Older revision		Revision as of 13:14, 26 August 2019
Line 51:		Line 51:
	*Multiple language integration in a python library (Used in CRAN Lister)		*Multiple language integration in a python library (Used in CRAN Lister)
	*Different programming methodologies explained to me by my mentors(eg [https://en.wikipedia.org/wiki/Test-driven_development TDD])		*Different programming methodologies explained to me by my mentors(eg [https://en.wikipedia.org/wiki/Test-driven_development TDD])
	*Work with tools; DVCS (git), issue tracker (phabricator forge), docker		*Work with tools; DVCS (git), issue tracker (phabricator forge), containerization/virtualization (docker)

	=== Activity reports:===		=== Activity reports:===

Archit agrawal: /* Description: */

2019-08-26T13:09:04Z

Description:

← Older revision		Revision as of 13:09, 26 August 2019
Line 3:		Line 3:

	=== Description:===		=== Description:===
	~~As Software Heritage works on archiving and sharing source code, one~~ of ~~the major tasks~~ is to ~~ingest~~ the ~~latest source code available in the database from time to time and from all the possible sources where you can fetch the source code using~~ listers and ~~ingest them using~~ loaders. [https://docs.softwareheritage.org/devel/swh-lister/index.html#swh-lister Listers] are components that crawl the APIs of software forges (e.g., Bitbucket, GitHub, Sourceforge, ...) and return a list of the software available in it ~~whereas [~~Loaders take a bundle of software (tarball, Git repository ...) and load it into Software Heritage, by adapting it so that it matches the archive data model. The goal of this project is to increase the archive coverage by making listers and loaders for different websites that which stores source code, so that Software Heritage can fetch as much source code as possible and store it in the database to preserve it for future generations.

			The goal of this project is to increase the archive coverage by making listers and loaders for different forges.
			[https://docs.softwareheritage.org/devel/swh-lister/index.html#swh-lister Listers] are components that crawl the APIs of software forges (e.g., Bitbucket, GitHub, Sourceforge, ...) and return a list of the software available in it. Loaders take a bundle of software (tarball, Git repository ...) and load it into Software Heritage, by adapting it so that it matches the archive data model.

	===Student: ===		===Student: ===

Ardumont: /* Learnings: */

2019-08-26T12:38:50Z

Learnings:

← Older revision		Revision as of 12:38, 26 August 2019
Line 41:		Line 41:

	=== Learnings: ===		=== Learnings: ===
	Working in Software Heritage was a wholesome experience. I got to learn a new thing almost every day. ~~It would me injustice id I say I can account all my learnings in~~ a ~~section of a blog, however here are a list of~~ few of most prominent ~~once~~:		Working in Software Heritage was a wholesome experience. I got to learn a new thing almost every day. Here is a few of the most prominent ones:
	*~~Working~~ on a huge codebase		*Work on a huge codebase
	*Plan and design before jumping to code		*Plan and design before jumping to code
	*~~Writing~~ clean and well-commented code		*Write clean and well-commented code
	*~~Difference~~ between doing projects in college and in the industry(Spoiler Alert: '''A lot''')		*Learn the difference between doing projects in college and in the industry(Spoiler Alert: '''A lot''')
	*Multiple language integration in a python library (Used in CRAN Lister)		*Multiple language integration in a python library (Used in CRAN Lister)
	*Different programming methodologies explained to me by my mentors(eg [https://en.wikipedia.org/wiki/Test-driven_development TDD])		*Different programming methodologies explained to me by my mentors(eg [https://en.wikipedia.org/wiki/Test-driven_development TDD])
	*~~Working~~ with git ~~and~~ forge		*Work with tools; DVCS (git), issue tracker (phabricator forge), docker
	*Working with Docker

	=== Activity reports:===		=== Activity reports:===

Ardumont: /* Work Done: */

2019-08-26T12:30:45Z

Work Done:

← Older revision		Revision as of 12:30, 26 August 2019
Line 27:		Line 27:
	*** [https://forge.softwareheritage.org/T1724 Maven Lister]		*** [https://forge.softwareheritage.org/T1724 Maven Lister]
	** [https://forge.softwareheritage.org/rDLS08ade29e6de0616a3964360454ab52b58c082b75 Add tests to PyPI Lister]		** [https://forge.softwareheritage.org/rDLS08ade29e6de0616a3964360454ab52b58c082b75 Add tests to PyPI Lister]
	** [https://forge.softwareheritage.org/rDLSf424f07c7e628eb7a19d25f4fdb749682d97a21f ~~Refractor~~ base tests for listers]		** [https://forge.softwareheritage.org/rDLSf424f07c7e628eb7a19d25f4fdb749682d97a21f Refactor base tests for listers]
	** [https://forge.softwareheritage.org/D1441 Add documentation on How to run a new lister]		** [https://forge.softwareheritage.org/D1441 Add documentation on How to run a new lister]
	* '''Loaders:'''		* '''Loaders:'''
Line 33:		Line 33:
	*** Ingesting source code from package managers is a process somewhat similar for all of the package managers. This calls for a common base implementation for loading content from package managers into the archive. I worked on this idea, analysed the steps required to make a loader and the implementation of present package manager loader. Came up with the plan to implement the base loader and made the pass([https://forge.softwareheritage.org/D1694 D1694], [https://forge.softwareheritage.org/D1810 D1810], [https://forge.softwareheritage.org/D1811 D1811], [https://forge.softwareheritage.org/D1812 D1812], [https://forge.softwareheritage.org/D1813 D1813], [https://forge.softwareheritage.org/D1814 D1814], [https://forge.softwareheritage.org/D1744 D1744]). However, after the recommendation from my mentor, we changed the approach to make the base loader. Instead of making the whole base loader in one go, we decided to break it into multiple steps(3 steps) and follow the incremental approach.		*** Ingesting source code from package managers is a process somewhat similar for all of the package managers. This calls for a common base implementation for loading content from package managers into the archive. I worked on this idea, analysed the steps required to make a loader and the implementation of present package manager loader. Came up with the plan to implement the base loader and made the pass([https://forge.softwareheritage.org/D1694 D1694], [https://forge.softwareheritage.org/D1810 D1810], [https://forge.softwareheritage.org/D1811 D1811], [https://forge.softwareheritage.org/D1812 D1812], [https://forge.softwareheritage.org/D1813 D1813], [https://forge.softwareheritage.org/D1814 D1814], [https://forge.softwareheritage.org/D1744 D1744]). However, after the recommendation from my mentor, we changed the approach to make the base loader. Instead of making the whole base loader in one go, we decided to break it into multiple steps(3 steps) and follow the incremental approach.
	**'''[https://forge.softwareheritage.org/D1824 GNU Loader]'''		**'''[https://forge.softwareheritage.org/D1824 GNU Loader]'''
	*** As part of the first step towards the implementation of Base Loader, GNU Loader was implemented.		*** As part of the first step towards the implementation of Base Loader, GNU Loader was implemented.

	===TO-DO:===		===TO-DO:===

Archit agrawal: /* Activity reports: */

2019-08-24T17:58:48Z

Activity reports:

← Older revision		Revision as of 17:58, 24 August 2019
Line 59:		Line 59:
	** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-06/msg00009.html Week 23 Second Week (Coding)]		** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-06/msg00009.html Week 23 Second Week (Coding)]
	** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-06/msg00016.html Week 24 Third Week (Coding)]		** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-06/msg00016.html Week 24 Third Week (Coding)]
	** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-06/msg00026.html Week 25 Fourth Week (Coding)]		** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-06/msg00026.html Week 25 Fourth Week (Coding)(Work Summary)]
	** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-06/msg00033.html Week 26 Fifth Week (First Evaluation)]		** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-06/msg00033.html Week 26 Fifth Week (First Evaluation)]
	* July 2019		* July 2019
Line 70:		Line 70:
	** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-08/msg00004.html Week 32 Eleventh Week (Coding)]		** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-08/msg00004.html Week 32 Eleventh Week (Coding)]
	** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-08/msg00008.html Week 33 Twelfth Week (Coding)]		** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-08/msg00008.html Week 33 Twelfth Week (Coding)]
	** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-08/~~msg00008~~.html Week 34 Thirteenth Week (Final Evaluation)]		** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-08/msg00011.html Week 34 Thirteenth Week (Final Evaluation)]

	== Links ==		== Links ==

Archit agrawal at 13:18, 23 August 2019

2019-08-23T13:18:07Z

← Older revision		Revision as of 13:18, 23 August 2019
Line 1:		Line 1:
	===Title: ===		===Title:===
	Increase archive coverage		'''Increase archive coverage'''

	=== Description:===		=== Description:===
Line 48:		Line 48:
	*Multiple language integration in a python library (Used in CRAN Lister)		*Multiple language integration in a python library (Used in CRAN Lister)
	*Different programming methodologies explained to me by my mentors(eg [https://en.wikipedia.org/wiki/Test-driven_development TDD])		*Different programming methodologies explained to me by my mentors(eg [https://en.wikipedia.org/wiki/Test-driven_development TDD])
	*Working on git and forge		*Working with git and forge
	*Docker		*Working with Docker
	~~===Experience:===~~

	=== Activity reports:===		=== Activity reports:===

Archit agrawal at 13:08, 23 August 2019

2019-08-23T13:08:55Z

@@ Line 1: / Line 1: @@
-* '''Title:''' Increase archive coverage
+===Title: ===
-* '''Description:''' As Software Heritage works on archiving and sharing source code, one of the major tasks is to ingest the latest source code available in the database from time to time and from all the possible sources where you can fetch the source code using listers and ingest them using loaders. Listers are components that crawl the APIs of software forges (e.g., Bitbucket, Gitorious, Sourceforge, ...) and return a list of the software available in it whereas Loaders take a bundle of software (tarball, Git repository ...) and load it into Software Heritage, by adapting it so that it matches the archive data model. The goal of this project is to increase the archive coverage by making listers and loaders for different websites that which stores source code, so that Software Heritage can fetch as much source code as possible and store it in the database to preserve it for future generations.
+Increase archive coverage
-* '''Learnings:'''
+=== Description:===
-**How to work on a huge codebase
+As Software Heritage works on archiving and sharing source code, one of the major tasks is to ingest the latest source code available in the database from time to time and from all the possible sources where you can fetch the source code using listers and ingest them using loaders. [https://docs.softwareheritage.org/devel/swh-lister/index.html#swh-lister Listers] are components that crawl the APIs of software forges (e.g., Bitbucket, GitHub, Sourceforge, ...) and return a list of the software available in it whereas [Loaders take a bundle of software (tarball, Git repository ...) and load it into Software Heritage, by adapting it so that it matches the archive data model. The goal of this project is to increase the archive coverage by making listers and loaders for different websites that which stores source code, so that Software Heritage can fetch as much source code as possible and store it in the database to preserve it for future generations.
-* '''TO-DO:'''
+===Student: ===
-* '''Activity reports:'''
+Archit Agrawal
 * May 2019
-** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-05/msg00003.html Week 20 Second Week(Community Bonding)]
+** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-05/msg00003.html Week 20 Second Week (Community Bonding)]
 ** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-05/msg00010.html Week 21 Third Week (Community Bonding)]
 ** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-05/msg00017.html Week 22 First Week (Coding)]
@@ Line 58: / Line 72: @@
 ** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-08/msg00008.html Week 33 Twelfth Week (Coding)]
 ** [https://sympa.inria.fr/sympa/arc/swh-devel/2019-08/msg00008.html Week 34 Thirteenth Week (Final Evaluation)]
 == Links ==
 * [https://forge.softwareheritage.org/source/swh-lister/    Lister source code repository]
 * [https://forge.softwareheritage.org/source/swh-loader-core/   Loader source code repository]

Archit agrawal at 17:58, 21 August 2019

2019-08-21T17:58:04Z

← Older revision		Revision as of 17:58, 21 August 2019
Line 15:		Line 15:
	**** [https://forge.softwareheritage.org/D1584 Packagist Lister]		**** [https://forge.softwareheritage.org/D1584 Packagist Lister]
	**** [https://forge.softwareheritage.org/D1610 CGit Lister]		**** [https://forge.softwareheritage.org/D1610 CGit Lister]
	*** [https://forge.softwareheritage.org/rDLS08ade29e6de0616a3964360454ab52b58c082b75 Add tests to PyPI Lister]		*** Did research on the methods that could be used to make following listers and made an implementation plan for the same
	*** [https://forge.softwareheritage.org/rDLSf424f07c7e628eb7a19d25f4fdb749682d97a21f Refractor base tests for listers]
	**[https://forge.softwareheritage.org/D1441 Add documentation on How to run a new lister*]
	*** Did research on the methods that could be used to make following listers and made an implementation plan for the same:
	**** [https://forge.softwareheritage.org/T1734 Launchpad Lister]		**** [https://forge.softwareheritage.org/T1734 Launchpad Lister]
	**** [https://forge.softwareheritage.org/T1777 Rubygem Lister]		**** [https://forge.softwareheritage.org/T1777 Rubygem Lister]
	**** [https://forge.softwareheritage.org/T1718 NuGET(.NET) Lister]		**** [https://forge.softwareheritage.org/T1718 NuGET(.NET) Lister]
	**** [https://forge.softwareheritage.org/T1724 Maven Lister]		**** [https://forge.softwareheritage.org/T1724 Maven Lister]
			*** [https://forge.softwareheritage.org/rDLS08ade29e6de0616a3964360454ab52b58c082b75 Add tests to PyPI Lister]
			*** [https://forge.softwareheritage.org/rDLSf424f07c7e628eb7a19d25f4fdb749682d97a21f Refractor base tests for listers]
			**[https://forge.softwareheritage.org/D1441 Add documentation on How to run a new lister*]
	** '''Loaders:'''		** '''Loaders:'''
	*** Base Loader		*** Base Loader
	*** GNU Loader		*** GNU Loader

	*'''Challenges:'''
	* '''Learnings:'''		* '''Learnings:'''
			**How to work on a huge codebase
			**How to plan and design before jumping to code
			**Writing clean and well-commented code
			**How are projects are made in the industry different from making projects in college(Spoiler Alert: '''A lot''')
			**Multiple language integration in a python library (Used in CRAN Lister)
			**Different programming methodologies explained to me by my mentors(eg [https://en.wikipedia.org/wiki/Test-driven_development TDD])
			**Working on git and forge
			**Docker

	* '''TO-DO:'''		* '''TO-DO:'''
	* '''Activity reports:'''		* '''Activity reports:'''
Line 55:		Line 63:

	* [https://forge.softwareheritage.org/source/swh-lister/ Lister source code repository]		* [https://forge.softwareheritage.org/source/swh-lister/ Lister source code repository]
	* [https://forge.softwareheritage.org/source/swh-loader-core// Loader source code repository]		* [https://forge.softwareheritage.org/source/swh-loader-core/ Loader source code repository]
	* see project [https://summerofcode.withgoogle.com/projects/#5658995887439872 on the GSoC portal]		* see project [https://summerofcode.withgoogle.com/projects/#5658995887439872 on the GSoC portal]

Archit agrawal at 15:04, 21 August 2019

2019-08-21T15:04:44Z

@@ Line 8: / Line 8: @@
 * '''Experience:'''
 * '''What I did:'''
 *'''Challenges:'''
 * '''Learnings:'''