Suggestion box: source code to add/Debian derivatives

Debian derivatives

Suggestions from Debian's pabs (pabs@d.o) to inject into Software Heritage archive all the distributions listed in the Debian derivatives census:

Hi folks,

At the DebConf16 talk we discussed importing Debian derivatives source
packages into the Software Heritage archive. Below is a rough guide
about how to do it. If you want more detail I can give it in person at
DebConf16, via email or via IRC.

First download the census page to get a list of derivatives:

https://anonscm.debian.org/cgit/dex/census.git/tree/Makefile.var#n69
https://anonscm.debian.org/cgit/dex/census.git/tree/bin/get-wiki-html
https://anonscm.debian.org/cgit/dex/census.git/tree/bin/wiki-html-to-derivatives-list

Then download the text versions of their wiki pages:

https://anonscm.debian.org/cgit/dex/census.git/tree/Makefile.deriv#n51
https://anonscm.debian.org/cgit/dex/census.git/tree/bin/get-wiki-text

Then extract the sources.list files from each wiki page:

https://anonscm.debian.org/cgit/dex/census.git/tree/Makefile.deriv#n57
https://anonscm.debian.org/cgit/dex/census.git/tree/bin/wiki-text-to-sources-list

Then download the apt meta-data into a subdirectory, the APT_CONFIG
environment variable must be set otherwise apt will use system hooks:

https://anonscm.debian.org/cgit/dex/census.git/tree/Makefile.deriv#n60
https://anonscm.debian.org/cgit/dex/census.git/tree/bin/get-package-lists
https://anonscm.debian.org/cgit/dex/census.git/tree/bin/fakegpgv
https://anonscm.debian.org/cgit/dex/census.git/tree/etc/apt.conf
https://anonscm.debian.org/cgit/dex/census.git/tree/Makefile.deriv#n6

Then you will need an importer for apt repositories that is based on
sources.list input files. The compare-source-package-list script from
the derivatives census has a loop around source packages, you would
just need to add downloading them using the apt acquire system.

https://anonscm.debian.org/cgit/dex/census.git/tree/bin/compare-source-package-list#n1149
https://anonscm.debian.org/cgit/dex/census.git/tree/Makefile.deriv#n63

In addition, a significant cache of source packages from derivatives, including disappeared ones, is available on lw08.debian.org