Seirl at 15:56, 16 January 2018

2018-01-16T15:56:44Z

← Older revision		Revision as of 15:56, 16 January 2018
Line 1:		Line 1:
			* '''WARNING: out of date draft blueprint'''
			* '''INFO: the Software Heritage implementation of the vault is now staying in its own repository with an up to date documentation, in [https://forge.softwareheritage.org/source/swh-vault/ swh-vault]'''

	= Software Heritage Vault =		= Software Heritage Vault =

Seirl: Created page with "= Software Heritage Vault = Software source code '''objects'''---e.g., individual source code files, tarballs, commits, tagged releases, etc.---are stored in the Software Her..."

2018-01-16T15:39:34Z

Created page with "= Software Heritage Vault = Software source code '''objects'''---e.g., individual source code files, tarballs, commits, tagged releases, etc.---are stored in the Software Her..."

New page

= Software Heritage Vault =

Software source code '''objects'''---e.g., individual source code files, tarballs, commits, tagged releases, etc.---are stored in the Software Heritage (SWH) Archive in fully deduplicated form. That allows direct access to individual artifacts but require some preparation, usually in the form of collecting and assembling multiple artifacts in a single '''bundle''', when fast access to a set of related artifacts (e.g., the snapshot of a VCS repository, the archive corresponding to a Git commit, or a specific software release as a zip archive) is required.

The '''Software Heritage Vault''' is a cache of pre-built source code bundles which are assembled opportunistically retrieving objects from the Software Heritage Archive, can be accessed efficiently, and might be garbage collected after a long period of non-use.

== Requirements ==

* '''Shared cache'''

The vault is a cache shared among the various origins that the SWH archive tracks. If the same bundle, originally coming from different origins, is requested, a single entry for it in the cache shall exist.

* '''Efficient retrieval'''

Where supported by the desired access protocol (e.g., HTTP) it should be possible for the vault to serve bundles efficiently (e.g., as static files served via HTTP, possibly further proxied/cached at that level). In particular, this rules out building bundles on the fly from the archive DB.

== API ==

All URLs below are meant to be mounted at API root, which is currently at https://archive.softwareheritage.org/api/1/. Unless otherwise stated, all API endpoints respond on HTTP GET method.

== Object identification ==

The vault stores bundles corresponding to different kinds of objects. The following object kinds are supported:

* directories
* revisions
* repository snapshots

The URL fragment <code>:objectkind/:objectid</code> is used throughout the vault API to fully identify vault objects. The syntax and meaning of :objectid for the different object kinds is detailed below.

=== Directories ===

* object kind: directory
* URL fragment: directory/:sha1git

where :sha1git is the directory ID in the SWH data model.

=== Revisions ===

* object kind: revision
* URL fragment: revision/:sha1git

where :sha1git is the revision ID in the SWH data model.

=== Repository snapshots ===

* object kind: snapshot
* URL fragment: snapshot/:sha1git

where :sha1git is the snapshot ID in the SWH data model. ('''TODO''' repository snapshots don't exist yet as first-class citizens in the SWH data model; see References below.)

== Cooking ==

Bundles in the vault might be ready for retrieval or not. When they are not, they will need to be '''cooked''' before they can be retrieved. A cooked bundle will remain around until it expires; at that point it will need to be cooked again before it can be retrieved. Cooking is idempotent, and a no-op in between a previous cooking operation and expiration.

To cook a bundle:

* POST /vault/:objectkind/:objectid

Request body: '''TODO''' something here in a JSON payload that would allow notifying the user when the bundle is ready.

Response: 201 Created

== Retrieval ==

* GET /vault/:objectkind

(paginated) list of all bundles of a given kind available in the vault; see Pagination. Note that, due to cache expiration, objects might disappear between listing and subsequent actions on them.

Examples:

* GET /vault/directory
* GET /vault/revision
* GET /vault/:objectkind/:objectid

Retrieve a specific bundle from the vault.

Response:

* 200 OK: bundle available; response body is the bundle
* 404 Not Found: missing bundle; client should request its preparation (see Cooking)

== References ==

* [https://wiki.softwareheritage.org/index.php?title=User:StefanoZacchiroli/Repository_snapshot_objects Repository snapshot objects]
* Amazon Web Services, [http://docs.aws.amazon.com/amazonglacier/latest/dev/amazon-glacier-api.html API Reference for Amazon Glacier]; specifically [http://docs.aws.amazon.com/amazonglacier/latest/dev/job-operations.html Job Operations]

= TODO =

* '''TODO''' pagination using HATEOAS
* '''TODO''' authorization: the cooking API should be somehow controlled to avoid obvious abuses (e.g., let's cache everything)
* '''TODO''' finalize repository snapshot proposal

[[Category:Blueprint]]

Vault Blueprint - Revision history

Seirl at 15:56, 16 January 2018

Seirl: Created page with "= Software Heritage Vault = Software source code '''objects'''---e.g., individual source code files, tarballs, commits, tagged releases, etc.---are stored in the Software Her..."