Seirl: Created page with "== Using Amazon Athena == The ''Software Heritage graph dataset'' is available as a public dataset in [https://aws.amazon.com/athena/ Amazon Athena]. === Setup === In order..."

2019-03-11T18:09:57Z

Created page with "== Using Amazon Athena == The ''Software Heritage graph dataset'' is available as a public dataset in [https://aws.amazon.com/athena/ Amazon Athena]. === Setup === In order..."

New page

== Using Amazon Athena ==

The ''Software Heritage graph dataset'' is available as a public dataset in [https://aws.amazon.com/athena/ Amazon Athena].

=== Setup ===

In order to query the dataset using Athena, you will first need to [https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/ create an AWS account and setup billing].

Once your AWS account is ready, you will need to install a few dependencies on your machine:

* Python 3
* The [https://docs.aws.amazon.com/cli/index.html aws cli]
* The [https://boto3.amazonaws.com/v1/documentation/api/latest/index.html boto3 Python package]

On Debian, the dependencies can be installed with the following commands:

<pre>sudo apt install python3 python3-boto3 awscli</pre>
Once the dependencies are installed, run:

<pre>aws configure</pre>

and add your AWS Access Key ID and your AWS Secret Access Key, to give Python access to your AWS account.

=== Create the tables ===

To import the schema of the dataset into your account, download the scripts from the [https://annex.softwareheritage.org/public/dataset/swh-graph-2019-01-28/athena/ athena/] folder, then run the following command:

<pre>./gen_schema.py</pre>

This will create the required tables in your AWS account. You can check that the tables were successfuly created by going to the [https://console.aws.amazon.com/athena/home Amazon Athena console] and selecting the "swh" database.

=== Run queries ===

From the console, once you have selected the "swh" database, you can directly run queries from the Query Editor.

Here is an example query that computes the most frequent file names in the archive:

<pre>SELECT FROM_UTF8(name, '?') AS name,
COUNT(DISTINCT target) AS cnt
FROM directory_entry_file
GROUP BY name
ORDER BY cnt DESC
LIMIT 1;</pre>
More documentation on Amazon Athena is available [https://docs.aws.amazon.com/athena/index.html here].

Graph Dataset on Amazon Athena - Revision history

Seirl: Created page with "== Using Amazon Athena == The ''Software Heritage graph dataset'' is available as a public dataset in [https://aws.amazon.com/athena/ Amazon Athena]. === Setup === In order..."