Difference between revisions of "User:StefanoZacchiroli/Content deduplication"

From Software Heritage Wiki
Jump to navigation Jump to search
Line 1: Line 1:
 
Some experiments on deduplicating contents at sub-file granularity.
 
Some experiments on deduplicating contents at sub-file granularity.
  
== Linux kernel, Git repo ==
+
== Datasets ==
 +
 
 +
=== Linux kernel, Git repo ===
  
 
* origin: git.kernel.org, on 2018-01-06
 
* origin: git.kernel.org, on 2018-01-06
Line 10: Line 12:
 
Use [https://en.wikipedia.org/wiki/Rabin_fingerprint Rabin fingerprints].
 
Use [https://en.wikipedia.org/wiki/Rabin_fingerprint Rabin fingerprints].
  
=== test 1 ===
+
=== test 1: linux git ===
  
 
Rabin fingerprint parameters:
 
Rabin fingerprint parameters:

Revision as of 12:53, 7 January 2018

Some experiments on deduplicating contents at sub-file granularity.

Datasets

Linux kernel, Git repo

  • origin: git.kernel.org, on 2018-01-06
  • 1.653.941 content blobs, for a total of 19 GB (compressed)

Rabin fingerprints

Use Rabin fingerprints.

test 1: linux git

Rabin fingerprint parameters:

  • prime: 3
  • window_size: 48 KB
  • min_block_size: 2 KB
  • avg_block_size: 8 KB
  • max_block_size: 64 KB

Results:

  • original size (uncompressed): 55.89 GB
  • dedup chunk size (uncompressed): 19.87 GB (35.55%)

References