Difference between revisions of "User:StefanoZacchiroli/Content deduplication"
(→Rabin fingerprints) |
|||
| Line 10: | Line 10: | ||
== Rabin fingerprints == | == Rabin fingerprints == | ||
| − | + | * Approach: use [https://en.wikipedia.org/wiki/Rabin_fingerprint Rabin fingerprints] | |
| + | * Implementation: [https://forge.softwareheritage.org/rDSNIPe26a6cee53d0a0e748f3f3c9b0934477eaf25b5b swh-dedup-blocks.py] | ||
=== test 1: linux git === | === test 1: linux git === | ||
Revision as of 13:02, 7 January 2018
Some experiments on deduplicating contents at sub-file granularity.
Contents
Datasets
Linux kernel, Git repo
- origin: git.kernel.org, on 2018-01-06
- 1.653.941 content blobs, for a total of 19 GB (compressed)
Rabin fingerprints
- Approach: use Rabin fingerprints
- Implementation: swh-dedup-blocks.py
test 1: linux git
Rabin fingerprint parameters:
- prime: 3
- window_size: 48 KB
- min_block_size: 2 KB
- avg_block_size: 8 KB
- max_block_size: 64 KB
Results:
- original size (uncompressed): 55.89 GB
- dedup chunk size (uncompressed): 19.87 GB (35.55%)
