User:StefanoZacchiroli/Content deduplication

From Software Heritage Wiki

< User:StefanoZacchiroli

Revision as of 12:12, 7 January 2018 by StefanoZacchiroli (talk | contribs)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Jump to navigation Jump to search

Some experiments on deduplicating contents at sub-file granularity using Rabin fingerprints.

Linux kernel, Git repo

origin: git.kernel.org, on 2018-01-06
1.653.941 content blobs, for a total of 19 GB (compressed)

test 1

Rabin fingerprint parameters:

prime: 3
window_size: 48 KB
min_block_size: 2 KB
avg_block_size: 8 KB
max_block_size: 64 KB

Retrieved from "https://wiki.softwareheritage.org/index.php?title=User:StefanoZacchiroli/Content_deduplication&oldid=748"