Difference between revisions of "User:StefanoZacchiroli/Content deduplication"
Jump to navigation
Jump to search
| Line 1: | Line 1: | ||
Some experiments on deduplicating contents at sub-file granularity. | Some experiments on deduplicating contents at sub-file granularity. | ||
| − | == Linux kernel, Git repo == | + | == Datasets == |
| + | |||
| + | === Linux kernel, Git repo === | ||
* origin: git.kernel.org, on 2018-01-06 | * origin: git.kernel.org, on 2018-01-06 | ||
| Line 10: | Line 12: | ||
Use [https://en.wikipedia.org/wiki/Rabin_fingerprint Rabin fingerprints]. | Use [https://en.wikipedia.org/wiki/Rabin_fingerprint Rabin fingerprints]. | ||
| − | === test 1 === | + | === test 1: linux git === |
Rabin fingerprint parameters: | Rabin fingerprint parameters: | ||
Revision as of 12:53, 7 January 2018
Some experiments on deduplicating contents at sub-file granularity.
Datasets
Linux kernel, Git repo
- origin: git.kernel.org, on 2018-01-06
- 1.653.941 content blobs, for a total of 19 GB (compressed)
Rabin fingerprints
Use Rabin fingerprints.
test 1: linux git
Rabin fingerprint parameters:
- prime: 3
- window_size: 48 KB
- min_block_size: 2 KB
- avg_block_size: 8 KB
- max_block_size: 64 KB
Results:
- original size (uncompressed): 55.89 GB
- dedup chunk size (uncompressed): 19.87 GB (35.55%)