User:StefanoZacchiroli/Content deduplication: Difference between revisions
Jump to navigation
Jump to search
No edit summary |
No edit summary |
||
| Line 1: | Line 1: | ||
Some experiments on deduplicating contents at sub-file granularity. | Some experiments on deduplicating contents at sub-file granularity. | ||
== Linux kernel, Git repo == | == Datasets == | ||
=== Linux kernel, Git repo === | |||
* origin: git.kernel.org, on 2018-01-06 | * origin: git.kernel.org, on 2018-01-06 | ||
| Line 10: | Line 12: | ||
Use [https://en.wikipedia.org/wiki/Rabin_fingerprint Rabin fingerprints]. | Use [https://en.wikipedia.org/wiki/Rabin_fingerprint Rabin fingerprints]. | ||
=== test 1 === | === test 1: linux git === | ||
Rabin fingerprint parameters: | Rabin fingerprint parameters: | ||
Revision as of 12:53, 7 January 2018
Some experiments on deduplicating contents at sub-file granularity.
Datasets
Linux kernel, Git repo
- origin: git.kernel.org, on 2018-01-06
- 1.653.941 content blobs, for a total of 19 GB (compressed)
Rabin fingerprints
Use Rabin fingerprints.
test 1: linux git
Rabin fingerprint parameters:
- prime: 3
- window_size: 48 KB
- min_block_size: 2 KB
- avg_block_size: 8 KB
- max_block_size: 64 KB
Results:
- original size (uncompressed): 55.89 GB
- dedup chunk size (uncompressed): 19.87 GB (35.55%)