Difference between revisions of "User:StefanoZacchiroli/Content deduplication"
Jump to navigation
Jump to search
m (StefanoZacchiroli moved page User:StefanoZacchiroli/Scratch/Content deduplication to User:StefanoZacchiroli/Content deduplication) |
|||
| Line 13: | Line 13: | ||
* Implementation: [https://forge.softwareheritage.org/rDSNIP359fea8207e69fee3ca02c41b513cfebc6096d0f swh-dedup-blocks.py] | * Implementation: [https://forge.softwareheritage.org/rDSNIP359fea8207e69fee3ca02c41b513cfebc6096d0f swh-dedup-blocks.py] | ||
| − | === test 1: linux git | + | === test 1 === |
| + | |||
| + | Dataset: linux.git | ||
Rabin fingerprint parameters: | Rabin fingerprint parameters: | ||
| Line 23: | Line 25: | ||
Results: | Results: | ||
| + | * average chunk size (effective): 9.37 KB | ||
* original size (uncompressed): 55.89 GB | * original size (uncompressed): 55.89 GB | ||
* dedup chunk size (uncompressed): 19.87 GB (35.55%) | * dedup chunk size (uncompressed): 19.87 GB (35.55%) | ||
| + | |||
| + | === test 2 === | ||
| + | |||
| + | Dataset: linux.git | ||
| + | |||
| + | Rabin fingerprint parameters: | ||
| + | * prime: 3 | ||
| + | * window_size: 48 KB | ||
| + | * min_block_size: 512 B | ||
| + | * avg_block_size: 2 KB | ||
| + | * max_block_size: 8 KB | ||
| + | |||
| + | Results: | ||
| + | * average chunk size (effective): 5.07 KB | ||
| + | * original size (uncompressed): 55.89 GB | ||
| + | * dedup chunk size (uncompressed): 16.19 GB (28.96%) | ||
== References == | == References == | ||
Revision as of 10:18, 8 January 2018
Some experiments on deduplicating contents at sub-file granularity.
Datasets
Linux kernel, Git repo
- origin: git.kernel.org, on 2018-01-06
- 1.653.941 content blobs, for a total of 19 GB (compressed)
Rabin fingerprints
- Approach: use Rabin fingerprints
- Implementation: swh-dedup-blocks.py
test 1
Dataset: linux.git
Rabin fingerprint parameters:
- prime: 3
- window_size: 48 KB
- min_block_size: 2 KB
- avg_block_size: 8 KB
- max_block_size: 64 KB
Results:
- average chunk size (effective): 9.37 KB
- original size (uncompressed): 55.89 GB
- dedup chunk size (uncompressed): 19.87 GB (35.55%)
test 2
Dataset: linux.git
Rabin fingerprint parameters:
- prime: 3
- window_size: 48 KB
- min_block_size: 512 B
- avg_block_size: 2 KB
- max_block_size: 8 KB
Results:
- average chunk size (effective): 5.07 KB
- original size (uncompressed): 55.89 GB
- dedup chunk size (uncompressed): 16.19 GB (28.96%)