User:StefanoZacchiroli/Content deduplication: Difference between revisions
Jump to navigation
Jump to search
m (StefanoZacchiroli moved page User:StefanoZacchiroli/Scratch/Content deduplication to User:StefanoZacchiroli/Content deduplication) |
No edit summary |
||
| Line 13: | Line 13: | ||
* Implementation: [https://forge.softwareheritage.org/rDSNIP359fea8207e69fee3ca02c41b513cfebc6096d0f swh-dedup-blocks.py] | * Implementation: [https://forge.softwareheritage.org/rDSNIP359fea8207e69fee3ca02c41b513cfebc6096d0f swh-dedup-blocks.py] | ||
=== test 1: linux git | === test 1 === | ||
Dataset: linux.git | |||
Rabin fingerprint parameters: | Rabin fingerprint parameters: | ||
| Line 23: | Line 25: | ||
Results: | Results: | ||
* average chunk size (effective): 9.37 KB | |||
* original size (uncompressed): 55.89 GB | * original size (uncompressed): 55.89 GB | ||
* dedup chunk size (uncompressed): 19.87 GB (35.55%) | * dedup chunk size (uncompressed): 19.87 GB (35.55%) | ||
=== test 2 === | |||
Dataset: linux.git | |||
Rabin fingerprint parameters: | |||
* prime: 3 | |||
* window_size: 48 KB | |||
* min_block_size: 512 B | |||
* avg_block_size: 2 KB | |||
* max_block_size: 8 KB | |||
Results: | |||
* average chunk size (effective): 5.07 KB | |||
* original size (uncompressed): 55.89 GB | |||
* dedup chunk size (uncompressed): 16.19 GB (28.96%) | |||
== References == | == References == | ||
Revision as of 10:18, 8 January 2018
Some experiments on deduplicating contents at sub-file granularity.
Datasets
Linux kernel, Git repo
- origin: git.kernel.org, on 2018-01-06
- 1.653.941 content blobs, for a total of 19 GB (compressed)
Rabin fingerprints
- Approach: use Rabin fingerprints
- Implementation: swh-dedup-blocks.py
test 1
Dataset: linux.git
Rabin fingerprint parameters:
- prime: 3
- window_size: 48 KB
- min_block_size: 2 KB
- avg_block_size: 8 KB
- max_block_size: 64 KB
Results:
- average chunk size (effective): 9.37 KB
- original size (uncompressed): 55.89 GB
- dedup chunk size (uncompressed): 19.87 GB (35.55%)
test 2
Dataset: linux.git
Rabin fingerprint parameters:
- prime: 3
- window_size: 48 KB
- min_block_size: 512 B
- avg_block_size: 2 KB
- max_block_size: 8 KB
Results:
- average chunk size (effective): 5.07 KB
- original size (uncompressed): 55.89 GB
- dedup chunk size (uncompressed): 16.19 GB (28.96%)