Difference between revisions of "User:StefanoZacchiroli/Content deduplication"
Jump to navigation
Jump to search
| Line 7: | Line 7: | ||
* origin: git.kernel.org, on 2018-01-06 | * origin: git.kernel.org, on 2018-01-06 | ||
* 1.653.941 content blobs, for a total of 19 GB (compressed) | * 1.653.941 content blobs, for a total of 19 GB (compressed) | ||
| + | * original size (uncompressed): 55.89 GB | ||
== Rabin fingerprints == | == Rabin fingerprints == | ||
| Line 26: | Line 27: | ||
Results: | Results: | ||
* average chunk size (effective): 9.37 KB | * average chunk size (effective): 9.37 KB | ||
| − | |||
* dedup chunk size (uncompressed): 19.87 GB (35.55%) | * dedup chunk size (uncompressed): 19.87 GB (35.55%) | ||
| Line 42: | Line 42: | ||
Results: | Results: | ||
* average chunk size (effective): 5.07 KB | * average chunk size (effective): 5.07 KB | ||
| − | |||
* dedup chunk size (uncompressed): 16.19 GB (28.96%) | * dedup chunk size (uncompressed): 16.19 GB (28.96%) | ||
Revision as of 10:19, 8 January 2018
Some experiments on deduplicating contents at sub-file granularity.
Datasets
Linux kernel, Git repo
- origin: git.kernel.org, on 2018-01-06
- 1.653.941 content blobs, for a total of 19 GB (compressed)
- original size (uncompressed): 55.89 GB
Rabin fingerprints
- Approach: use Rabin fingerprints
- Implementation: swh-dedup-blocks.py
test 1
Dataset: linux.git
Rabin fingerprint parameters:
- prime: 3
- window_size: 48 KB
- min_block_size: 2 KB
- avg_block_size: 8 KB
- max_block_size: 64 KB
Results:
- average chunk size (effective): 9.37 KB
- dedup chunk size (uncompressed): 19.87 GB (35.55%)
test 2
Dataset: linux.git
Rabin fingerprint parameters:
- prime: 3
- window_size: 48 KB
- min_block_size: 512 B
- avg_block_size: 2 KB
- max_block_size: 8 KB
Results:
- average chunk size (effective): 5.07 KB
- dedup chunk size (uncompressed): 16.19 GB (28.96%)