Difference between revisions of "User:StefanoZacchiroli/Content deduplication"
(→Random sample 0.1%) |
(→Rabin fingerprint chunking) |
||
| Line 61: | Line 61: | ||
=== test 4 === | === test 4 === | ||
| − | + | Dataset: random sample 0.1% | |
| − | + | Rabin fingerprint parameters: | |
| − | + | * prime: 3 | |
| − | + | * window_size: 48 KB | |
| − | + | * chunk size (min/avg/max): 512 B / 1 KB / 8 KB | |
| − | |||
| − | + | Results: | |
| − | + | * average chunk size (effective): 1.56 KB | |
| − | + | * dedup chunk size (uncompressed): '''244.09 GB (85.05%)''' | |
== References == | == References == | ||
Revision as of 17:40, 17 January 2018
Some experiments on deduplicating contents at sub-file granularity.
Contents
Datasets
Linux kernel, Git repo
- origin: git.kernel.org, on 2018-01-06
- 1.653.941 content blobs, for a total of 19 GB (compressed)
- original size (uncompressed): 55.89 GB
Random sample 0.1%
- random sample of Software Heritage contents
- 0.1%, Bernoulli sampling; 4.111.582 contents
- random sample done on 2018-01-09
Rabin fingerprint chunking
- Approach: use Rabin fingerprints as in LBFS
- Implementation: swh-dedup-blocks.py
test 1
Dataset: linux.git
Rabin fingerprint parameters:
- prime: 3
- window_size: 48 KB
- chunk size (min/avg/max): 2 KB / 8 KB / 64 KB
Results:
- average chunk size (effective): 9.37 KB
- dedup chunk size (uncompressed): 19.87 GB (35.55%)
test 2
Dataset: linux.git
Rabin fingerprint parameters:
- prime: 3
- window_size: 48 KB
- chunk size (min/avg/max): 512 B / 2 KB / 8 KB
Results:
- average chunk size (effective): 2.86 KB
- dedup chunk size (uncompressed): 9.09 GB (16.26%)
test 3
Dataset: linux.git
Rabin fingerprint parameters:
- prime: 3
- window_size: 48 KB
- chunk size (min/avg/max): 512 B / 1 KB / 8 KB
Results:
- average chunk size (effective): 1.72 KB
- dedup chunk size (uncompressed): 6.49 GB (11.60%)
test 4
Dataset: random sample 0.1%
Rabin fingerprint parameters:
- prime: 3
- window_size: 48 KB
- chunk size (min/avg/max): 512 B / 1 KB / 8 KB
Results:
- average chunk size (effective): 1.56 KB
- dedup chunk size (uncompressed): 244.09 GB (85.05%)
