Difference between revisions of "User:StefanoZacchiroli/Content deduplication"

From Software Heritage Wiki
Jump to navigation Jump to search
Line 5: Line 5:
 
* origin: git.kernel.org, on 2018-01-06
 
* origin: git.kernel.org, on 2018-01-06
 
* 1.653.941 content blobs, for a total of 19 GB (compressed)
 
* 1.653.941 content blobs, for a total of 19 GB (compressed)
 +
 +
== test 1 ==
 +
 +
Rabin fingerprint parameters:
 +
* prime: 3
 +
* window_size: 48 KB
 +
* min_block_size: 2 KB
 +
* avg_block_size: 8 KB
 +
* max_block_size: 64 KB

Revision as of 12:12, 7 January 2018

Some experiments on deduplicating contents at sub-file granularity using Rabin fingerprints.

Linux kernel, Git repo

  • origin: git.kernel.org, on 2018-01-06
  • 1.653.941 content blobs, for a total of 19 GB (compressed)

test 1

Rabin fingerprint parameters:

  • prime: 3
  • window_size: 48 KB
  • min_block_size: 2 KB
  • avg_block_size: 8 KB
  • max_block_size: 64 KB