 | TurboSFV - Blog | | | |
|
TurboSFV | 
2026-02-07 13:29:18 |
| TurboSFV v11.00 - Multipart checksum files |
TurboSFV v11.00 - Multipart checksum files
Notes to TurboSFV v11.00:
TurboSFV version 11 introduces multipart checksum files: Instead of one big checksum file, hash values are written to many smaller files, each with a number of hash values to be specified. Thus, the hash file creation job is split into
multiple parts: During each part, a list of files will be created, checksums for these files will be calculated and written to a checksum file with a specific part number. Then, the job continues exactly where it paused, with the next
part.
As behind each file in the list there is a data structure, which takes a certain amount of memory (for example for the file name or the calculated hash value), a huge list of files consumes a huge amount of memory. In worst case, this can
result in an out-of-memory error, which would probably make the previous extensive work useless. So the idea is to provide a way to avoid these possible problems, resulting in to split the output into smaller
pieces.
Another advantage is, that for a validation of checksums for a huge amount of files, this work must not be done all in once: We simple take one part of the checksum files, validate the checksums and continue, whenever it suits fine. In
case of validation errors, then a smaller file list makes it easier to handle that error, for example, if a file needs to be replaced with the
original.
Multipart checksum files are offered by the TurboSFV command-line versions, which are delivered with TurboSFV CE, PE and XE. Technically, a new /m switch signals the application, that it has to handle multipart checksum files. The
command-line version "create" expects a value together with the /m switch, specifying the number of checksums per hash file as a power of 10. For example, "/m:3" results in 1.000 checksums per file, while "/m:6" means 10^6 = 1.000.000 checksums.
The appropriate part number is automatically added to the name of the hash file, together with a separator, for example
"_1".
On the other hand, the command-line version "validate" can be instructed with /m switch (without value) to treat the passed checksum file as the first part of a multipart checksum file. Once finished with one part, it automatically tries
to continue with the next part, via incrementing the part number, until there are no more checksum files
available.
Also the service application, included in TurboSFV XE, can handle multipart checksum files: Similar settings are available in the job details for creation, validation and analysis jobs, which can be configured from within the main program.
The difference is, that the service application runs in the background as a service, without the need to have a user logged
on.
Summary: If we have to transfer many files - we are talking about millions of files - then multipart checksum files can really simplify the task, by splitting the job into smaller pieces. This helps during the creation of hash files, but
also for the validation of hash values. Finally, it helps to successfully finish the
task.
Feel free to add a comment regarding this new version here.
|
TurboSFV
Cologne, Germany |
|
|
|
|
|
|
|
|
