Useful to split up jobs over multiple CPU cores or even multiple computers. Examples:
tar -cf - /bigdirectory | splitjob -j 4 gzip > big.tar.gz
splitjob "ssh h1 bzip2" "ssh h2 bzip2" < f > f.bz2
Homepage
Download
Recent Releases
3.228 Aug 2021 16:27
minor feature:
Robustness against lots of false magic bytes within compressed data at parallel decompression as the number of retries now might be bigger than the number of parallel jobs
3.116 Jul 2021 20:16
minor feature:
Improved support for parallel decompression of files which recursively contains other files compressed with the same method. This could happen for tar archives or disk image files.
3.008 Apr 2020 11:43
minor bugfix:
Removed predefined support for gzip parallel decompression as gzip might fail silently when parallel decompression is attempted. Marked as a stable release as latest experimental feature now has been tested some years.
2.209 Mar 2019 22:36
major feature:
Added experimental support for parallel decompression
2.111 Nov 2017 22:50
major bugfix:
Bugfix: Fixed copy-paste error in code which caused writing outside allocated buffer when output data from called program was bigger than input data. This could happen also at compression if data is already compressed. In theory bugs like this could cause more or less random behavior. In practice this bug has caused corrupted backup archives. Any users of version 2.0 should upgrade to version 2.1 to avoid this bug!
2.015 Oct 2017 20:33
major feature:
Now accepts signal SIGUSR2 to increase number of parallel running jobs and SIGUSR1 to decrease number of parallel running jobs.
1.209 Oct 2017 21:11
minor feature:
Might now be able to recover if sub process fails even if some data has been read out from the sub process.
1.131 Jan 2015 18:51
minor feature:
Saving RAM by freeing up unused RAM in child processes.
1.014 Dec 2014 14:57
documentation:
No changes since previos beta version which have now been tested for some months without any problems found.
0.9.2beta25 Aug 2014 11:36
minor bugfix:
Avoiding short reads from stdin which could cause non optimal compression performance when read block sizes got smaller than intended.
0.9beta24 Jul 2014 23:11
minor feature:
First public release