Excessive Multi-Threaded GZip Compression with GZRepack

: *squish*

SWF gaiden (side story)

This is some kind of spin-off from my previous mini project. As I said back then Deflate is everywhere. SVGZ files are just gzipped SVGs for example. I thought it would be nice to have some specific tool for SVGZs, but since it's just GZip I decided to generalize it a little bit.

It's basically a logical sub step on the way to the next SWF re-compression utility. It spawns several worker threads which invoke different ZIP utilities with different switches. E.g. with 2 cores there are 2 worker threads by default, which means it can keep up to 12 cores busy with the current default configuration (it contains 12 tasks). At the end the smallest Deflate stream is extracted and put into a GZip file.

The tool is similar to PNGOUT, OptiPNG, or PNGCrush insofar as it clearly puts the priority on size over time — except that it's for GZ/TGZ/SVGZ/etc. files instead of PNG.

Usage is similar to PNGOUT

X:\>java -jar GZRepack.jar
java -jar GZRepack.jar <inFile> [outFile]
by Jos Hirth (http://kaioa.com)

If only one parameter is provided this file will be overwritten, provided that
it's a GZip file (gz, tgz, svgz, etc.). Otherwise a new file with ".gz"
extension will be created. If the new file isn't smaller the original file
won't be overwritten.

Options:
  /t<number of threads>   Number of worker threads (automatic if omitted)
  /y                      Assume Yes at the 'overwrite file?' prompt

Examples:
java -jar GZRepack.jar in.svg out.svgz     <- writes out.svgz
java -jar GZRepack.jar foo.svgz            <- writes foo.svgz if smaller
java -jar GZRepack.jar bar.tgz             <- writes bar.tgz if smaller
java -jar GZRepack.jar baz.txt             <- writes baz.txt.gz

Example output

X:\>java -jar GZRepack.jar shapes2.svgz
[1] X:\bin\kzip /b128 X:\tmp\gzrepack_tmp1.zip X:\tmp\gzrepack_tmp2368.svg
[...snip...]
[0] task & post task done


     bytes  tool & switches
-------------------------------------------------------------------------------
   752,734  kzip /b128 %archive %file
   753,023  kzip /b256 %archive %file
   754,515  kzip /b512 %archive %file
   756,590  kzip /b1024 %archive %file
   757,313  7z a -tzip %archive %file -mx9 -mfb128 -mpass=15
   757,362  7z a -tzip %archive %file -mx9 -mfb258 -mpass=15
   757,374  7z a -tzip %archive %file -mx9 -mfb256 -mpass=15
   757,934  7z a -tzip %archive %file -mx9 -mfb64 -mpass=15
   760,753  7z a -tzip %archive %file -mx9 -mfb32 -mpass=15
   771,035  kzip /b0 %archive %file
   817,979  7z a -tzip %archive %file -mx9 -mfb16 -mpass=15
   957,722  7z a -tzip %archive %file -mx9 -mfb8 -mpass=15


in :    917,596 shapes2.svgz (GZ)
out:    752,734 shapes2.svgz
chg:   -164,862 (82.03% of original; 17.97% smaller)

That file was saved with Inkscape. Its compression speed is pretty high, but as you can see it's possible to make it quite a bit smaller if you try hard enough.

tasks.conf

# example configuration for GZRepack

# %archive = zip file placeholder (absolute path)
# %file = file placeholder (absolute path)

kzip /b0 %archive %file
kzip /b128 %archive %file
kzip /b256 %archive %file
kzip /b512 %archive %file
kzip /b1024 %archive %file
7z a -tzip %archive %file -mx9 -mfb8 -mpass=15
7z a -tzip %archive %file -mx9 -mfb16 -mpass=15
7z a -tzip %archive %file -mx9 -mfb32 -mpass=15
7z a -tzip %archive %file -mx9 -mfb64 -mpass=15
7z a -tzip %archive %file -mx9 -mfb128 -mpass=15
7z a -tzip %archive %file -mx9 -mfb256 -mpass=15
7z a -tzip %archive %file -mx9 -mfb258 -mpass=15

# the last one is the post processing utility, which processes all zip files afterwards
# put DISABLE_POST_PROCESSING there if you want to skip this step

deflopt %archive

Edit this file if you want to add new tasks or remove some.

Files

gzrepack.zip (12kb — source included)

Dependencies

Directory structure

After the Zip is extracted and all additional executables are downloaded and copied over it should look like this:

GZRepack
│ GZRepack.jar
│ tasks.conf
├ bin
│   7z.exe
│   DeflOpt.exe
│   kzip.exe
└ tmp

Comments

Mon, 09/08/2008 - 06:15 — Anonymous

problem with fuzetsu

Hi, I have a problem and thought I'd bring it to the game's creator directly. I installed fuzetsu on my computer, and when I try to remove it, nothing happens.

Any fix? Thanks.

Mon, 09/08/2008 - 14:15 — jh

Re: fuzetsu

Go to the Java Webstart control panel. E.g. via Win+R then:

javaws (old versions of Java)

javaws -viewer (newer versions of Java)

Over there you can manage all cached Java Webstart applications.

If the "bin" directory of the JRE isn't in your path for whatever reason go to "C:\Program Files\Java\jre1.6.0_07\bin" (or so) and execute javaws.exe from there.

kaioa.com