You are not logged in.

#1 2020-03-15 11:48:06

Registered: 2015-08-10
Posts: 2,836

Try the ZStandard (zstd) compression method

(Only tested in Buster/Lithium and higher.)

ZStandard is a compression method developed by Facebook Inc. It depends on the inputs, but it is mostly just as effective and several times faster than XZ compression (e.g. via xz, or tar -J).

Why am I so stoked about this? I have a huge NAS storage where my document scanner stores large scan PDFs or images, which I tend to consume in Acrobat to make pretty PDFs out of. Since I don't really keep track of which scans I have already consumed and archived elsewhere, I make snapshot backups of said storage space about 4 times a year. Every time, I produced a XZ compressed tarball of roughly 1G in size; the compression takes between 15-30m on my laptop (single-threaded, but multi-threaded is not that much faster). Today, for the first time, I tried out compressing my tarballs with --zstd.

zstd is 10x faster and still compresses down to the same size. This is awesome; I'm going to use this compression method a lot more in the future.

Beware of possible defects in the format that might make it unsuitable for long-term archival storage -- it is well known that XZ has some problems in that area,  see I really haven't read that much about it.

apt-get install zstd

Then you can use

tar --zstd -cf archive.tar.zst /path/to/something
zstdcat /your/text/file.zst
zstd /file/to/compress
unzstd /file/to/compress.zst

Per aspera ad astra.


#2 2020-03-15 17:41:21

Registered: 2015-09-29
Posts: 4,638

Re: Try the ZStandard (zstd) compression method

So... let me make sure I'm getting this right:
- it is free, libre & opensource
- it is 10 times faster than XZ, but the 2 are related somehow? That part of your post is a little unclear to me...

BL quote proposals to this thread please.
how to ask smart questions | my repos / my repos | my blog
Thank you for posting direct image links!


Board footer

Powered by FluxBB