You are not logged in.

#1 2017-11-10 23:27:03

Martin
Member
From: Stockholm, Sweden
Registered: 2015-10-01
Posts: 161
Website

Creating a compressed archive -- what to use?

Earlier tonight I was about to archive the content of a folder when it struck me that maybe there is a 'best' choice other than one of the usual suspects such as zip or tar.gz. A quick search showed me that there is no shortage of alternatives.

What is best considering:

* Resulting file size
* Portability
* ? what else should one consider?

/Martin


"Problems worthy of attack
prove their worth by hitting back."
Piet Hein

Offline

#2 2017-11-10 23:30:37

Sector11
Tpyo Knig
From: 77345 ¡#
Registered: 2015-08-20
Posts: 5,083
Website

Re: Creating a compressed archive -- what to use?

Unarchiving.  I've had some fail.


BunsenLabs Forum Rules ---== I'm a Conky 1.9'er ==---
System:    Host: s12 Kernel: 3.16.0-4-amd64 x86_64 (64 bit gcc: 4.8.4)
Desktop: Openbox 3.5.2 dm: (startx) Distro: Debian GNU/Linux 8

Offline

#3 2017-11-10 23:31:44

Head_on_a_Stick
Moderator
From: London
Registered: 2015-09-29
Posts: 7,383
Website

Re: Creating a compressed archive -- what to use?

The best compression algorithm available in Debian seems to be https://packages.debian.org/stretch/lrzip

See https://lkml.org/lkml/2011/6/4/23


"The first principle is that you must not fool yourself - and you are the easiest person to fool" — Richard Feynman

Forum Rules   •   How to report a problem   •   bunsenlabs(7)

Offline

#4 2017-11-10 23:55:13

twoion
ほやほや
Registered: 2015-08-10
Posts: 1,895
Website

Re: Creating a compressed archive -- what to use?

Martin wrote:

* Resulting file size
* Portability
* ? what else should one consider?

Just some thoughts.

Resulting file size: Most compression formats are fast to decompress. The greatest difference is the time/complexity it takes to compress. Usually,  gzip or deflate yield already acceptable results and are quite fast, bzip2 or lzma/xz may achieve marginally better compression ratios but take significantly more time. lz4/lzo are as fast or even faster than gzip, but may yield worse compression ratios.

Usability + file size: Depending on what you want to do, I'd suggest to chose an archiving format that supports random access, which is useful when using large archives (like >100G/>1T) . tar does not! Then again, tar is good at archiving Unixy file systems by supporting stuff like --numeric-owner. For a single folder, ZIP is probably a good enough, and it supports random access, 7zip with LZMA seems to support random access too.

Speed: Not all compressors support parallel compression, and if they do, may lose portability with the standard (non-parallelized) decompressor.

Portability: lzo/lz4 are probably the least portable. But given that 7zip is available on all major platforms, portability shouldn't be an issue with any format you pick. Just don't use RAR (although it's a good  compressor, the unrar utility does not implement the full featureset of upstream rar (WinRAR)).

Integrity: Back up to 2 locations, and store some checksums external to the archive too. Some file formats recover better from corruptions than others (zip & rar tend to be very good at this due to containing separately compressed members ("directory formats"), no idea about 7z or gzip etc, but I think this is a very interesting point worth researching). Good guards against corruption of data at rest (bit rot) are extra copies and parity (RAID, mostly).

Last edited by twoion (2017-11-10 23:55:49)


Where time just endlessly spins…

Online

#5 2017-11-11 09:15:40

ohnonot
...again
Registered: 2015-09-29
Posts: 2,218
Website

Re: Creating a compressed archive -- what to use?

for portability, i always use zip.
it doesn't compress so well compared to, erm, .7za iirc.
but in addition to being well supported on all platforms, users also instantly recognize it. everybody knows what "a zip file" is.

edit:
if portability is no issue, i still have 2 aspects to consider: speed and compression.
usually, the better the compression, the more cpu cycles it takes to de/compress.
most utilities default to something that tries to be a perfect compromise.

Last edited by ohnonot (2017-11-11 09:17:48)

Offline

Board footer

Powered by FluxBB