You are not logged in.

#1 2017-01-16 20:30:57

geekosupremo
Member
Registered: 2016-04-27
Posts: 168

Opinion-ation! Alt FS on a single HDD server

I'm looking for something the internet has no shortage of ... opinions. I'd prefer them to be backed by at least personal experience.

So ... I'm maintaining a small file server here at work, a simple Samba file share off of Wheezy. And another computer has become available to be a server, which means fun time experimentation!

Part of me would like for the "new" Dual Core G2020 Intel Support page, 4GB DDR3 1333, 1TB HDD to be a data secure and snapshotable.

I've looked into LVM, and while not complicated, it's not simple. I'm constantly in lvm -help to remember which command to use to get anything done. Again not "the worst" but can't shake the feeling that there has to be something that's less commands aka easier.

This brings us to the wonderful world of CoW file systems!! Namely ZFS & Btrfs.

What I know:

  • ZFS has a longer period of being 'stable'

  • Btrfs became 'stable' last year (2015)

  • Btrfs is the production FS at Facebook and is default on OpenSuse Enterprise Linux as of version 12

  • BOTH are considered 'good'

  • ZFS has a troubled history with patents & Linux

  • Btrfs is considered a 'better' Linux citizen

  • BOTH have staunch supporters that say it's has saved their bacon

What I'd like to know:

  • Day to day are the commands annoying to deal with?

  • Is snapshot management a PITA?

  • How much time is spent/wasted on 'maintainence' commands - eg Btrfs rebalancing?

  • Does this seem better than LVM & associated user-land tooling?

I'm reasonably sure that one answer is: "If you're asking the question, you don't need it." <- While this may be true exploring the experiences of others seems a worthwhile endeavor.

This is a very basic file share not fancy or large DBs involved. Just good ol' Samba to group of Windows clients.

----- Current status -----

2017.02.20

While still stuggling with exactly how much configureation I'll need/not need to do, I stumbled across something super interesting.

Apparently the default FS in FreeBSD (and I'm assuming it's available to Open & Net flavors) which is UFS has a snapshot ability, already. I stumbled across it while searching for backup functions in BSD. Near the end of last week I was churning though docs about the UFS, since I think my lil box is underpowered for ZFS, and in there I came across a mention of a built in snapshot function. Today I'm looking up more about UFS (which is a pretty interesting, albeit not 'fresh' FS) and found out FreeBSD has some super useful "in the box" on the shelf right next to 'the box' under the "useful free addons" banner, aka a pkg or port, functions for dealing with snapshots, be it ZFS or UFS!

  • snapshot(8) - is a generic, manual snapshot management tool

  • periodic-snapshot(8) - is a command,built on the aforementioned snapshot, for easy automation of snapshot maintainence

I guess it's things like this that the BSD users smugly call a "unified OS" wink

2017.02.15

Ok. I'm in configuration hell. It seems this is relatively common, based on a number of forum posts I've seen. While not a 'flaw', per say, in FreeBSD but it is a frustration. Not giving up on learning it ... but I may be pursuing a path with a shorter "to running" path, since I want to get this up and going soon.

I've revised my plan yet again. As I have dug into both ZFS and btrfs it's become clear that the assumptions that both systems make are somewhat beyond the hardware stats of my little boxes.

Could I still use them? Sure, there is no glaring red lights, but there are no green lights either. The idea of these new FS are really cool, and I wish them the best, but I think they're like a Zorg ZF-1 when all I need is a pocket knife ... well ... maybe a multi-tool.

2017.02.06

Blurg. The HDD was a "recycle" and as it turns out had a hardware fault.

Soooo we're doing it all again on the FreeBSD front.

--- Why did I go with FreeBSD over the other options?

  • The FreeBSD Handbook is written in a format that my brain digests fairly easily, therefore it's learning curve is less steep than say OpenBSD

  • ZFS has more battle testing the Btrfs -- not that it's 'better' but it's survived a bit more and is now better documented

  • While I first jumped in on Debian-land, I have a soft spot for the BSDs

  • BSD Jails are slightly more sane (to me) than the LXC abstractions. I feel like LXC is trying to do more than I need/want it to.

  • I can always fall back on Debian/Ubuntu and LVM with little to no loss

My plan is to rsync the data to be used over to the new box and then try to run it for a month as a "backup" and see if the system continues to make sense. Then move it into "main" and the current to "backup".

2017.02.03

Installed FreeBSD 11.0 - RELEASE on to the bare metal drive/computer with ZFS as the main FS. It has dawned on me that I need to stop analizing and start doing. wink

2017.01.19

Started a VM for OpenBSD & FreeBSD with the intention of running ZFS on them.

- OpenBSD has been pushed to the side, for the moment. It's just a little bit more funky that I want to deal with right now. I'm sure it's a great option, but it feels like Arch it works but the package/port management is just weird enough that I'm struggling to follow to docs for it.

- FreeBSD - Release 11 - is installed and running, used the installer to set up the ZFS. Still configuring it, but so far their handbook hasn't lead me astray.

- Contemplating a TrueOS instead of pure FreeBSD since it has "sensible" defaults for what I'm trying to get going. <-- This idea has been put aside. After spinning up the VM I realized that TrueOS is a Rolling Release, so it is likely not suitable for my goals.

Last edited by geekosupremo (2017-02-20 23:42:22)

Offline

#2 2017-01-16 21:30:13

Head_on_a_Stick
Member
From: London
Registered: 2015-09-29
Posts: 8,759
Website

Re: Opinion-ation! Alt FS on a single HDD server

Zfs is possible with Debian but only as a user-buildable custom kernel module so it is quite tricky to actually use.

Btrfs would be my choice, I have been using it as my standard GNU/Linux filesystem for over a year now.

The CoW feature is very useful to me because all of my laptops are old with dodgy batteries and they tend to hard hang quite a lot — btrfs shrugs this off and often doesn't even fsck.

By comparison, the FFS filesystem on my OpenBSD laptop really doesn't like hard shutdowns and complains long and hard afterwards.

This contrasts rather sharply with the popular opinion of btrfs as a "data-eater" smile

Of course, the plural of anecdote is not evidence and btrfs is still relatively immature, especially when compared to zfs.

I think @twoion posted a thread with some very interesting filesystem comparisons, well worth searching out.

So, for brtfs:

geekosupremo wrote:

Day to day are the commands annoying to deal with?

Not at all, very simple and logical.

The man page is long but thorough and fairly readable.

Is snapshot management a PITA?

I don't find it to be but my snapshot system is *very* simple.

How much time is spent/wasted on 'maintainence' commands - eg Btrfs rebalancing?

Oh yeah, I forgot about that...

lol

Does this seem better than LVM & associated user-land tooling?

I have no idea, I looked at LVM briefly when I was (more of) a n00b and it was far too complicated; btrfs is *much* simpler.


“Et ignotas animum dimittit in artes.” — Ovid, Metamorphoses, VIII., 18.

Forum Rules   •   How to report a problem   •   Software that rocks

Offline

#3 2017-01-16 22:18:21

geekosupremo
Member
Registered: 2016-04-27
Posts: 168

Re: Opinion-ation! Alt FS on a single HDD server

Head_on_a_Stick wrote:

I think @twoion posted a thread with some very interesting filesystem comparisons, well worth searching out.

I believe this is the relevant thread - https://forums.bunsenlabs.org/viewtopic.php?id=1694

And that is quite informative! Thanks for mentioning it!

It does still dodge the ZFS bit but is an interesting insight into Btrfs.

Offline

#4 2017-01-17 21:41:56

twoion
ほやほや
Registered: 2015-08-10
Posts: 2,459

Re: Opinion-ation! Alt FS on a single HDD server

geekosupremo wrote:
Head_on_a_Stick wrote:

I think @twoion posted a thread with some very interesting filesystem comparisons, well worth searching out.

I believe this is the relevant thread - https://forums.bunsenlabs.org/viewtopic.php?id=1694

And that is quite informative! Thanks for mentioning it!

It does still dodge the ZFS bit but is an interesting insight into Btrfs.

ZFS is something powerful but you need to understand its architecture to manage it well and in order to be able to decide whether to run it or not. It merges lots of things provided by filesystem + lvm + softraid into one coherent system, i.e. it replaces a stack of software with an integrated solution, takes extra care to integrate features well, and provides many knobs to tune plus extras. If you want to use deduplication, you need more RAM than 4G else it'll crawl.

Otherwise, I suggest you do a test setup and study-by-doing. Same for btrfs, but that is IMHO nothing I would put on a prod server for commercial use yet.

Especially for ZFS, there is lots of good documentation.


What can you see / On the horizon?

Offline

#5 2017-01-17 21:45:00

Head_on_a_Stick
Member
From: London
Registered: 2015-09-29
Posts: 8,759
Website

Re: Opinion-ation! Alt FS on a single HDD server

twoion wrote:

If you want to use deduplication, you need more RAM than 4G else it'll crawl.

Just to reinforce this point: the recommended RAM for full deduplication in zfs (which is required for all of the error-correction facilities) is 16 GiB...

yikes


“Et ignotas animum dimittit in artes.” — Ovid, Metamorphoses, VIII., 18.

Forum Rules   •   How to report a problem   •   Software that rocks

Offline

#6 2017-01-18 00:04:54

geekosupremo
Member
Registered: 2016-04-27
Posts: 168

Re: Opinion-ation! Alt FS on a single HDD server

As expected, well thought out and insightful answers!

Spinning up some VirtualBox VMs now. wink

Offline

#7 2017-01-20 23:57:41

Head_on_a_Stick
Member
From: London
Registered: 2015-09-29
Posts: 8,759
Website

Re: Opinion-ation! Alt FS on a single HDD server

A quick word of warning about zfs...

It seems that gparted and similar tools can't manage zfs storage pools properly — I had a working zpool on /dev/sda5 with my main (btrfs) partition on /dev/sda3 and after I deleted sda5 and expanded sda3 (the two are contiguous despite the numbering) I was left with a completely unbootable Arch box due to the root partition having both zfs and btrfs signatures.

The damn thing kept me out for almost 40 minutes (!) until I figured out what had happened; I used:

wipefs /dev/sda3

to spot the extraneous zfs label and then I had to rebuild the zfs module (in my BL system) and clear the traces with:

# zpool labelclear /dev/sda3

“Et ignotas animum dimittit in artes.” — Ovid, Metamorphoses, VIII., 18.

Forum Rules   •   How to report a problem   •   Software that rocks

Offline

#8 2017-02-01 16:29:12

geekosupremo
Member
Registered: 2016-04-27
Posts: 168

Re: Opinion-ation! Alt FS on a single HDD server

Thanks for the real world use input @Head_on_a_Stick !!

As I dig into this documentation more, I think that I'm still over complicating the design of this server.

While snapshots sound brilliant, I think I may resort to a more simple, and possibly more sane, Samba share with specific folders mounted as shares.

Probably some sort of rsync cron tab or maybe attic for backup system.

This s*** is complicated. hmm

It's all still worth learning ... just ... complicated.

Last edited by geekosupremo (2017-02-01 16:31:22)

Offline

#9 2017-02-01 17:51:40

iMBeCil
WAAAT?
From: Edrychwch o'ch cwmpas
Registered: 2015-09-29
Posts: 643

Re: Opinion-ation! Alt FS on a single HDD server

^Didn't saw your post until now ... let me just say that LVM snapshots are not that complicated, three commands are enough ... For example, I do semiregular snapshots with zero downtime, and store copy to the external USB HDD. Commands are:

### create (temporary) snapshot 
lvcreate -L10G -s -n /dev/VG1debian/dom4-disk-snap /dev/VG1debian/dom4-disk
# dom4-disk = actual on line volume, contains running freebsd; size is 15G
# dom4-disk-snap = temporary snapshot volume; size is 10G, as I (over)estimate that during creating 
                   backup, it won't grow larger; to be safe, make it as large as original volume 

### create actual backup with dd
dd if=/dev/VG1debian/dom4-disk-snap of=/media/backup/backup-VM/dom4-disk-2017.02.01_10.36.13.dd
# I suppose one can use other tools instead of dd; note that we make backup of above created 
# temporary 'snapshot' volume 'dom4-disk-snap'.
# '/media/backup...' is external USB HDD.
# This takes the longest time, around 400 seconds (I think it is Intel i5, 8 GB RAM, USB3).
# Also, it is possible to insert compressing of the image, but it increases time for factor ~3.

### Removing temporary snapshot ...
lvremove -f /dev/VG1debian/dom4-disk-snap

Note that at no time we stopped and/or paused original volume, or made it inaccessible!

I don't know if it helps you, but I hope you got the idea how difficult or easy it is with LVM.


Postpone all your duties; if you die, you won't have to do them ..

Offline

#10 2017-02-01 18:12:32

geekosupremo
Member
Registered: 2016-04-27
Posts: 168

Re: Opinion-ation! Alt FS on a single HDD server

Thank you so much for your practical example! It does show a simpler path if I stay with LVM.

:high fives everyone in the thread:

Offline

#11 2017-02-01 18:23:22

iMBeCil
WAAAT?
From: Edrychwch o'ch cwmpas
Registered: 2015-09-29
Posts: 643

Re: Opinion-ation! Alt FS on a single HDD server

You're welcome.

Indeed, the LVM way is not that complicated. Actually, I'm sure zfs and/or btrfs is nobrainer, too, but only after one reads an real-life example  smile What I found out during setting up those backups, is that good planning is essential. I must have spent third of time to decide what I really want, where to put backups, what to backup, how often and how automatic the backup creation should be ... and so on.

BTW, what they say about btrfs is true: it is not yet production-ready. And, as I have heard from certain guy from IT, zfs on debian really needs a huge amounts of RAM, as HoaS emphasized already.

Anyway, good luck.


Postpone all your duties; if you die, you won't have to do them ..

Offline

#12 2017-02-02 00:28:37

geekosupremo
Member
Registered: 2016-04-27
Posts: 168

Re: Opinion-ation! Alt FS on a single HDD server

Agreed, @iMBeCil. The other advantage of LVM is once it's setup, it's more like the partition system I'm already familiar with.

I'm also realizing if I really want to use the ZFS for a redundant backup aka have a second box with a "live" backup via something like sent ZFS snapshots I'll need to commit to a fuller ZFS/BSD server stack. Which isn't bad ... but it does make the learning curve steeper since I'm also adding BSD to the mix.

Blurg.

All this thinking makes Homer go ... something ... something.

Offline

#13 2017-02-02 08:09:29

Head_on_a_Stick
Member
From: London
Registered: 2015-09-29
Posts: 8,759
Website

Re: Opinion-ation! Alt FS on a single HDD server

geekosupremo wrote:

adding BSD to the mix

FreeBSD (11.0) outperforms Linux anyway so you might as well just switch wholesale devil


“Et ignotas animum dimittit in artes.” — Ovid, Metamorphoses, VIII., 18.

Forum Rules   •   How to report a problem   •   Software that rocks

Offline

#14 2017-02-02 10:07:40

iMBeCil
WAAAT?
From: Edrychwch o'ch cwmpas
Registered: 2015-09-29
Posts: 643

Re: Opinion-ation! Alt FS on a single HDD server

^ and ^^ By all means, if one has enough time to learn new stuff, it is certainly worth to 'assemble' FreeBSD based file/web server. Lots gonna be learned ... for example one will learn that Debian is much more automatic (easier?) compared to FreeBSD.

And then master the zfs ... (I wish I have enough spare time to learn zfs on FreeBSD, and set it up on production machine.)

Head_on_a_Stick wrote:

FreeBSD (11.0) outperforms Linux anyway so you might as well just switch wholesale devil

Oh, you are naughty, aren't you smile ... From this benchmark one can only conclude that the X-less servers are faster (cf. FreeBSD vs DragonFly).  tongue Not to mention page 2, where FreeBSD is worse by factor 5 on certain test. I'm sure that with some tweaking, all distros would perform nearly the same.


Postpone all your duties; if you die, you won't have to do them ..

Offline

#15 2017-02-02 17:45:31

geekosupremo
Member
Registered: 2016-04-27
Posts: 168

Re: Opinion-ation! Alt FS on a single HDD server

That is an interesting set of tests.

It seems that the FreeBSD OotB install has a better tuned network stack. I'm sure with enough tweaking they could all be more competitive but that is a good data point.

I'm sure many of the skills I've picked up from my short stint in Linux-land will translate to FreeBSD if I go that way.

:sigh: I feel decisioned out right now. neutral

Offline

#16 2017-02-03 03:45:41

tynman
Member
Registered: 2015-10-13
Posts: 80

Re: Opinion-ation! Alt FS on a single HDD server

Head-on-a-stick wrote:

Zfs is possible with Debian but only as a user-buildable custom kernel module so it is quite tricky to actually use

I'm not sure what a " user-buildable custom kernel module" is, but I installed zfs in Debian 8 with:

> wget http://archive.zfsonlinux.org/debian/pool/main/z/zfsonlinux/zfsonlinux_6_all.deb
> dpkg -i zfsonlinux_6_all.deb
> apt-get update
> apt-get install lsb-release #(this step did nothing)
> apt-get install debian-zfs

It sets itself up to run as a service and has been pretty straightforward to work with.

Offline

#17 2017-02-03 09:31:47

Head_on_a_Stick
Member
From: London
Registered: 2015-09-29
Posts: 8,759
Website

Re: Opinion-ation! Alt FS on a single HDD server

^ That method of installation is no longer required because all of the necessary packages can now be installed directly from the Debian repositories.

I'm not sure what a " user-buildable custom kernel module" is

The zfs kernel module that is required to work with the filesystem cannot be included with the kernel and cannot be supplied directly by Debian due to licensing incompatibilities.

To install the module, the user must download the dkms package along with the appropriate kernel headers and then build it locally (just like Broadcom wireless modules and the like):

# apt install module-assistant
# m-a prepare
# apt install zfs-dkms zfsutils-linux # etc

https://packages.debian.org/search?keyw … ection=all


“Et ignotas animum dimittit in artes.” — Ovid, Metamorphoses, VIII., 18.

Forum Rules   •   How to report a problem   •   Software that rocks

Offline

#18 2017-02-03 10:41:10

Head_on_a_Stick
Member
From: London
Registered: 2015-09-29
Posts: 8,759
Website

Re: Opinion-ation! Alt FS on a single HDD server

geekosupremo wrote:

How much time is spent/wasted on 'maintainence' commands - eg Btrfs rebalancing?

For Btrfs, a systemd .timer unit can be used to run `btrfs scrub` automatically every month.

These unit files are derived from the Arch btrfs-progs package, I have adapted them for Debian:

# /etc/systemd/system/btrfs-scrub@.timer

[Unit]
Description=Monthly Btrfs scrub on %f

[Timer]
OnCalendar=monthly
AccuracySec=1d
Persistent=true

[Install]
WantedBy=multi-user.target
# /etc/systemd/system/btrfs-scrub@.service

[Unit]
Description=Btrfs scrub on %f

[Service]
Nice=19
IOSchedulingClass=idle
ExecStart=/sbin/btrfs scrub start -B %f

Simply create the .timer & .service unit files at the locations indicated above and then enable the .timer with:

sudo systemctl enable btrfs-scrub@-.timer

Then check the status with:

systemctl list-timers

The output should look something like this:

TheLab: ~ # systemctl list-timers --no-pager
NEXT                         LEFT                LAST                         PASSED        UNIT                         ACTIVATES
Sat 2017-02-04 09:36:36 GMT  22h left            Fri 2017-02-03 09:36:36 GMT  1h 1min ago   systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Mon 2017-02-06 00:00:00 GMT  2 days left         Wed 2017-02-01 19:10:37 GMT  1 day 15h ago fstrim.timer                 fstrim.service
Wed 2017-03-01 00:00:00 GMT  3 weeks 4 days left n/a                          n/a           btrfs-scrub@-.timer       btrfs-scrub@-.service

3 timers listed.
Pass --all to see loaded but inactive timers, too.

smile

Last edited by Head_on_a_Stick (2017-02-04 10:58:14)


“Et ignotas animum dimittit in artes.” — Ovid, Metamorphoses, VIII., 18.

Forum Rules   •   How to report a problem   •   Software that rocks

Offline

#19 2017-02-03 23:25:09

geekosupremo
Member
Registered: 2016-04-27
Posts: 168

Re: Opinion-ation! Alt FS on a single HDD server

@HoaS you may be ... a Head on a Stick ... but you're a splendid one! wink

For what it's worth I've jumped in on testing on "bare metal" with FreeBSD 11.0-RELEASE & ZFS. I have bootable media for all of the alternative installs I would want/need. Also I realized I didn't have enough spare space on my main box to house the needed test files on a VM. ops

I'll be updating the OP as I move forward. You all are fantastic for offering your INVALUABLE insights and real world experiences. I think this maybe the best technical forum I've ever participated in.

I'm done with over analizing ... time to do something!

Offline

#20 2017-02-16 00:38:19

geekosupremo
Member
Registered: 2016-04-27
Posts: 168

Re: Opinion-ation! Alt FS on a single HDD server

I've revised my plan yet again.

Offline

#21 2017-04-14 17:15:45

geekosupremo
Member
Registered: 2016-04-27
Posts: 168

Re: Opinion-ation! Alt FS on a single HDD server

There is no shortage of information about these COW file systems out on the interwebs. Most of it is geared toward 'enterprise' class problems, aka the issues that Facebook (btrfs) and Oracle (ZFS) deal with.

But this guide - [Aaron Toponce - pthree.org] - found on AskUbuntu maybe the best explained ZFS on Gnu Linux/Debian/Ubuntu for people that don't have 5,000,000 io operations a minute aka small business/home server/desktop.

The guide was written back in 2012, edited early 2013, BUT it breaks down the issue into digestable chunks and gives very useful, practical advice. While some of the install details may not be the most current, the thought process behind the why/how of ZFS still seemes relevant.

I'm still learning & putting this all together in my head/VMs but I think this may have the details I need to actually make a, somewhat, informed decision about if/when I would deploy ZFS.

Offline

#22 2017-04-15 16:22:27

iMBeCil
WAAAT?
From: Edrychwch o'ch cwmpas
Registered: 2015-09-29
Posts: 643

Re: Opinion-ation! Alt FS on a single HDD server

geekosupremo wrote:

But this guide - [Aaron Toponce - pthree.org] - found on AskUbuntu maybe the best explained ZFS on Gnu Linux/Debian/Ubuntu for people that don't have 5,000,000 io operations a minute aka small business/home server/desktop.

Excellent guide, excellent read, thanks geekosupremo for sharing ...

Now I have much clearer idea what ZFS really is, and what its capabilities are. As I suspected, it is overkill for my case ... with two disks only, 8GB RAM only, and really no need for extrasuperduper data consistency and 'durability' (essentially I need 'sandboxing' of few debians and FreeBSDs servers ...), LVM is just enough for me.

Best luck geekosupremo setting up the ZFS  wink Please share your experience when done.


Postpone all your duties; if you die, you won't have to do them ..

Offline

Board footer

Powered by FluxBB