Reading the reddit threads here and on homelab, you'd think ZFS was the bees knees, but I type in ZFS on a job board and there doesn't seem to be much demand.
I'm not some storage expert, but I've played around with ZFS in my own home lab for vm storage.
Recently I've been looking at building out a new server with new storage to replace my old machine. Before I go down this rabbit + money hole I was wondering what the job market out there is like for folks with ZFS skills.
Yea but usually they just say Windows
Nah man everyone is on ReFS now.
All the homies are using it for S2D.
Isn’t ReFS really lacking or have they fixed the problems?
The best part of ReFS is the fact that it does not support DFS.
It still cannot be a bootable drive. It still only has one purchasable driver in Linux. Datto is a big user all DATTO devices use ZFS.
Man, ReFS is super unstable. Had a friend that decided to use the "new" filesystem for an archive drive. Ended up spending a week recovering his data. That and Win 10/11 didn't support it except on Pro, Server 2019 didn't support booting to ReFS, some OS components don't work with ReFS. Stay far away, it's not ready for prime time.
The ability to make ReFS volumes was withdrawn from W10 Pro in order to allow for differentiation with the controversial new SKU "Windows Pro For Workstations" .
Pro for Workstations was a bit of a market-segmentation "scam" because any machine with an Intel Xeon-branded processor had to ship with the much more expensive Pro for Workstations SKU, per Microsoft's contracts with OEMs. Most of the OEMs offered much-cheaper Linux pre-installed on their Xeon workstations.
ReFS was sold down the river for some short-term revenue. Too bad, as Windows users could have gotten a lot more value from a reliable new filesystem than from "AI".
I've never worked with it but from what I've heard ReFS is anything but "resilient"
Microsoft: "It's copy on write so it will never get corrupted!"
ReFS: [gets corrupted]
Microsoft still recommends ReFS for exchange mailbox drives.
As far as I know ReFS doesn’t support chkdsk as it should do all the magic automatically.
Problem is that it doesn’t work, and when it breaks you can’t fix it.
Nah, it's still lacking.
Absolutely not. I've used ZFS, BTRFS, GlusterFS, CephFS, and every normal file system like ReFS, NTFS, FAT/ExFAT, XFS, EXT3/4... the only filesystem that I have lost data with has been ReFS, multiple times. I think the last time it happened I found out the version of repair utilities for ReFS doesnt even support the version of ReFS that comes with Windows (this was for server 2022 in year 2023 so maybe/hopefully this has changed).
I also have lost data with ReFS, multiple times. Supposedly WS2025 has a lot of fixes, but I don't really trust it, and probably would wait until that has been out 6-12 months before moving to it and using ReFS again. I had a multi-terabyte ReFS volume just not mount after a Patch Tuesday restart, and no tools would even recognize it, much less check the integrity. I poked at it for a few hours, threw in the towel, formatted the volume NTFS, and started a restore.
Even with btrfs configured in so-called "dangerous" configs like RAID5/6, I've not lost data directly due to the filesystem.
XFS, I've had to do some tap dancing to get it repaired, but eventually was able to get it back running. However, ReFS is the only one that I've had go dead and stay dead for no reason, after a normal reboot, with no repairs or way to access data possible.
RAID 5 is dangerous. No “so called” about it. Seen it first hand like a dozen times. It will always fail.
the only filesystem that I have lost data with has been ReFS, multiple times
You've never lost data on a FAT formatted disk? You must be a youngster. I mean, you never ran into the guy sticking his floppies to the computer with a magnet?
I wouldn't fault the filesystem for something like that.
I thought refs was a no-no for s2d as it causes redirected writes to your SAN and all writes basically proxy through one host because of it?
Only if you use a SAN backend. If you are running HCI nodes it’s recommended over NTFS
ZFS is a filesystem that builds-in software RAID and integrity protection, instead of leaving that to hardware as was done traditionally, or putting it in separate software layers as the Linux kernel has always preferred.
ZFS has been big for decades, but you shouldn't expect job listings to specify keywords unless the business principals care about it very much. I did once talk to an insurance company who really wanted a Linux LVM expert, and was even giving applicants a paper quiz. The first five questions were Linux basics, and the other 15 were very specific LVM questions that they wanted candidates to answer from memory. It was fair -- they said they were looking for an expert.
Probably a great way for somebody who doesn’t know how to evaluate Unix admins to find somebody who’s legit.
Imagine the guy who can recite LVM commands from memory but sucks as a Linux admin. Not happening.
Like those folks that have pieces of paper saying they are "certified"
Plenty of them but I have to keep telling them how to find the full log partition and how to set an ip either by conf files or ip address and change hostnames. Set up ufw or iptables....
That is an odd job requirement. The only times I touch a Linux LVM is when setting up the machine, or during recovery. The exception is Red Hat, where if I have to set up Linux software RAID under it in a RH supported configuration, I use the dm-integrity, md-raid, LVM2, and ext4 or XFS stack. If I have to expand a filesystem, I add another md-raid compound device to LVM as a physical volume and expand the logical volumes and their filesystems.
I do know you can do some funky things with md-raid and Linux LVM. For example, I've seen archive servers with wide RAID 1 arrays, like four-way mirroring. However, in general, LVM is sort of like awk, sed, and other tools. You use it and go on, as it is an assumed part of the job.
They never gave me any specific backstory, so I assumed that either there was a big, known, knowledge gap on their team, or they'd had some high-profile incident involving LVM.
We find that sites tend to either not think of LVM much, or be in love with its dynamic reconfigurability as a way to solve storage challenges. You and I are in the former category, and they were obviously in the latter. This was also right around twenty years ago, when SANs were rare and exotic, so they most likely didn't have any of the SAN abstractions available to them.
Storage drives eventually fail. I've loved zfs since Solaris because of this simple truth. When they fail, your zfs snapshot can get you up and running fast. Resilvering is a pain, but it works. Yes, other tech is out there with similar results. However, zfs is the bees knees as you can pay attention to your logs, and it will let you know when there's an issue.
You still want to have backups. You still want to set up your raidz or raidz2 set up with dual parity. And you still want to back up your stuff in another location, in the event of catastrophe.
ZFS isn't popular on job boards as newer tech stacks are perceived as sexier. However, ZFS is already future-proofed. There's constant improvements, and it works today all the way up to a billion terabytes.
Frankly, zfs isn’t all that complicated. I would expect anyone to be able to pick up zfs fairly quickly if they need it for the job, it’s really just a tool, it’s not a full time commitment. You don’t see anyone advertising for “grep specialists”, you’re simply expected to know it if you need it.
Also, zfs alone does not scale out, you need to pair it with something else if you want distributed storage, and that something else is going to be the more complicated part. So something like ceph is complicated enough that you could want someone specialised to administer a cluster.
Grep specialist... BAHAHAHAHAHA oh wait, that's AI isn't it?
Man the amount of linux+ certified people we've hired thay don't know what grep or "cat xyz.log | more" is makes me think I should just give in lol
My go to for searching through logs:
$ vim logfile.log
Once I’m in there:
/error\d\+
Simple RegExp that looks for anything that says error followed by numbers.
Looking for IP addresses? RegExp for that:
/\v\d{1,3}(\.\d{1,3}){3}
Honestly RegExp should be a required skill. I put it on my resume and I’ve had people ask me ”what’s ‘regexp’? But one dude knew. I had a whiteboard interview with an older sysadmin years ago, he asked me some softball “tell me how you would troubleshoot this…” - I grabbed the marker, went over to the board, and banged out something like:
grep -r -E '\b([0-9]{1,3}\.){3}[0-9]{1,3}\b' /var/log/httpd
Guy started laughing after I got to “\b(“ - let me finish my solution, said “Well shit, don’t know if you talked to (HR lady) about numbers yet, but I’d like to offer you the job!”
When younger people just starting out are looking for career advice, I always say “learn to script, and learn regexp. You’ll be unstoppable”
Why cat a log and pipe it to more?
depends what yer awkin about. or what it sed.
If your log is longer than the screen you can't scroll it with just cat, at least my env.
You can vi, vim, nano, etc. Or just that and then scroll down to read it.
Sure, but you can just do: more cuz.log
You don't need cat.
Well I might be the idiot in this scenario...
Most everyone cats whither it's needed or not: cat <file> | less
It's kind of a joke that cat is almost never actually needed when it's called.
Still, kind of nice to be able to do something like cat *.log | grep <search> | less
I'll often do a useless-use-of-cat if I'm incrementally building a long pipeline of commands, in order to give myself more flexibility if I need to add more filters to the start of the pipeline. Only once the pipeline's fully written will I then refactor to use the < operator or pass the input file as a parameter to the first command.
I mean you can do the same thing with grep directly but yeah, everyone does each command separately instead of memorizing every possible arg for every terminal program.
grep *.log -e "search" | less
In all fairness, I do the same shit but worse like chaining greps together with various invert flags and search expressions ( Fuck you regex, we will meet in hell )
cat *.log | grep -v -e "data" | grep -e "search" | grep -v -e "device" | ... | less
Nobody's an idiot here. We're all just doing and learning.
More mental model than command knowledge. Cat outputs, more/less scrolls, grep filters etc. While basically every command can be used as the everything command, its far easier to segment and remember what each of them does by limiting their scope to something reasonable
Sure, but it's a habit to break.
You're essentially reading the file twice and for large files it is noticeable.
The question is why you’d use more instead of less.
because less is more!
I wish I could give this more upvotes lol
When I started out I inherited a bunch of Solaris 8 and 9 boxes... They didn't have less!
That’s very fair- I use more or cat (I forget which) in Powershell.
cd /pub
more beer
Because they never ever studied for it.... just braindumps.
LPIC-1 has a volume of around 1-2 semesters.
Combination with Ceph provides a good solution.
It was nice on Solaris as you could roll things back by booting an old snapshot especially on SPARC. On Linux it was a bit of a pain as not all distros especially enterprise supported zfs on Linux.
Nowadays btrfs is solid enough that I trust it for similar use-cases. openSUSE's Snapper in particular automates btrfs snapshotting and has saved my ass on multiple occasions.
No raid 5 or encription support, problems with quotas and many snapshots...
Linux already has mdraid and LUKS, and I'm pretty sure those problems with quotas and snapshots got fixed years ago.
You can play with matroska style storage, using mdadm luks and btrfs lvm/ext4, suffering the slow performance (calling a block requires several translations, over different layers), and praying there is no problem.
Or just use zfs, which does all that all of the above tools do by itself, but better, optimizing the storage because zfs is aware of all the chain from filesystem to hardware block device, a proven enterprise grade storage solution.
And no, quota is not solved: https://wiki.archlinux.org/title/btrfs#Quota
Nor the raid5 bug: https://wiki.archlinux.org/title/btrfs#Multi-device_file_system
And if it you have a dirty shutdown you can't even mount root without doing btrfs rescue zero-log; no problem if it is your laptop as you can do it from initram, but on a headless server...
suffering the slow performance
I've yet to encounter any noticeable performance impact from any of those layers.
And if it you have a dirty shutdown you can't even mount root without doing btrfs rescue zero-log
I've done countless dirty shutdowns with btrfs over the years and haven't had to do that once. Even if that was actually an issue, how is that any worse than needing to fsck?
but on a headless server...
...you would pull up a console using the built-in lights-out management or equivalent. Or you'd pull up a console using a network KVM. Or you'd automate running recovery steps (like btrfs rescue zero-log) if the root FS fails to mount. Or you'd bake something like Dropbear into the initramfs so that you can SSH in. Or countless other approaches.
I've yet to encounter any noticeable performance impact from any of those layers.
On their own, have you ever stacked them? ( mdadm+luks+btrfs), because I did, not fun at all.
I've done countless dirty shutdowns with btrfs over the years and haven't had to do that once. Even if that was actually an issue, how is that any worse than needing to fsck?
lucky you, one time I used btrfs root on a raspy and every time It rebooted via power cycle I had to yank the microsd and fix btrfs on my laptop. I've yet to see an ext4 root failing to boot due to a power cycle, fsking on the boot or not (and I'm using a btrfs root on the laptop I'm typing this BTW),
...you would pull up a console using the built-in lights-out management or equivalent. Or you'd pull up a console using a network KVM. Or you'd automate running recovery steps (like btrfs rescue zero-log) if the root FS fails to mount. Or you'd bake something like Dropbear into the initramfs so that you can SSH in. Or countless other approaches.
Or I can just use a more reliable filesystem, which will boot after a power loss 100% of the time (in my experience).
On their own, have you ever stacked them? ( mdadm+luks+btrfs), because I did, not fun at all.
Not with btrfs, but I have with other filesystems on top of mdraid and LUKS (and LVM).
on a raspy
Well there's your issue lmao; SD cards are notorious for being flaky and for not behaving very nicely with normal filesystems, and I doubt you'd fare much better with ZFS on that card.
Or I can just use a more reliable filesystem, which will boot after a power loss 100% of the time (in my experience).
Aight, and in my experience btrfs has been that filesystem. So have ext4 and XFS and (yes) ZFS, to be clear.
In any case, you should have those above-listed techniques in your toolbox anyway, since boot failures can happen for any number of reasons (like hardware failures or bad updates or a script running sudo rm -rf ${WhoopsIMaedATypo}/* or what have you) and you'll therefore want some options to diagnose and possibly resolve the issue before having to walk all the way to that server and yank out its boot drive for offline diagnosis :)
I’ve had plenty of datacenter engineers tell me it’s not popular because most it’s not used by off-the-shelf solutions and/or it’s abstracted away from most storage engineers. Examples: Pure Storage, NetApp, etc.
Pretty much this.
I use it, but the stuff I do isn't enterprise. (Enterprise folks have a vastly different use case than research).
There's almost no storage case where you're buying raw hardware and configuring it yourself. Even if you have people with that skillset, you have better uses for their time, and you want to be able to hire someone to replace them at some point, without needing someone to reverse engineer your underlying storage set up.
We do it but we are an OEM for a full hardware and software solution. Meaning HPE saying "this config isn't valid" then I made it work and now it's valid!
But I can't get them on anything "new" to them. No zfs, keep vmware (lol no perpetuals)
It's popular in homelabs because it provides similar features as a enterprise storage, but for free. If you're going to use it in enterprise you also need to take the decision if it's really worth it to roll your own storage instead of buying an actual enterprise solution.
Also, this is mostly anecdotal but every time I have seen ZFS in large scale deployment it hasn't performed well... though mostly it's due to making mistakes such as enabling dedup on ZFS for VM storage, or thinking that RAID-Z2 has the same performance characteristics as RAID6, or simply overestimating the efficiency of SSD L2ARC and ZIL devices.
It doesn't help that installing ZFS on anything other than Ubuntu is a bit of a pain as well as you'll need to deal with dkms, or pre-compiled modules that at least used to lag a bit behind on kernel patches.
you havent even scratched the surface there bud
Building on this, ZFS brings an enterprise grade featsure set to the everyday user and organization.
you get RAM assissted caching, storage tiering, multi-protocol storage, full CoW, snapshots, restores, compression, flexible compression, encryption, etc.....PLUS, the whole logical volume management side of it.
pair this with a simple iSCSI target (target-cli, or iscitgtd?) and you can get an incredibly powerful and cheap Network Block Store comparable to 6 figure enterprise solutions from NetApp, 3Par, etc.
Also has significant protections against stuff like bitrot by assuming they drive will at some point lie about the contents on the disk.
It's just a tool. Home builders don't see a lot of job postings for people who "know hammers"... and Storage Architects / Admins don't see ZFS called out specifically either. Knowing it is just part of the toolkit for a larger role.
You're not going to find a lot of dev mapper, md and LVM jobs either.
But, at the same time, I don't think there's a demand for "Storage Spaces" experts either.
I think it's "assumed" that there's knowledge (?).
So... I think ZFS+ makes sense. ZFS, because of what it is, sort of "demands" good platform knowledge (how to construct a storage system). It's not that you can use ZFS "poorly", because you can, and it can still be useful, but there is an HW art that supports ZFS ideally, so knowing all of that makes for a better storage person.
Understanding all the above and then understanding how ZFS storage bricks can be utilized completes that picture, but again, I'd probably want even more (why not).
ZFS skills are really just understanding storage at a reasonable level. It's a pretty old technology so you won't see job postings with it listed but it is definitely still in use.
Outside of storage admin roles of course but that's extremely rare.
I'm not attacking, just putting things in a different perspective.
ZFS is newer than hardware RAID at least and it does a pretty good job of detecting corruption and replacing corrupt files. It's saved my ass more than a few times. That being said I don't like how much of a RAM hog ZFS is, which IMO can make virtualization require more RAM.
Have you found a newer tech that offers the resiliency that ZFS has without the RAM hogging?
ZFS has tuning nobs to knock down the RAM usage, but you take a perf hit. What I typically do is have a ZFS storage server and then NFS mount everything from that, and everything important is stored there, with ext4 used for boot disks.
nfs mounting zfs has always sucked for me.
You're probably hit by the sync part of NFS, i.e. when NFS writes to ZFS, NFS will always (unless you disable sync in ZFS) wait for ZFS to report back that the data is written to disk.
This can usually be mitigated by adding cache/journal drives to your zpool, or just have your zpool consist of super fast storage.
For example on VMware NFS is sync=always, while iscsi is sync=default, which is generally sync=off.
My write bytes plot looks like a sawtooth but worse. It's probably it. I'll go try this. Thank You.
ZFS is a fine solution. I'm not saying it's bad because it is old just that you don't see NTFS experience on job descriptions either.
Are there alternates that have different pros and cons? Sure. Btrfs (decent blend but less scalabe), XFs (performance focused)
Ideally you should be building your storage array knowing what file format you are going to use on it and optimize the platform for what it needs.
Define "RAM hogging"? ZFS only requires 768MB of RAM to run, but recommends 2GB, seems pretty low to me. It being too low will hurt performance a bit, but still works just fine.
You’ll want enough ram to store an entire txg plus some for cache.
Ok, my point still stands? Where do you think the 2GB recommendation comes from? That's more than large enough for 2.5Gbps throughput to hold an entire full transaction group. Again as I mentioned, very low ram will hurt performance, but it will still function even with 1GB of RAM. Not sure why somebody would want to do that though.
I've always been told for ZFS it's 16GB base +1GB per TB of storage, but I've never seen that being used. I can comfortabilty run a 16TB zpool in 8GB
The whole 1GB per TB is for large enterprise use. If you have hundreds or thousands of users with dozens or more concurrent reads and writes going on at once then yes you'll want a good size ARC so your performance isn't terrible.
Been using zfs for 20 years here about it's huge in the non windows area not something people specifically look for when it comes to hiring
Interesting ZFS on windows will be a thing one day soon: https://github.com/openzfsonwindows/openzfs/releases
Will take years to be stable , zfs on Linux took years to be stable
Yes, it did take a long time. But the development didn't really accelerate until Delphix and a bunch of other companies decided to make Linux the master and push out to FreeBSD and Illumos.
I think there is some demand for ZoW (ZFS on Windows). With some focused development around it. There won't have corruption issues you need to sort out. Once it works, it will work and be safe to use.
QUICK someone phone in Jim and Allen
ROFL! 🤣
If they don't answer can they phone Oracle?
You’re not gonna see that in a job description it’s to low level, just like you don’t see Fat, NTFS, EXT3 etc. what you will see is windows file servers or Linux file servers or SANS, or maybe even NFS, iscsi, or fiber channel. They can’t put every little detail on the job posting.
Why on earth would you think there would be job listings noting a specific filesystem lmao.
Managing ZFS is not that difficult. Even if done using the command line interface. You don't need to invest much time.
Take a look at TrueNas. (Core and Scale)
Yep. I've been using TrueNAS for well over 15+ years and it's been rock solid. Also, using ZFS on my ProxMox servers at home and work. Again, rock solid. ZFS is really easy to manage and maintain. Even in command line.
If you setup notification to e-mail via cron jobs to check the status of ZFS you're already ahead of the game.
Another factor is commercial support. Sun made some very fast, reliable storage systems based on ZFS. The offered worldwide support and a large, reputable company backing those systems. when Oracle bought them, sales carried on for a while by momentum and then fizzled.
Today, you won't find much demand for Solaris, Oracle ZFS or Sparx systems in help wanted ads.
What you will see are HPC, educational and large ISP type customers using the open source versions of ZFS and related technologies. These will be places that operate at a large enough scale to make maintaining these solutions worthwhile or are on a space where they just need max performance at lowest cost.
There used to be this comparison chart of linux FS's. There was one column "makes you kill your wife".
Of course RieserFS is the only one with that check mark.
What's all the hype with cars? I type cars into a job search engine and it doesn't return much
I’ve been looking for FAT12 jobs but nobody seems to put that under skills requirements in their hiring process. Maybe talk to recruiters to inform them of ZFS keyword importance in their applicant tracking systems.
Anti-Bitrot
It's secret power: replication
Are there really admin jobs that talk about storage protocols?
You likely won't get hired just on ZFS skills. However, knowing how to use it is a good thing. What ZFS brings to the table is a combination of logical volume management and the filesystem. This way, say I need a disk image for an iSCSI target, I can zfs create -s -V tank/zvol, and have in /dev/zfs/tank/zvol a raw volume. I can create a separate filesystem and mount it anywhere I feel like.
RAID rebuilds are nice under ZFS. Instead of rebuilding the entire RAID volume, just the stored data is done with a resilver, after a failed hard disk is replaced.
Snapshots are also nice. At a previous job, they had a vertical market application where the developer, in their genius, used millions of name:value pairs, in a subdirectory off of $HOME. Backing those up via a standard backup program was not possible. In fact, the file server, were it not ZFS, would run out of inodes and collapse. Backups in the normal way were not possible. However, with ZFS, I could snapshot the home directories and send those snapshots offsite, as well as send them into a Borg Backup as an additional method of backups.
If you know how to use a LVM and fdisk, as well as filesystem tools, you know how to use ZFS, pretty much. It isn't difficult.
The one thing that I see people do make a mistake with ZFS is assuming it can recover from single disk hardware failure. It can't. If you have ZFS on a removable hard drive, and it starts getting errors, ZFS can't do anything about it. However, if one has RAID-1 or RAID-6 (RAID-Z2), ZFS can not just route around faulty drives, but possibly rebuild.
ZFS is not the be all and end all. You cannot shrink a pool unless you remove vdevs (be it single drives, pairs, RAID-Z instances, etc.) Once a pool is deployed, there is not much you can do except evacuate all data and start from scratch if a core item isn't configured right. However, ZFS is intended for the enterprise where one is going to set up a RAID array and leave it be from there on out. If it needs expanded, eithe vdevs will be added, or the data copied off, the array rebuilt from scratch. If someone needs shrinking of drives or volumes, I'd highly recommend looking at something like btrfs, lvm2 and Linux md-raid. If one has RAID members of uneven sizes, then consider using md-raid or btrfs.
ZFS is also not great in low resource environments. If you want something for a Raspberry Pi with 1-2 gigs of RAM, I'd steer people towards btrfs, or how Synology does it -- btrfs on top of md-raid. If one wants bit-rot detection at the RAID level, then consider using dm-integrity with SHA-256 on the lowest layer, then build md-raid on top of that. This way, an error that makes it through the drives CRC or caused by a controller glitch will be passed to md-raid as a hard error.
tl;dr, ZFS has its place, and there are many use cases for it. It is worth knowing how to use, especially for things where you need to build a backup server or something with RAID volumes. However, it is just part of a job. ZFS also has pros and cons.
Overall, I have been happy with ZFS. On Ubuntu, when standing up MinIO servers, if a drive failed, it was smart enough to light the failed light on the array so I could replace the drive and have it resilver.
but I type in ZFS on a job board and there doesn't seem to be much demand
Have you seen any jobs asking for FAT32, NTFS, Ext4 or xfs? It's just a filesystem. Hardly worth putting on a job board
ZFS is good in general, but absolutely is over-hyped. There's a lot of downsides to it. If you're using it as RAID, you can't expand an array like you can with traditional software RAID, it eats a lot of RAM, and I believe it writes to your drives excessively. CoW isn't always the right solution either. It has upshots though, great snapshot and backup support, deduplication, and various data corruption protection abilities.
Me personally, I will always use traditional software RAID, but would consider ZFS for CoW filesystem, though it'd be a competition between that and BTRFS
It's probably depends on which options do you have. If you have no experience with zfs it surely will be a rabbit hole until you find out how to configure it. However if you have only two choices, hardware raid or software zfs zRAID I definitely choose zfs, because of flexibility, errors correction, etc.
100% on software RAID. Makes it real easy to move drives to another server and import the pools. Done.
This would be lumped under "storage" or "RAID" on a resume. Those terms would assume you are familiar with different filesystems and redundancy patterns, in the same way that an "experienced roofer" would already know what shingles and tar paper are.
Hiring managers don't tend to dive into specifics unless it's a core part of their business (e.g. if you're applying to work at iX systems with TrueNAS).
Just play with it in my lab and I like all features it comes with.
If you do it right. Easy to manage and see all the drives. No raid cards needed any more. Snapshots are simple abs quick. More advanced you can send a full backup easily to a remote system.
Xfs , ext4 do not compare. There is lvm and mdadm but it feels 20 years behind.
Tegile built their hybrid and all flash block SANs on ZFS, and they aren’t around any more
were getting rid of our Tegile, 5200 nVME has caused us nothing but trouble, controller after controller keeps failing, I think last year I replaced 3 within 8 months...if it's not the controller its the stupid sodimm that fails and needs replaced.
Moving to Pure nvme and not looking back.
We got in when Tegile was new and bleeding edge, worked awesome, had great support and relationship, then when WD acquired them their support started to goto shit.
yeah worked there for a year, most support calls I ever saw, horrible box
You wouldn't look for a job specific to ZFS, nor would you any other file system, unless you were like a developer or something, but it wouldn't be a sysadmin job.
ZFS itself is an incredible file system, it has some downsides, but for large scale data storage, it's one of the best out there. I've got pretty extensive experience with it and with managing servers running it, it's not let me down once (same can't be said for alternative things like ReFS lol).
But again, other than development, there isn't really a "ZFS job market", it'd be one of many tools under your belt as an admin/engineer/architect. Basically any large scale storage deployment I would design today would be ZFS or CephFS based, but I can also manage ReFS, NTFS, etc... they're all just file systems at the end of the day, and as an admin NOT a dev, I should know how to manage them all and know their ups and downs.
but I type in ZFS on a job board and there doesn't seem to be much demand.
Well, yeah. Have you ever seen many jobs for NTFS or HFS or exFAT? It's a filesystem, not a career.
It's a well engineered filesystem. Not perfect, but way more modern than something like NTFS or Ext4 on a RAID5, so you get some nice robustness and snapshot features. But part of the reason it's so nice is that once you have a big server set up, you pretty much only touch it in an active way when you are rolling around snapshot revisions, or replacing failed drives. If it was such a pain in the neck to manage that it required constant fiddling and many companies needed to post tons of ZFS gigs on job boards to hire full time ZFS admins to poke it at day-to-day in order to keep it from falling over, nobody would use it.
It's just a useful tool in the toolbox of a versatile system administrator who needs to keep bytes around.
Zfs tasks would be part of the OS admin. One example would be Solaris. I’m not sure if the latest version uses zfs. It’s been a while since I worked on them.
I am not going to repeat what others are saying, but there are ZFS is super advantageous! I worked on a project a few years ago at my old job to migrate all of our backup servers to ZFS, we were using LVM before. Why did we move them all to ZFS? Because compression, I work/worked in the webhosting business and all of the data we back up (Customer data) a lot of it is super compressible, depending on the server/partition on the server (Our servers ahd between 4 and 8 /home partitions) we were seeing on average 1.8x compression and we saved SOO much space!
Lets not forget deduplication, we opted not to use it because we would need close to 1TB of RAM in our backup servers to fully take advantage of it.
Before I left I was on the track of moving our fleets to use ZFS so that we could snapshot our VM filesystems instead of doing incremental file backups, however after I left the push was abandoned, but not ZFS backups.
ZFS is cool as shit and always has been. The kids I hang out with were raving about it in 2006.
I can't speak for others, but I manage ~12PB of storage on ZFS.
Here's what I like about zfs.
No fsck. At the scales I work at, it would be unrealistic to run an fsck on 780tb of storage.
The fact that the meta data and the file data is checksummed means I can fully validate my storage on a monthly basis.
Subvolumes, instant snapshots, clones, built-in compression.
I could go on and on..
I have been using ZFS for almost 20 years now. Maybe there is an assumption if you know Linux and storage, then ZFS isn't a stretch. On the other hand you will see requests for ceph expertise.
I mean if you want to manage Solaris machines then ZFS is a pretty good skill to have.
On most of my jobs we just have Vendor deploy iSCSI solution or other storage solution. We don't care what is running on the metal we need to use it for either fast storage or slow storage. We need to be notified when drive goes bad and needs to be replaced. I am sorry but I don't think it worth for me to learn this stuff if it would be like 1% of my job and I would have to be really good at it in order to practice is safely or otherwise I lose data.
Enterprises are moving away from solutions where you need very particular skillset which has to be at very high level, which is going to be used once a month or several months. It is easier to get a solution and just employ simpler admin who can keep the system running with support.
Recently I've been looking at building out a new server with new storage to replace my old machine. Before I go down this rabbit + money hole I was wondering what the job market out there is like for folks with ZFS skills.
Yeah, but you're not going to see it advertised much. You may see it as a tertiary item on a list of pre-reqs or "nice to haves" but most job listing will just say NAS, SAN, or some iteration of "Storage Admin". I've used it professionally, among other filesystems, for about a decade now. It's all going to depend on the size and scale of the business and if they're tied to any particular vendor. There have been situation where I've built a custom server solution running FreeNAS (now TrueNAS) and other situations where I've used it in an old mainframe environment (Solaris). It's nice to know because you'll never know when you'll run into it. It's also great for setting up SMB NAS boxes.
Think about how often you see NTFS, FAT32, XFS, BTRFS, or EXT mentioned in job posting? It's probably never.
Lots of Dell rack storage arrays use ZFS. Isilons were fun for that, my homelabing with FreeNAS became invaluable experience when I had to start managing a 15 node cluster of them. Love ZFS.
Frankly you shouldn't be seeing job postings talking about drive formats because no one freaking cares. You don't build your own RAID system in a real environment, you are using a company product like pure or whatever and dealing with their front end. If you actually care about the low level like that, you are doing it wrong.
The hype is its features & offerings. A filesystem is just that: a filesystem. No job listing will specify filesystems in their job listing. As a SysAdmin, you should already know how to work with filesystems. Even on the Linux Admin side you won't see job listings requiring ext4, xfs, or btrfs knowledge, or even something like requiring Oracle experience listing a job requirement as knowledge of gfs2. It's just a different filesystem with a different storage format & functions & features. It's up to you to read the documentation and/or the company wiki on working with that filesystem.
I think the primary issues are not technical.
Three isues that play a big role:
ZFS licensing is not GPL, which means the big distro's offer it as an addon rather then a core choice (and big distro's are where the job market will be).
There are additional licensing concerns. There's good reason to believe that NetApp would sue if a major distro make ZFS core. Many of the really cool features of ZFS seem concurrent with similar innovations at NetApp
Filesystem adoption is very conservative (for good reasons).
OpenZFS is CDDL and NetApp has nothing to do with it.
Ubuntu includes zfs in core
I upgraded to 22.whatever LTS, only to find out that a mission critical app for me didn't like that upgrade... was dreading the OS rollback... ZFS took... 10 seconds, to roll back to pre-OS upgrade, and all other updates, everything works again... that saved me hours right away.
This was new to me (thanks)
Worth noting, this is still awkward, from a licensing standpoint. From what I'm reading, the binary is considered GPL, but there is no Dual license, and that this is a new workaround.
Which is, pretty cool (overall).
My sense is that BTRFS and ZFS occupy much of the same space, and BTRFS (with all it's issues) seems to have an adoption edge.
ZFS is a redundant file system. The first backup company to use it was Datto back it 2007. It has a couple advantages.
Allows software block differential and allows you to save certain states of the file system.
Allows on the fly reversion to different state of the file system and on the fly VMs.
There are a ton of free drivers for ZFS file system in Linux. The big advantage over ReFS Windows version that has only 1 purchasable driver. AKA why Datto can not show ReFS drives on there Web Interface.
Disadvantages :
One big disadvantages is its tendency to bloat on a RAID Array. Extra data writing redundancy on top of redundancy. Windows ReFS does not suffer from it.
Some call this a disadvantage if you delete the pool all data is lost.
ZFS should always be ran with a backup ether local or cloud. They have been able to reduce the amount of bloat in the file system.
There is no Hype around ZFS. Raid and ZFS are identical but both have their pros and cons. If anyone tells your ZFS is perfect and better than standard raid they have no idea wtf they are talking about.
I use both in the daily and I can tell you for a fact both have issues.
Could you elaborate on that matter? I see only zfs preachers or hw raid fanatics here. Hardly noone sees that they both have issues and it depends on situation.
Well off the top of my head one major downside to ZFS is the ram and cpu requirements. It uses RAM and CPU as buffer\cache and to calculate parity instead of a on board battery like you would find on a Raid Controller. So technically if your power goes out and the system goes down dirty, if the data isnt commited\processed via memory yet to ZFS that data could be gone. Normal raid cards handle this by using a battery on the raid card.
Even on something as simple as TrueNAS its recommended no less then 16GB of ram (not crazy but its still a requirement that normal RAID doesnt have) and that requirement will go up based on the amount of data you have in your pool.
As ZFS is quite paranoid about hardware, the less hiding there is, the more it can cope with any hardware issues. RAID Controller configurations and ZFS may be very difficult to restore or reconfigure when it fails (i.e. hardware failure) and because of this JBOD is recommended for ZFS over using a Raid Controller with ZFS which is also another downside.
Here is a little info on it from Oracle talking about the basic requirements. Physical Memory is one of them https://docs.oracle.com/cd/E19253-01/819-5461/gitgn/index.html and from other Open Source products that typically use ZFS https://pve.proxmox.com/wiki/ZFS_on_Linux
ZFS is a software raid. So if the OS that manages the software dies (like TrueNAS drive used for the OS install) your ZFS pool is gone. It can be recovered in most cases if you have a ZFS config backup and dont modify the drives when they are off the ZFS pool. But its more iffy than doing it on a Raid Controller since again, it is a software raid. It has the same negatives as using Windows Software Raid which is why its not recommended an enterprise settings and why they use Raid Controllers, because that information is stored on a Raid Controller and isnt lost if power goes out or the OS dies.
However, ZFS does have pros also like allowance of mixed drives and sizes and the fact it is free. But another con is if you needs ZFS support you are relying mostly on the Open Source community or searching indepth for a ZFS expert which is harder to find than a Raid expert since it is more widely used.
Raid has its own issues as well such as maintenance, cost, can do mixed sizes but alot of raids dont like mixed drives. It is possible to do though.
However the major pro of Raid is that its a hardware raid and not a software raid, typically has better performance, caching etc... and is easier to find support for since it is used everywhere, especially in enterprise settings.
However both at the end of the day do the same thing. Raid type that can mirror/script data across drives and allow for a disk failure to occur without losing all data.
Its just good to be educated on the subject and dont let the ZFS fanboys fool you into thinking its better or more bullet proof than typical raid. Because it isnt. I would say they are both comparable with their own pros and cons. If you are OK with the cons of ZFS than go for it. I personally use a TrueNAS at the house and havnt really had any issues but in my VM Host servers I use a Raid Controller instead because I'm not OK with possibility of data loss incase of a power outage and the APC dies.
My major pullback with ZFS is the fact it is a software raid and because so it still has the negatives tied to using such a process. This is why I backup my TrueNAS an external disk as well. I personally wouldn't trust it with critical data like enterprise related functions but for home use, if done right it can work just fine.
My problem with ZFS is the inability to expand volumes easily/conveniently
Add or replace vdevs with zpool add or zpool replace. Couldn't be easier or more convenient.
You know very well what I mean.
If you have a 6 disk ZFS volume You can’t just add one and expand.
You know very well what I mean.
If you have a 6 disk ZFS volume You can’t just add one and expand.
I didn't until you added additional information, even with the additional information it's ambiguous but I think i know what you mean now. I would read "volume" to mean pool. I believe the above poster may have also read it similarly, suggestion you didn't know you can just add vdevs (either a single disk, mirror or raidz) to an existing pool. Or you can replace the disks in your existing vdevs with larger ones and grow that way. Using ZFS nomenclature of pool and vdev would help to avoid ambiguity.
What I now think you mean is you want to be able to take a disk and add it to an existing raidz (or z2, z3) vdev. Good news. "RAID-Z expansion feature" was merged into OpenZFS master branch in November and will allow you to add an additional disk to an existing raidz vdev one at a time. It hasn't been put included in a release yet, the chatter at the time suggested it was will come with a future 2.3.0 release.
If you're dying to get that feature you can always roll your own branch with that feature. Here's an example of me turning a 4 disk raidz3 (dumbest pool topology ever, please don't anyone do this with real hardware), and expand one disk at a time to 7 disks.
My only experience with ZFS is the TrueNAS OS, and from my friend who is an insane home labber told me that it's TERRIBLE for VMs. Tbf idk if he was referring to ZFS or TrueNAS in general.
I have 40 active Hyper VMs on zfs, 3 mirror vdevs + 40 GB of RAM and iSCSI sharing. It works great for 5 years.
The main rule is, you always should have at least 50% of free storage space.
"The main rule is, you always should have at least 50% of free storage space."
Are you agreeing with him?
To be clear, for iSCSI is 50% free space, for SMB is 20%.
50% free storage! More like 20% free. We keep 20% reserved on our iSCSI SANs at work.
Also, whoever said ZFS was bad for VMs is smoking crack, and TrueNAS is fine for that. I use iSCSI on TrueNAS Core to ESXi hypervisors in my homelab.
ZFS was great when the alternative was VxFS and VxVM (Veritas) but it came way too late to save Solaris. RHEL has XFS. AIX has a native VM and JFS2. Not even sure who still uses ZFS at this point.
ZFS kicks the ever living shit out of RAID. Start there.
Do you see a lot of job applications involving NTFS?