Not a file system developer but an ardent ZFS user. Here are my thoughts.
I pick ZFS every time because of its usability, reliability, and being able to easily understand what goes on without being a master of the OS ecosystem it runs on.
For example, pick ZFS, and sit with the ZFS handbook for 15 minutes and you already know how and what to do to:
[Create/modify/destroy] X [Vdev/Zpools/datasets]
A disk gives out, it is easy to k ow what to do to let ZFS deal with it without googling hard and relying on forums about what the right thing to do for your OS is - unless you already know it.
ZFS is not the fastest filesystem to pick with it being a COW fs. But, it does everything so well, feels solidly thought out that it gives a lot of confidence to the user which is very important aspect for a file system.
I hope bcachefs and many other modern filesystems that are vying for mainstream adoption like ZFS keeps these virtues from ZFS. Until then, I would happily try and play with them, but not think about using them in stuff I really care about.
Panics and other issues are pretty rare for regular users that don't push ZFS to its limits with over-utilized RAM or obscure kernel and/or ZFS features, but when you do encounter problems, you better know some kernel internals kung-fu if you want to keep using recent kernel and ZFS versions rather sooner.
(This argument is mainly about ZoL though; I can't speak for Illumos or FreeBSD that have somewhat tighter integration between ZFS and the rest of the kernel.)
Hmmm. True that when you encounter a panic or a serious error, and you lack serious system/kernel/OS skills, it can be real trouble. But still less so than picking a new/developing filesytstem. There are so many eyes on ZFS that one would hope someone with the skills you lack is already looking at that serious issue or at least will do if you file a bug report.
As a personal experience, I have two ZFS based file servers being hammered by researchers pushing it in many weird IO patterns, bursts requests and so for about a year and half without ever hitting a serious error. I'm amazed by how rock solid the system is. One of them even has a Optane cache for NFS writes- just to give you an idea of feature richness.
FreeBSD+ZFS on a dedicated server+JBODs. I keep the infrastructure dead simple and they are a charm.
That is only because the author (kent overstreet) based it on his former work bcache.
Fyi he now has fully persistent allocation info done so his attempt ti mainline it should start soon. Even though I still got open bug reports left I'm excited about this.
He checks his code against the xfs test suite so hopefully it really will be stable and useable.
My main problems with bcachefs are:
* it's mainly one developer
* when I tested it I had problems many times
* he documents his progress irregular(ly) (not frequently with pauses and at different places: bcachefs.org, patreon home, patreon posts)
* he doesn't answer questions (on his patreon posts or on github bug reports) (this could be a good thing if he only wants to concentrate on code)
* while waiting om bcachefs becoming mature, butterfs became finally mature (kind of)
So even thought I'm following the project I'm not sure if compression now has working disk space accounting or not and similar things, but since I see admirable progress I'm still looking forward for the first mainline release
I think there have been updates after every bigger piece of work recently. They were irregular because the work took different amount of time. He never promised scheduled updates as far as I remember.
I'm really only talking about documenting not patreon posts. For instance the patreon main page says compression is working with lz4 and zlib, the bcachefs homepage says lz4, zlib and zstd (with zstd having problems).
Also the mount times mentioned on the homepage are afaik (could be wrong) not that big of a ptoblem anymore and only slow because of the optional fsck
I do feel less confident trusting by data to one person hacking on one file system compared to a tried and tested file system that was built by Sun’s best engineers. CoW support in Linux is good but lack of snapshots still makes this an inferior choice to zfs.
I wish the GPL zealots wouldn’t bark that loud every time someone mentions ZFS on Linux. Ubuntu is shipping next release with ZFS on root. It will be awesome.
I thought the Ubuntu's decision that they can include it in their release was controversial but are they that convinced the license allows them to do that?
Well, it did originate from the realization that a good block caching layer designed for SSDs was most of the way to being a real filesystem in its own right. The caching is the biggest advantage bcachefs has over other filesystems, and the name does end in "FS". Perhaps it's a personal bias that makes you inclined to think dismissively of something as "just" a caching layer, when it's actually a really hard problem?
Perhaps it's a personal bias that makes you inclined to think dismissively of something as "just" a caching layer, when it's actually a really hard problem?
I was providing a possible answer to the question: why do people not give bcachefs more attention as a filesystem?
> Perhaps it's a personal bias that makes you inclined to think dismissively of something as "just" a caching layer, when it's actually a really hard problem?
That’s quite incendiary.
I also assumed it was a caching filesystem based on the name. Something you could attach to LVM or BTRFS to cache disk write or read on access.
Because despite the fact there is “fs” there is “cache” written there. And caching filesystems would indicate that instead of just using ram it would use some kind of traditional storage. It’s not unthinkable that this is a common thought process. What kind of “personal bias” could exist which alters what a caching filesystem would mean?
I didn't mean to be incendiary, just to point out that different experiences could be leading people to make inaccurate assumptions.
And I'm a bit confused by what you mean when you say "caching filesystem", since you seem to be using that term to talk about something that isn't a filesystem at all, rather than a filesystem that has advanced caching capabilities on its feature list. (cf. terms like "checksumming filesystem", "copy on write filesystem", "journalling filesystem") I was careful to refer to the original bcache and similar tools as "caching layers" rather than mislabel them as filesystems, but bcachefs is a proper filesystem.
A caching filesystem (as opposed to a standard block storage filesystem) would be one used for cached data specifically. ZFS has cache drives but the actual on disk layout does not resemble ZFS as we know it. That’s what I’m eluding to;
ZFS in this case is an all-in-one solution so it’s perfectly fine to refer to it as a feature of the filesystem, but externalising that to a separate agnostic product would result in this being exposed.
In fact CacheFS was designed exactly like this, and as you can see includes both “cache” and “FS” in the name.
The lack of a zero-copy sendfile for ZFS is one of several reasons that we (Netflix Open Connect) use UFS on our CDN nodes.
It would be interesting to see how he solved this. From a brief look at the FreeBSD code several years ago, I seem to recall that ARC dealt in 8K chunks (maybe because it is the sparc page size??), but x86_64 uses a 4K page size, and mmap() and sendfile() expect to deal in pages.
The other issue would be "taming" the ARC. We make heavy use of FreeBSD's SF_NOCACHE flag to sendfile(). This allows us to teach the VM system to keep hot content in the page cache, and drop less popular content.
Has anyone considered replacing ARC? It actually has fairly poor hit rates compared to other policies, with very modest improvements over LRU. I also wonder how much ghost entries contribute to space overhead, which is actually trivial to remove (ARC is an adaptive SLRU - so replace ghosts with a sampling hill climber).
Apple’s aborted port had this, but it was the source of a lot of bugs and instability. Probably could have been made stable with enough time and effort though.
It's incredible to me how many people are suddenly comfortable violating the license of ZFS, simply because Cannonical says it's okay? This is an open source project, under an OSI approved open source license (CDDL) which is quite known to not be GPL compatible. The license is derived from MPL 1.1, which is also known to not be GPL compatible. Why did they bother creating MPL 2, when we could have just ignored the GPL incompatibility in MPL 1.1?
You can't be someone who loses their mind over GPL violations, but then willfully misinterpret someone else's open source license because they have something you want. The developers involved specifically picked CDDL because it wasn't GPL compatible. That was their desire / intent, and so that's the license they went with. By ignoring the license and the clearly known intent of the developers in choosing that license, simply because they made something you really want, I believe that ZFS on Linux is really a form of theft in the open source world. It's the same sort of attitude that leads to GPL violations.
I don't agree with making things CDDL to be explicitly GPL incompatible, but I do believe in respecting the author of the software in question and the license he or she chose. If people choose licenses I don't agree with, I don't use their software. I don't just use it anyway and pretend like there's no license issue.
You don't have to be RMS to see that these people who are ignoring the licensing issues (not just shipping user mode tools like a FUSE implementation which is legit) honestly don't care about open source and licensing and copyleft. They just see something someone else made that they want and they take it and ignore the licensing and developer intent. How can we hope to enforce the GPL successfully but then ignore other OSI approved licenses when it suits us?
This is why big names in Linux kernel development aren't remotely interested in ZFS.
Not to mention the fact that the users of ZFS on Linux are all setting themselves up for Oracle to bring the licensing inquisition.
It's an open legal question if a user loadable module violates the CDDL/GPL. Until its fully litigated, its going to remain an open question.
As an example: the FSF's position is that yes, if you dynamically link against a GPLd binary that all the code has to be GPLd, and if one wants to link against a copy-left library in a non copy left program the library has to use the LGPL. However, the LGPL predates dynamic linking. It's from the days of static linking where the (L)GPLd could would be distributed together in a single binary. In the world of dynamic linking that isn't true anymore.
A primary example is libreadline. It is (or was?) GPLd and the FSF use this a hammer to try and get other code to be GPL. However, this (as far as I know) was never actually litigated.
As another example: the Linux kernel is under GPL2, Linus makes it clear that closed source programs can use the syscall interface without a problem, but its not clear that if he didn't say this that they couldn't. He includes the note to clarify his position that they can for those that would be concerned. i.e. the argument would be that without the syscall note, one couldn't ship any hardware device (say an appliance you stick in your rack) with linux and proprietary software. My gut is that most would find that difficult to swallow.
As an aside: I personally feel this has similarity to the google / oracle java lawsuit. If one thinks that APIs cannot be copyrighted then arguably anything that just uses APIs and doesn't embed the actual GPLd code one should be fine as the APIs aren't able to be copyrighted. If on the other hand one views APIs are able to copyrighted as well, then it would be much more clear that the FSF's position is correct. Though the usage might still be fair use (and to an extent one might argue that its even more fair use than Google's usage)
If this ever does get litigated, I'll be following closely as would be very interested (intellectually at least) in the result.
TLDR: until its actually litigated, everyone who claims to speak an absolute truth is speaking out of their ass. Lawyers are a conservative bunch in general so they'll give you the most conservative answer in terms of what you can do, that doesn't mean its the absolute truth in terms of what you can and cannot do.
What we do know is the intent of the developers in open sourcing their work and that those working on these ports are violating that intent. They specifically chose this license with the belief that it protected their software from being consumed by GNU/Linux.
The counterargument, maybe their license doesn't block us from using their work in the way they think it does, and until it's proven in court we won't know is extremely offensive and self-serving.
This is the same argument raised by many GPL violators. "The GPL hasn't been proven well enough in court in my jurisdiction so I'm just going to use your code in violation of your license terms."
This is theft, an open source license violation, and lawyering to defend that theft. It violates the principles of open source. We are literally discussing an OSI approved license. I believe that any responsible member of the open source community would respect the authors license and not try to lawyer their license out of your way so you can have their code on your terms.
If this was Linux devs and GPL, rather than Solaris devs and CDDL, everyone would be on the other side of this argument. It's hypocrisy and it should be called out.
Do we actually "know"? Cause I "know" that the opposition to GPL was not to make it incompatible with GPL (which is a point of contention actually, and hits deeply into question of when a code is derived code). The point was that making OpenSolaris GPL would block integration chances with NON-GPL projects.
In fact, at least one anecdote points out that Sun expected CDDL components to make it into Linux source, just like cases of other non-GPL components in the past (AFS, for example)
ZFS is a great filesystem and being able to possibly have a better implementation than we do at the moment, or at least, be able to spur additional development and optimization, is very welcome!
I hope that more people would help with bcachefs development. It seems like a modern alternative for ZFS with proper license and Linux integration.
Not a file system developer but an ardent ZFS user. Here are my thoughts.
I pick ZFS every time because of its usability, reliability, and being able to easily understand what goes on without being a master of the OS ecosystem it runs on.
For example, pick ZFS, and sit with the ZFS handbook for 15 minutes and you already know how and what to do to:
[Create/modify/destroy] X [Vdev/Zpools/datasets]
A disk gives out, it is easy to k ow what to do to let ZFS deal with it without googling hard and relying on forums about what the right thing to do for your OS is - unless you already know it.
ZFS is not the fastest filesystem to pick with it being a COW fs. But, it does everything so well, feels solidly thought out that it gives a lot of confidence to the user which is very important aspect for a file system.
I hope bcachefs and many other modern filesystems that are vying for mainstream adoption like ZFS keeps these virtues from ZFS. Until then, I would happily try and play with them, but not think about using them in stuff I really care about.
> I pick ZFS every time because of its … reliability, and being able to easily understand what goes on
https://github.com/zfsonlinux/zfs/issues?utf8=&q=is%3Aissue+...
Panics and other issues are pretty rare for regular users that don't push ZFS to its limits with over-utilized RAM or obscure kernel and/or ZFS features, but when you do encounter problems, you better know some kernel internals kung-fu if you want to keep using recent kernel and ZFS versions rather sooner.
(This argument is mainly about ZoL though; I can't speak for Illumos or FreeBSD that have somewhat tighter integration between ZFS and the rest of the kernel.)
Hmmm. True that when you encounter a panic or a serious error, and you lack serious system/kernel/OS skills, it can be real trouble. But still less so than picking a new/developing filesytstem. There are so many eyes on ZFS that one would hope someone with the skills you lack is already looking at that serious issue or at least will do if you file a bug report.
As a personal experience, I have two ZFS based file servers being hammered by researchers pushing it in many weird IO patterns, bursts requests and so for about a year and half without ever hitting a serious error. I'm amazed by how rock solid the system is. One of them even has a Optane cache for NFS writes- just to give you an idea of feature richness.
FreeBSD+ZFS on a dedicated server+JBODs. I keep the infrastructure dead simple and they are a charm.
https://bcachefs.org/
The name doesn't help. It doesn't sound like a real filesystem (it sounds like something that accelerates a real filesystem)
That is only because the author (kent overstreet) based it on his former work bcache.
Fyi he now has fully persistent allocation info done so his attempt ti mainline it should start soon. Even though I still got open bug reports left I'm excited about this.
Bcachefs got
He checks his code against the xfs test suite so hopefully it really will be stable and useable.My main problems with bcachefs are:
So even thought I'm following the project I'm not sure if compression now has working disk space accounting or not and similar things, but since I see admirable progress I'm still looking forward for the first mainline release> he documents his progress irregular(ly)
I think there have been updates after every bigger piece of work recently. They were irregular because the work took different amount of time. He never promised scheduled updates as far as I remember.
I'm really only talking about documenting not patreon posts. For instance the patreon main page says compression is working with lz4 and zlib, the bcachefs homepage says lz4, zlib and zstd (with zstd having problems).
Also the mount times mentioned on the homepage are afaik (could be wrong) not that big of a ptoblem anymore and only slow because of the optional fsck
btrfs no longer feels convincing when RedHat thinks it's no longer worth pursuing it.
I do feel less confident trusting by data to one person hacking on one file system compared to a tried and tested file system that was built by Sun’s best engineers. CoW support in Linux is good but lack of snapshots still makes this an inferior choice to zfs. I wish the GPL zealots wouldn’t bark that loud every time someone mentions ZFS on Linux. Ubuntu is shipping next release with ZFS on root. It will be awesome.
I thought the Ubuntu's decision that they can include it in their release was controversial but are they that convinced the license allows them to do that?
Well, it did originate from the realization that a good block caching layer designed for SSDs was most of the way to being a real filesystem in its own right. The caching is the biggest advantage bcachefs has over other filesystems, and the name does end in "FS". Perhaps it's a personal bias that makes you inclined to think dismissively of something as "just" a caching layer, when it's actually a really hard problem?
Perhaps it's a personal bias that makes you inclined to think dismissively of something as "just" a caching layer, when it's actually a really hard problem?
I was providing a possible answer to the question: why do people not give bcachefs more attention as a filesystem?
> Perhaps it's a personal bias that makes you inclined to think dismissively of something as "just" a caching layer, when it's actually a really hard problem?
That’s quite incendiary.
I also assumed it was a caching filesystem based on the name. Something you could attach to LVM or BTRFS to cache disk write or read on access.
Because despite the fact there is “fs” there is “cache” written there. And caching filesystems would indicate that instead of just using ram it would use some kind of traditional storage. It’s not unthinkable that this is a common thought process. What kind of “personal bias” could exist which alters what a caching filesystem would mean?
I didn't mean to be incendiary, just to point out that different experiences could be leading people to make inaccurate assumptions.
And I'm a bit confused by what you mean when you say "caching filesystem", since you seem to be using that term to talk about something that isn't a filesystem at all, rather than a filesystem that has advanced caching capabilities on its feature list. (cf. terms like "checksumming filesystem", "copy on write filesystem", "journalling filesystem") I was careful to refer to the original bcache and similar tools as "caching layers" rather than mislabel them as filesystems, but bcachefs is a proper filesystem.
A caching filesystem (as opposed to a standard block storage filesystem) would be one used for cached data specifically. ZFS has cache drives but the actual on disk layout does not resemble ZFS as we know it. That’s what I’m eluding to;
ZFS in this case is an all-in-one solution so it’s perfectly fine to refer to it as a feature of the filesystem, but externalising that to a separate agnostic product would result in this being exposed.
In fact CacheFS was designed exactly like this, and as you can see includes both “cache” and “FS” in the name.
https://lwn.net/Articles/100330/
unifying ARC with the OS block cache is huge for performance, i hope this feature is adopted by other ZFS implementations
The lack of a zero-copy sendfile for ZFS is one of several reasons that we (Netflix Open Connect) use UFS on our CDN nodes.
It would be interesting to see how he solved this. From a brief look at the FreeBSD code several years ago, I seem to recall that ARC dealt in 8K chunks (maybe because it is the sparc page size??), but x86_64 uses a 4K page size, and mmap() and sendfile() expect to deal in pages.
The other issue would be "taming" the ARC. We make heavy use of FreeBSD's SF_NOCACHE flag to sendfile(). This allows us to teach the VM system to keep hot content in the page cache, and drop less popular content.
> The other issue would be "taming" the ARC
Has anyone considered replacing ARC? It actually has fairly poor hit rates compared to other policies, with very modest improvements over LRU. I also wonder how much ghost entries contribute to space overhead, which is actually trivial to remove (ARC is an adaptive SLRU - so replace ghosts with a sampling hill climber).
Apple’s aborted port had this, but it was the source of a lot of bugs and instability. Probably could have been made stable with enough time and effort though.
> with the hope that Oracle may change and again open source their latest ZFS
Yeah, bud, good luck with that. Someone hasn't worked with Oracle.
Demo regarding unified ARC & Pagecache https://youtu.be/be0ph4b9vUE
Is that video a late april fools joke. I didn't yet watch it fully, but a presentation on zfs on Windows in an Internet Explorer window? Wtf?
You know, you say that, but...
https://github.com/openzfsonwindows/ZFSin
(I am aware this is not the codebase the presentation is running on or about.)
Didn't know that anyway. Thanks for the link
I hope the port is more stable than their server. Anyone have a mirror?
https://web.archive.org/web/20190418224604/https://www.cross...
Needs a repo and build/install instructions. Eager to see how this improves over ZFSonLinux
Not especially compelling as a link when the source repo is still empty, so we only have a video to go by.
It's incredible to me how many people are suddenly comfortable violating the license of ZFS, simply because Cannonical says it's okay? This is an open source project, under an OSI approved open source license (CDDL) which is quite known to not be GPL compatible. The license is derived from MPL 1.1, which is also known to not be GPL compatible. Why did they bother creating MPL 2, when we could have just ignored the GPL incompatibility in MPL 1.1?
You can't be someone who loses their mind over GPL violations, but then willfully misinterpret someone else's open source license because they have something you want. The developers involved specifically picked CDDL because it wasn't GPL compatible. That was their desire / intent, and so that's the license they went with. By ignoring the license and the clearly known intent of the developers in choosing that license, simply because they made something you really want, I believe that ZFS on Linux is really a form of theft in the open source world. It's the same sort of attitude that leads to GPL violations.
I don't agree with making things CDDL to be explicitly GPL incompatible, but I do believe in respecting the author of the software in question and the license he or she chose. If people choose licenses I don't agree with, I don't use their software. I don't just use it anyway and pretend like there's no license issue.
You don't have to be RMS to see that these people who are ignoring the licensing issues (not just shipping user mode tools like a FUSE implementation which is legit) honestly don't care about open source and licensing and copyleft. They just see something someone else made that they want and they take it and ignore the licensing and developer intent. How can we hope to enforce the GPL successfully but then ignore other OSI approved licenses when it suits us?
This is why big names in Linux kernel development aren't remotely interested in ZFS.
Not to mention the fact that the users of ZFS on Linux are all setting themselves up for Oracle to bring the licensing inquisition.
It's an open legal question if a user loadable module violates the CDDL/GPL. Until its fully litigated, its going to remain an open question.
As an example: the FSF's position is that yes, if you dynamically link against a GPLd binary that all the code has to be GPLd, and if one wants to link against a copy-left library in a non copy left program the library has to use the LGPL. However, the LGPL predates dynamic linking. It's from the days of static linking where the (L)GPLd could would be distributed together in a single binary. In the world of dynamic linking that isn't true anymore.
A primary example is libreadline. It is (or was?) GPLd and the FSF use this a hammer to try and get other code to be GPL. However, this (as far as I know) was never actually litigated.
As another example: the Linux kernel is under GPL2, Linus makes it clear that closed source programs can use the syscall interface without a problem, but its not clear that if he didn't say this that they couldn't. He includes the note to clarify his position that they can for those that would be concerned. i.e. the argument would be that without the syscall note, one couldn't ship any hardware device (say an appliance you stick in your rack) with linux and proprietary software. My gut is that most would find that difficult to swallow.
As an aside: I personally feel this has similarity to the google / oracle java lawsuit. If one thinks that APIs cannot be copyrighted then arguably anything that just uses APIs and doesn't embed the actual GPLd code one should be fine as the APIs aren't able to be copyrighted. If on the other hand one views APIs are able to copyrighted as well, then it would be much more clear that the FSF's position is correct. Though the usage might still be fair use (and to an extent one might argue that its even more fair use than Google's usage)
If this ever does get litigated, I'll be following closely as would be very interested (intellectually at least) in the result.
TLDR: until its actually litigated, everyone who claims to speak an absolute truth is speaking out of their ass. Lawyers are a conservative bunch in general so they'll give you the most conservative answer in terms of what you can do, that doesn't mean its the absolute truth in terms of what you can and cannot do.
What we do know is the intent of the developers in open sourcing their work and that those working on these ports are violating that intent. They specifically chose this license with the belief that it protected their software from being consumed by GNU/Linux.
The counterargument, maybe their license doesn't block us from using their work in the way they think it does, and until it's proven in court we won't know is extremely offensive and self-serving.
This is the same argument raised by many GPL violators. "The GPL hasn't been proven well enough in court in my jurisdiction so I'm just going to use your code in violation of your license terms."
This is theft, an open source license violation, and lawyering to defend that theft. It violates the principles of open source. We are literally discussing an OSI approved license. I believe that any responsible member of the open source community would respect the authors license and not try to lawyer their license out of your way so you can have their code on your terms.
If this was Linux devs and GPL, rather than Solaris devs and CDDL, everyone would be on the other side of this argument. It's hypocrisy and it should be called out.
Do we actually "know"? Cause I "know" that the opposition to GPL was not to make it incompatible with GPL (which is a point of contention actually, and hits deeply into question of when a code is derived code). The point was that making OpenSolaris GPL would block integration chances with NON-GPL projects.
In fact, at least one anecdote points out that Sun expected CDDL components to make it into Linux source, just like cases of other non-GPL components in the past (AFS, for example)
I am certainly looking forward to trying it out!
ZFS is a great filesystem and being able to possibly have a better implementation than we do at the moment, or at least, be able to spur additional development and optimization, is very welcome!