Linux file system developer: we're severely under-resourced
lore.kernel.org
I've said this previously, and I'll say it again: we're severely under-resourced. Not just XFS, the whole fsdevel community. As a developer and later a maintainer, I've learnt the hard way that there is a very large amount of non-coding work is necessary to build a good filesystem. There's enough not-really-coding work for several people. Instead, we lean hard on maintainers to do all that work. That might've worked acceptably for the first 20 years, but it doesn't now.
[…]
Dave and I are both burned out. I'm not sure Dave ever got past the 2017 burnout that lead to his resignation. Remarkably, he's still around. Is this (extended burnout) where I want to be in 2024? 2030? Hell no.
Reminds me of this xkcd post:
This problem is pretty common across most parts of the Linux space. Everyone wants to volunteer coding work, which is great, but not what's desperately needed right now.
The Linux community needs more than programmers, or else it will consist only of programmers. We need UI/UX experts, or we'll never have the simplicity and ease of use of iOS. We need accessibility designers or we'll never match up to the accessibility of MacOS. We need graphic designers and artists or we'll never look as good as Windows 11. We need PR professionals and marketing experts or we'll never be as notable as the Windows XP startup sound.
We don't have enough volunteers that fit into these categories. The next best thing you can do is contribute your money so that your favourite project can hire the people they need.
That's more the desktop environment than the Linux kernel though. Gnome, for example, is a simple, good looking, accessible desktop environment.
I love and use GNOME daily, but I think it's still the case that the interface "needs some getting used to" for a Windows/MacOS user. The design paradigm is just not familiar or self-explanatory to anyone who has regularly used desktop computers in the past decade.
why do you think thats unreasonable? its a different system
I think it is unreasonable because a Windows user (i.e. myself) can quickly get up to speed with MacOS within five minutes without the need for external instruction. I can manage a MacOS system perfectly fine even without any prior knowledge of how it works. I can figure out how to configure the settings to do what I need it to do without needing to search for how to do it online.
GNOME took almost a week to get used to and remember where things are located, such as what is located in Settings, how the task flow works, and so forth. I never got used to the "disappearing dock". I had to use an extension for that. GNOME is just way more different than the others. Meanwhile, my grandpa picked up Cinnamon as a lifelong Windows user within five minutes.
IDK, choosing between ctrl or the cmd key on macos always felt like a flip coin, I hated it
Why are you talking like gnome is the default Linux DE? Its not. As you said yourself cinnamon is better for some folks.
It's not, but what distros frequently top the list of "user-friendly" distros?
Ubuntu, PopOS, Fedora, and friends.
Maybe it's not how it should be, but that's currently how it is.
I never heard of PopOS, and fedora is definitely not meant to be user friendly. I'd have mentioned first Mint (which ships with cinnamon I believe) actually
I would encourage you to look up "user-friendly Linux distros" on your favourite search engine and check the first few results.
PopOS is System76's distro. It's quite popular among beginners and frequently recommended to those just starting with Linux. I don't personally use it.
Will do, it's been a while since I've been a beginner I suppose. Nonetheless, few years ago mint would have been the goto recommendation, at least in my circle. I think I even got my mom to try a live version out of curiosity
I mean going from windows to mac os also needs getting used to beacuse Obviusly they are not the same systems . Each of those has diffrent design philosophy. And out of all linux GUI gnome is the simplest( In my opinion also the best looking one ). And most of the rest also takes very little of getting used to since they are very similar to either windows or MacOS( elementary ).
What are the main issues in getting used to it do you think?
The first thing I always hear from people trying out gnome for the first time is along the lines of "Where is the minimize and maximum buttons?" and depending on what programs they use "where is the icon tray" (app indicators, or the "system tray" on Windows).
Whenever I try to explain the devs' philosophies regarding those, they quickly have lost excitement so generally these days I just start people on KDE.
The minimise and maximise buttons are in the same place they are on windows, to the left of the close button?
And by icon indicators, do you mean the dock? It's possible to pin that.
Gnome by default does not have those buttons enabled. Their design vision is for you to not actually have to minimize a window, but rather if you need to focus on a specific window either maximize it (in which you double click the app's header or drag it to the top of the screen), or move that window to a different workspace. The options are technically still in Gnome, and can be enabled via either dconf editor, or through the Gnome Tweaks app - however, a few distros enable it out of box. If you use a distro that has a more vanilla Gnome experience, such as Fedora, this won't be the case.
By icon tray / app indicators, I mean apps that show some sort of status or shortcut in the bottom right area of Windows / KDE (or the top right of macOS). On my desktop right now, that would be Discord, JetBrains Toolbox, and KSnip (the last two are extension icons).
One of the major reasons I outright hate Gnome, they have strong opinions on how the users should want to use their computers. "You shouldn't want to minimize windows, so we disable the button to do that."
No thanks.
Ugh, Gnome 3... I used to be all in on Gnome, then went to Mate after the Gnome Shell days. Last year I came full circle to KDE and for whatever reasons I had to dislike it in the past, KDE Plasma is a damn good environment these days.
Gnome used to be better for being simple and effective. Now you have to tweak it so much to get what you want, you might as well go with KDE.
Qt feels more dominant in the cross platform space these days too, with more new projects choosing Qt over GTK
I don't know a single ui person or artist that gives a single f about open source.
I think as a dev you naturally grow to respect the open source space because of how much you take from it.
I would love to be proven wrong though
I can name one at least: https://www.davidrevoy.com/
I mean, Microsoft and Apple have career pages stating exactly what is needed. Does Linux have something like this for volunteers? I think it's just very inaccessible to someone from the outside. Maybe start there, improve the recruitment of volunteers
This, very very much.
I can't code for shit, but am willing to put some time and effort into FOSS.
But I'm not sure what's needed, or where is needed.
Have these projects ever considered just asking for the help they need?
It's extremely inaccessible from the outside. Non-devs don't know what to do with it.
We brought Linux laptops into our company (1500+ laptops during the pandemic) and after a year, half the devs and design team switched to Macs. The Linux users who stayed are pretty much using the laptop as a glorified netbook where they just use Google doc suites. And whenever they encounter a problem... Searching leads to nerds arguing about what version of Linux they should be using instead. Or the answer requires them to go into terminal to fix a problem.
I have a bunch of users asking about Elementary OS, for the sole reason in that it "looks" good.
Honestly I would use it for that reason too, you can install the Pantheon desktop on NixOS, but the gnome ecosystem is just too good
This so so much. I want to contribute. The only time I ever have are when the barrier to getting setup and finding things I can do are low.
The "open" management of bigger open source problems is a kafkaesque nightmare. If you want to help make something better and change it somehow, you have to go on week-long journeys trying to figure out who is in control of that part of the project, who you can ask for guidance, who knows anything at all...
E.g. once I wanted to help package a new version of a software for a big linux distribution... and literally all the (~10) mantainers apparently wen't missing a few years ago. I managed to find one of the mantainers private reddit account and contacted them there, and they just made me a mantainer. And I still couldn't do shit because there is another dependecy which also needs to be updated, but it's mantainers are also all dead.
The effort of even getting to the point where you could contribute something meaningful, is like 100 times more than the effort of the actual contribution. It's completely rotten.
Unfortunately that's the nature of volunteer work, people just lose interest in some topics. It's worse the more niche a topic is. Sometimes your only option is to fork
How does someone like me, a PhD candidate who has a background in Human-Computer Interaction, contribute? Or the accessibility researchers I collaborate with who sit one office down and use and love Linux and make their own hacks for Linux to improve its accessibility?
I think part of the problem is a matchmaking (as as you said a PR) problem. I'm sure a lot of the experts you describe would love to contribute some cycles (including myself) but your post is the first time I'm hearing that people like me are needed in my HCI capacity. Otherwise I would have assumed there are plenty of programmers better than me available so why bother?
Start a blog where you get users to try and complete a task on a stock Linux install, and write about it.
"Given a stock Debian XFCE install, users were tasked to connect their Airpods pro. Here's what happened"
(Even doing this for Lemmy, I.e. "find a community about birds", would be insanely helpful)
A blog like this would help give focus for developers on what needs to be improved, in every part of the stack!
I like that. It seems the internet with theoretically good answers for things that can be difficult to find. Blog format is much easier to skim that SO, and as long as the post titles are valid and meaningful, it’s a quick check for usefulness before reading.
use open source software. whenever you see potential to improve, open an issue with detailed explaination.
try to solve the problem yourself. send pull requests. adapt them according to instructions by devs. when they're good, the will get merged.
if you use the software a lot, contribute whenever you think something is needed.
that second part is exactly why there is a lack of hci work in the linux space. Hci is a specialty, just like coding is. The standard ask of "create a pr with the code" is asking peopae who don't typically code to do so, in addition to doing the work of researching the problm, designing a solution, and then testing that solution for suitability.
Since the only mechanism most open source projects have for accepting contribution is code, and the ask is usually for code, there is never even an opportunity to submit design work.
I think most of the time if they have a Github/Gitlab repo open to the public opening issues, they will accept an issue that merely describes a problem that needs to be fixed along with how it might be fixed.
Something like this is generally appreciated:
...and if the maintainers ask for contact info, provide it and there you go.
nowadays, design work is code (or at least markup). Like .qml or .glade or even html/css.
encounter a problem and learning a new tool to solve it, is a pretty standard situation in IT. Designers have varying capabilities in that domain. While a print designer might have trouble learning a new software, a MMI-Architect or UI Designer should commonly be able to learn a new UI framework (designer software).
Generally, if you like to contribute to any project (not limited to software), you can't expect the project to adapt to your toolset.
But often it's enough to share an idea for it to get adapted.
Sure, you can pr the design files, but thats not how the messaging comes across. Even the "how to contribute" for most projects, if they have one, is usually entirely technical. The majority of designers (not all) I have worked with have been very shy about technical work, so having no clear "non-technical" contribution pathway is a deterent.
you can start by opening an issue, sharing your idea, uploading drafts etc. like I mentioned.
because what you wish contributing to is a technical matter. You can't modify airplane design without a single clue on aerodynamics. Same with software.
Design of UX is a separate craft from programing, to follow your own analogy, you don't need to know electrical engineering to design an airplane control panel
lol, of course you do. and A LOT more. Hence it's usually done by a team of experts. Pilots, eengineers, manufacturing, regulation/standards, safety,...
UX is just one of many parts of the expertise needed. same in software.
What you're claiming is wishful thinking that has nothing to do with real world workflows. I also don't know a single UI designer having the troubles you are pointing out.
That's the difference between designer and UI Designer. Just because you can draw an UI with your favourite tool, doesn't make you an UI design expert.
I feel like, with this reply, you are willfully glossing over my point. The issue at hand is that open source software is short on the ux design expertise. My claim is that by centering the programing expertise, and in fact by not going out of the way to be inviting to the non-programming expertise, open source projects are self-perpetuating these cycles.
We can find ways to invite good designers in, or we can continue with the "sufficient" design ost projects currently have.
I'm happy if people have ather strategies for overcoming the current problem, but the current aproach is not doing it
I think ui/ux has the opposite problem where there are not enough programmers. Idk if there is a shortage of ux people, but I do know that it takes a significant amount of work to implement and maintain ui projects.
If you are interested in helping out though, you should join the discussion forums for a project like KDE or Gnome. I doubt the Gnome people will give you the time of day, but there are many KDE projects that would appreciate volunteer work from someone like you. Browse the discussions at htltps://discuss.kde.org and see if you can find a project to latch on to, or find a project you think you can improve, and proactively reach out to the maintainer(s) with a proposal.
Just be mindful that it may take a while before your changes are implemented, since everyone is a volunteer with other jobs and responsibilities.
This plea for help is specifically for non-coding, but still deeply technical work.
Without too much knowledge, I have the strong feeling this is equally true for the Fediverse and Lemmy.
And while it is fairly obvious and straight forward how to contribute as a programmer, it's less so for all the other, equally important, tasks.
I know a thing or two about UI/UX, but tbh it feels very weird to me to butt in somewhere and tell people what to do. Because unfortunately, while I probably could design a good UI, I have no experience in implementing UIs.
I can deal with C++ so I thought I might as well learn Qt, but I couldn't even get Qt Creator or KDevelop to run properly on my distro and didn't find the time to get into it since.
This stuff sadly not that trivial which is probably why you don't find too many people who can do it.
Honestly this sounds wild to me, I've never had Qt Creator fail on me. Do you use a distro that doesn't distribute headers with packages?
I use KDE on top of Arch and had (very) broken theming and some missing packages I think. I just installed the package, started the thing and hoped for the best tho - I really should look up some guide
Arch is not intended as a distro for the average user, it's intended for a user who can use the advantages of a basic highly customizable system
I use Arch, btw
I'm not sure the UI/UX is anyhow related to kernel development.
I3/Sway/Hyprland/other WMs are already much more superior to anything that can be achieved in Win/Mac desktops in terms of flow and performance. Only polishing is needed. Other desktop options like Gnome/KDE are fine and more lightweight desktops are okay too. They need polishing too, but already great.
Isn't there some kind of volunteer-project market with easy access?
there's https://bountysource.com/ or https://www.vamp.sh/
Bound to specific problems and limited to projects willing to pay money.
Is it just me or does vamp.sh fail with an error on firefox?
Worked for a second before it threw me an error.
It's failing because it can't connect to some crypto web3 bullshit... It's probably best if it doesn't load tbh.
And Bountysource is also dead crypto shit. See: https://github.com/bountysource/core/issues/1586
I understand what you say, and I partly agree, but neither do I want the forced simplicity of iOS on Linux, nor graphic design like windows 11. And I'm pretty sure I'm not alone. Please don't uphold these as some holy grails, they are not.
They didn't say anything about "forced simplicity". Not everything is a slippery slope
Linux's power is coincidentally its weakness. I don't think the broader community wants that simplicity imposed.
Step 1: Don't murder your wife
This must be a reference to someone, right? Casual femicide sounds a bit off.
ReiserFS
https://arstechnica.com/gadgets/2023/08/the-torrid-saga-of-reiserfs-nears-its-end-with-obsolete-label-in-linux-kernel/
Awful and horrifying.
I think a lot of people don't know that about ReiserFS. 'Geek defense', wtf.
I'm old. I was around for it. It was a shock to the Linux community.
I remember when ReiserFS was prolific as 'a beowulf cluster of' and stage1 Gentoo cred, which are also likely all but forgotten.
Hard to believe this is old time trivia now. Time flies.
https://en.m.wikipedia.org/wiki/ReiserFS
Oof. The filesystem was great though.
I think I'm missing something?
Hans Reiser developed ReiserFS, which was a good fs for it's time, and was then convicted of murdering his wife.
Oh 😲
Username checks out.
The guy is definitely a genius.
On the bright side, he should have all the time in the world to code, after going to prison.
The linux kernel is very old school in how it is run and originally a big part of the DevSecOps movement was removing a lot of manual overhead.
Moving on to something like Gitea (codeberg) would give you a better diff view and is quicker/easier than posting a patch to a mailing list.
The branching model of the kernel is something people write up on paper that looks great (much like Gitflow) but is really time consuming to manage. Moving to feature branch workflow and creating a release branches as part of the release process allows a ton of things to be automated and simplified.
Similarly file systems aren't really device specific, so you could build system tests for them for benchmarking and standard use cases.
Setting up a CI to perform smoke testing and linting, is fairly standard.
Its really easy to setup a CI to trigger when a new branch/pr is created/updated, this means review becomes reduced to checking business logic which makes reviews really quick and easy.
Similarly moving on to a decent issue tracker, Jira's support for Epic's/stories/tasks/capabilities and its linking ability is a huge simplifier for long term planning.
You can do things like define OKR's and then attach Epics to them and Stories/tasks to epics which lets you track progress to goals.
You can use issues the way the linux community currently uses mailing lists.
Combined with a Kanban board for tracking, progress of tickets. You remove a ton of pain.
Although open source issue trackers are missing the key productivity enablers of Jira, which makes these improvements hard to realise.
The issue is people, the linux kernel maintainers have been working one way for decades. Getting them to adopt new tools will be heavily resisted, same with changing how they work.
Its like everyone outside, knows a breaking the ABI definition from the sub system implementation would create a far more stable ABI which would solve a bunch of issues and allow change when needed, except no one in the kernel will entertain the idea.
You offered a lot of suggestions, and I'm sure people will disagree over the specifics, but I think your overall point is excellent and not talked about enough. I wonder if anyone has ever even attempted a survey on the ages of maintainers/contributors? I bet it's skewing older fast.
Nothing wrong with that of course, especially given the project's age, complexity, and being written in C - but you're right, at some point you have to attract new talent - people can't maintain forever.
I'm a 29 year old developer - I didn't even know you could do git patches via email until recently. And while it's super cool, it also sounds kinda terrible, especially at the volume they must be receiving? Their own docs are saying the mailing lists receive some 500 emails per day and I can't imagine the merge process is fun.
So many doc pages are dedicated to how to submit a patch - which is great that it's documented, and I'm sure it will always be somewhat complicated for a large project - but it also feels like things that are all automatically handled by newer tools / bots which can automatically enforce style checks, etc.
I guess they could argue that the complicated process acts as a filter to people submitting PRs who don't know what they are doing, but I'd argue it also shuts out talented engineers who don't have 40 hours to learn how to submit a patch to a project on top of also learning the kernel and also fixing the bug in question.
From what little I read of their git process, does anyone know if there's anything preventing the maintainer of a subsystem from setting up a more modern method for receiving patches? As long as the upstream artifact to the kernel has the expected format?
What long term planning? Who is going to come up with that plan? Will everyone agree to that plan? Who will be paying for the resources to work on that plan?
I am not seeing how that would help. What are you going to do if there is no progress on something? Fire volunteer X because he didn't make progress on ticket Y (as he has no interest in ticket Y)?
Modern ticket system or issue tracker? Yes, absolutely. But Jira? Certainly not, considering Atlassian's business practices. A project like Linux deserves a system where they can maintain some control and it probably should be open source.
Yeah email is ancient and certainly terrible from a usability perspective if you're an outsider to the workflows, but at least it can't be shut off or taken away on a whim. Also it's universal and therefore accessible.
Yeah. If core Linux teams publicly adopt an open source KanBan board, issue tracker, and code review tool, I am a lot more likely to adopt and help maintain those products.
I have no interest in navigating the politics and history to join any core Linux teams, but I contribute in other parts of open source.
I guess I'm saying the only place I'm really interested in working with the core teams is on whatever DevOps tools they adopt.
We also can’t go back to fix SMTP either. Simply saying “we need to break ABI” is not a solution for the same reason, the fear Darrick mentions in the email is justified, it’s the fear of e-mails when you change something that breaks everyone’s widget in a way they don’t expect or care to learn about and demand you fix it for them. It’s all well and good to say that they should simply wash their hands of it, but that’s not an option when your inbox is flooded with automated bug reports that don’t explain the problem
I am actually arguing for a stable ABI.
The few times I have had to compile out of tree drivers for the linux kernel its usually failed because the ABI has changed.
Each time I have looked into it, I found code churn, e.g. changing an enum to a char (or the other way) or messing with the parameter order.
If I was empire of the world, the linux kernel would be built using conan.io, with device trees pulling down drivers as dependencies.
The Linux ABI Headers would move out into their own seperately managed project. Which is released and managed at its own rate. Subsystem maintainers would have to raise pull requests to change the ABI and changing a parameter from enum to char because you prefer chars wouldn't be good enough.
Each subsystem would be its own "project" and with a logical repository structure (e.g. intel and amd gpu drivers don't share code so why would they be in the same repo?) And built against the appropriate ABI version with each repository released at its own rate.
Unsupported drivers would then be forked into their own repositories. This simplifies depreciation since its external to the supported drivers and doesn't need to be refactored or maintained. If distributions can build them and want to include the driver they can.
Linus job would be to maintain the core kernel, device trees and ABI projects and provide a bill of materials for a selection of linux kernel/abi/drivers version which are supported.
Lastly since every driver is a descrete buildable component, it would make it far easier for distributions to check if the driver is compatible (e.g. change a dependency version and build) with the kernel ABI they are using and provide new drivers with the build.
None of this will ever happen. C/C++ developers loath dependency management and people can ve stringly attached to mono repos for some reason.
Kernel is not a monolithic application, and you cannot develop it like one. There are tons of actors: independent developers, small support companies (like Collabora), corporations, all with different priorities. There is a large number of independent forks (e.g. for obscure devices), that will never be merged, but need to merge e.g. security patches from the mainline. A single project management tool won't do, not your typical business grade tracking&reporting tool.
CI is already there. Not a central one—again, distributed across different organizations. Different organizations have different needs for CI, e.g. supporting weird architectures that they need to develop against.
There is a reason Torvalds created git—existing tools just wouldn't work. There might be a place for a similar revolution regarding a bugtracker…
Is there a link to help? I want to help
https://kernelnewbies.org/
Thank you
Same.
There are so many filesystems. So. Many.
Have any of them considered actually dropping some and pooling efforts into the more promising ones?
Isn't "anyone can fork if a project doesn't really fit their taste" sort of the curse of open source?
Swallowing your pride, merging into another project and taking a less glamorous role in that project is not as easy as it was to fork when steering your project.
This is generally speaking. I'm definitively not saying any of this is that case with the XFS project.
Ps. Murdering your wife is also something that seems to be bad for filesystems....
I don't think it's because of the ego. But if you're working with other people, you need to do a lot of non-coding (non-fun) things. Align thinking, find compromises, establish and follow processes. Things are easier and more fun hacking alone. No processes to limit you, no one telling you "this doesn't align with the vision of the project" (and the other way round - you don't have to maintain code contributed by other people with use cases not interesting to you) etc. For volunteer FOSS contributors, doing fun stuff is often a big part of the motivation to give their free time to the community.
Boy do I find mailinglists impossible to read.
Ontopic: I appreciate what these people are doing, but were I them, a lousy community would have me quit very quickly or never join the project to begin with. Maybe the should just quit and when shit hits the commercial fan, they'll either pay to get it fixed, get somebody on it to get it fixed, or move on.
Probably fixing whatever non-dev problems they have to make it a nice community to join wouldn't hurt either. For me, it's the lack of time and C code. C is a language I absolutely will not touch. There's not much worse than it IMO and it's refreshing to see rust slowly entering the kernel with all its tooling.
How can we help spread the load? Can we have volunteer sign ups and shift work?
I guess the best start would be to have a person to organize volunteers.
I think that might help, asking for volunteers. Segregating them based on their abilities (LinkedIn profile for instance). There are a lot of people who have a couple of hours a week they can spend for such a cause. Please check with your colleagues and see if asking for volunteers is possible.
Donating money is also an option, but in this economy, it's not sustainable to donate significant sums on a recurring basis for most.
I'm one of those people. How do we organize this?
Go door to door like the Jehovah Witnesses
Pretty much, just file system is boring work
i think the inner workings of an OS is pretty much awesome.
one has to look into to see the beauty.
What kind of non-coding work do you need done?
the email lists:
I don't think non-coding work means non-technical work - you'll likely still need to dig deep into the technical details to actually be able to help
There is also a link to docs: add maintainer entry profile for XFS
I’m having trouble understanding the email, is this for people to help with fs or xfs or both or are they effectively equal? The linked patch is for XFS which I understand Darrick has quit, is he still on fsdevel? Is this plea for either ?
The thread is an attempt to merge a new file system,
bcachefs. This is a large change, requiring a lot of review from experienced developers, and getting anyone to do this work turned out to be difficult. Darrick here started talking how, in general, all development of file systems in Linux is troubled by lack of manpower.So if this is a plea for help, where should I go to see what I can do?