What software stack would you have chosen for Lemmy?
Lemmy is an open-source, federated link aggregator and discussion platform similar to Reddit, Lobste.rs, or Hacker News. The software stack used in Lemmy includes Rust with Actix and Diesel for the backend, and TypeScript with Inferno for the frontend.
The developers chose Rust for its performance, safety, and concurrency features, which help in building a reliable and efficient backend. Actix is a popular Rust web framework that provides a lightweight and fast foundation for the server-side application. Diesel is a Rust ORM (Object-Relational Mapping) and query builder that simplifies database interactions. On the frontend, TypeScript offers better type safety and tooling compared to plain JavaScript, while Inferno is a fast and lightweight React-like library for building user interfaces. These technologies were chosen to create a performant, reliable, and easily deployable platform.
I read someone saying that the lack of contributors was due to the software stack being unconventional and takes people a while to get used to it. So I was curious to know what other people would have used.
So Rust is capital H Hard to hire good developers. There just ain't enough with experience in not only the language, but service management, the rest of the SDLC, SLAs, getting the best out of Diesel and knowing DBs, and the grizzled nature needed to be leading teams etc. There just isn't enough of it in production to have built up this industry level that python, js, php, c#, and java have. Typescript is also pretty niche still, and thus has similar problems, though not to the same extent.
And I've noticed pretty much every backend dev would like Rust, every front Typescript. And fullstack will learn and run everything and anything anyway. And so this is a temporary problem. Early adoptors, like lemmy, will see more people contribute, more learn, more launch more stuff. And that starts with open source projects.
And thus: I'd make exactly the same decisions. Lemmy isn't a bank, it's more a start-up, it doesn't pay well, it has to attract on other benefits. Attracting volunteers might be easier in desirable languages, will certainly be easier in ten years time say than Rails, Play, or Laravel might be. CV fluffing from OS contributions is a fantastic way to make yourself more marketable, and who doesn't want TS/Rust on that CV?
I think Rust is a perfect choice here. Considering the investments of the Linux kernel, AWS, Microsoft and so on, I think Rust is a future-proof bet.
That said it think the programming language is not everything. It seems to me that lemmy was written under the assumptions that there will be a lot more hosted instances that will fedrate but a lot of load seems to centralize on a handful of instances now (i.e. lemmy.world).
To support these it could make sense to rethink the system design to something that offers better support for high-load and high-availability scenarios.
Backend in Javascript (node) and frontend in PHP (web assembly), the only sane way.
The database should be made in Flash.
Nah, database as one huge JSON stored in local storage. Wait, did I accidentally invent MongoDB in a browser?
OpenSilverlight would like a word.
Just use Cobol. Lots of Cobol developers are idle these days.
Backend in Excel
I think Rust is a little hard to find contributors, and is also just hard in general. Too low level.
I would opt for a language like Kotlin, Go or TypeScript. Easier than Rust, more popular than, or similar to Rust popularity.
JS, Python and ruby might also be very easy and popular, but they're a little too loosey for me for a large project.
If I were to do it today, I would choose Scala purely out of interest. Such a cool language. But it's probably less practical and popular than the choices I named above
I'd argue the opposite - Rust is the perfect fit for the entire stack. We don't even need JS with libraries like Yew
What does yew do?
You can write fronted in it without using JS, thanks to WASM.
I agree. Rust has advantages, but none of them outweighs the negatives (complexity, difficult to find devs) for this particular use case.
I also agree that JVM would be a good platform. It's both performant enough and simple/conventional enough.
The thing with sum types, async support, trait system, is low level?
The backend is quite alright. The Rust backend makes it indimidating to approach, but I know it has many advantages.
The frontend could use LOTS of changes. I don't like Inferno, it's messy and confusing to work with. Instead, I would have opted for a Svelte+Tailwind stack for the UI.
Svelte is such a pleasure to work with.
Rails. Of course, I'm biased because it's what I know the best, but also because Mastodon is written with Rails so in terms of getting more contributors from a similar project I think Rails would have been a good choice.
I also looked into helping with Lemmy development, but ran into the same problem you've described: I don't know anything about any parts of the stack. I run my own instance so I can run and debug it well enough, but actively working on new features/bug fixes would require more time to get up to speed than I can put in currently.
Edit: I'm not one to complain about imaginary internet downvotes, but can we be better than reddit please? Downvote is not a disagree button. The question was what would I have chosen, not what the most popular web framework is this year. It's cool if you would have chosen something else. This whole thread has become "what framework does everyone like the most currently?"
and at some point in the future, the possibility of upgrading from Ruby to Crystal …
Rails is terrible. For something like Lemmy I'm confident it would just fall over (or require a much higher hosting burden). There's way too much data to render and retrieve in a system like Lemmy, the caching systems for JSON shuck (hello jbuilder caching objects instead of json), Ruby performance is awful, there's no mitigating factor to allow efficient blocking on IO bound requests (like coroutines or fibers), Ruby used a non-compacting garbage collector so it's subject to heap fragmentation and wasting memory, there's no static typing (or even type hinting), the Ruby ecosystem is built on Rails alone at this point (few if any libraries to make use of), the principle of "developer happiness" results in the need for tools like Rubocop to "resolve" invented problems WRT "how do you style X in a particular context" (among other things), and to top it off the framework makes doing the wrong thing easy and the right thing harder (Rails apps are so prone to N + 1 database queries).
It's also a ridiculously hard thing to hire for because basically nobody teaches Ruby or Rails. There's a very limited pool of Rails experts.
If some of that has changed I'd love to know, but after working in rails for ~3 years... I have no love for it. The only thing I'd ever use rails for is a small scale project, like a small business website/webstore. IMO, it's not suitable for websites that need to serve large volumes of data that doesn't cache well at the page level.
Well, it seems to work well enough for Mastodon. I'd also point out that there are websites far larger than Lemmy using Rails at scale (Shopify and GitHub being the primary examples). It scales quite well.
Without benchmarks, this is a subjective statement. Ruby 3 is quite speedy now, at least I find it to be. Performance is even better with YJIT enabled in Ruby 3.2. I think this "Ruby is slow!" stereotype is left over from before Ruby 1.9 when YARV was introduced. Is it as fast as a native language? No and it never will be, nor will any other interpreted language. But this idea that Ruby is somehow the slowest possible interpreted language has not been true for a very long time now.
I'm not entirely sure what you mean by this, but Ruby introduced fibers a while ago.
Not true anymore. Ruby has had a compacting GC for a while now. This was a difficult problem to solve because of Ruby's support for native extensions. Otherwise it would have been done much sooner.
You can use RBS (built into Ruby) or Sorbet (third-party Shopify project) for this if you want. Or don't, I personally prefer Ruby's duck typing. I don't think a language using someone's preferred type system makes it "terrible" just because it's not what someone likes. Other people are perfectly happy with it.
Ruby being a one-stop-shop for Rails has been a criticism of it for years. I'm not sure how that's a knock against Rails or Ruby though. I personally write all sorts of non-Rails things with Ruby as I love the syntax and greatly enjoy writing Ruby code over that of, say, Python. To each their own though. There's Sinatra, Jekyll, Fluentd, Chef, etc. all written in Ruby.
I personally love the configurability that Rubocop provides. Some people like the "one true way" of formatting for languages like Go (which also for some reason thinks single letter variable names everywhere is a great idea), but other linters like Eslint for JavaScript have the same level of configurability as Rubocop. Alternatively, just don't use them. Rubocop is not something you need to use to use Rails.
Drop in gem that will automatically solve 99% of N+1 issues: https://github.com/clio/jit_preloader
Again, subjective statement here. My company has no problem hiring Rails devs. Granted, we operate on the idea that any good engineer can get up to speed with a framework in a reasonable amount of time. I don't believe in the "you are only worth the frameworks that you know" idea. Give me three months and I'm confident I could be proficient & competent with any web framework. Not a true expert per se, but knowledgeable enough to get stuff done, which at the end of the day is what you need to do. Those are the people I want to hire. Not the person that's going to be afraid to jump into something new because they've never worked with it before.
Am I saying that Ruby/Rails is perfect and there's no problems? Of course not. There's no such thing as a perfect language or framework. But I also think that Ruby is unfairly demonized because of lack of familiarity and it being less used than JS and Python frameworks. And oh boy do I have complaints about JS frameworks. But I wouldn't go so far as to call any of them "terrible." Maybe it's not my/your cup of tea and that's okay.
I could be mistaken, but I believe these sites are normally "using rails" (as in they shell out to other languages, Go has become common, for performance critical end points or portions of those end points that need to work a bit harder). I also suspect they've got some pretty heavy modifications.
I haven't worked with Ruby since... 2.5(?). That's very fair about the JIT improvements.
The Ruby language itself has fibers, but last I knew Rails was not making use of fibers for ActiveRecord. The problem is effectively you're in an interpreted language (bad enough) without real parallelism (worse). So, when any IO occurs one of your non-parallel threads is blocked. Ruby jumps around to process the different threads, but this creates a context switching problem and you lose a lot of performance to the preemptive multitasking's context switches (and resources are hard locked up by the IO).
With a coroutines or fibers based approach (cooperative multitasking) you don't need multiple threads, the "callbacks" just fire into the appropriate place when IO is fulfilled (what NodeJS is famous for). You can get way better throughput and it becomes trivial to execute queries that aren't co-dependent in parallel (so your page that depends on 10 different queries gets resolved much faster, and your app can work on other pages while that data is being resolved in the background).
It's not a silver bullet, you can run into issues if you have CPU bound tasks on the event loop thread (and thus create latency issues as things just aren't getting finished). However, in a well designed web app that shouldn't happen anyways (this stuff should be in background job queues, or at the very least running in separate threads -- in Ruby that likely means processes unless the threading has significantly improved).
That's good to hear.
RBS is definitely newer (I stopped working with Ruby around 2018). Looks like Sobet was just getting started. Also good to hear.
I've changed my stance over the years from "it's a matter of preference" to "it's objectively bad." The lack of any type structure leads to a significantly larger requirement in unit testing. Unit testing can be good, don't get me wrong, but in our org it had gotten to the point we were writing unit tests that effectively tested "did the code change" ... and I don't think that's a good workflow. You're lacking a compiler to do type checks for you (when you refactor something) so you have to write all the code to catch things the type system would've taken care of for you.
Put another way, I don't consider a unit test for a function as trivial as:
To be superior to the type system telling me
y
no longer exists. However, ify
no longer exist, I still want to know before a customer bumps into the inevitable crash.The problem you run into here is a lack of mature libraries to leverage if you want to do anything non-trivial. e.g., our application needed to read from spreadsheets. The Ruby libraries either A) didn't have support for common formats like
xls
or B) would load the entire spread sheet into memory represented as Ruby objects (you can predict how well that performed :) ).Without potentially breaking a NDA, that made the effort to get efficient spread sheet processing much more challenging than it would've been if we'd picked a language with more reach; this is just the main example I think of, there were others.
I think we eventually started using something akin to that. However, I'm a big advocate for making the wrong thing look wrong/complicated, and Rails very often makes the wrong thing look simple. Note that gems like this don't really solve the problem they just inform you when you've made the mistake, or alternatively forcing your app into hammering your database even in situations when it doesn't need to (which can drive you back to my earlier points about the lack of cooperative mulitasking to deal with the IO bottlenecks).
I agree, but there is something to be said for hiring people that are extremely knowledgeable in the framework to help highlight solutions (like those you've mentioned here) vs "you're a great C++ dev, now go do Ruby!" It's not a deal breaker, but when you hire folks like that the time to get them up to speed is going to be far higher, and they're going to make mistakes that in a sense "aren't at their experience level" (particularly with the former point about making the wrong thing look wrong considered).
I'll concede my initial reply/stance was a bit bold, and you've definitely highlighted serious improvements in the ecosystem I've either overlooked or forgotten about... Perhaps I should've said was terrible; I'm not rooting for Ruby to fail, but I did have a bad experience.
I can't speak to what they're doing internally, but I would agree this is a safe assumption. I think the point though is that Rails, with some help of external services depending on use case, can certainly be scaled out to a level that is far above the average web service making it more than capable of handling 95% of use cases you could throw at it. In the case of Shopify, they've traditionally been great about contributing optimizations back to Rails so their performance optimizations are shared with the larger ecosystem (and has the benefit of keeping them closer to upstream).
To your point, Ruby recently introduced Ractors which are true parallelism. They're new enough that their use isn't widespread yet, but I've played around with them and it's definitely neat to have real parallelism in Ruby now. And for web services, this would depend too on the web server. For example, using Puma or Unicorn will have multiple worker processes so there is some parallelism between requests regardless.
Yeah, I get that. For what it's worth, I really can't think of a situation where there hasn't been some library written in Ruby for something that I needed shy of extremely esoteric stuff that I likely would have needed to write myself if working in another language anyway. But that's going to be highly dependent on a case-by-case basis. For what it's worth, I make it a point to use as few third-party libraries as I can unless they're highly popular. It's a problem in all languages that random person's pet project library, while highly useful, becomes abandonware far too often.
You might be referring to the Bullet gem which is just a notification that there's an N+1 and where to find it so it may be fixed. However, the JIT Preloader gem actually does automatically solve the problem of N+1s in nearly all cases (see the README for details if you're curious). It's the closest to a silver bullet solution for the N+1 problem as I've seen and I now give almost zero thought to N+1s anymore. I know the devs were wanting to get it merged into Rails to solve this problem for everyone, but I don't know what happened to that effort.
Right, I wouldn't portend that anyone can make an easy switch from, say, embedded systems to web development in a few weeks or even a few months. I meant more like if you're competent web developer the core concepts of building web backends/frontends don't vary all that much between frameworks. At the end of the day, the underlying concepts deal with HTTP and HTML/JS/CSS so if you have a solid understanding of the base concepts and system design for the backend it shouldn't be much trouble to switch the framework sitting on top of those, especially if you have a team around you that is effective at code reviews, answering questions, and generally investing in new employees. Like you said, switching from something totally unrelated is a different situation though.
Thanks for a thoughtful reply, by the way. :) I really shy away from getting into pointless internet fanboy debates over which tool/language/framework is "best" but always enjoy when there's thought out reasoning behind points.
I'd use what I've been experimenting with exclusively on personal projects: htmx, AlpineJs and Razor Pages on PostgreSQL AKA the HARP stack. Obviously, a hilarious acronym was needed.
Which might sound esoteric and hipster, but I'd contend it's pretty close to how we were building websites for decades before the cult of the SPA took over. For those not in the know, HARP is built with no fe frameworks, everything is rendered server side and html is swapped in the DOM on the fly. Htmx is a very tiny js library that makes backend requests to the server, and renders the returned htmx within the current page. AlpineJs is a client-side js library that acts like a modernized and simplified jQuery. Razor Pages is part of the ASP.NET web framework that runs on .Net, and produces html from Razor templates coded with C#. My professional work is on SQL Server, but I like PostgreSQL as the runner up because I'm not paying mssql out of my own pocket.
I'm wouldn't be concerned with hiring since I'd mostly just need C# developers with some designers. .Net developers are a dime a dozen, and many are seasoned vets with 15+ years experience building with .Net. It's easy to build a career with just C#/.net/asp.net so few of these devs are running around flipping frameworks every few hype cycles.
But I might have just shown my age and bias.
The behaviour you described is very similar to what Blazor is. No frontend framework, and the DOM updates are rendered on the fly. I'm using it a lot and it's very good.
I like Blazor and use it exclusively at my work (usually to build the same type of stuff I'd use a HARP approach in a personal prj).
Blazor is awesome, but really is attractive to backend .Net developers more than anyone else. However, Blazor has a bunch of downsides: Blazor Server is too chatty to build scalable public facing webapps. Blazor WASM has a massive initial payload, which makes it slow and heavy.
Also, it just really falls into being overkill for so much stuff on the web. Half the shit I'm paid to build with Blazor would be faster and cheaper with just some htmx. Most SPAs are attempting to build a sand castle with an excavator.
That's my case, i'm a backend developer, more specialized in databases than websites, so i choose blazor because it was easy for me to become proficient fast and build nice and fast apps.
Also i usually build Enterprise apps, not public facing ones, so my target is still good for blazor. I know there are methods to scale better with blazor server (like using signalr on a separate service, outside the webapp).
Also we use DevExpress components, so basically all UI controls are done for me, so my dev time is even lower.
I'll look at htmx, i'm curious at the tecnical differences and why it may be lighter than blazor.
Blazor WASM doesn't need to have a massive initial payload, that's only if you're making your projects too big, and aren't really structuring them correctly. To be fair, Microsoft hasn't really done anything to make this easier, or address this
(Shameless self promotion) - I however, have fixed that: https://github.com/RonSijm/RonSijm.Blazyload Your "landing page" should basically ideally be pretty much the only thing that's initially loaded, so that the initial load is fast. Once that is done, you can quietly lazy load the rest of the dlls you need
Definitely would have kept Rust, at least. Rust is one of the fastest growing languages out there (maybe fastest or second fastest), and it's incredible for the majority of application domains, not just performance related.
The issue of attracting contributors is a funny one right now. We happen to be living in a time of plentiful new and interesting languages. Rust, D, Vale, and Elixir, for example, have all caught my attention by bringing things to the table that older, better-established languages don't do as well. In order to benefit from them, and potentially help shape them into tools that benefit us even more, we have to use them. This is how we build a better future for ourselves, and for our field. IMHO, this is more important than maximizing the number potential contributors on every project. As a fairly new project, Lemmy is in a good position to make this trade-off.
(Also, having fewer contributors generally makes a project easier to manage, so the downside is not necessarily all down.)
I'm mainly glad they chose something with decent memory safety. This is important to me in systems that face the network or process external data.
Rust probably wouldn't have been my first choice, because my early impression is that it seems ugly and awkward, so I don't think I would enjoy using it. That doesn't make it an objectively bad tool for the job.
On the front end, I just wish it was less reliant on JavaScript, such that basic functionality worked when scripts are disabled in the browser. Outside of that, TypeScript seems like a fine choice.
I'd have gone with PHP. Modern PHP is really good, all the tools are mature, the few drawbacks it has are well understood and easily worked around, and it's an easy language to learn with a huge community of people who already know it.
For a project this large and widely used - I wouldn't use a framework and that's another area where PHP really shines. The standard library is very comprehensive, and all of the major frameworks are split into smaller modules which can be used standalone and interact with modules from other frameworks.
So you can bootstrap the project and get to a working proof of concept quickly by using well tested and well written third party code, then later when you need lower memory footprints/etc it's trivial to fork those modules and and reduce them to just the core features you are actually using in your project.
That's certainly an unpopular opinion. I like your bravery!
So, kbin then?
PHP is dead. Why not use TS.
What’s the current tech stack and why do you think they chose it?
I can continue a convo if you give me a bit more
Rust backend and Inferno frontend AFAIK
I don't have as much experience working with the stack, but from what I've read it seems like Rust is a pretty solid choice for the backend. It also seems like a lot of the upgrades people want are for the front end, so that's what would benefit the most from being simpler.
Typescript makes sense, and a handful of frameworks have typescript support. Would anyone know more about the benefits of using Inferno over something like Vue/Nuxt or plain React?
I'd honestly choose a similar stack for the back-end. I have limited experience with Rust, but my impressions so far is that it's a language that allows you to make changes with confidence that they'll work. I feel like starting something in Rust is somewhat difficult, but contributing is relatively simple.
For the front-end, I don't think the choice is as important, since I think that by virtue of being federated and being able to have multiple front-ends, it would almost be better for the front-end to be managed by other parts of the community. And I say that as a primarily front-end/developer-experience dev.
I would probably default to React since I'm familiar with it and it's very popular, but would probably be tempted to experiment with something better.
I don't think the software stack does matter that much. I've downloaded the Lemmy source and ran it from source, and my main caveat with it is the structure. I would have liked to see a much more modular design, with an add-on kind of system. Now, when an instance owner wants to add a feature, their only two options are either to: Try to get their specific feature into Lemmy main, or create a specific Lemmy fork with their own feature(s).
The entry level to just create your own Lemmy add-on would be much smaller than to try to get something new into the main repo. Plus it'd give you a sense of ownership when you'd see something like "50 instances have added your add-on" - over "I made a cool feature, it got PRed into main, and pretty much no one knows who build it"
For example, a great system is something like Home Assistant - Where this is a structure of "Addon-Repo" and "Addons" - with a Main Repo and you can add your own custom repo, or just install from .zip if you want that for some reason. Kind of similar as something like Nuget
If people could "contribute" with their own standalone things instead of having to PR stuff into a repo, you'd get much more contributions and user created add-ons
Do we want to end up with a WordPress situation however? Gate-keeping does not only have drawbacks
Interesting question. I'd choose .NET backend and Vue + Nuxt frontend and REST API between them.
The choice of stack would matter a lot less if the documentation were better. Last I checked, having to figure out the details of message semantics from scratch made writing better frontends or alternative server implementations in other languages ten times more difficult than it needs to be.
I quite like Rust so I'm biased. But I understand it's not always the best choice.
I'd probably go with Symfony for the backend and Angular for the frontend.
I wish it was easier to install.
It has the perfect stack
I'd go with Elixir + Phoenix for the backend. Though that would also mean not many people would manage to contribute, as it's not a very widely known or used language. It is very resource efficient, tho, so it should scale upwards very easily
Flask backend
Blazor frontend
I read flash and was horrified a second.
Can actionscript be used for a database?
If you've got the right attitude.
Why did you chose Blazor and then Flask instead of .NET for backend? I'm just curious.
Ngl I was going for worst possible stack. Not that either one is bad on its own, but like you said, I don't think anyone in their right mind would choose this combo.
Duh... Definitely LAMP
Good old PHP and HTML+vanilla JS
I'd probably go with a different database engine. Postgres is not scaling really well and poorly suitable to throughput intensive applications like tracking activities, upvotes etc.
Which one would you pick? Cockroach?
Probably a big data oriented NoSQL database like ScyllaDB. It provides eventual consistency with fast and efficient reads and writes, but high availability, which seems like a perfect fit for the case. CockroachDB favors consistency in place of availability which doesn't seem to be really a priority for Lemmy instances. You would rather have some upvotes, posts and comments missing sometimes than not being able to load the rest of the data.