Viewing lemmy posts by all tends to be dominated by a few communities

dryguy@lemmy.world to Fediverse@lemmy.world – 565 points –

I've noticed that there are a few communities that tend to dominate when viewing all. Some days it gets to where looking at all isn't very different than just looking at Memes@lemmy.ml or 196@lemmy.blahaj.zone.

Before someone says "you can just block communities you don't want to see," it's not that I never want to see them, it's that I want to be able to have a view that shows me what is new and popular in a wide variety of communities. I appreciate seeing a few good memes in my feed. The problem is when that's all I see. Changing the sort from active to hot or top x days doesn't have much effect on which communities dominate, so that isn't the solution either.

"You can just subscribe to communities you like". True, but that has the effect of narrowing what I see. I'd like a view that showed me new things I never thought to subscribe to.

Lemmy devs - if you are reading this - it would be nice to have a feed that limited the number of posts showing up from any particular community. It could be a simple cutoff of 2 or 3 posts, or maybe some sort of weighting function to cause additional posts from the same community to appear lower in the sort order for that feed.

I'd love to hear what devs and other users think about this.

Edit: To everyone saying "just sort be new" - yes, that has its uses, but it only solves part of the problem. I'd like a feed that shows me what is new and popular, but from more than just one or two communities.

104

I read something the other day that they’re working on a new sorting algorithm that would limit it to the top few posts from each community within a given time frame. Specifically to address this issue.

No idea on timeframe or further details, or if I even summarized it correctly lol.

3 more...

This has already been mentioned a few times. Just find the relevant issue on GitHub and give it an upvote.

Idk it seems like a problem that will sort itself out as Lemmy grows, and artificially limiting how many posts from a community can reach the front page seems like a suboptimal solution that’s going to have unintended consequences down the line.

I'd agree if the coms in question were niche but so far in my experience they never are. When it comes to communities dominating All, it's always bottom-of-the-barrel memes and porn. The posts from non-garbage communities that show up are usually by themselves

Everyone’s definition of bottom of the barrel will be different, and nobody’s personal content preferences should be forced on the community as a whole. If you really dislike those communities that much you can block them.

I do, my blocklist is quite large already

The point I'm making is that the communities that would be most affected (negatively) by it are the giant low effort meme/shitpost ones that don't "need" the exposure to thrive because they're general interest communities

Right now those communities are more important than ever. They are what's going to bring more people here and grow the fediverse. I don't want to start hiding popular content at a time when Lemmy most needs to be popular.

Is the goal of Lemmy to follow the reddit playbook where quantity is more important than quality? I much prefer thoughtful, specific content to "mass appeal" content. There's no shortage of places to find the latter, why does it need to be the focus here?

Because this is donation-funded. Having a big audience is the only thing that can ensure financial stability long term.?

And I don’t think content that is funny rather than informative is inherently bad or less important. There’s nothing wrong with this place being fun and not just some stuffy content classroom.

The real issue I see is that it's all reposted reddit content so Lemmy looks like a crappy clone instead of being its own thing. For those who enjoy those posts, why would they switch to a site that has the same posts but with empty comments sections?

I'm not saying it's some huge problem or anything. I just don't think limiting the amount of frontpage posts per community will negatively affect the site

I joined because I still want the same content without the asshole owner. I’m here for the different power structure, not because I hated Reddit’s content.

Doesn't Reddit totally weight by size of community tho? Not saying we should just ape the old site but I suspect it's actually necessary for smaller communities to grow naturally.

I think there are better ways to highlight smaller communities and grow them more organically, like a community dedicated to new and small communities (sorry if I fucked up that link, I'm new here) could highlight a new community each day worthy of our attention. Reddit used to have a subreddit of the day.

Right now the number one thing federated social media needs is just more users. I worry they'll feel discouraged if they stop seeing the content that gets the most upvotes right now.

I think I'm going to have to disagree with you on that. New communities is fine but as someone who's been active over there for a while it definitely seems that the issue is that it's impossible for small communities to get any traction on the main page. I think the most important thing for long term growth is making sure that new users don't come over here and find that all the communities they are interested in are dead.

If a community has grown a lot, unfortunately it takes up all the posts on the homepage. If I were a developer, I would sort by weighted success. For example, if the "x" community has 1000 subscribers and the post gets 100 likes, it has ten percent success. If the other 'y' community has 100 people and gets 11 likes, it has 11 percent success. This overrides the post in community x because the post in community y is more successful. This is the logic of 'weighted success'. With this logic, a better ranking formula can be created.

The thing is, a 10% upvoted post on a 10,000 people community is more popular than a 90% upvoted post on a 1000 people community - those 10,000 people in the former community are 10,000 people interested enough in that kind of thing to subscribe, whilst only 1000 people are interested enough in the other kind of thing.

So it does make sense to put the former higher up in the global page when sorted by popularity because globally that post was more popular.

However I do think there should be someway to as a user push down posts from certain communities without outright blocking the whole thing: maybe som throttling-down based on the rate of posts per time (i.e. the upvote threshold for posts from a community to come out in All depends on the number of otherwise qualifying posts in the last X days/hours, thus explicitly targetting the "flooding with posts" itself) rather than the straight count of upvotes or the proportion of upvotes that you suggest.

That said in the meanwhile I'm really tempted to block the more generic meme communities.

General popularity is not a good metric IMO. If I like a community, then it shouldn't matter if a million people like or it's only me and my cousin. If the community likes the content, I want to see it.

It's trust between the members of a community.

However, weighted sorting is not a solution too, upvotes counts are not linear. Maybe, quantile sort?

What's liked by the general population is a good metric for providing general stuff to the general population and that's what we're talking about in All.

That average can however deviate a lot from the sweet spot for some people, quite possibly a large minority (even the majority depending on how concentrated or not people's tastes are around it).

Something that looks at your previous choices (or even generally stated choices in the form of communities you subscribe to or block) similarly to what some search engines and some social media sites will do, can shift that toward more your own specific tastes, but that's computationally more expensive and requires more users and more user data to get better results (basically it's finding certain kinds of users and local minima which are more satisfactory to them).

I suspect something like an AI solution (not LLM, just a much simpler neural network) running on your own device that tries to predict what you're going to click on and learns with what you do (or not) is the only way for a personalized "no fluff on my feed" solution, but that's for apps running on top of Lemmy, not the Lemmy engine.

2 more...

Another, related issue to this is that I find the three frontpage categories (All/Local/Subscribed) to be too blunt of a sorting method.

I would really love if we had the option to create subcategories to our Subscribed feed, so we could group related communities and end up with different frontpages for News/Sports/Memes etc.

Being able to filter by language would also be super helpful.

Im sure the Swedish gaming community is great, some lovely Svens and Ingas but I dont speak swedish. Being able to autofilter subs by default language would be awesome.

Isn't that what reddit's "multi Reddit" was all about?

I always hated multi-reddit because it never worked the way I wanted it to.

I wanted it to give me an r/all setup for a specific sets of subreddits. What it did was just give posts from every subreddit in the list, including ones that hadn't had posts in months. It was just trash to try to use.

Yes. It's been discussed on the Lemmy GitHub so it's in the works, but that's all I know.

I generally use “New Comments” as my sorting and it’s a little bit better, but still the same spammy communities end up on top

Memes, memes as far at the eye can see

So many memes

I don’t hate it yet, since they’re the memes of my childhood/teen years, so tbh I find them funnier than the post 2016/18 bullshit

I'd love some kind of per community bias adjustment even for subscribed communities. Like, I don't really want to remove them cause memes are great, but because !memes@lemmy.world and 196 post so often my subscribed feed is pretty dominated by them no matter how I sort it.

For "All" some kind of adjustment based on subscribers makes sense, but I don't even know if that's possible given the way Lemmy works. Maybe a "show me less" button that moves the same bias adjustment just for communities you're not subscribed to?

Note that this is one of the advantages of having an account on a smaller and/or more focused instance or having multiple accounts.

All "Alls" are not the same. Actually, the "All" displayed on a given instance is everything local to that instance and everything from other instances to which someone on that instance has subscribed. So if nobody from that instance has subscribed to a particular community on another instance, then for all intents and purposes, it just doesn't exist. Even on "All".

Granted that it's somewhat unlikely for an instance to not have someone somewhere along the way subscribe to some notably popular community, it is possible, and the smaller and more focused the instance is, the more likely it is.

Thanks for teaching me. I can see that causing challenges down the road.

For example, I'm always on the lookout for all things quilting. If someone names their quilting community "Fabric Hordes" (not impossible, just look at phenomenon like r/animetitties) it wouldn't come up in my explicit searches, and is very unlikely to be sought out or found by others in my instance.

Right, but there are lots of ways around that.

There's already been a fair amount of demand for some method to group communities by interest, so it's essentially guaranteed that somebody is going to provide some way to do that, and likely multiple somebodies are going to figure out multiple ways.

So, would it be wise and helpful for a mod or bot per instance to subscribe to as many communities as possible to help the instance's feed?

Hell no.

How does that "help" their feed? What possible benefit could there be in using a bot to subscribe willy-nilly to every community out there, no matter how shitty it is?

I mean - if some instance owner wants to do that, that's their choice, and I guess there are people out there who would like the resulting instance filled to the brim with every bit of garbage that exists anywhere in the fediverse, so it's safe to assume that somebody will do it sooner or later. Personally, I think the idea is repulsive though.

Maybe I wasn't clear enough in that other post - I think that the fact that each instance has a different "All" depending on what the members there have subscribed to is a good thing. It means that different instances have different feels, and over time, as they get more established, that's going to be even more the case.

So for instance, a notably tech-oriented instance is going to end up displaying pretty much every tech-oriented community on the fediverse on its All because somebody on the instance will have subscribed to it, pretty much no matter what it is, AND at the same time, all of the stuff nobody's interested in just won't be there at all, because nobody bothered to subscribe to it in the first place.

Granted that that's not going to appeal to people who want to be flooded with every bit of garbage on the entire fediverse when they click All, but they can just go away and sign up with some other instance that gives them what they want. Which I'm sure is exactly what the people who sought out a tech-oriented instance in the first place would prefer anyway.

How does that "help" their feed? What possible benefit could there be in using a bot to subscribe willy-nilly to every community out there, no matter how shitty it is?

I don't know... You made it sound like the only way for me to have more communities show up in my All feed on lemmy.ca would be if a volunteer on lemmy.ca, be it a bot or mod, subscribed to all the communities they could find. Hense seeing top posts from All communities. And I only meant for a bot or mod to just subscribe. Not to repost everything.

If that isn't how it works, sorry. It is why I asked if that would work or not.

You don't need a mod or a bot to do it for you. You can go find communities on other instances and subscribe to them, and that all by itself gets them added to the All feed on your home instance.

Or you could just register an account on a different instance that slready has more stuff on it, like lemmy.world.

Ah. I had noticed that lemmy.world's all seemed different than lemm.ee's, which in turn was also different than kbin. That's good to know

This was an issue Reddit used to have circa 2015. Front page was all League of Legends posts, and then it was all The_Donald posts. Then Reddit screwed up their algorithm and it was literally 100% The_Donald posts.

I don't need another t_d vietnam flashbacks man, don't jinx it.

Fun fact. r/leagueoflegends was one of the most filtered subreddits from r/all at one point.

The important takeaway here is that it took a long time before it was actually good. They had to try a bunch of different sorting algorithms before they found one that really worked and let you see your small subs just as much as your big ones.

It might take a while here too unfortunately.

I don't know about anyone else's experience, but I've noticed that any time I click into a post view and then back out, I'm taken to the first page of posts, no matter if I was 2 or 3 pages on. If the redirect respected where it came from instead of going to home, that could reduce the impact of post-order sorting. Also if the list pager had more options than 'prev' and 'next' (maybe a few numbered pages between or beyond) I could get beyond that 3rd page without getting there feeling like an illustration of the schlemiel the painter's problem

I found a workaround to that problem, just do a page refresh after going back from a post to the list of posts.

Fyi the devs aren't reading this (and probably won't be before long, since they are busy just coding a lot of features). Best place to ask for this is on the issue tracker (first check if it hasn't been asked before), even better implement it yourself if you can!

I'm right there with you - I'm making an app called flemmy, and I have 12 more tasks on my list before I'm putting it on the play store - should be this weekend at the latest. Iphone build shouldn't be that far behind. I can also make a desktop build if anyone wants it, but right now have no intentions to host a site myself - I strongly feel the data the app collects shouldn't leave your device

Version 1 is about creating something close to the Reddit apps I used to use, and it's there - just needs a little more polish (and to let you post... I'm more of a commenter, so I forgot that was a thing for an embarrassingly long time)

I can support all sorts of filters, from keywords to hiding specific posts to "snoozing" communities. I can also save your place when you change sort methods or accounts - it's what I always wanted for Reddit.

Also, I have support for redirecting links - Twitter to nitter, YouTube to pipe, etc.

Version 2 is going to focus on your feed. Already I connect to multiple servers (it's a real headache, but the foundation is there), so next is stitching feeds together and custom feed algorithms. What you mention is at the top of my list - a way to tweak the feed based on all sorts of filters.

Ideally, I want it to adapt to you - using upvotes and comments to tweak your feed. All on your phone - it's amazing what you can do on a phone when you're not interested in data collection

I just made !flemmy@Lemmy.world, I'll post some screenshots when I need a longer break

Awesome, that you develop tools for lemmy! Thank you.

A question: Did you consider to publish your app on the F-Droid-Store? I only use apps from there and the only usable app for lemmy is jerboa. An alternative to that would be awesome, as it has quite a lot of problems.

Browse by New. Bam! Problem solved! Stop being slaves of the Algo. Actively search for stuff you like while also actively block those you don't like.

Better solution: don't use Lemmy as your feed, use a feed reader (RSS). There are per-channel feeds that you can sort and filter using parameters.

Doing things this way will also help create the open web we all want to see, where "forum" is not a synonym for "Reddit" or "Lemmy", where you can also follow the goings-on in other places and not miss anything.

Yeah, I really think it's important to not see Lemmy as one singular community, or a lot of important use cases will go ignored.

yes! yes! yes! I have the same problem.

My suggestion is to add a view with subscriptions ordered by the selected criterion (i.e. new/active/hot/top) and below each community there is vertical list with posts from that community sorted.

This would allow see what's up with the communities we follow and then jump in those communities if we find anything interesting.

A quick, but a little dirty solution for this, would be communities having “tags” in their metadata. This wouldn’t prevent spam, or an accumulation of four trillion tags, but you could easily add “only these tags,” or “not these tags,” to any feed. User objects have metadata that is used like this (as the “bot” flag) already. I’m just familiar enough with the code to know it wouldn’t be a slam dunk, but it’s also not a breaking change or re-write!

Tags would be great, and a much better way of controlling your feed than blocking communities or instances. Just because I usually don't want to see memes or shitposts doesn't mean I'm never in the mood for them. It could also potentially help people who don't want porn in their feed but want to keep non-porn NSFW visible.

Maybe they could even integrate with Mastodon's hashtags

@dryguy The algorithm needs to be improved. It needs to adjust for the number of boosts/favorites that large communities get. You should be seeing posts from all of the subs you are subscribed too and not just the most popular ones.

I like to use "Hot" instead of active as it seems to fetch posts from a more diverse set of communities

I've also heard that view by new comments is solid as well

Btw, It's already possible to block all bot posts

Ohhh is that what unchecking the "show bot account" does I thought it removed the badge indicating a user was a bot account. Thanks!

Top from the past 6 or 12 hours works better than Hot IMO. It feels more like the traditional reddit front page.

I think there should still be a strictly chronological feed that isn't algorithmic, but blocking the bots that import a lot of low quality content from reddit would be easier than blocking the individual subs.

I'm definitely not suggesting that any of the existing feeds need to be done away with. They all have their uses.

Can't you just sort by new?

That just gives you what's new. I'm looking for what is new and popular. I usually browse by top 6 hours.

Sorting by new works pretty well for me. There's still the couple of them showing up, but there's a lot more variety to what I see that way.

I'm with you 100%. I did find that searching by hot comes out a little better. But I think I'm finally ready to just block the meme and the rule subs.

Also there's just not as much content here as. Reddit. That will eventually change,

Most communities I'm in just seem to post about Reddit and its soooo boring, like get a life and make original content

I think this problem is relegated to being registered to larger instances. On one hand you get lots to choose from when it comes to your local feed, similar to how Mastodon works, on the other hand you have the problem you just mentioned. I know you bring up the problem of narrowing what you see, but if you're going to be on a big instance, there really is no other way than to subscribe to those communities you have a specific interest in. I'm on a smaller instance and I just take the time to go through communities that I may have an even fleeting interest in and subscribe to them, avoiding larger generic communities. This is the same strategy I employ when it comes to my news feed as well. I only subscribe to RSS feeds from sites I have an interest in.

What if you make an alt account that blocks big prominent communities when sorting through /all so you can view the top posts from smaller communities?

But then you have to change accounts, which I understand isn't difficult, though it's still something that needs to be done proactively by the user. Would be nice to just have a better content ranking algorithm that gives smaller communities a chance.

I fully agree. On reddit i would use the all frontpage to find new communities. Here it doesnt work.

As more people join these kinds of things will change and evolve. Hopefully the site infrastructure will adapt to it as well.

Someone from reddit should quit and recreate their algorithm for front page.