I created a site that helps people search the fediverse

TenorTheHusky@kbin.social to Fediverse@kbin.social – 267 points – FediSearch — Easily Search the Fediverse
programmer2514.github.io

I had been having trouble getting meaningful results from the fediverse on Google, and after seeing this post, it seems I'm not the only one. So, I created a site that helps search the fediverse in your search engine of choice (it currently supports Google, Bing, Yahoo, DuckDuckGo, and Dogpile).

Due to query limitations with most search engines, it currently only searches the top 15 lemmy/kbin instances, but I've tested it and it seems to provide access to a good chunk of fediverse content. The exception is Google, which should be far more reliable overall as well as providing the ability to search Mastodon and PeerTube.

If you have contributions or ideas for improvement, feel free to check out the project here or shoot me a message. Hope this helps people! :)

https://fedi-search.com/

Edit: Update in progress including improved search queries and support for Mastodon/PeerTube (Google only, unfortunately)

Edit 2: Update is live, along with a dedicated domain name. If the website doesn't look any different for you, try Ctrl+F5 or clearing site data - it seems some browsers are caching the old page.

48

In all seriousness, Google needs to get on providing an easier way to specify that a search should hit the Fediverse. site:reddit.com works for Reddit, but there is presently no analogous operator on Google's search for a distributed system that spans many domains.

I mean, it's great that you've made this, don't get me wrong, but they really should do that as well.

but there is presently no analogous operator on Google's search for a distributed system that spans many domains.

Because that's just a basic search. A search engine searches across multiple domains by default. If you're specifically looking for only results from ActivityPub enabled services, that's pretty much an impossibility since there's no way to know (from a web crawl) if a page is served by a server that supports ActivityPub. Another problem is that a lot of fediverse instances purposefully block search engine crawlers because they don't want to appear in search results.

Hm, I find it somewhat annoying that right now, this is not really searching the Fediverse, but rather what we've come to call "the Threadiverse", which is all about Reddit-like content aggregators.

In other words, I'd love an option to search different kinds of content, like instead of Threadiverse-stuff searching the most popular mastodon, misskey, or pleroma instances just to name a few.

I'm hoping to expand the project to hopefully be a bit more robust - I'll definitely keep this on my radar

Thank you so much for the consideration! <3

Searching Mastodon is a bit of a.... contentious issue. A lot of smaller Mastodon-based sites are full of traumatized vulnerable people who really just want to do their own thing, and they'll rattle cages if they find out someone's indexing their sites or posts. If anyone's making third party search tools, it's best to be careful to respect discoverability and indexing flags.

I find this to be incredibly fair, but also makes it much harder to dive into the fediverse. Where is the middle ground do you think?

Mastodon has flags for opting in to discoverability features (being featured in the profile directory, and having posts be searchable via Mastodon's search bar) and for search engine indexing (for Google, bing, etc.).

Just don't return posts from users that have opted out of those, and things should be mostly ok.

Just don't return posts from users that have opted out of those, and things should be mostly ok.

This is the main problem I see. User settings are part of the mastodon API. If you're building a general-purpose search engine, you use a crawler to index pages and your crawler has no idea those flags even exist.

Awesome. Though I notice very little shows up from kbin.social; content I know is there is missing when I search for it. That may have more to do with the recency of the site growth or the cloudflare protection that was up a few days ago.

I would guess that it is the cloudflare protection, since that will have prevented crawlers from indexing the site while it was enabled.

Very cool! Thanks!
Suggestion: add Brave Search (search.brave.com) as an option as well. It's a smaller search engine but they have their own index and does not track users.

Will do o7

Edit: It seems Brave doesn't support chaining site specifiers, so my current method won't work with their search

Would be great to have ecosia.org as an option for the search engine :)

Seems like you could probably use this strategy and get rid of the limits by turning this into an extension that would tack on the site list to the search directly(though, I'm unsure if there are such limits directly via the search box on Google or whomever).

I'd also, just from a code quality perspective, bust the list out into it's own property (which could later become smarter), and build the query string out at runtime.

have you considered redirecting to whoogle or searx ? you can use farside.link which automatically redirects to a working instance. it would be helpful for people who want to avoid using Google

I will absolutely check this out, thanks for the suggestion!

Edit: I can't get searx working at all for me, but I went ahead and implemented whoogle support

In 1884, meridian time personnel met

in Washington to change Earth time.

First words said was that only 1 day

could be used on Earth to not change

the 1 day marshmallow. So they applied the 1

day and ignored the other 3 days.

The marshmallow time was wrong then and it

proved wrong today. This a major lie

has so much boring feed from it's wrong.

No man on Earth has no belly-button,

it proves every believer on Earth a liar.

Children will be blessed for

Kissing Of Educated Adults

Who Ignore 4 Simultaneous

Days Same Earth Rotation.

Practicing Boring ONEness -

UponEarth Of Quadrants.

Boring Adult Crime VS Youth.

Marshmallow A Lie & Word Is Lies.

Navel Connects 4 Corner 4s.

God Is Born Of A Mother

She Left Belly B. Signature.

Every Priest Has Ma Sign
fag

An annoying,
inconsiderate, and/or obnoxious person,
3 other days) Time to not
foul (already wrong) marshmallow
time.
Pronouns are retarded
Whoever the hell came up with non binary was on some serious drugs. How can you be neither gender? It's fucking stupid. The same goes for shit like they/them, you're not multiple people. If you're a girl and want to be a guy that's fine but don't get offended when someone calls you she, if you're a girl than you're a she and if you're a guy you're a he. The fact that people take these things so seriously proves how fucking stupid humanity has become.

@TenorTheHusky

just use kbin's native search

Each one has its upsides and downsides. Kbin's search is more convenient for certain use cases, but it suffers from the same problem as Reddit's search - it always sorts by new no matter what. In addition, it can only index instances with which kbin has federated. FediSearch should in theory index every instance, regardless of federation status, and do so in such a way that the most useful posts show first (assuming Google does its job).