How does Lemmy feel about "open source" machine learning, akin to the Fediverse vs Social Media?

brucethemoose@lemmy.world to Ask Lemmy@lemmy.world – 3 points –

Obviously there's not a lot of love for OpenAI and other corporate API generative AI here, but how does the community feel about self hosted models? Especially stuff like the Linux Foundation's Open Model Initiative?

I feel like a lot of people just don't know there are Apache/CC-BY-NC licensed "AI" they can run on sane desktops, right now, that are incredible. I'm thinking of the most recent Command-R, specifically. I can run it on one GPU, and it blows expensive API models away, and it's mine to use.

And there are efforts to kill the power cost of inference and training with stuff like matrix-multiplication free models, open source and legally licensed datasets, cheap training... and OpenAI and such want to shut down all of this because it breaks their monopoly, where they can just outspend everyone scaling , stealiing data and destroying the planet. And it's actually a threat to them.

Again, I feel like corporate social media vs fediverse is a good anology, where one is kinda destroying the planet and the other, while still niche, problematic and a WIP, kills a lot of the downsides.

12

I think it’s amazing. I’m running Ollama with a bunch of open-source llms. You’re right. It’s so good. The problem is keeping up to date on what the newest development is.

The pace of progress is so fast and it’s really difficult to know what the cool kids are experimenting with this moment.

Oh, and if your hardware is AMD or Nvidia, you should really give exllama a shot.

If it's Apple, you should investigate kobold.cpp and more "nitty gritty" llama.cpp backends.

I have largely negative feelings towards ollama for a lot of reasons, but one of them is that it hides a lot of the knobs to get the absolute best out of LLMs, and understand how they work.

Honestly a big problem is that the community for filtering the news has "collapsed."

The only reasonable congregation was basically /r/localllama, and due to a number of factors (including, apparently, a Reddit bug that was driving away traffic according to a mod), and its shrunken a ton.

Twitter, linkedin, youtube and such are awful and full of straight up lies. Huggingface is just impossible to navigate and filter. There are a few niche aggregators, but they come and go.

Hence I was hoping lemmy would grow its existing ML communities, but most of lemmy seems broadly anti AI, even anti open source AI, hence this post to get a feel if that's true.

I read localllama through redlib but I don’t contribute. I am not technical enough to contribute and I don’t understand the math.

I have been looking at YouTube for some videos to try to explain it, but I haven’t found anything that is in the sweet spot between “video for non-technical people” and “video for people with PhD and quantum physics”

It's a giant mess. Even the technical vidoes tend to be theoretical, and are either obsolete or do nothing to help you actually run them.

I would know nothing if I hadn't been following the community since the Pygmalion/ESRGAN days

I've spent the past 2 years looking for the open source AI community, but haven't really found it. I've tinkered with Stable Diffusion and Ollama and I want to learn more, but haven't found the right places online yet.

Hate to suggested it but have you checked reddit localllama?

Very much pro Open Source AI. Especially as a concept digital public good. With https://petals.dev/ being the most promising option that regard (imagine something like RAG for the arch wiki with very large models supported by the community!).

It feel very enthusiasts right now. Where I feel like I'm just on the cusp of having usable set up.

I personally really want a full Dev that just takes gitlab issues and runs codes against tests until it passes, and then cycles between attempting to explain what it doing and refactoring until that explanation is reasonably simple, then submit PR.

At the moment I am trying to use it as a copilot (ollama lama3, continue, and devonAI vscode plugins) all on my MacBook (my Linux machine were too small gpu wise, at least first time I attempted). That said it ok for questions no real luck on a decent experience for actually making anything.

The next step to me for it to move from enthusiast to hobbiest would be:

  1. Models that just work on my machine. I had to do a lot of trial and error just get performant models.
  2. Models just my use case. I don't know what model support tooling, or multimodal inputs. What models are actually optimized for programing, for actions (ala openinterpretor), for reviewing documents, etc.
  3. For federated (like pedals.dev) I feel like I need some sane data guardrails. I don't want my medical documents anywhere near "bittorrent style" anything, but would absolutely love to leverage it for better outcome on opensource projects without secrets file. This also feeds into point 2 to me.
  4. More sane RAG. Maybe even IPFS links to caches or DBs for popular data sources (like code docs for example).

I feel like there has to be a better way for this. Maybe its just selinux rules for data tags for locking down my local system and some routing config file at the root of my projects. Idk tbh

Honestly I am not sold on petals, it leaves so many technical innovations behind and its just not really taking off like it needs to.

IMO a much cooler project is the AI Horde: A swarm of hosts, but no splitting. Already with a boatload of actual users.

And (no offense) but there are much better models to use than ollama llama 8b, and which ones completely depends on how much RAM your Mac has. They get better and better the more you have, all the way out to 192GB. (Where you can squeeze in the very amazing Deepseek Code V2)

The splitting is 80% of the cool factor for me. Rather than bog down the one node that can handle those cooler models, and have more contribution opportunities.

I wonder honestly if a petals network could be a target host on horde lol

The problem is that splitting models up over a network, even over LAN, is not super efficient. The entire weights need to be run through for every half word.

And the other problem is that petals just can't keep up with the crazy dev pace of the LLM community. Honestly they should dump it and fork or contribute to llama.cpp or exllama, as TBH no one wants to split up LLAMA 2 (or even llama 3) 70B, and be a generation or two behind for a base instruct model instead of a finetune.

Even the horde has very few hosts relative to users, even though hosting a small model on a 6GB GPU would get you lots of karma.

The diffusion community is very different, as the output is one image and even the largest open models are much smaller. Lora usage is also standardized there, while it is not on LLM land.

I guess to me be able to serve the 408b model even though I'm on a laptop is just awesome to me.

Also I saw Lora was an option for Petals but I haven't messed with it at all.