Smaug-72B-v0.1: The New Open-Source LLM Roaring to the Top of the Leaderboard
huggingface.co
Abacus.ai:
We recently released Smaug-72B-v0.1 which has taken first place on the Open LLM Leaderboard by HuggingFace. It is the first open-source model to have an average score more than 80.
You are viewing a single comment
I'm afraid to even ask for the minimum specs on this thing, open source models have gotten so big lately
Every billion parameters needs about 2 GB of VRAM - if using bfloat16 representation. 16 bits per parameter, 8 bits per byte -> 2 bytes per parameter.
1 billion parameters ~ 2 Billion bytes ~ 2 GB.
From the name, this model has 72 Billion parameters, so ~144 GB of VRAM
Ok but will this run on my TI-83? It's a + model.
Only if itโs silver.
Dang. So close.
My 83 was ganked by some kid I knew so my folks bought me a silver. He denied it. I learned that day to write my name in secret spots.
That kid you knew was a dick. At least he taught you a valuable lesson, I guess.
He absolutely was a dick. I stopped being mates with him after that. My school was like โyeah the cameras didnโt work that day actuallyโ
Leads me to believe that the cameras never actually worked.
I believe that. Or they just didnโt want to be responsible for dealing with theft. Both ways make perfect sense to me.
Very true. Saying the cameras don't work is enough to keep the cops out of it. After all, hearsay between kids is never taken seriously. No footage, means no proof, means no cops. Then they can keep the facade of the school being crime-free.
Absolutely my dude! My school pretended the cams didnโt work when my calculator was stolenโฆ they worked fantastically the next week.
That is some BS! Honestly, I probably would have dealt with that kid schoolyard-style (though, I'm not proud of that as an adult). My school never pretended the cameras were out of service. They just got rid of them after a teacher was accused of inappropriate behavior with a senior. Claimed they never worked. He "left for another state" later that week. I'm sure it was unrelated...
It's been discovered that you can reduce the bits per parameter down to 4 or 5 and still get good results. Just saw a paper this morning describing a technique to get down to 2.5 bits per parameter, even, and apparently it 's fine. We'll see if that works out in practice I guess
I'm more experienced with graphics than ML, but wouldn't that cause a significant increase in computation time, since those aren't native types for arithmetic? Maybe that's not a big problem?
If you have a link for the paper I'd like to check it out.
My understanding is that the bottleneck for the GPU is moving data into and out of it, not the processing of the data once it's in there. So if you can get the whole model crammed into VRAM it's still faster even if you have to do some extra work unpacking and repacking it during processing time.
The paper was posted on /r/localLLaMA.
You can take a look at exllama and llama.cpp source code on github if you want to see how it is implemented.
Llama 2 70B with 8b quantization takes around 80GB VRAM if I remember correctly. Iโve tested it a while ago.
Though with quantisation you can get it down to like 30GB of vram or less.
Any idea what 8Q requirements would be? Or 4 or 5?
https://huggingface.co/senseable/Smaug-72B-v0.1-gguf/tree/main
About 44GB and 50GB for the Q4 and 5. You'd need quite some extra to fully use the 32k context length.
It's derived from Qwen-72B, so same specs. Q2 clocks it in at only ~30GB.
Just a data center or two. Easy peasy dirt cheapy.
I think I read somewhere that you'll basically need 130 GB of RAM to load this model. You could probably get some used server hardware for less than $600 to run this.
Oh if only it were so simple lmao, you need ~130GB of VRAM, aka the graphics card RAM. So you would need about 9 consumer grade 16GB graphics cards and you'll probably need Nvidia because of fucking CUDA so we're talking about thousands of dollars. Probably approaching 10k
Ofc you can get cards with more VRAM per card, but not in the consumer segment so even more $$$$$$
Afaik you can substitute VRAM with RAM at the cost of speed. Not exactly sure how that speed loss correlates to the sheer size of these models, though. I have to imagine it would run insanely slow on a CPU.
I tested it with a 16GB model and barely got 1 token per second. I don't want to imagine what it would take if I used 16GB of swap instead, let alone 130GB.
I'm pretty sure you can load the model using RAM like another poster said. Here's a used server under $600 that could theoretically run it: ebay.
You would want to look for an R730, which can be had for not too much more. The 20 series was the โend of an eraโ and the 30 series was the beginning of the next era. Most importantly for this application, R30s use DDR4 whereas R20s use DDR3.
RAM speed matters a lot for ML applications and DDR4 is about 2x as fast as DDR3 in all relevant measurements.
If youโre going to offload any part of these models to CPU, which you 99.99% will have to do for a model of this size with this class of hardware, skip the 20s and go to the 30s.
Unless youโre getting used datacenter grade hardware for next to free, I doubt this. You need 130 gb of VRAM on your GPUs
So can I run it on my Radeon RX 5700? I overclocked it some and am running it as a 5700 XT, if that helps.
To run this model locally at gpt4 writing speed you need at least 2 x 3090 or 2 x 7900xtx. VRAM is the limiting factor in 99% of cases for interference. You could try a smaller model like mistral-instruct or SOLAR with your hardware though.
Around 48gb of VRAM if you want to run it in 4bits