CodeInvasion

@CodeInvasion@sh.itjust.works
1 Post – 28 Comments
Joined 1 years ago

I am an LLM researcher at MIT, and hopefully this will help.

As others have answered, LLMs have only learned the ability to autocomplete given some input, known as the prompt. Functionally, the model is strictly predicting the probability of the next word^+^, called tokens, with some randomness injected so the output isn’t exactly the same for any given prompt.

The probability of the next word comes from what was in the model’s training data, in combination with a very complex mathematical method to compute the impact of all previous words with every other previous word and with the new predicted word, called self-attention, but you can think of this like a computed relatedness factor.

This relatedness factor is very computationally expensive and grows exponentially, so models are limited by how many previous words can be used to compute relatedness. This limitation is called the Context Window. The recent breakthroughs in LLMs come from the use of very large context windows to learn the relationships of as many words as possible.

This process of predicting the next word is repeated iteratively until a special stop token is generated, which tells the model go stop generating more words. So literally, the models builds entire responses one word at a time from left to right.

Because all future words are predicated on the previously stated words in either the prompt or subsequent generated words, it becomes impossible to apply even the most basic logical concepts, unless all the components required are present in the prompt or have somehow serendipitously been stated by the model in its generated response.

This is also why LLMs tend to work better when you ask them to work out all the steps of a problem instead of jumping to a conclusion, and why the best models tend to rely on extremely verbose answers to give you the simple piece of information you were looking for.

From this fundamental understanding, hopefully you can now reason the LLM limitations in factual understanding as well. For instance, if a given fact was never mentioned in the training data, or an answer simply doesn’t exist, the model will make it up, inferring the next most likely word to create a plausible sounding statement. Essentially, the model has been faking language understanding so much, that even when the model has no factual basis for an answer, it can easily trick a unwitting human into believing the answer to be correct.

—-

^+^more specifically these words are tokens which usually contain some smaller part of a word. For instance, understand and able would be represented as two tokens that when put together would become the word understandable.

3 more...

Someone did the math and realized we would need a 130% tariff on all goods to replace current income tax revenue.

People’s number one concern is inflation. If that tariff is created we will see 100% inflation over night!

1 more...

The only upside I can think of is they'd actually start caring about the planet instead of thinking they'll be dead in 100 years anyway.

You do realize that every posted on the Fediverse is open and publicly available? It’s not locked behind some API or controlled by any one company or entity.

Fediverse is the Wikipedia of encyclopedias and any researcher or engineer, including myself, can and will use Lemmy data to create AI datasets with absolutely no restrictions.

AFAIK, there’s nothing stopping any company from scraping Lemmy either. The whole point pf reddit limiting API usage was so they could make money like this.

Outside of morals, there is nothing to stop anybody from training on data from Lemmy just like there’s nothing stopping me from using Wikipedia. Most conferences nowadays require a paragraph on ethics in the submission, but I and many of my colleagues would have no qualms saying we scraped our data from open source internet forums and blogs.

I am a satellite software engineer turned program manager. This is not unexpected in this current environment, however the conditions that created the environment are abnormal.

This solar cycle is much stronger than past cycles. I'm on mobile, so I can't get a good screenshot, but you can go here to see this cycle and the last cycle, as well as an overlay of a normal cycle https://www.swpc.noaa.gov/products/solar-cycle-progression

As solar flux increases, the atmosphere expands considerably, causing more drag than predicted. During periods of solar minimum, satellites can remain in a very low orbit with minimal station keeping. However, at normal levels of solar maximum, 5 year orbits can easily degrade to 1 year orbits. Forecasters says we are still a year away from solar maximum, and flux is already higher than last cycle's all time high (which was also an anomalously strong cycle). So it will get worse before it gets better.

TLDR: Satellites are falling out of the sky because the sun is angy

This is the only response required. I'm quickly becoming exhausted of reading everyone's epiphany on "enshittification" as if it's some natural eventuality. Yes the money must eventually come, but not always at the expense of platform quality. If anything the results we see from "enshittification" are due to the fact that most businesses fail eventually due to poor leadership.

Just to echo what you have already said, money today is simply more expensive than it used to be. We even see the impacts of macro monetary decisions on households.

Buying a house or a car on loan is far more expensive than it would have been a year and half ago. A $500,000 house in 2021 would cost $2,000 a month at 2.75% interest and 20% down. Today same that payment is $2,800 or 40% more expensive at 7.75% interest.

Modern companies live on revolving debt, so if their suddenly gets 40% more expensive and that same amount of money is also less valuable at the same time (inflation), then they need to make up the difference somehow.

Corporations are trying to find the balance between squeezing more revenue to pay their ever increasing debt bills while also not destroying the environment that attracted the users (their products) in the first place. Twitter and Reddit are just going about it horrifically because of poor business leadership and decision making. Netflix's approach appears to be sustainable, and there is no doubt that YouTube will be fine in the long run.

This is not meant to be apologetic to the decisions made by Twitter and Reddit. They've made their bed through their own horrible decisions, and now they've got to sleep in it.

2 more...

You are looking at two different tax systems. The effective US tax rate (the rate you actually pay is much much less). Our household makes $300k per year, and we have a $650k net worth. Our income taxes every year? Less than 7% of that, which is absurdly low. The ultra wealthy are taxed even less than that. The US is propped up by taxes from the middle-class because the more you makes, the easier it becomes to optimize and lower your effective tax rate. We need to tax the rich more.

I’m an AI researcher at one of the world’s top universities on the topic. While you are correct that no AI has demonstrated self-agency, it doesn’t mean that it won’t imitate such actions.

These days, when people think AI, they mostly are referring to Language Models as these are what most people will interact with. A language model is trained on a corpus of documents. In the event of Large Language Models like ChatGPT, they are trained on just about any written document in existence. This includes Hollywood scripts and short stories concerning sentient AI.

If put in the right starting conditions by a user, any language model will start to behave as if it were sentient, imitating the training data from its corpus. This could have serious consequences if not protected against.

You used the phrase correctly, and your English is great!

The above commenter was rudely stating that your observation is not correct as it's has always been this way.

I don't go around using that word because of how many people find it disrespectful. But, and I ask this out of honest curiousity, why is it offensive in the first place?

I see it as synonymous with 'idiot' or 'stupid' when used colloquially. The argument that it's a medical term doesn't really hold as 'idiot' and 'moron' are also medical terms that refer to a lacking of intellectual acuity. In many ways 'retarded' has the same meaning both colloquially and medically. To be mentally retarded is to be mentally slowed or lacking that similar mental acuity that 'idiot' or 'moron' convey.

Retarded just means slow and it's a perfectly apt description. Where I think people get confused is when retardation is linked with a specific attribute like physical retardation or emotional retardation, those convey very different meanings.

I'm not saying that we should start using it again, but that I find it odd how society has latched onto a very specific word and labelled it as bad in the matter of a decade. At the end of the day, any word that can be used to insult or demean, is rude. It's not the word being used, it's what is meant by them. The term 'Cis-gender ' is also being used in a highly exclusionary way and often times is conveyed as an insult. However, it's real meaning is not insulting in the least.

The victims have beed identified as the owner of the flight school, a flight instructor, and a student.

https://www.wtnh.com/new-england-news/no-survivors-after-plane-crash-in-greenfield/

To add to this insight, there are many recent publications showing the dramatic improvements of adding another modality like vision to language models.

While this is my conjecture that is loosely supported by existing research, I personally believe that multimodality is the secret to understanding human intelligence.

much less serious

EXCEPT when it came to grammar and spelling! Honestly, I miss that kind of pedantry from the old days.

I'm convinced that we should use the same requirements to fly an airplane as driving a car.

As a pilot, there are several items I need to log on regular intervals to remain proficient so that I can continue to fly with passengersor fly under certain conditions. The biggest one being the need for a Flight Review every two years.

If we did the bare minimum and implemented a Driving Review every two years, our roads would be a lot safer, and a lot less people would die. If people cared as much about driving deaths as they did flying deaths, the world would be a much better place.

I fell like the only way is to become a dictatorship like El Salvador, and then hope the person in power is actually trying to make the country a better place.

Be right back, I'm going to read up on the latest of El Salvador.

Edit: Bukele has good intentions, but it's clear that he justifies the means with the ends, including jailing non-criminal dissenters, removing elected officials, and ignoring the supreme court to do it. Only 30% of the 70,000 jailed (nearly 1% of the country's population) have gang ties according to human rights groups.

At the same time, Bukele has a 90% popularity rating. The people clearly want this. I just wonder how and when the falsely imprisoned might be freed.

Agreed.

Nevertheless, the Federal regulators will have an uphill battle as mentioned in the article.

Neither "puffery" nor "corporate optimism" counts as fraud, according to US courts, and the DOJ would need to prove that Tesla knew its claims were untrue.

The big thing they could get Tesla on is the safety record for autosteer. But again there would need to be proof it was known.

I am a pilot and this is NOT how autopilot works.

There is some autoland capabilities in the larger commercial airliners, but autopilot can be as simple as a wing-leveler.

The waypoints must be programmed by the pilot in the GPS. Altitude is entirely controlled by the pilot, not the plane, except when on a programming instrument approach, and only when it captures the glideslope (so you need to be in the correct general area in 3d space for it to work).

An autopilot is actually a major hazard to the untrained pilot and has killed many, many untrained pilots as a result.

Whereas when I get in my Tesla, I use voice commands to say where I want to go and now-a-days, I don’t have to make interventions. Even when it was first released 6 years ago, it still did more than most aircraft autopilots.

1 more...

"Beep... Beep... Beep..." -Sputnik

Tenet is criminally under rated at 69%. Easily one of my favorite movies of all time.

1 more...

"If it wasn't hard, it wouldn't be worth doing"

You are absolutely correct! I just couldn't think of a way to further dive into that nuance, but I also wanted the example to be relatable and tangible. Thank you!

Culturally, 18% is the absolute minimum nowadays. An average tip is north of 20%.

I typically do 20% and round to the next dollar. So if the meal was $56.14, I would calculate $5.60 as approximately 10%, double it to $11.20 and then add 66 cents for a tip of $11.86 do the final total is $68.00.

The servers only enter the total line in the system, so this makes it easier for them.

The actual study claims that top 10% is $41k and accounts for 50% of carbon emissions. No where does it normalize incomes for those from Kenya as the article claims. So these incomes are viewed globally. If you are in the US and make more than $20/hr hours a week, you are top 10%.

$67/hr makes you top 1%.

Others are calling to eat the rich without realizing that the global rich includes low wage earners flipping burgers at McDonald's (I'm in Boston and minimum wage is $15/hr and an assistant manager can be hired for $22/hr).

https://oxfamilibrary.openrepository.com/bitstream/10546/621551/2/cr-climate-equality-201123-en.pdf

As an Alaskan voter, ranked choice is the only reason we have a female, Native American, Democrat congressional representative instead of Sarah Palin filling Don Young's deep red legacy. RCV is equitable and works, but not in the way progressives hope. It allows for the most centrist candidate to be chosen that appeals to the most possible people. A two party system just becomes a battle of political extremes. And like it or not, being progressive is far left for a reason, especially in America. And I consider myself fairly progressive leaning.

The honeymoon phase for Lemmy is over. No one cares anymore about someone expressing themselves (I still do--this is based on my observations).

All politicians, including Bernie must cater to the rich and wealthy. It's our jobs as voters to discern the true reason why they do it. It's a cruel reality that without the support of the wealthy, no one will make it to congress to effect change. Every successful politician knows this very well. They know not to bite the hand that feeds, so each one (*that works for progress and not personal gain) must walk a very thin line to effect that change while also making sure they aren't removed from office for fear that whatever they did could quickly be undone.

Bernie is in it for the right reasons. Biden and Clinton were in it for the right reasons. Having personally interacted with a few politicians in private settings (no cameras or hidden recorders) I can say that even some Republicans are in it for the right reasons, but differ in ideology and how to effect that change. In the end they play a fictional role when tv cameras are around.

Personally, I was most impressed by my interactions with Marco Rubio of all people. He's exceptionally bright, and cares very deeply about the protection of this country, and it's a very good thing he is vice chairman (or chairman during Republican senates) of the Senate Intelligence Committee as it seems to be his calling. It's just unfortunate that in front of the cameras, he needs to dumb himself down in order to get elected by Republican voters.

It's literally an area chart. There's no way to "skew" it in this format. $800B of $6,300B in spending is always going to be 12.6% no matter how you slice it.

4 more...