Greg Rutkowski Was Removed From Stable Diffusion, But AI Artists Brought Him Back - Decrypt
decrypt.co
Greg Rutkowski, a digital artist known for his surreal style, opposes AI art but his name and style have been frequently used by AI art generators without his consent. In response, Stable Diffusion removed his work from their dataset in version 2.0. However, the community has now created a tool to emulate Rutkowski's style against his wishes using a LoRA model. While some argue this is unethical, others justify it since Rutkowski's art has already been widely used in Stable Diffusion 1.5. The debate highlights the blurry line between innovation and infringement in the emerging field of AI art.
You are viewing a single comment
What blurry line? An artist doesn’t what his art stolen from him. Seems pretty cut and dry to me.
I don’t disagree but stolen is a bit of a stretch
Nothing was stolen.
Drawing inspiration from someone else by looking at their work has been around for centuries.
Imagine if the Renaissance couldn't happen because artists didn't want their style stolen.
I don't fully understand how this works, but if they've created a way to replicate his style that doesn't involve using his art in the model, how is it problematic? I understand not wanting models to be trained using his art, but he doesn't have exclusive rights to the art style, and if someone else can replicate it, what's the problem?
This is an honest question, I don't know enough about this topic to make a case for either side.
TL;DR The new method still requires his art.
LoRA is a way to add additional layers to a neural network that effectively allow you to fine tune it's behaviour. Think of it like a "plugin" or a "mod"
LoRas require examples of the thing you are targeting. Lots of people in the SD community build them for particular celebrities or art styles by collecting examples of the that celebrity or whatever from online.
So in this case Greg has asked Stable to remove his artwork which they have done but some third party has created an unofficial LoRA that does use his artwork to mod the functionality back in.
In the traditional world the rights holder would presumably DMCA the plugin but the lines are much blurrier with LoRA models.
Great explanation, thanks!
Do you know how they recreated his style? I couldn’t find such information or frankly have enough understanding to know how.
But if they either use his works directly or works created by another GAI with his name/style in the prompt, my personal feeling is that would still be unethical, especially if they charge money to generate his style of art without compensating him.
Plus, I find that the opt-out mentality really creepy and disrespectful
I still have trouble understanding the distinction between "a human consuming different artists, and replicating the style" vs "software consuming different artists, and replicating the style".
there's no distinction. people are just robophobic.
LORA's are created on image datasets, but these images are just available anywhere. It's really not much different from you taking every still of The Simpsons and using it. What I don't understand is how these are seen as problematic because a majority of end users utilizing AI are doing it under fair use.
No one charges for LORA's or models AFAIK. If they do, it hasn't come across the Stable Diffusion discords I moderate.
People actually selling AI generated art is also a different story and that's where it falls outside of fair use if the models being used contain copy-written work. It seems pretty cut and dry, artists complained about not being emulated by other artists before AI so it's only reasonable that it happens again. If people are profiting off it, it should be at least giving compensation to the original artist (if it could be adjusted so that per-token payments are given as royalties to the artist). However, on the other hand think about The Simpsons, or Pokemon, or anything that has ever been sold as a sticker/poster/display item.
I'm gonna guess that a majority of people have no problem with that IP theft cause it's a big company. Okay... so what if I love Greg but he doesn't respond to my letters and e-mails begging him to commission him for a Pokemon Rutkowski piece? Under fair use there's no reason I can't create that on my own, and if that means creating a dataset of all of his paintings that I paid for to utilize it then it's technically legal.
The only thing here that would be unethical or illegal is if his works are copywritten and being redistributed. They aren't being redistributed and currently copy-written materials aren't protected from being used in AI models, since the work done from AI can't be copywritten. In other words, while it may be disrespectful to go against the artists wishes to not be used in AI, there's no current grounds for it other than an artist not wanting to be copied... which is a tale as old as time.
TL;DR model and LORA makers aren't charging, users can't sell or copywrite AI works, and copywritten works aren't protected from being used in AI models (currently). An artist not wanting to be used currently has no grounds other than making strikes against anything that is redistributing copies of their work. If someone is using this LORA to recreate Greg Rutkowski paintings and then proceeds to give or sell them then the artist is able to claim that there's theft and damages... but the likelihood of an AI model being able to do this is low. The likelihood of someone selling these is higher, but from my understanding artistic styles are pretty much fair game anyway you swing it.
I understand wanting to protect artists. Artists also get overly defensive at times - I'm not saying that this guy is I actually am more on his side than my comment makes it out, especially after how he was treated in the discord I moderate. I'm more just pointing out that there's a slippery slope both ways and the current state of U.S. law on it.
Generally speaking, the way training works is this:
You put together a folder of pictures, all the same size. It would've been 1024x1024 in this case. Other models have used 768z768 or 512x512. For every picture, you also have a text file with a description.
The training software takes a picture, slices it into squares, generates a square the same size of random noise, then trains on how to change that noise into that square. It associates that training with tokens from the description that went with that picture. And it keeps doing this.
Then later, when someone types a prompt into the software, it tokenizes it, generates more random noise, and uses the denoising methods associated with the tokens you typed in. The pictures in the folder aren't actually kept by it anywhere.
From the side of the person doing the training, it's just put together the pictures and descriptions, set some settings, and let the training software do its work, though.
(No money involved in this one. One person trained it and plopped it on a website where people can download loras for free...)
That's really the big thing, not just here but any material that's been used to train on without permission or compensation. The difference is that most of it is so subtle it can't be picked out, but an artist style is obviously a huge parameter since his name was being used to call out those particular training aspects during generations. It's a bit hypocritical to say you aren't stealing someone's work when you stick his actual name in the prompt. It doesn't really matter how many levels the art style has been laundered, it still originated from him.
It is unconditionally impossible to own an artistic style. "Stealing a style" cannot be done.
And yet the artist's name is used to push the weights towards pictures in their style. I don't know what the correct semantics are for it, nor the legalities. That's part of the problem, the tech is ahead of our laws, as is usually the case.
That's not even vaguely new in the world of art.
Imitating style is the core of what art is. It's absolutely unconditionally protected by copyright law. It's not even a .01 out of 10 on the scale of unethical. It's what's supposed to happen.
The law might not cover this yet, but any law that restricts the fundamental right to build off of the ideas of others that are the core of the entirety of human civilization is unadulterated evil. There is no part of that that could possibly be acceptable to own.
I totally agree with you on protecting the basics of creativity and growth. I think the core issue is using "imitate" here. Is that what the LLM is doing, or is that an anthropomorphism of some sense that there's intelligence guiding the process? I know it seems like I'm nitpicking things to further my point, but the fact that this is an issue to many even outside artwork says there is a question here of what is and isn't okay.
The AI is not intelligent. That doesn't matter.
Nothing anyone owns is being copied or redistributed. The creator isn't the tool; it's the person using the tool.
AI needs two things to work, an algorithm and data. If training is allowed to anyone, anyone can create their own algorithms and use the AI as a tool to create innovative new messages with some ideas borrowed from other work.
If data is proprietary, they cannot. But Disney still can. They'll just as successfully flood out all the artists who can't use AI because they don't have a data set, but now they and the two other companies in the world who own IP are basically a monopoly (or tri- or whatever) and everyone else is screwed.
It's only using his name because the person who created the LORA trained it with his name. They could have chosen any other word.
True, and then because it's a black box there wouldn't be a known issue at all. Or maybe it would be much less of an issue because the words might have blended others into the mix, and his style wouldn't be as obvious in the outputs, and/or it would be easier to dismiss. Did the training involve actual input of his name, or was that pulled from the source trained on? How much control was in the training?
Just wait until you can copywrite a style. Guess who will end up owning all the styles.
Spoiler, it's wealthy companies like Disney and Warner. Oh you used cross hatching? Disney owns the style now you theif.
Copyright is fucked. Has been since before the Mickey mouse protection act. Our economic system is fucked. People would rather fight each other and new tools instead of rallying against the actual problem, and it's getting to me.
You're right, copyright won't fix it, copyright will just enable large companies to activate more of their work extract more from the creative space.
But who will benefit the most from AI? The artists seem to be getting screwed right now, and I'm pretty sure that Hasbro and Disney will love to cut costs and lay off artists as soon as this blows over.
Technology is capital, and in a capitalist system, that goes to benefit the holders of that capital. No matter how you cut it, laborers including artists are the ones who will get screwed.
Me, I'll benefit the most. I've been using a locally running instance of the free and open source AI software Stable Diffusion to generate artwork for my D&D campaigns and they've never looked more beautiful!
Same here. It's awesome being able to effectively "commission" art for any random little thing the party might encounter. And sometimes while generating images there'll be surprising details that give me new ideas, too. It's like brainstorming with ChatGPT but in visual form.
Is drawing Mickey Mouse in a new pose copying the style or copying Mickey Mouse?
You said it yourself. You're drawing Micky mouse in a new pose, so you're copying Mickey mouse.
Drawing a cartoon in the style of Mickey mouse isn't the same thing.
You can't have a copyright on "big oversized smile, exaggerated posture, large facial features, oversized feet and hands, rounded contours and a smooth style of motion".
The second.
I'm not sure how that's relevant here, though. There is nothing at all being copied but an aesthetic.
Although to me it is interesting that, even without literal copying, a generator might be capable of potentially emulating some key features of a specified source. Can this sometimes arguably extend beyond just "an aesthetic"? We've all seen examples similar to this one (from the SD online demo, default setting, with a familiar public-domain source) — https://i.imgur.com/PUJs3RL.png
I don't, but another poster noted that it involves using his art to create the LoRA.
I don't know about creepy and disrespectful, but it does feel like they're saying "I know the artist doesn't want me to do this, but if he doesn't specifically ask me personally to stop, I'm going to do it anyway."
You're pretty spot on. It's not much different from a human artist trying to copy his style by hand but without reproducing the actual drawings.
His art was not "stolen."
If I look at someone's paintings, then paint something in a similar style did I steal their work? Or did I take inspiration from it?
No, you used it to inform your style.
You didn't drop his art on to a screenprinter, smash someone else's art on top, then try to sell t-shirts.
Trying to compare any of this to how one, individual, human learns is such a wildly inaccurate way to justify stealing a someone's else's work product.
If it works correctly it's not a screenprinter, it's something unique as the output.
The fact that folks can identify the source of various parts of the output, and that intact watermarks have shown up, shows that it doesn't work like you think it does.
They can't, and "intact" watermarks don't show up. You're the one who is misunderstanding how this works.
When a pattern is present very frequently the AI can learn to imitate it, resulting in things that closely resemble known watermarks. This is called "overfitting" and is avoided as much as possible. But even in those cases, if you examine the watermark-like pattern closely you'll see that it's usually quite badly distorted and only vaguely watermark-like.
Yes, because "imitate" and "copy" are different things when stealing from someone.
I do understand how it works, the "overfitting" was just laying clear what it does. It copies but tries to sample things in a way that won't look like clear copies. It had no creativity, it is trying to find new ways of making copies.
If any of this was ethical, the companies doing it would have just asked for permission. That they didn't says a everything you need to know.
I don't usually have these kinds discussions anymore, I got tired of conversations like this back in 2016, when it became clear that people will go to the ends of the earth to justify unethical behavior as long as the people being hurt by it are people they don't care about.
And we're back to you calling it "stealing", which it certainly is not. Even if it was copyright violation, copyright violation is not stealing.
You should try to get the basic terminology right, at the very least.
Just because you've redefined theft in a way that makes you feel okay about it doesn't change what they did.
They took someone else's work product, fed it into their machine then used that to make money.
They stole someone's labor.
I haven't "redefined" it, I'm using the legal definition. People do sometimes sloppily equate copyright violation with theft in common parlance, but they're in for a rude awakening if they intend to try translating that into legal action.
Using that term in an argument like this is merely trying to beg the question of whether it's wrong, since most everyone agrees that stealing is wrong you're trying to cast the action of training an AI as something everyone will by default agree is wrong. But it's not stealing, no matter how much you want it to be, and I'm calling that rhetorical trick out here.
If you want to argue that it's wrong you need to argue against the actual process that's happening, not some magical scenario where the AI trainers are somehow literally robbing people.
Taking someone's work product and converting it, without compensation and consent, into your profit is theft of labor.
Adding extra steps, like, say, training an AI, doesn't absolve the theft of labor.
We're it ethical, the companies doing it would have asked for permission and been given cinsent. They didn't.
That's not what's going on here. The finished product contains only the style of the artist that the AI was trained on, and style is not copyrightable. Which is a damn good thing, as humans have been learning from each other's "work products" and mimicking each others' styles since time immemorial.
BTW, theft of labor means failing to pay wages or provide employee benefits owed to an employee by contract or law. You're using that term incorrectly too, Greg Rutkowski wasn't hired to do anything for the people who trained the AI off of his work.
No, I'm not using it incorrectly, I'm just not concerned with the legal definition as I'm not a lawyer or anyone tied up in this mess.
If you do a thing, and it takes time and skill to do it, then someone copies it, they stole your labor.
Saying they "copied his style", the style he spent a lifetime crafting, then trying to say they didn't benefit, at no cost, to the labor he put into crafting that style because "well actually, the law says..." is a bad argument as it tries to minimize what they did.
If their product could not exist without his labor, and they did not pay him for that labor, they stole his labor.
For, like, the fourth time in this thread: were this ethical, they would have asked for permission, they didn't.
Does that mean the AI is not smart enough to remove watermarks, or that it's so smart it can reproduce them?
It means that it's stupid enough that it reproduces them - poorly.
It's not smart or stupid. It does what it's been trained on, nothing more.
LLMs and directly related technologies are not AI and possess no intelligence or capability to comprehend, despite the hype. So, they are absolutely the former, though it's rather like a bandwagon sort of thing (x number of reference images had a watermark, so that's what the generated image should have).
That's debatable. LLMs have shown emergent behaviors aside from what was trained, and they seem to be capable of comprehending relationships between all sorts of tokens, including multi-modal ones.
Anyway, Stable diffusion is not an LLM, it's more of a "neural network hallucination machine" with some cool hallucinations, that sometimes happen to be really close to some or parts of the input data. It still needs to be "smart" enough to decompose the original data into enough and the right patterns, that it can reconstruct part of the original from the patterns alone.
Thanks for the clarification!
LLMs have indeed shown interesting behaviors but, from my experience with the technology and how it works, I would say that any claims of intelligence being possessed by a system that is only an LLM would be suspect and require extraordinary evidence to prove that it is not mistaken anthropomorphizing.
I don't think an LLM alone can be intelligent... but I do think it can be the central building block for a sentient self-aware intelligent system.
Humans can be thought of as being made of a set of field-specific neural networks, tied together by a looping self-evaluating multi-modal LLM that we call "conscience". The ability of an LLM to consume its own output, is what allows it to be used as the conscience loop, and current LLMs being trained on human language with all its human nuance, is an extra bonus.
Probably some other non-text multi-modal neural networks capable of consuming their own output could also be developed and be put in a loop, but right now we have LLMs, and we kind of understand most of what they're saying, and they kind of understand most of what we're saying, so that makes communication easier.
I mean, it is anthropomorphizing, but in this case I think it makes sense because it's also anthropogenic, since these human language LLMs get trained on human language.
Absolutely agreed with most of that. I think that LLMs and similar technologies are incredible and have great potential to be components of artificial intelligences. LLMs by themselves are more akin to "virtual intelligences" portrayed in the Mass Effect games, but currently generally with fewer guard rails to prevent hallucinations.
I suspect there may be a few other concurrent "loops", likely not as well compared to LLMs (though some might be) running in our meat computers and their inefficiency and poor fidelity likely ends up being part of the factors that make our consciousness. Otherwise, your approximation makes a lot of sense. Still a lot to learn about our meat computers but, I really do hope we, as a species, succeed in making the world a bit less lonely (by helping other intelligence emerge).
There is some discussion about people "with an internal monologue", and people "without". I wonder if those might be some different ways of running that loop, or maybe some people have one loop take over others... and the whole "dissociative personality disorder" could be multiple loops competing for being the main one at different times.
Related to fidelity, some time ago I read an interesting thing: consciousness means having brainwaves out of sync, when they get in sync people go unconscious. From a background in electronics, I've always assumed the opposite (system clock and such), but apparently our consciousness emerges from the asynchronous differences, meaning the inefficiencies and poor fidelity might be a feature, not a bug.
Anyway, right now, as someone suffering from insomnia, I'd happily merge with some AI just to get a "pause" button.
It's like staring yourself blind at artworks with watermarks until you start seeing artworks with blurry watermarks in your dreams