Debunking Devin: "First AI Software Engineer" Upwork lie exposed!

vallode@lemmy.world to Technology@lemmy.world – 73 points – Debunking Devin: "First AI Software Engineer" Upwork lie exposed!
youtube.com

YouTuber Internet of Bugs examines the latest demo from Cognition that showcases their "first AI software engineer" allegedly solving UpWork programming tasks.

10

Am I one of the few who just doesn't use AI at all? I don't have to generate tons of code for work at the moment and brand new projects that I've been given are small--meaning I wouldn't necessarily use it to generate starter boilerplate. I have coworkers that love copilot or spend much longer prompting ChatGPT than they would if they wrote code themselves. A majority of my time is spent modelling the problem, gathering rejuirements, researching others' solutions online (likely this step could be better AI-assisted?), not actually implementing a solution in code.

Anyway, I'm not super anti-AI in software development, and I see where it could be useful. Maybe it just isn't for me yet. The current hype around it as well as the attitude of big-tech exceptionalism ("AI can salve all our problems") feels a bit like a bubble, at least regarding the current generation of LLMs and ML

One way it can be useful is when you use it as a more verbal variant of rubber duck debugging. You'll need to state the issue that you're facing, including the context and edge cases. In doing so, the problem will also become more clear to you yourself.

Contrary to a rubber duck, it can then actually suggest some approach vectors, which you can then dismiss or investigate further.

This is how I use LLMs right now, and there have been a few times it's been genuinely helpful. Mind you, most of the time it's been helpful, it's because it hallucinates some nonsense that gets me in the right direction, but that's still at least a little better than the duck.

That was my experience as well with GPT 3.5. But the hit ratio is a lot better with GPT 4, and other models like Mixtral and its derivatives.

I'm in a similar boat. The challenges I run into at work are often resultant of the weird infrastructure we use in our datacenters and our weird legacy software stacks, so there's no reason to believe AI will have anything in its training set that will help me. People don't realize that AI is only as good as their problem is ubiquitous. If you spend your time working on weird legacy systems, it can't possibly predict all the weird needles you need to thread when developing solutions.

I think it's a sensible position to be in. I tend to use AI in order to build awareness of it's capabilities. I find that sometimes it is useful for brainstorming or "tip of my tongue" searches but as you say the actual coding capabilities are exaggerated.

To me Cognition AI is doing what most other hype-based startups do which is generate good headlines so that VC money can keep pouring in. It is up to us to spot and question things that don't quite make sense... and anything with the words "AI understands X" don't make sense currently ^^

Does anyone have an alternative title for this that doesn't sound clickbaity as f***? I'm kind of afraid to feed whatever media organization came up with that headline.

It is a typical Youtube clickbait title but the Youtuber in question doesn't seem to be nefarious, it does what it says on the tin :P I'd say the description covers what the video is about better.