No doubt fuzzers (vibecoded or otherwise) can be powerful, but can't you just mark all "/" as potential divide by zero errors?
I guess sometimes developers think they "know" some variable won't be zero, but unless it checked explicitly or by the compiler, that shouldn't be trusted.
I mean there could be a guard clause? But yeah, seems like this could be statically evaluated like how some IDEs see a null check and don’t complain about nullability within the same scope.
The open ended bug hunt is the best use case for these agents. Finding nothing costs a few dollars, finding a division by zero in ffmpeg pays for itself.
It’s interesting how AI may both raise and lower the quality of software. It’s very easy to send an AI agent on an open-ended bug hunt, and if it wastes a bunch of time and effort and finds nothing, no big deal. Time is much more important for a human developer with a salary.
No, you frame it wrong. It's not reviewing output, it's (at the very least in this particular case) reviewing the outcome, - and without LLMs you ain't getting to review anything.
This ship has sailed, LLMs do find bugs, do save time, and do tasks dweebs considered art or craft (unable to do real either) with flying colors.
I don't care if you call it an over-engineered looping machine or what, there are concrete benefits to using LLMs for this. They work faster than developing your own looping algorithm and more often produce useful results than not.
I dislike AI, but if AI finds real bugs then this is in my opinion objectively a positive thing. Of course the question is what constitutes a real bug.
The fruits of using LLMs to code.
You'll waste far more time finding what it quietly and subtly wrecked than you would have if you just coded it yourself.
It’s obviously Claude 69 with time travel functionality, that’s too dangerous to release to public. They’re working on space-time limiting sandbox to prevent these issues.
No doubt fuzzers (vibecoded or otherwise) can be powerful, but can't you just mark all "/" as potential divide by zero errors?
I guess sometimes developers think they "know" some variable won't be zero, but unless it checked explicitly or by the compiler, that shouldn't be trusted.
What are you suggesting and how would it be different than how SIGFPE already works?
I mean there could be a guard clause? But yeah, seems like this could be statically evaluated like how some IDEs see a null check and don’t complain about nullability within the same scope.
The open ended bug hunt is the best use case for these agents. Finding nothing costs a few dollars, finding a division by zero in ffmpeg pays for itself.
It’s interesting how AI may both raise and lower the quality of software. It’s very easy to send an AI agent on an open-ended bug hunt, and if it wastes a bunch of time and effort and finds nothing, no big deal. Time is much more important for a human developer with a salary.
Finding the bugs with LLMs is easy. Reviewing the output, cleaning it up, and making sure it doesn't break something else is the hard part.
No, you frame it wrong. It's not reviewing output, it's (at the very least in this particular case) reviewing the outcome, - and without LLMs you ain't getting to review anything.
This ship has sailed, LLMs do find bugs, do save time, and do tasks dweebs considered art or craft (unable to do real either) with flying colors.
> LLMs do find bugs, do save time
They find bugs but whether they save time is nowhere near as clear as you try to insinuate here.
They save time in finding bugs.
And for me they also save time in fixing bugs.
> It’s very easy to send an AI agent on an open-ended bug hunt, and if it wastes a bunch of time and effort and finds nothing, no big deal.
No big deal? It’s not like it’s free… tokens cost money.
I don't care if you call it an over-engineered looping machine or what, there are concrete benefits to using LLMs for this. They work faster than developing your own looping algorithm and more often produce useful results than not.
But what's your expectation of the net?
I dislike AI, but if AI finds real bugs then this is in my opinion objectively a positive thing. Of course the question is what constitutes a real bug.
Unfiltered models will help build exploits for the bugs they find, so there is some means of measuring their efficacy.
Generating an incorrect input file seems to be the easiest task of all for any fuzzer.
Generating correct input to get deep into the call stack and then finding something is the hard part.
send patches
...they did.
Where?
They only suggested a basic guard, chich can be useless if this case never happens
The fruits of using LLMs to code. You'll waste far more time finding what it quietly and subtly wrecked than you would have if you just coded it yourself.
Those sneaky LLMs going 7 years into the past and committing as a human:
https://code.ffmpeg.org/FFmpeg/FFmpeg/commit/8eda3c7f91e1a5b...
It’s obviously Claude 69 with time travel functionality, that’s too dangerous to release to public. They’re working on space-time limiting sandbox to prevent these issues.
You're not reading it right. The bug was found using a vibecoded fuzzer.
I wonder from where Claude stole this fuzzer.
Or it used something called an "analogy" which is a valid way to solve new problems.