I had been playing with and building fuzzers long before AFL, but its release and its ability to explore the code space in a smart way fundamentally changed the game. Never once have I improved a codebase so it could be fuzzed and didn't get a payoff of it finding bugs.
And the field is relatively young. Like the other year built a fuzzer with a transformer model as the internal learning engine and other cool stuff like https://antithesis.com
I know fuzzing has been around forever, but it is still surprising how many engineers I encounter who have never used one. I hate fixing bugs and love tools that just find bugs and gave me exact reproducible cases.
It was nice to see GoLang get a built in fuzzer to try to help it become more mainstream.
And these days with AI where you need ways it both can't gamify (a unit test that simply returns 'pass') as well as automatic ways to eval generate code fuzzing is a great fit. And not to mention that it is cheaper to fuzz all night than continuously burn tokens looking for ways to crash the code.
jonhohle 23 hours ago [-]
I used to think writing safe code in C/C++ was a matter of skill. However, after finding more than a few issues with AFL at a FAANG in code written by engineers with the highest engineering title, I changed my mind. Incredible tool.
fasterik 22 hours ago [-]
It is a matter of skill to a large extent, though. A skilled programmer will write fewer bugs on average than an unskilled one. They'll know the patterns to avoid and have a higher bar for quality. Nobody can write 100% bug-free code, independent of programming language. Also, job title might correlate with skill but it doesn't rule out incompetence at the highest levels.
tikhonj 19 hours ago [-]
Writing safer C is a matter of skill.
Writing safe C is a matter of either radically different (much, much slower) process or better tooling or both. Skill is not sufficient.
kragen 21 hours ago [-]
Sometimes writing shitty code is a result of competence, not incompetence. I've seen much better programmers than I ever will be writing terrible code because it was the fastest way to get a prototype running and find out whether the problem was worth writing good code for.
eptcyka 9 hours ago [-]
I love how it takes 2 comments to go from *Writing safe/good C code is really hard* to *Sometimes shitty code is good, actually*. In the context of using fuzzers to find bugs in existing code.
embedding-shape 8 hours ago [-]
Happens in all HN comment threads, everyone answers from their own perspective and context without explicitly specifying it, so if you don't read the sub-thread (first person talks about production software, last person talks about prototypes, in this case), it'll seem like people talk past each other, but in reality everyone is sharing truths that are applicable to their specific environment and context.
eptcyka 4 hours ago [-]
In this case, I feel like people are commenting for the sake of expressing their opinion, without much regard if it is materially contributing to the topic at hand. And now, neither am I. :(
kragen 4 hours ago [-]
I was commenting on why jonhohle might have been "finding more than a few issues with AFL at a FAANG in code written by engineers with the highest engineering title". I don't know the context he was finding them in, because he didn't think it was relevant enough to mention, and my whole claim here is that you have to take that context into account.
ambicapter 22 hours ago [-]
It doesn't really matter, the most skilled programmers are usually working on harder stuff, so the bug rate remains fairly consistent.
Neywiny 22 hours ago [-]
I think it's mostly a testing thing. Valgrind is a friend a lot of people haven't heard of for c/c++. Setting up good regression tests is an art. Maintaining them is even harder.
rfgplk 10 hours ago [-]
C++ actually has numerous footguns that even experienced developers are completely unaware of. Main reason being is that C++ follows three models in parallel
* the C++ standard
* the CPU arch details
* the OS details
The standard itself delegates a large chunk of functionality to be "platform specific" which will inherently depend either on the CPU or the underlying OS. In fact C++ itself is entirely silent about the ISA other than loosely specifying the virtual machine the language is supposed to run on. So you can get scenarios where some code is perfectly safe on one arch (say alignment doesn't matter) while being severe UB/a failure condition on another (alignment matters and you fault the second you touch misaligned data). There are also other issues like integer over/underflow, fp issues, SIMD issues, memory access issues, stack/heap issues; all of which you MUST address.
Other languages don't quite have these problems because the compiler/runtime handles them and they instead give you a unified interface behind which to write your code.
Conscat 18 hours ago [-]
I think that good use of static analysis and fuzz testing is a part of the skill to detail-oriented programming languages.
PorciiVorbesc 21 hours ago [-]
>I used to think writing safe code in C/C++ was a matter of skill.
It's a matter of how many fucks the devs gave about doing quality work. You can find inexperienced people deliver quality work simply because they're passionate and they care (Half-Life was made in part by people with no prior game or coding experience), and you can find supposed rockstars deliver slop because for whatever reason they don't care, they just want to punch in, collect paychecks, then punch out.
FAANG stopped being a benchmark of skill a while ago. It's a skill of griding leetcode and 7 stage interviews to get it, then resting and vesting till you can jump ship to something better or get mass layd off.
pfdietz 20 hours ago [-]
> It's a matter of how many fucks the devs gave about doing quality work.
It's more a matter of how much management cares about quality, and how much time and how many resources they're willing to devote to it.
And that's a function of how much the quality of the code actually matters in the market.
PorciiVorbesc 20 hours ago [-]
FANG doesn't have resources, but indie coders and game studios do?
bawolff 16 hours ago [-]
You can have all the resources in the world but it won't matter if you spend them on something else.
pfdietz 20 hours ago [-]
Notice I said how much management is willing to devote, not how much they have.
nxobject 20 hours ago [-]
I think you’re a bit optimistic about the quality of a triple-A game’s codebase that was rushed to a holiday release date. I have slightly more tolerance for bugginess in a game.
sczi 19 hours ago [-]
Has there ever existed a C or C++ codebase of significant size that hasn't had exploitable memory corruption issues? I don't think we can keep saying it's a question of the dev's passion or skill or whatever, if no dev has ever managed to do it.
That's not to say memory corruption issues are the only kind of issues, and rewriting a mature C/C++ codebase into a memory safe language may well introduce more issues than it fixes...
muvlon 12 hours ago [-]
An example that comes to mind is postfix (the SMTP daemon). It hasn't had zero memory safety issues, but nearly so. Certainly an abnormally low amount given the size/complexity of the project and the number of eyes on it.
It's also had a pretty 'normal' (read: higher) amount of non-memory related bugs such as SMTP smuggling of filesystem permission TOCTOU, so it's not like its devs are somehow really good at just not writing bugs.
Maybe they have chosen a software architecture that allows them to avoid complicated memory-related logic? I genuinely don't know, would love to hear from a postfix developer.
uecker 13 hours ago [-]
Outside of specially designed and verified code, probably not. Although there are certainly many projects where there are so few memory-safety issues that is is irrelevant in practice compared to other issues. Most of the issues I see are very easily preventable by using a string and a buffer abstractions that do bounds checking.
Also I think it is noteworthy that memory-safety issues are really well-defined and can automatically be found using sanitizers and fuzzers or verification tools. This is not the case for many other correctness issues and once memory safety issues are fixed this will remain as a major problem for all code.
AlotOfReading 5 hours ago [-]
[dead]
18 hours ago [-]
printer_in_pink 10 hours ago [-]
It is a largely a skill issue.
My first litmus test is: is the codebase using the standard library? If so, it's probably insecure garbage.
Nothing in libc is usable in a high quality code base, hell not even the humble sin() function is sensible with its retarded errno nonsense.
rfgplk 10 hours ago [-]
Yea, one of the reasons I rolled my own libc equivalent is how actually awful glibc (and the STL) are. If you actually read the glibc source you'll find countless comments such as "TODO: we need to account for X failure state" or ridiculous UB prone codepaths (signal trampolines, TLS, threading for instance). As in, these are things that can and SHOULD be guarded by the libc itself instead they just allow plain UB to occur.
The STL is awful by extension because it delegates ALL of this work to libc (anything that interacts with the OS) and they integrate this deeply into every single container.
18 hours ago [-]
rwmj 6 hours ago [-]
This is quite timely. I just radically increased fuzzing of hivex (a Windows Registry parser) by switching to persistent mode. Execs/sec went from around 440 to 70,000, although the test cases aren't quite comparable. Documentation for persistent mode is quite terrible though.
I had been playing with and building fuzzers long before AFL, but its release and its ability to explore the code space in a smart way fundamentally changed the game. Never once have I improved a codebase so it could be fuzzed and didn't get a payoff of it finding bugs.
And the field is relatively young. Like the other year built a fuzzer with a transformer model as the internal learning engine and other cool stuff like https://antithesis.com
I know fuzzing has been around forever, but it is still surprising how many engineers I encounter who have never used one. I hate fixing bugs and love tools that just find bugs and gave me exact reproducible cases. It was nice to see GoLang get a built in fuzzer to try to help it become more mainstream.
And these days with AI where you need ways it both can't gamify (a unit test that simply returns 'pass') as well as automatic ways to eval generate code fuzzing is a great fit. And not to mention that it is cheaper to fuzz all night than continuously burn tokens looking for ways to crash the code.
Writing safe C is a matter of either radically different (much, much slower) process or better tooling or both. Skill is not sufficient.
* the C++ standard
* the CPU arch details
* the OS details
The standard itself delegates a large chunk of functionality to be "platform specific" which will inherently depend either on the CPU or the underlying OS. In fact C++ itself is entirely silent about the ISA other than loosely specifying the virtual machine the language is supposed to run on. So you can get scenarios where some code is perfectly safe on one arch (say alignment doesn't matter) while being severe UB/a failure condition on another (alignment matters and you fault the second you touch misaligned data). There are also other issues like integer over/underflow, fp issues, SIMD issues, memory access issues, stack/heap issues; all of which you MUST address.
Other languages don't quite have these problems because the compiler/runtime handles them and they instead give you a unified interface behind which to write your code.
It's a matter of how many fucks the devs gave about doing quality work. You can find inexperienced people deliver quality work simply because they're passionate and they care (Half-Life was made in part by people with no prior game or coding experience), and you can find supposed rockstars deliver slop because for whatever reason they don't care, they just want to punch in, collect paychecks, then punch out.
FAANG stopped being a benchmark of skill a while ago. It's a skill of griding leetcode and 7 stage interviews to get it, then resting and vesting till you can jump ship to something better or get mass layd off.
It's more a matter of how much management cares about quality, and how much time and how many resources they're willing to devote to it.
And that's a function of how much the quality of the code actually matters in the market.
That's not to say memory corruption issues are the only kind of issues, and rewriting a mature C/C++ codebase into a memory safe language may well introduce more issues than it fixes...
It's also had a pretty 'normal' (read: higher) amount of non-memory related bugs such as SMTP smuggling of filesystem permission TOCTOU, so it's not like its devs are somehow really good at just not writing bugs.
Maybe they have chosen a software architecture that allows them to avoid complicated memory-related logic? I genuinely don't know, would love to hear from a postfix developer.
Also I think it is noteworthy that memory-safety issues are really well-defined and can automatically be found using sanitizers and fuzzers or verification tools. This is not the case for many other correctness issues and once memory safety issues are fixed this will remain as a major problem for all code.
The STL is awful by extension because it delegates ALL of this work to libc (anything that interacts with the OS) and they integrate this deeply into every single container.
https://github.com/libguestfs/hivex/commit/2254c62832c568804...