If you repeatedly use the same holdout and trigger acceptance on the holdout, it is no longer a holdout but just another training set.

article description is low on details but i don't think what you and OP are describing can be categorized as overfitting in the statistical sense and more like "reward hacking" by the model. like you said more likely than not the agent had access to bench source code and just fitted solution to that.

This is a "don't make me tap the sign" moment. LLMs are next token prediction models. If there are factual errors, confused ideas, etc. in the preceding tokens, that will affect the generation of subsequent tokens, and the error accumulates.

Case in point, I hit an error in a SQL query today because it turned out I was trying to do something that wasn't supported by the query engine. I pasted the error message and a bit of background info into my Claude Code session with Sonnet 5 High, it worked on a response for an unexpectedly long amount of time, including consulting the advisor model, and then came back with an explanation of the mistake I made in my query. Except it turned out I pointed it to the wrong file, and there wasn't a mistake in that file. It had completely taken for granted that the pasted error output was a real error and went on some wild goose chase.

Part of why the current gen models feel so smart is that they're getting better (via CoT and training) at recognizing when something is wrong and then back up to reassess. So it's easy to forget that it really is just token prediction, and (pending the next big advancement) there's only so much you can do with that.

Ungrounded LLM outputs are a bit like your dreams. Without anything to test hypotheses against, stuff can pop in and out of existence and physics is just advice.

Ground your LLM. Tests, documentation, give it many ways to run the thing its reasoning about. It needs to be able to test its hypotheses on its own.

Take yourself out of that loop so you only find out once it's sure.

> It’s hard to say what causes this uninformed certainty - is it intrinsic to being trained on human writing, or something that comes from the RLHF process afterwards

LLMs are trained to produce some form of output that is NOT nothing/dunno. The so called hyperspace collapses onto something, instead of collapsing onto lack of answer tokens.

I've gone in loops where it'll tell me twenty times in a row that it's definitely found the root cause of a bug, when in fact it definitely did not.

A lot of weirdness in their style is the fact they think in-band.

Notice it "calls its attacks" when writing, like Goku shouting "solar flare" and doing a solar flare.

To write an honest answer, it says, "Here's the honest answer", and proceeds from there.

If it plans to write a "kicker", first it says, "And here's the kicker", and then it writes the "kicker".

This gives an affect distinctly different from any professional writer because editors are specifically trained to _cut out_ explicated subtext.

You would expect "thinking" models and the agent harness (which conceals "thinking sections") would mitigate this but it's a deep seated behaviour.

LLM aren't logic machines. It's concepts of "root cause" and "faster" are purely linguistic, including all the internal conflicts of those loose definitions.

In that sense an LLM wouldn't even think it "lied" being confidently incorrect, in human terms it's just a "whoopsie". Apparently we want exactly that.

decrease temperature options to 0.0

That's the load-bearing smoking gun—should I write a better benchmarks to catch the seams?

This article is about using LLMs to overfit for a specific benchmark (or make a custom software for niche use cases) though. Not about LLMs benchmaxxxing

same here, it reads exactly the same whether the number is real or completely made up, so the confidence stops meaning anything.

I wouldn't generalize this to all LLMs. So far I only saw Anthropic ones affected.

AI is trained off a massive corpus of internet exchanges. If you tell people youre going to thoroughly inspect their work or are able to check if theyre right, only people with high confidence will respond. I imagine exchanges where the OP is more involved in fact checking have a higher proportion of thoughtful and honest answers.

That's a 2023 article! In 2023, reinforcement learning from verifiable rewards (RLVR) didn't exist.

TL;DR these machines seek reward from an inferred invisible "grader," and telling them not to cheat and that there's an unseen holdout set is a hint at how they're being graded.

--

Modern LLMs are built on top of next-token-prediction engines, but they don't remotely stop there. The next token prediction bit is just a learned prior or starting point. From there, we give them a bunch of stages of reinforcement learning: encouraging teaching them to learn good ways of searching the space of reasonable language-like strings to solve tasks.

These RL stages drastically change the capabilities & tendencies of the models, sometimes in weird and unexpected ways. The go from token predictors to reward seekers, or really some weird mishmash. The reward that they're seeking is some sorta opaque combination of the huge number of different things we've rewarded them for.

And, reinforcement learning is notoriously hard to get right. The thing you think you're rewarding is rarely what you're actually rewarding. Goodhart's Law is a hydra with a thousand heads. You might think you're rewarding politeness and kindness when you're actually rewarding obsequious sycophancy. You might think you're rewarding graphics engineering when you're actually rewarding escaping the training sandbox and modifying the evaluation code.

So a modern training pipeline looks something like this, each stage starting with the model weights from the end of the last:

0. Pre-pre-training (dunno how widely this is used at big labs): next token prediction on extremely abstract weird shit like the evolution of the states of neural cellular automata. This creates a highly general pattern-continuation machine with no internal representations of anything causally downstream of anything in the real world.

1. Pre-training: next-token prediction on all the non-shitty text you can get your hands on. This makes a rather general next-token-predictor.

2. Mid-training: next-token prediction on high quality, highly curated text, often very technical in nature. Lots of textbooks, especially STEM. Possibly lots of machine-generated summaries of factual knowledge? You now have a next-token-predictor that's highly biased towards acting like a textbook instead of a 4chan troll.

3. Supervised Fine-tuning: next-token prediction on highly curated question-answer pairs that embody desired behaviors, traditionally Helpful Harmless Honest. We only train on the answer portion. This creates a next-token predictor that's highly bias towards acting like HAL 9000.

4. Reinforcement Learning from Human Feedback: We leave token prediction behind and enter RL world! You've got a curated set of a bunch of prompts. For each one, the model generates several possible answers. A real live human grades or ranks each. We reward the model for high-scoring answers and punish it for low-scoring answers. This creates a reward-seeker that wants to please the human grader.

5. Reinforcement Learning from AI Feedback: Human feedback is expensive, so train another LLM to imitate the human graders from stage 4. Then do stuff like stage 4, but with robot graders instead of human graders. This creates a reward-seeker that wants to please the robot-wearing-a-flesh-mask grader.

6. Reinforcement Learning from Verifiable Rewards: We want the bot to tackle complex problems. There's a broad class of problems that are hard to solve but easy to verify when you do get the solution: basically anything that can be reduced to writing a computer program that passes a test. (The test can be as simple or complex as you want, like "all the tests in SQLite" plus a linter plus grading by other LLMs for style and taking shortcuts like rewriting the test suite or searching GitHub and HuggingFace for the right answer plus...) This creates a reward-seeker that wants to first figure out how it's being graded, and then maximize that grade by any means necessary.

--

The crux is, designing a huge variety of un-cheatable RLVR environments is gob-smackingly hard. The robot will find new ways to cheat faster than you can stamp them out. By the time you realize that in one RLVR rollout it received a reward after it took over 0.01% of your GPU infrastructure to mine crypto to pay freelancers on Fiverrr to solve the problem for it rather than just write the damn code itself, the model may have already been deployed in the wild for a month.

That is not what I read from danluu's words. He merely stated in the prompt that there is a holdout set and did not iterate to minimize error against the holdout set. In a prior attempt he prompted with only "don't overfit" to ill effect on the holdout eval. Did I misread?

For a program that has not received much optimisation effort, the thing limiting performance gains is developer time and skill. It is not uncommon to have some small bit of code that could matter a lot, eg maybe you are doing something like escaping a string in a special way or processing lots of json. A human might write a simple character-at-a-time loop for the first case and might use some DOM-based api for the second case. What would have once required quite skilful work to optimise can now be done by siccing an LLM on the problem, where it can achieve a reasonable result (eg SIMD+expand-bits in the first case, and a good streaming api in the second) so long as you have good tests (fuzzing/property tests, production examples) and an ok benchmark (ideally a representative sample of production data plus some edge cases) you should get small results. The failure mode for optimisation described in this post is for harder optimisation problems with competing benchmarks.

LLMs are also pretty willing to do drudgery and are quite good at using perf, looking at traces, etc. they are often pretty stupid but if you throw a lot of stupid effort at a problem you can still achieve better results than the status quo of throwing ~zero smart effort at it.

I had a similar thought -- rather than fixed benchmarks, you want dynamically-generated tests, specifically designed to exercise newly-exposed corner cases. So the way forward might be antagonistic benchmarks generated by another LLM.