I've been casually spot-checking other animals in other vehicles, because my absolute dream situation here is to catch an AI lab that's demonstrably better at pelicans on bicycles than other combinations.
Catching a lab cheating specifically on my one dumb benchmark would be really funny.
Dylan's methodology here - generating 1008 SVGs across an 8x6 combination - is significantly more robust than anything I was considering.
His conclusion:
> Nothing jumped out at me. I couldn’t find a case where the pelican-bicycle images looked noticeably better than the rest of that model’s grid.
lukev 1 days ago [-]
What if they’re not pelicanmaxxing, but svgmaxxxing in general?
Because otherwise using a LLM to generate complex svgs is pretty niche and what I thought made this a good benchmark when it was new - generalized programming and spatial knowledge.
Obviously image gen in svg format is not a particularly hard problem if tackled directly on its own.
qq66 1 days ago [-]
But that's a genuine worthwhile capability. It's like benchmarkmaxxing on a weightlifting competition by getting really strong.
sandcat_ 19 hours ago [-]
Yes, but the original purpose of the benchmark (simonw, please correct me if I'm wrong!) was to test whether new models were good at novel problem solving. Things they haven't been trained on. So yes, getting better at generating SVGs is great news (and it seems they have been) but this particular benchmark still strikes me as largely worthless now, unless SVGs happen to be what you care about in particular when a new model drops.
Instead, it feels like a more appropriate benchmark for the original purpose would be to come up with new, novel problems each time, and compare across all models (including previous ones).
Throwing my hat in the ring: Generate a pelican shaped crossword where all the clues are related to bicycles.
(Haiku 4.5: https://imgur.com/a/N112Nxo, I'm trying some others but it's very slow! Opus has been at it for about 20 minutes.)
phreeza 12 hours ago [-]
I think a lot of LLM progress in the last 2 years or so is really from densifying the training data to niches like this and not so much from general problem solving capability. I guess the hope is that the deluge of super specific training data will pay off through better overall competence. cf https://www.dwarkesh.com/p/the-sample-efficiency-black-hole
qq66 16 hours ago [-]
I do have a private benchmark I use for that. I have a repository with an in-progress codebase and a very large planning document, I tell the new models to simply finish everything in the planning document, and I never push those changes. All of the recent models have essentially gotten it perfect which means that the models are all equivalently good for my level of needs.
londons_explore 10 hours ago [-]
I have noticed that codebases I have used an AI on seem to perform better in future AI models than ones I have not.
This makes me think AI companies are using chat history to train the next model.
tln 9 hours ago [-]
AI generated code is usually full of comments, human code is usually less commented.
It may be that AI performs better on such code.
TomatoCo 19 hours ago [-]
I've mostly been running LLMs on my own hardware so the phrase "been at it for about 20 minutes" gave me pause. My first instinct was to ask "but on what hardware?" but I suppose one advantage of closed models is that there's a somewhat-consistent cognitive-effort-to-time ratio.
sandcat_ 19 hours ago [-]
Fair, I don’t know if this helps but it was Opus 4.8 (high effort) via Claude Desktop, but through my org’s LLM gateway so hard to compare really. But it’s usually pretty quick.
(don’t tell my boss.)
amelius 12 hours ago [-]
How they will approach it eventually will probably be to let a diffusion style model render the image, then run some algorithm to convert that to svg.
If you stick to the benchpress, it's just "benchmaxxing".
isoprophlex 14 hours ago [-]
I'm about ready for maxxmaxxing, where there's just maximally more of everything all the time
Sharlin 14 hours ago [-]
I’m waiting for the inevitable counterreaction to all this maxxing business, dubbed "minmaxxing"
Paul_Clayton 8 hours ago [-]
In D&D, min-maxing (hyphen and one x) is already a used term for character optimization, minimizing undesirables and maximizing desirables (I guess).
Minimizing peaks is probably not a good strategy generally. Even low peaks may have some benefit ("if you know your problem is in this domain and performance is critical, this tool offers a 3% advantage").
Minimizing troughs may be more attractive. People generally seem to react more strongly to negatives and competitors can devise benchmarks which emphasize one's troughs.
While a universal expert would be convenient and broad knowledge aids some forms of creativity, specialization has substantial advantages.
Trading max performance (primary metric of concern) for some improvement in a secondary metric often makes sense and that could be an interpretation of "minmaxxing", reducing the over-emphasis of a single metric which would otherwise be maximized.
Sharlin 7 hours ago [-]
Yes, it was a subtle referencing to rpg minmaxing, and also the minimax algorithm, beyond the more surface-level idea that "maximalist" trends tend to be followed by "minimalist" ones as a natural counterreaction.
ryukoposting 1 days ago [-]
How useful actually is this? It generates SVGs of pelicans on bicycles, sure, and some of them are (almost) spatially correct. But, none of them look good.
AI image generation suffers from this more generally. You can generate pictures of pelicans, sure. Newer models clearly generate images with more pelican-ness than before. But all of it is still uglier than sin. Drawing things accurately is one thing, making results that someone might actually want to use (without embarrassing themselves) is something else.
derektank 24 hours ago [-]
Just today I asked Claude Fable to take an SVG map and add a circle with a 150 mile radius around a specific city, cutting off the circle at the edge of certain boundaries. It came back a minute later with more or less exactly what I wanted, saving me maybe 5 to 10 minutes of photoshop time. It’s almost certainly not perfect, but I didn’t need it to be perfect, just a legible representation for a group of 20ish people.
somenameforme 14 hours ago [-]
OTOH I had Fable laying out a 3d broadcast room, and as part of the assets it sourced, it grabbed a camera. Makes sense. There were a lot of problems with the scene - backwards facing props and clipping everywhere. But the true coup de grace was a mansized, more like a giant sized, 35mm camera pointing at everything. I was seriously confused for a minute until I realized what had happened.
cmckn 23 hours ago [-]
I used Claude to plan some landscaping, and it gave me SVG diagrams of flower beds showing where to place plants and their approximate mature size. It was pretty useful.
zahlman 1 days ago [-]
Even if the models don't break through any particular "uglier than sin" barrier, with a bit more work, presumably the SVGs could become importable into an editor that would let a human apply taste and discretion. Seems to me like a heck of a head-start.
As for conventional diffusion-model stuff, I happen to think there are some pieces of AI art that still look really good even knowing they're AI.
fiddlerwoaroof 1 days ago [-]
AI can generate a fairly satisfactory SVG for a favicon now (programmer art quality at least).
Kamshak 24 hours ago [-]
Great for charts, illustrations etc, can do much more with SVG than pretty pictures. Good spatial understanding in an SVG can be really nice for model to have
SirHackalot 22 hours ago [-]
I used it to generate some icons for my app recently, I found it a pleasantly useful feature already in its current state. Before this I would’ve had to close the code editor, open something like Illustrator, use something like a tablet (or struggle with the mouse and keyboard), and export the result. For something like a generic icon for a button, or a simple geometric figure, this always felt too tedious. On the other hand, my apps looked visually impressive before the AI and people valued my artistic ability more. So there’s that… Not every software engineer also had an eye for design. With AI generation, they can bridge that gap.
amarant 1 days ago [-]
Vibecoding a SVG based metroidvania as we speak! This is gonna be lit!
joshmoody24 20 hours ago [-]
Would it be useful if the models were actually good at it?
seebeen 13 hours ago [-]
I love this comment.
tempfile 11 hours ago [-]
Ugh, they're just goodmaxxing.
OutOfHere 24 hours ago [-]
The fear is that the SVGmaxxing is limited to "X doing Y". If such 'template maxxing' exists, it will break for other templates, e.g. "X not doing Y", "X and Y doing Z", "X doing Y doing Z", etc.
nl 20 hours ago [-]
I use LLMs for 3D CAD design in OpenSCAD.
There seems to be a very strong correlation between models that are good at SVG and models that are good at 3D CAD.
Anecdote I know, but there does seem to be generalization going on here.
somenameforme 14 hours ago [-]
Which models have you found good for working with 3d stuff?
nl 13 hours ago [-]
ChatGPT 5.5 and Sol 5.6, Fable are good.
I haven't really tested Opus 4.8, but 4.7 wasn't nearly as good as ChatGPT 5.5.
somenameforme 13 hours ago [-]
Interesting! The reason I asked is I've had poor results with Fable and 3d stuff. Its spatial awareness seems poor - doing things like rotating left and then right back, and then pitching nonsensically, just to try to capture a segment of a scene for a verification pass. And its placement often results in clipping, misrotations, and so on.
It could well be that the exact domain matters more than the bigger picture concepts, like 3d. One of the ever fewer reminders that this tech is still just fundamentally a token prediction algorithm.
21asdffdsa12 11 hours ago [-]
The failure seems to be that a scene is inherently hierarchical with a graph with offset, rotation, scale and a name - and this seems to be transferred back poorely fro example during 3d generation.
zarzavat 20 hours ago [-]
If a model can improve at drawing "X doing Y" and that prompt wasn't in the training set then it means it has improved its internal mapping from text-to-spatial-to-text.
sysguest 1 days ago [-]
well that holds IF svgmaxxing is 100% "code-writing-maxxing"
...which.. hmm I dunno if they are same or not
tsimionescu 1 days ago [-]
No, the point is that a general-ish ability to draw good SVGs is a useful ability in itself. People need SVGs for all sorts of purposes, and if AI can generate one for them, that's mostly useful (discussions about art and employment etc notwithstanding).
That said, I think this would correlate relatively little with general programming ability. They're not unrelated, of course, but being able to generate code that paints an accurate + esthetically pleasing image is quite different from generating code that achieves a non-spatial goal.
runarberg 23 hours ago [-]
If people need to generate good-ish SVG why not simply use a specialized model for a much better result and for far cheaper and quicker?
Why do LLMs need to be able to do this as well, but worse, slower and more expensive?
nl 20 hours ago [-]
For example I do SVG diagrams to explain architecture and specific data flows in a multi-tier app.
LLMs do a great job because they understand both code and SVGs well.
The irony of having to post it as a PNG isn't lost on me...
18 hours ago [-]
runarberg 18 hours ago [-]
This is not a strong use case. I could have made this in Microsoft word when I was 13 years old.
Why on earth would you let an LLM do this when it would take you 10 min to do this in Figma or Inkscape or even just Word
nl 18 hours ago [-]
Because it reads the code and generates this in a single pass in 5 minutes.
I haven't read the code.
Why would I do it in a slower, more difficult way for something that's going to be outdated in 2 hours?
simonw 17 hours ago [-]
I'd be very impressed by anyone who can produce that in Figma or Word in ten minutes.
runarberg 17 hours ago [-]
These are rectangles with some fill and strokes, textboxes lines and arrows. If the text has already been written, and the author has a clear idea of the flow all it takes are some a dozen or so points and click, and setting a couple of fills, then copy pastes, and finally some resizes, selects and moves.
This design is trivial. I admit it would be hard to achieve in Word (or at least for me because I don’t know how to make a diagram in Word any more) but Figma and Inkascape are made to do these things, and have optimized UI for that (personally I would have just used mermaid though).
I think you may have lost your faith in human capabilities just a little bit if you think drawing stuff like this takes any time or effort at all. Compared to designing and architecturing the system, drawing the diagram is trivial. Now I know that my parent did neither but it seems like they vibe-coded the whole thing. I’m sure they will end up with a fun little toy from the whole endeavor they can play with for 2 weeks before abandoning. Maybe the author will even feel bad about the carbon footprint of this whole exercise and buy some carbon offsets to make up for it.
nl 15 hours ago [-]
> I’m sure they will end up with a fun little toy from the whole endeavor they can play with for 2 weeks before abandoning.
That's exactly the idea - except it's more like 2 hours before I prototype the next version.
> Maybe the author will even feel bad about the carbon footprint of this whole exercise and buy some carbon offsets to make up for it.
The passive aggressiveness of this is perfectly weighted and admirably phrased.
runarberg 8 hours ago [-]
I am glad you get to have fun while people you will never meet and live in Sweden or Virginia have to suffer higher electricity prices. But we all get to enjoy the warmer climate together. But at least you can enjoy your self having the plagiarizer build you a single use toy that only took the carbon footprint of entire nations to create.
yorwba 22 hours ago [-]
Yeah, if you want to generate an aesthetically pleasing SVG, you'd be better off asking a pixel-based image-generation model for "vector art" and then deconstructing it into an equivalent SVG with something like LayerPeeler. https://layerpeeler.github.io/
exhaze 22 hours ago [-]
Agree directionally - even back in sonnet 3.5 days, I was helping some friends by showing them how to create intermediate representations for SVG building blocks mapping to parametrizable functions that can be used to make interactive SVG-rendered visualizations for various medical needs.
wasabi991011 1 days ago [-]
I don't see why that's true. LLMs don't have to only be good at code-writing.
timClicks 23 hours ago [-]
If they're optimizing for SVG generation, then that's an excellent outcome in my opinion. Vector images shouldn't be "pretty niche".
21asdffdsa12 11 hours ago [-]
And SVG is fantastic to draw easter eggs or hidden messages. Just a insult to the top of the 4 in 404.
knollimar 23 hours ago [-]
Please it's almost my primary bench for drafting understanding.
I'm almost about to post that xkcd 810
rtpg 10 hours ago [-]
> Obviously image gen in svg format is not a particularly hard problem if tackled directly on its own.
I don't understand the truth to this point, and there's a clear difficulty in establishing structure, a thing that you don't need to actually do with raster image gen, right?
Like if you say "please give me 9 circles" you expect circles in the image and not just a bunch of pixel that are vaguely circle-like, right?
simonw 23 hours ago [-]
Gemini have absolutely been SVGmaxxing. They've openly talked about it.
brikym 1 days ago [-]
Then it's great. A year ago I couldn't get any AI to draw a simple company logo in SVG or even convert from raster. No doubt the Pelicans put pressure on the labs to fix the awful SVG situation. Now I can even make a decent Peli in 3D.
exhaze 22 hours ago [-]
Not an expert on this, so won’t speculate regarding what traits would create robustness specifically attributed to SVG visual representation capability, but felt you might find this paper on reasoning models trained on physical world video data becoming better at general reasoning interesting:
Generating images not by pixels but by code and instructions is niche to you?
netsec_burn 1 days ago [-]
Addressed in the article, in case you're curious.
Dylan16807 19 hours ago [-]
Addressed by three inconclusive sentences. When someone goes into deeper detail on the topic, we shouldn't assume they failed to read the article just because they said "what if" instead of mentioning those three sentences.
lukev 1 days ago [-]
Well, it’s mentioned as a limitation of the analysis, very much not ruled out (or in.)
That simonw is causing labs to do extra fine-tuning runs for this seems highly probable :)
Theodores 9 hours ago [-]
As a human SVG-maxxer, I doubt there is the training data out there for true SVG maxxing.
SVG has its neat SVGO tool that optimises SVG code, this is very clever but it won't spot easy wins such as when a group of elements can be mirrored, or an element re-used with a transform. Neither will it use the full smorgasbord of features. SVGO is not AI, however, it does a fab job of taking bloated files from Adobe Illustrator and getting something good to work with.
The bar for SVG is really low, mostly just paths, compressed and not human readable. SVG should be a human readable format, so 'circle radius 10' rather than two hundred points at six decimal places to draw the same circle.
You have to RTFM to do cool things with SVG and there is a lack of appreciation of the format amongst developers and designers.
Eggpants 18 hours ago [-]
I've been pretty happy with LLM svg based data plots I've asked, including log scaled axises and histograms. Definitely a first world problem of course.
mgrandl 13 hours ago [-]
But did they create those plots “by hand”, or did they use one of million open source libraries that create SVG plots? In my experience even SOTA models suck at generating SVG graphics like logos.
kelvinjps10 24 hours ago [-]
SVG are really useful you can create images that don't have the AI look
6thbit 22 hours ago [-]
I think svg is a balanced test because of the level of indirection and the required 'conceptualization' of physical elements then expressed through code.
meowface 14 hours ago [-]
They arguably might be, but I kind of want them to be!
kaliqt 1 days ago [-]
Funnily enough, not that niche, because I have tried many times to do it as part of a wider project.
Melatonic 16 hours ago [-]
Probably the case but not a bad one to "max"
nbardy 20 hours ago [-]
Then they’re legitimately getting better at svg which is a valuable tool.
Simon I hope from this day hence, your bio always includes:
"Simon Willison, among other things, is an advocate for the inclusion of pelican geometry in LLM training datasets."
exhaze 22 hours ago [-]
“Today, the NASDAQ fell by 18% moments after Simon Willison published his latest blog evaluating the Legend 7.4 model that rendered an animated SVG of a pelican on a bicycle and the pelican fell off”
shawabawa3 15 hours ago [-]
When asked for comment an anthropic spokesperson said "it's very unusual for the pelican to fall off. They are built to very rigorous SVG engineering standards"
julianz 19 hours ago [-]
I asked Gemini 3.6 Flash for a pūkeko on a unicycle yesterday and the results were very stylish. The thinking notes were cool too, making sure it got all the high points of what it means to be a pūkeko (distinctive swamp hen found in NZ and Australia for anyone not familiar) and then figuring out how the legs would work with the pedals.
packageman 21 hours ago [-]
I worked with image analytics years ago. In 2024 and cynical about LLM image capability, I tested different models to create "Two female octopi in a bar each having their own drink. One is wearing a yellow hat. The other is wearing a red hat. They have no human features." It went badly I think because the training visual training data associated with the word "female" was overwhelmingly biased in quantity and obviated the 'no human features' instruction even after iterated instructions to remove.
So I tried "Make an image of a Djibouti cab with a camel sitting in the passenger seat. Give the camel no human anatomical features." Still bad.
Two years later the graphics and perspectives of the tools are so much better. But that isn't the big story. What really stands out is the models are no longer adding human female anatomies to camels and octopi. The gates and filtering based on instructions have matured enough so that the pelican/bicycle deductive reasoning puzzle is less problematic. But until an LLM anticipates something like impressionism from Parisian artists rebelling against the rules of the French Academy, continue to reserve a place for human artistry.
tl;dr simonw, your spot-check nailed it just as well as any extensive methodology. LLM's no longer need to cheat this part of the test (better to hack the question than the tool).
Sharlin 13 hours ago [-]
Back in 2024 we didn’t really yet have image models that used bigger language models as their text encoders, so prompt understanding was still rather hit and miss, especially with negations involved.
exhaze 23 hours ago [-]
What if they’re just unsuccessful at it and something about a model grokking how to create a realistic visual representation of a pelican on a bicycle ends being key to the next OOM of capability unhobbling.
And since we have established this silly routine once, I must keep going and ask - yes, you have quite a collection of bona fide pelicans you’ve seen and photographed.
Have you physically seen all the other animals you’ve evaluated as well?
I was born in Soviet Russia, so trust but verify and if still around, maybe there pelican make LLM draw Simon ride bicycle and notice Python code improve.
gopalv 1 days ago [-]
> Catching a lab cheating specifically on my one dumb benchmark would be really funny.
Similar thing happened when TPC came up with SQL benchmarks.
If you're not good at TPC, your engineering team is no good.
If you're good at TPC, then (as a customer) we will actually include you in a bake-off benchmark for our specific problem.
Winning on it is the price of admittance into the game, especially in a crowded market.
But how narrowly you benchmarket matters, you can't just hard-code that specific scenario & not fix anything adjacent while you're at it.
For example when it comes to GPUs, the "Quack3" (sic) benchmark on ATI cards comes to mind.
ryandrake 22 hours ago [-]
Yea, I worked for a competitor to ATI back in the day and we were definitely Quakemaxxing. We were not so unethical as to try to detect the .EXE name so to put the GPU into a "cheating" mode only when that benchmark was run, but we did run Quake 3 pretty much constantly while trying to eke out a few more FPS...
exhaze 22 hours ago [-]
Wasn’t that almost the norm for both new OS drivers and subsequently GPU SoC firmware of ‘gamer cards’ to literally optimize for the latest AAA titles? I feel in that case, the interests aligned - in a world where I had a chance to actually play Crysis, I wouldn’t care why the frame rate was decent, no?
zkmon 11 hours ago [-]
Why is it hard to come up with tests that closely resemble real-world usecases where the models are being used?
simonw 6 hours ago [-]
It's not that hard - there are plenty of those.
They're mostly not very funny though.
philipwhiuk 7 hours ago [-]
Would it be any more applicable?
We see the same problem in database benchmarking.
The good thing about the deliberately-non-real world pelican case is that it gives a general impression of how much the model is improving because it's not likely that it's being specifically targeted at it, rather than a 'real world case' which might have been specifically optimised for.
docheinestages 1 days ago [-]
I think a more fundamental test is SVG art creation in general. Perhaps a pipeline to take any image, caption it, ask the LLM for an SVG, rasterize to an image, and finally either use a deterministic visual similarity check or ask another LLM to be the judge and score how close the SVG is to the original image.
zahlman 1 days ago [-]
Fidelity to the original is definitely not how humans would measure "art" in this context.
docheinestages 1 days ago [-]
True, maybe we can call the generated SVG something else than art.
gilleain 1 days ago [-]
Perhaps also vary the bird? Wikipedia tells me pelicans are in the order _Pelecaniformes_ so shoebills or herons might do.
JadoJodo 21 hours ago [-]
It's a shame you started with pelicans on a bicycle and not snakes on a plane.
usef- 20 hours ago [-]
It's better to be original
Melatonic 16 hours ago [-]
It's all fun and games until that Pelican becomes self aware and asks themselves - why am I riding this bicycle ? I could be creating a Terminator 2 esque dystopian future myself.
And that's how we end up with PelicanSkynet
graemep 9 hours ago [-]
I am not sure being better at pelicans on bicycles would be proof of cheating. Could all the commentary on the web on pelican on bicycle SVGs not influence training without anything being done deliberately?
DonHopkins 10 hours ago [-]
A pelican on a bicycle is like a canary in a coal mine.
And a man without religion is like a fish without a bicycle.
cyberax 1 days ago [-]
> I've been casually spot-checking other animals in other vehicles
Snakes on a plane, weasels on a diesel, spiders on a glider, baboons on a balloon, goats on a boat.
mattertoast 1 days ago [-]
[dead]
mauvehaus 1 days ago [-]
> All 21 pelican-bicycle images, across all seven labs, face right. No other animal/vehicle combination does that.
> However, facing right is common: 60% of all 1,008 images do it. How common depends on the animal and the vehicle, and bicycles are one of the two vehicles where it’s strongest
Of course the pelican on the bicycle is facing right. The drivetrain on a bicycle is on the right side. If you want any representation of a bicycle that shows the drivetrain you're going to show the right side of it if you want to do so without the frame occluding it. It's an excellent bet that their training data reflects this.
As near as I can tell, all of the bicycles are shown facing right, regardless of the direction the animal is facing (GPT 5.6-Terra, Sample 1/3). Also, in every case where the rider has legs (i.e. not the whale) both of the rider's legs are on the right side of the bicycle. This suggests a pretty serious lack of actual understanding of how a bicycle works.
cheesecakegood 21 hours ago [-]
Interestingly, there was an artist a few years back who (for an unrelated project) had almost 400 people across a range of ages draw a bicycle and 75% of those faced left! So this seems to actually go slightly against the human drawing intuition.
On the other hand, I notice that the prompt says to draw a pelican riding a bicycle, implying motion... and since most of us read left to right, I think it's usually natural to draw an object in motion moving left to right as well, which means the bicycle should be facing right. So maybe that specific setup is more natural here.
Either way, for humans bicycles are actually really hard to draw from memory. In fact, I substitute teach, and sometimes as an activity I have my students draw bicycles from memory in 60 seconds. Most make pretty serious errors, usually the frame or chain connections: they can tell it's wrong but still can't draw a more correct one. I use it as an object lesson about the difference between recognition and recall - most students never realize that much of their studying can end up being the former, when tests and life almost always ask for the latter. This helps explain why many students go from "that makes perfect sense" when going over review problems to a total mind blank only a few minutes later (especially in math!).
ctidd 21 hours ago [-]
It’s a very strong convention in bicycle photography. The drive side is considered nicer looking, it’s where you get to show off the components, etc.
Independent of what humans may do with the same prompt, training datasets will be full of traditional bike photos, which will all be drive side.
It’s also one of the easiest yellow flags to look for in used bike listings (not low-end, but anything enthusiast level). If a bike is photographed on the wrong side, there’s a non-negligible chance it’s stolen, because the seller is presumed to know better if they’re an enthusiast themself.
sheept 17 hours ago [-]
That's interesting. I think the bikes-facing-left bias probably comes from how humans use bikes: the kickstand is on the left, so people likely hold and approach bikes from its left.
Looking online, the kickstand is apparently on the left to avoid the gears. Based on the other comment about bike photography, it's interesting the same design choice makes humans and cameras/LLMs see bikes from different sides.
OliverGuy 11 hours ago [-]
If you are a bike nerd and like taking photos of your bike(s) you'll almost always try and have it drive train towards the camera, which would be front on the right side. Possibly this is where is this coming from? Generally also true for retail listings too
Sharlin 13 hours ago [-]
I think in general if you ask people to draw vehicles or animals (including humans) from the side, there will be this left-facing bias. Perhaps this is true only in left-to-right writing cultures, hard to say.
amelius 12 hours ago [-]
It's the other way around: the kickstand is on the left because people jump on their bikes by swinging their right leg. Therefore looking at a bike from the other side feels awkward because you don't feel the same urge to jump on it and ride it.
userbinator 20 hours ago [-]
I wonder how the skill at drawing a bicycle compares between the general population, and those who own or ride one regularly.
Tepix 14 hours ago [-]
Perhaps japanese LLMs have a reason to do it differently:
To add to to this looking at the #baaw (bikes against a wall) tag on insta It was over 200 pics before I found a pic of a bike that wasn't facing left-to-right
> This suggests a pretty serious lack of actual understanding of how a bicycle works
an... understanding? there is no understanding here at all
stusmall 1 days ago [-]
I'm glad someone ran the numbers on this. Every single Simon Willison post of an SVG is followed with someone dismissing it saying "I'm sure they train on it by now." This is despite a good blog post with sound logic on how easy that is to catch. [1] Glad to see someone took the time for a quantitative analysis of dumb little animals riding dumb little bikes.
I agree with the conclusion and am happy to see this blog post, but this killed a bit of credibility for me:
> Using a single LLM judge for scoring. Every score here comes from one model, GPT-5.6 Luna, looking at one image at a time. I didn’t do much alignment and didn’t check how often it agrees with itself on a re-run.
Having used a similar setup (with previous gen LLMs) to evaluate the 3D models that my product[0] generates, it turned out there was no correlation at all. LLM judgments were very much random and I assume judging SVGs is not that far from judging 3D models. I guess I have to re-test this with current gen.
I don't think this small amount generalization to other animals and vehicles is strong evidence they haven't trained on this, either directly or more generally.
conception 24 hours ago [-]
Honest question how could they possibly train on this as there are no good SVG pelicans to train off right? So they’re just training off a bunch of bad ones which should lead to just bad pelicans, but the pelicans are getting better.
unholiness 21 hours ago [-]
Training on generating SVGs directly at all is already fairly niche. Generating full scenes with a cartoony character is even nicher. But there's plenty of non-pelican cartoony SVG content out there (created, not written, by humans with vector design tools), and more importantly, plenty of vision models to give feedback on the output (just raster as a png). You could easily hill climb this niche skill, if you cared.
alexthehurst 23 hours ago [-]
It’s not hard for a visual model to score the quality of that output though, which would be a pretty good fitness function.
retinaros 9 hours ago [-]
I did that. This works to some extent but why bother. Sft brings you 99,9% there already. Rl helps more with syntax errors. Svg is code
raincole 20 hours ago [-]
They can easily afford 1000 human made pelican svg files if they want. I think you underestimate how much resources SotA AI companies have.
(I'm not saying that they did that. I'm just saying they can.)
dd8601fn 6 hours ago [-]
If I were running one of these mega AI companies I would set aside a tiny team to produce and release a pelican model, explicitly trained for this.
The very best svg pelican on a bile generation model. Just for laughs.
elicash 1 days ago [-]
There's that version of the argument version, but there's also the softer version: that there used to be no training material of illustrated pelicans on bicycles, but now you have actual artistically talented individuals drawing it and that could improve the performance even though the AI labs are sucking it up no differently than everything else.
This post proves that hasn't happened yet, either. Although maybe the bad results posted online are being trained on and that explains the UNDER performance.
elliotto 21 hours ago [-]
Bike nerd + AI nerd here. The author's observation that all bicycle images face right almost certainly has to do with the convention to photograph a bicycle from the right. From the right, you see the drivetrain - this is good for aesthetics, but also for marketing - the drivetrain is branded and labelled and a buyer will want to know what model it is.
There is a bunch of guidance online on how to photograph bikes, and every sales image of a bike will be from the right. You can anecdotally observe this by google imaging 'bicycle for sale'.
SyneRyder 23 hours ago [-]
Huh. They're not "Pelicanmaxxing"... they're Ottermaxxing.
Take a look at the GLM 5.2 and Deepseek V4 "animal on a plane" examples. In every case, the animal is standing on top of the plane, a clear misunderstanding of the concept of "animal on a plane"... with the exception of the Otter. The otters are sitting in a seat on the plane, looking out the window.
That's Ethan Mollick's "Otter On A Plane Using WiFi" image benchmark.
(Sometimes the Racoon is sitting inside the plane as well, but the racoon is a common backup benchmark. I'm surprised it wasn't also holding a sign saying that it loves trash.)
Also, Grok seemed to really really enjoy "whale on a plane" in that second round, and kudos to GPT Terra for deciding after 3 rounds that the user was terrible at spelling and generated "Antelope On A Plain".
EDIT: I promise I'm a human, but I did just notice my "that's not x... that's y" construction at the start. I am rather Claudepilled — my apologies.
twodave 6 hours ago [-]
To be fair, when I see that pattern after a relevant question, it isn't much of an AI tell. Consider this exchange: A:"Are you hungry?" B:"I'm not hungry, I'm thirsty."
When LLMs use the pattern, they are often setting up a straw man and then knocking it over.
Otterly99 9 hours ago [-]
Thank you for the share, I'm glad AI labs are getting their priorities straight.
a3w 9 hours ago [-]
But it does come with an Otter refusal to other tasks given.
bnfcl 1 days ago [-]
This is funny, I actually did a similar experiment just yesterday.
Looking for evidence of the same, but with another twist: checking if the models would choose to create a pelican on a bicycle, if no specific bird or method of transportation was specified.
I find your analysis much more convincing than TFA, since it doesn't require a subjective evaluation and is more robust to animal/transport complexity.
bnfcl 1 days ago [-]
Thanks! Because I think that models are becoming better at creating SVGs in general. If you look at Claude Fable 5 and Kimi K3 for example.
Simple as they are, there are some really aesthetically pleasing penguins on skateboards in there, including from less capable models. (In fact, I would say the Opus series got progressively worse at it over time.)
bnfcl 10 hours ago [-]
Agreed!
Wowfunhappy 1 days ago [-]
> The more plausible story is SVGmaxxing
Exactly--and you have to ask yourself at this point what "maxxing" really means, since "get better at drawing SVGs" is a useful skill.
beering 1 days ago [-]
Really awful how the AI labs are skillmaxxing /s
Pelicans aside, we need to remember that benchmarks are the only good quantitative way we have of comparing models. If someone has complaints about “benchmaxxing”, please ask them to contribute a better benchmark! It is valuable work and very appreciated.
dbt00 1 days ago [-]
It's a problem because of Goodhart's law.
If you train towards the test, you aren't necessarily improving overall fitness, but you are destroying the value of that test over time because you're decreasing its correlation with overall fitness.
Dylan16807 19 hours ago [-]
Avoiding "benchmaxxing" helps keep benchmarks be a good quantitative way to compare models! It's hard to come up with new benchmarks, and it's a waste of everyone's time if we have to churn through them constantly. Effort to keep AI labs working on general skills and not "teaching to the test" is effort well spent.
Wowfunhappy 1 days ago [-]
> If someone has complaints about “benchmaxxing”, please ask them to contribute a better benchmark!
I don't think I'd go that far!
When someone says a model has been benchmaxxed, what they really mean is that it performs better in benchmarks compared to their real world experience. That's a real thing, I've certainly experienced it with some models.
...my take is that some things in life just resist quantitative measurements. Who is the best job candidate? What is the best programming language? Add AI models to the pile.
waterproof 23 hours ago [-]
I recently had the following conversation with Claude:
Me: how many P's are in the following text? [Pasted text]
Claude: There are 14 P's, all lowercase (no capital P's)
Me: how many in "strawberry"?
Claude: there are 3 R's in the word "strawberry".
ickyforce 23 hours ago [-]
I tried this with Gemini 3.6 Flash
Me:how many P's are in the following text? [Pasted text with 10 P's]
Gemini: There are 9 "P"s (1 uppercase P and 8 lowercase ps) in the provided text. [List of words except the one missed]
What I find interesting, though, is how the "feature" isn't really exciting anymore, or because it creates SVGs that aren't "perfect" that we maybe feel underwhelmed sometimes (at least I do).
However thinking about it...if someone asked me to manually create an SVG, or hell even draw a quick doodle on a bit of paper of the same, I'd still probably be much slower than an LLM and potentially end up sketching less accurate anatomy than the machine.
I think the general "organic task" stuff has been mostly sorted out, but in personal and professional experiences using AI to try to _do_ something, I've found less so recently problems with hallucinations and moreso problems with attention.
For example GPT5.6 still has issues where if I provide it with a list of documents and then ask it to raise questions from that information. Then provide it with additional documents that answer some of those questions and ask it to summarise which outstanding questions there are again, it still asks questions that have become irrelevant with the additional documents - but when this is pointed out it knows exactly what to do and produces the correct list of outstanding questions.
I'm sure frontier models are doing all sorts of crazy stuff with attention already, but it seems to me like we almost need some hierarchical attention mechanism like KVL (with Level added) so that it's aware not only of semantic connections between tokens in the context but also of where there are gaps, missing links to assist the model in becoming aware of its own attention span (I guess).
apwheele 1 days ago [-]
So this is not my experience at all for asking about simple SVG icons for web-pages. Here is one of the examples I have tried for in the past, make a simple cartoon SVG knife for a map icon for a crime map.
Can see the images for ChatGPT/Claude (Sonnet 5), and Gemini are all quite bad.
Jagged edge of LLMs. How do you explain being able to generate very complicated shapes in the Pelican example but cannot make a much simpler icon without just alluding to it is in the training data?
Topfi 24 hours ago [-]
Thanks for sharing another solid data point. I fear you won't get an answer from my experience [0]. Unfortunately, the blog post decided to forgo the very models that I found to be the worst offenders:
> Here is what Qwen3.6-35B-A3B via Openrouter provided for a sloth riding a skateboard: https://imgur.com/a/Dy8fvR5
> Like Grok 4 Fasts attempt at a mushroom in a rowboat, it is barely recognisable as anything despite both Qwen3.6-35B-A3B and Grok 4 Fast having no issue with more popular (i.e. benchmarked) examples. [...]
> And here is Opus 4.7 [which simonw claimed to provide a worse pelican vs Qwen], again via Openrouter: https://imgur.com/a/Qus1Enf
Anyone who hasn't witnessed such deltas either hasn't looked at enough examples, a sufficient variety of models, or both. And they are, unfortunately, not limited to "SVGMaxxing", but a wide range of evals.
If you look at the raster image ChatGPT generated, that is fine. It is just this example (and other simple SVG icons I have asked for) result in pretty bad SVGs. It just makes me highly suspicious that the LLMs are learning shape primitives and extrapolating to new shapes, vs just having a big dictionary of prior examples and stitching them together.
robocat 23 hours ago [-]
Don't judge the dog's technique. The miracle is that it's dancing.
I would guess most programmers struggle to create SVG icons - I don't find it easy. The average person even more so.
Are we best to assume an LLM is a blind programmer? Any HN comments from blind programmers tasked with creating SVG icons? Only relevant comment I could find from ctoth was about accessibility: https://news.ycombinator.com/item?id=7185771
Projecting how you think onto what the LLM is doing or should be doing, is probably a mistake on your part.
I recently spent a little time trying to understand exactly why Gemini was misexplaining $X. $X = {why the generated LaTeX visually didn't match what it was asked to do}. It was enlightening.
throwaway6s1df 22 hours ago [-]
I don’t know how anyone with a neutral view can confidently take this:
> Direction: All 21 pelican-bicycle images, across all seven labs, face right. No other animal/vehicle combination does that.
> However, facing right is common: 60% of all 1,008 images do it…
and the tables in “Evidence #5” to be anything but evidence the models have likely trained on pelican on bicycle data more than others.
The data clearly shows:
- 100% pelican on bicycle facing right
- significant skew to the right for bicycle-like vehicles
- significant preference for right facing for birds
Averaging those extreme results to “60%” to make it sound like it’s pretty fair because it’s close to “50%” isn’t statistically sound.
The methodology is generally unsound. There is no actual scoring with a well defined rubric, it’s just vibed with a single model (GPT 5.6 Luna).
The “not better at drawing” evidence are equally hard to take seriously when there is no clear, non-subjective indication of what better or worse is.
dllu 1 days ago [-]
I feel like getting LLMs to spit out an SVG is akin to getting a human artist to draw something by just reciting a list of coordinates. It's insanely hard and unnatural.
Image generation models nowadays can easily generate a photorealistic pelican riding a bicycle, where the bicycle has perfect structure. But it is, of course, only a raster image.
It seems that we're missing a kind of step to decompose an image into a list of instructions (say, SVG paths, or even brush strokes with a real brush) to reproduce it properly. Doing so would probably need a true understanding of the structure of the scene, which is something that AI still struggles with to this day.
staticshock 1 days ago [-]
The pelican on a bicycle test is specifically about generating an SVG, fyi, not a raster.
dllu 1 days ago [-]
I know. I'm just thinking about how to make AI create SVGs better... in theory, a sufficiently smart AI could "generate an image in its head", think about it, and then output the SVG paths to produce said image. Intuitively that would be somewhat closer to how human artists convert artistic visions into a sequence of arm movements while holding a brush (obviously, humans don't hold a fully formed, photorealistic image in the head while drawing, but rather vague concepts, but still).
cherioo 1 days ago [-]
I don’t quite agree. Good human artist can visualize in their mind how to draw a picture, i think. Which i think is no different than LLM doing SVG drawing in their “head”. Anthropic’s recent post call this head-space “workspace”.
It just might feel foreign to human who does not have a SVG trained head-space.
dllu 24 hours ago [-]
Sorry, my comment was confusingly worded, but I am precisely thinking about the head-space thing you are talking about. My point is: the fact that image generation models can generate a perfect pelican riding on a topologically correct and highly detailed bike frame [1], suggests that AI does have the capability to correctly understand it internally. But something is disconnected when we just let an LLM directly output the SVG, and all the nuanced understanding is lost.
As such, modern LLMs still kind of suck at generating pelican bike SVGs with obvious errors:
* some omitted the bottom of the diamond which connects from the pedals to the rear wheel
* some added an extra connection from the pedals to the front wheel, making it impossible to steer
* none could align the head tube with the fork
* none added a correct offset to the fork
* none could generate the chain properly in a way that attaches to the two sprockets correctly
... whereas these errors do not appear in the raster image. (To be fair, the raster image has other weirdnesses, like the bird having arms and only one leg)
If we could harness the sort of internal thinking that must have happened when it generated the highly consistent raster image, but make it output SVG instead, we'd get much better SVGs. So this would be indeed "visualize in their mind how to draw a picture" before outputting the SVG.
Image models that support text output like Image2, or general text models that can read images like Claude can vectorize raster images. But they aren't very good at it, doing it manually in Inkscape still produces better quality even when done by non-artists.
jboss10 12 hours ago [-]
Some of these are quite nice(I like gemma 3.5 flash's work)
Nice article! Thanks for experimenting and writing it.
I'm curious if some of the animals / vehicules might force the models to use more tokens than others, and I could not find the token counts in the shared data, is it possible to publish it please? :)
2001zhaozhao 19 hours ago [-]
I feel like the simplest Pelicanmaxxing method is just to teach the model that whenever a user asks for a svg illustration of something with no other clarification, it should default to making it as detailed and pretty as possible. This would make every single svg from that model look better and not just the pelican on a bicycle
ianberdin 15 hours ago [-]
We have a little bit different conclusions.
Same pelican grid + MacBook Pro in 3D. Also with end cost.
There's every chance here I'm just being annoying pedant, but generating a bunch of SVGs of animals on vehicles doesn't mean that it's good at generating SVGs in general, just SVGs of animals on vehicles.
On the flip side, GPT 5.6 Sol did a pretty convincing render of a burglar eating salami.
I'd be curious to see how the other models on random things that are completely tangential to pelicans or bicycles.
richardw 23 hours ago [-]
And a whole universe of random tests got baked into the AI’s training data. Websites of antelopes driving trains and hammerhead sharks swinging in a tyre swing were created. It was a short while until AI became so focused on animals that it gave up competing with developers. Life became sane again.
anshumankmr 16 hours ago [-]
For an MVP I am building I asked to make a brain SVG and every attempt it has been doing it has been hillariously wrong and I had told Opus4.8/Fable to pick some SVG it could find online and it went ahead and still used its own thing that turned not too good.
(full disclosure not particularly that good at front end stuff so relying heavily on Claude and Codex for it)
ertgbnm 1 days ago [-]
I've had the feeling that labs aren't pelicanmaxxing specifically but that they do have some sort of RL environment for SVGs that they are letting the AIs overcook in. Specifically I'm thinking of the gemini 3.1 pro annoucnement that seemed to have a huge leap in animated SVG performance but not much else impressive about it.
So they aren't pelicanmaxxing but they are benchmaxxing in a way. The benefit of the pelican was originally that uplift on the pelican signaled an overall uplift on intelligence. I don't believe that is the case anymore and it is just another jagged edge of model intelligence.
ListeningPie 12 hours ago [-]
The blog has it's own comment section under the article that's completely blank and yet clearly there is a lot of interest with 197 comments. Having an empty comment sections sends the wrong signal.
bluealienpie 1 days ago [-]
AI rating AI? Am I missing something.
Nnnes 23 hours ago [-]
I'm surprised (but not really) that you're the only comment I see even mentioning it. The ratings may be even lower quality than the SVGs.
Obviously they're all a bit cartoon-y, what else do you expect from SVGs. But I'm not convinced you could find a single human on Earth over the age of 4 who would seriously give the vehicle in GPT/1/whale/plane a 5/5.
Browse through the options a bit and the rest is not that much better. Grok/2/cat/plane, one of the more accurate planes, got a 2/5. For the most part, vehicles entirely missing do get a 1/5, except for whatever it is in Gemini/1/heron/plane scoring 4. Animals inside planes get completely random vehicle scores I guess.
The cats are all orange, except for a few of the skateboard cats that are black. I'm sure there's nothing to read into there...
Well, I've convinced myself that the next effective test of multimodal models will be whether their judgments of LLM-generated SVG airplanes are anywhere close to reasonable.
zahlman 1 days ago [-]
That seems to be how we signal "objectivity" nowadays.
recursive 24 hours ago [-]
Missing something? You're missing the boat! Have some AI-prepared koolaid before you get left behind.
nostrademons 1 days ago [-]
It's really refreshing to see someone publish a null result.
I fear there this reveals something else - not about the nature of models but of our community. The intro of the article seemed to affirm the importance of HN as a tastemaker, potentially influencing decision-makers on the scale of billions or trillions. For the 15 or so years I’ve been hanging around here, that only feels like mild hyperbole… a good LaunchHN can reshape the future, right?
It made me realize that for this time around, we’re not at the center anymore. The future of LLMs is the stuff of nations and AIG is what the labs actually care about. They aren’t pelicanmaxxing just as much as they really aren’t revenue/margin/marketing maxing. They just want our attention and ideas so they can show growth and acquire FLOPS.
The apparent fact that they aren’t catering to this community (who frankly decides what goes and stays in prod) leaves me feeling a bit defeated somehow. And also awesome?! Like there is a culture here that runs deeper than any technology and it has fought hard to maintain its identity. Props to dang and all for keeping the astroturfing so imperceptible that I can say this.
In any case, a great little piece of citizen-science dcastm. Lmk if there is a way I can chip in towards token costs.
munk-a 1 days ago [-]
It's a method to grade LLM output - as such it's something that will receive focus in correcting for. As soon as people who have a say in where funding is going noticed it as a metric the labs started caring about their performance in it. In the best case the labs are focusing on improving SVG capabilities in general and optimizing Pelican production as part of that initiative - but now that it's a known measure it is no longer reliable.
pjmlp 15 hours ago [-]
I would expect so.
In the last century, as spec tests for C and C++ compilers, databases, Java application servers became trendy, all vendors were optimising for great articles on the respective technical magazines.
11 hours ago [-]
rldjbpin 11 hours ago [-]
the pelican bike combo is truly an HN phenomenon, and i think the hypothesis carry a lot of weight about the assumption of the mindshare this website truly has in the industry.
very nice approach to test it and might be a nice way to "grid search" evals in other use cases perhaps.
oasisbob 23 hours ago [-]
As a unicyclist, all the sideways-riding caught my eye for being especially silly.
However, I'm very surprised that most of the models make the same sideways mistake with only some of the animals, and they do it consistently.
With most of the models, cat, raccoons, and otters are almost always riding sideways. Why is that?
pbhjpbhj 12 hours ago [-]
Pose bias? Most pictures of animals like penguins are face on (I think they're mainly at 45°, and birds are general profile; so perhaps it's a more generalised bias)? Or perhaps the 'recognisable' is correlated with face-on?
inigyou 8 hours ago [-]
Why do all of the images just say "failed"?
robviren 1 days ago [-]
Why let your dreams be dreams? This is a perfect example of following a hypothesis. I love when people dive into an esoteric subject and just go full swing. Reminds me a CGP Grey and the name Tiffany. Sometimes you just need to know.
oaxacaoaxaca 24 hours ago [-]
Hilarious question. Imagine someone woke up from a 7 year coma and read this title lol
antonyragleap 15 hours ago [-]
The interesting question is whether this generalizes beyond pelicans to benchmarks the model hasn't seen before.
i'm puzzled by the choice to have an llm judge the images.
Copenjin 16 hours ago [-]
I'm also puzzled by the fact that I expected EVERYONE to point this out but yours is the only comment about it.
Even if using LLM is the only way to do things at scale it does not mean that it's always the right tool.
davidkunz 16 hours ago [-]
Maybe they're animalsonvehiclesmaxxing.
jonatron 1 days ago [-]
OK, so we've done animal_vehicle, how about new SVG ideas each time? I just tried "make an SVG of a man sitting in a chair at a computer behind a desk" which gives more interesting results than the animalVehicle test.
ninju 1 days ago [-]
There probably good set of images of that description already so it does exercise the inference capability of the model
luciana1u 10 hours ago [-]
the benchmark was supposed to measure if models can solve novel problems and instead it became a benchmark for how fast labs can solve the benchmark
BeetleB 1 days ago [-]
Oh great! You've now made it a lot easier for LLMs to train on this dataset!
Your next iteration will need different animals and different transportation options. You'll run out after a few iterations.
anuramat 1 days ago [-]
"benchmaxxing by generalizing" is not really benchmaxxing
That’s an impressive image, but what a mistake it was to click the second link (on mobile without an ad blocker). I wouldn’t send it to anyone I respect ...
johndough 23 hours ago [-]
Thanks for pointing that out. I haven't noticed any adds in years with Firefox and Ublock Origin extension.
I'll look for a better image host in the future. I guess the economic incentives makes them all turn bad after a while.
ACCount37 1 days ago [-]
The name is "Recraft V4", and from looking it up: yeah, it sure seems like whatever black magic they use for SVG generation kicks ass.
johndough 24 hours ago [-]
Oops, autocorrect. Sorry about that.
Rooster61 1 days ago [-]
I find it humorous that the animal + plane combo appears to be such an outlier. I assume this is due to the models assuming the user mean plain and misspelled it in the prompt.
ramses0 1 days ago [-]
I think it's actually due to "pelican on a plane" isn't the same as "pelican on an airplane" (Sonnet5 @ Flamingo x Plane), some consistent and warranted semantic/linguistic confusion!
flsw 1 days ago [-]
I noticed this happens especially with herons. My guess is it's because the model links "heron" to Heron's formula and the Cartesian plane
NitpickLawyer 1 days ago [-]
GLM has 2 combos of "on a plane" literally sitting inside a plane, with a window and a bit of wing showing. That's funny.
zahlman 1 days ago [-]
... Is that not how it should be interpreted?
gpjanik 12 hours ago [-]
TLDR: the experiment asks for in-distribution responses and gets those.
The right answer here is to ask a LLM to create a scene similar in quality to those, but completely out of distribution.
I asked GPT 5.6 Sol to give me a pelican playing football on San Siro while smoking a cigarette, in AC Milan's t-shirt. While this sounds like higher complexity of a problem, the generations from current models often include additional details like scene composition, scarf, etc., I don't ask for, so I wanted to see what here is memorization vs. composition skill.
"write svg code of a fish playing football on san siro in ac milan's t shirt, with raybans on and a cigarette."
Try that on GPT 5.6 Sol, Fable, or whatever other model. It's chaos.
ionwake 10 hours ago [-]
Absolutely fantastic top tier post
Copenjin 16 hours ago [-]
Using an LLM to judge drawing giving a rating? Not the best idea.
PeterStuer 14 hours ago [-]
Any benchmark gaining some, even a little, traction before a model release date should at this point be considered tainted. Create your own, never publish it or write about it in any detail.
MarcellusDrum 10 hours ago [-]
The article clearly shows that this benchmark is not tainted.
tomas789 1 days ago [-]
Having an objective score is quite difficult. Maybe it would be better to do a pairwise comparison and calculate ELO?
NitpickLawyer 1 days ago [-]
Just click through the models. At a glance (and highly subjective) I don't see anything jumping out as oom worse than anything else. I only noticed a model placing the animal inside a plane (with seat and small window) but other than that, they all seem similar inside each model to me.
javier123454321 1 days ago [-]
If you want to, go ahead, but it seems to me the author already exceeded the energy expenditure that this question warranted.
stri8ted 1 days ago [-]
You seem to assume training on pelican would not result in improved performance on other similar tasks. Why?
altcognito 1 days ago [-]
He didn't. That's why the article exists. You have to do the science to see if it does.
He was asking the question - do we see gains across other tasks? The underlying question was: Is the additional attention given to this specific task creating a false impression of progress?
HarHarVeryFunny 1 days ago [-]
Either you've memorized the outline (or detailed component shapes) of, say, a horse, or you haven't. Memorizing the outline of a pelican isn't going to help you with the horse.
You could train a model to do something a bit different like a pencil sketch, or vector graphic sketch, of something given a photo of it, and expect that to be a generalized skill, but if you are asking the model to do it "from memory" then memorizing a pelican is no substitute for not having memorized a horse.
6thbit 22 hours ago [-]
What would be an alternative format or process with a similar effort to drawing SVGs?
bvrmn 15 hours ago [-]
it's quite interesting scoring and conclusions. For me Gemini renders are definitely top outliers for all combinations.
comrade1234 1 days ago [-]
Hilarious. Could you imagine being a programmer at an AI company and this is your assigned task?
40four 15 hours ago [-]
Okay, first off, without honestly reading the whole article (I tried but I just don’t have the patience), a quick red flag is the final analysis is only through Fable? Isn’t that inherently going to introduce unwanted bias?
Whatever. Doesn’t really matter much. My next thought is, I get that requiring an SVG is adding an extra layer of complexity as far as the art goes, but why is nobody talking about that the actual art is absolute trash?
I get it. It’s basically a meme at this point and it’s a fun game to play with the models. But my thought is it should be illuminating to anyone who is an artist that LLMs are still a long way off from taking your job :)
pbronez 8 hours ago [-]
Nice use of regression analysis to understand your experimental dataset
> But again, some combinations might be just harder to draw than others.
> To account for that, I fit a fixed-effects regression on all 1,008 images: score ~ lab + animal × vehicle, plus per-lab interaction terms for pelican, bicycle, and the pelican-bicycle cell, with robust standard errors. The animal × vehicle terms absorb the inherent difficulty of all 48 combinations. The interactions measure each lab’s benchmark-specific boost relative to the average lab, with confidence intervals.
RobRivera 24 hours ago [-]
Chasing metrics
Chasing dragons
Tomato, tomato
andy99 1 days ago [-]
If an AI researcher was going to pelicanmaxx, they would almost certainly apply the augmentations mentioned in the article during training, e.g. randomly selecting animals and conveyances. You’d want a model that generalizes well, just sfting in that specific prompt would be pretty bush league for a frontier lab.
I don’t have any reason to believe they are gaming the benchmark, just saying. I do find the idea of a data labeller having to generate thousands of svgs of different animals on different modes of transportation quite funny though.
cute_boi 1 days ago [-]
At this point, I think there are so many pelican images in the pretraining data that drawing a pelican no longer makes sense as a model evaluation task.
IshKebab 23 hours ago [-]
Thanks for not using AI to write this. So much more pleasant to read.
busymom0 1 days ago [-]
How does attempt 2 by Llama 4 Maverick look like a bald eagle??
ck2 1 days ago [-]
I am not sure if this is how it works but let's say there was a reddit thread talking about the pelican benchmark and in it someone posts mockup examples of what an ideal result would look like
aren't some LLM going to digest that thread at some point and indirectly learn from it?
basically my point is originally this was a good benchmark because it was an absurd never-seen-before thing, but now that it is in content, some models are going to get a benefit in education?
you'd need the "AI" equivalent of an old-school "google whack", something with no previous results
They ingest so much data that a couple of reddit threads do not move the needle.
It is the reinforcement learning that produces more tangible results with less data, but it is something that the AI labs specifically selects and it is not picked up unknowingly
j45 1 days ago [-]
The models definitely seem to pay attention to the tests.
Since the tests can be generally gamed with directing descriptions at it non-deterministically, there's a greater chance the questions solution can be found.
Of course, hopefully the models are instead adding patterns and types of questions as well and it makes the models more capable, but it may be limited in how it transfers to other types of questions in breadth or depth.
> Every single one is a pelican, on a bicycle, first try. When every student gets an A, the exam has stopped grading.
Numerous pelicans and their bikes are clearly horribly malformed. In fact none of the bike frames are correct. Fable and Opus come close, but the top of the diamond is disconnected in Fable's case and the head tube is misaligned with the front fork in Opus's case.
And of course, as the parent post shows, labs don't actually seem to be training on the pelican bike case.
ErrantX 1 days ago [-]
Agreed. And more; the Macbooks are pretty much the same - some are god approximations, some are terrible, all of them are recognisably a MacBook. And if you start using it they can train on it.
The problem isn't the test, its that is a public test.
Simon has previously said he has a list of secret prompts (at least one of which he "burned" as a demonstration a while ago). That's what makes it a good test - his commentary on the public test is something of a proxy for non-public tests. This makes it a good benchmark.
dcchambers 1 days ago [-]
It's incredible that each model has it's own style that remains relatively consistent throughout all of the different generated examples.
"Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes."
Or the more pop layman version
"When a measure becomes a metric/KPI, it ceases to be a good measure."
Story time, I live in Argentina, and we don't have Big Macs, the main Mc Donald's brand, here, because during the CFK presidency, one of her tactics was to Goodhart economic metrics. Even the informal obscure ones like the [Big Mac Index](https://en.wikipedia.org/wiki/Big_Mac_Index), I don't know the precise details, but the Big Mac ended up being a very cheap item, like 2 or 3 times cheaper than actual menu items, but it was never on the advertised menu, and it also ended up being very small compared to the other burgers, so it wasn't even like a hack, a shrinkflation type of deal.
But hey, anyone who read the Big Mac Index table would never find Argentina at the bottom of that list along with a couple of other countries with bad brands, so the ploy worked. And now we live with the aftershock, the brand never really turned around, other brands with ridiculous names took over it like the McTasty, which makes me sound like that skit from Tarantino's Pulp Fiction.
zahlman 1 days ago [-]
How did the president manage to influence McDonalds' local business decisions? And how did that lead to McDonald's pulling out of the country?
TZubiri 21 hours ago [-]
McDonald's complies with local laws and operates with local juristic entities. While McDonald's US owns McDonalds Argentina (Arcos Dorados Argentina SA), and there's some level of control that occurs in the US "Global" headquarters, some that occurs at the National HQ level, and some that occurs in the individual location. Obviously a president would be able to influence 2 of those levels, it's not necessary for her to influence the Global executives.
To be clear McDonald's didn't pull out of the country, like in many other countries, it adapted to the local market. Same way as in india it serves non meat variants owing to the high vegetarian population, consider that mcdonald's operates in Venezuela and China, and operated in Russia up until the Russia-Ukraine war. It takes a lot for MCD to pull out of a country.
As for the exact mechanisms of presidential price control on MCD Argentina, I don't have the specifics here, but I can get pretty close. There have been 2 broad mechanisms to exhert price control during CFK's 8 years of presidency and during his Husband's 4, let's call them official and unofficial.
The official mechanisms would be passing laws or presidential decrees (DNU) that don't go through congress, as well as influencing regulation of executive ministries/departments like central bank norms, exchange rates. Some measures like 'precios cuidados' were placed for this very specific purpose, it's very possible Big Macs were under this specific scheme, I do not recall, I was a bit young.
The unofficial methods would be less public, but well known, in the food industry it was especially common, it's well established that food are one of the first and most common targets of price controls. I have heard direct accounts from family members about high ranking government officials setting up meetings with producers in food markets to give orders of lowering prices, one going as far as brandishing a firearm by placing it on a table while discussing the subject.
Writing this out loud I realize that this explains the later over-correction of argentina that allowed freemarket capitalist libertarianism to rise. The optimal strategy in democracy seems to be polarization, so both extremisms seem to symbiotically feed off each other. Not a country of moderateness this one.
edifierxuhao 11 hours ago [-]
[flagged]
tim_tihub 9 hours ago [-]
[dead]
Ilya85 1 days ago [-]
[flagged]
Ilya85 1 days ago [-]
[flagged]
TheSpacerr 13 hours ago [-]
[dead]
sbseitz 1 days ago [-]
I wish I could downvote this for Pelicanmaxxing lmao.
influx 1 days ago [-]
Would you prefer the term Pelicangate?
sbseitz 22 hours ago [-]
Yass!
theandrewbailey 1 days ago [-]
We're going to keep maxxmaxxing forever.
sbseitz 22 hours ago [-]
I believe you are correct!
weregiraffe 16 hours ago [-]
Forevermaxxing
andrewstuart 1 days ago [-]
The pelican prompt is ridiculous.
Test the LLLM against things you want it to do.
Asking questions that are absurd is like interviewing developers and asking absurd questions on the grounds that it tests creative and critical thinking.
Remember these Microsoft interview questions designed to identify the best developers?
"If you could eliminate one U.S. state, which one would it be?"
"How would you move Mount Fuji?"
Absurd interview questions have an air of legitimacy due to the quasi sophisticated justifications put forward for why they are good tests.
Absurd interview questions are not good tests of people or LLMs.
Relevant questions are good tests.
user- 1 days ago [-]
The whole point of "AI" is arbitrary task completion. Why isn't a SVG drawing relevant for that?
zahlman 1 days ago [-]
> Asking questions that are absurd is like interviewing developers and asking absurd questions on the grounds that it tests creative and critical thinking.
Yes; what's wrong with that?
Do you suppose that it doesn't test those qualities?
aakresearch 13 hours ago [-]
I, personally, think it does not. For the absurd task which has no testable solution, there is no grounded basis to judge creativity or critical thinking behind an answer. The only things that remain are "vibes" or "hilarity". Well, if these are what you want to test for, I do not disagree. But it isn't something I'd be interested in.
ErrantX 1 days ago [-]
As I understand it; the point is to ask for an SVG which would demonstrate a conceptual understanding of what is being asked for and that is an important test IMO.
What sufficiently hard, but useful, problem would you ask the model for?
BigTTYGothGF 1 days ago [-]
> Test the LLLM against things you want it to do
I agree, it is ridiculous to ask an LLM to replace an artist.
simonw 1 days ago [-]
> The pelican prompt is ridiculous
Yes, deliberately so.
It was never intended as a meaningful benchmark. The surprising thing was that for the first ~12 months performance on the stupid pelican benchmark did seem to correspond to the performance of the models on other tasks.
That pattern no longer holds - Fable 5 and GPT-5.6 have both been out-pelicaned by lesser models now.
Rendered at 20:53:01 GMT+0000 (Coordinated Universal Time) with Vercel.
I've been casually spot-checking other animals in other vehicles, because my absolute dream situation here is to catch an AI lab that's demonstrably better at pelicans on bicycles than other combinations.
Catching a lab cheating specifically on my one dumb benchmark would be really funny.
Dylan's methodology here - generating 1008 SVGs across an 8x6 combination - is significantly more robust than anything I was considering.
His conclusion:
> Nothing jumped out at me. I couldn’t find a case where the pelican-bicycle images looked noticeably better than the rest of that model’s grid.
Because otherwise using a LLM to generate complex svgs is pretty niche and what I thought made this a good benchmark when it was new - generalized programming and spatial knowledge.
Obviously image gen in svg format is not a particularly hard problem if tackled directly on its own.
Instead, it feels like a more appropriate benchmark for the original purpose would be to come up with new, novel problems each time, and compare across all models (including previous ones).
Throwing my hat in the ring: Generate a pelican shaped crossword where all the clues are related to bicycles.
(Haiku 4.5: https://imgur.com/a/N112Nxo, I'm trying some others but it's very slow! Opus has been at it for about 20 minutes.)
This makes me think AI companies are using chat history to train the next model.
It may be that AI performs better on such code.
(don’t tell my boss.)
reminds me of this Key and Peele skit
If you stick to the benchpress, it's just "benchmaxxing".
Minimizing peaks is probably not a good strategy generally. Even low peaks may have some benefit ("if you know your problem is in this domain and performance is critical, this tool offers a 3% advantage").
Minimizing troughs may be more attractive. People generally seem to react more strongly to negatives and competitors can devise benchmarks which emphasize one's troughs.
While a universal expert would be convenient and broad knowledge aids some forms of creativity, specialization has substantial advantages.
Trading max performance (primary metric of concern) for some improvement in a secondary metric often makes sense and that could be an interpretation of "minmaxxing", reducing the over-emphasis of a single metric which would otherwise be maximized.
AI image generation suffers from this more generally. You can generate pictures of pelicans, sure. Newer models clearly generate images with more pelican-ness than before. But all of it is still uglier than sin. Drawing things accurately is one thing, making results that someone might actually want to use (without embarrassing themselves) is something else.
As for conventional diffusion-model stuff, I happen to think there are some pieces of AI art that still look really good even knowing they're AI.
There seems to be a very strong correlation between models that are good at SVG and models that are good at 3D CAD.
Anecdote I know, but there does seem to be generalization going on here.
I haven't really tested Opus 4.8, but 4.7 wasn't nearly as good as ChatGPT 5.5.
It could well be that the exact domain matters more than the bigger picture concepts, like 3d. One of the ever fewer reminders that this tech is still just fundamentally a token prediction algorithm.
...which.. hmm I dunno if they are same or not
That said, I think this would correlate relatively little with general programming ability. They're not unrelated, of course, but being able to generate code that paints an accurate + esthetically pleasing image is quite different from generating code that achieves a non-spatial goal.
Why do LLMs need to be able to do this as well, but worse, slower and more expensive?
LLMs do a great job because they understand both code and SVGs well.
Edit:
An example for a synth I'm buulding: https://imgur.com/a/U694Ek7
The irony of having to post it as a PNG isn't lost on me...
Why on earth would you let an LLM do this when it would take you 10 min to do this in Figma or Inkscape or even just Word
I haven't read the code.
Why would I do it in a slower, more difficult way for something that's going to be outdated in 2 hours?
This design is trivial. I admit it would be hard to achieve in Word (or at least for me because I don’t know how to make a diagram in Word any more) but Figma and Inkascape are made to do these things, and have optimized UI for that (personally I would have just used mermaid though).
I think you may have lost your faith in human capabilities just a little bit if you think drawing stuff like this takes any time or effort at all. Compared to designing and architecturing the system, drawing the diagram is trivial. Now I know that my parent did neither but it seems like they vibe-coded the whole thing. I’m sure they will end up with a fun little toy from the whole endeavor they can play with for 2 weeks before abandoning. Maybe the author will even feel bad about the carbon footprint of this whole exercise and buy some carbon offsets to make up for it.
That's exactly the idea - except it's more like 2 hours before I prototype the next version.
> Maybe the author will even feel bad about the carbon footprint of this whole exercise and buy some carbon offsets to make up for it.
The passive aggressiveness of this is perfectly weighted and admirably phrased.
I'm almost about to post that xkcd 810
I don't understand the truth to this point, and there's a clear difficulty in establishing structure, a thing that you don't need to actually do with raster image gen, right?
Like if you say "please give me 9 circles" you expect circles in the image and not just a bunch of pixel that are vaguely circle-like, right?
https://arxiv.org/abs/2210.05359
That simonw is causing labs to do extra fine-tuning runs for this seems highly probable :)
SVG has its neat SVGO tool that optimises SVG code, this is very clever but it won't spot easy wins such as when a group of elements can be mirrored, or an element re-used with a transform. Neither will it use the full smorgasbord of features. SVGO is not AI, however, it does a fab job of taking bloated files from Adobe Illustrator and getting something good to work with.
The bar for SVG is really low, mostly just paths, compressed and not human readable. SVG should be a human readable format, so 'circle radius 10' rather than two hundred points at six decimal places to draw the same circle.
You have to RTFM to do cool things with SVG and there is a lack of appreciation of the format amongst developers and designers.
https://xkcd.com/810/
Mission fucking accomplished. https://xkcd.com/810/
"Simon Willison, among other things, is an advocate for the inclusion of pelican geometry in LLM training datasets."
So I tried "Make an image of a Djibouti cab with a camel sitting in the passenger seat. Give the camel no human anatomical features." Still bad.
Two years later the graphics and perspectives of the tools are so much better. But that isn't the big story. What really stands out is the models are no longer adding human female anatomies to camels and octopi. The gates and filtering based on instructions have matured enough so that the pelican/bicycle deductive reasoning puzzle is less problematic. But until an LLM anticipates something like impressionism from Parisian artists rebelling against the rules of the French Academy, continue to reserve a place for human artistry.
tl;dr simonw, your spot-check nailed it just as well as any extensive methodology. LLM's no longer need to cheat this part of the test (better to hack the question than the tool).
And since we have established this silly routine once, I must keep going and ask - yes, you have quite a collection of bona fide pelicans you’ve seen and photographed.
Have you physically seen all the other animals you’ve evaluated as well?
I was born in Soviet Russia, so trust but verify and if still around, maybe there pelican make LLM draw Simon ride bicycle and notice Python code improve.
Similar thing happened when TPC came up with SQL benchmarks.
If you're not good at TPC, your engineering team is no good.
If you're good at TPC, then (as a customer) we will actually include you in a bake-off benchmark for our specific problem.
Winning on it is the price of admittance into the game, especially in a crowded market.
But how narrowly you benchmarket matters, you can't just hard-code that specific scenario & not fix anything adjacent while you're at it.
For example when it comes to GPUs, the "Quack3" (sic) benchmark on ATI cards comes to mind.
They're mostly not very funny though.
We see the same problem in database benchmarking.
The good thing about the deliberately-non-real world pelican case is that it gives a general impression of how much the model is improving because it's not likely that it's being specifically targeted at it, rather than a 'real world case' which might have been specifically optimised for.
And that's how we end up with PelicanSkynet
And a man without religion is like a fish without a bicycle.
Snakes on a plane, weasels on a diesel, spiders on a glider, baboons on a balloon, goats on a boat.
> However, facing right is common: 60% of all 1,008 images do it. How common depends on the animal and the vehicle, and bicycles are one of the two vehicles where it’s strongest
Of course the pelican on the bicycle is facing right. The drivetrain on a bicycle is on the right side. If you want any representation of a bicycle that shows the drivetrain you're going to show the right side of it if you want to do so without the frame occluding it. It's an excellent bet that their training data reflects this.
Citation: https://www.rei.com/c/bikes
Edited to add:
As near as I can tell, all of the bicycles are shown facing right, regardless of the direction the animal is facing (GPT 5.6-Terra, Sample 1/3). Also, in every case where the rider has legs (i.e. not the whale) both of the rider's legs are on the right side of the bicycle. This suggests a pretty serious lack of actual understanding of how a bicycle works.
https://www.gianlucagimini.it/portfolio-item/velocipedia/
On the other hand, I notice that the prompt says to draw a pelican riding a bicycle, implying motion... and since most of us read left to right, I think it's usually natural to draw an object in motion moving left to right as well, which means the bicycle should be facing right. So maybe that specific setup is more natural here.
Either way, for humans bicycles are actually really hard to draw from memory. In fact, I substitute teach, and sometimes as an activity I have my students draw bicycles from memory in 60 seconds. Most make pretty serious errors, usually the frame or chain connections: they can tell it's wrong but still can't draw a more correct one. I use it as an object lesson about the difference between recognition and recall - most students never realize that much of their studying can end up being the former, when tests and life almost always ask for the latter. This helps explain why many students go from "that makes perfect sense" when going over review problems to a total mind blank only a few minutes later (especially in math!).
Independent of what humans may do with the same prompt, training datasets will be full of traditional bike photos, which will all be drive side.
It’s also one of the easiest yellow flags to look for in used bike listings (not low-end, but anything enthusiast level). If a bike is photographed on the wrong side, there’s a non-negligible chance it’s stolen, because the seller is presumed to know better if they’re an enthusiast themself.
Looking online, the kickstand is apparently on the left to avoid the gears. Based on the other comment about bike photography, it's interesting the same design choice makes humans and cameras/LLMs see bikes from different sides.
https://www.cyclingweekly.com/news/japan-unveils-new-olympic...
https://www.instagram.com/explore/search/keyword/?q=%23baaw
an... understanding? there is no understanding here at all
1. https://simonwillison.net/2025/Nov/13/training-for-pelicans-...
> Using a single LLM judge for scoring. Every score here comes from one model, GPT-5.6 Luna, looking at one image at a time. I didn’t do much alignment and didn’t check how often it agrees with itself on a re-run.
Having used a similar setup (with previous gen LLMs) to evaluate the 3D models that my product[0] generates, it turned out there was no correlation at all. LLM judgments were very much random and I assume judging SVGs is not that far from judging 3D models. I guess I have to re-test this with current gen.
[0]: https://grandpacad.com
(I'm not saying that they did that. I'm just saying they can.)
The very best svg pelican on a bile generation model. Just for laughs.
This post proves that hasn't happened yet, either. Although maybe the bad results posted online are being trained on and that explains the UNDER performance.
There is a bunch of guidance online on how to photograph bikes, and every sales image of a bike will be from the right. You can anecdotally observe this by google imaging 'bicycle for sale'.
Take a look at the GLM 5.2 and Deepseek V4 "animal on a plane" examples. In every case, the animal is standing on top of the plane, a clear misunderstanding of the concept of "animal on a plane"... with the exception of the Otter. The otters are sitting in a seat on the plane, looking out the window.
That's Ethan Mollick's "Otter On A Plane Using WiFi" image benchmark.
https://www.oneusefulthing.org/p/the-recent-history-of-ai-in...
(Sometimes the Racoon is sitting inside the plane as well, but the racoon is a common backup benchmark. I'm surprised it wasn't also holding a sign saying that it loves trash.)
Also, Grok seemed to really really enjoy "whale on a plane" in that second round, and kudos to GPT Terra for deciding after 3 rounds that the user was terrible at spelling and generated "Antelope On A Plain".
EDIT: I promise I'm a human, but I did just notice my "that's not x... that's y" construction at the start. I am rather Claudepilled — my apologies.
When LLMs use the pattern, they are often setting up a straw man and then knocking it over.
Looking for evidence of the same, but with another twist: checking if the models would choose to create a pelican on a bicycle, if no specific bird or method of transportation was specified.
My version of it: https://www.modelbias.ai/pelican-on-a-bicycle-test
https://www.modelbias.ai/pelican-on-a-bicycle-test/result/12... https://www.modelbias.ai/pelican-on-a-bicycle-test/result/12...
In my tests it did create bicycles the most, but this is just a general bias I believe, as tested here: https://www.modelbias.ai/prompt/transport
Exactly--and you have to ask yourself at this point what "maxxing" really means, since "get better at drawing SVGs" is a useful skill.
Pelicans aside, we need to remember that benchmarks are the only good quantitative way we have of comparing models. If someone has complaints about “benchmaxxing”, please ask them to contribute a better benchmark! It is valuable work and very appreciated.
If you train towards the test, you aren't necessarily improving overall fitness, but you are destroying the value of that test over time because you're decreasing its correlation with overall fitness.
I don't think I'd go that far!
When someone says a model has been benchmaxxed, what they really mean is that it performs better in benchmarks compared to their real world experience. That's a real thing, I've certainly experienced it with some models.
...my take is that some things in life just resist quantitative measurements. Who is the best job candidate? What is the best programming language? Add AI models to the pile.
Me: how many P's are in the following text? [Pasted text]
Claude: There are 14 P's, all lowercase (no capital P's)
Me: how many in "strawberry"?
Claude: there are 3 R's in the word "strawberry".
Me:how many P's are in the following text? [Pasted text with 10 P's]
Gemini: There are 9 "P"s (1 uppercase P and 8 lowercase ps) in the provided text. [List of words except the one missed]
Me: How many in strawberry?
Gemini: Something went wrong (1096)
However thinking about it...if someone asked me to manually create an SVG, or hell even draw a quick doodle on a bit of paper of the same, I'd still probably be much slower than an LLM and potentially end up sketching less accurate anatomy than the machine.
I think the general "organic task" stuff has been mostly sorted out, but in personal and professional experiences using AI to try to _do_ something, I've found less so recently problems with hallucinations and moreso problems with attention.
For example GPT5.6 still has issues where if I provide it with a list of documents and then ask it to raise questions from that information. Then provide it with additional documents that answer some of those questions and ask it to summarise which outstanding questions there are again, it still asks questions that have become irrelevant with the additional documents - but when this is pointed out it knows exactly what to do and produces the correct list of outstanding questions.
I'm sure frontier models are doing all sorts of crazy stuff with attention already, but it seems to me like we almost need some hierarchical attention mechanism like KVL (with Level added) so that it's aware not only of semantic connections between tokens in the context but also of where there are gaps, missing links to assist the model in becoming aware of its own attention span (I guess).
https://x.com/CrimeDecoder/status/2080008114615537766
Can see the images for ChatGPT/Claude (Sonnet 5), and Gemini are all quite bad.
Jagged edge of LLMs. How do you explain being able to generate very complicated shapes in the Pelican example but cannot make a much simpler icon without just alluding to it is in the training data?
> Here is what Qwen3.6-35B-A3B via Openrouter provided for a sloth riding a skateboard: https://imgur.com/a/Dy8fvR5
> Like Grok 4 Fasts attempt at a mushroom in a rowboat, it is barely recognisable as anything despite both Qwen3.6-35B-A3B and Grok 4 Fast having no issue with more popular (i.e. benchmarked) examples. [...]
> And here is Opus 4.7 [which simonw claimed to provide a worse pelican vs Qwen], again via Openrouter: https://imgur.com/a/Qus1Enf
Anyone who hasn't witnessed such deltas either hasn't looked at enough examples, a sufficient variety of models, or both. And they are, unfortunately, not limited to "SVGMaxxing", but a wide range of evals.
[0] https://news.ycombinator.com/item?id=48951229
Are we best to assume an LLM is a blind programmer? Any HN comments from blind programmers tasked with creating SVG icons? Only relevant comment I could find from ctoth was about accessibility: https://news.ycombinator.com/item?id=7185771
Projecting how you think onto what the LLM is doing or should be doing, is probably a mistake on your part.
I recently spent a little time trying to understand exactly why Gemini was misexplaining $X. $X = {why the generated LaTeX visually didn't match what it was asked to do}. It was enlightening.
> Direction: All 21 pelican-bicycle images, across all seven labs, face right. No other animal/vehicle combination does that.
> However, facing right is common: 60% of all 1,008 images do it…
and the tables in “Evidence #5” to be anything but evidence the models have likely trained on pelican on bicycle data more than others.
The data clearly shows:
- 100% pelican on bicycle facing right
- significant skew to the right for bicycle-like vehicles
- significant preference for right facing for birds
Averaging those extreme results to “60%” to make it sound like it’s pretty fair because it’s close to “50%” isn’t statistically sound.
The methodology is generally unsound. There is no actual scoring with a well defined rubric, it’s just vibed with a single model (GPT 5.6 Luna).
The “not better at drawing” evidence are equally hard to take seriously when there is no clear, non-subjective indication of what better or worse is.
Image generation models nowadays can easily generate a photorealistic pelican riding a bicycle, where the bicycle has perfect structure. But it is, of course, only a raster image.
It seems that we're missing a kind of step to decompose an image into a list of instructions (say, SVG paths, or even brush strokes with a real brush) to reproduce it properly. Doing so would probably need a true understanding of the structure of the scene, which is something that AI still struggles with to this day.
It just might feel foreign to human who does not have a SVG trained head-space.
As such, modern LLMs still kind of suck at generating pelican bike SVGs with obvious errors:
* some omitted the bottom of the diamond which connects from the pedals to the rear wheel
* some added an extra connection from the pedals to the front wheel, making it impossible to steer
* none could align the head tube with the fork
* none added a correct offset to the fork
* none could generate the chain properly in a way that attaches to the two sprockets correctly
... whereas these errors do not appear in the raster image. (To be fair, the raster image has other weirdnesses, like the bird having arms and only one leg)
If we could harness the sort of internal thinking that must have happened when it generated the highly consistent raster image, but make it output SVG instead, we'd get much better SVGs. So this would be indeed "visualize in their mind how to draw a picture" before outputting the SVG.
[1] https://chatgpt.com/s/m_6a611c29c02481918fbb0f165eb83594
https://s3.eu-west-1.amazonaws.com/images.dylancastillo.co/p...
https://s3.eu-west-1.amazonaws.com/images.dylancastillo.co/p...
I'm curious if some of the animals / vehicules might force the models to use more tokens than others, and I could not find the token counts in the shared data, is it possible to publish it please? :)
Same pelican grid + MacBook Pro in 3D. Also with end cost.
https://playcode.io/blog/macbook-svg-benchmark
On the flip side, GPT 5.6 Sol did a pretty convincing render of a burglar eating salami.
I'd be curious to see how the other models on random things that are completely tangential to pelicans or bicycles.
So they aren't pelicanmaxxing but they are benchmaxxing in a way. The benefit of the pelican was originally that uplift on the pelican signaled an overall uplift on intelligence. I don't believe that is the case anymore and it is just another jagged edge of model intelligence.
Obviously they're all a bit cartoon-y, what else do you expect from SVGs. But I'm not convinced you could find a single human on Earth over the age of 4 who would seriously give the vehicle in GPT/1/whale/plane a 5/5.
Browse through the options a bit and the rest is not that much better. Grok/2/cat/plane, one of the more accurate planes, got a 2/5. For the most part, vehicles entirely missing do get a 1/5, except for whatever it is in Gemini/1/heron/plane scoring 4. Animals inside planes get completely random vehicle scores I guess.
The cats are all orange, except for a few of the skateboard cats that are black. I'm sure there's nothing to read into there...
Well, I've convinced myself that the next effective test of multimodal models will be whether their judgments of LLM-generated SVG airplanes are anywhere close to reasonable.
It made me realize that for this time around, we’re not at the center anymore. The future of LLMs is the stuff of nations and AIG is what the labs actually care about. They aren’t pelicanmaxxing just as much as they really aren’t revenue/margin/marketing maxing. They just want our attention and ideas so they can show growth and acquire FLOPS.
The apparent fact that they aren’t catering to this community (who frankly decides what goes and stays in prod) leaves me feeling a bit defeated somehow. And also awesome?! Like there is a culture here that runs deeper than any technology and it has fought hard to maintain its identity. Props to dang and all for keeping the astroturfing so imperceptible that I can say this.
In any case, a great little piece of citizen-science dcastm. Lmk if there is a way I can chip in towards token costs.
In the last century, as spec tests for C and C++ compilers, databases, Java application servers became trendy, all vendors were optimising for great articles on the respective technical magazines.
very nice approach to test it and might be a nice way to "grid search" evals in other use cases perhaps.
However, I'm very surprised that most of the models make the same sideways mistake with only some of the animals, and they do it consistently.
With most of the models, cat, raccoons, and otters are almost always riding sideways. Why is that?
Even if using LLM is the only way to do things at scale it does not mean that it's always the right tool.
Your next iteration will need different animals and different transportation options. You'll run out after a few iterations.
I'll look for a better image host in the future. I guess the economic incentives makes them all turn bad after a while.
The right answer here is to ask a LLM to create a scene similar in quality to those, but completely out of distribution.
I asked GPT 5.6 Sol to give me a pelican playing football on San Siro while smoking a cigarette, in AC Milan's t-shirt. While this sounds like higher complexity of a problem, the generations from current models often include additional details like scene composition, scarf, etc., I don't ask for, so I wanted to see what here is memorization vs. composition skill.
"write svg code of a fish playing football on san siro in ac milan's t shirt, with raybans on and a cigarette."
Try that on GPT 5.6 Sol, Fable, or whatever other model. It's chaos.
He was asking the question - do we see gains across other tasks? The underlying question was: Is the additional attention given to this specific task creating a false impression of progress?
You could train a model to do something a bit different like a pencil sketch, or vector graphic sketch, of something given a photo of it, and expect that to be a generalized skill, but if you are asking the model to do it "from memory" then memorizing a pelican is no substitute for not having memorized a horse.
Whatever. Doesn’t really matter much. My next thought is, I get that requiring an SVG is adding an extra layer of complexity as far as the art goes, but why is nobody talking about that the actual art is absolute trash?
I get it. It’s basically a meme at this point and it’s a fun game to play with the models. But my thought is it should be illuminating to anyone who is an artist that LLMs are still a long way off from taking your job :)
> But again, some combinations might be just harder to draw than others.
> To account for that, I fit a fixed-effects regression on all 1,008 images: score ~ lab + animal × vehicle, plus per-lab interaction terms for pelican, bicycle, and the pelican-bicycle cell, with robust standard errors. The animal × vehicle terms absorb the inherent difficulty of all 48 combinations. The interactions measure each lab’s benchmark-specific boost relative to the average lab, with confidence intervals.
Tomato, tomato
I don’t have any reason to believe they are gaming the benchmark, just saying. I do find the idea of a data labeller having to generate thousands of svgs of different animals on different modes of transportation quite funny though.
aren't some LLM going to digest that thread at some point and indirectly learn from it?
basically my point is originally this was a good benchmark because it was an absurd never-seen-before thing, but now that it is in content, some models are going to get a benefit in education?
you'd need the "AI" equivalent of an old-school "google whack", something with no previous results
* https://en.wikipedia.org/wiki/Googlewhack
It is the reinforcement learning that produces more tangible results with less data, but it is something that the AI labs specifically selects and it is not picked up unknowingly
Since the tests can be generally gamed with directing descriptions at it non-deterministically, there's a greater chance the questions solution can be found.
Of course, hopefully the models are instead adding patterns and types of questions as well and it makes the models more capable, but it may be limited in how it transfers to other types of questions in breadth or depth.
I think we should stop using pelican benchmark.
> Every single one is a pelican, on a bicycle, first try. When every student gets an A, the exam has stopped grading.
Numerous pelicans and their bikes are clearly horribly malformed. In fact none of the bike frames are correct. Fable and Opus come close, but the top of the diamond is disconnected in Fable's case and the head tube is misaligned with the front fork in Opus's case.
And of course, as the parent post shows, labs don't actually seem to be training on the pelican bike case.
The problem isn't the test, its that is a public test.
Simon has previously said he has a list of secret prompts (at least one of which he "burned" as a demonstration a while ago). That's what makes it a good test - his commentary on the public test is something of a proxy for non-public tests. This makes it a good benchmark.
"Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes."
Or the more pop layman version
"When a measure becomes a metric/KPI, it ceases to be a good measure."
Story time, I live in Argentina, and we don't have Big Macs, the main Mc Donald's brand, here, because during the CFK presidency, one of her tactics was to Goodhart economic metrics. Even the informal obscure ones like the [Big Mac Index](https://en.wikipedia.org/wiki/Big_Mac_Index), I don't know the precise details, but the Big Mac ended up being a very cheap item, like 2 or 3 times cheaper than actual menu items, but it was never on the advertised menu, and it also ended up being very small compared to the other burgers, so it wasn't even like a hack, a shrinkflation type of deal.
But hey, anyone who read the Big Mac Index table would never find Argentina at the bottom of that list along with a couple of other countries with bad brands, so the ploy worked. And now we live with the aftershock, the brand never really turned around, other brands with ridiculous names took over it like the McTasty, which makes me sound like that skit from Tarantino's Pulp Fiction.
To be clear McDonald's didn't pull out of the country, like in many other countries, it adapted to the local market. Same way as in india it serves non meat variants owing to the high vegetarian population, consider that mcdonald's operates in Venezuela and China, and operated in Russia up until the Russia-Ukraine war. It takes a lot for MCD to pull out of a country.
As for the exact mechanisms of presidential price control on MCD Argentina, I don't have the specifics here, but I can get pretty close. There have been 2 broad mechanisms to exhert price control during CFK's 8 years of presidency and during his Husband's 4, let's call them official and unofficial.
The official mechanisms would be passing laws or presidential decrees (DNU) that don't go through congress, as well as influencing regulation of executive ministries/departments like central bank norms, exchange rates. Some measures like 'precios cuidados' were placed for this very specific purpose, it's very possible Big Macs were under this specific scheme, I do not recall, I was a bit young.
The unofficial methods would be less public, but well known, in the food industry it was especially common, it's well established that food are one of the first and most common targets of price controls. I have heard direct accounts from family members about high ranking government officials setting up meetings with producers in food markets to give orders of lowering prices, one going as far as brandishing a firearm by placing it on a table while discussing the subject.
Writing this out loud I realize that this explains the later over-correction of argentina that allowed freemarket capitalist libertarianism to rise. The optimal strategy in democracy seems to be polarization, so both extremisms seem to symbiotically feed off each other. Not a country of moderateness this one.
Test the LLLM against things you want it to do.
Asking questions that are absurd is like interviewing developers and asking absurd questions on the grounds that it tests creative and critical thinking.
Remember these Microsoft interview questions designed to identify the best developers?
"If you could eliminate one U.S. state, which one would it be?"
"How would you move Mount Fuji?"
Absurd interview questions have an air of legitimacy due to the quasi sophisticated justifications put forward for why they are good tests.
Absurd interview questions are not good tests of people or LLMs.
Relevant questions are good tests.
Yes; what's wrong with that?
Do you suppose that it doesn't test those qualities?
What sufficiently hard, but useful, problem would you ask the model for?
I agree, it is ridiculous to ask an LLM to replace an artist.
Yes, deliberately so.
It was never intended as a meaningful benchmark. The surprising thing was that for the first ~12 months performance on the stupid pelican benchmark did seem to correspond to the performance of the models on other tasks.
That pattern no longer holds - Fable 5 and GPT-5.6 have both been out-pelicaned by lesser models now.