As I looked through the images I was unimpressed entirely, at first. But, then I started thinking, these look a little... "childish" to me.
Childish as in... A newish artist who is drawing a concept rather than light / forms (Which is something artists typically do as they understand drawing more and more).
The rose in the vase specifically - some models understood that there was supposed to be shading, reflections, the concept of refraction - others just drew "blue = glass" and "green = stem" and "red = rose".
Really odd to look at, considering if I saw any of these drawings from a human kid, I would say "good job buddy" and put it on the fridge. I'm expecting these to get better as models improve, and perhaps the artistic progression will be there along with it...
jvanderbot 7 hours ago [-]
I call this "symbol drawing". Beginning artists do this. They think "This is a head, a head is round. This is where eyes go, eyes are shaped like this", and the whole thing ends up being a collage of symbols vs a representation of the space and experience of viewing a face. When I used to be OK at drawing, it was because I forced myself to use touch instead of sight to compose images. So weird to explain, but I'd feel the 3d to get the lighting and such better.
A lot of art that someone smarter than me told me to appreciate seems to follow the pattern of hitting the space and/or experience while minimizing the use of symbols. Impressionistic paintings esp avoid symbols IMHO, while bizzaro picassos abuse symbols outright and still hit the experience they are going for.
MarcScott 12 hours ago [-]
Grok consistently draws images that, if I were its parent, I would refuse to put on the fridge door.
userbinator 20 hours ago [-]
Really odd to look at, considering if I saw any of these drawings from a human kid, I would say "good job buddy" and put it on the fridge.
The Grok ones in particular gave me that thought. Most of them really look like what a kid would do when given the same tools, while the other models' output have distinctly more "AI-ness" to them (for lack of a better term.)
danesparza 7 hours ago [-]
I took the same prompts to Gemini and was stunned by the results. The are completely different from the images shown in the article (and genuinely good art pieces that were generated).
jstanley 6 hours ago [-]
Were you using the Gemini image generator though? The article is about using LLMs to do drawing via tool calls.
ralusek 21 hours ago [-]
What's interesting is that the way in which they're childish is actually extremely human. In fact, one of the ways that you're often taught to draw more realistically is to stop thinking of the concepts as icons you're drawing the outlines of, and instead sort of blur your eyes and see things as they are: hues and values. In other words, become a camera or a printer that has no idea what it's capturing or printing other than a grid of values. That is how you achieve realism.
The fact that it has clearly iconified these concepts in its mind and is tracing the outlines of the things it thinks/expects to go where is very human.
NichoPaolucci 4 hours ago [-]
Yes - precisely what I was getting at with the childish comment.
It felt a bit surreal to see a machines rendition of something that very closely maps to a younger human as they explore + understand more about the world. Scary, even (to me). This was the first time I’ve seen LLM output and thought “wow, maybe it is learning.”
cj 21 hours ago [-]
Parrots can also sound extremely human, but it’s only mimicry.
How do we know the difference?
elcritch 20 hours ago [-]
Yes, but Parrots don't mimic the stages of learning speech development like children do. They just memorize a phrase.
These SOTA LLMs aren't trying to mimic existing children's drawings, but interestingly they're following somewhat similar progression that human children do as they develop.
diegof79 19 hours ago [-]
They don’t develop. We should stop anthropomorphizing LLMs.
AI labs are improving the ML techniques used to build better models; it’s a big difference.
15 hours ago [-]
perching_aix 10 hours ago [-]
That's very clearly what they're referring to also, or at least I certainly doubt this would be lost on them, or most anyone here, at this point.
diegof79 5 hours ago [-]
I imagine that it should be clear to everyone here now, until I read threads like this comparing child development to the advances in LLM building.
It’s like saying that a new M-Apple chip developed like a child. There is a clear tendency to give human like attributes to LLMs, probably caused by the terms used in ML: training, learning, etc.
perching_aix 1 hours ago [-]
I'm not so sure about this view.
For me, it was the same genre of comment as e.g. comments that call out how vision models tend to be structured surprisingly similarly to how the visual cortex is structured, even without specifically attempting to make that happen, and within the boundaries of our understanding of either things of course.
So not so much anthropomorphization, and more a recognition of the unintentional, and surprising, biomimicry.
Though to be honest, I'll be damned if any of the pictures in the post look anything like any child drawing I've ever seen, Grok's or any of the other models.
ETH_start 15 hours ago [-]
Isn't LLM training, the development?
ipaddr 15 hours ago [-]
LLMs come complete they don't train on the fly based on interaction with the world. LLMs are wikipedia if they stopped allowing edits and all of the previous edits are training.
saidnooneever 12 hours ago [-]
there is infact RL for LLMs and its unlikely the frontier labs do not use it in some ways
tadfisher 15 hours ago [-]
The LLMs themselves are not doing the developing or learning.
ben_w 10 hours ago [-]
Both statements are true without contradiction.
A specific model develops as it passes through training.
AI labs change the architecture between models to allow them to surpass the previous models' best scores.
It's also going to keep being a hard sell to say "stop anthropomorphizing LLMs" when the models anthropomorphise themselves.
But besides that: Yeah, sure, they're not human, they're a cargo-cult mimicry of by and of minds that popped out of evolution doing gradient decent on intergenerational survivability. So what? Still interesting when the result of such cargo-culting incidentally echoes what our natural-selection-not-engineered moist electrochemistry happens to do.
richk449 18 hours ago [-]
If you want to know if something is a human or a parrot, you could give it a paint brush and ask it to paint the Mona Lisa.
saidnooneever 12 hours ago [-]
you can also use your eyes, hands or other senses :').
reminds me of those jokes?
what is difference between a parot and a human?
silence
rly? u cant tell the difference between a parot and a human? :')
CamperBob2 18 hours ago [-]
What's your parrot's take on the Jacobian Conjecture?
cj 7 hours ago [-]
My parrot doesn't have a take, but it sure can repeat one it heard on Reddit.
CamperBob2 4 hours ago [-]
So, you're saying that it was somebody on Reddit who disproved the conjecture, and not a "stochastic parrot?"
cj 3 hours ago [-]
You asked for my parrot's take, not how to disprove a conjecture.
Edit: And not a great example to use to prove human-ness considering humans didn't disprove the conjecture?
CamperBob2 4 minutes ago [-]
Parrot analogies for frontier AI are so stupid that they prove human-ness by dint of being a mistake no thinking entity would make.
19 hours ago [-]
jnathsf 19 hours ago [-]
GPT 5.6 Sol had the best two drawings (rose and starry nights) but even more impressive was how efficient it was RE cost/time/tokens vs Fable (3.4M vs 14.6M / $7.74 vs $161!). OpenAI has quietly innovated around inference - this is will be a growing differentiator even against open models.
lubujackson 19 hours ago [-]
I do think they are going to stretch their lead in value if Anthropic doesn't wake up and stop YOLOing tokens. Kimi is an amazing achievement, but it has the same (or worse) kitchen sink approach as Fable.
At work, even if Fable is technically better I much prefer Sol because it is so much faster and concise.
theplumber 18 hours ago [-]
To me GPT is quite slow. I use it with max reasoning though but Fable at max as well
dajonker 13 hours ago [-]
For me personally, there is no point in using more reasoning than medium or sometimes high. It just gets so much slower and expensive, meaning that i have to wait longer to evaluate the results and provide feedback or continue to the next task.
Seriously, what's going on there ? Why is it so different from others? Is it just behind technologically/training wise or it's using something fundamentally different?
jorl17 22 hours ago [-]
Grok 4.5 is...something else.
It performs much better than composer2.5 (while being as fast). It's not Opus, but I think it's not that far off. Definitely better than sonnet for what I've been doing.
On the other hand, I think they probably heavily adapted the training data so that it really is extremely focused on code. I just recently ran my personal "poetry benchmark" on it (where I give it ~850 poems I've written over my life and ask it to comment the corpus as a whole), and it's whack. It tries to write in portuguese (most of the poems are portuguese) and code-switches constantly and mixes up words to the point of making what it writes almost unreadable (e.g. it writes stuff like "You can't QoS that that look for beast poems", in portuguese, all messed up). The quality of the analysis is also quite bad (I'd say it's definitely behind Sonnet).
So I really think they either threw away data that wasn't tied to coding so that they could fine-tune it to that, or somehow they've got such an unbalanced dataset that coding ends up dominating either way. To me, its disastrous performance in this drawing "competition" fits this narrative.
userbinator 20 hours ago [-]
"You can't QoS that" sounds like the title of a nerdcore rap song.
jorl17 19 hours ago [-]
Some of my poetry has clear IT jargon, but it's a very small portion of it (<1%). Some of my teenage poetry revolved a lot around the idea of wanting to become science, knowledge, and machine, and be rid of feeling altogether (to become an idea that has no body, or to be come the mathematical equations that define the world), with some very amateur odes written glorifying science and machine (a clear pastiche of Álvaro de Campos with a modern twist). But, again, this is not the majority of the work, far from it.
For some reason, OpenAI models, Gemini (and apparently Grok too), love to latch onto this and obsess over this idea that it's "programming poetry" or "poetry for the IT crowd". Often OpenAI and Gemini try to write the "equations of my poetry" (granted, I do write about a cyclical relationship between thinking, feeling and writing a lot, and I do have ONE poem which ends with a Q.E.D.).
I'm giving this context to say that it is very bizarre. It's as if they latch onto it and act as if it's a core or highly distinguished part of the poetry, when it really isn't. Anthropic models, on the other hand, absolutely do not do this, and have never done it.
I really don't understand why this happens. Maybe it's because it has a lot of portuguese, I don't know. And even though the "QoS" is clearly the wrong token being generated, I have had situations where gemini spoke of some phase of my poetry as the "Q&A part" (really, no joke...)
In any case, it's why it's my personal benchmark after all :D
saidnooneever 12 hours ago [-]
might also be U shaped memory issue did you try changing the order of your poems to see if it focusses on different ones? maybe it just happened to have those old ones in points it memory was focused at. (ofc its not good but would be an alternative idea to it focussing on IT things/code to produce such results)
jorl17 6 hours ago [-]
Yep, I did. It didn’t have any effect. Randomizing the poems has stopped having a significant effect in the last year or so on the analysis as a whole.
It still somewhat affects the way the AI looks at the poems, especially if it has to make lists of the “best”, but it is not a very pronounced effect nowadays. Maybe it gives some preference for earlier and latter poems but not a lot.
Back when I started doing it, these holes in the context were super obvious, just like you described. It would mixup entire periods and also hallucinate or forget about specific sections. That’s precisely why I started randomizing them as part of my experiments.
Nowadays the frontier models are much better and don’t really mix anything up.
giancarlostoro 22 hours ago [-]
They focused a little too much on Grok Imagine.
Ekshef 22 hours ago [-]
Did Elon tell some poor engineer to give grok a prompt injection for drawing,"make it look like one of my childhood drawings!" Just like the Tesla truck?
visiondude 20 hours ago [-]
the starry night one is soo funny
eth0up 21 hours ago [-]
The razor-wire at the bottom for Starry Night was clever, and very Grok. Really shows its military spirit.
Edit: I just don't see the point of redacting the Mona Lisa
vidarh 12 hours ago [-]
Redacting? Is that not just the model using the smudge tool it was given.
eth0up 6 hours ago [-]
Is this not someone being near infinitely obtuse?
HN, the only environment in the world where people will scientifically falsify a joke.
vidarh 6 hours ago [-]
If I saw something I thought resembled a joke I wouldn't have made the comment.
eth0up 4 hours ago [-]
ESL?
Allow me to explain the theory behind the humor...
Grok's art had heavy use of what strongly resembles a permanent marker with lateral-ish scribbles similar to what one might see when someone is trying to mark something as discarded. Documents, especially those procured by the US government when responding to FOIA requests, contain many 'strike-through' style black marks resembling bold permanent-marker lines, which obfuscate the words they wish the reader to remain ignorant of. In many cases, this is the entire document, especially where unredacted words might otherwise serve in any way to clarify the nature of the FOIA inquiry, i.e. answer the question.
Grok's art, resembled to me, and others potentially appreciating the joke, redactions more than art. In some forms of humor, the presenter of the joke feigns obtuseness, or a mildly self-deprecating, stupid perspective. But the stupidity of the perspective actually reveals a relevant concept, or idea.
Often, such jokes can be deciphered using common sense, or various razors, including both occam and hanlon, the former self-suggesting that the joker is probably not so stupid as to think Grok was really redacting anything, and the latter self-suggesting that there was probably no malice involved.
yojo 21 hours ago [-]
I read the balls as “houses”, though the phallic spire emerging from them dead-center is also very on-brand
eth0up 20 hours ago [-]
I must admit, I have a natural tendency to overlook C&Bs, but solid catch -- it's there. Perhaps that explains the Mona Lisa redaction; Grok probably put more effort into that one.
bdcravens 18 hours ago [-]
The Grok ones are amusing, almost comically bad. However, whenever I've tried to pass an image creation request to any of the Opus models, it's been far worse, like first week of using Microsoft Paint bad (while ChatGPT would create social media quality images using the same prompts)
vidarh 12 hours ago [-]
Opus doesn't have access to an image model, while ChatGPT does (it's not multi-modal, but it can prompt OpenAI's image model). So Opus by default is forced to write code to generate images, or generate SVG's as text. That's putting it in a worse situation than the linked article, where all the models were given rudimentary tools.
Give Opus access to an image model, and it will use that to create images just fine just like ChatGPT.
tylerrobinson 21 hours ago [-]
The Grok ones are so weird that they cross into uncanny and surreal. Truly bizarre and capable of eliciting feelings from me, if only bad feelings…
fastball 14 hours ago [-]
The most interesting result for me is that they apparently prompted the models to optimize for SSIM, but many of the models trend worse over time. I suppose because viewing the canvas always comes after drawing, and they didn't give "revert to previous" capability as part of the toolkit.
Which in turn kinda jives with my experience of using these models for code: to some extent they only seem to have a concept of "forward", which invariably leads to "write more code to fix previous problems created", rather than taking a step back and removing broken things entirely.
d1sxeyes 12 hours ago [-]
Stopping problem at work. I wonder if results would improve if we gave the agents an “undo” or “revert” tool.
hombre_fatal 19 hours ago [-]
GPT-5.6 Sol is the knock out here. Some of those results are really human/charming.
The rectangular smudge tool is a weird tool in the first place, but it's cute to see the models try to use it.
js8 17 hours ago [-]
I really like the cabin from Sol. It could be an art piece.
hombre_fatal 5 hours ago [-]
Its rose is also insane. Not just the shading of the flower but its clever use of the smudge tool to soften the background and then soften the vase so it looked more like glass.
And I like how its starry night decided to accentuate the church/steeple in the village backdrop.
Pretty cool how it did these with a scrappy little tool palette. What a fun idea.
js8 3 hours ago [-]
I found it funny that Fable reinterpreted the Starry Night as "an oil-well in a desert", while Grok went for a "Stalin statue in a prison camp" angle.
raybb 6 hours ago [-]
A year or so ago I was trying to get llms to make pixel art by asking them to create a 2d array of color codes. I'm no prompt guru but they all failed at anything beyond a basic shape. I tried at many resolutions from 16x16, 32x32, 64x64, and maybe a little larger but basically only then quit small ones were even recognizeable.
vunderba 6 hours ago [-]
There’s a related benchmark along these lines that uses an LLM with a discrete “instructive graphic language” to get models to generate Minecraft voxel structures with varying degrees of success.
This is picking the wrong tool for the job twice over. Half of it is seeing if text-to-image mechanics exist in general purpose LLMs, the other half is seeing if they can reason their way about generating the 2d array. Imagine asking this of yourself and doing it through just a train of thought. Yes, I could probably draw a basic smiley face in the color of your choice in a 16x16 without antialiasing in 15 minutes.
The way the LLM describes its own work at the end is hilarious.
traes 12 hours ago [-]
Perhaps there was something in the prompt or settings preventing this, but I'm surprised (and slightly disappointed) that none of the models approached this the way I would: download an image of the Mona Lisa, run all of the drawing functions many times to construct a forward model of the drawing implements, and attempt to solve some kind of explicit inverse problem through either ML or a classical algorithm to minimize some difference metric. Were they just restricted from running code or are the models unindustrious without a very specific prompt?
tulio_ribeiro 13 hours ago [-]
Weird choice of SSIM/RMSE. By feeding these back into the model, the agent is actively degrading its artistic output. This is only valid for the target reference I think?
Alternatively, I think much better results could be had by computing the cosine similarly using DINOv2 ONNX (@xenova/transformers).
My guess is that the results are greatly limited by relying on the current metrics.
Even better, attach toModelOutput directly to drawTool so it returns the rendered canvas image immediately upon drawing.
Grok's look like a truly disturbed child. Like the drawings from that kid in "The Ring".
sashank_1509 14 hours ago [-]
Where do these models even have this data to learn from. There must be massive computer use datasets? I have a hard time believing it’s emergent if it’s able to do something this good.
LastTrain 20 hours ago [-]
Why include Grok when it is clearly not even in the same category as the other three?
taf2 21 hours ago [-]
Would love to see the results if they had used /goal or similar
pavel_lishin 23 hours ago [-]
I think this is just an ad.
noodlescb 23 hours ago [-]
If it is, it's a horrible ad. My main takeaway is that all of them were god awful at image generation and Fable was 20x more expensive and still awful.
Legend2440 22 hours ago [-]
Well, your takeaway is bad. This isn't image generation, it's tool use with a digital paintbrush.
boomskats 21 hours ago [-]
And image tokenisation.
dizzard 20 hours ago [-]
I wonder how much better a harness could get for drawing
aqfamnzc 5 hours ago [-]
I'm surprised that the harness wasn't better described here. Is the draw tool really just asking LLMs to provide a set of coordinates/parameters for generating shapes? Honestly, if so, the results are shockingly good. Think of how well a typical human artist would do with such a crude method of applying brushstrokes.
thomasfromcdnjs 18 hours ago [-]
I think a lot better
- the best image related stuff I've seen is where the harness is constantly cropping and looking closer at things (likely helps a lot for computer vision in general)
- also it would be interesting because the harness could almost have its own "palette" as if it could play with blank squares and different strokes over lapping or blending before applying to the main canvas
wolttam 16 hours ago [-]
Would be great to see Kimi here.
eth0up 22 hours ago [-]
Claude was clearly 'pushing back' on the coziness of the cabin. But I think it did best with the cat. Grok, I fear, is making a case for euthanasia. It's suffering and I think it would be cruel to let it continue. Someone pull the plug. .
AlienRobot 21 hours ago [-]
The difference in cost is pretty incredible.
elcritch 20 hours ago [-]
Rather the difference of cost of Claude is pretty incredible. Then again, I guess they gotta make it rain while they can.
usef- 13 hours ago [-]
He mentions they used quite different methods to get to the end-result. I'd love to know how much changing the prompt impacts things: like telling claude to only draw and blend, similar to sol, etc.
There's likely not training data for this and Claude seemed to spend a lot of time blending the background.
mbmbn 13 hours ago [-]
Shows about the same level of technical mastery that present days painters.
reactordev 19 hours ago [-]
Should have compared against FLUX Klein and Z-Image Turbo...
roguedemon 15 hours ago [-]
That was a good testcase. Pretty cool.
fwip 20 hours ago [-]
Article is AI-written.
maxall4 18 hours ago [-]
I very much enjoyed Grok 4.5's rendition of the Mona Lisa as Elon Musk with tentacles.
SXX 21 hours ago [-]
Now add Deepseek, GLM and Kimi :-)
felixfurtak 19 hours ago [-]
I just tried with Deepseek and got this for the Mona Lisa
Now this is a cool test. Sol's the best in each one, while Grok looks like the work of that person who "fixed" that Jesus painting..
20 hours ago [-]
flintapi 13 hours ago [-]
[flagged]
geroge_kyaw 23 hours ago [-]
Useless They are not image generation models.
Legend2440 22 hours ago [-]
That's what makes it an interesting challenge.
samuelknight 22 hours ago [-]
Not useless. LLMs are the most general purpose computer algorithms ever created. They are getting smarter and cheaper at a geometric rate. What is a bad idea today could have useful applications tomorrow.
munk-a 22 hours ago [-]
> cheaper at a geometric rate
Citation needed - my company is paying more than ever for code generation. I have no reason to believe (given anecdotes) that anyone finds themselves in the opposite situation.
beering 18 hours ago [-]
You’re paying more in total because the overall cost has gone down. This is normal economics when something useful gets cheaper. If you were to use original GPT-4 to do as much, you’d be paying an order of magnitude more.
ssl-3 20 hours ago [-]
It keeps getting cheaper and better -- for me.
I get a lot more use (read: cheaper per interaction) and much better quality results from the $20 that I spend on this stuff every month than I did several years ago.
(And several years before that, it was all essentially unobtanium.)
KolinFirz 23 hours ago [-]
Tell them to draw LeBron James or Heisenberg. Everyone will refuse.
leobg 13 hours ago [-]
I'm missing some context here.
First of all, this is a marketing piece, made for engagement.
Second, in production, you would adjust your approach to the model's strengths and weaknesses. For example, if you found that a model does not respond to the tools you are offering, then you would find a way to make it more responsive. Also, we know that different models handle prompts and system prompts differently. So comparing several models from different providers against exactly the same prompt is quite a naive way of exploring the capabilities of those models or comparing them to each other. You may have just happened to “speak the language“ of one model better than that of another. Why would that be the model’s fault and not yours?
Third, none of these results mean anything if you do not run each process on each model several times to check for consistency. Your model might just have gotten lucky. This point is even more true when you're talking about a progressive task where one step builds on the previous one. The model might just have painted itself into a corner.
Rendered at 20:37:21 GMT+0000 (Coordinated Universal Time) with Vercel.
Childish as in... A newish artist who is drawing a concept rather than light / forms (Which is something artists typically do as they understand drawing more and more).
The rose in the vase specifically - some models understood that there was supposed to be shading, reflections, the concept of refraction - others just drew "blue = glass" and "green = stem" and "red = rose".
Really odd to look at, considering if I saw any of these drawings from a human kid, I would say "good job buddy" and put it on the fridge. I'm expecting these to get better as models improve, and perhaps the artistic progression will be there along with it...
A lot of art that someone smarter than me told me to appreciate seems to follow the pattern of hitting the space and/or experience while minimizing the use of symbols. Impressionistic paintings esp avoid symbols IMHO, while bizzaro picassos abuse symbols outright and still hit the experience they are going for.
The Grok ones in particular gave me that thought. Most of them really look like what a kid would do when given the same tools, while the other models' output have distinctly more "AI-ness" to them (for lack of a better term.)
The fact that it has clearly iconified these concepts in its mind and is tracing the outlines of the things it thinks/expects to go where is very human.
It felt a bit surreal to see a machines rendition of something that very closely maps to a younger human as they explore + understand more about the world. Scary, even (to me). This was the first time I’ve seen LLM output and thought “wow, maybe it is learning.”
How do we know the difference?
These SOTA LLMs aren't trying to mimic existing children's drawings, but interestingly they're following somewhat similar progression that human children do as they develop.
AI labs are improving the ML techniques used to build better models; it’s a big difference.
It’s like saying that a new M-Apple chip developed like a child. There is a clear tendency to give human like attributes to LLMs, probably caused by the terms used in ML: training, learning, etc.
For me, it was the same genre of comment as e.g. comments that call out how vision models tend to be structured surprisingly similarly to how the visual cortex is structured, even without specifically attempting to make that happen, and within the boundaries of our understanding of either things of course.
So not so much anthropomorphization, and more a recognition of the unintentional, and surprising, biomimicry.
Though to be honest, I'll be damned if any of the pictures in the post look anything like any child drawing I've ever seen, Grok's or any of the other models.
A specific model develops as it passes through training.
AI labs change the architecture between models to allow them to surpass the previous models' best scores.
It's also going to keep being a hard sell to say "stop anthropomorphizing LLMs" when the models anthropomorphise themselves.
But besides that: Yeah, sure, they're not human, they're a cargo-cult mimicry of by and of minds that popped out of evolution doing gradient decent on intergenerational survivability. So what? Still interesting when the result of such cargo-culting incidentally echoes what our natural-selection-not-engineered moist electrochemistry happens to do.
reminds me of those jokes?
what is difference between a parot and a human? silence rly? u cant tell the difference between a parot and a human? :')
Edit: And not a great example to use to prove human-ness considering humans didn't disprove the conjecture?
At work, even if Fable is technically better I much prefer Sol because it is so much faster and concise.
I think using this skill, Claude drew it better than the examples given. https://claude.ai/share/bc0867dc-034f-4865-bb71-acb83bfa36a1
Seriously, what's going on there ? Why is it so different from others? Is it just behind technologically/training wise or it's using something fundamentally different?
It performs much better than composer2.5 (while being as fast). It's not Opus, but I think it's not that far off. Definitely better than sonnet for what I've been doing.
On the other hand, I think they probably heavily adapted the training data so that it really is extremely focused on code. I just recently ran my personal "poetry benchmark" on it (where I give it ~850 poems I've written over my life and ask it to comment the corpus as a whole), and it's whack. It tries to write in portuguese (most of the poems are portuguese) and code-switches constantly and mixes up words to the point of making what it writes almost unreadable (e.g. it writes stuff like "You can't QoS that that look for beast poems", in portuguese, all messed up). The quality of the analysis is also quite bad (I'd say it's definitely behind Sonnet).
So I really think they either threw away data that wasn't tied to coding so that they could fine-tune it to that, or somehow they've got such an unbalanced dataset that coding ends up dominating either way. To me, its disastrous performance in this drawing "competition" fits this narrative.
For some reason, OpenAI models, Gemini (and apparently Grok too), love to latch onto this and obsess over this idea that it's "programming poetry" or "poetry for the IT crowd". Often OpenAI and Gemini try to write the "equations of my poetry" (granted, I do write about a cyclical relationship between thinking, feeling and writing a lot, and I do have ONE poem which ends with a Q.E.D.).
I'm giving this context to say that it is very bizarre. It's as if they latch onto it and act as if it's a core or highly distinguished part of the poetry, when it really isn't. Anthropic models, on the other hand, absolutely do not do this, and have never done it.
I really don't understand why this happens. Maybe it's because it has a lot of portuguese, I don't know. And even though the "QoS" is clearly the wrong token being generated, I have had situations where gemini spoke of some phase of my poetry as the "Q&A part" (really, no joke...)
In any case, it's why it's my personal benchmark after all :D
It still somewhat affects the way the AI looks at the poems, especially if it has to make lists of the “best”, but it is not a very pronounced effect nowadays. Maybe it gives some preference for earlier and latter poems but not a lot.
Back when I started doing it, these holes in the context were super obvious, just like you described. It would mixup entire periods and also hallucinate or forget about specific sections. That’s precisely why I started randomizing them as part of my experiments.
Nowadays the frontier models are much better and don’t really mix anything up.
Edit: I just don't see the point of redacting the Mona Lisa
HN, the only environment in the world where people will scientifically falsify a joke.
Allow me to explain the theory behind the humor...
Grok's art had heavy use of what strongly resembles a permanent marker with lateral-ish scribbles similar to what one might see when someone is trying to mark something as discarded. Documents, especially those procured by the US government when responding to FOIA requests, contain many 'strike-through' style black marks resembling bold permanent-marker lines, which obfuscate the words they wish the reader to remain ignorant of. In many cases, this is the entire document, especially where unredacted words might otherwise serve in any way to clarify the nature of the FOIA inquiry, i.e. answer the question.
Grok's art, resembled to me, and others potentially appreciating the joke, redactions more than art. In some forms of humor, the presenter of the joke feigns obtuseness, or a mildly self-deprecating, stupid perspective. But the stupidity of the perspective actually reveals a relevant concept, or idea.
Often, such jokes can be deciphered using common sense, or various razors, including both occam and hanlon, the former self-suggesting that the joker is probably not so stupid as to think Grok was really redacting anything, and the latter self-suggesting that there was probably no malice involved.
Give Opus access to an image model, and it will use that to create images just fine just like ChatGPT.
Which in turn kinda jives with my experience of using these models for code: to some extent they only seem to have a concept of "forward", which invariably leads to "write more code to fix previous problems created", rather than taking a step back and removing broken things entirely.
The rectangular smudge tool is a weird tool in the first place, but it's cute to see the models try to use it.
And I like how its starry night decided to accentuate the church/steeple in the village backdrop.
Pretty cool how it did these with a scrappy little tool palette. What a fun idea.
https://github.com/Ammaar-Alam/minebench
I made a lower effort but similar scaffold for LLMs to do iterative drawing in Nov 2024, with Sonnet 3.5 as the artist: https://paritybits.me/llm-drawing-with-eyes-open/
Quite a difference.
Alternatively, I think much better results could be had by computing the cosine similarly using DINOv2 ONNX (@xenova/transformers).
My guess is that the results are greatly limited by relying on the current metrics.
Even better, attach toModelOutput directly to drawTool so it returns the rendered canvas image immediately upon drawing.
https://www.tryai.dev/?error=server_error&error_code=unexpec...
- the best image related stuff I've seen is where the harness is constantly cropping and looking closer at things (likely helps a lot for computer vision in general) - also it would be interesting because the harness could almost have its own "palette" as if it could play with blank squares and different strokes over lapping or blending before applying to the main canvas
There's likely not training data for this and Claude seemed to spend a lot of time blending the background.
Citation needed - my company is paying more than ever for code generation. I have no reason to believe (given anecdotes) that anyone finds themselves in the opposite situation.
I get a lot more use (read: cheaper per interaction) and much better quality results from the $20 that I spend on this stuff every month than I did several years ago.
(And several years before that, it was all essentially unobtanium.)
First of all, this is a marketing piece, made for engagement.
Second, in production, you would adjust your approach to the model's strengths and weaknesses. For example, if you found that a model does not respond to the tools you are offering, then you would find a way to make it more responsive. Also, we know that different models handle prompts and system prompts differently. So comparing several models from different providers against exactly the same prompt is quite a naive way of exploring the capabilities of those models or comparing them to each other. You may have just happened to “speak the language“ of one model better than that of another. Why would that be the model’s fault and not yours?
Third, none of these results mean anything if you do not run each process on each model several times to check for consistency. Your model might just have gotten lucky. This point is even more true when you're talking about a progressive task where one step builds on the previous one. The model might just have painted itself into a corner.