> We wanted to see why Uzbekistan didn’t jump out, so we reproduced it in our comment (Extended Data Fig 1). It turns out that Uzbekistan wasn’t even the biggest outlier, but that the version they had published had the axes cropped so you couldn’t see the outliers (see red boxes in our version). This seemed indicative of a different issue, which is why we documented it in the comment.
Cropping the chart to hide the outliers is so bad that I can't tell if they're incompetent or malicious. I wouldn't be surprised if this is the kind of thing an LLM would produce in the hands of an operator not paying too much attention, but the paper was published in the time period before LLMs were everywhere in publishing.
throw310822 10 hours ago [-]
> Cropping the chart to hide the outliers is so bad that I can't tell if they're incompetent or malicious
It reminds me of the famous "hide the decline", when the climate scientists working on the famous "hockey stick" paper discussed how to hide in the graphs the recent decline of proxy temperatures while measured temperatures kept growing (which would put in question the general reliability of the proxies).
WarOnPrivacy 3 hours ago [-]
PSA: The divergence issue (aka hide the decline) refers to some tree rings in certain high altitude regions where the width of rings after 1960 would flatten or decline.
However, there is evidence in the literature that the use of the
bristlecone pine series as a temperature proxy may not be valid
(suppressing "warm period" in the hockey stick handle); and that
bristlecones do exhibit CO2-fertilized growth over the last 150
years (enhancing warming in the hockey stick blade).
Further driving misreadings of the report is where the
researchers refer to a a standard scientific practice as a
trick. This standard practice is replacing the unreliable
tree-ring data from 1960 onwards with actual thermometer readings.
ref: https://en.wikipedia.org/wiki/Wegman_Report
The hockey stick report does not refer to a decline in rates of warming. And since then, subsequent data has not only confirmed the hockey stick graph, it has pulled it slightly sharper.
Featured two decades later now in the AR6 SPM is a longer Hockey
Stick with an even sharper blade. And no longer just for the Northern
Hemisphere, it now covers the whole globe. The recent warming is seen
not only to be unprecedented over the past millennium, but tentatively,
the past hundred millennia.
ref: https://www.realclimate.org/index.php/archives/2021/08/a-tale-of-two-hockey-sticks/
I feel keeping science safe from bad actors and their misguided followers is our duty to do. My personal thanks to all who toil against corruption machines.
zmgsabst 19 hours ago [-]
This is partly why I don’t worry about “LLM slop” — we had plenty of artisanal human slop before.
saaaaaam 13 hours ago [-]
Artisanal human slop took time - hours, days, weeks - and some effort to produce. AI slop can be produced in seconds, minutes or hours, at the click of a button.
renegade-otter 13 hours ago [-]
Creating misleading data still took effort. Now there is none needed. Slop will simply flood the zone and good content will become increasingly rare. See the problem?
jryle70 10 hours ago [-]
Well, you can also use LLM to find issues, can't you? If it's quick to introduce an error, it's also quick to find and fix the error.
inigyou 9 hours ago [-]
Bonus - when you ask it to find an error but there are none, it will make one up!
nneonneo 23 hours ago [-]
Figure 1a (the leftmost subfigure in TFA’s lead image) shows the data for Uzbekistan from the DOSEv1 dataset (green), DOSEv2 dataset (red) and World Bank (black). The authors of the retracted study used the DOSEv2 dataset in order to model climate effects on the economy at a sub-national level, as opposed to the country-level analyses used in prior work. However, it looks like the DOSEv2 data was just bad for all 14 provinces in Uzbekistan (a 90% drop in GDP for all provinces in 2020!).
The typical correlation between weather and the economy is going to be fairly noisy across the dataset, but if you have 14 extra datapoints all saying there’s a catastrophic GDP crash in one year together with some coincidental weather effect, that’s going to bias the model hard. Notably, they also extrapolate losses forward all the way to 2100, so the effects of such a bias will compound.
MarkusQ 1 days ago [-]
We need something akin to the international geophysical year, but for data integrity. Make it an interdisciplinary priority to clean house and root out papers that are hanging by a thread of included / excluded outliers, biased samples, and outright fraud. It would be humbling, but we'd be in much better shape afterwards.
fliglr 19 hours ago [-]
Given what happened this past 2 weeks, I can't imagine that would be so popular right now
MarkusQ 6 hours ago [-]
This past two weeks specifically? I'm not sure what you're referring to.
In any case, getting the facts straight generally isn't that popular, but it can be very effective.
foxglacier 23 hours ago [-]
It can only be done by outsiders. Everyone involved is incentivized to hide mistakes and fraud.
electroglyph 23 hours ago [-]
Good on them for the retraction. It's good to see science at work.
noopprod 17 hours ago [-]
All else aside I mean how can they even claim to predict what an economy will do in 100 years anyway, it's going to adapt to complex higher order effects. Maybe climate change will increase GDP of everyone has to hire a worker to fan them with palm leaves.
inigyou 9 hours ago [-]
Comparison: 100 years ago there was a global gold standard, Germany didn't exist, only a few people had cars, there were no computer machines no matter how rich you were, no transistors, no TV but many people listened to broadcast radio instead, stock trading was also for rich people and nobody was using the market to see how well the economy was doing, science fiction was about going to Venus because it was thought to be more habitable than Mars, protons had only just been discovered but not neutrons yet, and east of the Mediterranean was the Ottoman Empire.
foxglacier 4 hours ago [-]
By that same trickery, Germany didn't exist 40 years ago either. Neither is very different.
foxglacier 4 hours ago [-]
> increase GDP of everyone has to hire a worker to fan them with palm leaves
If that's how GDP worked, wouldn't it generally decrease with time as technology automates tasks?
What's funny about these future cost of climate change studies is even this incorrectly-pessimistic one just say that in 2100 we'll be about as rich as we would have been in 2090. 2090 level climate-change-free wealth sounds fantastic! That's not a disaster. There are doomers who're sure it'll be the end of civilization or otherwise a huge disaster but there are no quantified predictions showing that.
ktoyame 23 hours ago [-]
Makes me wonder how many more papers out there have hard-to-pin-down errors like that
And how useful potentially AI could be to spot those (even if retrospectively)
tootie 1 days ago [-]
I'm confused as to what the actual issue was. What was the data for which Uzbekistan was the outlier and why?
rao-v 1 days ago [-]
The article suggests it's unreasonable numbers in the original Uzbekistan data source and that other datapoints may have been worse, the authors just didn't correctly execute their basic checks.
"It turns out that Uzbekistan wasn’t even the biggest outlier, but that the version they had published had the axes cropped so you couldn’t see the outliers..."
tjwebbnorfolk 23 hours ago [-]
> had the axes cropped so you couldn’t see
Almost sounds intentional...
> This seemed indicative of a different issue, which is why we documented it in the comment.
Yea, that different issue is fraud.
rdtsc 9 hours ago [-]
They are dancing around the accusation to help authors save face. Extreme incompetence (as in don’t let these people near 100 feet of any Excel spreadsheet level) could be another explanation but given the cropping issue it’s probably intentional
madaxe_again 23 hours ago [-]
They don’t specify, but based on the period they’re talking about I’d put money on it being related to the cotton scandal, to pripiski - that is, the Soviet tendency to make up production figures. When glasnost happened in ‘88 the fiction collapsed, although not immediately - most cotton producers continued to bullshit about their numbers until the mid 90s, while the industry dwindled due to lack of water for irrigation and desertification.
beepbooptheory 20 hours ago [-]
This feels like it could be right, but then maybe its just interesting that there is exactly one outlier here like this, and not more?
esafak 1 days ago [-]
My read is that the model had too much variance; more regularization was needed.
Ozzie-D 21 hours ago [-]
The cascading effect is what makes this particularly dangerous. One bad data point doesn't just produce one wrong conclusion, it gets cited, incorporated into meta-analyses, and eventually shapes policy. By the time someone traces it back to a cropped chart and a suspicious outlier, the conclusions drawn from it have their own citation momentum. The fix isn't just better peer review, it's making raw datasets reproducible enough that anomalies like a 90% GDP drop across 14 provinces get flagged automatically before publication.
throw310822 8 hours ago [-]
You have just described 90% of climate science. It's solid at the foundation (the basic physics of the atmosphere, the predicted trends for the future) and mostly bullshit all the way down from there, each layer building on the uncertainties and biases of the previous.
t1234s 22 hours ago [-]
Uzbecky nitwits
Rendered at 22:21:56 GMT+0000 (Coordinated Universal Time) with Vercel.
Cropping the chart to hide the outliers is so bad that I can't tell if they're incompetent or malicious. I wouldn't be surprised if this is the kind of thing an LLM would produce in the hands of an operator not paying too much attention, but the paper was published in the time period before LLMs were everywhere in publishing.
It reminds me of the famous "hide the decline", when the climate scientists working on the famous "hockey stick" paper discussed how to hide in the graphs the recent decline of proxy temperatures while measured temperatures kept growing (which would put in question the general reliability of the proxies).
The typical correlation between weather and the economy is going to be fairly noisy across the dataset, but if you have 14 extra datapoints all saying there’s a catastrophic GDP crash in one year together with some coincidental weather effect, that’s going to bias the model hard. Notably, they also extrapolate losses forward all the way to 2100, so the effects of such a bias will compound.
In any case, getting the facts straight generally isn't that popular, but it can be very effective.
If that's how GDP worked, wouldn't it generally decrease with time as technology automates tasks?
What's funny about these future cost of climate change studies is even this incorrectly-pessimistic one just say that in 2100 we'll be about as rich as we would have been in 2090. 2090 level climate-change-free wealth sounds fantastic! That's not a disaster. There are doomers who're sure it'll be the end of civilization or otherwise a huge disaster but there are no quantified predictions showing that.
And how useful potentially AI could be to spot those (even if retrospectively)
"It turns out that Uzbekistan wasn’t even the biggest outlier, but that the version they had published had the axes cropped so you couldn’t see the outliers..."
Almost sounds intentional...
> This seemed indicative of a different issue, which is why we documented it in the comment.
Yea, that different issue is fraud.