Please report success/failure after each test. Asking me to read and compare 30 writing samples to get any feedback at all means I won't finish. Telling me immediately when I got one wrong lets me recognize patterns and improve my guesses.
StilesCrisis 1 days ago [-]
I did five, then gave up and just pressed A until I reached the end. I got 3/5 right.
neoncontrails 1 days ago [-]
Exactly the same here. 4/5.
dozerly 1 days ago [-]
Yea, I did two and then harrumphed in annoyance that I was expected to do all 10.
DonHopkins 1 days ago [-]
We must do something about this immediately! Immediately! Immediately!
Not sure what that says about me or the LLM but I guess I shouldn't worry too much about watermarking ruining the outputs…
andai 1 days ago [-]
> Telling me immediately when I got one wrong lets me recognize patterns and improve my guesses.
Wouldn't this make it a worse measurement?
Noumenon72 2 hours ago [-]
I don't want to know whether Claude can pass the Turing test of someone who has never seen Claude's style before. I want to know whether I will start detecting the watermark everywhere and come to loathe its taint, and I want to learn how to identify it so I don't get tricked. Not interested in doing anything extra to help the website.
neoncontrails 1 days ago [-]
Well, the alternative is a lot of us spammed A to get to the end, so the data quality is already horrendous.
stranded22 1 days ago [-]
Yes. Did one - saw that I wouldn’t get feedback until I have completed all 10 (if at all) and noped out.
qarl2 1 days ago [-]
I believe this is to test the theory that people can detect watermarked text.
Teaching you how to identify watermarked text while the experiment is running would ruin the data.
Phemist 24 hours ago [-]
Maybe detecting watermarked text is a skill to be attained. Not allowing proper feedback and training will not allow people to notice the difference on time, thus ruining the data?
Best practice is to allow a number (scaled based on complexity of task) of training rounds (with short feedback loops) prior to letting people loose on the regular samples.
lemarchr 23 hours ago [-]
That would be a different experiment.
qarl2 23 hours ago [-]
That's fine. But the author's goal here is to determine if anyone can tell. So he's probably not going to ruin his experiment.
ipaddr 24 hours ago [-]
Detecting watermarked text is hard. Detecting AI garbage text is easy but then classfiying the garbage further is beyond us.
thaumasiotes 23 hours ago [-]
> I believe this is to test the theory that people can detect watermarked text.
> Teaching you how to identify watermarked text while the experiment is running would ruin the data.
But that's complete nonsense. If it's possible to teach someone how to identify watermarked text, then you've already proven that people can detect watermarked text.
fwlr 3 days ago [-]
Utterly imperceptible, even when studied under the microscope in a way that LLM text very rarely is in practice.
It will be interesting to see whose concerns are assuaged (perhaps they genuinely though mistakenly believed it would degrade quality), and whose concerns are heightened (perhaps their real objection is that their AI-generated text will become detectable).
Yizahi 23 hours ago [-]
My personal concern after the initial news broke about the topic, was not that generated content would become worse, but that to verify the presence of the watermark, one need to upload the complete and unedited source back to these megacorpos, and likely separately to each of them in a few years. Basically this is a truly novel mechanism to legally siphon any new and not yet stolen intellectual property and do that in perpetuity. And there is an absolutely disastrous loss of responsibility here, because soon any organization which requires content integrity (say science, education, literature etc.) will add a new requirement to accept that the submitted new work will be legally and "voluntarily" fed into the LLM megacorpos (with implied loss of property right).
Basically we are on the verge of moment when intellectual property will cease to exist outside of a handful of max security closed sites for military or similar work. Everything else will be automatically and now legally fed into the maw of LLMs.
The fact that watermarking work so spectacularly transparent and undetectable only allows this moment to happen sooner.
Ygg2 23 hours ago [-]
If you suspect that, make sure your text mentions at random gremlins, groblings and jack bunnies. I.e. pre poison your input.
Yizahi 12 hours ago [-]
If you get a choice to do anything like that, then the whole issue in my top comment is irrelevant. My concern was about not having such choice at a global scale.
red_admiral 1 days ago [-]
If anyone notices degraded quality, that would imply they could break the crypto behind the watermark.
For an analogy, distinguishing AES ciphertext from random bits without the key would be counted as breaking AES (the more precise statement of this is called AEAD).
rcxdude 1 days ago [-]
I'm not sure the watermark has been demonstrated to have that kind of property. Even if you use a CSRNG as the 'key' once you're feeding it through the token selection process it's going to risk re-introducing certain correlations.
kalkin 23 hours ago [-]
How would you re-introduce correlations on top of a CSPRNG without a cryptographic break?
rcxdude 11 hours ago [-]
Because in order to not break the token selection process you are making decisions biased by the LLM's output, and in order to make the watermark detectable without the entire context, you are making those decisions based on a relatively small number of preceding tokens.
To give an example in the extreme: if you make the bias total and only make that decision based on the preceding token, there are certain token pairs that your model will never output, and this will be pretty obvious even to people just reading the text (because of any given common two-token phrase, there's a 50% chance you would just disappear in watermarked text). You can make this less extreme and more hidden by increasing the window and reducing the bias, but at the cost of reducing the signal. I don't know exactly what the tradeoff curve looks like, so it might be that you can reach set of parameters where the bias is in principle undetectable without the key but still reliably detectable for realistic lengths of text segments, but I would not assume that this is definitely the case.
kalkin 3 hours ago [-]
This doesn't change the temperature of the model. It's not going to make something that's a 70% chance suddenly a 100% chance. It also doesn't change the context length. At most, it seems like it may reduce some of the variation between different requests sampled from the same prompt with nonzero temperature, if I'm reading this right from the Google paper that Anthropic says they're implementing:
> For our experiments, we configure SynthID-Text to be single-sequence non-distortionary; this preserves text quality and provides good detectability, while having some reduction to inter-response diversity. We call this configuration ‘non-distortionary SynthID-Text’ (and where not otherwise specified, ‘SynthID-Text’ also refers to this).
Great, but SynthID watermarking isn't a cryptographically-secure scheme, nor could one be made, because all the things necessary for the scheme to be crypto-secure are also things that go against being human-comprehensible.
The SynthID watermark scheme relies on the LLM's own logits being a map into acceptable substitutions for specific words. For example, if we have the sentence "The quick brown fox jumps over the lazy dog", the LLM might give the last word[0] as 98.5% "dog", 1% "hound", 0.5% "canine". In normal decoding we sample the logits after scaling them against a temperature parameter; in SynthID we additionally bias the logits to encode output from the watermark CSPRNG - e.g. the CSPRNG might decide to boost the third logit of this particular token to 65%.
But wait! The LLM might have selected valid synonyms for the token, but those synonyms will never be correct because this is a very well-known sentence. To humans, it's one sentence-long token, in the sense that most English-speaking humans who encounter this sentence will see the first few words and already know the ending. It thus has one bit of entropy[1]. Adding SynthID watermarks to this sentence necessarily has to change the words and thus adds more information - namely, that some process outside of the LLM itself has modified the text.
We don't need to cryptographically distinguish AES from random if both the AES (SynthID) and random (normal decoding) happens to also be constrained to form a valid English sentence.
[0] I am too lazy to account for multi-token words. Let's just assume the LLM was trained with a pretokenizer that splits exclusively on whitespace and every word in the training set got assigned a token.
[1] Compressible information
red_admiral 13 hours ago [-]
That doesn't match my understanding of how the watermark works. Let's say we have your distribution of (0.985, 0.01, 0.005). The AI somehow has to sample that distribution, let's say the pre-watermark model picks as many uniform (i.i.d.) bits as it needs to do the sampling, then does a Smirnov transform or something to get the correct distribution.
The AI with watermark picks its bits from a CSPRNG, then does the same transform to get the distribution. The result I think is that you can reduce distinguishing the distribution to breaking the CSPRNG.
In this example the entropy after the transform is around 0.13 bits, so the watermarking scheme cannot add any useful information. Once you have hundreds or prehaps thousands of bits of entropy, it starts to become feasible to add information into that 'spare' space.
usef- 23 hours ago [-]
Your example is not realistic: it won't arbitrarily change a 0.5% token up to 65%. If it did it would absolutely degrade performance in a very measurable way. It's only biasing the randomness for tokens where it does have multiple plausible choices.
For constrained quotes like "quick brown..." the synthid watermark would be weak, but that's by design.
bastawhiz 1 days ago [-]
I've read that watermarking should in theory be impossible to detect except by the entity that watermarked it. Which is sensible, and I mostly understand at a high level.
But what I don't know and don't understand is what happens if you watermark watermarked text. Does it test positive for both watermarks? Only the second? Indeterminate?
Or maybe I'm misunderstanding. Can you tell that it's watermarked, but only the entity who put the watermark in place can test if it's theirs? My confusion about watermarking multiple times still stands, though.
Regardless of what happens when you watermark multiple times, no matter the outcome, it weakens the watermark. Which, depending on the threat model, kind of makes it moot. I can't imagine a serious situation where a watermark can be weakened in any way and still be useful. Even "this came from an LLM" isn't a valid signal if you can just watermark ANY text through purely mechanical means.
It's also not clear to me how this will affect mainstream LLMs. If all output text is watermarked, there MUST be an escape hatch. Otherwise, JSON schemas will break (or provide holes where unwatermarked text can be exfiltrated through MCP), "return this text exactly with no changes" will be impossible, and writing diffs will break.
I feel like I must be missing something.
mhitza 1 days ago [-]
If you are refering in terms of the code of practice part of the EU AI Act
> I've read that watermarking should in theory be impossible to detect except by the entity that watermarked it
This is a carveout exception, for watermarking. In the spirit of those terms it should be machine identifiable.
In my opinion they should have thought better about this, paricularly for text, because in its current forms it is easy to lead next to a new "tamper-proof" requirement, which in practice is DRM. And we do not need more DRM.
For images, music there is metadata already where such information can be stored. And if end users are found using unlabeled AI their accounts could be ban from these platforms. Not something the social platforms might want, but it's a saner approach than trying to reinvent the secret printer dots on all generated media.
i think practical question is whether the detector survives ordinary transformations of the text. if for say i paraphrase a watermarked answer with another model, do we expect the original signal to disappear and the second model's signal to replace it?
skybrian 1 days ago [-]
There are multiple ways to paraphrase and the paraphrasing model is going to use its own random number generator whenever it thinks there's more than one possible choice. (Not really binary; the RNG will have more or less effect.)
So it will definitely be watermarked by the paraphrasing model. But the question is whether the original signal survives at all. There might be a weak signal that's detectable with enough text?
billyp-rva 1 days ago [-]
If its paraphrased to any significant degree I'd expect the original to not survive. The second model's watermark would of course be there regardless.
throw310822 1 days ago [-]
Does it still apply with zero entropy?
red_admiral 1 days ago [-]
No. Anthropic's example is completing the sentence "Isaac Newton's most famous work was called Principia ..." has only one correct answer, so nothing to watermark.
andy99 1 days ago [-]
The watermarking is an inextricable part of the token generation, it just using a known pseudo random sequence for the sampling. It’s not a transform that can be applied later.
bastawhiz 8 hours ago [-]
Any LLM generation is just a transform of input to output.
rcxdude 1 days ago [-]
This obviously stops working as soon as you don't have the entire context. To reliably detect a subset of the LLM's output you need to do something more sophisticated but also more invasive.
pickledish 23 hours ago [-]
(not sure why you were downvoted, you are correct)
1 days ago [-]
Jowsey 24 hours ago [-]
Interestingly, it seems almost every set of three seems to follow a pattern: one passage of the three will have a key word or phrase swapped in the first sentence. That is, for every set of 3 passages, two will start with ~identical sentences, and one will have a key word or token changed.
I caught onto this early and used it every time, and ended up getting 2/10, which is worse than random chance. I smell trickery!
lacker 1 days ago [-]
This is like giving you three outputs from md5sum and asking you to guess for which one the input ended in a "q". There's no way to tell unless you break the RNG.
pllbnk 1 days ago [-]
Yeah, it's a pointless exercise. I hope the author is just trolling given that he is knowledgeable in the field.
> Here are three 64-character hex strings. Two are random. One is HMAC-SHA256(secret_key, "anthropic"). You don't have the key. Which one is the HMAC?
josh-sematic 1 days ago [-]
I think the point is probably to help convince people that the watermarking doesn’t perceptibly impact quality, which is a concern some people have (whether well founded or not).
ipaddr 24 hours ago [-]
"doesn’t perceptibly impact quality" no not perceptibly but it does.
BoiledCabbage 23 hours ago [-]
> no not perceptibly but it does.
Similar to how a single particle of dust landing on your shoulder makes you weigh more.
Yes it does - but anyone arguing that is completely missing the point.
pllbnk 4 hours ago [-]
Even then, LLMs are based on a lossy compression, so the quality is harmed by design.
NotPractical 3 days ago [-]
Could do with some context on how watermarking works. Objectively speaking it should be impossible to tell.
marcyb5st 1 days ago [-]
My understanding is that watermarking in prose is basically a bias when sampling tokens. For a system that knows the average probability for each possible token in the LLM vocabulary it is possbile to quantify said bias given enough text.
For a human that doesn't reason in tokens and therefore doesn't know anything about their probability distribution, it should be impossible to tell. Relying on fancy words/constructs within sentences should not give you any signal as well, since you don't know if the the prompt included instructions for that.
elikoga 2 days ago [-]
I disliked the fact that the experiment only covered prose, which my eyes glossed over and made me actually do random entries to pass on and see the results. I'd love to see it on a more accurate output distribution like commented code
gjm11 1 days ago [-]
I don't think anyone is, or plans to be, watermarking AI-generated code as opposed to text.
[EDITED to add:] As pointed out by a helpful comment below, I was misremembering: Anthropic do apply their watermarking to code, they just say that it will have negligible impact on the actual code (because there's generally less scope for variation in that) but e.g. it will have its usual effect on comments in the code.
> code—which in very many cases has to be exact—has generally less watermarking than some other forms of text.
Generally less watermarking. Not no watermarking.
demibabs 1 days ago [-]
What do you mean? The watermarking applies to all text outputs, including code.
It’s just much less effective since code is low entropy.
red_admiral 1 days ago [-]
Why not? It would help with a lot of potential legal issues.
reactordev 1 days ago [-]
Wow I actually got a 7/10. It was hard to tell at first but there are signs that tipped me off to which one probably had a higher score out of the multiple choice.
petters 1 days ago [-]
You likely just got lucky.
arcwhite 3 days ago [-]
Interesting, I did very badly, 3/10!
Yizahi 23 hours ago [-]
3/10 too, and I've actually tried to discern the answer. My working theory was to pick texts which jumped between more to less frequent words (subjectively of course), but I was very wrong, basically I couldn't spot a watermarked trait at all.
bastawhiz 1 days ago [-]
Same, doing worse than random chance seems like an interesting signal though, but I'm not sure what it's a signal of.
DHowett 1 days ago [-]
Only one third of the options at each stage are watermarked, so 3/10 seems well within random chance.
8 hours ago [-]
abathur 1 days ago [-]
My sense of the concern here is that watermarking may somehow deprive someone or something of value regardless of whether or not they can tell, so I briefly pondered trying to rank these from best to worst and see if any set of those votes meaningfully deviated from ~average.
That said, I read the first triple and found all three tortured enough that I can't be bothered with the rest.
Call me persuaded, I guess.
rrr_oh_man 1 days ago [-]
It feels all of them are terribly written. I don't know why.
StilesCrisis 1 days ago [-]
Because it's AI slop? Not that surprising.
Lerc 1 days ago [-]
I was never going to do very well on this. My ADHD was itching after the third one. I suspect it would have been sooner but I had a bit of extra focus from the suprise that it selected an answer for the first question when I tried to scroll.
To avoid that on the following questions I just held my finger on my phone to avoid a click. That eventually selected some text, and I instinctively tapped to deselect. That triggered another random pick, then I just tapped through to the end because I was fed up.
cellis 8 hours ago [-]
Did 3 then just assumed the longest text was the correct one. Too much reading
DecoPerson 21 hours ago [-]
Holy moly PLEASE:
- Do not make tapping the text act as selection. I was trying to scroll and accidentally advanced twice.
- Add back and/or reset test buttons.
- Give immediate feedback, rather than asking me to read 10 x 3 long texts.
Very frustrating. I was very interested in the results but this combination of problems made this site worthless.
madarcho 3 days ago [-]
If SynthID is a google technology, then this is likely just us training their ai again, captcha all over again.
jamienk 23 hours ago [-]
The writing reminded me of the SAT test reading section. Impossible to focus.
smallerize 1 days ago [-]
Google's SynthID page says they can watermark text, but it also says that it can only detect the watermark on "image, video or audio". Does that mean that the text watermarks can't actually be used as watermarks?
Jabbles 24 hours ago [-]
What is the meaning of the numbers?
> Weighted mean detector score 0.5307
Does it mean that that passage would be rated a 53% chance of being watermarked? So you would need a passage 10x as long to be reasonably sure of providence?
smikhanov 1 days ago [-]
It takes a lot of patience to read this much slop voluntarily.
1 days ago [-]
kshmir 1 days ago [-]
Thought there were only 2 options!
foundry27 1 days ago [-]
Me too lol.
It was only at question #8 that I realized there was a third option, and while I’d love to say that accounts for how I got a 1/10, after reviewing the third options I doubt it would’ve made a difference.
IshKebab 23 hours ago [-]
Ha me too. That explains why 4/10 is "slightly better than random chance".
aizk 1 days ago [-]
I had a moment I thought was concrete watermarking the other day.
Claude wrote the sentence... "since the compute buffer estimate has some sl..."
Now you'd think the right word would be slack, but Claude wrote... slop?
Which does seem off but, those two letters could be tokens very close in probability.
bombcar 19 hours ago [-]
We’re going to get people passing AI text through various competing AIs, aren’t we?
Deep Fried AI Slop™ here we come!
raver1975 23 hours ago [-]
ai;dr
jibal 1 days ago [-]
This is stupid --- way too long and wordy. Make your test worth taking. And even then it's theoretically impossible to detect the watermark so what even is the point? If it's to check whether the watermarking actually has that property, this is not at all a reliable way to do that.
IshKebab 23 hours ago [-]
The watermarking only works with long wordy text.
But yes I agree, it's pointless because we already know that nobody can detect this.
doublerabbit 21 hours ago [-]
Your result: 7/10
You did slightly better than random chance
... That was random chance. Got bored and just clicked on random boxes.
AiToolsGem 24 hours ago [-]
[flagged]
MagicMoonlight 1 days ago [-]
[dead]
lolokbro 1 days ago [-]
[flagged]
jdw64 1 days ago [-]
8/10. It was harder to distinguish than I expected. If they had applied something like a humanizer skill, it probably would have been nearly impossible to tell.
kalkin 23 hours ago [-]
It sounds like you have a publishable paper - you've broken Aaronsen's watermarking scheme. Congratulations!
(Or maybe you got lucky.)
19 hours ago [-]
Rendered at 22:21:24 GMT+0000 (Coordinated Universal Time) with Vercel.
https://www.youtube.com/watch?v=jLO7VrRij_M
Not sure what that says about me or the LLM but I guess I shouldn't worry too much about watermarking ruining the outputs…
Wouldn't this make it a worse measurement?
Teaching you how to identify watermarked text while the experiment is running would ruin the data.
Best practice is to allow a number (scaled based on complexity of task) of training rounds (with short feedback loops) prior to letting people loose on the regular samples.
> Teaching you how to identify watermarked text while the experiment is running would ruin the data.
But that's complete nonsense. If it's possible to teach someone how to identify watermarked text, then you've already proven that people can detect watermarked text.
It will be interesting to see whose concerns are assuaged (perhaps they genuinely though mistakenly believed it would degrade quality), and whose concerns are heightened (perhaps their real objection is that their AI-generated text will become detectable).
Basically we are on the verge of moment when intellectual property will cease to exist outside of a handful of max security closed sites for military or similar work. Everything else will be automatically and now legally fed into the maw of LLMs.
The fact that watermarking work so spectacularly transparent and undetectable only allows this moment to happen sooner.
For an analogy, distinguishing AES ciphertext from random bits without the key would be counted as breaking AES (the more precise statement of this is called AEAD).
To give an example in the extreme: if you make the bias total and only make that decision based on the preceding token, there are certain token pairs that your model will never output, and this will be pretty obvious even to people just reading the text (because of any given common two-token phrase, there's a 50% chance you would just disappear in watermarked text). You can make this less extreme and more hidden by increasing the window and reducing the bias, but at the cost of reducing the signal. I don't know exactly what the tradeoff curve looks like, so it might be that you can reach set of parameters where the bias is in principle undetectable without the key but still reliably detectable for realistic lengths of text segments, but I would not assume that this is definitely the case.
> For our experiments, we configure SynthID-Text to be single-sequence non-distortionary; this preserves text quality and provides good detectability, while having some reduction to inter-response diversity. We call this configuration ‘non-distortionary SynthID-Text’ (and where not otherwise specified, ‘SynthID-Text’ also refers to this).
https://www.nature.com/articles/s41586-024-08025-4
The SynthID watermark scheme relies on the LLM's own logits being a map into acceptable substitutions for specific words. For example, if we have the sentence "The quick brown fox jumps over the lazy dog", the LLM might give the last word[0] as 98.5% "dog", 1% "hound", 0.5% "canine". In normal decoding we sample the logits after scaling them against a temperature parameter; in SynthID we additionally bias the logits to encode output from the watermark CSPRNG - e.g. the CSPRNG might decide to boost the third logit of this particular token to 65%.
But wait! The LLM might have selected valid synonyms for the token, but those synonyms will never be correct because this is a very well-known sentence. To humans, it's one sentence-long token, in the sense that most English-speaking humans who encounter this sentence will see the first few words and already know the ending. It thus has one bit of entropy[1]. Adding SynthID watermarks to this sentence necessarily has to change the words and thus adds more information - namely, that some process outside of the LLM itself has modified the text.
We don't need to cryptographically distinguish AES from random if both the AES (SynthID) and random (normal decoding) happens to also be constrained to form a valid English sentence.
[0] I am too lazy to account for multi-token words. Let's just assume the LLM was trained with a pretokenizer that splits exclusively on whitespace and every word in the training set got assigned a token.
[1] Compressible information
The AI with watermark picks its bits from a CSPRNG, then does the same transform to get the distribution. The result I think is that you can reduce distinguishing the distribution to breaking the CSPRNG.
In this example the entropy after the transform is around 0.13 bits, so the watermarking scheme cannot add any useful information. Once you have hundreds or prehaps thousands of bits of entropy, it starts to become feasible to add information into that 'spare' space.
For constrained quotes like "quick brown..." the synthid watermark would be weak, but that's by design.
But what I don't know and don't understand is what happens if you watermark watermarked text. Does it test positive for both watermarks? Only the second? Indeterminate?
Or maybe I'm misunderstanding. Can you tell that it's watermarked, but only the entity who put the watermark in place can test if it's theirs? My confusion about watermarking multiple times still stands, though.
Regardless of what happens when you watermark multiple times, no matter the outcome, it weakens the watermark. Which, depending on the threat model, kind of makes it moot. I can't imagine a serious situation where a watermark can be weakened in any way and still be useful. Even "this came from an LLM" isn't a valid signal if you can just watermark ANY text through purely mechanical means.
It's also not clear to me how this will affect mainstream LLMs. If all output text is watermarked, there MUST be an escape hatch. Otherwise, JSON schemas will break (or provide holes where unwatermarked text can be exfiltrated through MCP), "return this text exactly with no changes" will be impossible, and writing diffs will break.
I feel like I must be missing something.
> I've read that watermarking should in theory be impossible to detect except by the entity that watermarked it
This is a carveout exception, for watermarking. In the spirit of those terms it should be machine identifiable.
In my opinion they should have thought better about this, paricularly for text, because in its current forms it is easy to lead next to a new "tamper-proof" requirement, which in practice is DRM. And we do not need more DRM.
For images, music there is metadata already where such information can be stored. And if end users are found using unlabeled AI their accounts could be ban from these platforms. Not something the social platforms might want, but it's a saner approach than trying to reinvent the secret printer dots on all generated media.
So it will definitely be watermarked by the paraphrasing model. But the question is whether the original signal survives at all. There might be a weak signal that's detectable with enough text?
I caught onto this early and used it every time, and ended up getting 2/10, which is worse than random chance. I smell trickery!
> Here are three 64-character hex strings. Two are random. One is HMAC-SHA256(secret_key, "anthropic"). You don't have the key. Which one is the HMAC?
Similar to how a single particle of dust landing on your shoulder makes you weigh more.
Yes it does - but anyone arguing that is completely missing the point.
For a human that doesn't reason in tokens and therefore doesn't know anything about their probability distribution, it should be impossible to tell. Relying on fancy words/constructs within sentences should not give you any signal as well, since you don't know if the the prompt included instructions for that.
[EDITED to add:] As pointed out by a helpful comment below, I was misremembering: Anthropic do apply their watermarking to code, they just say that it will have negligible impact on the actual code (because there's generally less scope for variation in that) but e.g. it will have its usual effect on comments in the code.
> code—which in very many cases has to be exact—has generally less watermarking than some other forms of text.
Generally less watermarking. Not no watermarking.
It’s just much less effective since code is low entropy.
That said, I read the first triple and found all three tortured enough that I can't be bothered with the rest.
Call me persuaded, I guess.
To avoid that on the following questions I just held my finger on my phone to avoid a click. That eventually selected some text, and I instinctively tapped to deselect. That triggered another random pick, then I just tapped through to the end because I was fed up.
- Do not make tapping the text act as selection. I was trying to scroll and accidentally advanced twice.
- Add back and/or reset test buttons.
- Give immediate feedback, rather than asking me to read 10 x 3 long texts.
Very frustrating. I was very interested in the results but this combination of problems made this site worthless.
> Weighted mean detector score 0.5307
Does it mean that that passage would be rated a 53% chance of being watermarked? So you would need a passage 10x as long to be reasonably sure of providence?
It was only at question #8 that I realized there was a third option, and while I’d love to say that accounts for how I got a 1/10, after reviewing the third options I doubt it would’ve made a difference.
Deep Fried AI Slop™ here we come!
But yes I agree, it's pointless because we already know that nobody can detect this.
... That was random chance. Got bored and just clicked on random boxes.
(Or maybe you got lucky.)