Artificial intelligence (AI) generates texts in an instant. As a result, AI detectors will be just as bad at recognizing an AI text as someone tossing a coin.
Reliably determining whether a text is made by AI or not may be impossible. That appears from new mathematical proof.
The ease with which AI models generate texts that are indistinguishable from human texts is already leading to problems: students roll their essays out of chatGTP, and mass disinformation campaigns put on feet.
READ ALSO
We don’t need miracles to save the world
If we focus on revolutionary solutions such as carbon capture and geoengineering…
Ideas to combat such abuse include a kind of watermark hidden in AI texts, or searching texts for patterns that only an AI would produce.
Computer scientist Soheil Feizi and his colleagues at the University of Maryland in the United States now claim that they have mathematically proven that these AI unmasking techniques are not reliable. This is due to the combination of paraphrasing programs and AI models. If a student has an essay written by chatGTP, and then runs it through a paraphrasing program, it drastically reduces the effectiveness of a watermark. Also, the texts of language models will become mathematically more similar to human language as the models improve.
Detector errors
To demonstrate this, Feizi and his team used AI-based paraphrasing tools to reword an AI-generated text. They entered that new text into several AI text detectors. The majority of the detectors then only had an efficiency of 50 percent. ‘We see a huge drop in the performance of the detectors. They fall to about the accuracy of an arbitrary predictor,’ says Feizi.
The researchers have used a mathematical proof, the so-called impossible result, show that AI detectors are getting more and more difficult. The reason is that the word choice of AI models is becoming more and more human. That is why the detectors will either incorrectly label too many texts as AI, or too few, so that the real AI texts are no longer singled out. The mathematical proof is a pre-publication, so has none yet peer review undergo.
Consequences
“For all practical purposes, even the best detector, whether it already exists or is yet to be developed, is not going to be very good,” says Feizi. ‘Such a model will actually be very close to just tossing a coin. We will never be able to reliably determine whether a text was created by a human or by an AI model. I think we have to learn to live with that fact,” says Feizi.
Computer scientist Yulan He from King’s College London suggests that we should try to understand the consequences of AI models, rather than spend a lot of time building AI detectors. “What risks do these AI models bring to our lives and how can we use them as useful AIs for ourselves?”