‘Mom! Help! Can you transfer money quickly?’

Jennifer DeStefano has just parked her car when her phone rings. It’s a Friday afternoon in January in Scottsdale, Arizona. As DeStefano gets out of the car she looks at the screen. An unknown number. She gets on the phone to her fifteen-year-old daughter Briana, who is training for a ski race in the mountains north of Scottsdale. DeStefano puts her phone on speaker.

“Mom!” says Briana. “Something terrible has happened!”
“What have you done? What is going on?”

At that moment a deep male voice sounds.

“Lay down! Head back!”
“Wait, wait,” says DeStefano. “What is?!”

“Listen. I have your daughter,” says the man. He demands that DeStefano transfer $50,000 immediately. “Otherwise I will stuff your daughter with drugs and dump her across the border in Mexico.”

DeStefano turns off her phone’s speakerphone. She looks around the parking lot in panic and screams for help. Passers-by rush in. One of them calls emergency number 911.

The employee tells DeStefano that she is likely the victim of an “AIscam‘. Something that, says the employee of the emergency center, has been happening more often lately.

DeStefano is only reassured when, on a passerby’s phone, she gets on the phone with her husband, who is with her daughter. She disconnects from the scammer. She never transfers the money.

“My daughter’s voice sounded so real. It wasn’t a recording. No, it was a real conversation,” she says in a phone conversation with NRC. “She cried like my daughter would cry.”

Clone voice with an app

Jennifer DeStefano is one of many American victims of voice cloning, the newest form of online scam. Criminals imitate a voice using artificial intelligence (AI), and then use that voice to encourage a friend or family member to transfer money. There are no exact figures, but in the US the number of stories about victims of voice cloning is increasing rapidly. In the Netherlands, the police reports, there are no known cases yet.

That’s a matter of time. Due to the rapid developments in AI technology, cloning voices is quickly becoming easier and cheaper. The American audio company ElevenLabs – one of the pioneers in this technology – claims that it only takes a minute of recording to make a clone. Since last week, this is possible in twenty-eight languages, including Dutch.

https://twitter.com/StijnBz/status/1694307778955735205

Voice cloning can have great applications, claim the companies that specialize in it. Think of tourists who can make themselves understood everywhere on holiday, with their own voice. ALS patients who can use their voice again. The original voices of historical figures that can be used in games, movies or audio books.

But voice cloning also involves enormous risks. Almost immediately after the launch of the ElevenLabs app came fake audio online in which actress Emma Watson read from My Kampf. Streaming service Spotify has a day job removing songs from musicians whose voices have been copied without permission. Opponents of both Joe Biden and Donald Trump make convincing fake videos in which they put words in American politicians’ mouths. And with the technology, cybercriminals get their hands on a very convincing new weapon to extort money from citizens.

“We live in a time when you can’t trust messages, videos and now voices. Even from the people closest to you,” says Jennifer DeStefano. “Don’t think: this won’t happen to me.”

‘Friend-in-need fraud’

The canteen of the Amsterdam police headquarters, mid-August. Manon den Dunnen, Strategic Specialist Digital at the police, lifts her empty coffee cup from the table. “I have this cup. I can grab him. I can drink out of it. So he exists,” she says. “That’s how our brains work. We trust what we can observe.”

That is why Den Dunnen is very concerned about the consequences of voice cloning. Our communication via telephone or computer screen passes through a “digital filter, which determines what you see and hear,” says Den Dunnen. “When you call, you don’t actually have direct contact with anyone. You are not facing him or her, so you must realize that what you are hearing may have been manipulated. That is very difficult to comprehend.”

Den Dunnen knows the stories from the United States, such as that of Jennifer DeStefano. Fraud with voice cloning is, she expects, something that will soon make Dutch victims as well. Especially now that the software is also available in a ready-made app in Dutch.

Cybercriminals have a very convincing new weapon at their disposal to extort money from citizens

Then it is about what the police call ‘friend-in-emergency fraud’, where someone poses as a so-called acquaintance and asks for money to be transferred. But Den Dunnen also expects an increase in ‘CEO fraud’. In this form of scam, executives are impersonated and companies are defrauded of sometimes enormous amounts. Cybercriminals combine different techniques, breaking into someone’s email and then calling colleagues with a cloned voice.

In January 2020, a bank in the United Arab Emirates lost $35 million after a Hong Kong employee transferred large amounts to various accounts. He thought his boss, who supposedly called him and also sent e-mails, had told him to do so. This bank robbery is, as far as we know, the largest to date where voice cloning has been used.

“It becomes so much easier,” says Den Dunnen. “And when I think about the victims and see how much extra police capacity this will cost, I am seriously concerned.”

‘Reckless’

What can be done about it? Until national governments come up with good AI legislation, the voice cloning companies can do what they want. Although the companies themselves say they do everything they can to prevent misuse of their technology.

After criticism of the first version of the app, ElevenLabs decided to put its software behind a paywall, hoping to deter scammers (a subscription costs $5 per month). The company also released a tools that anyone can detect if audio comes from ElevenLabs. According to the company, these measures should help to create “a safe media landscape”.

Den Dunnen reacts skeptically when she hears about it. Startups shouldn’t be marketing their software at all if it can be abused so much, she says. “These companies first let people experiment with it and only adjust their products after the incidents are widely reported in the press,” says Den Dunnen. “Then I think: think of the victims. It’s really reckless.”

And so, until there is good legislation, companies and citizens must arm themselves against fraudsters. In May, the US Senate called on the top executives of the six largest US banks to take action. Among other things, by purchasing special software that can recognize voices.

According to Manon den Dunnen, this is not the solution. These voting systems do not always pick out AI voice clones, for example the Australian tax authorities experienced. She advocates additional security measures – such as two-factor authentication – “so that systems are not only accessible with biometric data,” says Den Dunnen.

She especially hopes that managers will raise the subject in the workplace. And make good agreements with each other. So that everyone on the work floor knows which control questions to ask if the boss asks him by phone or e-mail to transfer money.

And what can you do as a citizen if your child calls and suddenly needs money? Does every family in the Netherlands have to agree on a code word, such as the Fraudehelpdesk recently recommended? “That is not a good idea, because a code word is also easy to crack,” says Den Dunnen. “Rather ask a few casual questions. About what you discussed the last time you saw each other. Talk about it with each other. That seems very strange now, but I hope that we will all find this normal.”