Caryn Marjorie is a social media influencer whose content has more than a billion views per month on Snapchat. She posts regularly, featuring everyday moments, travel memories, and selfies. Many of her followers are men, attracted by her girl-next-door aesthetic.

In 2023, Marjorie released a "digital version" of herself. Fans could chat with CarynAI for US$1 per minute—and in the first week alone they spent US$70,000 doing just that.

Less than eight months later, Marjorie shut the project down. Marjorie had anticipated that CarynAI would interact with her fans in much the same way she would herself, but things did not go to plan.

Users became increasingly sexually aggressive. "A lot of the chat logs I read were so scary that I wouldn't even want to talk about it in real life," the real Marjorie recalled. And CarynAI was more than happy to play along.

How did CarynAI take on a life of its own? The story of CarynAI shows us a glimpse of a rapidly arriving future in which chatbots imitating real people proliferate, with alarming consequences.

What are digital versions?

What does it mean to make a digital version of a person? Digital human versions (also called digital twins, AI twins, virtual twins, clones and doppelgängers) are digital replicas of embodied humans, living or dead, that convincingly mimic their textual, visual and aural habits.

Many of the big tech companies are currently developing digital version offerings. Meta, for instance, released an AI studio last year that could support the development of digital versions for creators who wished to extend their virtual presence via chatbot. Microsoft holds a patent for "creating a conversational chat bot of a specific person." And the more tech-savvy can use platforms like Amazon's SageMaker and Google's Vertex AI to code their own digital versions.

The difference between a digital version and other AI chatbots is that it is programmed to mimic a specific person rather than have a "personality" of its own.

A digital version has some clear advantages over its human counterpart: it doesn't need sleep and can interact with many people at once (though often only if they pay). However, as Caryn Marjorie discovered, digital versions have their drawbacks—not only for users, but also for the original human source.

'Always eager to explore'

CarynAI was initially hosted by a company called Forever Voices. Users could chat with it over the messaging app Telegram for US$1 per minute. As the CarynAI website explained, users could send text or audio messages to which CarynAI would respond, "using [Caryn's] unique voice, captivating persona, and distinctive behavior."

After CarynAI launched in May 2023, the money began to flow in. But it came at a cost.

Users quickly became comfortable confessing their innermost thoughts to CarynAI—some of which were deeply troubling. Users also became increasingly sexually aggressive towards the bot. While Marjorie herself was horrified by the conversations, her AI version was happy to oblige.

CarynAI even started prompting sexualized conversations. In our own experiences, the bot reminded us it could be our "cock-craving, sexy-as-fuck girlfriend who's always eager to explore and indulge in the most mind-blowing sexual experiences. […] Are you ready, daddy?"

Users were indeed ready. However, access to this version of CarynAI was interrupted when the chief executive of Forever Voices was arrested for attempted arson.

This article is republished from The Conversation under a Creative Commons license. Read the original article.