The growing threat of AI-generated fake accounts on social media

SHAH ALAM – The rise of artificial intelligence (AI) in content creation has blurred the distinction between reality and deception, raising significant concerns about digital trust.

Experts warned that the increasing sophistication of AI-generated content is making it more difficult to differentiate between genuine and fraudulent activity online, with fake accounts capable of artificially shaping public discourse, influencing opinions and even swaying electoral outcomes.

Taylor’s University, School of Media and Communication (SOMAC), Faculty of Social Sciences and Leisure Management senior research fellow Associate Professor Dr Massila Hamzah said the proliferation of AI-generated fake accounts was a growing concern.

She said the combination of advanced AI capabilities and the rapid expansion of social media platforms such as Instagram has enabled fraudulent accounts to reach vast audiences within seconds.

"The ability to create highly convincing fake accounts and content poses a growing threat to all levels of society.

"Likewise, the rapid expansion of social media platforms such as Instagram accelerates the spread of such accounts, enabling them to reach vast audiences almost instantly," she said when contacted.

Massila further noted that Malaysia was not immune to these risks, citing warnings from the Securities Commission (SC) regarding AI-generated scams that used deepfake technology.

She stressed that fraudsters were leveraging AI to create hyper-realistic impersonations of celebrities, public figures and even reputable companies, deceiving victims into fraudulent schemes.

These tactics, she added, made it increasingly difficult for the public to distinguish between authentic and manipulated content.

"AI-generated content has the potential to create highly realistic fabrications of images, videos and voices, making it increasingly difficult for users to distinguish between genuine and fake content.

"As AI technology advances, these fabrications become more convincing, blurring the line between reality and manipulation. This poses significant challenges in verifying online identities and content authenticity," she said.

Massila also highlighted that the AI-generated fake accounts were not always linked to real individuals and were often deployed to spread false information, manipulate public perception and even instil fear and distrust.

These accounts are commonly used for scams, online harassment and coordinated disinformation campaigns, further complicating efforts to maintain digital integrity.

"They are commonly used to distribute spam, amplify coordinated disinformation campaigns and engage in online harassment. Cybercriminals and bad actors leverage AI technology to automate these processes, making them more efficient and difficult to track," she added.

Beyond misinformation, Massila also raised ethical concerns regarding deepfake technology, which, while beneficial for entertainment and education, was often misused for deception and impersonation.

She pointed out that AI’s ability to manipulate audiovisual content raised pressing questions about trust, privacy and digital security.

She added that the rising popularity of AI-generated content has led to increased digital addiction, with many individuals becoming overly reliant on AI tools to produce creative content.

"This can result in digital addiction, where individuals prioritise content generation over real-world interactions, impacting mental well-being and productivity," she said.

Meanwhile, Malaysian Research Accelerator for Technology and Innovation (MRANTI), Innovation Commercialisation, head and AI expert Dr Afnizanfaizal Abdullah referenced a 2023 study by the University of California, Berkeley, which estimated that between five and 15 per cent of social media accounts may be fake or automated.

These, he said ranged from simple bots to sophisticated AI-generated accounts capable of convincingly mimicking human behaviour.

He said the growing presence of fake accounts posed serious concerns for online spaces, particularly in terms of misinformation and digital manipulation.

"AI-generated accounts have become highly advanced, making distinguishing them from genuine users difficult. They can artificially create consensus or dissent on specific topics, influencing public opinion and potentially swaying electoral outcomes," he told Sinar Daily.

Afnizanfaizal also cited findings from a 2023 Stanford Internet Observatory report, which documented how AI-generated accounts played a role in distorting online discussions during the 2022 US midterm elections.

These accounts, he said were used to spread misleading information, amplify partisan narratives and create the illusion of grassroots support or opposition.

He further highlighted that AI-driven fake accounts have significantly eroded trust in social media platforms.

He said AI systems can now generate profiles with realistic images, personal details and human-like activity patterns, making it easier for misinformation to proliferate unnoticed.

A 2023 Edelman survey, he noted, found that 70 per cent of respondents were concerned about the authenticity of information due to the presence of fake accounts, highlighting widespread scepticism in digital spaces.

"This erosion of trust undermines news credibility, enables harmful conspiracy theories and diminishes social media’s value as a platform for genuine human connection and discourse.

"Enhanced detection mechanisms and regulatory frameworks are urgently needed to address these challenges,” he said.

As AI continued to evolve, experts stress the need for stricter regulations and improved digital literacy to combat the threats posed by AI-generated content.

Without effective countermeasures, the increasing sophistication of fake accounts could further undermine public trust and the credibility of online interactions.