Deepfake democracy: The AI manipulation influencing the 2024 election in India

During the election campaigns for the state legislatures of Madhya Pradesh in central India and Rajasthan in the west last November, police registered multiple cases for deepfake videos targeting senior politicians including Modi, Shivraj Singh Chauhan, Kailash Vijayvargia (all BJP) and Kamal Nath (Congress). The deepfake content production is often outsourced to private consulting firms, which rely on social media networks for distribution, spearheaded by WhatsApp.

A political consultant who requested anonymity told Al Jazeera that numbers of ordinary citizens with no public profile are registered on WhatsApp and used for the campaigns to make it harder for anyone to directly trace them back to parties, candidates, consultants and AI firms.

This consultant ran six campaigns in assembly elections last year for both the BJP and Congress. “In Rajasthan, we were using phone numbers of construction labourers to run our network on WhatsApp,” they said, “where deepfakes were primarily circulated.”

Meanwhile, AI-manipulated audios are particularly valuable tools in smaller constituencies, “targeting candidates with forged call recordings about arranging ‘black money’ for elections or threatening someone to buy votes,” the consultant said, whose own candidate was targeted with one such recording. The recordings are generally masked with candidates’ voices to cast them as evidence of corruption.

“Manipulating voters by AI is not being considered a sin by any party,” they added. “It is just a part of the campaign strategy.”

India has 760 million internet users – more than 50 percent of the population – behind only China.

Among all the requests, one from a constituency in southern Rajasthan stood out to Jadoun. Ahead of the state election in November, the caller requested that Jadoun alter a problematic but authentic video of their candidate – ​​whose party he did not disclose – to make a realistic deepfake. The aim: to claim that the original was a deepfake, and the deepfake the original.

“The opposition had a troubling video of their candidate and they wanted to spread it quicker on social media to claim it is a deepfake,” he said, bursting into awkward laughter. “Political campaigns are getting weirder.”

Threats to election integrity

Indian laws currently do not define “deepfakes” clearly, said Anushka Jain, a policy researcher at Goa-based Digital Futures Lab. The police have been using laws against defamation, fake news or violation of a person’s modesty, combined with the Information Technology Act, to try and tackle individual cases. But often, they’re playing whack-a-mole.

“The police are prosecuting on the effect of the deepfake and not because it is a deepfake itself,” she said.

Analysts say that the Election Commission of India (ECI), an autonomous body that conducts polling, needs to catch up with the shifting nature of political campaigns.

In the days leading up to the voting in Telangana state elections last year, ruling Bharat Rashtra Samithi party leaders repeatedly warned their followers on social media to stay alert against deepfakes deployed by the Congress party. They also appealed to the ECI against the deepfake clip that the Congress shared on the morning of the vote.

But the video remains online and the party never received notice from the ECI, two Congress leaders aware of the issue told Al Jazeera.

Al Jazeera has sought comments from the ECI but is yet to receive a response.

“Even if one person is misled into believing something and that changes his mind, it vitiates the purity of the election process,” said SY Quraishi, former chief election commissioner of India. “Deepfakes have made the problem of rumour-mongering during the polls graver by a thousand times.”

Quraishi said that deepfakes need to be moderated in real time to minimise the damage they can cause to Indian democracy.

“The ECI needs to take action before the damage is done,” he said. “They need to be a lot more prompt.”

‘Truth is out of reach’

The Indian government has been pressing major tech companies, including Google and Meta, to actively make efforts to moderate deepfakes on their platforms. IT minister Rajeev Chandrasekhar has met officials from these firms as part of deliberations over the threats posed by deepfakes.

By asking the tech sector to take the lead, the government escapes any criticism that it is trying to selectively censor selective deepfakes, or that it is trying to crack down on emerging AI technologies more broadly.

But by passing the buck to private companies, the government is raising questions about the sincerity of its intent to regulate manipulative content, said Prateek Waghre, the executive director of India’s Internet Freedom Foundation, a leading New Delhi-based tech policy think-tank. “It is almost wishful thinking,” he said.

Arguing that the tech companies have not been able to deal with the existing problems with content moderation, Waghre said that “the rise of AI now” has compounded challenges. And the current approach to content moderation ignores what’s really at the heart of the problem, he said.

“You are not solving the problem,” he said. “The design [of algorithms] is just flawed.”

On February 16, major tech companies signed an accord at the Munich Security Conference to voluntarily adopt “reasonable precautions” to prevent artificial intelligence tools from being used to disrupt democratic elections around the world. But the vaguely worded pact left many advocates and critics disappointed.

YouTube has announced that it will enable people to request the removal of AI-generated or altered content that simulates an identifiable person, including their face or voice, using its privacy request process.

“I’m not very optimistic about the platform’s capabilities to detect deepfake,” said Ravi Iyer, managing director of the Neely Center for Ethical Leadership and Decision Making at the University of Southern California’s Marshall School of Business. “With low digital literacy and rising consumption of videos, this poses a grave risk to India’s election integrity.”

Identifying every AI-manipulated media is not a reasonable task, Iyer said, so companies need to redesign algorithms that don’t promote polarising content. “Companies are the ones with the money and resources, they need to take reasonable steps to tackle the rise of deepfakes,” he said.

The Internet Freedom Foundation has published an open letter urging electoral candidates and parties to voluntarily refrain from using deepfake technology ahead of the national elections. Waghre isn’t confident that many will bite, but he said it’s worth a try.

Meanwhile, political campaigns are bolstering their AI armouries – and some, like Reddy, the national coordinator for social media at Congress, concede that the future looks dark.