India’s New Delhi — A seven-second video went viral on social media on November 30, 2018, as voters lined up early to cast ballots in legislative elections to select Telangana’s new administration. Telangana is a state in southern India.
The image, which was posted on X by the state-based Congress party, which is currently in opposition nationally, featured KT Rama Rao, the head of the state’s Bharat Rashtra Samiti, urging people to support the Congress.
A senior Congress leader who asked to remain anonymous claimed that the party “operated unofficially” by sharing it extensively on several WhatsApp groups. After all, it was viewed over 500,000 times on the party’s official X account.
It was fake.
The Congress party leader told Al Jazeera, “Of course, even though it looks completely real, it was AI-generated.” “However, a typical voter would not be able to tell the difference; voting had already begun [when the video was uploaded], and [the opposition campaign] had no time to limit the damage.”
The well-timed deepfake served as a symbol of the deluge of artificial intelligence (AI)-generated, or manipulated, media that has tainted several state elections in India in recent months and is now threatening to drastically alter the country’s upcoming national elections.
In the largest elections in history and throughout the world, India’s approximately one billion voters will choose their next national government between March and May. The world became aware of the dangers posed by false AI-generated material in January when phony, sexually graphic photos of singer Taylor Swift surfaced on social media. Deepfakes are a “threat to democracy,” according to India’s information technology minister Ashwini Vaishnaw, who expressed this fear in November. Prime Minister Narendra Modi has repeated these views.
However, managers of over 40 previous campaigns told Al Jazeera that teams from all of India’s major parties, including the Congress and Modi’s Bharatiya Janata Party, are using deepfakes to sway voters due to the easier access to portable artificial intelligence tools. While some AI tools for creating deepfakes are freely available, others can be subscribed to for as little as 10 cents every video.
‘Creating perception’
Arguably the tech-savvy party in India, the BJP, has pioneered the use of illusions in political advertising. The party began using 3D holographic projections of Modi in 2012 so that he could “campaign” concurrently in dozens of locations. The tactic was extensively used in the 2014 general elections that resulted in Modi’s election.
While there was not much deception involved, Manoj Tiwari, a BJP member of parliament, was one of the first people in the world to employ deepfakes for political campaigning in February 2020. Ahead of the legislative assembly elections in the nation’s capital, Tiwari spoke to voters in three different languages—Hindi, Haryanvi, and English—reaching three different groups of people in the diverse metropolis. The only real video was the Hindi one: The other two were deepfakes, in which artificial intelligence (AI) was utilized to mimic his speech and facial expressions while also manipulating his lip movement. This made it nearly impossible to tell that they weren’t real just by looking at them.
employing lifelike footage of the legendary politician and former movie writer M Karunanidhi at campaign rallies, the Dravida Munnetra Kazhagam (DMK), which governs the southern state of Tamil Nadu, has been employing artificial intelligence (AI) to bring back its iconic leader M Karunanidhi in recent months.
According to analysts and campaign managers, the 2024 elections may significantly increase the use of deepfakes.
“Politics is about creating perception; with AI tools [of voice and video modulation] and a click, you can turn the perception on its head in a minute,” said Arun Reddy, the national coordinator for social media at the Congress, who oversaw the party’s tech-savvy Telangana election. He added that the team was bursting with ideas to incorporate AI in campaigning, but that they didn’t have enough “trained people” to execute them all.
Reddy is strengthening his team – as are other parties.
“AI will have a resounding effect in creating the narrative,” Reddy told Al Jazeera. “The political AI-manipulated content will increase multifold, much more than what it ever was.”
‘Campaigns are getting weirder’
From the desert town of Pushkar in western India, 30-year-old Divyendra Singh Jadoun runs an AI startup, The Indian Deepfaker. Launched in October 2020, his company cloned the voice of Rajasthan state’s Congress chief ministerial candidate Ashok Gehlot for his team to send personalised messages on WhatsApp, addressing each voter by their name, during November assembly elections. The Indian Deepfaker is currently working with the team of Sikkim’s Chief Minister Prem Singh Tamang for holograms during upcoming campaigns. Sikkim is one of India’s smallest states in the northeast, perched on the Himalayas between India, Bhutan and China.
That’s the clean, official work, he said. But in recent months, he has been swamped by what he describes as “unethical requests” from political campaigns. “The political parties reach out indirectly via international numbers on WhatsApp, burner handles on Instagram, or connect on Telegram,” Jadoun told Al Jazeera in a phone interview.
In the November election, his company denied more than 50 such requests, he said, where potential clients wanted videos and audio altered to target political opponents, including with pornography. As a startup, Jadoun said his company is particularly careful to avoid any legal trouble. “And it is a very unethical use of AI,” he added. “But I know many people who are doing it for very low prices and are readily available now.”
During the election campaigns for the state legislatures of Madhya Pradesh in central India and Rajasthan in the west last November, police registered multiple cases for deepfake videos targeting senior politicians including Modi, Shivraj Singh Chauhan, Kailash Vijayvargia (all BJP) and Kamal Nath (Congress). The deepfake content production is often outsourced to private consulting firms, which rely on social media networks for distribution, spearheaded by WhatsApp.
A political consultant who requested anonymity told Al Jazeera that numbers of ordinary citizens with no public profile are registered on WhatsApp and used for the campaigns to make it harder for anyone to directly trace them back to parties, candidates, consultants and AI firms.
This consultant ran six campaigns in assembly elections last year for both the BJP and Congress. “In Rajasthan, we were using phone numbers of construction labourers to run our network on WhatsApp,” they said, “where deepfakes were primarily circulated.”
Meanwhile, AI-manipulated audios are particularly valuable tools in smaller constituencies, “targeting candidates with forged call recordings about arranging ‘black money’ for elections or threatening someone to buy votes,” the consultant said, whose own candidate was targeted with one such recording. The recordings are generally masked with candidates’ voices to cast them as evidence of corruption.
“Manipulating voters by AI is not being considered a sin by any party,” they added. “It is just a part of the campaign strategy.”
India has 760 million internet users – more than 50 percent of the population – behind only China.
Among all the requests, one from a constituency in southern Rajasthan stood out to Jadoun. Ahead of the state election in November, the caller requested that Jadoun alter a problematic but authentic video of their candidate – whose party he did not disclose – to make a realistic deepfake. The aim: to claim that the original was a deepfake, and the deepfake the original.
“The opposition had a troubling video of their candidate and they wanted to spread it quicker on social media to claim it is a deepfake,” he said, bursting into awkward laughter. “Political campaigns are getting weirder.”
Threats to election integrity
Indian laws currently do not define “deepfakes” clearly, said Anushka Jain, a policy researcher at Goa-based Digital Futures Lab. The police have been using laws against defamation, fake news or violation of a person’s modesty, combined with the Information Technology Act, to try and tackle individual cases. But often, they’re playing whack-a-mole.
“The police are prosecuting on the effect of the deepfake and not because it is a deepfake itself,” she said.
Analysts say that the Election Commission of India (ECI), an autonomous body that conducts polling, needs to catch up with the shifting nature of political campaigns.
In the days leading up to the voting in Telangana state elections last year, ruling Bharat Rashtra Samithi party leaders repeatedly warned their followers on social media to stay alert against deepfakes deployed by the Congress party. They also appealed to the ECI against the deepfake clip that the Congress shared on the morning of the vote.
But the video remains online and the party never received notice from the ECI, two Congress leaders aware of the issue told Al Jazeera.
Al Jazeera has sought comments from the ECI but is yet to receive a response.
“Even if one person is misled into believing something and that changes his mind, it vitiates the purity of the election process,” said SY Quraishi, former chief election commissioner of India. “Deepfakes have made the problem of rumour-mongering during the polls graver by a thousand times.”
Quraishi said that deepfakes need to be moderated in real time to minimise the damage they can cause to Indian democracy.
“The ECI needs to take action before the damage is done,” he said. “They need to be a lot more prompt.”
‘Truth is out of reach’
The Indian government has been pressing major tech companies, including Google and Meta, to actively make efforts to moderate deepfakes on their platforms. IT minister Rajeev Chandrasekhar has met officials from these firms as part of deliberations over the threats posed by deepfakes.
By asking the tech sector to take the lead, the government escapes any criticism that it is trying to selectively censor selective deepfakes, or that it is trying to crack down on emerging AI technologies more broadly.
But by passing the buck to private companies, the government is raising questions about the sincerity of its intent to regulate manipulative content, said Prateek Waghre, the executive director of India’s Internet Freedom Foundation, a leading New Delhi-based tech policy think-tank. “It is almost wishful thinking,” he said.
Arguing that the tech companies have not been able to deal with the existing problems with content moderation, Waghre said that “the rise of AI now” has compounded challenges. And the current approach to content moderation ignores what’s really at the heart of the problem, he said.
“You are not solving the problem,” he said. “The design [of algorithms] is just flawed.”
On February 16, major tech companies signed an accord at the Munich Security Conference to voluntarily adopt “reasonable precautions” to prevent artificial intelligence tools from being used to disrupt democratic elections around the world. But the vaguely worded pact left many advocates and critics disappointed.
YouTube has announced that it will enable people to request the removal of AI-generated or altered content that simulates an identifiable person, including their face or voice, using its privacy request process.
“I’m not very optimistic about the platform’s capabilities to detect deepfake,” said Ravi Iyer, managing director of the Neely Center for Ethical Leadership and Decision Making at the University of Southern California’s Marshall School of Business. “With low digital literacy and rising consumption of videos, this poses a grave risk to India’s election integrity.”
Identifying every AI-manipulated media is not a reasonable task, Iyer said, so companies need to redesign algorithms that don’t promote polarising content. “Companies are the ones with the money and resources, they need to take reasonable steps to tackle the rise of deepfakes,” he said.
The Internet Freedom Foundation has published an open letter urging electoral candidates and parties to voluntarily refrain from using deepfake technology ahead of the national elections. Waghre isn’t confident that many will bite, but he said it’s worth a try.
Meanwhile, political campaigns are bolstering their AI armouries – and some, like Reddy, the national coordinator for social media at Congress, concede that the future looks dark.
“Most people using AI are out there to distort the facts. They want to create a perception that’s not based on truth,” said Reddy. “Combine the penetration of social media in India with the rise of AI, the truth will be out of reach of people in the elections now.”