False AI-Generated Content Regarding the Iran War Floods X

When disinformation specialist Tal Hagin asked Grok to verify a post on X regarding Iranian missiles allegedly hitting Tel Aviv, Elon Musk’s AI chatbot faltered significantly.

Grok consistently misidentified the date and location of a video that was originally posted on X by an Iranian state-owned media outlet on Sunday. Subsequently, the chatbot attempted to justify its response by sharing an AI-generated image.

“Now Grok is responding with AI-generated nonsense,” Hagin remarked. “It’s cooked, I tell you.”

This interaction encapsulates the surreal environment that X has devolved into since the US and Israel intensified their offensive against Iran on February 28. As reported by WIRED at that time, the platform was inundated with disinformation from accounts disseminating fake and repurposed videos.

As the situation has progressed, the disinformation wave has only worsened. Recently, it has been amplified by AI-generated images and videos, while Grok has repeatedly provided inaccurate information when asked to validate claims made on the platform. AI-generated images are being circulated by paid accounts with blue checkmarks and Iranian officials, aiming to exaggerate the damage.

The rise of easily accessible AI image- and video-generation tools has resulted in increasingly sophisticated fake content. For instance, on March 2, Iranian officials and state media shared AI-generated videos depicting a high-rise building in Bahrain engulfed in flames. Many found these visuals to appear convincingly realistic: one image showing a US B-2 bomber being downed by Iran, with US troops captured, garnered over a million views before it was removed, while images of Delta Force members being apprehended by Iranian forces reached over 5 million views prior to deletion.

Some AI-generated content on X is less believable. One video claims to show Iranian forces producing missiles inside a cave, yet it has still been shared by numerous accounts, accumulating over a million views.

The Iranian government is also leveraging AI to promote blatantly antisemitic narratives. Accounts within a pro-regime propaganda network on X have shared AI-generated content depicting Orthodox Jews guiding American soldiers to war or celebrating American casualties, according to researchers from the Institute of Strategic Dialogue (ISD), who shared their findings with WIRED.

Additionally, several accounts in this pro-regime network circulated a fake video that reportedly showcased a group of young girls walking past President Donald Trump dressed only in underwear. The post received over 6.8 million views, per ISD, before being taken down, though it continues to be circulated by other accounts on X.

“What’s especially unique about this conflict is the significant increase in AI-generated content I find myself debunking,” Hagin tells WIRED. “This is likely because AI has advanced enough to deceive journalists, combined with the ease of creating this AI-generated material with no repercussions. The longer we operate without regulations against AI misuse, the greater the damage caused. I fear the surge of AI-driven fake news could push us beyond the point of a fact-based reality unless we implement change immediately.”

When the wave of AI-generated fakes began to overwhelm the platform last week, X announced it would briefly demonetize blue checkmark accounts that post AI-generated videos related to armed conflict without appropriate labeling. X did not respond to inquiries regarding the number of accounts it had demonetized since this policy was enacted. Until recently, several Iranian officials seemed to be paying X for its premium service, which provided their accounts with blue checkmarks, increased visibility, and opportunities to generate revenue from their posts.