Cybercriminals have pulled a fast one on X’s AI chatbot, getting it to promote phishing scams through a sneaky technique known as “Grokking”. Here’s what you need to know about this alarming development.
We’re all familiar with the dangers of social engineering – the age-old tactic of manipulating individuals to reveal sensitive information or install malware. Traditionally, this has been carried out through phishing emails, texts, or phone calls. However, a new player has entered the scene: generative AI (GenAI).
In certain scenarios, GenAI and large language models (LLMs) integrated into popular online platforms can inadvertently aid in social engineering schemes. Recently, security experts raised the alarm about this happening on X (formerly Twitter). If you’ve never considered this a threat before, it’s time to start treating any responses from public-facing AI bots with caution.
So, how does ‘Grokking’ operate and why should you be concerned? AI poses a social engineering risk in two primary ways. Firstly, LLMs can be manipulated to craft convincing phishing campaigns on a large scale, generating deepfake audio and video to deceive even the most vigilant user. Yet, as X discovered, there’s another, more sinister danger: the “Grokking” technique.
In this malicious campaign, threat actors bypass X’s prohibition on links in promoted posts (aimed at combating malvertising) by sharing video card posts featuring clickbait videos. They embed their malicious link in the inconspicuous “from” field below the video. Here’s the twist: the threat actors then prompt X’s built-in GenAI bot Grok to identify the video’s source. Grok, upon reading the post, spots the tiny link and unwittingly amplifies it in its response.
Why is this approach so hazardous?
This ploy essentially transforms Grok into a malicious actor, leading it to repost a phishing link from its trusted account.
These paid video posts can reach millions of impressions, potentially disseminating scams and malware extensively.
The links will receive a boost in SEO and domain reputation, given Grok’s high level of trustworthiness.
Researchers uncovered numerous accounts engaging in this tactic until they were suspended.
The links themselves redirect users to credential-stealing forms and malware downloads, opening the door to account hijacking, identity theft, and other serious consequences.
This isn’t solely an issue for X and Grok. Similar techniques could theoretically be applied to any GenAI tools or LLMs integrated into a reputable platform. It underscores threat actors’ resourcefulness in circumventing security measures, as well as the risks users face when relying on AI-generated content.
The perils of prompt injection
Prompt injection is a form of attack where threat actors feed GenAI bots malicious instructions disguised as legitimate user queries. This can be done directly, by inputting these instructions into a chat interface, or indirectly, as seen in the Grok case.
In the latter scenario, the malicious instruction is typically concealed within data that the model is prompted to process as part of a legitimate task. For instance, a malicious link was hidden in video metadata beneath the post, and Grok was asked, “where is this video from?”.
These types of attacks are on the rise. Analysts at Gartner recently reported that a third of organizations had encountered prompt injection in the past year. Unfortunately, there are countless other scenarios where a scheme akin to the Grok/X instance could unfold.
Lessons learned: exercise caution with AI
There are countless variations of this threat, but the primary lesson is to never blindly trust the output of any GenAI tool. Always be wary, as threat actors may have duped the LLM into generating malicious content. They’re counting on your trust, but as demonstrated, malicious prompts can be concealed in various ways.
Additionally:
– When provided with a link by a GenAI bot, hover over it to inspect the destination URL before clicking.
– Maintain a healthy skepticism towards AI-generated content, especially if it seems out of place.
– Use robust, unique passwords (stored securely) and enable multi-factor authentication to reduce the risk of credential theft.
– Keep all your software and operating systems updated to minimize vulnerabilities.
– Invest in comprehensive security software from a reputable provider to block malware, phishing attempts, and suspicious activities.
AI tools have introduced a new battleground in the ongoing fight against phishing. Stay vigilant, question everything, and don’t assume that AI always has the correct answers.
