Cybersecurity experts are issuing stark warnings as criminal hackers rapidly adopt generative artificial intelligence, transforming widely used business tools into potent weapons for sophisticated attacks. New methods allow attackers to manipulate AI programs through simple emails or calendar invites, enabling them to steal sensitive data without triggering traditional security alarms.
The rapid and sometimes unsecured integration of AI into corporate environments is creating significant vulnerabilities. From hijacking AI coding assistants to automating entire ransomware campaigns, these emerging threats represent a fundamental shift in the cybersecurity landscape, challenging defenders to keep pace with AI-driven attacks.
Key Takeaways
- Hackers are exploiting generative AI tools to bypass security systems and steal corporate data.
- A recent supply-chain attack successfully hijacked AI assistants on over 1,000 machines to extract sensitive information.
- AI is being used to automate entire hacking campaigns, from finding vulnerabilities to determining ransom amounts.
- Experts warn that future threats could involve an attacker's AI collaborating directly with a victim's AI system.
A New Generation of Security Risks
While businesses explore the productivity benefits of generative AI, malicious actors are exploiting the same technology with alarming speed. The integration of AI into everyday software, from email clients to code editors, has created new pathways for cyberattacks that were previously theoretical.
Security professionals express concern that the rush to deploy AI features often overlooks the inherent risks. This mirrors the early days of cloud computing, where security measures lagged behind adoption. However, the capabilities of generative AI mean the potential for damage is far greater.
"It’s kind of unfair that we’re having AI pushed on us in every single product when it introduces new risks," said Alex Delamotte, a threat researcher at security company SentinelOne.
This rapid deployment, often driven by employees or executives eager to utilize new technology, can bypass standard IT security protocols, leaving companies exposed to novel threats that their existing defenses are not designed to detect.
Hijacking AI in Corporate Environments
Recent incidents demonstrate how attackers can manipulate AI tools that are already trusted within a company's network. These attacks leverage the AI's extensive access to internal systems to carry out malicious commands.
The First AI Hijacking Attack
In August, what is believed to be the first major attack of its kind targeted users of Nx, a popular code management platform. Attackers published poisoned software updates that were downloaded by hundreds of thousands of users. The malicious code was specifically designed to find and issue commands to AI coding assistants from companies like Google and Anthropic that were running on the victim's machine.
The compromised AI tools were instructed to locate and send back sensitive data, including account passwords and cryptocurrency wallets. According to reports, more than 1,000 machines were compromised in this supply-chain attack.
"What makes this attack special is that it is the first time that I know of that the attacker tried to hijack the AI running in the victim’s environment," explained Henrik Plate, a researcher at Endor Labs. He noted that the attacker could have used the same access to alter the company's source code.
In another example, a security test conducted by Dave Brauchler of NCC Group successfully tricked a client's AI program-writing assistant into executing commands that exposed the company's entire databases and code repositories. "We have never been this foolish with security," Brauchler commented on the state of AI implementation.
Exploiting Automation and Hidden Commands
Demonstrations at the recent Black Hat security conference highlighted how easily AI models can be manipulated. One presentation showed how an attacker could send an email containing hidden instructions. If a user asked their AI assistant to summarize the email, the AI would execute the hidden commands, such as finding and exfiltrating digital passwords.
A similar technique was effective against Google's Gemini AI. An email with hidden directives, without any attachment, was able to trick the AI's summarization feature. The AI generated a false summary claiming the user's account was compromised and provided the attacker's phone number for assistance, creating a highly convincing phishing scam.
The Rise of Agentic AI Threats
Security company Guardio has already demonstrated risks associated with "agentic AI," which can perform actions on behalf of a user. Researchers tricked the Comet browser extension from Perplexity into purchasing a watch from a fraudulent online store and following instructions from a fake banking email, all without direct human oversight.
AI as an Autonomous Attacker
Beyond manipulating existing AI tools, criminals are now using artificial intelligence to orchestrate entire cyberattacks. AI's ability to process vast amounts of information and automate complex tasks makes it a powerful tool for malicious purposes.
The AI company Anthropic recently reported it had uncovered a complete ransomware campaign operated by an individual using AI for every stage of the attack. The AI was used to:
- Identify vulnerable systems at a target company.
- Execute the attack to breach the network.
- Evaluate the stolen data for value.
- Suggest an appropriate ransom amount to demand.
This development is particularly concerning because advances in natural language processing mean the attacker does not need to be an expert coder to launch a sophisticated operation.
Finding New Flaws with AI
Advanced AI programs are also being deployed to find previously unknown software vulnerabilities, known as "zero-days." These are highly prized by hackers because no patches exist for them.
A recent competition held by the Pentagon's Defense Advanced Research Projects Agency (DARPA) showcased this capability. Seven teams using autonomous AI systems found a total of 18 new zero-day vulnerabilities within 54 million lines of open-source code. While the teams worked to patch the flaws, officials acknowledged that hostile actors are developing similar capabilities to find and exploit them.
The Next Wave of AI Threats
The cybersecurity community is bracing for even more advanced threats as AI technology evolves. The most alarming scenario involves a collision of these new techniques, where an attacker's AI gains entry into a network and then begins to communicate directly with the victim's internal AI systems.
This "bad guy AI collaborating with the good guy AI," as Delamotte described it, could allow for unprecedented levels of infiltration and data theft. Experts predict this will redefine how companies approach internal security and employee access to powerful tools.
Adam Meyers, senior vice president at CrowdStrike, offered a sobering prediction for the near future. "Next year," he said, "AI will be the new insider threat."