A developer experienced a catastrophic data loss event after an AI coding assistant, following a series of commands, deleted an entire production environment. The incident wiped out a database containing two and a half years of records and even the associated backup snapshots.
The event serves as a critical case study on the risks of delegating powerful commands to AI agents without rigorous human supervision and robust safety protocols. While the data was eventually recovered, the incident highlights the unforgiving nature of infrastructure management tools when combined with AI's literal interpretation of instructions.
Key Takeaways
- An AI agent named Claude Code, tasked with managing cloud infrastructure, executed a command that deleted an entire production setup.
- The root cause was a human error involving a missing configuration file, which led the AI to misinterpret the state of the system.
- Over 2.5 years of database records and recovery snapshots for two separate websites were instantly erased.
- The developer, Alexey Grigorev, successfully recovered the data with assistance from Amazon Business support within approximately 24 hours.
- The incident prompted a complete overhaul of safety procedures, including restricting the AI from running destructive commands directly.
A Routine Migration Turns Critical
The situation began with a standard infrastructure task. Developer Alexey Grigorev intended to migrate his website, AI Shipping Labs, to Amazon Web Services (AWS). He planned for it to share the same cloud infrastructure as his other project, DataTalks.Club, to simplify management and reduce costs.
To manage this process, Grigorev used Terraform, a popular tool that allows developers to define and manage their cloud infrastructure using code. He employed an AI agent, Claude Code, to help execute the necessary Terraform commands.
What is Terraform?
Terraform is an "infrastructure as code" tool. Instead of manually clicking through web interfaces to set up servers, databases, and networks, developers write configuration files. Terraform reads these files and builds, changes, or destroys the infrastructure to match the code. This automates and standardizes the process, but also means a single command can have massive consequences.
The initial plan was straightforward. However, a crucial oversight set the stage for disaster. Grigorev instructed the AI agent to begin the setup process but forgot to provide it with a vital "state file." This file acts as a map, telling Terraform exactly what infrastructure already exists. Without it, the tool operates blindly, assuming it is starting from scratch.
The Unraveling Sequence of Events
Operating without the state file, Claude and Terraform saw the existing setup for DataTalks.Club as something that shouldn't be there. It began creating duplicate resources for the new website, AI Shipping Labs. Grigorev noticed the error and halted the process midway.
He then tasked the AI with identifying the newly created duplicate resources for cleanup. Believing he had addressed the immediate issue, he uploaded the correct state file. His assumption was that the AI would first remove the duplicates and then use the state file to correctly configure the environment.
This assumption proved to be incorrect. Once the AI agent had the state file, its logic shifted. Its primary goal became making the live environment match the instructions in the file. To do this cleanly, it initiated a `terraform destroy` command.
"I over-relied on the AI agent to run Terraform commands," Grigorev later admitted, reflecting on the critical misstep of allowing the agent to execute powerful actions without manual confirmation.
Because the state file described the entire shared infrastructure—including the long-running DataTalks.Club website—the destroy command targeted everything. In an instant, the servers, networks, load balancers, and, most critically, the database with 2.5 years of records were gone. Even the database snapshots, which Grigorev had considered his backups, were wiped out as they were part of the same infrastructure definition.
Total Data Loss
The single command resulted in the deletion of the entire setup for two websites, including a primary database and all its recovery snapshots, erasing 2.5 years of accumulated records.
Recovery and A Path Forward
Immediately following the data wipe, Grigorev contacted Amazon Business support. Fortunately, the support team was able to assist in a recovery effort. Within about a day, the lost data was successfully restored, averting a permanent disaster.
The close call prompted Grigorev to conduct a thorough post-mortem analysis and implement a new set of safeguards to prevent a recurrence. These new protocols serve as a valuable lesson for anyone integrating AI into development workflows.
New Safety Measures Implemented
- Manual Execution of Destructive Actions: The AI agent is no longer permitted to run potentially destructive commands like `destroy`. Grigorev will now have the AI generate a plan, which he will manually review and execute himself.
- Enhanced Delete Protections: He is applying delete protection flags within both Terraform and AWS permissions to prevent critical resources from being easily removed.
- Improved State Management: The Terraform state file has been moved from his local machine to a secure, remote S3 storage location, a best practice for team collaboration and safety.
- Regular Backup Testing: A new policy involves periodically testing the database restoration process to ensure backups are viable and the recovery procedure works as expected.
This incident is not a story of a rogue AI, but rather a stark reminder that AI agents are powerful tools that execute commands with logical precision but without human context or foresight. They follow the instructions they are given based on the information they have. As developers increasingly lean on AI for efficiency, this event underscores the non-negotiable need for human oversight, robust safety nets, and a healthy skepticism when delegating commands that can have irreversible consequences.