How Drift Gets In
Infrastructure drift begins the moment someone makes a change outside of Terraform. This is almost never malicious and almost always understandable. An engineer gets paged at midnight, needs to open a security group to restore service, and does it in the console because it is faster. A third-party service adds a tag to a resource it manages. A cloud provider changes a default attribute on an existing resource type. Each of these is a legitimate action in its own context, and none of them announce themselves to your state file.
The result is that your Terraform state — the file that records what Terraform believes exists — and the actual state of the infrastructure quietly diverge. The longer this goes undetected, the more the gap widens. Teams that do not run terraform plan regularly are frequently surprised to discover, on the next deployment, that Terraform wants to undo changes that have been live for months. Sometimes those changes were intentional. Sometimes they were forgotten. Either way, the surprise is the problem.
Why the State File Is Not Reality
The Terraform state file is a snapshot, not a live view. When Terraform runs a plan or apply, it refreshes its knowledge of the world by querying the provider APIs — but between runs, it has no way of knowing what has changed. A state file that was accurate six months ago and has not been the basis of an apply since may be significantly wrong. Resources may have been deleted manually. Attributes may have drifted. New resources may exist that Terraform has never been told about.
The "set and forget" failure mode is particularly common in stable infrastructure. A team sets up a production environment with Terraform, deploys it, and then does not touch it for a year because it is working. During that year, accumulated console changes, provider migrations, and incident remediations all run outside Terraform. When someone finally runs terraform plan again, the delta between state and reality produces a terrifying list of planned changes — most of which, if blindly applied, would destroy things that are working.
Detecting Drift
The primary detection tool is terraform plan -refresh=true, which queries provider APIs to reconcile the state file with actual resource attributes before computing what changes would be needed. The exit code semantics matter here: exit code 0 means no changes, exit code 1 means an error, and exit code 2 means there are changes to apply. Automating on exit code 2 in a scheduled CI job — running a speculative plan that never applies but alerts when the output is non-empty — gives you continuous drift detection without manual oversight.
For resources that exist in the cloud but not in Terraform state at all, terraform import is the mechanism for bringing them under management. This is more labour-intensive: you must write the resource configuration, then import the existing resource ID, then run a plan to confirm the configuration matches reality. But it is the correct path for closing the gap between what exists and what Terraform knows about, particularly after an incident where resources were created manually to restore service.
Structural Prevention
Detection is a reactive measure. Prevention is structural. The most effective preventive control is removing — or at minimum restricting — console write access to production infrastructure. If the only path to a production change is a pull request that runs through Terraform, drift cannot occur through the console. AWS Service Control Policies, Azure Policy, and OPA or Sentinel in Terraform Cloud can enforce this at the API layer, blocking direct resource modifications that bypass the IaC pipeline entirely.
Workspace isolation and state locking address a different drift vector: concurrent Terraform operations. State locking prevents two applies from running simultaneously against the same state file, which would produce corrupted state. Workspace isolation ensures that separate environments — development, staging, production — each have their own state file and cannot interfere with each other. Neither of these prevents a developer from making a manual console change, but they prevent the class of drift caused by Terraform operations themselves running in inconsistent conditions.
Remediation Without Destruction
When drift is detected, the immediate instinct is to run terraform apply and restore the declared state. This is the wrong instinct. The correct first question is: was this drift intentional? If a security group rule was added manually to respond to an incident, removing it by applying Terraform may restore the outage you just fixed. The plan output needs to be reviewed by someone who understands the context of the changes, not treated as a machine-safe operation.
The remediation options are: update the Terraform configuration to codify the intentional drift and commit it, or revert the manual change and let Terraform's state win. The choice depends entirely on whether the drift represents a decision that should be preserved. The worst outcome is a blind apply that destroys working infrastructure because no one reviewed what the plan was actually going to do.
Drift as a First-Class Concern
The teams that handle drift well have stopped treating it as an occasional problem to fix and started treating it as a metric to monitor. Scheduled drift detection runs — daily or more frequently in active environments — produce alerts when plans are non-empty. Pull request gates require a clean plan before merging infrastructure changes. Post-incident reviews include a Terraform reconciliation step that codifies any manual changes made during the incident.
The underlying principle is that infrastructure drift is not a Terraform problem or a cloud provider problem — it is a process problem. The tools for detecting and preventing it exist and work. What fails is the cultural default of treating Terraform as a deployment tool rather than as the authoritative definition of what your infrastructure should be. When the state file is treated as truth, drift becomes visible immediately. When it is treated as a rough guide, drift accumulates until it causes an incident.