Mastering Terraform State Management: Learn to Recover from Disasters!

Effectively managing Terraform state is essential for maintaining reliable infrastructure as code workflows. In large enterprise environments, knowing how to modify and recover the state file during unforeseen issues is critical to keeping your deployments consistent, secure, and operational.

In this guide, we’ll walk through importing resources, moving resources, removing resources, unlocking state, and backing up or restoring state using the Terraform CLI, Azure CLI, and the Azure Portal.

We’ll cover all the common methods to ensure you’re comfortable managing, troubleshooting, and restoring Terraform state when needed.

• 1.Backup and Restore State
  • Backing up state before changes such as version upgrades or state modifications
  • Restoring state to recover from corruption, accidental state modifications and/or issues
• 2.Import Resources
  • The dreaded Resources already exist error, generally due to being deployed manually and are not in state
  • An incomplete terraform apply that failed but created the resource and didn’t write the resource to state
  • Importing resources to move state files (more so legacy environments or code being restructured)
• 3.Move Resources
  • This is generally due to module changes in code, for example resource keys may change when adding new features to modules or even provider updates
• 4.Remove Resources
  • Generally to clean up state i.e. you want to manually manage a resource for some bespoke solution
  • Removing resources for decommissioning activities
• 5.Unlock State
  • Generally when you run a terraform command that involves writing to state like an apply, the state file is locked to avoid any other modifications.
  • A graceful shutdown was not successful and the state file lock was not released.

1. Backup and Restore State

Note: You should always have a backup of your state file and its always best practice to take a backup before you make significant changes to state including performing any upgrades.

You can use the Terraform CLI to directly pull down a copy of your state file. Please keep in mind if you have your Terraform state stored in a remote storage account, that you will need to have network connectivity and permissions to the blob container (or S3 bucket) from where you are running the command.

In an Enterprise environment, you made not have direct access to the storage account, so here are some options:

• Use Terraform code blocks (import blocks, remove blocks and moved blocks)
• Use Azure Portal to manually download the state file
• Enable Blob versioning on the Terraform state blob container
• Execute the commands from a build agent

# Terraform CLI

Backup State

To save a local copy of the current state, use the following commands.

terraform init
terraform state pull > backup.tfstate

Restore State

First you need to edit the file backup.tfstate in Notepad or VSCode. Update the serial by increasing it by 1. The serial will need to be higher than the current state file serial before you push it up (please keep this in mind in the event that you have a very old backup and there have been changes since you took a backup)

{
  "version": 4,
  "terraform_version": "1.11.0",
  "serial": 12, <--- [Serial needs to be increased before pushing]
  "lineage": "cdce38ed-6720-ea86-1568-5cee9efc78fd",
...
}

Once you have increased the serial, run the following command:

terraform state push backup.tfstate

This will push the backup state file back to the remote backend.

> Azure Portal

Note: If Blob versioning is enabled, that is the recommended option rather than the following manual steps!

Backup State

Option 1: (blob versioning)

• Use blob versioning, you only need to restore as all write actions are version controlled.

Option 2: (snapshots)

1. Go to the storage account
2. Browse to the blob container and find the state file.
3. Click on Snapshots
4. Click Create snapshot

Option 3: (manual)

1. Go to the storage account
2. Browse to the blob container and find the state file.
3. Use Download to save a backup locally.
4. To restore, use Upload to replace the existing state file.

Restore State

Using blob versioning:

1. Go to the storage account
2. Browse to the blob container and find the state file.
3. Click on Versions
4. Locate the Version Id by Modified date that you want to restore
5. Click the 3 dots on the right hand side of the Version
6. Click Make current version

Using Snapshots:

1. Go to the storage account
2. Browse to the blob container and find the state file.
3. Click on Snapshots
4. Locate the Name by Modified date that you want to restore
5. Click the 3 dots on the right hand side of the Version
6. Click Promote

2. Import Resources

When a resource already exists in Azure but is missing from Terraform state, use terraform import to bring it under Terraforms control.

You will need to have the following ready before importing:

• Terraform code already written for the resource you want to import
• Know what the target resource reference will be in state
• The Resource ID of the resource you want to import.

Note: The Terraform resource page has import information: azurerm_network_security_group

# Terraform CLI

If there are existing resources deployed with similar code, you can run the following command to see what the reference will look like in state.

terraform state list

Example: module.nsg.azurerm_network_security_group.this

Import Command

To import the resource, run the following command and you should get a import successful message!

terraform import \
  module.nsg.azurerm_network_security_group.this \
  /subscriptions/<subscription_id>/resourceGroups/tf-resources/providers/Microsoft.Network/networkSecurityGroup/nsg-prod-ae-1

> Azure Portal

You can generally get the Resource ID from the Azure portal under the properties configuration blade.

1. Navigate to the Resource.
2. Open the Properties blade.
3. Copy the Resource ID and use it in the terraform import command.

#> Terraform Code

Import blocks were released in Terraform v1.5.0 and later. The feature allows you to import resources using Terraform code, now why would you want to do this?

The import will be executed within the code, this means that the import will run from the build agent (or client) that has permissions and network connectivity to manage and deploy resources to Azure. This is the recommended way if you have any network restrictions to execute terraform state commands locally.

Add the following to import.tf (or in any other Terraform configuration file)

import {
  to = module.nsg.azurerm_network_security_group.this
  id = "/subscriptions/<subscription_id>/resourceGroups/tf-resources/providers/Microsoft.Network/networkSecurityGroup/nsg-prod-1"
}

Run terraform apply once it has been applied once, it will not run again but its best to remove the code once the apply has run!

3.Move Resources

When refactoring Terraform code, you may need to move resources to avoid configuration changes or destroys. Use terraform state mv to update the state.

A simple example of this is to move a native resource into a module (technically you could remove it from state and then import it but a move would be a single command).

# Before
resource "azurerm_network_security_group" "old" {
...
}

# After
module "nsg3" {
  source  = "Azure/avm-res-network-networksecuritygroup/azurerm"
  ...
}

# Terraform CLI

Move in State

terraform state mv \
  azurerm_network_security_group.old \
  module.nsg3.azurerm_network_security_group.this

#> Terraform Code

Moved blocks were released in Terraform v1.3.x and later. This is the recommended way if you have any network restrictions to execute terraform state commands locally.

Add the following to moved.tf (or in any other Terraform configuration file)

moved {
  from = azurerm_network_security_group.old
  to = module.nsg3.azurerm_network_security_group.this
}

Run terraform apply once it has been applied once, it will not run again but its best to remove the code once the apply has run!

4. Remove Resources

You want to manually manage a VM outside of Terraform. To manually remove a resource from Terraform state without deleting it in Azure, use terraform state rm.

# Terraform CLI

Remove from State

terraform state rm \
  azurerm_linux_virtual_machine.vm

This removes the resource from Terraform state while leaving it in Azure.

#> Terraform Code

Removed blocks were released in Terraform v1.7 and later. More information on how to execute this can be found at: # Removing a Resource from the Terraform State Using the “removed” block

Note: There appears to be a limitation at the time of writing that you cannot remove instances of a module, you can only remove the entire module. For example you can only remove module.vm.azurerum_virtual_machine and not module.vm.azurerum_virtual_machine.myfavoritevm. The terraform state rm command is the recommendation option if this does not meet your requirements!

Add the following to removed.tf (or in any other Terraform configuration file) Note: You need to add the lifecycle parameter to ensure that the resource is not destroyed!

removed {  
  from = null_resource.null  
  
  lifecycle {  
    destroy = false  
  }  
}

Run terraform apply once it has been applied once, it will not run again but its best to remove the code once the apply has run!

5. Unlock State

Terraform locks the state during write operations. If an error leaves the lock in place, you need to manually unlock it.

Generally the error will appear with the LOCK_ID. You can use either of the following to release the Terraform state lock.

# Terraform CLI

terraform force-unlock -force <LOCK_ID>

You can find the lock ID in the error message or backend storage.

# Azure CLI

If you use an Storage Account for state backend and the lock is caused by a blob lease, you can break the lease:

az storage blob lease break \
  --account-name <storage_account_name> \
  --container-name <container_name> \
  --blob-name <state_file_name>

> Azure Portal

1. Go to the storage account.
2. Navigate to the blob container where your state file is stored.
3. Locate the state file, click Break Lease in the toolbar.

Closing Thoughts

I hope this guide gave you the clarity and confidence to manage your Terraform state like a pro. If you’d like to see these concepts in action, check out the accompanying video above where I walk through real-world examples step by step.

Thanks for reading, and I hope this helped simplify one of the most critical parts of Terraform workflows!