How To Sync Terraform Config And .tfstate File For Existing AWS Resources

Terraform is a great (and dominant) infrastructure automation tool. It is multi-cloud, can build all sorts of resources, and in some cases supports API calls to build resources before the native tooling from cloud providers does.

However, it’s dependent on a state file that is local, and only reflects resources created by terraform, and a local configuration file to describe resources. It’s not able to reach out to a cloud account and create a configuration and .tfstate file based on the existing resources that were built via another method. Or at least, it isn’t able to yet.

The scaffolding for this functionality exists within Terraform for the AWS cloud, and is called the “import” functionality. It’s able to map a single existing resource to a single configuration block for the same resource type and fill in the info, which is of course a manual and tedious process. And imagine if you have hundreds (or thousands!) of resources. It isn’t a feasible way to move forward.

Terraforming (link) is a wrapper around terraform and is able to map multiple resources at the same time to configuration blocks, as well as build .tfstate files for multiple existing resource types. Still, it’s a little awkward to use — only a single resource type is able to be imported at the same time, and if a command is run against a non-existing resource type (say you don’t have a batch configuration, and run a sync against the batch resource), it wipes out the existing .tfstate entirely, removing your progress.

Clearly, the tools could use some help. So I wrote some.

I imagine both of these tools (terraform import & terraforming) will eventually get this same functionality. In fact, both of these tools are open source, and I’ll work on adding this functionality natively to both of these tools via PRs.

However, for the time being, I’m publishing my code which permits:

Creating from scratch a .tfstate file for every terraforming supported resource in an AWS region. Creating a single (monolithic) configuration file for each existing resource in an AWS region

This code assumes you don’t have an existing .tfstate file — in fact, it wipes out your existing local .tfstate file and builds a new one. So please back up your .tfstate and configuration files before running this tool.

However, if your’e new to terraform and want to sync the configuration to an existing AWS region and look at the config for all the resources that exist there, this is a neat shortcut.

Rather than post the code here and update each time I (or you! The code is open source) update it.

Here’s the state of the code today:

# Generate .tf and .tfstate for existing AWS VPC resources
#  Author: Kyler Middleton, 7.9.2019

# Limitations/bugs
# SG rules are stripped of descriptions during import for some reason


#/bin/bash -e

# Establish directory for temp config files
echo "Creating temp directory to stash files"
mkdir temp

# Export single region
echo "Which region do you want to run against, e.g. us-east-2"
read region
export AWS_REGION=$region

# Find all files in AWS, generate terraform config for them in individual files in temp folder
echo "Utilizing terraforming to sync config files for all resources into temp folder"
terraforming help | grep terraforming | grep -v help | awk '{print "terraforming", $2, ">", "temp/"$2".tf";}' | bash

# Build the .tfstate file as empty so it can be iterated on
echo "Over-writing terraform.tfstate file with an empty shell"
cat <<EOL > terraform.tfstate
{
  "version": 1,
  "serial": 12,
  "modules": [
    {
      "path": [
        "root"
      ],
      "outputs": {
      },
      "resources": {
      }
    }
  ]
}
EOL

# Identify which resource .tf files aren’t empty (resources exist of that type) and generate tfstate for each existing resource
echo "Identify resource types which exist, generate tfstate for them"
find temp -type f -not -empty -name '*.tf' -exec basename {} \; | sed -e 's/.tf//' $1 | awk '{print "terraforming", $1, "--tfstate", "--merge=terraform.tfstate", "--overwrite";}' | bash

# Delete previous terraform_config.tf if it exists
echo "Removing terraform_config.tf old files"
rm -f terraform_config.tf

# Build single file with all .tf configuration in it
echo "Building a single terraform_config.tf file with configuration for all existing resources"
find temp -type f -not -empty -name '*.tf' -exec basename {} \; | sed -e 's/.tf//' $1 | awk '{print "terraforming", $1, ">>", "terraform_config.tf";}' | bash

# remove previous version.tf file if exists
echo "Removing versions.tf file which block 0.12upgrade process"
rm -f versions.tf

# Fixup file for terraform 0.12, will automatically response yes
echo "Auto-upgrading config and tfstate to 0.12 terraform standard"
echo "yes" | terraform 0.12upgrade

# Format all local scripts
echo "Formatting scripts via terraform fmt"
terraform fmt

# Remove temp directory, no longer needed
echo "Deleting temp directory and all contents"
rm -rd temp

And here’s the link to the GitHub repo where the current state is stored.

https://github.com/KyMidd/TerraformingAllResources

I hope it’s useful to you. Please add any corrections, comments, and feature additions you’d like via pull requests. And if you know how to update the terraform or terraforming source code to add these functionalities and make my code obsolete, please do so!

That would be the best case scenario.

Thanks all. Good luck out there!
kyler