Using Ephemeral Resources in Terraform

Last year, I did a workshop at B-Sides in Idaho Falls on secrets management in Terraform. I had a pretty solid outline and felt confident in my ability to "do it live." What I hadn't counted on was a provider in an underlying module being deprecated that week and completely blowing the whole thing to bits.

It was rough.

Maybe this small discovery in the registry with a quick implementation is my shot at redemption.

(As always, the solution is up on GitHub so you can just look at it and not have to read the SEO slop.)

The Problem

Whether you were aware or not, Terraform stores everything in the state file. The state file is just JSON. So while a terraform state show resource.name may shoe (sensisitve_data) when you look at it, the JSON file knows what the value is. It has to. That has always meant locking down your remote states and using local states for anything super critical or sensitive. These both come with their own tradeoffs.

This is an issue that has always been on my radar. I believe everything can or should be done through code, so that little security wrinkle is something I could never let go. Likewise, I think that you should be able to use tools like Terraform to manage your keys and rotation of those keys.

The Solution

While taking another crack at this idea of rotating keys as code (RKaC?), I discovered a new resource type in the Terraform registry for the random provider. The provider has had the ability to create a password, but this idea of an ephemeral password was pretty exciting. This new type provides the ability to create a resource without the sensitive data being stored in the state at all.

The Implementation

I'll skip talking about the main.tf file because it's pretty dead simple.

What will talk about though is the resources.tf file, which is where all the magic happens:

ephemeral "random_password" "password_update" {
  length  = 16
  special = false
  upper   = true
  lower   = true
  numeric = true
}

resource "google_secret_manager_secret" "ephemeral_secret" {
  secret_id = "my-secret-id" // Replace with your secret ID
  replication {
    auto {}
  }
}

resource "google_secret_manager_secret_version" "ephemeral_secret_version" {
  secret         = google_secret_manager_secret.ephemeral_secret.id
  secret_data_wo = ephemeral.random_password.password_update.result
}

In order:

1. We create a random password that is ephemeral. This means that it will only be created on the next terraform apply and will be forgotten/ignored.
2. We create a new secret in Google Secret Manager.
3. We create a new version of the secret using the result from the random provider creating the password and sets it to the secret_data_wo which makes sure that the information is not stored in the state file.

On the next terraform apply, the password will be created and once the apply has completed it will be forgotten and that object itself will be ignored going forward. If you check the state file, you will see that there is no mention of the random_password.password_update resource and you will see that the secret_data_wo field for google_secret_manager_secret_version.ephemeral_secret_version has no value. This means that your state file is completely clean and the only thing that knows the value of the password is your secrets manager.

Assuming your IAM is configured correctly and you're using LUA and all of those other fun acronyms...

A Closing Thought

This is a big move for Terraform. I looked and the GCS Provider has also started adding ephemeral resources for keys and secrets. The less that information is in the state file, the better.

That all said, I can not stress enough the nature of this resource is exactly what is says on the tin...ephemeral. If you goof up and need it to come back, it won't, you'll just have to create a new resource. It's a price worth paying, but a fact worth knowing.