How We Manage our Terraform State Outside of Our Cloud

How Other Companies Manage Their Terraform State

Every Implementation I've ever seen has used the cloud vendor's object storage implmentation, whether AWS's S3, Linode's Object Storage or Azure's Blob Storage.

The typical workflow goes something like this: 1. Manually create a bucket to hold your State. 2. Set up your your permissions to allow the machine running Terraform to be able to access the bucket. 3. Set up Terraform to use the bucket as a backend.

This model works for most applications - but it could be better.

Pitfalls of using Object Storage to Manage State

There are a lot of good reasons to manage your state in your cloud provider, but there are also reasons not to.

Vendor Lock-In

The number one reason that Dominion Solutions' Devops team decided to use an alternative location to store their terraform state in another location was due to the fact that we do not want to migrate the state from one provider to another. If we choose to leave our cloud provider, Linode tomorrow, it would take some adjustments to our terraform stacks, and we'd be off.

It Costs Money

Story time! When I was just getting started in my career, in order to make ends meet, I took a job doing something completely related to my chosen career path waiting tables in a diner. As a young man, I would regularly come up with ideas for how to improve the efficiency of various tedious tasks that folks in the diner would be responsible. I would bring them up to the owner, and every time I suggested an idea, his response was always the same "Costs Money, Son!" It wasn't until much later that I realized exactly what he was getting at. Why would you pay extra for something that you're already getting from another investment, or in this case, for free?

Dominion Solutions' Solution

We manage our Terraform state using GitLab Managed Terraform State. Doing so allows us to manage our state in the same code repository that we are using to create the terraform state. This keeps the operations tidy in that we no longer have to go find the correct file, and properly done, we can integrate the terraform deployment directly into our Gitlab Workflow.

Getting Started with GitLab Managed Terraform State

Setting up for the first time

We need to add the following line to our main.tf file

terraform {
  backend "http" {}
}

Support for Multiple Environments

We quickly realized that we wanted to use Terraform to manage Multiple Environments.

In our case, we are using the following environments: * Development * Staging * Production

This presented a challenge, as we tried different things for support across environments. Locally, we tried using Terraform Workspaces but this did not work out with the remote storage. Instead, we developed a script to change the environments.

#!/usr/bin/env bash
function processParams {
    echo "Process Params ${#}"
    if [[  0 -eq $# ]]
    then
        getHelp
    fi

    while [[ -n "$1" ]]
    do
        case $1 in
            -e | --environment)
                shift;
                STATE_NAME=$1
                echo "State Name: ${STATE_NAME}"
                shift;
            ;;
            -i | --initialize)
                INITIIALIZE=true
            ;;
            *)
                getHelp
            ;;
        esac
    done;
}

function getHelp {
    echo "changeEnv.sh -h
changeEnv.sh -e <newEnvironment>
    -e, --environment   The environment to switch to.
    -h, --help          Show this help message and exit"
    exit 0
}

source .env
INITIIALIZE=false
processParams $@

executable=("terraform init"
    -backend-config="address=https://gitlab.com/api/v4/projects/<project-id>/terraform/state/${STATE_NAME}"
    -backend-config="lock_address=https://gitlab.com/api/v4/projects/<project-id>/terraform/state/${STATE_NAME}/lock"
    -backend-config="unlock_address=https://gitlab.com/api/v4/projects/<project-id>/terraform/state/${STATE_NAME}/lock"
    -backend-config="username=${GITLAB_USER_NAME}"
    -backend-config="password=${GITLAB_ACCESS_TOKEN}"
    -backend-config="lock_method=POST"
    -backend-config="unlock_method=DELETE"
    -backend-config="retry_wait_min=5"
)
if [[ $INITIIALIZE -ne true ]]
then
 executable+=(-migrate-state)
fi
echo "${executable[@]}"

${executable[@]}

Bringing it all together

We use a custom image with both python and terraform installed. In our case we need both so that we can run ansible in tandem. With our gitlab-ci.yml, we bring it all together and use the whole process to get our state managed:

deploy_dev:
  stage: deploy
  before_script:
    - pip3 install -r requirements.txt
    - ansible-galaxy collection install -r requirements.yml
    - ./changeEnv.sh -e "${CI_ENVIRONMENT_NAME}"
  script:
    - terraform init
    - terraform apply -auto-approve -var-file=applications/dominion_solutions/env/${CI_ENVIRONMENT_NAME}.tfvars
    - ansible-playbook -i inventories/${CI_ENVIRONMENT_NAME} playbooks/main.yml
  environment: development
  rules:
    - if: $COMMIT_BRANCH == "development"

Potential risks of using GitLab for State Management

GitLab is moving away from continuing to provide the Terraform Builders in order to promote their OpenTofu tool. You can still use the GitLab-Created Terraform containers if you would like, and the instructions are included here: https://gitlab.com/gitlab-org/terraform-images

All in all, we still use Gitlab to manage our terraform state, and don't think their support for the state management is going anywhere anytime soon.

Dominion Solutions LLC is a GitLab Channel Partner.

Send us an email to get up to date pricing and get started.