Skip to content
Snippets Groups Projects
Select Git revision
No matching results
An error occurred while fetching branches. Retry the search.
An error occurred while fetching tags. Retry the search.
Blame Permalink
README.md 20.50 KiB

OpenTofu CI/CD Component

This project is home to the OpenTofu CI/CD component and it's related assets, like the gitlab-tofu wrapper script and OCI images containing that script together with an OpenTofu version.

Read more:

Note: Please make sure to use a released version of this CI/CD component. You find all releases on the Releases Overview Page.

:recycle: Migrating from the Terraform CI/CD templates? Check this out.

Usage

include:
  - component: $CI_SERVER_FQDN/components/opentofu/full-pipeline@<VERSION>
    inputs:
      # The version must currently be specified explicitly as an input,
      # to find the correctly associated images. # This can be removed
      # once https://gitlab.com/gitlab-org/gitlab/-/issues/438275 is solved.
      version: <VERSION> # component version
      opentofu_version: <OPENTOFU_VERSION>

stages: [validate, test, build, deploy, cleanup]

---

# ... or without the destroy jobs:
include:
  - component: $CI_SERVER_FQDN/components/opentofu/validate-plan-apply@<VERSION>
    inputs:
      # The version must currently be specified explicitly as an input,
      # to find the correctly associated images. # This can be removed
      # once https://gitlab.com/gitlab-org/gitlab/-/issues/438275 is solved.
      version: <VERSION> # component version
      opentofu_version: <OPENTOFU_VERSION>

stages: [validate, build, deploy]

A concrete example may look like this:

# Using version `0.10.0`:
include:
  - component: $CI_SERVER_FQDN/components/opentofu/full-pipeline@0.10.0
    inputs:
      # The version must currently be specified explicitly as an input,
      # to find the correctly associated images. # This can be removed
      # once https://gitlab.com/gitlab-org/gitlab/-/issues/438275 is solved.
      version: 0.10.0 # component version
      opentofu_version: 1.6.1

stages: [validate, test, build, deploy, cleanup]

---

# ... in case you absolutely know what you are doing and are
# aware that this may introduce breaking changes, you may use the latest release:
include:
  - component: $CI_SERVER_FQDN/components/opentofu/full-pipeline@~latest
    inputs:
      # The version must currently be specified explicitly as an input,
      # to find the correctly associated images. # This can be removed
      # once https://gitlab.com/gitlab-org/gitlab/-/issues/438275 is solved.
      version: latest # component version
      opentofu_version: 1.6.1

stages: [validate, test, build, deploy, cleanup]

Or import all jobs as hidden templates ready to be extended:

include:
  - component: $CI_SERVER_FQDN/components/opentofu/job-templates@<VERSION>
    inputs:
      # The version must currently be specified explicitly as an input,
      # to find the correctly associated images. # This can be removed
      # once https://gitlab.com/gitlab-org/gitlab/-/issues/438275 is solved.
      version: <VERSION> # component version
      opentofu_version: <OPENTOFU_VERSION>

stages: [...]

fmt:
  extends: [.opentofu:fmt]

...

OpenTofu Version

The OpenTofu version can be specified with the opentofu_version input. More details can be found here.

Base Image OS

The GitLab OpenTofu images come in multiple base image variants:

  • alpine (default)
  • debian

The base image OS can be specified with the base_os input.

GitLab-managed Terraform state backend

This component - by leveraging the gitlab-tofu CLI internally - automatically configures the GitLab-managed Terraform state backend. The only thing required is that the Terraform configuration must specify an empty http backend block, like this:

terraform {
  backend "http" {}
}

We recommend having a dedicated backend.tf file inside your root_dir with the aforementioned block.

Access to Terraform Module Registry

Similar to automatically configuring the [GitLab-managed Terraform state backend] the component also sets up credentials to authenticate with the Terraform Module Registry of the project the pipeline runs in. It basically sets the TF_TOKEN_<domain> variable to the $CI_JOB_TOKEN, where <domain> is the GitLab instance domain, for example for GitLab.com this would set TF_TOKEN_gitlab_com to the $CI_JOB_TOKEN. However, it'll only do so if the variable is not already provided. Thus, if you want to authenticate differently or to another Terraform Module Registry, you may just provide the TF_TOKEN_<domain> variable yourself, e.g. via CI/CD variables.

Access to GitLab via glab or GitLab Terraform Provider

The GitLab CLI glab is pre-installed in all the images. If you want to use glab or the GitLab Terraform Provider we recommend configuring a CI/CD variable called GITLAB_TOKEN. This will automatically authenticate both tools. For glab you can just start using it, for the Terraform Provider you just need to define the provider requirement. Tofu will do the rest.

Opinionated Templates

This component repository also provides some templates that may often be used, for example one that only runs validation (fmt and validate), plan and an apply, but no destructive actions.

Job Templates

Instead of including the full-pipeline or another opinionated template, it's also possible to include individual jobs and compose your own pipeline, for example, to just run the fmt job you can do:

include:
  - component: $CI_SERVER_FQDN/components/opentofu/fmt@<VERSION>
    inputs:
      # The version must currently be specified explicitly as an input,
      # to find the correctly associated images. # This can be removed
      # once https://gitlab.com/gitlab-org/gitlab/-/issues/438275 is solved.
      version: <VERSION>
      opentofu_version: 1.6.1
      root_dir: tofu/

Or you can also include the job-templates template, that will include all available OpenTofu jobs as hidden job templates prefixed with .opentofu:. Those are especially useful when you want to minimize your includes and you want to extend the jobs:

include:
  - component: $CI_SERVER_FQDN/components/opentofu/job-templates@<VERSION>
    inputs:
      # The version must currently be specified explicitly as an input,
      # to find the correctly associated images. # This can be removed
      # once https://gitlab.com/gitlab-org/gitlab/-/issues/438275 is solved.
      version: <VERSION>
      opentofu_version: 1.6.1

plan:
  extends: [.opentofu:plan]
  parallel:
    matrix:
      - TF_ROOT: test/
      - TF_ROOT: prod/

Have a look at the full-pipeline for how it's constructed.

The following job components exist:

  • fmt: Check formatting of configuration files.
  • validate: Validate configuration.
  • test: Test configuration.
  • plan: Plan an apply or destroy.
  • apply: Apply a configuration.
  • destroy: Destroy a configuration.
  • delete-state: Delete the GitLab-managed Terraform state.
  • custom-command: Run a custom OpenTofu command.
  • module-release: Release an OpenTofu module to the GitLab Terraform Module Registry.

Have a look at the individual template spec to learn about the available inputs.

Inputs

Please checkout the individual templates for the input definitions. The catalog page beautifully renders the inputs for each templates - check it out!

Available OpenTofu Versions

The following OpenTofu versions are available with this component via the opentofu_version input:

Variables

(:construction: This section is work in progress)

Have a look at the src/gitlab-tofu.sh script and how the TF_-prefixed variables are being used. You may set them according to your needs.

Auto-forwarded predefined CI variables

The gitlab-tofu script auto-forwards some "popular" predefined CI/CD variables as OpenTofu variables.

The forwarded variables are:

  • CI_JOB_ID
  • CI_COMMIT_SHA
  • CI_JOB_STAGE
  • CI_PROJECT_ID
  • CI_PROJECT_NAME
  • CI_PROJECT_NAMESPACE
  • CI_PROJECT_PATH
  • CI_PROJECT_URL

To use them in your OpenTofu configuration you can define a string variable with the same name but in lower snake_case. For example the CI_PROJECT_NAME CI/CD variable can be accessed in the OpenTofu configuration like this:

variable "ci_project_name" {
  type        = string
  description = "The name of the directory for the project."
}

Install additional tools

The gitlab-opentofu container image deliberately comes with minimal tooling to keep the image size small and be the least common denominator for our users.

However, it is sometimes necessary to install additional tools. To do that you can overwrite the included jobs with a before_script entry. The gitlab-opentofu image uses alpine as its base image and therefore apk can be used to install the tools. For example to install jq:

include:
  - component: $CI_SERVER_FQDN/components/opentofu/validate-plan@<VERSION>
    inputs:
      version: <VERSION>
      opentofu_version: 1.6.1

plan:
  before_script:
    - apk add jq

Best Practices

This section is a collection of some best practices. Feel free to contribute more that generally apply. If a best practice really becomes the de-facto standard we may make it the default behavior if possible.

Lockfile Handling

If you commit the Lockfile (.terraform.lock.hcl) to your repository we recommend setting either the TF_INIT_FLAGS (handled by this component) or TF_CLI_ARGS_init (handled by OpenTofu directly) to -lockfile=readonly to prevent any changes to the lockfile during the pipeline job and with that ensuring that OpenTofu really uses the locked dependencies.

Examples

Here are some example repositories to demonstrate how this component maybe used:

  • timofurrer/opentofu-test: uses multiple environments configured for different kinds of pipelines with a single branch.

Please contribute your own examples!

Releases & Versioning

This project currently releases tagged commits. An overview of releases can be found on the Releases page and a Changelog can be found here.

Each release is accessible in the CI/CD Catalog.

Component Versions

The component release versions follow Semantic Versioning 2.0.0.

Image Versions

This project releases multiple OCI image variants that can be used with the component. The intention is that the images used in a component have the same version and or not mixed. Due to the limitations described in https://gitlab.com/gitlab-org/gitlab/-/issues/438275 it's currently required to provide the component version in the component include field and as the version input. Check out the Usage section for examples.

There are alpine and debian variants available.

Each component release deploys the following images:

  • $CI_TEMPLATE_REGISTRY_HOST/components/opentofu/gitlab-opentofu:<VERSION>-opentofu<OPENTOFU_VERSION>-<OS_VARIANT>
  • $CI_TEMPLATE_REGISTRY_HOST/components/opentofu/gitlab-opentofu:<VERSION>-opentofu-<OS_VARIANT>
    • Includes the latest stable OpenTofu version at the time of releasing the component
  • $CI_TEMPLATE_REGISTRY_HOST/components/opentofu/gitlab-opentofu:<VERSION>-<OS_VARIANT>
    • Includes the latest stable OpenTofu version at the time of releasing the component

In the above examples <VERSION> references the component version, <OPENTOFU_VERSION> an OpenTofu release, from here and OS_VARIANT either alpine or debian.

The release notes contain a full list of images deployed to the registry.

Note: unfortunately, these image versions are not SemVer compatible, because - indicates a prerelease (which they are not in this case). However, we cannot use the alternative + which would indicate build metadata as we'd like. See https://github.com/distribution/distribution/issues/1201

Image Signing

Every released image is signed using sigstore/cosign.

You can use the following command to verify the signatures:

VERSION=X.Y.Z # put a released components/opentofu version here
IMAGE_REF=... # put a released components/opentofu image reference here
cosign verify "${IMAGE_REF}" --certificate-identity="https://gitlab.com/components/opentofu//.gitlab-ci.yml@refs/tags/${VERSION}" --certificate-oidc-issuer="https://gitlab.com"

For example, for image ref registry.gitlab.com/components/opentofu/gitlab-opentofu:0.34.0-opentofu1.6.0-alpine and version 0.34.0:

cosign verify "registry.gitlab.com/components/opentofu/gitlab-opentofu:0.34.0-opentofu1.6.0-alpine" \
    --certificate-identity "https://gitlab.com/components/opentofu//.gitlab-ci.yml@refs/tags/0.34.0" \
    --certificate-oidc-issuer "https://gitlab.com"

For self-managed mirrors the OIDC issuer must be changed, too.

Using with Renovate

To keep the component versions up to date you could use Renovate.

Renovate users who use the component input opentofu_version should include the following extends so that the OpenTofu version is raised to a maximum of the version suitable for the component:

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["local>components/opentofu"],
  ...
}

The above renovate config allows to update the version input together with the component include version if the version input has a # component version comment suffix, like so:

include:
  - component: $CI_SERVER_FQDN/components/opentofu/validate-plan-apply@<VERSION>
    inputs:
      # The version must currently be specified explicitly as an input,
      # to find the correctly associated images. # This can be removed
      # once https://gitlab.com/gitlab-org/gitlab/-/issues/438275 is solved.
      version: <VERSION> # component version
      opentofu_version: <OPENTOFU_VERSION>

stages: [validate, test, build, deploy, cleanup]

(You may need to adjust the path to the components/opentofu to match your mirror.) Fore more details refer to the Renovate documentation.

Some more example configurations for your renovate.json:

  • Package Rule to update all CI-Components 
    {
  		"matchFileNames": [
  			".gitlab-ci.yaml",
      ".gitlab-ci.yml",
  			"templates/**/*.yaml",
      "templates/**/*.yml"
  		],
  		"groupName": "Pipeline",
  		"semanticCommitType": "ci",
  		"automerge": true
  },
  • Package rule to pin only major.minor versions: 
    {
  	  "matchManagers": ["gitlabci"],
  	  "extractVersion": "^(?<version>\\d+\\.\\d+)"
  },
  • Package rule to target a specific component: 
    {
    "matchPackageNames": ["components/opentofu"],
    "matchManagers": ["gitlabci"]
},

Example Repositories:

Usage on self-managed

GitLab CI/CD components are not yet distributed and available on self-managed GitLab instances. (see details here). It's also not possible to just include CI/CD components across instance, thus an include like - component: gitlab.com/components/opentofu/full-pipeline@~latest won't work from a self-managed instance. However, you could mirror this project from GitLab.com onto any self-managed instance using a repository pull mirror.

If the component is being mirrored to another path than components/opentofu, then you also need to change that path in the include:component and additionally provide the correct image_registry_base input.

See also the official GitLab documentation for it here.

If you want to save runner resources you may disable the unit and integration tests by setting the SKIP_TESTS CI/CD variable to true.

The pipeline of this component respects the GitLab Dependency Proxy configuration by detecting the CI_DEPENDENCY_PROXY_DIRECT_GROUP_IMAGE_PREFIX environment variable and configuring buildah to use it when building the container images.

Migrating from the Terraform CI/CD templates

When migrating from the GitLab Terraform CI/CD templates you can use the following migration rules:

  • Used Terraform.gitlab-ci.yml -> Migrate to validate-plan-apply.
  • Used Terraform/Base.gitlab-ci.yml -> Migrate to job-templates.
    • Migrate the .terraform: job prefix to .opentofu:.
  • Used the kics-iac-sast job -> Additionally include the Jobs/SAST-IaC.latest.gitlab-ci.yml template.
  • Migrate the following job names:
    • build -> plan
    • deploy -> apply
  • Migrate the TF_ROOT variable to the root_dir input.
    • Although the TF_ROOT variable is still used and maybe overwritten after the import on individual jobs.
  • Migrate the TF_STATE_NAME variable to the state_name input.
    • Although the TF_STATE_NAME variable is still used and maybe overwritten after the import on individual jobs.
  • Migrate the TF_AUTO_DEPLOY variable to custom rules inputs.
  • Used other variables -> Use the same variables with this component.

The same rules apply for the latest templates. We also recommend to check out the Usage section for more details about the available templates and inputs.

OpenTofu component inputs vs. Terraform template variables

This OpenTofu CI/CD component makes use of inputs whereas the Terraform CI/CD templates used variables. We recommend that you use the inputs with the OpenTofu component where available and required. However, if needed you may overwrite the jobs and set the variables you like.

Can I use this component with Terraform?

Probably. Although, we don't officially support it or maintain any compatibility layer where necessary.

The OpenTofu CI/CD component job mainly interface with the gitlab-tofu script that is distributed with the gitlab-opentofu container image used as the base image for the jobs. This base image also contains the tofu binary.

If you'd want to use Terraform instead you may provide your own container image that contains at least a script called gitlab-tofu so that it's compatible with the component jobs. Everything else in the job can be custom, like replacing tofu with terraform.

You may provide the image_registry_base input to any of the component includes, pointing to the container registry URI hosting the container image. The container image name can be configured in the image_name input. The image has be versioned so that it is compatible with the image versioning of this project.

Contributing

See the CONTRIBUTING.md guide.