Skip to content
README.md 14.6 KiB
Newer Older
Timo Furrer's avatar
Timo Furrer committed
<!-- This document is generated by `make docs` from `.gitlab/README.md` -->

# OpenTofu CI/CD Component
Fabio Pitino's avatar
Fabio Pitino committed

This project is home to the **OpenTofu CI/CD component** and it's related assets,
Timo Furrer's avatar
Timo Furrer committed
like the `gitlab-tofu` wrapper script and OCI images containing that script
together with an OpenTofu version.

Read more:

- [CI/CD components](https://docs.gitlab.com/ee/ci/components)
- [Development guide for GitLab CI/CD components](https://docs.gitlab.com/ee/development/cicd/components)
- [CI/CD Catalog](https://docs.gitlab.com/ee/ci/components/index.html#cicd-catalog)

**Note**: Please make sure to use a released version of this CI/CD component.
You find all releases on the [Releases Overview Page](https://gitlab.com/components/opentofu/-/releases).

♻️ **Migrating from the Terraform CI/CD templates?** Check **[this](#migrating-from-the-terraform-cicd-templates)** out.

[[_TOC_]]

Timo Furrer's avatar
Timo Furrer committed
## Usage

```yaml
include:
  - component: $CI_SERVER_FQDN/components/opentofu/full-pipeline@<VERSION>
Timo Furrer's avatar
Timo Furrer committed
    inputs:
      # The version must currently be specified explicitly as an input,
Timo Furrer's avatar
Timo Furrer committed
      # to find the correctly associated images. # This can be removed
Timo Furrer's avatar
Timo Furrer committed
      # once https://gitlab.com/gitlab-org/gitlab/-/issues/438275 is solved.
      version: <VERSION>
      opentofu_version: <OPENTOFU_VERSION>
Timo Furrer's avatar
Timo Furrer committed

Timo Furrer's avatar
Timo Furrer committed
stages: [validate, test, build, deploy, cleanup]
Timo Furrer's avatar
Timo Furrer committed

---

# ... or without the destroy jobs:
include:
  - component: $CI_SERVER_FQDN/components/opentofu/validate-plan-apply@<VERSION>
Timo Furrer's avatar
Timo Furrer committed
    inputs:
      # The version must currently be specified explicitly as an input,
      # to find the correctly associated images. # This can be removed
      # once https://gitlab.com/gitlab-org/gitlab/-/issues/438275 is solved.
      version: <VERSION>
      opentofu_version: <OPENTOFU_VERSION>

stages: [validate, build, deploy]
Timo Furrer's avatar
Timo Furrer committed
```

A concrete example may look like this:

```yaml
# Using version `0.10.0`:
Timo Furrer's avatar
Timo Furrer committed
include:
  - component: $CI_SERVER_FQDN/components/opentofu/full-pipeline@0.10.0
Timo Furrer's avatar
Timo Furrer committed
    inputs:
      # The version must currently be specified explicitly as an input,
Timo Furrer's avatar
Timo Furrer committed
      # to find the correctly associated images. # This can be removed
Timo Furrer's avatar
Timo Furrer committed
      # once https://gitlab.com/gitlab-org/gitlab/-/issues/438275 is solved.
      version: 0.10.0
Timo Furrer's avatar
Timo Furrer committed
      opentofu_version: 1.6.1
Timo Furrer's avatar
Timo Furrer committed

Timo Furrer's avatar
Timo Furrer committed
stages: [validate, test, build, deploy, cleanup]
Timo Furrer's avatar
Timo Furrer committed

Timo Furrer's avatar
Timo Furrer committed
---

# ... in case you absolutely know what you are doing and are
# aware that this may introduce breaking changes, you may use the latest release:
Timo Furrer's avatar
Timo Furrer committed
include:
  - component: $CI_SERVER_FQDN/components/opentofu/full-pipeline@~latest
Timo Furrer's avatar
Timo Furrer committed
    inputs:
      # The version must currently be specified explicitly as an input,
Timo Furrer's avatar
Timo Furrer committed
      # to find the correctly associated images. # This can be removed
Timo Furrer's avatar
Timo Furrer committed
      # once https://gitlab.com/gitlab-org/gitlab/-/issues/438275 is solved.
      version: latest
Timo Furrer's avatar
Timo Furrer committed
      opentofu_version: 1.6.1
Timo Furrer's avatar
Timo Furrer committed

Timo Furrer's avatar
Timo Furrer committed
stages: [validate, test, build, deploy, cleanup]
Timo Furrer's avatar
Timo Furrer committed
```

Or import all jobs as hidden templates ready to be extended:

```yaml
include:
  - component: $CI_SERVER_FQDN/components/opentofu/job-templates@<VERSION>
    inputs:
      # The version must currently be specified explicitly as an input,
      # to find the correctly associated images. # This can be removed
      # once https://gitlab.com/gitlab-org/gitlab/-/issues/438275 is solved.
      version: <VERSION>
      opentofu_version: <OPENTOFU_VERSION>

stages: [...]

fmt:
  extends: [.opentofu:fmt]

...
```

### Opinionated Templates

This component repository also provides some templates that may often be used,
for example one that only runs validation (`fmt` and `validate`), plan and an apply,
but no destructive actions.

Timo Furrer's avatar
Timo Furrer committed
- [`validate-plan`](templates/validate-plan.yml)
- [`validate-plan-apply`](templates/validate-plan-apply.yml)
- [`validate-plan-destroy`](templates/validate-plan-destroy.yml)
Timo Furrer's avatar
Timo Furrer committed
Instead of including the `full-pipeline` or another opinionated template,
it's also possible to include individual jobs
Timo Furrer's avatar
Timo Furrer committed
and compose your own pipeline, for example, to just run the `fmt` job you can do:

```yaml
include:
  - component: $CI_SERVER_FQDN/components/opentofu/fmt@<VERSION>
Timo Furrer's avatar
Timo Furrer committed
    inputs:
      # The version must currently be specified explicitly as an input,
      # to find the correctly associated images. # This can be removed
      # once https://gitlab.com/gitlab-org/gitlab/-/issues/438275 is solved.
      version: <VERSION>
Timo Furrer's avatar
Timo Furrer committed
      opentofu_version: 1.6.1
Timo Furrer's avatar
Timo Furrer committed
      root_dir: tofu/
```

Timo Furrer's avatar
Timo Furrer committed
Or you can also include the `job-templates` template, that will include
all available OpenTofu jobs as hidden job templates prefixed with `.opentofu:`.
Those are especially useful when you want to minimize your includes and
you want to extend the jobs:

```yaml
include:
  - component: $CI_SERVER_FQDN/components/opentofu/job-templates@<VERSION>
Timo Furrer's avatar
Timo Furrer committed
    inputs:
      # The version must currently be specified explicitly as an input,
      # to find the correctly associated images. # This can be removed
      # once https://gitlab.com/gitlab-org/gitlab/-/issues/438275 is solved.
      version: <VERSION>
      opentofu_version: 1.6.1

plan:
  extends: [.opentofu:plan]
  parallel:
    matrix:
      - TF_ROOT: test/
      - TF_ROOT: prod/
```

Timo Furrer's avatar
Timo Furrer committed
Have a look at the [`full-pipeline`](templates/full-pipeline.yml) for how it's constructed.

The following job components exist:

- [`fmt`](templates/fmt.yml)
- [`validate`](templates/validate.yml)
Timo Furrer's avatar
Timo Furrer committed
- [`test`](templates/test.yml)
Timo Furrer's avatar
Timo Furrer committed
- [`plan`](templates/plan.yml)
- [`apply`](templates/apply.yml)
- [`destroy`](templates/destroy.yml)
- [`delete-state`](templates/delete-state.yml)
Timo Furrer's avatar
Timo Furrer committed
- [`custom-command`](templates/custom-command.yml)
Timo Furrer's avatar
Timo Furrer committed

Have a look at the individual template spec to learn about the available inputs.

Timo Furrer's avatar
Timo Furrer committed
### Inputs

| Name | Default | Description |
| ---- | ------- | ----------- |
Timo Furrer's avatar
Timo Furrer committed
| `stage_validate` | `validate` | Defines the validate stage. This stage includes the `fmt` and `validate` jobs. |
Timo Furrer's avatar
Timo Furrer committed
| `stage_test` | `test` | Defines the test stage. This stage includes the `test` job. |
Timo Furrer's avatar
Timo Furrer committed
| `stage_build` | `build` | Defines the build stage. This stage includes the `plan` job. |
| `stage_deploy` | `deploy` | Defines the deploy stage. This stage includes the `apply` job. |
| `stage_cleanup` | `cleanup` | Defines the cleanup stage. This stage includes the `destroy` and `delete-state` jobs. |
| `version` | `latest` | Version of this component. Has to be the same as the one in the component include entry. |
Timo Furrer's avatar
Timo Furrer committed
| `opentofu_version` | `1.7.1` | OpenTofu version that should be used. Must be one of `1.7.1`, `1.7.0`, `1.7.0-alpha1`, `1.6.2`, `1.6.1`, `1.6.0`. |
| `image_registry_base` | `$CI_REGISTRY/components/opentofu` | Host URI to the job images. Will be combined with `image_name` to construct the actual image URI. |
| `image_name` | `gitlab-opentofu` | Image name for the job images. Hosted under `image_registry_base`. |
Timo Furrer's avatar
Timo Furrer committed
| `root_dir` | `${CI_PROJECT_DIR}` | Root directory for the OpenTofu project. |
| `state_name` | `default` | Remote OpenTofu state name. |
| `auto_apply` | `false` | Whether the apply job is manual or automatically run. |
| `auto_destroy` | `false` | Whether the destroy job is manual or automatically run. |
| `plan_artifacts_access` | `none` | Access level for the plan artifact. See https://docs.gitlab.com/ee/ci/yaml/#artifactsaccess for possible values. |
Timo Furrer's avatar
Timo Furrer committed

### Available OpenTofu Versions

The following OpenTofu versions are available with this component via the `opentofu_version` input:

Timo Furrer's avatar
Timo Furrer committed
- [`1.7.1`](https://github.com/opentofu/opentofu/releases/tag/v1.7.1)
Ben Brown's avatar
Ben Brown committed
- [`1.7.0`](https://github.com/opentofu/opentofu/releases/tag/v1.7.0)
- [`1.7.0-alpha1`](https://github.com/opentofu/opentofu/releases/tag/v1.7.0-alpha1)
Timo Furrer's avatar
Timo Furrer committed
- [`1.6.2`](https://github.com/opentofu/opentofu/releases/tag/v1.6.2)
Timo Furrer's avatar
Timo Furrer committed
- [`1.6.1`](https://github.com/opentofu/opentofu/releases/tag/v1.6.1)
Timo Furrer's avatar
Timo Furrer committed
- [`1.6.0`](https://github.com/opentofu/opentofu/releases/tag/v1.6.0)
Timo Furrer's avatar
Timo Furrer committed
### Variables

(🚧 *This section is work in progress*)

Have a look at the [`src/gitlab-tofu.sh`](src/gitlab-tofu.sh) script and how the `TF_`-prefixed
variables are being used. You may set them according to your needs.

### Install additional tools

The `gitlab-opentofu` container image deliberately comes with minimal tooling
to keep the image size small and be the least common denominator for our users.

However, it is sometimes necessary to install additional tools. To do that you
can overwrite the included jobs with a `before_script` entry. The `gitlab-opentofu`
image uses `alpine` as its base image and therefore `apk` can be used to install
the tools. For example to install `jq`:

```yaml
include:
  - component: $CI_SERVER_FQDN/components/opentofu/validate-plan@<VERSION>
    inputs:
      version: <VERSION>
      opentofu_version: 1.6.1

plan:
  before_script:
    - apk add jq
```
Timo Furrer's avatar
Timo Furrer committed

### Best Practices

This section is a collection of *some* best practices.
Feel free to contribute more that generally apply.
If a best practice really becomes the de-facto standard
we may make it the default behavior if possible.

#### Lockfile Handling

If you commit the Lockfile (`.terraform.lock.hcl`) to your repository
we recommend setting either the `TF_INIT_FLAGS` (handled by this component)
or `TF_CLI_ARGS_init` (handled by OpenTofu directly) to `-lockfile=readonly`
to prevent any changes to the lockfile during the pipeline job and with
that ensuring that OpenTofu really uses the locked dependencies.

Timo Furrer's avatar
Timo Furrer committed
## Releases & Versioning

This project currently releases tagged commits.
An overview of releases can be found on the [Releases page](https://gitlab.com/components/opentofu/-/releases)
and a Changelog can be found [here](CHANGELOG.md).
Timo Furrer's avatar
Timo Furrer committed

Each release is accessible in the [CI/CD Catalog](https://gitlab.com/explore/catalog).

### Component Versions

The component release versions follow [Semantic Versioning 2.0.0](https://semver.org/).

### Image Versions

This project releases multiple OCI image variants that can be used with the component.
The intention is that the images used in a component have the same version and or not mixed.
Due to the limitations described in https://gitlab.com/gitlab-org/gitlab/-/issues/438275
Timo Furrer's avatar
Timo Furrer committed
it's currently required to provide the component version in the `component` include field
and as the `version` input. Check out the [Usage](#Usage) section for examples.

Each component release deploys the following images:

- `$CI_REGISTRY/components/opentofu/gitlab-opentofu:<VERSION>-opentofu<OPENTOFU_VERSION>`
- `$CI_REGISTRY/components/opentofu/gitlab-opentofu:<VERSION>-opentofu`
Timo Furrer's avatar
Timo Furrer committed
  - Includes the latest stable OpenTofu version at the time of releasing the component
- `$CI_REGISTRY/components/opentofu/gitlab-opentofu:<VERSION>`
Timo Furrer's avatar
Timo Furrer committed
  - Includes the latest stable OpenTofu version at the time of releasing the component

In the above examples `<VERSION>` references the component version and `<OPENTOFU_VERSION>`
Timo Furrer's avatar
Timo Furrer committed
an OpenTofu release, from [here](https://github.com/opentofu/opentofu/releases).

*Note: unfortunately, these image versions are not SemVer compatible,
because `-` indicates a prerelease (which they are not in this case).
However, we cannot use the alternative `+` which would indicate build metadata
as we'd like.
See https://github.com/distribution/distribution/issues/1201*

## Usage on self-managed

GitLab CI/CD components are not yet distributed and available on self-managed GitLab instances.
(see details [here](https://gitlab.com/gitlab-org/gitlab/-/issues/415638)).
It's also not possible to just include CI/CD components across instance, thus an include like
`- component: gitlab.com/components/opentofu/full-pipeline@~latest` won't work from a
self-managed instance.
However, you could mirror this project from GitLab.com onto any self-managed instance using
a [repository pull mirror](https://docs.gitlab.com/ee/user/project/repository/mirror/pull.html).

Timo Furrer's avatar
Timo Furrer committed
If the component is being mirrored to another path than `components/opentofu`, then you also
need to change that path in the `include:component` and additionally provide the correct
`image_registry_base` input.
See also the official GitLab documentation for it
[here](https://docs.gitlab.com/ee/ci/components/#use-a-gitlabcom-component-in-a-self-managed-instance).

If you want to save runner resources you may disable the unit and integration tests
by setting the `SKIP_TESTS` CI/CD variable to `true`.

## Migrating from the Terraform CI/CD templates

When migrating from the GitLab Terraform CI/CD templates you can use the following migration rules:

- Used `Terraform.gitlab-ci.yml` -> Migrate to `validate-plan-apply`.
- Used `Terraform/Base.gitlab-ci.yml` -> Migrate to `job-templates`.
    - Migrate the `.terraform:` job prefix to `.opentofu:`.
- Used the `kics-iac-sast` job -> Additionally include the `Jobs/SAST-IaC.latest.gitlab-ci.yml` template.
- Migrate the following job names:
    - `build` -> `plan`
    - `deploy` -> `apply`
- Migrate the `TF_ROOT` variable to the `root_dir` input.
    - Although the `TF_ROOT` variable is still used and maybe overwritten after the import on individual jobs.
- Migrate the `TF_STATE_NAME` variable to the `state_name` input.
    - Although the `TF_STATE_NAME` variable is still used and maybe overwritten after the import on individual jobs.
- Migrate the `TF_AUTO_DEPLOY` variable to the `auto_apply` input.
Timo Furrer's avatar
Timo Furrer committed
- Used other variables -> Use the same variables with this component.

The same rules apply for the `latest` templates.
We also recommend to check out the [Usage](#Usage) section for more details about the available templates and inputs.

Timo Furrer's avatar
Timo Furrer committed
### OpenTofu component `inputs` vs. Terraform template `variables`

This OpenTofu CI/CD component makes use of [`inputs`](https://docs.gitlab.com/ee/ci/yaml/#specinputs)
whereas the Terraform CI/CD templates used [`variables`](https://docs.gitlab.com/ee/ci/yaml/#variables).
We recommend that you use the `inputs` with the OpenTofu component where available and required.
However, if needed you may overwrite the jobs and set the `variables` you like.

## Can I use this component with Terraform?

Probably. Although, we don't officially support it or maintain any compatibility layer where necessary.

The OpenTofu CI/CD component job mainly interface with the [`gitlab-tofu`](src/gitlab-tofu.sh) script
that is distributed with the `gitlab-opentofu` container image used as the base image for the jobs.
This base image also contains the `tofu` binary.

If you'd want to use Terraform instead you may provide your own container image
that contains at least a script called `gitlab-tofu` so that it's compatible with the component jobs.
Everything else in the job can be custom, like replacing `tofu` with `terraform`.

You may provide the `image_registry_base` input to any of the component includes, pointing to the
container registry URI hosting the container image. The container image name can be configured in
the `image_name` input. The image has be versioned so that it is compatible
with [the image versioning of this project](#image-versions).

Timo Furrer's avatar
Timo Furrer committed
## Contributing

See the [CONTRIBUTING.md](CONTRIBUTING.md) guide.