GitLab CI template for GitOps¶

This project implements a template to integrate GitOps deployment strategy with your GitLab CI/CD pipelines.

More specifically, this template allows you to trigger a change to a JSON or YAML file in a remote Git repository from your project pipeline.

For example, it will allow you to automatically update an image tag in a Helm Values file as soon as a new version of the container image is produced.

Important

This technique is certainly not the only nor the best way to implement a GitOps deployment strategy but in some cases it might help.

Whenever possible, we recommend using dedicated tools, such as:

Auto image update with FluxCD,
Argo CD Image Updater with ArgoCD,
even Renovate can help you implement the same very smartly.

Usage¶

This template can be used both as a CI/CD component or using the legacy include:project syntax.

Use as a CI/CD component¶

Add the following to your .gitlab-ci.yml:

include:
  # 1: include the component
  - component: $CI_SERVER_FQDN/to-be-continuous/gitops/gitlab-ci-gitops@1.0.0

# 2: GITOPS_TRIGGER_XXX variables are defined as project CI/CD variables

Use as a CI/CD template (legacy)¶

Add the following to your .gitlab-ci.yml:

include:
  # 1: include the template
  - project: "to-be-continuous/gitops"
    ref: "1.0.0"
    file: "/templates/gitlab-ci-gitops.yml"

# 2: GITOPS_TRIGGER_XXX variables are defined as project CI/CD variables

`gitops-trigger` job¶

This job triggers a change to a JSON or YAML file in a remote Git repository.

It uses the following variable:

Input / Variable	Description	Default value
`image` / `GITOPS_IMAGE`	The Docker image used to run GitOps (`git` + `yq`)	`docker.io/alpine/git:latest`
`GITOPS_TRIGGER_ONPROD`	The GitOps changes to trigger on the production branch (JSON) - see below	none (disabled)
`GITOPS_TRIGGER_ONINTEG`	The GitOps changes to trigger on the integration branch (JSON) - see below	none (disabled)
`GITOPS_TRIGGER_ONDEV`	The GitOps changes to trigger on the development branches (JSON) - see below	none (disabled)
`GITOPS_TRIGGER_ONRELEASE`	The GitOps changes to trigger on release tags (JSON) - see below	none (disabled)

Triggers definition (JSON)¶

The GitOps changes to trigger must be defined as a JSON structure, defining:

whether to trigger automatically or manually,
the target Git repositories and branch,
the files to modify,
the yq eval command to apply (supports late variable expansion).

the triggers definition MUST be managed as a project CI/CD variable (outside of the project .gitlab-ci.yaml then) for two reasons:

target repository URL bears their own authentication credentials,
allows a loose coupling between the source project and the target repositories to trigger (you don't necessarilly want to update the source project .gitlab-ci.yaml every time you change the triggers).

The triggers definition is formatted as follows:

{
  "when": "<whether to trigger changes manually or automatically (one of 'manual' or 'on_success')>",
  "commit_message": "<optional /The commit message template to use for GitOps trigger (supports late variable expansion)>",
  "triggers": [
    {
      "url": "<The target repository URL to push changes, including credentials>",
      "branch": "<The target repository branch to push changes to>",
      "commit_message": "<optional / The repo specific commit message template to use (supports late variable expansion)>",
      "dry_run": "<optional / true or false>",
      "changes": [
        {
          "file": "<The target file to apply the 'yq eval' command>",
          "yq_eval": "<The 'yq eval' command to apply (supports late variable expansion) - see https://mikefarah.gitbook.io/yq/operators/eval>"
        },
        ...
      ]
    },
    ...
  ]
}

The triggers definition must respect this JSON schema.

Example¶

{
  "when": "manual",
  "commit_message": "chore(gitops): update deployment values triggered from commit #$CI_COMMIT_SHA (see $CI_PIPELINE_URL)",
  "triggers": [
    {
      "url": "https://token:$MY_RW_ACCESS_TOKEN@$CI_SERVER_FQDN/path/to/values/project.git",
      "branch": "main",
      "changes": [
        {
          "file": "values_preprod.yml",
          "yq_eval": ".myapp.image.tag = \"$${docker_tag}@$${docker_digest}\""
        }
      ]
    }
  ]
}

Late variable expansion mechanism

The template supports a late variable expansion mechanism for the commit_message and yq_eval JSON fields.

It allows you to inject dynamically evaluated variables (coming from the pipeline execution context), using the %{somevar} (or $${somevar}) syntax.

The above example makes use of it to inject $docker_tag and $docker_digest variables, propagated by the Docker template.

on the other hand, the here-above commit_message field doesn't need to use this technique here as $CI_COMMIT_SHA and $CI_PIPELINE_URL are not late evaluated.

Target repository URL bears authentication

This template doesn't support any fancy mechanism to provide authentication, but instead assumes the target Git repository URL bears its own authentication credentials.

Example: https://user:suP3rpA55w0rd@gitlab.acme.corp/path/to/values/project

make sure to manage those credentials safely, as any other secret:

never store them directly in your source code,
manage them as project or group CI/CD variables:
masked to prevent them from being inadvertently displayed in your job logs,
protected if you want to secure some secrets you don't want everyone in the project to have access to (for instance production secrets).

Variants¶

Vault variant¶

This variant allows delegating your secrets management to a Vault server.

Configuration¶

In order to be able to communicate with the Vault server, the variant requires the additional configuration parameters:

INPUT / Variable	Description	Default value
`TBC_VAULT_IMAGE`	The Vault Secrets Provider image to use (can be overridden)	`registry.gitlab.com/to-be-continuous/tools/vault-secrets-provider:master`
`vault-base-url` / `VAULT_BASE_URL`	The Vault server base API url	must be defined
`vault-oidc-aud` / `VAULT_OIDC_AUD`	The `aud` claim for the JWT	`$CI_SERVER_URL`
`VAULT_ROLE_ID`	The AppRole RoleID	none
`VAULT_SECRET_ID`	The AppRole SecretID	none

By default, the variant will authentifacte using a JWT ID token. To use AppRole instead the VAULT_ROLE_ID and VAULT_SECRET_ID should be defined as secret project variables.

Usage¶

Then you may retrieve any of your secret(s) from Vault using the following syntax:

@url@http://vault-secrets-provider/api/secrets/{secret_path}?field={field}

With:

Name	Description
`secret_path` (path parameter)	this is your secret location in the Vault server
`field` (query parameter)	parameter to access a single basic field from the secret JSON payload

Example¶

include:
  # main template
  - project: "to-be-continuous/gitops"
    ref: "1.0.0"
    file: "/templates/gitlab-ci-gitops.yml"
  # Vault variant
  - project: "to-be-continuous/gitops"
    ref: "1.0.0"
    file: "/templates/gitlab-ci-gitops-vault.yml"

variables:
  # audience claim for JWT
  VAULT_OIDC_AUD: "https://vault.acme.host"
  # Vault server Url
  VAULT_BASE_URL: "https://vault.acme.host/v1"
  # Secrets managed by Vault
  GITOPS_TRIGGER_ONPROD: "@url@http://vault-secrets-provider/api/secrets/b7ecb6ebabc231/my-srv/gitops/prod?field=trigger"

GitLab CI template for GitOps¶

Usage¶

Use as a CI/CD component¶

Use as a CI/CD template (legacy)¶

gitops-trigger job¶

Triggers definition (JSON)¶

Example¶

Variants¶

Vault variant¶

Configuration¶

Usage¶

Example¶

`gitops-trigger` job¶