GitLab - Running your First CI/CD Pipeline

To run a pipeline you need to define the file .gitlab-ci.yml.

This file contains STAGES that can run multiple JOBS by order of declaration .

# GitLab terraform-images WAS example adapted.
# https://gitlab.com/gitlab-org/terraform-images/-/tree/master/ 

# GitLab Example Terraform image
image: 
  name: "$CI_TEMPLATE_REGISTRY_HOST/gitlab-org/terraform-images/releases/terraform:1.2.8"
  #name: "$CI_TEMPLATE_REGISTRY_HOST/gitlab-org/terraform-images/releases/1.2:latest"  

# Latest Terraform stable image and CLI tool. Not necessarily the lastest Hashicorp release.
#image: registry.gitlab.com/gitlab-org/terraform-images/stable:latest
  
# Variation
# image: https://registry.gitlab.com/gitlab-org/terraform-images/releases:1.2
 
variables:
  TF_ROOT: ${CI_PROJECT_DIR}
  TF_ADDRESS: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/terraform/state/${CI_PROJECT_NAME} 
  VAULT_SERVER_URL: "https://vault.infoitech.co.uk"
  VAULT_AUTH_ROLE: "terraform-pid-51" 

# Cache a list of files and directories between jobs.
cache:
  key: example-production
  paths:
    - ${TF_ROOT}/.terraform

# Runs before each job's script commands.
before_script:
  - cd ${TF_ROOT}

# Define statges that contain groups of jobs.
# The order of the the items defines the execution order for jobs.
stages:
  - prepare
  - validate
  - get-secrets
  - build
  - deploy

# Declare the INIT job.
init:
  # stage - define which stage this job runs.
  stage: prepare
  environment:
    name: staging
  # All jobs require one script keyword.
  script:    
    - gitlab-terraform version    
    - gitlab-terraform init    

# Validade job.
validate:
  stage: validate
  environment:
    name: staging
  script:
    - gitlab-terraform init
    - gitlab-terraform validate

# Get Secrets from Vault
get-secrets:
  stage: get-secrets
  environment:
    name: staging
  image: 
    name: vault:latest
    entrypoint:
      - '/usr/bin/env'
      - 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'  
  script:
    # Check job's ref name
    - echo $CI_COMMIT_REF_NAME
    # and is this ref protected
    - echo $CI_COMMIT_REF_PROTECTED
    # Vault address can be provided here or as a CI/CD Variable
    - export VAULT_ADDR="${VAULT_SERVER_URL}"
    # Authenticate and get token. Token expiry time and other properties can be configured.
    # when configuring JWT auth.    
    - export VAULT_TOKEN="$(vault write -field=token auth/jwt/login role=terraform-pid-51 jwt=$CI_JOB_JWT)"    
    # Now use the vault token to read the secret and store it in an environment variable.
    # Proxmox API ID    
    - echo "PM_API_TOKEN_ID=$(vault kv get -field=username kv-v2/tucana/hv2/proxmox/api)" >> proxmox-secrets.env
    # Proxmox API Token
    - echo "PM_API_TOKEN_SECRET=$(vault kv get -field=token kv-v2/tucana/hv2/proxmox/api)" >> proxmox-secrets.env
  artifacts:
    reports:
      dotenv: proxmox-secrets.env    

plan:
  stage: build  
  environment:
    name: staging
  script:        
    - export
    - gitlab-terraform plan
    - gitlab-terraform plan-json
  # Artifacts specify which files to save as job artifacts.
  # Job artifacts are a list of files and directories that are attached to the job when it succeeds, fails or always.
  # Artifacts types : https://docs.gitlab.com/ee/ci/yaml/artifacts_reports.html#artifactsreportsterraform
  artifacts:
    name: plan
    paths:
      - ${TF_ROOT}/plan.cache
    reports:
      terraform: ${TF_ROOT}/plan.json

# Separate apply job for manual launching Terraform as it can be destructive
# action.
apply:
  stage: deploy  
  environment:
    name: production
  script:
    - export
    - gitlab-terraform apply
  # Dependencies is used to download artifacts from previous jobs.
  # If not defined all artifacts are downloaded.
  dependencies:
    - plan
    - get-secrets
  when: manual
  # rules is used to include or exclude jobs in pipelines.
  # replaces only/except
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual

Jobs

JOBS can use the default docker image or an declared image to execute. Even a local image can be created, uploaded to the gitlab registry and used.

Gitlab has a huge documentation and more information about JOBS can be found here.

The Gitlab Graphical UI will show you the stages and the JOBS inside each stage.

Through this screen you can click on the STAGES and run clicking on the play button.

Here we can see the JOB output. Make sure to not expose secrets here and set the proper permissions to the ARTIFACTS.

Directory file structure.

/home/tiago/Documents/projects/hv2/terraform
├── .git
├── .gitlab-ci.yml
├── .gitlab-ci.yml.hashicorp-image
├── ll
├── main.tf
└── README.md

Manual JOBS

My terraform repository. The APLY JOB on runs when manually run. This JOB is dangerous it can as long as create to destroy infrastructure as well.

You can see in the image below the APPLY JOB reaching to the proxmox API creating  or destroying resources.

Resources