GitLab - Download Artifacts from Parent Pipelines.

GitLab allows you to create Jobs that triggers other project's pipelines as seen on the diagram below.

The diagram above illustrates a trigger job that is used to start another project pipeline. The terraform get-resources-usage trigers another project pipeline that queries one of my hypervisors and returns its usage as a TXT file. This file is added as an artifact of the pipeline.

Pipeline artifacts can be downloaded by downstream pipelines ( GitLab names them parent/child pipelines ) using a built in feature needs:pipeline:job

However, GitLab still does not have a built in feature to allow the opposite.

We need to download artifacts from a downstream pipeline in our terraform pipeline. The artifact contains the hypervisor resources usage that will be presented to the staff member deploying the terraform config.

This can be achieved with the code below.

# ...
# ^ Other Jobs and configurations above.

# Trigger a downstream pipeline.
get-resources-usage:
  inherit:
    variables: false
  stage: get-resources-usage
  trigger:
    project: "infoitech/infrastructure/tucana/hv2/proxmox/get-resources-usage"
    strategy: depend

plan:
  stage: build  
  environment:
    name: staging
  variables:
    UPSTREAM_API_URL: ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}
    # TO-DO -> Query from VAULT.
    UPSTREAM_API_TOKEN: "XXXXXXX"
    DOWNSTREAM_API_TOKEN: "XXXXXXXX"
    
  script:            
    # Create Terraform JSON Plan.
    - gitlab-terraform plan-json        
    # DOWNLOAD ARTIFACTS FROM DOWNSTREAM PIPELINES     
    # Install missing packages and update xargs.    
    - apk update
    - apk add curl jq unzip findutils
    # Fetch the IDs of all child pipelines, fetch the IDs of their "deploy" jobs and    
    # download the artifacts of all of these jobs into `artifacts-<jobid>.zip`    
    - >
      curl -sS --header "PRIVATE-TOKEN: ${UPSTREAM_API_TOKEN}" "${UPSTREAM_API_URL}/pipelines/${CI_PIPELINE_ID}/bridges?per_page=100"
      | jq '.[].downstream_pipeline | ((.project_id|tostring)+"/pipelines/"+(.id|tostring)+"/jobs")'
      | xargs -i curl -sS --header "PRIVATE-TOKEN: ${DOWNSTREAM_API_TOKEN}" "${CI_API_V4_URL}/projects/{}"
      | jq -j '.[] | select(.name == "deploy") | ( (.pipeline.project_id|tostring)+"/jobs/"+(.id|tostring) )'
      | xargs --delimiter=/ -n 3 /bin/sh -c ' curl -sS --header "PRIVATE-TOKEN: $DOWNSTREAM_API_TOKEN" --output artifacts-$2.zip "$CI_API_V4_URL/projects/$0/$1/$2/artifacts"'       
    # Unzip all downloaded artifacts
    - unzip -o \*.zip
    
    # Show Terraform Plan
    - gitlab-terraform plan
    # Display the artifacts of all child pipelines.
    # In this pipeline the TXT will contain resources usage of the node terraform is deploying to. 
    - cat *.txt
  artifacts:
  # Artifacts specify which files to save as job artifacts.
  # Job artifacts are a list of files and directories that are attached to the job when it succeeds, fails or always.
  # Artifacts types : https://docs.gitlab.com/ee/ci/yaml/artifacts_reports.html#artifactsreportsterraform
    name: plan
    paths:
      - ${TF_ROOT}/plan.cache
      - ${TF_ROOT}/plan.json
    reports:
      terraform: ${TF_ROOT}/plan.json
...
# Other Jobs below VVV.

This thread has more information about this issue and how the GitLab team is working to fix it.