Arch Linux - Docker Node - Deploying GitLab Container - ( Part IV )

I have decided to deploy an internal git to version control my scripts and server configuration.

We will deploy the GitLab Enterprise edition. GitLab recommends installing the Entreprise Edition instead of the Community version to allow easy migration if we decide to pay for the Enterprise Version.

The official docker image can be found here.

The first prerequisite is to have docker installed.

[root@docker0 ~]# docker version
Server:
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d88bc
  Built:            Wed Aug  4 10:58:48 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.5.5
  GitCommit:        72cec4be58a9eb6b2910f5d10f1c01ca47d231c0.m
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Let's follow the gitlab instructions on how to deploy the image.

1) Set up Volumes Location

Configure a new environment variable pointing to the directory where the data will reside.

export GITLAB_HOME=/root/docker-projects/vlan.65-gitlab

[root@docker0 vlan.65-gitlab]# ll ../
total 8.0K
drwxr-xr-x 5 root root 4.0K Sep 22 21:50 vlan.55-pihole
drwxr-xr-x 2 root root 4.0K Sep 24 15:08 vlan.65-gitlab

2) Create Container Network

We are going to have our container directly attached to our network through MACVLAN.

docker network ls

NETWORK ID     NAME        DRIVER    SCOPE
1e0cc00dd141   bridge      bridge    local
1567bbd600cb   host        host      local
c848bfb101a1   macvlan55   macvlan   local
3f7b6cc3c82a   none        null      local

docker network create -d macvlan \
> --subnet=192.168.65.0/24 \
> --gateway=192.168.65.1 \
> --opt parent=ens18.65 \
> macvlan65

ip -c l

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 5a:a4:8c:e9:00:83 brd ff:ff:ff:ff:ff:ff
    altname enp0s18
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:32:86:1d:e8 brd ff:ff:ff:ff:ff:ff
4: ens18.55@ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 5a:a4:8c:e9:00:83 brd ff:ff:ff:ff:ff:ff
12: ens18.65@ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 5a:a4:8c:e9:00:83 brd ff:ff:ff:ff:ff:ff

4) Define Logging

The default driver provided by docker is in JSON and it creates a quite big log entry. We need to change to journald that has smaller entries preventing disk space exhaustion.

5) Prometheus Metrics

The memory allocated to Prometheus' metrics is not enough and it has to be increased to at least 256MB.

6) Self-signed Certs for Backend

GitLab is running behind a HAproxy instance on my firewalls. Because HAproxy will deal with the let's encrypt certificates it can be disabled in our GitLab server, but to avoid traffic in plaintext flowing thorugh the network I will enable self-signed certificates.

Create a configuration file for the SSL certificate and change its contents to reflect your installation. The subjectAltName are important if you are planning to run CI/CD pipelines. Without the correct FQDN runners registration will fail.

touch ssl-cert.conf

# Key:
#  - <SK> = Key to a config subsection

# Main entry point
# since our command on the CLI is `req`, OpenSSL is going to look for a matching entry-point
# This lets you store multiple command configs together, in a single file
[req]
# algorithm and number of bits to use when creating the private key
# rsa:2048
default_bits = 4096
# Same as `-nodes` argument, to prevent encryption of private key (passphrase)
encrypt_key = no
# Explicitly tells OpenSSL which message digest algorithm to use
# Good practice to specify, since older versions might default to MD5 (insecure)
default_md = sha256
# If you don't use `-keyout` in the CLI, this determines the private key filename
default_keyfile = domain.key
# <SK> These are values that are used to *distinguish* the certificate, such as the country and organization
# These values are normally collected via Q&A prompt in the CLI if config file is not used
distinguished_name = req_distinguished_name
# Ensures that distinguished_name values will be pulled from this file, as
#     opposed to prompting the user in the CLI
prompt = no
# <SK> Used for extensions to the self-signed cert OpenSSL is going to generate for us
x509_extensions = x509_extensions

[req_distinguished_name]
# - These are all values that are used to *distinguish* the certificate, such as 
#     the country and organization
# - Many of these have shorter keys that should be used for non-prompt values,
#     and long keys that should have a prompt string to display to the user, and
#     optionally a default value if the prompt is skipped (see below note)
# - For long keys, if you use fieldName with `_default` at the end, the value
#     will be used if prompt!==true, or if the user skips the prompt in the CLI

# Long = countryName
C = UK
# Long = stateOrProvinceName
ST = Surrey
# Long = localityName
L = London
# Long = organizationName
O = infoitech
# Long = organizationalUnitName
OU = headquarters
# Long = commonName
# Pay extra attention to common name - You can only define one, and it is the
#     value that is displayed to the user. Should NOT include protocol, but can
#     be in format of domain.tld, www.domain.tld, or even wildcard, to share a
#     common cert across multiple subdomains - `*.domain.tld`.
# Also, any value that you use here !*** MUST ***! be ALSO included in the SAN
#     (subject alternative name) section (subjectAltName), if you choose to
#     include that section. See: https://stackoverflow.com/a/25971071/11447682
CN = git.infoitech.co.uk

[x509_extensions]
# <SK> Used for (generically) custom field-value pairs that should be associated
#   with the cert, such as extra DNS names, IP addresses, and emails
subjectAltName = @alternate_names
subjectKeyIdentifier   = hash
authorityKeyIdentifier = keyid:always,issuer:always
basicConstraints       = CA:TRUE
keyUsage               = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment, keyAgreement, keyCertSign

[alternate_names]
# Extra domain names to associate with our cert
#  - These can be a mix of wildcard, IP address, subdomain, etc.
DNS.1 = git.infoitech.co.uk
DNS.2 = localhost
DNS.3 = 127.0.0.1
DNS.4 = 192.168.10.65
# Etc.
# See:
# - https://www.openssl.org/docs/man1.1.1/man5/x509v3_config.html#Subject-Alternative-Name
# - https://en.wikipedia.org/wiki/Subject_Alternative_Name

sudo openssl req -x509 -nodes -days 365 -newkey rsa:4096 -keyout ./ssl/git.infoitech.co.uk.key -out ./ssl/git.infoitech.co.uk.crt -config ssl-cert.conf

chmod -R 0755 ssl/

7) Redirect HTTP to HTTPS

Let's edit the NGINX settings for GitLab and our local docker registry that is enabled on this deployment.

8) Backups

Two folders are mounted in the host and a third can be set in a remote location. Also, GitLab was configured for 15 days retention.

The data backup will be in the already mounted folder ./data/backups.

Two cron jobs will be created on the host to execute the backups.

01 00 * * 1,3,6 docker exec -t gitlab /bin/sh -c 'gitlab-ctl backup-etc && cd /etc/gitlab/config_backup && cp $(ls -t | head -n1) /secret/gitlab/backups/'
10 00 * * 1,3,6 docker exec -t gitlab gitlab-backup

We first backup the configuration and 10 minutes later GitLab's data every Monday, Wednesday and Saturday.

9) Check Docker Compose Config

Let's check our docker compose file for errors.

docker-compose config

ERROR: The Compose file './docker-compose.yml' is invalid because:
Unsupported config option for services.gitlab: 'driver'
services.gitlab.logging contains an invalid type, it should be an object

I have missed indention on the logging driver section.

docker-compose config

version: "3.8"

services:
  gitlab:
    container_name: gitlab
    image: 'gitlab/gitlab-ee:latest'
    restart: always
    hostname: 'git'
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        # HTTPS Enabled
        external_url "https://git.infoitech.co.uk"
        registry_external_url "https:registry.infoitech.co.uk"
        # Step 7
        # Redirect port 80 to 443  
        nginx['redirect_http_to_https'] = true
        registry_nginx['redirect_http_to_https'] = true
        # Step 6
        # Disable lets encrypt because self-signed certs will be used.
        letsencrypt['enable'] = false
        # Step 8 - Backups
        ## Limit backup lifetime to 15 days
   		gitlab_rails['backup_keep_time'] = 1209600  
        # Enable the upload of large files.
        gitlab_rails['lfs_enabled'] = true
        # Enable local docker registry for auto deployment.
        registry['enable'] = true
        # Enable SSH
        gitlab_rails['gitlab_shell_ssh_port'] = 22
        # Add any other gitlab.rb configuration here, each on its own line
    dns:
      - 192.168.65.1
    networks:
            macvlan65:
                  ipv4_address: 192.168.65.10
    ports:
      - '80:80'
      - '443:443'
      - '22:22'
    volumes:
      - '$GITLAB_HOME/config:/etc/gitlab'
      - '$GITLAB_HOME/logs:/var/log/gitlab'
      - '$GITLAB_HOME/data:/var/opt/gitlab'
      # Step 6
      # Mount the SSL certs folder in the host.
      - '$GITLAB_HOME/ssl:/etc/gitlab/ssl'
      # Step 8 - Backups
      - '$GITLAB_HOME/backups/config:/secret/gitlab/backups'

    # Step 4
    # Fix - Docker containers exhausts space due to the json-file
    logging: 
      options:
        tag: "{{.ImageName}}/{{.Name}}/{{.ID}}"
      driver: journald
    
    # Step 5
    # Fix - mount not having enough space in Docker container.
    shm_size: 512m
    
networks:
  macvlan65:
          external: true

10) Deploying the container

Follow the commands below.

docker-compose up -d

Pulling gitlab (gitlab/gitlab-ee:latest)...
latest: Pulling from gitlab/gitlab-ee
35807b77a593: Pull complete
5de1c691df49: Pull complete
a9608775a0bd: Pull complete
c51cf31e3c16: Pull complete
803835994a8a: Pull complete
6b08dcdbeee2: Pull complete
f6ba44e7c742: Pull complete
c81892b985f4: Pull complete
Digest: sha256:f27e65c1ed79df94f1add05c77ef6f37d46441343e06787a5cec17a78ed552a1
Status: Downloaded newer image for gitlab/gitlab-ee:latest
Creating gitlab ... done

[root@docker0 vlan.65-gitlab]# docker container ls
CONTAINER ID   IMAGE                     COMMAND             CREATED          STATUS                             PORTS     NAMES
2dea512ed892   gitlab/gitlab-ee:latest   "/assets/wrapper"   30 seconds ago   Up 13 seconds (health: starting)             gitlab
36dcaad5bf68   pihole:Dockerfile         "/s6-init"          46 hours ago     Up 46 hours (healthy)                        pihole

You can use the command below to log into the container and check it status. Also, the command is useful for troubleshooting possible issues.

docker exec -it gitlab /bin/bash

gitlab-ctl status

run: alertmanager: (pid 655) 28s; run: log: (pid 672) 25s
run: gitaly: (pid 290) 88s; run: log: (pid 322) 86s
run: gitlab-exporter: (pid 608) 46s; run: log: (pid 613) 43s
run: gitlab-workhorse: (pid 534) 64s; run: log: (pid 545) 63s
run: grafana: (pid 700) 16s; run: log: (pid 716) 12s
run: logrotate: (pid 262) 100s; run: log: (pid 270) 99s
run: nginx: (pid 558) 58s; run: log: (pid 568) 57s
run: postgres-exporter: (pid 681) 22s; run: log: (pid 691) 18s
run: postgresql: (pid 332) 82s; run: log: (pid 495) 79s
run: prometheus: (pid 636) 34s; run: log: (pid 652) 31s
run: puma: (pid 499) 76s; run: log: (pid 506) 75s
run: redis: (pid 274) 94s; run: log: (pid 282) 93s
run: redis-exporter: (pid 619) 40s; run: log: (pid 630) 37s
run: registry: (pid 575) 52s; run: log: (pid 599) 49s
run: sidekiq: (pid 510) 70s; run: log: (pid 521) 67s
run: sshd: (pid 29) 120s; run: log: (pid 28) 120s

Resources