Arch Linux - Docker Node - Deploying GitLab Container - ( Part IV )

Arch Linux - Docker Node - Deploying GitLab Container - ( Part IV )

I have decided to deploy an internal git to version control my scripts and server configuration.

We will deploy the GitLab Enterprise edition. GitLab recommends installing the Entreprise Edition instead of the Community version to allow easy migration if we decide to pay for the Enterprise Version.

The official docker image can be found here.

The first prerequisite is to have docker installed.

[root@docker0 ~]# docker version
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d88bc
  Built:            Wed Aug  4 10:58:48 2021
  OS/Arch:          linux/amd64
  Experimental:     false
  Version:          v1.5.5
  GitCommit:        72cec4be58a9eb6b2910f5d10f1c01ca47d231c0.m
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2d
  Version:          0.19.0
  GitCommit:        de40ad0

Let's follow the gitlab instructions on how to deploy the image.

1) Set up Volumes Location

Configure a new environment variable pointing to the directory where the data will reside.

export GITLAB_HOME=/root/docker-projects/vlan.65-gitlab
[root@docker0 vlan.65-gitlab]# ll ../
total 8.0K
drwxr-xr-x 5 root root 4.0K Sep 22 21:50 vlan.55-pihole
drwxr-xr-x 2 root root 4.0K Sep 24 15:08 vlan.65-gitlab
Folder Permissions
Local location Container location Usage
$GITLAB_HOME/data /var/opt/gitlab For storing application data.
$GITLAB_HOME/logs /var/log/gitlab For storing logs.
$GITLAB_HOME/config /etc/gitlab For storing the GitLab configuration files.

2) Create Container Network

We are going to have our container directly attached to our network through MACVLAN.

docker network ls
1e0cc00dd141   bridge      bridge    local
1567bbd600cb   host        host      local
c848bfb101a1   macvlan55   macvlan   local
3f7b6cc3c82a   none        null      local
docker network create -d macvlan \
> --subnet= \
> --gateway= \
> --opt parent=ens18.65 \
> macvlan65
ip -c l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 5a:a4:8c:e9:00:83 brd ff:ff:ff:ff:ff:ff
    altname enp0s18
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 
    link/ether 02:42:32:86:1d:e8 brd ff:ff:ff:ff:ff:ff
4: ens18.55@ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 5a:a4:8c:e9:00:83 brd ff:ff:ff:ff:ff:ff
12: ens18.65@ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 5a:a4:8c:e9:00:83 brd ff:ff:ff:ff:ff:ff

4) Define Logging

The default driver provided by docker is in JSON and it creates a quite big log entry. We need to change to journald that has smaller entries preventing disk space exhaustion.

5) Prometheus Metrics

The memory allocated to Prometheus' metrics is not enough and it has to be increased to at least 256MB.

6) Self-signed Certs for Backend

GitLab is running behind a HAproxy instance on my firewalls. Because HAproxy will deal with the let's encrypt certificates it can be disabled in our GitLab server, but to avoid traffic in plaintext flowing thorugh the network I will enable self-signed certificates.

sudo openssl req -x509 -nodes -days 365 -newkey rsa:4096 -keyout ./ssl/ -out ./ssl/
chmod -R 0755 ssl/
Run these commands inside the folder of the docker-compose file in the docker host.

7) Redirect HTTP to HTTPS

Let's edit the NGINX settings for GitLab and our local docker registry that is enabled on this deployment.

8) Backups

Two folders are mounted in the host and a third can be set in a remote location. Also, GitLab was configured for 15 days retention.

The data backup will be in the already mounted folder ./data/backups.

Two cron jobs will be created on the host to execute the backups.

01 00 * * 1,3,6 docker exec -t gitlab /bin/sh -c 'gitlab-ctl backup-etc && cd /etc/gitlab/config_backup && cp $(ls -t | head -n1) /secret/gitlab/backups/'
10 00 * * 1,3,6 docker exec -t gitlab gitlab-backup

We first backup the configuration and 10 minutes later GitLab's data every Monday, Wednesday and Saturday.

9) Check Docker Compose Config

Let's check our docker compose file for errors.

docker-compose config
ERROR: The Compose file './docker-compose.yml' is invalid because:
Unsupported config option for services.gitlab: 'driver'
services.gitlab.logging contains an invalid type, it should be an object

I have missed indention on the logging driver section.

docker-compose config
version: "3.8"

    container_name: gitlab
    image: 'gitlab/gitlab-ee:latest'
    restart: always
    hostname: 'git'
        # HTTPS Enabled
        external_url ""
        registry_external_url ""
        # Step 7
        # Redirect port 80 to 443  
        nginx['redirect_http_to_https'] = true
        registry_nginx['redirect_http_to_https'] = true
        # Step 6
        # Disable lets encrypt because self-signed certs will be used.
        letsencrypt['enable'] = false
        # Step 8 - Backups
        ## Limit backup lifetime to 15 days
   		gitlab_rails['backup_keep_time'] = 1209600  
        # Enable the upload of large files.
        gitlab_rails['lfs_enabled'] = true
        # Enable local docker registry for auto deployment.
        registry['enable'] = true
        # Enable SSH
        gitlab_rails['gitlab_shell_ssh_port'] = 22
        # Add any other gitlab.rb configuration here, each on its own line
      - '80:80'
      - '443:443'
      - '22:22'
      - '$GITLAB_HOME/config:/etc/gitlab'
      - '$GITLAB_HOME/logs:/var/log/gitlab'
      - '$GITLAB_HOME/data:/var/opt/gitlab'
      # Step 6
      # Mount the SSL certs folder in the host.
      - '$GITLAB_HOME/ssl:/etc/gitlab/ssl'
      # Step 8 - Backups
      - '$GITLAB_HOME/backups/config:/secret/gitlab/backups'

    # Step 4
    # Fix - Docker containers exhausts space due to the json-file
        tag: "{{.ImageName}}/{{.Name}}/{{.ID}}"
      driver: journald
    # Step 5
    # Fix - mount not having enough space in Docker container.
    shm_size: 512m
          external: true

10) Deploying the container

Follow the commands below.

docker-compose up -d
Pulling gitlab (gitlab/gitlab-ee:latest)...
latest: Pulling from gitlab/gitlab-ee
35807b77a593: Pull complete
5de1c691df49: Pull complete
a9608775a0bd: Pull complete
c51cf31e3c16: Pull complete
803835994a8a: Pull complete
6b08dcdbeee2: Pull complete
f6ba44e7c742: Pull complete
c81892b985f4: Pull complete
Digest: sha256:f27e65c1ed79df94f1add05c77ef6f37d46441343e06787a5cec17a78ed552a1
Status: Downloaded newer image for gitlab/gitlab-ee:latest
Creating gitlab ... done
[root@docker0 vlan.65-gitlab]# docker container ls
CONTAINER ID   IMAGE                     COMMAND             CREATED          STATUS                             PORTS     NAMES
2dea512ed892   gitlab/gitlab-ee:latest   "/assets/wrapper"   30 seconds ago   Up 13 seconds (health: starting)             gitlab
36dcaad5bf68   pihole:Dockerfile         "/s6-init"          46 hours ago     Up 46 hours (healthy)                        pihole

You can use the command below to log into the container and check it status. Also, the command is useful for troubleshooting possible issues.

docker exec -it gitlab /bin/bash
[root@docker0 vlan.65-gitlab]#
gitlab-ctl status
run: alertmanager: (pid 655) 28s; run: log: (pid 672) 25s
run: gitaly: (pid 290) 88s; run: log: (pid 322) 86s
run: gitlab-exporter: (pid 608) 46s; run: log: (pid 613) 43s
run: gitlab-workhorse: (pid 534) 64s; run: log: (pid 545) 63s
run: grafana: (pid 700) 16s; run: log: (pid 716) 12s
run: logrotate: (pid 262) 100s; run: log: (pid 270) 99s
run: nginx: (pid 558) 58s; run: log: (pid 568) 57s
run: postgres-exporter: (pid 681) 22s; run: log: (pid 691) 18s
run: postgresql: (pid 332) 82s; run: log: (pid 495) 79s
run: prometheus: (pid 636) 34s; run: log: (pid 652) 31s
run: puma: (pid 499) 76s; run: log: (pid 506) 75s
run: redis: (pid 274) 94s; run: log: (pid 282) 93s
run: redis-exporter: (pid 619) 40s; run: log: (pid 630) 37s
run: registry: (pid 575) 52s; run: log: (pid 599) 49s
run: sidekiq: (pid 510) 70s; run: log: (pid 521) 67s
run: sshd: (pid 29) 120s; run: log: (pid 28) 120s


GitLab + Runner + Registry + CI/CD Stack Setup and Management Best Practices by GitOps #1
With this article series, I wanna tell end-to-end experience and best practices, while our infrastructure transformation at Trendyol Tech…
compose-spec/ at master · compose-spec/compose-spec
The Compose specification. Contribute to compose-spec/compose-spec development by creating an account on GitHub.
How to reset user password | GitLab
Documentation for GitLab Community Edition, GitLab Enterprise Edition, Omnibus GitLab, and GitLab Runner.
NGINX settings | GitLab
Documentation for GitLab Community Edition, GitLab Enterprise Edition, Omnibus GitLab, and GitLab Runner.
files/gitlab-config-template/gitlab.rb.template · master · / omnibus-gitlab
This project creates full-stack platform-specific downloadable packages for GitLab.

Share Tweet Send
You've successfully subscribed to Infoitech - [B]logging
Great! Next, complete checkout for full access to Infoitech - [B]logging
Welcome back! You've successfully signed in
Success! Your account is fully activated, you now have access to all content.