Vault - Create a Hashicorp Vault Container (Part I)

I am moving my infrastructure towards automation and with it comes the challenge of moving secrets around.

We don't want secrets hard-coded into our code and flying around all over the place. GitLab suggests using hashicorp Vault to solve these issues.

High Level Diagram of GitLab and Vault Working Together

Without further ado let's start the deployment.

1) Dockerfile

I had to change the official docker image due to their entry script automatically adding IP and running Vault in dev mode.

FROM alpine:3.14

# This is the release of Vault to pull in.
ARG VAULT_VERSION=1.11.3

# Create a vault user and group first so the IDs get set the same way,
# even as the rest of this may change over time.
RUN addgroup vault && \
    adduser -S -G vault vault

# Set up certificates, our base tools, and Vault.
RUN set -eux; \
    apk add --no-cache ca-certificates gnupg openssl libcap su-exec dumb-init tzdata && \
    apkArch="$(apk --print-arch)"; \
    case "$apkArch" in \
        armhf) ARCH='arm' ;; \
        aarch64) ARCH='arm64' ;; \
        x86_64) ARCH='amd64' ;; \
        x86) ARCH='386' ;; \
        *) echo >&2 "error: unsupported architecture: $apkArch"; exit 1 ;; \
    esac && \
    VAULT_GPGKEY=C874011F0AB405110D02105534365D9472D7468F; \
    found=''; \
    for server in \
        hkps://keys.openpgp.org \
        hkps://keyserver.ubuntu.com \
        hkps://pgp.mit.edu \
    ; do \
        echo "Fetching GPG key $VAULT_GPGKEY from $server"; \
        gpg --batch --keyserver "$server" --recv-keys "$VAULT_GPGKEY" && found=yes && break; \
    done; \
    test -z "$found" && echo >&2 "error: failed to fetch GPG key $VAULT_GPGKEY" && exit 1; \
    mkdir -p /tmp/build && \
    cd /tmp/build && \
    wget https://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_linux_${ARCH}.zip && \
    wget https://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_SHA256SUMS && \
    wget https://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_SHA256SUMS.sig && \
    gpg --batch --verify vault_${VAULT_VERSION}_SHA256SUMS.sig vault_${VAULT_VERSION}_SHA256SUMS && \
    grep vault_${VAULT_VERSION}_linux_${ARCH}.zip vault_${VAULT_VERSION}_SHA256SUMS | sha256sum -c && \
    unzip -d /tmp/build vault_${VAULT_VERSION}_linux_${ARCH}.zip && \
    cp /tmp/build/vault /bin/vault && \
    if [ -f /tmp/build/EULA.txt ]; then mkdir -p /usr/share/doc/vault; mv /tmp/build/EULA.txt /usr/share/doc/vault/EULA.txt; fi && \
    if [ -f /tmp/build/TermsOfEvaluation.txt ]; then mkdir -p /usr/share/doc/vault; mv /tmp/build/TermsOfEvaluation.txt /usr/share/doc/vault/TermsOfEvaluation.txt; fi && \
    cd /tmp && \
    rm -rf /tmp/build && \
    gpgconf --kill dirmngr && \
    gpgconf --kill gpg-agent 

# /vault/logs is made available to use as a location to store audit logs, if
# desired; /vault/file is made available to use as a location with the file
# storage backend, if desired; the server will be started with /vault/config as
# the configuration directory so you can add additional config files in that
# location.
RUN mkdir -p /vault/logs && \
    mkdir -p /vault/file && \
    mkdir -p /vault/config && \
    mkdir -p /vault/plugins && \
    mkdir -p /vault/ssl  

# Copy Script to create vault certs
# COPY 01-create-certs.sh /vault
# Create the self signed certs.
# RUN cd /vault && \
#    /bin/bash 01-create-certs.sh
# Delete the script.
#RUN rm /vault/01-create-certs.sh

# Change file permissions.
RUN chown -c -R vault:vault /vault

# Delete unused packages and folders
RUN apk del gnupg openssl && \
    rm -rf /root/.gnupg

# Expose the logs directory as a volume since there's potentially long-running
# state in there
VOLUME /vault/logs

# Expose the plugin folder.
VOLUME /vault/plugins

# Expose the certs folders in case you are generating your own certs.
VOLUME /vault/ssl

# Expose the file directory as a volume since there's potentially long-running
# state in there
VOLUME /vault/file

# 8200/tcp is the primary interface that applications use to interact with
# Vault.
# The official image exposes 8200 as seen above. However, I will be running on 443.
EXPOSE 443

# The entry point script uses dumb-init as the top-level process to reap any
# zombie processes created by Vault sub-processes.
#
# For production derivatives of this container, you shoud add the IPC_LOCK
# capability so that Vault can mlock memory.
COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
ENTRYPOINT ["docker-entrypoint.sh"]

# By default you'll get a single-node development server that stores everything
# in RAM and bootstraps itself. Don't use this configuration for production.
# CMD ["server", "-dev"]

# The official image runs by default in dev mode. The documentation is explicit :
# Warning: Never, ever, ever run a "dev" mode server in production
# Let's run it in server mode.
CMD ["server"]
/root/docker-projects/vlan.65-vault/Dockerfile

Updated entrypoint script.

#!/usr/bin/dumb-init /bin/sh
set -e

# Note above that we run dumb-init as PID 1 in order to reap zombie processes
# as well as forward signals to all processes in its session. Normally, sh
# wouldn't do either of these functions so we'd leak zombies as well as do
# unclean termination of all our sub-processes.

# Prevent core dumps
ulimit -c 0

# Custom condition added to skip the auto IP configuration.
# If VAULT_SKIP_IP_AUTO_CONFIG is set skip auto configuration and use only the config file.
# This change was made to allow full control over the config through volume mounts.
if [ -z "$VAULT_SKIP_IP_AUTO_CONFIG" ]; then
    # Allow setting VAULT_REDIRECT_ADDR and VAULT_CLUSTER_ADDR using an interface
    # name instead of an IP address. The interface name is specified using
    # VAULT_REDIRECT_INTERFACE and VAULT_CLUSTER_INTERFACE environment variables. If
    # VAULT_*_ADDR is also set, the resulting URI will combine the protocol and port
    # number with the IP of the named interface.
    get_addr () {
        local if_name=$1
        local uri_template=$2
        ip addr show dev $if_name | awk -v uri=$uri_template '/\s*inet\s/ { \
          ip=gensub(/(.+)\/.+/, "\\1", "g", $2); \
          print gensub(/^(.+:\/\/).+(:.+)$/, "\\1" ip "\\2", "g", uri); \
          exit}'
    }

    if [ -n "$VAULT_REDIRECT_INTERFACE" ]; then
        export VAULT_REDIRECT_ADDR=$(get_addr $VAULT_REDIRECT_INTERFACE ${VAULT_REDIRECT_ADDR:-"http://0.0.0.0:8200"})
        echo "Using $VAULT_REDIRECT_INTERFACE for VAULT_REDIRECT_ADDR: $VAULT_REDIRECT_ADDR"
    fi
    if [ -n "$VAULT_CLUSTER_INTERFACE" ]; then
        export VAULT_CLUSTER_ADDR=$(get_addr $VAULT_CLUSTER_INTERFACE ${VAULT_CLUSTER_ADDR:-"https://0.0.0.0:8201"})
        echo "Using $VAULT_CLUSTER_INTERFACE for VAULT_CLUSTER_ADDR: $VAULT_CLUSTER_ADDR"
    fi
fi


# VAULT_CONFIG_DIR isn't exposed as a volume but you can compose additional
# config files in there if you use this image as a base, or use
# VAULT_LOCAL_CONFIG below.
VAULT_CONFIG_DIR=/vault/config

# You can also set the VAULT_LOCAL_CONFIG environment variable to pass some
# Vault configuration JSON without having to bind any volumes.
if [ -n "$VAULT_LOCAL_CONFIG" ]; then
    echo "$VAULT_LOCAL_CONFIG" > "$VAULT_CONFIG_DIR/local.json"
fi

# If the user is trying to run Vault directly with some arguments, then
# pass them to Vault.
if [ "${1:0:1}" = '-' ]; then
    set -- vault "$@"
fi

# Look for Vault subcommands.
if [ "$1" = 'server' ]; then
    shift
    set -- vault server \
        -config="$VAULT_CONFIG_DIR" \
        # Disabling dev options, since the container will run in server mode.
        #-dev-root-token-id="$VAULT_DEV_ROOT_TOKEN_ID" \
        #-dev-listen-address="${VAULT_DEV_LISTEN_ADDRESS:-"0.0.0.0:8200"}" \
        "$@"
elif [ "$1" = 'version' ]; then
    # This needs a special case because there's no help output.
    set -- vault "$@"
elif vault --help "$1" 2>&1 | grep -q "vault $1"; then
    # We can't use the return code to check for the existence of a subcommand, so
    # we have to use grep to look for a pattern in the help output.
    set -- vault "$@"
fi

# If we are running Vault, make sure it executes as the proper user.
if [ "$1" = 'vault' ]; then
    if [ -z "$SKIP_CHOWN" ]; then
        # If the config dir is bind mounted then chown it
        if [ "$(stat -c %u /vault/config)" != "$(id -u vault)" ]; then
            chown -R vault:vault /vault/config || echo "Could not chown /vault/config (may not have appropriate permissions)"
        fi

        # If the logs dir is bind mounted then chown it
        if [ "$(stat -c %u /vault/logs)" != "$(id -u vault)" ]; then
            chown -R vault:vault /vault/logs
        fi

        # If the file dir is bind mounted then chown it
        if [ "$(stat -c %u /vault/file)" != "$(id -u vault)" ]; then
            chown -R vault:vault /vault/file
        fi
    fi

    if [ -z "$SKIP_SETCAP" ]; then
        # Allow mlock to avoid swapping Vault memory to disk
        setcap cap_ipc_lock=+ep $(readlink -f $(which vault))

        # In the case vault has been started in a container without IPC_LOCK privileges
        if ! vault -version 1>/dev/null 2>/dev/null; then
            >&2 echo "Couldn't start vault with IPC_LOCK. Disabling IPC_LOCK, please use --cap-add IPC_LOCK"
            setcap cap_ipc_lock=-ep $(readlink -f $(which vault))
        fi
    fi

    if [ "$(id -u)" = '0' ]; then
      set -- su-exec vault "$@"
    fi
fi

exec "$@"
/root/docker-projects/vlan.65-vault/docker-entrypoint.sh

Use the command below to build the image:

docker build --no-cache --pull -t "vault:Dockerfile" .

2) Docker Compose

Let's create the compose file to bring the Vault container up.

version: "3.8"

services:
  vault:
    container_name: vault
    image: 'vault:Dockerfile'
    restart: always
    hostname: 'vault'
    environment:
      # Enable localhost plaintext manipulation of Vault.
      # This options allow shell manipulation of the Vault Server
      # over the container shell. If the Server is unsealed the data flows in plaintext.
      VAULT_ADDR: 'http://127.0.0.1:8200'
      TZ: 'Europe/London'
      VAULT_SKIP_IP_AUTO_CONFIG: 'true'
    dns:
      - 192.168.65.1
    networks:
            macvlan65:
                  ipv4_address: 192.168.65.12
    ports:
      - '443:443'    
    volumes:
      # Vault - Logs
      - './logs:/vault/logs'
      # Vault - Plugins
      - './plugins:/vault/plugins'
      # Vault - Self-signed certificates
      - './ssl:/vault/ssl'
      # Vault - Encrypted Secrets File
      - './file:/vault/file'
      # Vault - Config
      - './config:/vault/config'
    cap_add:
      - IPC_LOCK
    logging: 
      options:
        tag: "{{.ImageName}}/{{.Name}}/{{.ID}}"
      driver: journald
networks:
  macvlan65:
          external: true

3) SSL Certificates

  • Self-Signed Certs

We need to create self-signed certs to our Vault server. The easiest way to do that is executing the below script. Remember to adapt the configuration to your needs.

  #!/usr/bin/env bash
# https://github.com/sethvargo/vault-kubernetes-workshop/blob/master/scripts/07-create-certs.sh

set -Eeuo pipefail

IP1="192.168.65.12"
IP2="127.0.0.1"

DNS1="vault.infoitech.co.uk"
DNS2="localhost"

DIR="$(pwd)/ssl"

rm -rf "${DIR}"
mkdir -p "${DIR}"

# Create the conf file
cat > "${DIR}/openssl.cnf" << EOF
[req]
default_bits = 4096
encrypt_key  = no
default_md   = sha256
prompt       = no
utf8         = yes
distinguished_name = req_distinguished_name
req_extensions     = v3_req
[req_distinguished_name]
C  = UK
ST = Surrey
L  = Tucana
O  = infoitech
CN = vault.infoitech.co.uk
[v3_req]
basicConstraints     = CA:FALSE
subjectKeyIdentifier = hash
keyUsage             = digitalSignature, keyEncipherment
extendedKeyUsage     = clientAuth, serverAuth
subjectAltName       = @alt_names
[alt_names]
IP.1  = ${IP1}
IP.2  = ${IP2}
DNS.1 = ${DNS1}
DNS.2 = ${DNS2}
EOF

# Generate Vault's certificates and a CSR
openssl genrsa -out "${DIR}/vault.key" 4096

openssl req \
  -new -key "${DIR}/vault.key" \
  -out "${DIR}/vault.csr" \
  -config "${DIR}/openssl.cnf"

# Create our CA
openssl req \
  -new \
  -newkey rsa:4096 \
  -days 3660 \
  -nodes \
  -x509 \
  -subj "/C=UK/ST=London/L=Tucana/O=Vault CA" \
  -keyout "${DIR}/ca.key" \
  -out "${DIR}/ca.crt"

# Sign CSR with our CA
openssl x509 \
  -req \
  -days 365 \
  -in "${DIR}/vault.csr" \
  -CA "${DIR}/ca.crt" \
  -CAkey "${DIR}/ca.key" \
  -CAcreateserial \
  -extensions v3_req \
  -extfile "${DIR}/openssl.cnf" \
  -out "${DIR}/vault.crt"

# Export combined certs for vault
cat "${DIR}/vault.crt" "${DIR}/ca.crt" > "${DIR}/vault-combined.crt"
/root/docker-projects/vlan.65-vault/01-create-certs.sh
  • Let's Encrypt Certs

It is also recommended and possible to use Let's Encrypt certificates with Vault. You will need to generate the the certs and renew it before expiration 3 months later.

I currently have the acme package installed in my firewalls and with the CloudFlare API it auto renew the certificates every 3 months. I will just need to figure out an automation solution to move the certs once renewed.

The certificate files are located in /tmp/acme/<<CERT_NAME>>/vault.infoitech.co.uk.

vault.infoitech.co.uk/
├── backup
├── ca.cer
├── fullchain.cer
├── vault.infoitech.co.uk.cer
├── vault.infoitech.co.uk.conf
├── vault.infoitech.co.uk.csr
├── vault.infoitech.co.uk.csr.conf
└── vault.infoitech.co.uk.key
Folder Structure

we need to copy fullchain.cer and vault.infoitech.co.uk.key to <<PROJECT_PATH>>/ssl.

4) Permissions

The vault container is deployed with  a group and user called vault and because the permissions of local folders are passed to the container mounts. We need to create a vault user in our docker host.

useradd -r -s /usr/bin/nologin vault

Let's set the configuration file and SSL certs with the permissions as below:

-rwxr--r-- 1 root  root  1.6K Sep  3 14:06 01-create-certs.sh*
drwxr-xr-x 2 vault vault 4.0K Sep  3 17:28 config/
-rwxr--r-- 1 vault vault  519 Sep  3 17:28 config/config.hcl*
drwxr-xr-x 5 vault vault 4.0K Sep  3 17:08 file/
drwxr-xr-x 2 vault vault 4.0K Sep  1 10:05 logs/
drwxr-xr-x 2 vault vault 4.0K Sep  2 15:03 plugins/
drwxr-xr-x 2 root  root  4.0K Sep  3 16:59 ssl/
-rw-r--r-- 1 root  root  1.9K Sep  3 16:59 ssl/ca.crt
-rw------- 1 root  root  3.2K Sep  3 16:59 ssl/ca.key
-rw-r--r-- 1 root  root    41 Sep  3 16:59 ssl/ca.srl
-rw-r--r-- 1 root  root   580 Sep  3 16:59 ssl/openssl.cnf
-rw-r--r-- 1 vault vault 3.9K Sep  3 16:59 ssl/vault-combined.crt
-rw-r--r-- 1 root  root  2.1K Sep  3 16:59 ssl/vault.crt
-rw-r--r-- 1 root  root  1.9K Sep  3 16:59 ssl/vault.csr
-rw-r----- 1 root  vault 3.2K Sep  3 16:59 ssl/vault.key

5) Vault Configuration

The configuration below will bring the container up and allow us to manage it. You need to adapt to your needs and also check Vault's documentation for further information.

storage "file" {
    path = "/vault/file"
}

listener "tcp" {
    address       = "192.168.65.12:443"
    tls_min_version = "tls12"
    tls_cert_file = "/vault/ssl/fullchain.cer"
    tls_key_file  = "/vault/ssl/vault.infoitech.co.uk.key"
}

listener "tcp" {
    address         = "127.0.0.1:8200"
    tls_disable     = "true"
}

api_addr = "https://vault.infoitech.co.uk"
cluster_addr = "https://vault.infoitech.co.uk"

ui = true

plugin_directory = "/vault/plugins"

log_level = "info"

default_lease_ttl = "168h"
max_lease_ttl = "720h"
/root/docker-projects/vlan.65-vault/config/config.hcl

6) Starting the container

Finally, let's bring the container up.

docker-compose up -d

You can check the container logs to make sure that it started without errors.

docker container logs vault
==> Vault server configuration:
             Api Address: https://vault.infoitech.co.uk
                     Cgo: disabled
         Cluster Address: https://vault.infoitech.co.uk:444
              Go Version: go1.17.13
              Listener 1: tcp (addr: "192.168.65.12:443", cluster address: "192.168.65.12:444", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
              Listener 2: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
               Log Level: info
                   Mlock: supported: true, enabled: true
           Recovery Mode: false
                 Storage: file
                 Version: Vault v1.11.3, built 2022-08-26T10:27:10Z
             Version Sha: xxXXXXXxxxXXXXXxx
2022-09-03T17:28:25.579+0100 [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
2022-09-03T17:28:25.628+0100 [INFO]  core: Initializing version history cache for core
==> Vault server started! Log data will stream in below:

7) Root Token

Vault uses tokens for authentication. The root token is generated during the initial setup and is the only method that cannot be disabled nor the root token expires.

The root token can do anything in Vault. It is extremely important to secure this token. Vault documentation explains how tokens work in more depth.

There is also an excellent tutorial from hashicorp on how to manipulate tokens that worth reading.

When you access the ui for the first time. Vault will generate the root token and you can use it for the initial setup.

8) Troubleshooting

You can use the docker container logs to inspect the container for errors and if running but not working as expected.

It is possible to login into the container to further investigate. Use vault --help for a list of commands.

docker exec -it vault /bin/sh
/ # vault status
Key                Value
---                -----
Seal Type          shamir
Initialized        true
Sealed             true
Total Shares       1
Threshold          1
Unseal Progress    0/1
Unseal Nonce       n/a
Version            1.11.3
Build Date         2022-08-26T10:27:10Z
Storage Type       file
HA Enabled         false
/ #

The file structure of the project is as below.

/root/docker-projects/vlan.65-vault
├── 01-create-certs.sh
├── config
│   └── config.hcl
├── docker-compose.yml
├── docker-entrypoint.sh
├── Dockerfile
├── file
│   ├── ...
├── logs
├── plugins
├── README.md
└── ssl
    ├── ca.crt
    ├── ca.key
    ├── ca.srl
    ├── openssl.cnf
    ├── vault-combined.crt
    ├── vault.crt
    ├── vault.csr
    └── vault.key
Project File Structure

Conclusion

This article describes how to spin up a basic Vault container. For more information check the docs and the resources below.

Resources

Official Hashicorp Vault Repository

Bash Parameter Expansion

Bash IF Statements Explained

Vault Configuration Docs

Hashicorp HCL - Docs

Dockerfile Reference Guide

Hashicorp Self-Signed TLS Certificates Example

Build Certificate Authority (CA) in Vault with an offline Root