I am moving my infrastructure towards automation and with it comes the challenge of moving secrets around.
We don't want secrets hard-coded into our code and flying around all over the place. GitLab suggests using hashicorp Vault to solve these issues .
High Level Diagram of GitLab and Vault Working Together Without further ado let's start the deployment.
1) Dockerfile I had to change the official docker image due to their entry script automatically adding IP and running Vault in dev mode.
FROM alpine:3.14
# This is the release of Vault to pull in.
ARG VAULT_VERSION=1.11.3
# Create a vault user and group first so the IDs get set the same way,
# even as the rest of this may change over time.
RUN addgroup vault && \
adduser -S -G vault vault
# Set up certificates, our base tools, and Vault.
RUN set -eux; \
apk add --no-cache ca-certificates gnupg openssl libcap su-exec dumb-init tzdata && \
apkArch="$(apk --print-arch)"; \
case "$apkArch" in \
armhf) ARCH='arm' ;; \
aarch64) ARCH='arm64' ;; \
x86_64) ARCH='amd64' ;; \
x86) ARCH='386' ;; \
*) echo >&2 "error: unsupported architecture: $apkArch"; exit 1 ;; \
esac && \
VAULT_GPGKEY=C874011F0AB405110D02105534365D9472D7468F; \
found=''; \
for server in \
hkps://keys.openpgp.org \
hkps://keyserver.ubuntu.com \
hkps://pgp.mit.edu \
; do \
echo "Fetching GPG key $VAULT_GPGKEY from $server"; \
gpg --batch --keyserver "$server" --recv-keys "$VAULT_GPGKEY" && found=yes && break; \
done; \
test -z "$found" && echo >&2 "error: failed to fetch GPG key $VAULT_GPGKEY" && exit 1; \
mkdir -p /tmp/build && \
cd /tmp/build && \
wget https://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_linux_${ARCH}.zip && \
wget https://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_SHA256SUMS && \
wget https://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_SHA256SUMS.sig && \
gpg --batch --verify vault_${VAULT_VERSION}_SHA256SUMS.sig vault_${VAULT_VERSION}_SHA256SUMS && \
grep vault_${VAULT_VERSION}_linux_${ARCH}.zip vault_${VAULT_VERSION}_SHA256SUMS | sha256sum -c && \
unzip -d /tmp/build vault_${VAULT_VERSION}_linux_${ARCH}.zip && \
cp /tmp/build/vault /bin/vault && \
if [ -f /tmp/build/EULA.txt ]; then mkdir -p /usr/share/doc/vault; mv /tmp/build/EULA.txt /usr/share/doc/vault/EULA.txt; fi && \
if [ -f /tmp/build/TermsOfEvaluation.txt ]; then mkdir -p /usr/share/doc/vault; mv /tmp/build/TermsOfEvaluation.txt /usr/share/doc/vault/TermsOfEvaluation.txt; fi && \
cd /tmp && \
rm -rf /tmp/build && \
gpgconf --kill dirmngr && \
gpgconf --kill gpg-agent
# /vault/logs is made available to use as a location to store audit logs, if
# desired; /vault/file is made available to use as a location with the file
# storage backend, if desired; the server will be started with /vault/config as
# the configuration directory so you can add additional config files in that
# location.
RUN mkdir -p /vault/logs && \
mkdir -p /vault/file && \
mkdir -p /vault/config && \
mkdir -p /vault/plugins && \
mkdir -p /vault/ssl
# Copy Script to create vault certs
# COPY 01-create-certs.sh /vault
# Create the self signed certs.
# RUN cd /vault && \
# /bin/bash 01-create-certs.sh
# Delete the script.
#RUN rm /vault/01-create-certs.sh
# Change file permissions.
RUN chown -c -R vault:vault /vault
# Delete unused packages and folders
RUN apk del gnupg openssl && \
rm -rf /root/.gnupg
# Expose the logs directory as a volume since there's potentially long-running
# state in there
VOLUME /vault/logs
# Expose the plugin folder.
VOLUME /vault/plugins
# Expose the certs folders in case you are generating your own certs.
VOLUME /vault/ssl
# Expose the file directory as a volume since there's potentially long-running
# state in there
VOLUME /vault/file
# 8200/tcp is the primary interface that applications use to interact with
# Vault.
# The official image exposes 8200 as seen above. However, I will be running on 443.
EXPOSE 443
# The entry point script uses dumb-init as the top-level process to reap any
# zombie processes created by Vault sub-processes.
#
# For production derivatives of this container, you shoud add the IPC_LOCK
# capability so that Vault can mlock memory.
COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
ENTRYPOINT ["docker-entrypoint.sh"]
# By default you'll get a single-node development server that stores everything
# in RAM and bootstraps itself. Don't use this configuration for production.
# CMD ["server", "-dev"]
# The official image runs by default in dev mode. The documentation is explicit :
# Warning: Never, ever, ever run a "dev" mode server in production
# Let's run it in server mode.
CMD ["server"]
/root/docker-projects/vlan.65-vault/Dockerfile Updated entrypoint script.
#!/usr/bin/dumb-init /bin/sh
set -e
# Note above that we run dumb-init as PID 1 in order to reap zombie processes
# as well as forward signals to all processes in its session. Normally, sh
# wouldn't do either of these functions so we'd leak zombies as well as do
# unclean termination of all our sub-processes.
# Prevent core dumps
ulimit -c 0
# Custom condition added to skip the auto IP configuration.
# If VAULT_SKIP_IP_AUTO_CONFIG is set skip auto configuration and use only the config file.
# This change was made to allow full control over the config through volume mounts.
if [ -z "$VAULT_SKIP_IP_AUTO_CONFIG" ]; then
# Allow setting VAULT_REDIRECT_ADDR and VAULT_CLUSTER_ADDR using an interface
# name instead of an IP address. The interface name is specified using
# VAULT_REDIRECT_INTERFACE and VAULT_CLUSTER_INTERFACE environment variables. If
# VAULT_*_ADDR is also set, the resulting URI will combine the protocol and port
# number with the IP of the named interface.
get_addr () {
local if_name=$1
local uri_template=$2
ip addr show dev $if_name | awk -v uri=$uri_template '/\s*inet\s/ { \
ip=gensub(/(.+)\/.+/, "\\1", "g", $2); \
print gensub(/^(.+:\/\/).+(:.+)$/, "\\1" ip "\\2", "g", uri); \
exit}'
}
if [ -n "$VAULT_REDIRECT_INTERFACE" ]; then
export VAULT_REDIRECT_ADDR=$(get_addr $VAULT_REDIRECT_INTERFACE ${VAULT_REDIRECT_ADDR:-"http://0.0.0.0:8200"})
echo "Using $VAULT_REDIRECT_INTERFACE for VAULT_REDIRECT_ADDR: $VAULT_REDIRECT_ADDR"
fi
if [ -n "$VAULT_CLUSTER_INTERFACE" ]; then
export VAULT_CLUSTER_ADDR=$(get_addr $VAULT_CLUSTER_INTERFACE ${VAULT_CLUSTER_ADDR:-"https://0.0.0.0:8201"})
echo "Using $VAULT_CLUSTER_INTERFACE for VAULT_CLUSTER_ADDR: $VAULT_CLUSTER_ADDR"
fi
fi
# VAULT_CONFIG_DIR isn't exposed as a volume but you can compose additional
# config files in there if you use this image as a base, or use
# VAULT_LOCAL_CONFIG below.
VAULT_CONFIG_DIR=/vault/config
# You can also set the VAULT_LOCAL_CONFIG environment variable to pass some
# Vault configuration JSON without having to bind any volumes.
if [ -n "$VAULT_LOCAL_CONFIG" ]; then
echo "$VAULT_LOCAL_CONFIG" > "$VAULT_CONFIG_DIR/local.json"
fi
# If the user is trying to run Vault directly with some arguments, then
# pass them to Vault.
if [ "${1:0:1}" = '-' ]; then
set -- vault "$@"
fi
# Look for Vault subcommands.
if [ "$1" = 'server' ]; then
shift
set -- vault server \
-config="$VAULT_CONFIG_DIR" \
# Disabling dev options, since the container will run in server mode.
#-dev-root-token-id="$VAULT_DEV_ROOT_TOKEN_ID" \
#-dev-listen-address="${VAULT_DEV_LISTEN_ADDRESS:-"0.0.0.0:8200"}" \
"$@"
elif [ "$1" = 'version' ]; then
# This needs a special case because there's no help output.
set -- vault "$@"
elif vault --help "$1" 2>&1 | grep -q "vault $1"; then
# We can't use the return code to check for the existence of a subcommand, so
# we have to use grep to look for a pattern in the help output.
set -- vault "$@"
fi
# If we are running Vault, make sure it executes as the proper user.
if [ "$1" = 'vault' ]; then
if [ -z "$SKIP_CHOWN" ]; then
# If the config dir is bind mounted then chown it
if [ "$(stat -c %u /vault/config)" != "$(id -u vault)" ]; then
chown -R vault:vault /vault/config || echo "Could not chown /vault/config (may not have appropriate permissions)"
fi
# If the logs dir is bind mounted then chown it
if [ "$(stat -c %u /vault/logs)" != "$(id -u vault)" ]; then
chown -R vault:vault /vault/logs
fi
# If the file dir is bind mounted then chown it
if [ "$(stat -c %u /vault/file)" != "$(id -u vault)" ]; then
chown -R vault:vault /vault/file
fi
fi
if [ -z "$SKIP_SETCAP" ]; then
# Allow mlock to avoid swapping Vault memory to disk
setcap cap_ipc_lock=+ep $(readlink -f $(which vault))
# In the case vault has been started in a container without IPC_LOCK privileges
if ! vault -version 1>/dev/null 2>/dev/null; then
>&2 echo "Couldn't start vault with IPC_LOCK. Disabling IPC_LOCK, please use --cap-add IPC_LOCK"
setcap cap_ipc_lock=-ep $(readlink -f $(which vault))
fi
fi
if [ "$(id -u)" = '0' ]; then
set -- su-exec vault "$@"
fi
fi
exec "$@"
/root/docker-projects/vlan.65-vault/docker-entrypoint.sh Use the command below to build the image:
docker build --no-cache --pull -t "vault:Dockerfile" .
2) Docker Compose Let's create the compose file to bring the Vault container up.
version: "3.8"
services:
vault:
container_name: vault
image: 'vault:Dockerfile'
restart: always
hostname: 'vault'
environment:
# Enable localhost plaintext manipulation of Vault.
# This options allow shell manipulation of the Vault Server
# over the container shell. If the Server is unsealed the data flows in plaintext.
VAULT_ADDR: 'http://127.0.0.1:8200'
TZ: 'Europe/London'
VAULT_SKIP_IP_AUTO_CONFIG: 'true'
dns:
- 192.168.65.1
networks:
macvlan65:
ipv4_address: 192.168.65.12
ports:
- '443:443'
volumes:
# Vault - Logs
- './logs:/vault/logs'
# Vault - Plugins
- './plugins:/vault/plugins'
# Vault - Self-signed certificates
- './ssl:/vault/ssl'
# Vault - Encrypted Secrets File
- './file:/vault/file'
# Vault - Config
- './config:/vault/config'
cap_add:
- IPC_LOCK
logging:
options:
tag: "{{.ImageName}}/{{.Name}}/{{.ID}}"
driver: journald
networks:
macvlan65:
external: true
3) SSL Certificates We need to create self-signed certs to our Vault server. The easiest way to do that is executing the below script. Remember to adapt the configuration to your needs.
#!/usr/bin/env bash
# https://github.com/sethvargo/vault-kubernetes-workshop/blob/master/scripts/07-create-certs.sh
set -Eeuo pipefail
IP1="192.168.65.12"
IP2="127.0.0.1"
DNS1="vault.infoitech.co.uk"
DNS2="localhost"
DIR="$(pwd)/ssl"
rm -rf "${DIR}"
mkdir -p "${DIR}"
# Create the conf file
cat > "${DIR}/openssl.cnf" << EOF
[req]
default_bits = 4096
encrypt_key = no
default_md = sha256
prompt = no
utf8 = yes
distinguished_name = req_distinguished_name
req_extensions = v3_req
[req_distinguished_name]
C = UK
ST = Surrey
L = Tucana
O = infoitech
CN = vault.infoitech.co.uk
[v3_req]
basicConstraints = CA:FALSE
subjectKeyIdentifier = hash
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
subjectAltName = @alt_names
[alt_names]
IP.1 = ${IP1}
IP.2 = ${IP2}
DNS.1 = ${DNS1}
DNS.2 = ${DNS2}
EOF
# Generate Vault's certificates and a CSR
openssl genrsa -out "${DIR}/vault.key" 4096
openssl req \
-new -key "${DIR}/vault.key" \
-out "${DIR}/vault.csr" \
-config "${DIR}/openssl.cnf"
# Create our CA
openssl req \
-new \
-newkey rsa:4096 \
-days 3660 \
-nodes \
-x509 \
-subj "/C=UK/ST=London/L=Tucana/O=Vault CA" \
-keyout "${DIR}/ca.key" \
-out "${DIR}/ca.crt"
# Sign CSR with our CA
openssl x509 \
-req \
-days 365 \
-in "${DIR}/vault.csr" \
-CA "${DIR}/ca.crt" \
-CAkey "${DIR}/ca.key" \
-CAcreateserial \
-extensions v3_req \
-extfile "${DIR}/openssl.cnf" \
-out "${DIR}/vault.crt"
# Export combined certs for vault
cat "${DIR}/vault.crt" "${DIR}/ca.crt" > "${DIR}/vault-combined.crt"
/root/docker-projects/vlan.65-vault/01-create-certs.sh It is also recommended and possible to use Let's Encrypt certificates with Vault. You will need to generate the the certs and renew it before expiration 3 months later.
I currently have the acme package installed in my firewalls and with the CloudFlare API it auto renew the certificates every 3 months. I will just need to figure out an automation solution to move the certs once renewed.
The certificate files are located in /tmp/acme/<<CERT_NAME>>/vault.infoitech.co.uk
.
vault.infoitech.co.uk/
├── backup
├── ca.cer
├── fullchain.cer
├── vault.infoitech.co.uk.cer
├── vault.infoitech.co.uk.conf
├── vault.infoitech.co.uk.csr
├── vault.infoitech.co.uk.csr.conf
└── vault.infoitech.co.uk.key
Folder Structure we need to copy fullchain.cer
and vault.infoitech.co.uk.key
to <<PROJECT_PATH>>/ssl
.
4) Permissions The vault container is deployed with a group and user called vault
and because the permissions of local folders are passed to the container mounts. We need to create a vault user in our docker host.
useradd -r -s /usr/bin/nologin vault
Let's set the configuration file and SSL certs with the permissions as below:
-rwxr--r-- 1 root root 1.6K Sep 3 14:06 01-create-certs.sh*
drwxr-xr-x 2 vault vault 4.0K Sep 3 17:28 config/
-rwxr--r-- 1 vault vault 519 Sep 3 17:28 config/config.hcl*
drwxr-xr-x 5 vault vault 4.0K Sep 3 17:08 file/
drwxr-xr-x 2 vault vault 4.0K Sep 1 10:05 logs/
drwxr-xr-x 2 vault vault 4.0K Sep 2 15:03 plugins/
drwxr-xr-x 2 root root 4.0K Sep 3 16:59 ssl/
-rw-r--r-- 1 root root 1.9K Sep 3 16:59 ssl/ca.crt
-rw------- 1 root root 3.2K Sep 3 16:59 ssl/ca.key
-rw-r--r-- 1 root root 41 Sep 3 16:59 ssl/ca.srl
-rw-r--r-- 1 root root 580 Sep 3 16:59 ssl/openssl.cnf
-rw-r--r-- 1 vault vault 3.9K Sep 3 16:59 ssl/vault-combined.crt
-rw-r--r-- 1 root root 2.1K Sep 3 16:59 ssl/vault.crt
-rw-r--r-- 1 root root 1.9K Sep 3 16:59 ssl/vault.csr
-rw-r----- 1 root vault 3.2K Sep 3 16:59 ssl/vault.key
5) Vault Configuration The configuration below will bring the container up and allow us to manage it. You need to adapt to your needs and also check Vault's documentation for further information.
storage "file" {
path = "/vault/file"
}
listener "tcp" {
address = "192.168.65.12:443"
tls_min_version = "tls12"
tls_cert_file = "/vault/ssl/fullchain.cer"
tls_key_file = "/vault/ssl/vault.infoitech.co.uk.key"
}
listener "tcp" {
address = "127.0.0.1:8200"
tls_disable = "true"
}
api_addr = "https://vault.infoitech.co.uk"
cluster_addr = "https://vault.infoitech.co.uk"
ui = true
plugin_directory = "/vault/plugins"
log_level = "info"
default_lease_ttl = "168h"
max_lease_ttl = "720h"
/root/docker-projects/vlan.65-vault/config/config.hcl 6) Starting the container Finally, let's bring the container up.
docker-compose up -d
You can check the container logs to make sure that it started without errors.
docker container logs vault
==> Vault server configuration:
Api Address: https://vault.infoitech.co.uk
Cgo: disabled
Cluster Address: https://vault.infoitech.co.uk:444
Go Version: go1.17.13
Listener 1: tcp (addr: "192.168.65.12:443", cluster address: "192.168.65.12:444", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
Listener 2: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
Log Level: info
Mlock: supported: true, enabled: true
Recovery Mode: false
Storage: file
Version: Vault v1.11.3, built 2022-08-26T10:27:10Z
Version Sha: xxXXXXXxxxXXXXXxx
2022-09-03T17:28:25.579+0100 [INFO] proxy environment: http_proxy="" https_proxy="" no_proxy=""
2022-09-03T17:28:25.628+0100 [INFO] core: Initializing version history cache for core
==> Vault server started! Log data will stream in below:
7) Root Token Vault uses tokens for authentication. The root token is generated during the initial setup and is the only method that cannot be disabled nor the root token expires.
The root token can do anything in Vault. It is extremely important to secure this token. Vault documentation explains how tokens work in more depth .
There is also an excellent tutorial from hashicorp on how to manipulate tokens that worth reading .
When you access the ui
for the first time. Vault will generate the root token and you can use it for the initial setup.
8) Troubleshooting You can use the docker container logs
to inspect the container for errors and if running but not working as expected.
It is possible to login into the container to further investigate. Use vault --help
for a list of commands.
docker exec -it vault /bin/sh
/ # vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed true
Total Shares 1
Threshold 1
Unseal Progress 0/1
Unseal Nonce n/a
Version 1.11.3
Build Date 2022-08-26T10:27:10Z
Storage Type file
HA Enabled false
/ #
The file structure of the project is as below.
/root/docker-projects/vlan.65-vault
├── 01-create-certs.sh
├── config
│ └── config.hcl
├── docker-compose.yml
├── docker-entrypoint.sh
├── Dockerfile
├── file
│ ├── ...
├── logs
├── plugins
├── README.md
└── ssl
├── ca.crt
├── ca.key
├── ca.srl
├── openssl.cnf
├── vault-combined.crt
├── vault.crt
├── vault.csr
└── vault.key
Project File Structure Conclusion This article describes how to spin up a basic Vault container. For more information check the docs and the resources below.
Resources Official Hashicorp Vault Repository
Bash Parameter Expansion
Bash IF Statements Explained
Vault Configuration Docs
Hashicorp HCL - Docs
Dockerfile Reference Guide
Hashicorp Self-Signed TLS Certificates Example
Build Certificate Authority (CA) in Vault with an offline Root