This is part 2 of 3 in our blog series on how to build out a secrets management environment using HashiCorp Vault® (“Vault”, for short). In our first blog of the series (part 1) we covered what Vault is. We also went through an overview of the solution. In this second part of the series, we build out Vault:
Review Prerequisites
Install Vault on the Linux Server
Configure the Vault Service
This will position us so that in our final blog (part 3) we can retrieve a secret from Vault. We will use a Python script to do that. And, finally, we will cover some bonus material.
Building the Environment
This is where we will spend the bulk of our time and effort! First, I list out the prerequisites that are needed to build out both the server and the desktop. Then, I cover two of the three major steps to build out the environment. We will cover major step #3 in our next blog entry.
Prerequisites
Ubuntu Server
An Ubuntu server 20.04 LTS with a sudo user and an accessible IP address is required for hosting Vault.
The unzip and jq packages also need to be installed.
Issue the commands in Figure 1 to install these two packages on the Ubuntu server.
sudo apt-get install unzip
sudo apt-get install jq Figure 1 - Install unzip and jq
Windows 10 Desktop
A Windows 10 desktop system with an IP address that can access the Ubuntu server is required for accessing Vault.
Visual Studio Code or your preferred IDE that can run Python scripts
Python Plugin for VS Code
Python v3.x – I recommend installing this for all users and adding it to the PATH variable.
Python requests library – Run the pip command in Figure 2 inside the VS Code console to add the requests library for Python into VS Code.
pip install requestsFigure 2 - Install Python requests Library
Create or obtain an SSL Certificate for Vault
An SSL certificate for the Ubuntu server is required. You can either obtain one from an already established CA (certificate authority) or you can create one on your own using OpenSSL. For proof-of-concept purposes, a self-signed certificate (created on your own) is acceptable. Best practice in production is to obtain a signed certificate from a trusted CA. I’ve described the two options below. I used a self-signed SSL certificate (option #2 below).
Option 1: Obtain an SSL Certificate from a CA
If you are working in an environment with a PKI, request a certificate from your CA (ask your CA administrator if you don’t know how). Make sure to include subject alternative name(s) (SAN) that represent the Vault server you are building. Use the DNS name and/or IP address for the Vault server you will build below. In these instructions we will be using the IP address of the server, but you can substitute the DNS name if you have that configured. The public key must be in the CER, CRT, or PEM formats. The private key must be in the PEM format.
Note: If you are starting with a PFX file you will need to convert and split it into the public and private keys using a tool like OpenSSL. When converting using OpenSSL, it may add values to the files which will cause Vault to fail to load the certificate. Keep only the info between and including “-----BEGIN” and “END-----".
Option 2: Generate a self-signed SSL certificate using OpenSSL
To do this, you will need OpenSSL installed. If you do not have it installed already on your Ubuntu server, there are a couple options for loading OpenSSL on your server. The easiest way is to use Ubuntu’s app-get command (sudo apt-get install -y openssl). Alternatively, you can download the latest version of OpenSSL from the openssl.org site: https://www.openssl.org/source/.
Create the OpenSSL Configuration File
You need to create an OpenSSL configuration file to ensure you create the correct type of certification. Use the following steps to create the configuration file. On the Ubuntu command line, type the command in Figure 3:
cat <<EOF>>vaultcert.cnfFigure 3 - Create vaultcert.cnf
This command will create a flat file with everything until the EOF characters at the end.
Paste in the text in Figure 4.
Important Note: Be sure to modify {{{YOUR VAULT IP}}} at the end. This should be the IP of your Ubuntu server.
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
C = US
ST = PA
L = Lancaster
O = TestCompany
OU = TestOrg
CN = vault.test
[v3_req]
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
IP.1 = {{{YOUR VAULT IP}}}
EOF Figure 4 - vaultcert.cnf
Run the command in Figure 5 to generate the certificate and key. Note: It must be pasted in as a single line on the command line.
openssl req -x509 -nodes -days 730 -newkey rsa:2048 -keyoutvaultserver.key -outvaultserver.cer -config vaultcert.cnfFigure 5 - Generate Certificate and Key
This will generate two files in your current directory: vaultserver.key and vaultserver.cer. You will use the vaultserver.key and vaultserver.cer files when setting up the Vault configuration files below. These will be the files used for the SSL connection to the Vault.
Major Step #1: Installing Vault on the Linux Server
Once the Ubuntu system is up, log in with a user account with sudo access.
Create the vault user as a system service user. The user in this exercise will be called “vault”. Issue the command in Figure 6 to create the user.
sudouseradd -r --system vaultFigure 6 - Add vault User
Download and copy the Vault binary to the correct directory. Issue the commands in Figure 7.
sudo mkdir /usr/local/bin/vault.d
wget https://releases.hashicorp.com/vault/1.7.0-rc2/vault_1.7.0-rc2_linux_amd64.zip
unzip vault_1.7.0-rc2_linux_amd64.zip
cp ~/vault /usr/local/bin/vault.d/ Figure 7 - Download Vault Binary
Note: Check the URL above (https://releases.hashicorp.com/vault/...) and modify it, if needed. I used HashiCorp Vault® 1.7.0 RC2. The latest version can be found at https://releases.hashicorp.com/vault/. Make sure to download the zip file for the hardware architecture you are running your Ubuntu server on.
Set the ownership and permissions on the Vault binary.
Issue the two commands in Figure 8 to change the user and group ownership to both be “vault” and the rights to be 775.
sudo chown -R vault:vault /usr/local/bin/vault.d
sudo chmod -R 775 /usr/local/bin/vault.d Figure 8 - Set Rights on vault.d
Create the Vault configuration folder. Then move the Vault server certificate and key files into the certs directory. Use the commands in Figure 9.
sudo mkdir /etc/vault.d/
sudo mkdir /etc/vault.d/certs
sudo mv vaultserver.cer /etc/vault.d/certs
sudo mv vaultserver.key /etc/vault.d/certs Figure 9 - Move certificate and key files
Run the command in Figure 10 and paste the text in Figure 11 into the command window. This snippet will create the vault configuration file, vault.hcl.
cat <<EOF >> vault.hclFigure 10 - Create vault.hcl File
ui = true
diable_mlock=true
HTTP listener
listener "tcp" {
address="127.0.0.1:8200"
tls_disable = 1
}
listener "tcp" {
address = "0.0.0.0:8200"
tls_disable = "false"
tls_cert_file = "/etc/vault.d/certs/vaultserver.cer"
tls_key_file = "/etc/vault.d/certs/vaultserver.key"
}
storage "raft" {
path = "/opt/raft/data"
node_id="node1"
}
cluster_addr="http://127.0.0.1:8201"
api_addr="http://127.0.0.1:8200"
EOF Figure 11 - vault.hcl
Move the vault.hcl file into the config directory and set the permissions. Use the commands in Figure 12.
mv vault.hcl /etc/vault.d
sudo chown -R vault:vault /etc/vault.d
sudo chmod -R 774 /etc/vault.d/ Figure 12 - Move vault.hcl File
Create the Vault Raft storage directory and set permissions. Issue the commands in Figure 13.
This is where Vault will store the encrypted secrets. (Note: “Raft” is a proprietary storage format that HashiCorp developed to store the secrets in an encrypted format. For more information about Raft, you can read HashiCorp’s documentation here.)
sudo mkdir /opt/raft
sudo mkdir /opt/raft/data
sudo chown -R vault:vault /opt/raft
sudo chmod -R 775 /opt/raft Figure 13 - Setup raft Directory
Major Step #2: Configuring the Vault Service
We will create a vault service definition for systemd.
Run the command in Figure 14 and paste the text in Figure 15 into the command window to create the service definition (configuration) file.
cat << EOF >> vault.serviceFigure 14 - Create vault.service File
[Unit]
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/vault.d/vault.hcl
StartLimitBurst=3
[Service]
User=vault
Group=vault
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
Capabilities=CAP_IPC_LOCK+ep
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/local/bin/vault.d/vault server -config=/etc/vault.d/vault.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
StartLimitInterval=60
StartLimitIntervalSec=60
StartLimitBurst=3
LimitNOFILE=65536
LimitMEMLOCK=infinity
[Install]
WantedBy=multi-user.target
EOF Figure 15 - vault.service File
Run the commands in Figure 16 to copy the vault service definition file into the system directory, configure “read” permissions on the file, and then startup the Vault service.
sudo mv vault.service /etc/systemd/system
sudo chmod 644 /etc/systemd/system/vault.service
sudo chown root:root /etc/systemd/system/vault.service
sudo systemctl enable vault.service
sudo systemctl start vault.service Figure 16 - Start Vault Service
Verify that the Vault service is running by typing the command in Figure 17 to see the status of running services.
systemctl --type=serviceFigure 17 - Verify Vault Service is Running
You should see the output in Figure 18.
Figure 18 - systemctl Output
If the vault service fails to start, it is usually a permissions issue, a missing directory, or a missing file. To troubleshoot the issue, run the command in Figure 19 to see the console output of the Vault binary. This may help you determine why the service is not starting.
sudo journalctl -u vaultFigure 19 - Run journalctl Command
Summary and What’s Next
Now that we’ve built out our Vault environment, we are ready for Major Step #3: Setting Up the Vault itself. And, finally, we will retrieve a secret from Vault. We’ll cover that in our final blog of the series (part 3).
Reach Out to Us
If you need more information about how to implement Hashicorp Vault® or another vaulting solution, or if you need help in automating your CI/CD pipeline, reach out to us. CMG (Career Mentor Group) specializes in automation and integration. If you want, you can even schedule a meeting with us to discuss possible solutions or help with DevOps resources.