Skip to main content

Key/Value Secrets Engine - API Calls

Challenge​

The KV secrets engine v1 does not provide a way to version or roll back secrets. This made it difficult to recover from unintentional data loss or overwrite when more than one user is writing at the same path.

Solution​

Run the version 2 of KV secrets engine which can retain a configurable number of secret versions. This enables older versions' data to be retrievable in case of unwanted deletion or updates of the data. In addition, its Check-and-Set operations can be used to protect the data from being overwritten unintentionally.

Step 1: Check the KV secrets engine version​

Ensure $VAULT_TOKEN value is set on the command line.

(Persona: admin)

The Vault server started in dev mode, automatically enables v2 of the KV secrets engine at the secret/ path. Verify that KV secrets engine is enabled and is set to version 2.

Display all the enabled secrets engine.

curl --header "X-Vault-Token: $VAULT_TOKEN" \
       $VAULT_ADDR/v1/sys/mounts | jq

NOTE: This example uses jq to process the JSON output for readability.

Step 2: Write secrets​

To understand how the versioning works, let's write some test data.

Create an API request payload containing some test data.

tee payload.json <<EOF
{
  "data": {
    "name": "ACME Inc.",
    "contact_email": "[email protected]"
  }
}
EOF

Write some data at secret/customer/acme.

curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request POST \
      --data @payload.json \
      $VAULT_ADDR/v1/secret/data/customer/acme | jq

Notice that the endpoint for KV v2 is /secret/data/<path>; therefore to write secrets at secret/customer/acme, the API endpoint becomes /secret/data/customer/acme.

Create an API request payload containing the data to update the current data.

tee payload.json <<EOF
{
  "data": {
    "name": "ACME Inc.",
    "contact_email": "[email protected]"
  }
}
EOF

Now you have two versions of the secret/customer/acme data.

Read back the secret.

curl --header "X-Vault-Token: $VAULT_TOKEN" \
      $VAULT_ADDR/v1/secret/data/customer/acme | jq -r ".data"

Step 3: Retrieve a specific version of secret​

You may run into a situation where you need to view the secret before an update.

Get version 1 of the secret defined at the path secret/customer/acme

curl --header "X-Vault-Token: $VAULT_TOKEN" \
      $VAULT_ADDR/v1/secret/data/customer/acme\?version=1 | jq -r ".data"

Get the metadata of the secret defined at the path secret/customer/acme.

curl --header "X-Vault-Token: $VAULT_TOKEN" \
      $VAULT_ADDR/v1/secret/metadata/customer/acme | jq -r ".data"

Step 4: Specify the number of versions to keep​

By default, the kv-v2 secrets engine keeps up to 10 versions. Let's limit the maximum number of versions to keep to be 4.

Create an API request payload specifying the max_versions to 4.

tee payload-config.json<<EOF
{
  "max_versions": 4,
  "cas_required": false
}
EOF

Configure the secrets engine at path secret/ to limit all secrets to a maximum of 4 versions.

curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --request POST \
   --data @payload-config.json \
   $VAULT_ADDR/v1/secret/config

Every secret stored for this engine are set to a maximum of 4 versions.

Display the secrets engine configuration settings.

curl --header "X-Vault-Token: $VAULT_TOKEN" \
   $VAULT_ADDR/v1/secret/config | jq -r ".data"

Configure the secret at path secret/customer/acme to limit secrets to a maximum of 4 versions.

curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --request POST \
   --data @payload.json \
   $VAULT_ADDR/v1/secret/metadata/customer/acme

The secret can also define the maximum number of versions.

Create four more secrets at the path secret/customer/acme.

curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --request POST \
   --data @payload.json \
   $VAULT_ADDR/v1/secret/data/customer/acme

Get the metadata of the secret defined at the path secret/customer/acme.

curl --header "X-Vault-Token: $VAULT_TOKEN" \
      $VAULT_ADDR/v1/secret/metadata/customer/acme | jq -r ".data"

The metadata displays the current_version and the history of versions stored. Secrets stored at this path are limited to 4 versions. Version 1 and 2 are deleted.

Verify that version 1 of the secret defined at the path secret/customer/acme are deleted.

curl --header "X-Vault-Token: $VAULT_TOKEN" \
      $VAULT_ADDR/v1/secret/data/customer/acme\?version=1 | jq

The output should look as follow:

{
  "errors": []
}

Step 5: Delete versions of secret​

Delete version 4 and 5 of the secrets at path secret/customer/acme.

curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --request POST \
   --data '{ "versions":[4,5] }' \
   $VAULT_ADDR/v1/secret/delete/customer/acme

Get the metadata of the secret defined at the path secret/customer/acme.

curl --header "X-Vault-Token: $VAULT_TOKEN" \
      $VAULT_ADDR/v1/secret/metadata/customer/acme | jq -r ".data.versions"

The metadata on versions 4 and 5 reports its deletion timestamp (deletion_time); however, the destroyed parameter is set to false.

Undelete version 5 of the secrets at path secret/customer/acme.

curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --request POST \
   --data '{ "versions":[5] }' \
   $VAULT_ADDR/v1/secret/undelete/customer/acme

Step 6: Permanently delete data​

Destroy version 4 of the secrets at path secret/customer/acme.

curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --request POST \
   --data '{ "versions":[4] }' \
   $VAULT_ADDR/v1/secret/destroy/customer/acme

Get the metadata of the secret defined at the path secret/customer/acme.

curl --header "X-Vault-Token: $VAULT_TOKEN" \
      $VAULT_ADDR/v1/secret/metadata/customer/acme | jq -r ".data.versions"

The metadata indicates that Version 4 is destroyed.

Delete all versions of the secret at the path secret/customer/acme

curl --header "X-Vault-Token: $VAULT_TOKEN" \
      --request DELETE \
      $VAULT_ADDR/v1/secret/metadata/customer/acme

Step 7: Configure automatic data deletion​

As of Vault 1.2, you can configure the length of time before a version gets deleted. For example, if your organization requires data to be deleted after 10 days from its creation, you can configure the K/V v2 secrets engine to do so by setting the delete_version_after parameter.

Configure the secrets at path secret/test to delete versions after 40 seconds.

curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --request POST \
   --data '{ "delete_version_after": "40s" }' \
   $VAULT_ADDR/v1/secret/metadata/test

Create a secret at the path secret/test.

curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --request POST \
   --data '{"data": {"message": "data1"}}' \
   $VAULT_ADDR/v1/secret/data/test | jq -r ".data"

Again, create a secret at the path secret/test.

curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --request POST \
   --data '{"data": {"message": "data2"}}' \
   $VAULT_ADDR/v1/secret/data/test

Again, create a secret at the path secret/test.

curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --request POST \
   --data '{"data": {"message": "data3"}}' \
   $VAULT_ADDR/v1/secret/data/test

Get the metadata of the secret defined at the path secret/test

curl --header "X-Vault-Token: $VAULT_TOKEN" \
      $VAULT_ADDR/v1/secret/metadata/test | jq -r ".data"

The metadata displays a deletion_time set on each version. After 40 seconds, the data gets deleted automatically. The data has not been destroyed.

Get version 1 of the secret defined at the path `secret/test.

curl --header "X-Vault-Token: $VAULT_TOKEN" \
      $VAULT_ADDR/v1/secret/data/test\?version=1 | jq -r ".data"

Step 8: Check-and-Set operations​

The v2 of KV secrets engine supports a Check-And-Set operation to prevent unintentional secret overwrite. When you pass the cas flag to Vault, it first checks if the key already exists.

Display the secrets engine configuration settings.

vault read secret/config

The cas_required setting is false. The KV secrets engine defaults to disable the Check-And-Set operation.

Create an API request payload that sets cas_required to true.

tee payload-cas.json<<EOF
{
  "max_versions": 10,
  "cas_required": true
}
EOF

Configure the secrets engine at path secret/ to enable Check-And-Set.

curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data @payload-cas.json \
    $VAULT_ADDR/v1/secret/config

Configure the secret at path secret/partner to enable Check-And-Set.

curl --header "X-Vault-Token: $VAULT_TOKEN" \
   --request POST \
   --data @payload.json \
   $VAULT_ADDR/v1/secret/metadata/partner

Once check-and-set is enabled, every write operation requires the cas parameter with the current version of the secret. Set cas to 0 when a secret at that path does not already exist.

Create an API request payload with secret data and the cas value to 0.

tee payload.json <<EOF
{
  "options": {
    "cas": 0
  },
  "data": {
    "name": "Example Co.",
    "partner_id": "123456789"
  }
}
EOF

Create a new secret at the path secret/partner.

curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data @payload.json \
    $VAULT_ADDR/v1/secret/data/partner | jq -r ".data"

Create an API request payload with updated secret data and the cas value to 1.

tee payload.json <<EOF
{
  "options": {
    "cas": 1
  },
  "data": {
    "name": "Example Co.",
    "partner_id": "ABCDEFGHIJKLMN"
  }
}
EOF

Overwrite the secret at the path secret/partner.

curl --header "X-Vault-Token: $VAULT_TOKEN" \
    --request POST \
    --data @payload.json \
    $VAULT_ADDR/v1/secret/data/partner | jq -r ".data"