Enabling data catalogs on Hybrid Manager

Hybrid Manager's data catalog requires additional configuration to manage Lakehouse data storage. This involves creating a Kubernetes secret with a confounding key that the Lakekeeper service can use to store encrypted data.

Creating a secret with the key

For EKS installations using the eks-install-secrets.sh script, you don't need to create a key and secret. In that case, skip this step and go to Backing up the confounding key.

Other EKS setups and operating systems require manual secret creation with the confounding key.

1. Create a confounding key and store it in a variable:

   → WrapCopy

   PG_CONFOUNDING_KEY=$(dd if=/dev/urandom bs=32 count=1 2>/dev/null | base64)

   Note

   • A confounding key is a randomized string that's at least 32 bytes long.
   • Create a confounding key for each Hybrid Manager deployment.

2. Create a namespace for the service:

   → WrapCopy

   kubectl create namespace upm-lakekeeper

3. Create a secret that references the new confounding key and is stored in the dedicated namespace:

   → WrapCopy

   kubectl apply -f - <<EOF
     apiVersion: v1
     kind: Secret
     metadata:
       name: pg-confounding-key
       namespace: upm-lakekeeper
     stringData:
       PG_CONFOUNDING_KEY: ${PG_CONFOUNDING_KEY}
   EOF

   After you've configured the secret, continue with Hybrid Manager installation.

Backing up the confounding key

The Hybrid Manager administrator must keep the confounding key safe and back it up.

Fetch the key:

kubectl get secrets -n upm-lakekeeper pg-confounding-key -o yaml

Store the key safely.