Simplifying Kubernetes Secrets Management with External Secrets

Secure your secrets in cloud-native apps with External Secrets on Oracle Kubernetes Engine. Follow best practices for secure secret management.

In a Kubernetes cluster, managing secrets effectively is crucial for maintaining the security and confidentiality of sensitive information. External Secrets is a Kubernetes extension that allows you to store and manage secrets in external systems, providing a centralized and secure approach to secrets management. In this article, you will learn how to set up External Secrets for different use cases, including Oracle Vault, Oracle Cloud Infrastructure Registry (OCIR), and TLS secrets for utilization with an Ingress Controller.

As Kubernetes adoption grows, securing sensitive information like API keys and credentials becomes crucial. Integrating external secrets with your Kubernetes cluster offers enhanced security, simplified management, granular access control, and easier scalability. Centralizing secrets storage, automating provisioning and rotation, and enforcing least privilege access are just a few benefits.

Setting Up External Secrets for Oracle Vault

Oracle Vault is a robust and secure secrets management service. To set up External Secrets for Oracle Vault, follow these steps:

Setting up Oracle Vault

1. Log in to the OCI Console and navigate to the Vault service.
2. Create a new vault and configure policies and access rules to meet your specific requirements.

Creating the Instance Principle to allow OKE to communicate with Oracle Vault

1. Add all compute instances of OKE to the dynamic group "k8-secrets-vault-group" from the "dev" compartment.
2. In the OCI Console, go to the Identity and Access Management (IAM) section.
3. Select "Policy" and create a new policy.
4. Assign necessary permissions to the policy, granting access to the vault and other required resources.

Allow dynamic-group k8-secrets-vault-group to manage vaults in compartment dev
Allow dynamic-group k8-secrets-vault-group to manage keys in compartment dev
Allow dynamic-group k8-secrets-vault-group to manage secret-family in compartment dev

these policy statements empower the dynamic group "k8-secrets-vault-group" to have granular control over managing vaults, keys, and secret families within the "dev" compartment in Oracle Cloud. It's important to ensure that these permissions are assigned appropriately based on the specific requirements and security considerations of your organization.

Setting Up External Secrets in Your Kubernetes Cluster / OKE

Before diving into specific use cases, let's first understand how to set up External Secrets in your Kubernetes cluster. Follow these steps:

Install the External Secrets Controller

The External Secrets Controller is responsible for managing the lifecycle of external secrets. Install it using Helm or by deploying the YAML manifests provided by the official External Secrets repository.

helm repo add external-secrets https://charts.external-secrets.io
helm update
helm install external-secrets \
    external-secrets/external-secrets \
      -n external-secrets \
      --create-namespace \
      --set installCRDs=true

The above-mentioned instructions will add the External Secrets repository to Helm, update the Helm repositories, and install the External Secrets operator in the designated namespace.

Choose an External Secrets Provider

External Secrets supports various providers, such as AWS Secrets Manager, HashiCorp Vault, Oracle Vault, and more. Select the provider that aligns with your requirements and integrate it with your Kubernetes cluster.

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: oci-vault-global-css
spec:
  provider:
    oracle:
      vault: <Oracle Vault OCID>
      region: me-jeddah-1

For our scenario, I have decided to utilize Oracle Vault for managing external secrets in conjunction with Oracle Kubernetes Engine (OKE). This choice allows us to securely store and access secrets within our OKE deployments.

kubectl apply -f vault-store.yaml

Define External Secret Specifications

Define the specifications for the external secrets you want to manage. These specifications typically include the provider-specific details required to fetch and store secrets securely.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: demo-es
spec:
  refreshInterval: 3m
  secretStoreRef:
    kind: SecretStore
    name: oci-vault-global-css # Must match SecretStore on the cluster
  target:
    name: DEMO_KEY # Name for the secret on the cluster
    creationPolicy: Owner
  dataFrom:
    - key: DEMO_KEY

kubectl apply -f demo-es.yaml

Verify and Test

Ensure the External Secrets Controller can successfully retrieve and inject secrets from Oracle Vault into your Kubernetes pods. Test the setup thoroughly to validate its functionality.

Example-1 Setting Up External Secret for OCIR

Oracle Cloud Infrastructure Registry (OCIR) is a fully managed container registry. To integrate OCIR with External Secrets, follow these steps:

Define External Secret Specifications

Specify the necessary configurations to fetch and manage secrets related to OCIR. These configurations might include registry credentials, image paths, and other relevant details.

apiVersion: external-secrets.io/v1beta1
kind: ClusterExternalSecret
metadata:
  name: ocir-global-ces
spec:
  externalSecretName: "ocir-global-sec"
  refreshTime: 3s
  namespaceSelector:
    matchLabels: {}
  externalSecretSpec:
    refreshInterval: 3s
    secretStoreRef:
      name: oci-vault-global-css
      kind: ClusterSecretStore
    target:
      template:
        type: kubernetes.io/dockerconfigjson
        data:
          .dockerconfigjson: "{{ .ocirsecret | toString }}"
      name: ocir-global-sec
      creationPolicy: Owner
    data:
      - secretKey: ocirsecret
        remoteRef:
          key: OCIR_AUTH_CONFIG

Your secret value in your OCI vault will look like this

{"auths":{"your.private.registry.example.com":{"username":"janedoe","password":"xxxxxxxxxxx","email":"jdoe@example.com","auth":"c3R...zE2"}}}

Generate Docker Config JSON Value

You can generate this docker config JSON value by creating the temporary secret in your Kubernetes cluster for OCIR, execute the following command:

kubectl create secret docker-registry ocirsecret --docker-server=jed.ocir.io --docker-username=[OCI Namespace]/[OCI Account UserName] --docker-password='[OCIR APP Password]' --docker-email=[OCI Account Email]

Retrieve the secret value using the following command:

kubectl get secret ocirsecret -n <namespace> | -o jsonpath="{.data\.dockerconfigjson}" | base64 -d

Make sure to replace <namespace> it with the appropriate namespace in your cluster.

When you apply the 'ocir-ces.yaml' file, it will transform into an external secret accessible across all namespaces. Consequently, you will have the capability to make use of this secret.

Validate and Utilize OCIR Secret

Ensure that the External Secrets Controller can successfully retrieve secrets of OCIR from Oracle Vault and use them within your OKE  environment. Validate the setup by deploying pods that utilize the OCIR secrets.

Example-2 Setting Up External Secret for TLS with an Ingress Controller

TLS (Transport Layer Security) certificates are crucial for securing communication over the internet. To set up External Secrets for TLS certificates with an Ingress Controller, follow these steps:

Acquire TLS Certificates

Obtain the necessary TLS certificates from a trusted certificate authority (CA) or generate self-signed certificates.

Store Certificates in an Oracle Vault

Store the TLS certificates CRT and KEY file content securely on Oracle Vault. Ensure the necessary permissions and access controls are in place.

Define External Secret Specifications

apiVersion: external-secrets.io/v1beta1
kind: ClusterExternalSecret
metadata:
  name: tls-global-ces
spec:
  externalSecretName: "tls-global-sec"
  refreshTime: 3s
  namespaceSelector:
    matchLabels: {}
  externalSecretSpec:
    refreshInterval: 3s
    secretStoreRef:
      name: oci-vault-global-css
      kind: ClusterSecretStore
    target:
      template:
        type: kubernetes.io/tls
        data:
          tls.crt: "{{ .tlscrtsecret | toString }}"
          tls.key: "{{ .tlskeysecret | toString }}"
      name: idwisecom-tls-global-sec
      creationPolicy: Owner
    data:
      - secretKey: tlscrtsecret
        remoteRef:
          key: TLS_CRT
      - secretKey: tlskeysecret
        remoteRef:
          key: TLS_KEY

Configure Ingress Controller

Integrate the External Secrets provider with your Ingress Controller to fetch and utilize the TLS certificates during ingress configuration. This enables secure communication with your applications.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ing
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "10m"

spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - api.example.com
      secretName: tls-global-sec
  rules:
    - host: api.example.com
      http:
        paths:
          - pathType: Prefix
            backend:
              service:
                name: "example-svc"
                port:
                  number: 80
            path: /

Verify and Monitor

Validate that the Ingress Controller successfully retrieves the TLS certificates from the External Secrets Oracle Vault and applies them to the ingress resources. Monitor the setup to ensure the certificates are regularly updated and remain valid.

Conclusion

Setting up External Secrets with Oracle Kubernetes Engine on Oracle Cloud provides a secure and efficient method for managing secrets in your cloud-native applications. By integrating with Oracle Vault, you can leverage its robust secret management capabilities while benefiting from the flexibility and scalability of Kubernetes.

In this article,  I've covered the Oracle Cloud side setup, installation, and configuration of the External Secrets Controller, and the basic setup for TLS and Docker. With this foundation in place, you can now securely manage your secrets and ensure their availability to your Kubernetes applications.

Remember to always follow best practices for secret management and regularly review and update your access permissions to maintain a secure environment.

Resources

1. https://external-secrets.io
2. https://external-secrets.io/v0.4.4/provider-oracle-vault/

To increase your range of knowledge, I recommend reading my other blog on Create cloud system architecture diagrams without using any design tool

Hi! I am Safoor Safdar a Senior SRE. Read More. Don't hesitate to reach out! You can find me on Linkedin, or simply drop me an email at me@safoorsafdar.com