AEWS [3기] 10주차 - K8S 시크릿 관리

Kubernetes

AEWS [3기] 10주차 - K8S 시크릿 관리

yu3papa 2025. 4. 10. 11:36

1. Valut 개요

HashiCorp Vault는 신원 기반(identity-based)의 시크릿 및 암호화 관리 시스템입니다. 이 시스템은 인증(authentication) 및 인가(authorization) 방법을 통해 암호화 서비스를 제공하여 비밀에 대한 안전하고 감사 가능하며 제한된 접근을 보장합니다.
시크릿(Secret)이란 접근을 철저히 통제하고자 하는 모든 것을 의미하며, 예를 들어 토큰, API 키, 비밀번호, 암호화 키 또는 인증서 등이 이에 해당합니다. Vault는 모든 시크릿에 대해 통합된 인터페이스를 제공하면서, 엄격한 접근 제어와 상세한 감사 로그 기록 기능을 제공합니다.

대표적인 시크릿의 종류
- 비밀번호
- Cloud Credentials : AWS, GCP, Azure, NCP
- Database Credentials : MySQL,
- SSH Key
- Token, API Key : GitHub, Telegram, Slack, OpenAI, Claude
- 인증서(PKI, TLS 등)

Vault의 동작방식?

https://developer.hashicorp.com/vault/docs/what-is-vault#how-does-vault-work
Vault는 주로 **토큰(Token)**을 기반으로 작동하며, 이 토큰은 클라이언트의 **정책(Policy)**과 연결되어 있습니다. 각 정책은 경로(path) 기반으로 설정되며, 정책 규칙은 클라이언트가 해당 경로에서 수행할 수 있는 작업과 접근 가능성을 제한합니다.
Vault에서는 토큰을 수동으로 생성해 클라이언트에 할당할 수도 있고, 클라이언트가 로그인하여 토큰을 직접 획득할 수도 있습니다.

1. 인증 (Authenticate): Vault에서 인증은 클라이언트가 Vault에 자신이 누구인지 증명할 수 있는 정보를 제공하는 과정입니다. 클라이언트가 인증 메서드를 통해 인증되면, 토큰이 생성되고 정책과 연결됩니다.
2. 검증 (Validation): Vault는 Github, LDAP, AppRole 등과 같은 신뢰할 수 있는 외부 소스를 통해 클라이언트를 검증합니다.
3. 인가 (Authorize): 클라이언트는 Vault의 보안 정책과 비교됩니다. 이 정책은 Vault 토큰을 사용하여 클라이언트가 접근할 수 있는 API 엔드포인트를 정의하는 규칙의 집합입니다. 정책은 Vault 내 특정 경로나 작업에 대한 접근을 허용하거나 거부하는 선언적 방식으로 권한을 제어합니다.
4. 접근 (Access): Vault는 클라이언트의 신원에 연관된 정책을 기반으로 토큰을 발급하여 비밀, 키, 암호화 기능 등에 대한 접근을 허용합니다. 클라이언트는 이후 작업에서 해당 Vault 토큰을 사용할 수 있습니다.

호텔 체크인 절차에 비유한 Vault의 동작방식 이해

2. 실습환경 구성

Jenkins (Docker Compose) + K8S v1.32 (kind 로 구성)

Jenkins 컨테이너 기동 -> Docker Compose 이용

# 작업 디렉토리 생성 후 이동
[yu3papa@iworks ~]$ mkdir ~/cicd-labs; cd $_

# docker-compose.yaml 파일 생성
[yu3papa@iworks cicd-labs]$ cat <<EOT > docker-compose.yaml

services:

jenkins:

container_name: jenkins

image: jenkins/jenkins

restart: unless-stopped

networks:

- cicd-network

ports:

- "8080:8080"

- "50000:50000"

volumes:

- /var/run/docker.sock:/var/run/docker.sock

- jenkins_home:/var/jenkins_home

volumes:

jenkins_home:

networks:

cicd-network:

driver: bridge

EOT

# Jenkins 실행
[yu3papa@iworks cicd-labs]$ docker compose up -d
[yu3papa@iworks cicd-labs]$ docker compose ps
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
jenkins jenkins/jenkins "/usr/bin/tini -- /u…" jenkins 2 minutes ago Up 2 minutes 0.0.0.0:8080->8080/tcp, [::]:8080->8080/tcp, 0.0.0.0:50000->50000/tcp, [::]:50000->50000/tcp

# Jenkins 초기 암호 확인
[yu3papa@iworks cicd-labs]$ docker compose exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword
3d8b78ff7041438fa99dd56083db39e0

# Jenkins 웹 접속 >> 추천 플러그인 설치 >> 계정 / 암호 입력 >> admin / qwe123

# Valult 플러그인 설치
# https://plugins.jenkins.io/hashicorp-vault-plugin/

K8S v1.32 클러스터 구성 (kind 이용)

# 자신의 IP 확인
[yu3papa@iworks cicd-labs]$ MyIP=192.168.10.4

# kind-3node.yaml 파일 생성
[yu3papa@iworks cicd-labs]$ cat > kind-3node.yaml <<EOF

kind: Cluster

apiVersion: kind.x-k8s.io/v1alpha4

networking:

apiServerAddress: "127.0.0.1" # $MyIP로 설정하셔도 됩니다.

nodes:

- role: control-plane

extraPortMappings:

- containerPort: 30000

hostPort: 30000

- containerPort: 30001

hostPort: 30001

- containerPort: 30002

hostPort: 30002

- containerPort: 30003

hostPort: 30003

- containerPort: 30004

hostPort: 30004

- containerPort: 30005

hostPort: 30005

- containerPort: 30006

hostPort: 30006

- role: worker

EOF

# K8S v1.32 클러스터 구성
[yu3papa@iworks cicd-labs]$ kind create cluster --config kind-3node.yaml --name myk8s --image kindest/node:v1.32.2
Creating cluster "myk8s" ...
✓ Ensuring node image (kindest/node:v1.32.2) 🖼
✓ Preparing nodes 📦 📦 📦
✓ Writing configuration 📜
✓ Starting control-plane 🕹️
✓ Installing CNI 🔌
✓ Installing StorageClass 💾
✓ Joining worker nodes 🚜
Set kubectl context to "kind-myk8s"
You can now use your cluster with:

kubectl cluster-info --context kind-myk8s

Thanks for using kind! 😊

# 확인
[yu3papa@iworks cicd-labs]$ kubectl get no
NAME STATUS ROLES AGE VERSION
myk8s-control-plane Ready control-plane 32s v1.32.2
myk8s-worker Ready <none> 17s v1.32.2
myk8s-worker2 Ready <none> 17s v1.32.2

[yu3papa@iworks cicd-labs]$ kubectl cluster-info
Kubernetes control plane is running at https://127.0.0.1:40623
CoreDNS is running at https://127.0.0.1:40623/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Argo CD 설치 및 기본 설정 --> helm_chart 이용

# namespace 생성 및 values.yaml 파일 생성
[yu3papa@iworks cicd-labs]$ kubectl create ns argocd
namespace/argocd created

[yu3papa@iworks cicd-labs]$ cat <<EOF > argocd-values.yaml

dex:

enabled: false

server:

service:

type: NodePort

nodePortHttps: 30002

extraArgs:

- --insecure # HTTPS 대신 HTTP 사용

EOF

# 설치
[yu3papa@iworks cicd-labs]$ helm repo add argo https://argoproj.github.io/argo-helm
"argo" has been added to your repositories
[yu3papa@iworks cicd-labs]$ helm install argocd argo/argo-cd --version 7.8.13 -f argocd-values.yaml --namespace argocd
NAME: argocd
LAST DEPLOYED: Sat Apr 12 08:51:08 2025
NAMESPACE: argocd
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
In order to access the server UI you have the following options:

1. kubectl port-forward service/argocd-server -n argocd 8080:443

 and then open the browser on http://localhost:8080 and accept the certificate

2. enable ingress in the values file `server.ingress.enabled` and either
 - Add the annotation for ssl passthrough: https://argo-cd.readthedocs.io/en/stable/operator-manual/ingress/#option-1-ssl-passthrough
 - Set the `configs.params."server.insecure"` in the values file and terminate SSL at your ingress: https://argo-cd.readthedocs.io/en/stable/operator-manual/ingress/#option-2-multiple-ingress-objects-and-hosts

After reaching the UI the first time you can login with username: admin and the random password generated during the installation. You can find the password by running:

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d

(You should delete the initial secret afterwards as suggested by the Getting Started Guide: https://argo-cd.readthedocs.io/en/stable/getting_started/#4-login-using-the-cli)

# 설치 확인
[yu3papa@iworks cicd-labs]$ kubectl get pod,svc,ep,secret,cm -n argocd
NAME READY STATUS RESTARTS AGE
pod/argocd-application-controller-0 1/1 Running 0 112s
pod/argocd-applicationset-controller-cccb64dc8-wj6pz 1/1 Running 0 112s
pod/argocd-notifications-controller-7cd4d88cd4-5x7bj 1/1 Running 0 112s
pod/argocd-redis-6c5698fc46-j2sdg 1/1 Running 0 112s
pod/argocd-repo-server-5f6c4f4cf4-zknxv 1/1 Running 0 112s
pod/argocd-server-7cb958f5fb-4prmx 1/1 Running 0 112s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/argocd-applicationset-controller ClusterIP 10.96.208.156 <none> 7000/TCP 112s
service/argocd-redis ClusterIP 10.96.29.247 <none> 6379/TCP 112s
service/argocd-repo-server ClusterIP 10.96.135.189 <none> 8081/TCP 112s
service/argocd-server NodePort 10.96.105.104 <none> 80:30080/TCP,443:30002/TCP 112s

NAME ENDPOINTS AGE
endpoints/argocd-applicationset-controller 10.244.1.4:7000 112s
endpoints/argocd-redis 10.244.2.2:6379 112s
endpoints/argocd-repo-server 10.244.2.3:8081 112s
endpoints/argocd-server 10.244.1.5:8080,10.244.1.5:8080 112s

NAME TYPE DATA AGE
secret/argocd-initial-admin-secret Opaque 1 109s
secret/argocd-notifications-secret Opaque 0 112s
secret/argocd-redis Opaque 1 117s
secret/argocd-secret Opaque 3 112s
secret/sh.helhttp://m.release.v1.argocd.v1 helm.sh/release.v1 1 2m36s

NAME DATA AGE
configmap/argocd-cm 9 112s
configmap/argocd-cmd-params-cm 32 112s
configmap/argocd-gpg-keys-cm 0 112s
configmap/argocd-notifications-cm 1 112s
configmap/argocd-rbac-cm 4 112s
configmap/argocd-redis-health-configmap 2 112s
configmap/argocd-ssh-known-hosts-cm 1 112s
configmap/argocd-tls-certs-cm 0 112s
configmap/kube-root-ca.crt 1 3m19s

# 최초 접속 암호 확인
[yu3papa@iworks cicd-labs]$ kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d ;echo
r6xu8oml25Zamg8u

# Argo CD 웹 접속 주소 확인 : 초기 암호 입력 (admin 계정)

3. Kubernetes에 Vault 설치

Helm을 사용한 Vault 배포

# 네임스페이스 생성 및 Helm Repo 추가
[yu3papa@iworks cicd-labs]$ kubectl create namespace vault
namespace/vault created

[yu3papa@iworks cicd-labs]$ helm repo add hashicorp https://helm.releases.hashicorp.com
"hashicorp" has been added to your repositories

# Helm Chart 설정 Values 설정 및 배포
[yu3papa@iworks cicd-labs]$ cat <<EOF > override-values.yaml

global:

enabled: true

tlsDisable: true # Disable TLS for demo purposes

server:

image:

repository: "hashicorp/vault"

tag: "1.19.0"

standalone:

enabled: true

replicas: 1 # 단일 노드 실행

config: |

ui = true

disable_mlock = true

cluster_name = "vault-local"

listener "tcp" {

address = "[::]:8200"

cluster_address = "[::]:8201"

tls_disable = 1

}

storage "raft" { # Raft 구성 권장

path = "/vault/data"

node_id = "vault-dev-node-1"

}

service:

enabled: true

type: NodePort

port: 8200

targetPort: 8200

nodePort: 30000 # Kind에서 열어둔 포트 중 하나 사용

injector:

enabled: true

ui:

enabled: true

serviceType: "NodePort"

EOF

[yu3papa@iworks cicd-labs]$ helm upgrade vault hashicorp/vault -n vault -f override-values.yaml --install
Release "vault" does not exist. Installing it now.
NAME: vault
LAST DEPLOYED: Sat Apr 12 09:47:02 2025
NAMESPACE: vault
STATUS: deployed
REVISION: 1
NOTES:
Thank you for installing HashiCorp Vault!

Now that you have deployed Vault, you should look over the docs on using
Vault with Kubernetes available here:

https://developer.hashicorp.com/vault/docs

Your release is named vault. To learn more about the release, try:

# 배포확인
[yu3papa@iworks cicd-labs]$ k -n vault get pods,svc,pvc
NAME READY STATUS RESTARTS AGE
pod/vault-0 0/1 Running 0 4m40s <-- 아직 초기화가 되어있지 않아 Readness Prove가 실패한 상태임
pod/vault-agent-injector-56459c7545-2g599 1/1 Running 0 4m40s

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/vault NodePort 10.96.225.32 <none> 8200:30000/TCP,8201:31361/TCP 4m40s
service/vault-agent-injector-svc ClusterIP 10.96.154.91 <none> 443/TCP 4m40s
service/vault-internal ClusterIP None <none> 8200/TCP,8201/TCP 4m40s
service/vault-ui NodePort 10.96.85.208 <none> 8200:30635/TCP 4m40s

NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
persistentvolumeclaim/data-vault-0 Bound pvc-ad979667-095f-458c-aa06-cae79e5adf03 10Gi RWO standard <unset> 4m40s

vautl-0 POD의 Readness Prove가 실패하여 정상적으로 컨테이너가 실행되지 않고 있음

[yu3papa@iworks cicd-labs]$ k -n vault logs vault-0
...(생략)...
2025-04-12T00:58:50.306Z [INFO]  core: seal configuration missing, not initialized
2025-04-12T00:58:55.307Z [INFO]  core: security barrier not initialized
2025-04-12T00:58:55.307Z [INFO]  core: seal configuration missing, not initialized
2025-04-12T00:58:57.307Z [INFO]  core: security barrier not initialized
2025-04-12T00:58:57.307Z [INFO]  core: seal configuration missing, not initialized
2025-04-12T00:59:02.306Z [INFO]  core: security barrier not initialized
2025-04-12T00:59:02.306Z [INFO]  core: seal configuration missing, not initialized

Vault 초기화 및 잠금해제

# vault 를 디폴트 네임스페이스로 변경
[yu3papa@iworks cicd-labs]$ kubectl config set-context --current --namespace=vault
Context "kind-myk8s" modified.

# Vault Status 명령으로 Sealed 상태확인
[yu3papa@iworks cicd-labs]$ kubectl exec -ti vault-0 -- vault status
Key Value
--- -----
Seal Type shamir
Initialized false
Sealed true
Total Shares 0
Threshold 0
Unseal Progress 0/0
Unseal Nonce n/a
Version 1.19.0
Build Date 2025-03-04T12:36:40Z
Storage Type raft
Removed From Cluster false
HA Enabled true
command terminated with exit code 2

# init-unseal.sh 을 사용하여 Vault Unseal 자동화
[yu3papa@iworks cicd-labs]$ cat <<EOF > init-unseal.sh

#!/bin/bash

# Vault Pod 이름

VAULT_POD="vault-0"

# Vault 명령 실행

VAULT_CMD="kubectl exec -ti \$VAULT_POD -- vault"

# 출력 저장 파일

VAULT_KEYS_FILE="./vault-keys.txt"

UNSEAL_KEY_FILE="./vault-unseal-key.txt"

ROOT_TOKEN_FILE="./vault-root-token.txt"

# Vault 초기화 (Unseal Key 1개만 생성되도록 설정)

\$VAULT_CMD operator init -key-shares=1 -key-threshold=1 | sed \$'s/\\x1b\\[[0-9;]*m//g' | tr -d '\r' > "\$VAULT_KEYS_FILE"

# Unseal Key / Root Token 추출

grep 'Unseal Key 1:' "\$VAULT_KEYS_FILE" | awk -F': ' '{print \$2}' > "\$UNSEAL_KEY_FILE"

grep 'Initial Root Token:' "\$VAULT_KEYS_FILE" | awk -F': ' '{print \$2}' > "\$ROOT_TOKEN_FILE"

# Unseal 수행

UNSEAL_KEY=\$(cat "\$UNSEAL_KEY_FILE")

\$VAULT_CMD operator unseal "\$UNSEAL_KEY"

# 결과 출력

echo "[🔓] Vault Unsealed!"

echo "[🔐] Root Token: \$(cat \$ROOT_TOKEN_FILE)"

EOF

[yu3papa@iworks cicd-labs]$ chmod +x init-unseal.sh
[yu3papa@iworks cicd-labs]$ ./init-unseal.sh
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 1.19.0
Build Date 2025-03-04T12:36:40Z
Storage Type raft
Cluster Name vault-local
Cluster ID 850856c2-c1ca-ac7a-4546-c3a960c7d3b6
Removed From Cluster false
HA Enabled true
HA Cluster n/a
HA Mode standby
Active Node Address <none>
Raft Committed Index 32
Raft Applied Index 32
[🔓] Vault Unsealed!
[🔐] Root Token: hvs.K8yolBx1TyV0SXgbs09kJezX

# vault status 명령을 사용하여 Unseal 되었는지 확인
[yu3papa@iworks cicd-labs]$ kubectl exec -ti vault-0 -- vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 1.19.0
Build Date 2025-03-04T12:36:40Z
Storage Type raft
Cluster Name vault-local
Cluster ID 850856c2-c1ca-ac7a-4546-c3a960c7d3b6
Removed From Cluster false
HA Enabled true
HA Cluster https://vault-0.vault-internal:8201
HA Mode active
Active Since 2025-04-12T01:14:35.953503534Z
Raft Committed Index 37
Raft Applied Index 37

# UI에 접속 --> 30000번 포트에서 NodePort 로 접속가능

Valut CLI 설정

# HashiCorp 공식 저장소를 추가하여 Vault CLI를 패키지로 설치
sudo dnf install -y dnf-plugins-core
sudo rpm --import https://apt.releases.hashicorp.com/gpg
sudo dnf config-manager --add-repo https://rpm.releases.hashicorp.com/RHEL/hashicorp.repo
sudo dnf install vault -y

# VAULT_ADDR 환경변수 설정
[yu3papa@iworks cicd-labs]$ export VAULT_ADDR='http://localhost:30000'
[yu3papa@iworks cicd-labs]$ vault status
Key                     Value
---                     -----
Seal Type               shamir
Initialized             true
Sealed                  false
Total Shares            1
Threshold               1
Version                 1.19.0
Build Date              2025-03-04T12:36:40Z
Storage Type            raft
Cluster Name            vault-local
Cluster ID              850856c2-c1ca-ac7a-4546-c3a960c7d3b6
Removed From Cluster    false
HA Enabled              true
HA Cluster              https://vault-0.vault-internal:8201
HA Mode                 active
Active Since            2025-04-12T01:14:35.953503534Z
Raft Committed Index    39
Raft Applied Index      39

[yu3papa@iworks cicd-labs]$ vault login
Token (will be hidden):
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                hvs.K8yolBx1TyV0SXgbs09kJezX
token_accessor       CfbN5ObmXsplunknR5FtxkvF
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

KV(Key-Value) 시크릿 엔진 활성화 및 샘플 구성

Version1 : KV 버전관리 불가
Version2 : KV 버전관리 가능

현재 Secret Engine은 1개가 존재합니다.

여기에 Vault KV version 2 엔진을 활성화하고 샘플 데이터를 저장하는 실습을 진행하겠습니다.

# KV v2 형태로 엔진 활성화
[yu3papa@iworks cicd-labs]$ vault secrets enable -path=secret kv-v2
Success! Enabled the kv-v2 secrets engine at: secret/

# 샘플 시크릿 저장
[yu3papa@iworks cicd-labs]$ vault kv put secret/sampleapp/config \
 username="demo" \
 password="p@ssw0rd"
======== Secret Path ========
secret/data/sampleapp/config

======= Metadata =======
Key Value
--- -----
created_time 2025-04-12T01:52:18.272876766Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 1

# 입력된 데이터 확인
[yu3papa@iworks cicd-labs]$ vault kv get secret/sampleapp/config
======== Secret Path ========
secret/data/sampleapp/config

======= Metadata =======
Key Value
--- -----
created_time 2025-04-12T01:52:18.272876766Z
custom_metadata <nil>
deletion_time n/a
destroyed false
version 1

====== Data ======
Key Value
--- -----
password p@ssw0rd
username demo

4. Vault Sidecar 연동 (Vault Agent)

학습목표

[Vault Agent](https://developer.hashicorp.com/vault/docs/agent-and-proxy/agent) Sidecar 패턴을 이해하고, 앱(Pod)이 시크릿을 볼륨으로 마운트하여 사용하도록 구성한다.
Vault Agent Injector 구성을 위해 Helm Chart(https://github.com/hashicorp/vault-helm) 에서 활성화 해야하며, https://github.com/hashicorp/vault-k8s을 활용한다.

Vault Agent Injector는 Kubernetes Pod 내부에 Vault Agent를 자동으로 주입해주는 기능입니다. 이를 통해 어플리케이션이 Vault로부터 자동으로 비밀 정보를 받아올 수 있게 됩니다.

Vault - Kubernetes 연동시 동작흐름 (https://medium.com/@muppedaanvesh/a-hand-on-guide-to-vault-in-kubernetes-%EF%B8%8F-1daf73f331bd)

Step1. Vault AppRole 방식 인증 구성

# 1. AppRole 인증 방식 활성화
[yu3papa@iworks cicd-labs]$ vault auth enable approle || echo "AppRole already enabled"
Success! Enabled approle auth method at: approle/

[yu3papa@iworks cicd-labs]$ vault auth list
Path Type Accessor Description Version
---- ---- -------- ----------- -------
approle/ approle auth_approle_1499dea3 n/a n/a
token/ token auth_token_f371b0c1 token based credentials n/a

# 2. 정책 생성
[yu3papa@iworks cicd-labs]$ vault policy write sampleapp-policy - <<EOF
path "secret/data/sampleapp/*" {
 capabilities = ["read"]
}
EOF
Success! Uploaded policy: sampleapp-policy

# 3. AppRole Role 생성
[yu3papa@iworks cicd-labs]$ vault write auth/approle/role/sampleapp-role \
 token_policies="sampleapp-policy" \
 secret_id_ttl="12h" \
 token_ttl="12h" \
 token_max_ttl="14h"
Success! Data written to: auth/approle/role/sampleapp-role

# 4. Role ID 및 Secret ID 추출 및 저장
[yu3papa@iworks cicd-labs]$ ROLE_ID=$(vault read -field=role_id auth/approle/role/sampleapp-role/role-id)
[yu3papa@iworks cicd-labs]$ SECRET_ID=$(vault write -f -field=secret_id auth/approle/role/sampleapp-role/secret-id)
[yu3papa@iworks cicd-labs]$ echo "ROLE_ID: $ROLE_ID"
ROLE_ID: 56f53a99-89c5-e2f7-359b-c5e65b7ea75d
[yu3papa@iworks cicd-labs]$ echo "SECRET_ID: $SECRET_ID"
SECRET_ID: 45ae3a27-ac94-3c66-b52f-6cf12524c9a3

# 5. 파일로 저장
[yu3papa@iworks cicd-labs]$ mkdir -p approle-creds
[yu3papa@iworks cicd-labs]$ echo "$ROLE_ID" > approle-creds/role_id.txt
[yu3papa@iworks cicd-labs]$ echo "$SECRET_ID" > approle-creds/secret_id.txt

# 6. (옵션) Kubernetes Secret으로 저장
[yu3papa@iworks cicd-labs]$ kubectl create secret generic vault-approle -n vault \
 --from-literal=role_id="${ROLE_ID}" \
 --from-literal=secret_id="${SECRET_ID}" \
 --save-config \
 --dry-run=client -o yaml | kubectl apply -f -
secret/vault-approle created

[yu3papa@iworks cicd-labs]$ k get cm
NAME DATA AGE
kube-root-ca.crt 1 90m
vault-config 1 89m

Step2. Vault Agent Sidecar 연동

Vault Agent는 vault-agent-config.hcl 설정을 통해 연결할 Vault의 정보와, Template 구성, 렌더링 주기, 참조할 Vault KV 위치정보 등을 정의합니다.

# 1. Vault Agent 설정 파일 작성 및 생성 (vault-agent-config.hcl) - HCL
[yu3papa@iworks cicd-labs]$ cat <<EOF | kubectl create configmap vault-agent-config -n vault --from-file=agent-config.hcl=/dev/stdin --dry-run=client -o yaml | kubectl apply -f -
vault {
 address = "http://vault.vault.svc:8200"
}

auto_auth {
 method "approle" {
 config = {
 role_id_file_path = "/etc/vault/approle/role_id"
 secret_id_file_path = "/etc/vault/approle/secret_id"
 remove_secret_id_file_after_reading = false
 }
 }

 sink "file" {
 config = {
 path = "/etc/vault-agent-token/token"
 }
 }
}

template_config {
 static_secret_render_interval = "20s"
}

template {
 destination = "/etc/secrets/index.html"
 contents = <<EOH
 <html>
 <body>
 username: {{ with secret "secret/data/sampleapp/config" }}{{ .Data.data.username }}{{ end }}
 password: {{ with secret "secret/data/sampleapp/config" }}{{ .Data.data.password }}{{ end }}
 </body>
 </html>
EOH
}
EOF
configmap/vault-agent-config created

# 2. 샘플 애플리케이션 + Sidecar 배포(수동방식)
# Nginx + Vault Agent 생성
[yu3papa@iworks cicd-labs]$ kubectl apply -n vault -f - <<EOF

apiVersion: apps/v1

kind: Deployment

metadata:

spec:

replicas: 1

selector:

matchLabels:

app: nginx-vault-demo

template:

metadata:

labels:

app: nginx-vault-demo

spec:

containers:

- name: nginx

image: nginx:latest

ports:

- containerPort: 80

volumeMounts:

- name: html-volume

mountPath: /usr/share/nginx/html

- name: vault-agent-sidecar

image: hashicorp/vault:latest

args:

- "agent"

- "-config=/etc/vault/agent-config.hcl"

volumeMounts:

- name: vault-agent-config

mountPath: /etc/vault

- name: vault-approle

mountPath: /etc/vault/approle

- name: vault-token

mountPath: /etc/vault-agent-token

- name: html-volume

mountPath: /etc/secrets

volumes:

- name: vault-agent-config

configMap:

- name: vault-approle

secret:

secretName: vault-approle

- name: vault-token

emptyDir: {}

- name: html-volume

emptyDir: {}

EOF
deployment.apps/nginx-vault-demo created

# 3. SVC 생성
[yu3papa@iworks cicd-labs]$ kubectl apply -f - <<EOF

apiVersion: v1

kind: Service

metadata:

spec:

type: NodePort

selector:

app: nginx-vault-demo

ports:

- protocol: TCP

port: 80

targetPort: 80

nodePort: 30001 # Kind에서 설정한 Port

EOF
service/nginx-service created

# 4. 생성된 컨테이너 확인
# 파드 내에 사이드카 컨테이너 추가되어 2/2 확인
[yu3papa@iworks cicd-labs]$ kubectl get pod -l app=nginx-vault-demo
NAME READY STATUS RESTARTS AGE
nginx-vault-demo-7776649597-wkhts 2/2 Running 0 45s

[yu3papa@iworks cicd-labs]$ kubectl describe pod -l app=nginx-vault-demo
...(생략)...
Containers:
 nginx:
 Container ID: containerd://8a2c1a3976cdffe84c2c74a1de39f7261419a5be6bcfb8c669f1d6ed98fd777f
 Image: nginx:latest
 Image ID: docker.io/library/nginx@sha256:09369da6b10306312cd908661320086bf87fbae1b6b0c49a1f50ba531fef2eab
 Port: 80/TCP
 Host Port: 0/TCP
 State: Running
 Started: Sat, 12 Apr 2025 11:32:16 +0900
 Ready: True
 Restart Count: 0
 Environment: <none>
 Mounts:
 /usr/share/nginx/html from html-volume (rw)
 /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7864f (ro)
 vault-agent-sidecar:
 Container ID: containerd://1f909d82546746d025e11edf0c4916f3a30832fd31efc1d02e4f43e34871cf4b
 Image: hashicorp/vault:latest
 Image ID: docker.io/hashicorp/vault@sha256:ee674e47dcf85849aadf255b5341f76c0e1a474bc5fa9be9cdfff2a2edf9a628
 Port: <none>
 Host Port: <none>
 Args:
 agent
 -config=/etc/vault/agent-config.hcl
 State: Running
 Started: Sat, 12 Apr 2025 11:32:37 +0900
 Ready: True
 Restart Count: 0
 Environment: <none>
 Mounts:
 /etc/secrets from html-volume (rw)
 /etc/vault from vault-agent-config (rw)
 /etc/vault-agent-token from vault-token (rw)
 /etc/vault/approle from vault-approle (rw)
 /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7864f (ro)

# 볼륨 마운트 확인
[yu3papa@iworks cicd-labs]$ kubectl exec -it deploy/nginx-vault-demo -c vault-agent-sidecar -- ls -l /etc/vault-agent-token
total 4
-rw-r----- 1 vault vault 95 Apr 12 02:32 token
[yu3papa@iworks cicd-labs]$ kubectl exec -it deploy/nginx-vault-demo -c vault-agent-sidecar -- cat /etc/vault-agent-token/token ; echo
hvs.CAESIAS4760wQybYD_Nu3EvElQRsz8-znwiWQ-EpSf8ZYGB2Gh4KHGh2cy5YRUpaZnhueFlVMTRkUW52QjFRbnZHTEo
[yu3papa@iworks cicd-labs]$ kubectl exec -it deploy/nginx-vault-demo -c vault-agent-sidecar -- cat /etc/vault/agent-config.hcl
vault {
 address = "http://vault.vault.svc:8200"
}

auto_auth {
 method "approle" {
 config = {
 role_id_file_path = "/etc/vault/approle/role_id"
 secret_id_file_path = "/etc/vault/approle/secret_id"
 remove_secret_id_file_after_reading = false
 }
 }

 sink "file" {
 config = {
 path = "/etc/vault-agent-token/token"
 }
 }
}

template_config {
 static_secret_render_interval = "20s"
}

template {
 destination = "/etc/secrets/index.html"
 contents = <<EOH
 <html>
 <body>
 username: {{ with secret "secret/data/sampleapp/config" }}{{ .Data.data.username }}{{ end }}
 password: {{ with secret "secret/data/sampleapp/config" }}{{ .Data.data.password }}{{ end }}
 </body>
 </html>
EOH
}
[yu3papa@iworks cicd-labs]$ kubectl exec -it deploy/nginx-vault-demo -c vault-agent-sidecar -- ls -al /etc/vault/approle
total 0
drwxrwxrwt 3 root root 120 Apr 12 02:32 .
drwxrwxrwx 4 root root 99 Apr 12 02:32 ..
drwxr-xr-x 2 root root 80 Apr 12 02:32 ..2025_04_12_02_32_04.1424053900
lrwxrwxrwx 1 root root 32 Apr 12 02:32 ..data -> ..2025_04_12_02_32_04.1424053900
lrwxrwxrwx 1 root root 14 Apr 12 02:32 role_id -> ..data/role_id
lrwxrwxrwx 1 root root 16 Apr 12 02:32 secret_id -> ..data/secret_id
[yu3papa@iworks cicd-labs]$ kubectl exec -it deploy/nginx-vault-demo -c vault-agent-sidecar -- cat /etc/secrets/index.html
 <html>
 <body>
 username: demo
 password: p@ssw0rd
 </body>
 </html>

# mutating admission
[yu3papa@iworks cicd-labs]$ kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io
NAME WEBHOOKS AGE
vault-agent-injector-cfg 1 107m

# 5. 실제 배포된 화면 확인

# 6. KV 값 변경 후 확인

5. Jenkins + Vault (AppRole) - CI

젠킨스 CI/CD 도구에서 민감한 정보를 Valut에서 가져오는 실습을 진행해 보겠습니다.

학습목표

Vault KV Store에 저장한 username, password을 Jenkins을 활용해서 획득하는 방안
CI 파이프라인에서 정적(Static) 시크릿을 외부에 저장하고 관리할 경우 사용할 수 있습니다.

Vault - Jenkins Plugin with AppRole 인증방식 워크플로우

1. 젠킨스 워커가 Vault에 인증
2. Vault는 토큰을 반환
3. 워커는 이 토큰을 사용해 작업에 해당하는 역할의 Wrapped SecretID를 요청
4. Vault는 Wrapped SecretID를 반환
5. 워커는 작업 러너를 생성하고, Wrapped SecretID를 변수로 전달
6. 러너 컨테이너는 Wrapped SecretID의 unwrap을 요청
7. Vault는 SecretID를 반환
8. 러너는 RoleID와 SecretID를 사용해 Vault에 인증
9. Vault는 필요한 시크릿 정보를 읽을 수 있는 정책이 포함된 토큰을 반환
10. 러너는 이 토큰을 사용해 Vault에서 시크릿을 가져옴

Jenkins에서 Vault 설정 및 Credentials 추가

Jenkins Pipeline Job 생성

Jenkins UI → New Item → Pipeline 선택

Pipeline 코드 작성

pipeline {

agent any

environment {

VAULT_ADDR = 'http://192.168.10.4:30000' // 실제 Vault 주소로 변경!!!

}

stages {

stage('Read Vault Secret') {

steps {

withVault([

vaultSecrets: [

[

path: 'secret/sampleapp/config',

engineVersion: 2,

secretValues: [

[envVar: 'USERNAME', vaultKey: 'username'],

[envVar: 'PASSWORD', vaultKey: 'password']

]

configuration: [

vaultUrl: "${VAULT_ADDR}",

vaultCredentialId: 'vault-approle-creds'

]

]) {

sh '''

echo "Username from Vault: $USERNAME"

echo "Password from Vault: $PASSWORD"

'''

script {

echo "Username (env): ${env.USERNAME}"

echo "Password (env): ${env.PASSWORD}"

}

Build를 수행하고 실행결과를 확인

6. ArgoCD + Vault Plugin (Kubernetes Auth/AppRole) - CD

ArgoCD Vault Plugin 소개

Argo CD에는 다양한 시크릿 관리 도구(HashiCorp Vault, IBM Cloud Secrets Manager, AWS Secrets Manager 등)플러그인을 통해 Kubernetes 리소스에 주입할 수 있도록 지원합니다.
플러그인을 통해 Operator 또는 CRD(Custom Resource Definition)에 의존하지 않고 GitOps와 Argo CD로 시크릿 관리 문제를 해결할 수 있습니다.
특히 Secret 뿐만 아니라, deployment, configMap 또는 기타 Kubernetes 리소스에도 사용할 수 있습니다.

Step 1. ArgoCD Vault Plugin을 위한 Credentials 활성화 - AppRole 인증

[yu3papa@iworks cicd-labs]$ kubectl apply -f - <<EOF

kind: Secret

apiVersion: v1

metadata:

namespace: argocd

type: Opaque

stringData:

VAULT_ADDR: "http://vault.vault:8200"

AVP_TYPE: "vault"

AVP_AUTH_TYPE: "approle"

AVP_ROLE_ID: 56f53a99-89c5-e2f7-359b-c5e65b7ea75d #Role_ID

AVP_SECRET_ID: 45ae3a27-ac94-3c66-b52f-6cf12524c9a3 #Secret_ID

EOF
secret/argocd-vault-plugin-credentials created

[yu3papa@iworks cicd-labs]$ k -n argocd get secret
NAME                              TYPE                 DATA   AGE
argocd-initial-admin-secret       Opaque               1      4h1m
argocd-notifications-secret       Opaque               0      4h1m
argocd-redis                      Opaque               1      4h1m
argocd-secret                     Opaque               3      4h1m
argocd-vault-plugin-credentials   Opaque               5      98s
sh.helhttp://m.release.v1.argocd.v1 helm.sh/release.v1   1      4h1m

Step 2. ArgoCD Vault Plugin 설치

ArgoCD Vault Plugin 설치 방법은 2가지가 있으며 현재는 Installation via a sidecar container 방식을 사용하는 것을 권장합니다.

→ 이번 스터디에서는 편의상 Helm으로 배포한 ArgoCD에 Kustomize을 활용해 기존 YAML에 대한 Patch을 적용합니다.

[yu3papa@iworks cicd-labs]$ git clone https://github.com/hyungwook0221/argocd-vault-plugin.git
Cloning into 'argocd-vault-plugin'...
remote: Enumerating objects: 2610, done.
remote: Counting objects: 100% (258/258), done.
remote: Compressing objects: 100% (129/129), done.
remote: Total 2610 (delta 192), reused 129 (delta 129), pack-reused 2352 (from 2)
Receiving objects: 100% (2610/2610), 1.72 MiB | 8.76 MiB/s, done.
Resolving deltas: 100% (1535/1535), done.
[yu3papa@iworks cicd-labs]$ cd argocd-vault-plugin/manifests/cmp-sidecar

# argocd 네임스페이스를 디폴트로 설정
[yu3papa@iworks cmp-sidecar]$ kubectl config set-context --current --namespace=argocd
Context "kind-myk8s" modified.

# 생성될 메니페스트 파일에 대한 확인
[yu3papa@iworks cmp-sidecar]$ kubectl kustomize .

apiVersion: v1

data:

avp-helm.yaml: "---\napiVersion: argoproj.io/v1alpha1\nkind: ConfigManagementPlugin\nmetadata:\n

\ name: argocd-vault-plugin-helm\nspec:\n allowConcurrency: true\n\n # Note:

this command is run _before_ any Helm templating is done, therefore the logic

is to check\n # if this looks like a Helm chart\n discover:\n find:\n command:\n

\ - sh\n - \"-c\"\n - \"find . -name 'Chart.yaml' && find

. -name 'values.yaml'\"\n generate:\n # **IMPORTANT**: passing `${ARGOCD_ENV_HELM_ARGS}`

effectively allows users to run arbitrary code in the Argo CD \n # repo-server

(or, if using a sidecar, in the plugin sidecar). Only use this when the users

are completely trusted. If\n # possible, determine which Helm arguments are

needed by your users and explicitly pass only those arguments.\n command:\n

\ - sh\n - \"-c\"\n - |\n helm template $ARGOCD_APP_NAME

-n $ARGOCD_APP_NAMESPACE ${ARGOCD_ENV_HELM_ARGS} . |\n argocd-vault-plugin

generate -\n lockRepo: false\n"

avp-kustomize.yaml: |

---

apiVersion: argoproj.io/v1alpha1

kind: ConfigManagementPlugin

metadata:

spec:

allowConcurrency: true

# Note: this command is run _before_ anything is done, therefore the logic is to check

# if this looks like a Kustomize bundle

discover:

find:

command:

- find

- "."

- -name

- kustomization.yaml

generate:

command:

- sh

- "-c"

- "kustomize build . | argocd-vault-plugin generate -"

lockRepo: false

avp.yaml: |

apiVersion: argoproj.io/v1alpha1

kind: ConfigManagementPlugin

metadata:

spec:

allowConcurrency: true

discover:

find:

command:

- sh

- "-c"

- "find . -name '*.yaml' | xargs -I {} grep \"<path\\|avp\\.kubernetes\\.io\" {} | grep ."

generate:

command:

- argocd-vault-plugin

- generate

- "."

lockRepo: false

kind: ConfigMap

metadata:

---

apiVersion: apps/v1

kind: Deployment

metadata:

spec:

selector:

matchLabels:

app.kubernetes.io/name: argocd-repo-server

template:

metadata:

labels:

app.kubernetes.io/name: argocd-repo-server

spec:

automountServiceAccountToken: true

containers:

- command:

- /var/run/argocd/argocd-cmp-server

envFrom:

- secretRef:

image: quay.io/argoproj/argocd:v2.7.9

securityContext:

runAsNonRoot: true

runAsUser: 999

volumeMounts:

- mountPath: /var/run/argocd

- mountPath: /home/argocd/cmp-server/plugins

- mountPath: /tmp

- mountPath: /home/argocd/cmp-server/config/plugin.yaml

subPath: avp-helm.yaml

- mountPath: /usr/local/bin/argocd-vault-plugin

subPath: argocd-vault-plugin

- command:

- /var/run/argocd/argocd-cmp-server

image: quay.io/argoproj/argocd:v2.7.9

securityContext:

runAsNonRoot: true

runAsUser: 999

volumeMounts:

- mountPath: /var/run/argocd

- mountPath: /home/argocd/cmp-server/plugins

- mountPath: /tmp

- mountPath: /home/argocd/cmp-server/config/plugin.yaml

subPath: avp-kustomize.yaml

- mountPath: /usr/local/bin/argocd-vault-plugin

subPath: argocd-vault-plugin

- command:

- /var/run/argocd/argocd-cmp-server

image: quay.io/argoproj/argocd:v2.7.9

securityContext:

runAsNonRoot: true

runAsUser: 999

volumeMounts:

- mountPath: /var/run/argocd

- mountPath: /home/argocd/cmp-server/plugins

- mountPath: /tmp

- mountPath: /home/argocd/cmp-server/config/plugin.yaml

subPath: avp.yaml

- mountPath: /usr/local/bin/argocd-vault-plugin

subPath: argocd-vault-plugin

- image: quay.io/argoproj/argocd:v2.8.13

initContainers:

- args:

- curl -L https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v$(AVP_VERSION)/argocd-vault-plugin_$(AVP_VERSION)_linux_amd64

-o argocd-vault-plugin && chmod +x argocd-vault-plugin && mv argocd-vault-plugin

/custom-tools/

command:

- sh

- -c

env:

- name: AVP_VERSION

value: 1.18.0

image: registry.access.redhat.com/ubi8

volumeMounts:

- mountPath: /custom-tools

volumes:

- configMap:

- emptyDir: {}

# -k 옵션으로 kusomize 실행
[yu3papa@iworks cmp-sidecar]$ kubectl apply -n argocd -k .
configmap/cmp-plugin created
Warning: resource deployments/argocd-repo-server is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
deployment.apps/argocd-repo-server configured

Step 3. 샘플 Application 배포하여 Vault와 동기화

Step 3-1) Application.yaml 작성

[yu3papa@iworks cmp-sidecar]$ kubectl apply -n argocd -f - <<EOF

apiVersion: argoproj.io/v1alpha1

kind: Application

metadata:

namespace: argocd

finalizers:

- resources-finalizer.argocd.argoproj.io

spec:

destination:

namespace: argocd

server: https://kubernetes.default.svc

project: default

source:

path: infra/helm

repoURL: https://github.com/hyungwook0221/spring-boot-debug-app

targetRevision: main

plugin:

env:

- name: HELM_ARGS

value: -f new-values.yaml

syncPolicy:

automated:

prune: true

selfHeal: true

EOF
Warning: metadata.finalizers: "resources-finalizer.argocd.argoproj.io": prefer a domain-qualified finalizer name including a path (/) to avoid accidental conflicts with other finalizer writers
application.argoproj.io/demo created

Step 3-2) Application 배포시 참조하는 new-values.yaml 확인

serviceAccount:

create: true

image:

repository: luafanti/spring-boot-debug-app

tag: main

pullPolicy: IfNotPresent

replicaCount: 1

resources:

memoryRequest: 256Mi

memoryLimit: 512Mi

cpuRequest: 500m

cpuLimit: 1

probes:

liveness:

initialDelaySeconds: 15

path: /actuator/health/liveness

failureThreshold: 3

successThreshold: 1

timeoutSeconds: 3

periodSeconds: 5

readiness:

initialDelaySeconds: 15

path: /actuator/health/readiness

failureThreshold: 3

successThreshold: 1

timeoutSeconds: 3

periodSeconds: 5

ports:

http:

value: 8080

management:

value: 8081

envs:

- name: VAULT_SECRET_USER

value: <path:secret/data/sampleapp/config#username>

- name: VAULT_SECRET_PASSWORD

value: <path:secret/data/sampleapp/config#password>

log:

level:

spring: "info"

service: "info"

Step 3-3) 실제 배포시 적용된 화면 확인

ArgoCD Vault Plugin 적용된 화면

Application 배포화면

Deployment에 적용된 env 값 확인 :

다음 실습을 위해 ArgoCD App 삭제

[yu3papa@iworks cmp-sidecar]$ kubectl delete applications demo
application.argoproj.io "demo" deleted

7. 실습 리소스 정리

kind로 작성한 K8S 클러스터 삭제

[yu3papa@iworks ~]$ kind delete cluster --name myk8s
Deleting cluster "myk8s" ...
Deleted nodes: ["myk8s-worker" "myk8s-worker2" "myk8s-control-plane"]

Docker Compose로 실행한 Jenkins 삭제

[yu3papa@iworks ~]$ cd ~/cicd-labs/

[yu3papa@iworks cicd-labs]$ docker compose down
[+] Running 2/2
✔ Container jenkins Removed 0.2s
✔ Network cicd-labs_cicd-network Removed

'Kubernetes' 카테고리의 다른 글

Cilium Study [1기] (1주차) - eBPF (2)	2025.07.19
AEWS [3기] 12주차 - Amazon VPC Lattice for Amazon EKS (0)	2025.04.26
AEWS [3기] 9주차 - EKS Upgrade (1)	2025.04.01
AEWS [3기] 8주차 - K8S CI/CD 와 kustomize (1)	2025.03.29
AEWS [3기] 7주차 - EKS Mode - Fargate (0)	2025.03.22

현재글AEWS [3기] 10주차 - K8S 시크릿 관리

yu3papa 님의 블로그

yu3papa 님의 블로그 입니다.

Today :
Yesterday :

yu3papa 님의 블로그