쿠버네티스는 1년에 3개의 Minor 버전을 출시하고, 가장 최근 3개의 Minor 버전에 대해서 Patch를 제공하고 있습니다.
| The Kubernetes project maintains release branches for the most recent three minor releases (1.32, 1.31, 1.30). Kubernetes 1.19 and newer receive approximately 1 year of patch support. Kubernetes 1.18 and older received approximately 9 months of patch support. |
그래서 약 1년 정도 Patch를 지원받고, 민감한 보안 이슈의 경우 예외적으로 18개월 정도까지 Patch를 지원하기도 합니다.
Version Skew Policy
version skew 는 쿠버네티의 구성요소들간의 Version 편차를 허용하는 정책으로 예를 들면 HA로 구성된 Control-Plane의 kube-apiserver의 경우 1대의 Control-plane은 1.32 버전으로 업그레이드 해도, 다른 Control-plane의 kube-apiserver 구성요소는 1.31도 허용하는 정책입니다.
- kube-apiserver : HA apiserver 경우 newest 와 oldest 가 1개 마이너 버전 가능
- newest `kube-apiserver` is at 1.32
- other `kube-apiserver` instances are supported at 1.32 and 1.31
- kubelet/kube-proxy : kubelet 는 apiserver 보다 높을수 없음. 추가로 apiserver 보다 3개 older 가능
- `kube-apiserver` is at 1.32
- `kubelet` is supported at 1.32, 1.31, 1.30, and 1.29
- 만약 apiserver 가 HA로 1.32, 1.31 있다면 kubelet 는 1.32는 안되고, 1.31, 1.30, 1.29 가능
- kubectl : one minor version (older or newer) of `kube-apiserver`
- `kube-apiserver` is at 1.32
- `kubectl` is supported at 1.33, 1.32, and 1.31
쿠버네티스 버전 업그레이드는 위의 Version Skey Policy를 확인하고 업그레이드 절차를 철저히 준비하는게 가장 중요합니다. 그래서 온프레미스 환경에서 업그레이드 계획을 준비하는 과정으로 진행해 보겠습니다.
가상의 온프레미스 환경 K8S 에서 업그레이드 계획 짜기
환경 소개
- K8S 1.28 버전으로 클러스터로 구성되어 있고, 3대의 Control-Plane, 3대의 Worker-node, HW LB로 운영중인 환경입니다.
각 구성요소의 상세버전은 아래와 같습니다.
- Ubuntu 22.04 (kernel 5.15)
- Containerd 1.7.z
- K8S Version 1.28.2 - kubeadm, kubelet
- CNI : Cilium 1.4.z
- CSI : OpenEBS 3.y.z
위와 같은 K8S 1.28 버전에서 1.32 버전을 목표로 업그레이드 계획을 아래의 절차대로 진행해 보겠습니다.
- 버전 호환성 검토
- K8S(kubelet, apiserver..) 1.32 요구 커널 버전 확인 : 예) user namespace 사용 시 커널 6.5 이상 필요 - Docs
- net.ipv4.ip_local_reserved_ports (since Kubernetes 1.27, needs kernel 3.16+);
net.ipv4.tcp_keepalive_time (since Kubernetes 1.29, needs kernel 4.5+);
net.ipv4.tcp_fin_timeout (since Kubernetes 1.29, needs kernel 4.6+);
net.ipv4.tcp_keepalive_intvl (since Kubernetes 1.29, needs kernel 4.5+);
net.ipv4.tcp_keepalive_probes (since Kubernetes 1.29, needs kernel 4.5+);
net.ipv4.tcp_syncookies (namespaced since kernel 4.6+).
net.ipv4.tcp_rmem (since Kubernetes 1.32, needs kernel 4.15+).
net.ipv4.tcp_wmem (since Kubernetes 1.32, needs kernel 4.15+).
net.ipv4.vs.conn_reuse_mode (used in ipvs proxy mode, needs kernel 4.1+);
- net.ipv4.ip_local_reserved_ports (since Kubernetes 1.27, needs kernel 3.16+);
- containerd 버전 조사 : 예) user namespace 에 대한 CRI 지원 시 containerd v2.0 및 runc v1.2 이상 필요 - Docs
- K8S 호환 버전 확인 - Docs
-
Kubernetes Version containerd Version CRI Version 1.29 1.7.11+, 1.6.27+ v1 1.3 2.0.0+, 1.7.13+, 1.6.28+ v1 1.31 2.0.0+, 1.7.20+, 1.6.34+ v1 1.32 2.0.1+, 1.7.24+, 1.6.36+ v1
-
- K8S 호환 버전 확인 - Docs
- CNI(Cilium) 요구 커널 버전 확인 : 예) BPF-based host routing 기능 필요 시 커널 5.10 이상 필요 - Docs
- CNI 이 지원하는 K8S 버전 확인 - Docs
- CNI 경우 보통 DaemonSet으로 모둔 노드에서 설치되므로 업그레이드시 네트워킹이 단절될 수 있습니다.
- CSI : 기존에 사용하는 CSI 맞게 K8S 버전별 요구사항 확인
- 애플리케이션 요구사항 검토…
- K8S(kubelet, apiserver..) 1.32 요구 커널 버전 확인 : 예) user namespace 사용 시 커널 6.5 이상 필요 - Docs
- 업그레이드 방법 결정 : in-place vs blue-green
- Dev - Stg - Prd 별 각기 다른 방법 vs 모두 동일 방법
- in-place 전략
- 기존 클러스터 구성요소를 교체하는 전략
- blue-green 전략
- K8S 클러스터를 목표버전으로 업그레이드하기 위해 목표 버전의 클러스터를 구성하고 교체하는 전략
- 결정된 방법으로 업그레이드 계획 수립 : 예시) in-place 전략으로 결정했다고 하고 진행
- 사전 준비
- (옵션) 각 작업 별 상세 명령 실행 및 스크립트 작성, 작업 중단 실패 시 롤백 명령/스크립트 작성
- 모니터링 설정
- (1) ETCD 백업
- (2) CNI(cilium) 업그레이드
- Control-Plane 노드 순차 업그레이드 : 1.28 → 1.29 → 1.30
- Control-Plane과 Data-plane간에 3개 이상의 버전차이는 허용하지 않기 때문에 1.28 버전에서 1.32 버전으로 한번에 업그레이드 할 수는 없습니다.
- 노드 1대씩 작업 진행 1.28 → 1.29
- (3) Ubuntu OS(kernel) 업그레이드 → 재부팅
- (4) containerd 업그레이드 → containerd 서비스 재시작
- (5) kubeadm 업그레이드 → kubelet/kubectl 업그레이드 설치 → kubelet 재시작
- 노드 1대씩 작업 진행 1.29 → 1.30
- (5) kubeadm 업그레이드 → kubelet/kubectl 업그레이드 설치 → kubelet 재시작
- 작업 완료 후 CP 노드 상태 확인
- Data-Plane 노드 순차 업그레이드 : 1.28 → 1.29 → 1.30
- 노드 1대씩 작업 진행 1.28 → 1.29
- (6) 작업 노드 drain 설정 후 파드 Evicted 확인, 서비스 중단 여부 모니터링 확인
- (7) Ubuntu OS(kernel) 업그레이드 → 재부팅
- (8) containerd 업그레이드 → containerd 서비스 재시작
- (9) kubeadm 업그레이드 → kubelet 업그레이드 설치 → kubelet 재시작
- 노드 1대씩 작업 진행 1.29 → 1.30
- (9) kubeadm 업그레이드 → kubelet 업그레이드 설치 → kubelet 재시작
- 작업 완료 후 DP 노드 상태 확인 ⇒ (10) 작업 노드 uncordon 설정 후 다시 상태 확인
- 노드 1대씩 작업 진행 1.28 → 1.29
- K8S 관련 전체적인 동작 1차 점검
- 애플리케이션 파드 동작 확인 등
- CP 노드 순차 업그레이드 : 1.30 → 1.31 → 1.32
- 노드 1대씩 작업 진행 x 버전별 반복
- kubeadm 업그레이드 → kubelet/kubectl 업그레이드 설치 → kubelet 재시작
- 작업 완료 후 CP 노드 상태 확인
- 노드 1대씩 작업 진행 x 버전별 반복
- DP 노드 순차 업그레이드 : 1.30 → 1.31 → 1.32
- 노드 1대씩 작업 진행 x 버전별 반복
- 작업 노드 drain 설정 후 파드 Evicted 확인, 서비스 중단 여부 모니터링 확인
- kubeadm 업그레이드 → kubelet 업그레이드 설치 → kubelet 재시작
- 작업 완료 후 DP 노드 상태 확인 ⇒ 작업 노드 uncordon 설정 후 다시 상태 확인
- 노드 1대씩 작업 진행 x 버전별 반복
- K8S 관련 전체적인 동작 2차 점검
- 애플리케이션 파드 동작 확인 등
온프레미스 환경에서 K8S 버전 업그레이드는 위의 ①번부터 ⑩번까지의 과정으로 세심하게 확인하면서 업그레이드를 진행하여야 하는데, EKS는 매니지드 서비스이기 때문여 업그레이드의 많은 부분을 AWS가 해주기 때문에 좀 더 간단하게 업그레이드를 할 수 가 있습니다.
Amazon EKS Upgrades: Strategies and Best Practices
이번 실습은 "Amazon EKS Upgrades: Strategies and Best Practices" 워크샵을 이용하여 진행합니다.
https://catalog.us-east-1.prod.workshops.aws/workshops/693bdee4-bc31-41d5-841f-54e3e54f8f4a/en-US
Amazon EKS 클러스터 업그레이드 워크숍의 목적은 고객에게 Amazon EKS 클러스터 업그레이드를 계획하고 실행할 수 있는 모범 사례를 제공하는 일련의 실험실을 소개하는 것입니다. 우리는 In-Place, Blue/Green 등 다양한 클러스터 업그레이드 전략을 탐구하고 각 전략의 실행 세부 사항을 자세히 살펴볼 것입니다. 이 워크숍에서는 EKS 테라폼 청사진, ArgoCD 등을 활용하여 전체 EKS 클러스터 업그레이드 프로세스를 안내합니다.
Workshop 실습환경
미국(오레곤) 리전을 사용하여 EKS는 1.25 버전이며, 노드 정보입니다.
Add-On 구성은 아래와 같습니다.
실습하는 쉘의 환경을 확인합니다.
- IDE 정보 확인 : 버전(1.25.16), 노드(AL2, 커널 5.10.234, containerd 1.7.25
| ec2-user:~/environment:$ whoami ec2-user ec2-user:~/environment:$ pwd /home/ec2-user/environment ec2-user:~/environment:$ export declare -x ASSETS_BUCKET="ws-event-f4fef182-00c-us-west-2/d2117abb-06fa-4e89-8a6b-8e2b5d6fc697/assets/" declare -x AWS_DEFAULT_REGION="us-west-2" declare -x AWS_PAGER="" declare -x AWS_REGION="us-west-2" declare -x BROWSER="/usr/lib/code-server/lib/vscode/bin/helpers/browser.sh" declare -x CLUSTER_NAME="eksworkshop-eksctl" declare -x COLORTERM="truecolor" declare -x EC2_PRIVATE_IP="192.168.0.227" declare -x EKS_CLUSTER_NAME="eksworkshop-eksctl" declare -x GIT_ASKPASS="/usr/lib/code-server/lib/vscode/extensions/git/dist/askpass.sh" declare -x HISTCONTROL="ignoredups" declare -x HISTSIZE="1000" declare -x HOME="/home/ec2-user" declare -x HOSTNAME="ip-192-168-0-227.us-west-2.compute.internal" declare -x IDE_DOMAIN="d4bryeda0kxpr.cloudfront.net" declare -x IDE_PASSWORD="yaAEowcTYoFvg0wBBTA8ITTDr7OOF8kA" declare -x IDE_URL="https://d4bryeda0kxpr.cloudfront.net" declare -x INSTANCE_IAM_ROLE_ARN="arn:aws:iam::678866666676:role/workshop-stack-IdeIdeRoleD654ADD4-aVMldSuUNfaa" declare -x INSTANCE_IAM_ROLE_NAME="workshop-stack-IdeIdeRoleD654ADD4-aVMldSuUNfaa" declare -x INVOCATION_ID="1be1447601e849cc9b12522faa8b1a40" declare -x JOURNAL_STREAM="8:41801" declare -x LANG="C.UTF-8" declare -x LESSOPEN="||/usr/bin/lesspipe.sh %s" declare -x LOGNAME="ec2-user" declare -x LS_COLORS="rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.webp=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.m4a=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.oga=01;36:*.opus=01;36:*.spx=01;36:*.xspf=01;36:" declare -x MAIL="/var/spool/mail/ec2-user" declare -x NODE_EXEC_PATH="/usr/lib/code-server/lib/node" declare -x OLDPWD declare -x PATH="/usr/lib/code-server/lib/vscode/bin/remote-cli:/home/ec2-user/.local/bin:/home/ec2-user/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin" declare -x PROMPT_COMMAND="__vsc_prompt_cmd_original" declare -x PS1=$'\\[\E]633;A\a\\]\\u:\\w:$ \\[\E]633;B\a\\]' declare -x PWD="/home/ec2-user/environment" declare -x REGION="us-west-2" declare -x SHELL="/bin/bash" declare -x SHLVL="1" declare -x SYSTEMD_COLORS="false" declare -x SYSTEMD_EXEC_PID="21119" declare -x S_COLORS="auto" declare -x TERM="xterm-256color" declare -x TERM_PROGRAM="vscode" declare -x TERM_PROGRAM_VERSION="1.91.1" declare -x TF_STATE_S3_BUCKET="workshop-stack-tfstatebackendbucketf0fc9a9d-qll9it5dhvaw" declare -x TF_VAR_aws_region="us-west-2" declare -x TF_VAR_eks_cluster_id="eksworkshop-eksctl" declare -x USER="ec2-user" declare -x VSCODE_GIT_ASKPASS_EXTRA_ARGS="" declare -x VSCODE_GIT_ASKPASS_MAIN="/usr/lib/code-server/lib/vscode/extensions/git/dist/askpass-main.js" declare -x VSCODE_GIT_ASKPASS_NODE="/usr/lib/code-server/lib/node" declare -x VSCODE_GIT_IPC_HANDLE="/tmp/vscode-git-d98a512aae.sock" declare -x VSCODE_IPC_HOOK_CLI="/tmp/vscode-ipc-c20938da-33a2-4191-aaf2-2be2d2ab9c17.sock" declare -x VSCODE_PROXY_URI="https://d4bryeda0kxpr.cloudfront.net/proxy/{{port}}/" declare -x which_declare="declare -f" # 실습 EKS 환경은 테라폼으로 배포되어 있고, tfstate 파일이 S3에 저장되어 있습니다. # s3 버킷 확인 ec2-user:~/environment:$ aws s3 ls 2025-03-30 05:34:11 workshop-stack-tfstatebackendbucketf0fc9a9d-qll9it5dhvaw ec2-user:~/environment:$ aws s3 ls s3://workshop-stack-tfstatebackendbucketf0fc9a9d-qll9it5dhvaw 2025-03-30 05:55:13 933780 terraform.tfstate # 환경변수(테라폼 포함) 및 단축키 alias 등 확인 ec2-user:~/environment:$ cat ~/.bashrc ...(생략)... export EKS_CLUSTER_NAME=eksworkshop-eksctl export CLUSTER_NAME=eksworkshop-eksctl export AWS_DEFAULT_REGION=us-west-2 export REGION=us-west-2 export AWS_REGION=us-west-2 export TF_VAR_eks_cluster_id=eksworkshop-eksctl export TF_VAR_aws_region=us-west-2 export ASSETS_BUCKET=ws-event-f4fef182-00c-us-west-2/d2117abb-06fa-4e89-8a6b-8e2b5d6fc697/assets/ export TF_STATE_S3_BUCKET=workshop-stack-tfstatebackendbucketf0fc9a9d-qll9it5dhvaw alias k=kubectl alias kgn="kubectl get nodes -o wide" alias kgs="kubectl get svc -o wide" alias kgd="kubectl get deploy -o wide" alias kgsa="kubectl get svc -A -o wide" alias kgda="kubectl get deploy -A -o wide" alias kgp="kubectl get pods -o wide" alias kgpa="kubectl get pods -A -o wide" aws eks update-kubeconfig --name eksworkshop-eksctl # EKS 자격증명이 되어 있음 export AWS_PAGER="" # eks 플랫폼 버전 확인 ec2-user:~/environment:$ aws eks describe-cluster --name $EKS_CLUSTER_NAME | jq {
"cluster": {
"name": "eksworkshop-eksctl",
"arn": "arn:aws:eks:us-west-2:678866666676:cluster/eksworkshop-eksctl",
"createdAt": "2025-03-30T05:38:02.500000+00:00",
"version": "1.25",
"roleArn": "arn:aws:iam::678866666676:role/eksworkshop-eksctl-cluster-20250330053739974300000004",
"resourcesVpcConfig": {
"subnetIds": [
"subnet-042922ea084483364",
"subnet-082eeb7c16623ae93",
"subnet-040fd59c3e2770cb9"
],
"securityGroupIds": [
"sg-034e81419ef24390b"
],
"clusterSecurityGroupId": "sg-0291a3f8e88f9cdae",
"vpcId": "vpc-0b70a20653beed1df",
"endpointPublicAccess": true,
"endpointPrivateAccess": true,
"publicAccessCidrs": [
"0.0.0.0/0"
]
},
"kubernetesNetworkConfig": {
"serviceIpv4Cidr": "172.20.0.0/16",
"ipFamily": "ipv4",
"elasticLoadBalancing": {
"enabled": false
}
},
"logging": {
"clusterLogging": [
{
"types": [
"api",
"audit",
"authenticator"
],
"enabled": true
},
{
"types": [
"controllerManager",
"scheduler"
],
"enabled": false
}
]
},
"identity": {
"oidc": {
}
},
"status": "ACTIVE",
"certificateAuthority": {
"data": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJTHVVZ0R5NGkrM1F3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TlRBek16QXdOVE00TURSYUZ3MHpOVEF6TWpnd05UUXpNRFJhTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUURpQkQwd0JVaDUyRWxlek0rZUh3YVZNVC9sa0xhTXgwN2tneEhTN0cwMkt2MzNHT2p6MDdSSDlzeHcKbk5CZ2ZEUjJwVis1TERzTXgyVktWcm1EQUNHdUs4eC84L2dmd2dIQ25OV0hETklNSC9EM0hLTnlZUEdrVjhwZQo5Y2ZrL3hUYlRHQnhoNTNyVXREcGEyU2lEZ0h3YzRzWUZ0aVJjUXFtOUt0aStJY2JSN2ZNcWdlNDBQRHhHeW1ZCkU5d1V6OUtFeWpjRHlCYloxK1Fna1hoQzBoeTNNb0NWSVlMQmRJOUJ1b1N1NXJtcmJXdU5sQkdxczdPcnpyd2kKdHZNSk9ad3psaWM2QWQ4QXRFTE40OWxEVXZtaVJvY05ROFcyeW10VnVPQXRBRUlLenJVakwxQlc4Rlc5Q1d1NQpiT01ZanlSVlY4TmJsM1JWdnYvS2YvVVNZRW52QWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJTanpveFZoOUhaTW5HK3VXTDlkRWFzakhVOUlEQVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQmY5ZWVTT2x5dQpNYVEvY3l0V1htVXpHWHVUbkZQZG11NGVidTA1VGhQN3YwNnNBNkh0SWZuQ3FVU0NrMHZZUlpRL3M0UW0ySVIxCkFYMS9PWE9LQ0IxRXNDaldDNXRDblpRMHluZkhlMlNWZjJHdmhmYlRCQmdkc2haZDZjaDBtZGl1TzJVUzN3R1gKSDBzZ1g2bm1NY2g0TEc1RnVlNHVldEdVbGxGVEJXeWNBVnZJMEcyMGx4dnZGUUhCbVlzUmNkRWd3TlFOSEdTdwp2SXJsbStjOGtQK0VIcmlMcTh1eU15THhiUTA1Y0RyUlUzY0FpWDQzL0hibnM2WWlPcENZcTRGZlRNZldJOUJFCmpqYjJLaHRuMHJQTmlMNlplMEQyWW9ZWHY5YzNDV24vTVhhRU5weU1zek1CYUNUSDJCMnVYNVNaT3FSbGlUNlUKS2E2MzExNHpkUkhECi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"
},
"platformVersion": "eks.44",
"tags": {
"karpenter.sh/discovery": "eksworkshop-eksctl",
"terraform-aws-modules": "eks",
"Blueprint": "eksworkshop-eksctl",
"GithubRepo": "github.com/aws-ia/terraform-aws-eks-blueprints"
},
"encryptionConfig": [
{
"resources": [
"secrets"
],
"provider": {
"keyArn": "arn:aws:kms:us-west-2:678866666676:key/215f54d9-8a7f-425b-9da4-387f409cc1de"
}
}
],
"health": {
"issues": []
},
"accessConfig": {
"authenticationMode": "API_AND_CONFIG_MAP"
},
"upgradePolicy": {
"supportType": "EXTENDED"
}
}
}
# EKS 관리콘솔에서 확인해보면  |
eks 클러스터 정보를 확인해 보겠습니다.
| ec2-user:~/environment:$ eksctl get cluster NAME REGION EKSCTL CREATED eksworkshop-eksctl us-west-2 False # 노드그룹정보를 확인하면 2개의 managed nodegroup 이 2개임 --> 다양한 업그레이드 실습을 위해서임 ec2-user:~/environment:$ eksctl get nodegroup --cluster $CLUSTER_NAME CLUSTER NODEGROUP STATUS CREATED MIN SIZE MAX SIZE DESIRED CAPACITY INSTANCE TYPE IMAGE ID ASG NAME TYPE eksworkshop-eksctl blue-mng-2025033005474923890000002b ACTIVE 2025-03-30T05:47:51Z 1 2 1 m5.large,m6a.large,m6i.large AL2_x86_64 eks-blue-mng-2025033005474923890000002b-8ecaf2cb-c650-7b98-f580-92eede51bb96 managed eksworkshop-eksctl initial-20250330054749233900000029 ACTIVE 2025-03-30T05:47:51Z 2 10 2 m5.large,m6a.large,m6i.large AL2_x86_64 eks-initial-20250330054749233900000029-52caf2cb-c64d-9fc0-0588-dd6a67b41eb4 managed  # AutoScaling 그룹을 확인하면 blue, initial 말고도 selfmng 도 있는것을 확인할수 있고 내가 직접 관리하는 노드여서, 관리형 노드그룹에는 보이지 않습니다. 관리주체가 AWS가 아니고 내가 직접 관리하여야 합니다. # 참고로 kapenter 노드도 Self-managed로 표시됩니다.   # Fargete도 관리하는 워커노드도 1대 존재합니다.--> 워크샵의 다양한 실습을 위해 ec2-user:~/environment:$ eksctl get fargateprofile --cluster $CLUSTER_NAME NAME SELECTOR_NAMESPACE SELECTOR_LABELS POD_EXECUTION_ROLE_ARN SUBNETS TAGS STATUS fp-profile assets <none> arn:aws:iam::678866666676:role/fp-profile-2025033005474370920000001f subnet-082eeb7c16623ae93,subnet-042922ea084483364,subnet-040fd59c3e2770cb9 Blueprint=eksworkshop-eksctl,GithubRepo=github.com/aws-ia/terraform-aws-eks-blueprints,karpenter.sh/discovery=eksworkshop-eksctl ACTIVE # 4개의 Add-on 을 사용하고 있습니다 ec2-user:~/environment:$ eksctl get addon --cluster $CLUSTER_NAME 2025-04-01 13:15:56 [ℹ] Kubernetes version "1.25" in use by cluster "eksworkshop-eksctl" 2025-04-01 13:15:56 [ℹ] getting all addons 2025-04-01 13:15:58 [ℹ] to see issues for an addon run `eksctl get addon --name <addon-name> --cluster <cluster-name>` NAME VERSION STATUS ISSUES IAMROLE UPDATE AVAILABLE CONFIGURATION VALUES aws-ebs-csi-driver v1.41.0-eksbuild.1 ACTIVE 0 arn:aws:iam::678866666676:role/eksworkshop-eksctl-ebs-csi-driver-2025033005471464490000001d coredns v1.8.7-eksbuild.10 ACTIVE 0 v1.9.3-eksbuild.22,v1.9.3-eksbuild.21,v1.9.3-eksbuild.19,v1.9.3-eksbuild.17,v1.9.3-eksbuild.15,v1.9.3-eksbuild.11,v1.9.3-eksbuild.10,v1.9.3-eksbuild.9,v1.9.3-eksbuild.7,v1.9.3-eksbuild.6,v1.9.3-eksbuild.5,v1.9.3-eksbuild.3,v1.9.3-eksbuild.2 kube-proxy v1.25.16-eksbuild.8 ACTIVE 0 vpc-cni v1.19.3-eksbuild.1 ACTIVE 0  |
kubectl 을 이용하여 클러스터 정보를 확인합니다.
| # CAPACITYTYPE이 spot으로 표현되는것이 kapenter 입니다. ec2-user:~/environment:$ kubectl get node --label-columns=eks.amazonaws.com/capacityType,node.kubernetes.io/lifecycle,karpenter.sh/capacity-type,eks.amazonaws.com/compute-type NAME STATUS ROLES AGE VERSION CAPACITYTYPE LIFECYCLE CAPACITY-TYPE COMPUTE-TYPE fargate-ip-10-0-32-234.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-2d5f260 fargate ip-10-0-13-173.us-west-2.compute.internal Ready <none> 18h v1.25.16-eks-59bf375 spot ip-10-0-14-181.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-59bf375 self-managed ip-10-0-16-129.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-59bf375 self-managed ip-10-0-26-237.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-59bf375 ON_DEMAND ip-10-0-32-44.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-59bf375 ON_DEMAND ip-10-0-6-153.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-59bf375 ON_DEMAND # kapenter는 nodepool을 사용하고 있어서 레이블을 확인해 봅니다. ec2-user:~/environment:$ kubectl get node -L eks.amazonaws.com/nodegroup,karpenter.sh/nodepool NAME STATUS ROLES AGE VERSION NODEGROUP NODEPOOL fargate-ip-10-0-32-234.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-2d5f260 ip-10-0-13-173.us-west-2.compute.internal Ready <none> 18h v1.25.16-eks-59bf375 default ip-10-0-14-181.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-59bf375 ip-10-0-16-129.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-59bf375 ip-10-0-26-237.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-59bf375 initial-20250330054749233900000029 ip-10-0-32-44.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-59bf375 initial-20250330054749233900000029 ip-10-0-6-153.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-59bf375 blue-mng-2025033005474923890000002b ec2-user:~/environment:$ kubectl get nodepools NAME NODECLASS default default ec2-user:~/environment:$ kubectl get nodeclaims -o yaml apiVersion: v1
items:
- apiVersion: karpenter.sh/v1beta1
kind: NodeClaim
metadata:
annotations:
karpenter.k8s.aws/ec2nodeclass-hash: "5256777658067331158"
karpenter.k8s.aws/ec2nodeclass-hash-version: v2
karpenter.k8s.aws/tagged: "true"
karpenter.sh/nodepool-hash: "12028663807258658692"
karpenter.sh/nodepool-hash-version: v2
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"karpenter.k8s.aws/v1beta1","kind":"EC2NodeClass","metadata":{"annotations":{},"labels":{"argocd.argoproj.io/instance":"karpenter"},"name":"default"},"spec":{"amiFamily":"AL2","amiSelectorTerms":[{"id":"ami-0ee947a6f4880da75"}],"role":"karpenter-eksworkshop-eksctl","securityGroupSelectorTerms":[{"tags":{"karpenter.sh/discovery":"eksworkshop-eksctl"}}],"subnetSelectorTerms":[{"tags":{"karpenter.sh/discovery":"eksworkshop-eksctl"}}],"tags":{"intent":"apps","managed-by":"karpenter","team":"checkout"}}}
creationTimestamp: "2025-03-31T19:14:27Z"
finalizers:
- karpenter.sh/termination
generateName: default-
generation: 1
labels:
env: dev
karpenter.k8s.aws/instance-category: c
karpenter.k8s.aws/instance-cpu: "2"
karpenter.k8s.aws/instance-cpu-manufacturer: intel
karpenter.k8s.aws/instance-ebs-bandwidth: "500"
karpenter.k8s.aws/instance-encryption-in-transit-supported: "false"
karpenter.k8s.aws/instance-family: c4
karpenter.k8s.aws/instance-generation: "4"
karpenter.k8s.aws/instance-hypervisor: xen
karpenter.k8s.aws/instance-memory: "3840"
karpenter.k8s.aws/instance-size: large
karpenter.sh/capacity-type: spot
karpenter.sh/nodepool: default
kubernetes.io/arch: amd64
kubernetes.io/os: linux
node.kubernetes.io/instance-type: c4.large
team: checkout
topology.k8s.aws/zone-id: usw2-az2
topology.kubernetes.io/region: us-west-2
topology.kubernetes.io/zone: us-west-2a
name: default-8t24l
ownerReferences:
- apiVersion: karpenter.sh/v1beta1
blockOwnerDeletion: true
kind: NodePool
name: default
uid: e7eb9787-64be-45eb-9440-0130875176fe
resourceVersion: "979813"
uid: 7df9a4dd-9f22-4f3b-89d6-2eddf8eb42b8
spec:
nodeClassRef:
name: default
requirements:
- key: kubernetes.io/os
operator: In
values:
- linux
- key: env
operator: In
values:
- dev
- key: kubernetes.io/arch
operator: In
values:
- amd64
- key: karpenter.sh/capacity-type
operator: In
values:
- on-demand
- spot
- key: karpenter.k8s.aws/instance-family
operator: In
values:
- c4
- c5
- m5
- m6a
- m6i
- r4
- key: topology.kubernetes.io/zone
operator: In
values:
- us-west-2a
- key: node.kubernetes.io/instance-type
operator: In
values:
- c4.2xlarge
- c4.4xlarge
- c4.8xlarge
- c4.large
- c4.xlarge
- c5.12xlarge
- c5.18xlarge
- c5.24xlarge
- c5.2xlarge
- c5.4xlarge
- c5.9xlarge
- c5.large
- c5.metal
- c5.xlarge
- m5.12xlarge
- m5.16xlarge
- m5.24xlarge
- m5.2xlarge
- m5.4xlarge
- m5.8xlarge
- m5.large
- m5.metal
- m5.xlarge
- m6a.12xlarge
- m6a.16xlarge
- m6a.24xlarge
- m6a.2xlarge
- m6a.4xlarge
- m6a.8xlarge
- m6a.large
- m6a.xlarge
- m6i.12xlarge
- m6i.16xlarge
- m6i.24xlarge
- m6i.2xlarge
- m6i.4xlarge
- m6i.8xlarge
- m6i.large
- m6i.xlarge
- r4.16xlarge
- r4.2xlarge
- r4.4xlarge
- r4.8xlarge
- r4.large
- r4.xlarge
- key: team
operator: In
values:
- checkout
- key: karpenter.sh/nodepool
operator: In
values:
- default
resources:
requests:
cpu: 180m
memory: 120Mi
pods: "5"
taints:
- effect: NoSchedule
key: dedicated
value: CheckoutApp
status:
allocatable:
cpu: 1930m
ephemeral-storage: 17Gi
memory: 2878Mi
pods: "29"
capacity:
cpu: "2"
ephemeral-storage: 20Gi
memory: 3552Mi
pods: "29"
conditions:
- lastTransitionTime: "2025-03-31T19:15:36Z"
message: ""
reason: Initialized
status: "True"
type: Initialized
- lastTransitionTime: "2025-03-31T19:14:30Z"
message: ""
reason: Launched
status: "True"
type: Launched
- lastTransitionTime: "2025-03-31T19:15:36Z"
message: ""
reason: Ready
status: "True"
type: Ready
- lastTransitionTime: "2025-03-31T19:15:14Z"
message: ""
reason: Registered
status: "True"
type: Registered
imageID: ami-0ee947a6f4880da75
nodeName: ip-10-0-13-173.us-west-2.compute.internal
providerID: aws:///us-west-2a/i-040568a2c6002ca45
kind: List
metadata:
resourceVersion: ""
ec2-user:~/environment:$ kubectl get nodeclaims NAME TYPE ZONE NODE READY AGE default-8t24l c4.large us-west-2a ip-10-0-13-173.us-west-2.compute.internal True 18h  # 노드의 가용영역 확인 ec2-user:~/environment:$ kubectl get node --label-columns=node.kubernetes.io/instance-type,kubernetes.io/arch,kubernetes.io/os,topology.kubernetes.io/zone NAME STATUS ROLES AGE VERSION INSTANCE-TYPE ARCH OS ZONE fargate-ip-10-0-32-234.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-2d5f260 amd64 linux us-west-2c ip-10-0-13-173.us-west-2.compute.internal Ready <none> 18h v1.25.16-eks-59bf375 c4.large amd64 linux us-west-2a ip-10-0-14-181.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-59bf375 m5.large amd64 linux us-west-2a ip-10-0-16-129.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-59bf375 m5.large amd64 linux us-west-2b ip-10-0-26-237.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-59bf375 m5.large amd64 linux us-west-2b ip-10-0-32-44.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-59bf375 m5.large amd64 linux us-west-2c ip-10-0-6-153.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-59bf375 m5.large amd64 linux us-west-2a ec2-user:~/environment:$ kubectl cluster-info Kubernetes control plane is running at https://52C16A04B8574AC4AA8D5815B27D9AF2.gr7.us-west-2.eks.amazonaws.com CoreDNS is running at https://52C16A04B8574AC4AA8D5815B27D9AF2.gr7.us-west-2.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. # 현재 K8S v1.25.16 버전임 ec2-user:~/environment:$ kubectl get nodes -owide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME fargate-ip-10-0-32-234.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-2d5f260 10.0.32.234 <none> Amazon Linux 2 5.10.234-225.910.amzn2.x86_64 containerd://1.7.25 ip-10-0-13-173.us-west-2.compute.internal Ready <none> 18h v1.25.16-eks-59bf375 10.0.13.173 <none> Amazon Linux 2 5.10.230-223.885.amzn2.x86_64 containerd://1.7.23 ip-10-0-14-181.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-59bf375 10.0.14.181 <none> Amazon Linux 2 5.10.230-223.885.amzn2.x86_64 containerd://1.7.23 ip-10-0-16-129.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-59bf375 10.0.16.129 <none> Amazon Linux 2 5.10.230-223.885.amzn2.x86_64 containerd://1.7.23 ip-10-0-26-237.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-59bf375 10.0.26.237 <none> Amazon Linux 2 5.10.234-225.910.amzn2.x86_64 containerd://1.7.25 ip-10-0-32-44.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-59bf375 10.0.32.44 <none> Amazon Linux 2 5.10.234-225.910.amzn2.x86_64 containerd://1.7.25 ip-10-0-6-153.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-59bf375 10.0.6.153 <none> Amazon Linux 2 5.10.234-225.910.amzn2.x86_64 containerd://1.7.25 |
| ec2-user:~/environment:$ kubectl get crd NAME CREATED AT applications.argoproj.io 2025-03-30T05:52:03Z applicationsets.argoproj.io 2025-03-30T05:52:03Z appprojects.argoproj.io 2025-03-30T05:52:03Z cninodes.vpcresources.k8s.aws 2025-03-30T05:43:32Z ec2nodeclasses.karpenter.k8s.aws 2025-03-30T05:52:28Z eniconfigs.crd.k8s.amazonaws.com 2025-03-30T05:45:19Z ingressclassparams.elbv2.k8s.aws 2025-03-30T05:52:29Z nodeclaims.karpenter.sh 2025-03-30T05:52:28Z nodepools.karpenter.sh 2025-03-30T05:52:28Z policyendpoints.networking.k8s.aws 2025-03-30T05:43:32Z securitygrouppolicies.vpcresources.k8s.aws 2025-03-30T05:43:32Z targetgroupbindings.elbv2.k8s.aws 2025-03-30T05:52:29Z # helm으로 배포된 App 확인 -> CodeCommit의 git에 있는 정보를 이용하여 프로비전될때 App 이 배포되어 있음 ec2-user:~/environment:$ helm list -A NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION argo-cd argocd 1 2025-03-30 05:51:59.982473164 +0000 UTC deployed argo-cd-5.55.0 v2.10.0 aws-efs-csi-driver kube-system 1 2025-03-30 05:52:29.518505175 +0000 UTC deployed aws-efs-csi-driver-2.5.6 1.7.6 aws-load-balancer-controller kube-system 1 2025-03-30 05:52:31.319524371 +0000 UTC deployed aws-load-balancer-controller-1.7.1 v2.7.1 karpenter karpenter 1 2025-03-30 05:52:29.004740889 +0000 UTC deployed karpenter-0.37.0 0.37.0 metrics-server kube-system 1 2025-03-30 05:51:59.974677692 +0000 UTC deployed metrics-server-3.12.0 0.7.0 # apps가 ArgoCD의 App-of-app 으로 구성되어 있습니다. ec2-user:~/environment:$ kubectl get applications -n argocd NAME SYNC STATUS HEALTH STATUS apps Synced Healthy assets Synced Healthy carts Synced Healthy catalog Synced Healthy checkout Synced Healthy karpenter Synced Healthy orders Synced Healthy other Synced Healthy rabbitmq Synced Healthy ui OutOfSync Healthy ec2-user:~/environment:$ kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE argocd argo-cd-argocd-application-controller-0 1/1 Running 0 2d7h argocd argo-cd-argocd-applicationset-controller-74d9c9c5c7-f8vnj 1/1 Running 0 2d7h argocd argo-cd-argocd-dex-server-6dbbd57479-jtmv5 1/1 Running 0 2d7h argocd argo-cd-argocd-notifications-controller-fb4b954d5-99p8x 1/1 Running 0 2d7h argocd argo-cd-argocd-redis-76b4c599dc-5nt64 1/1 Running 0 2d7h argocd argo-cd-argocd-repo-server-6b777b579d-vgk59 1/1 Running 0 2d7h argocd argo-cd-argocd-server-86bdbd7b89-66hdr 1/1 Running 0 2d7h assets assets-7ccc84cb4d-mqs5h 1/1 Running 0 2d7h carts carts-7ddbc698d8-q7vjh 1/1 Running 0 2d7h carts carts-dynamodb-6594f86bb9-x6785 1/1 Running 0 2d7h catalog catalog-857f89d57d-b69ch 1/1 Running 3 (2d7h ago) 2d7h catalog catalog-mysql-0 1/1 Running 0 2d7h checkout checkout-558f7777c-d6gcv 1/1 Running 0 18h checkout checkout-redis-f54bf7cb5-5b2fm 1/1 Running 0 18h karpenter karpenter-5696f77444-f8cnl 1/1 Running 1 (2d7h ago) 2d7h karpenter karpenter-5696f77444-jrqkg 1/1 Running 0 2d7h kube-system aws-load-balancer-controller-654b56467-l2rhm 1/1 Running 0 2d7h kube-system aws-load-balancer-controller-654b56467-nz5jk 1/1 Running 0 2d7h kube-system aws-node-5m4fr 2/2 Running 0 18h kube-system aws-node-5wcq2 2/2 Running 0 2d7h kube-system aws-node-759tc 2/2 Running 0 2d7h kube-system aws-node-cp5pj 2/2 Running 0 2d7h kube-system aws-node-cx49r 2/2 Running 0 2d7h kube-system aws-node-sxcsr 2/2 Running 0 2d7h kube-system coredns-98f76fbc4-rljqj 1/1 Running 0 2d7h kube-system coredns-98f76fbc4-wc52p 1/1 Running 0 2d7h kube-system ebs-csi-controller-6b575b5f4d-4p74v 6/6 Running 0 2d7h kube-system ebs-csi-controller-6b575b5f4d-dt2z4 6/6 Running 0 2d7h kube-system ebs-csi-node-5d95n 3/3 Running 0 2d7h kube-system ebs-csi-node-fcb47 3/3 Running 0 2d7h kube-system ebs-csi-node-g4zzf 3/3 Running 0 2d7h kube-system ebs-csi-node-mznsn 3/3 Running 0 2d7h kube-system ebs-csi-node-tf7ml 3/3 Running 0 2d7h kube-system ebs-csi-node-x7vnp 3/3 Running 0 18h kube-system efs-csi-controller-5d74ddd947-q8kb7 3/3 Running 0 2d7h kube-system efs-csi-controller-5d74ddd947-qsbjt 3/3 Running 0 2d7h kube-system efs-csi-node-7v2db 3/3 Running 0 18h kube-system efs-csi-node-bw5cd 3/3 Running 0 2d7h kube-system efs-csi-node-kpzxs 3/3 Running 0 2d7h kube-system efs-csi-node-mlwnn 3/3 Running 0 2d7h kube-system efs-csi-node-xwgwb 3/3 Running 0 2d7h kube-system efs-csi-node-zxs9z 3/3 Running 0 2d7h kube-system kube-proxy-92h65 1/1 Running 0 2d7h kube-system kube-proxy-djdnq 1/1 Running 0 18h kube-system kube-proxy-hhs2s 1/1 Running 0 2d7h kube-system kube-proxy-m5sm5 1/1 Running 0 2d7h kube-system kube-proxy-nzgwr 1/1 Running 0 2d7h kube-system kube-proxy-qtrws 1/1 Running 0 2d7h kube-system metrics-server-785cd745cd-vcqv8 1/1 Running 0 2d7h orders orders-5b97745747-wbcvx 1/1 Running 2 (2d7h ago) 2d7h orders orders-mysql-b9b997d9d-h9xwd 1/1 Running 0 2d7h rabbitmq rabbitmq-0 1/1 Running 0 2d7h ui ui-5dfb7d65fc-9xh92 1/1 Running 0 2d7h # 중요한 요소는 PDB가 걸려 있습니다. ec2-user:~/environment:$ kubectl get pdb -A NAMESPACE NAME MIN AVAILABLE MAX UNAVAILABLE ALLOWED DISRUPTIONS AGE karpenter karpenter N/A 1 1 2d7h kube-system aws-load-balancer-controller N/A 1 1 2d7h kube-system coredns N/A 1 1 2d7h kube-system ebs-csi-controller N/A 1 1 2d7h # ArgoCD는 NLB로 등록되어 있습니다. ec2-user:~/environment:$ kubectl get svc -n argocd argo-cd-argocd-server NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE argo-cd-argocd-server LoadBalancer 172.20.130.241 k8s-argocd-argocdar-06bc31bc9d-9d58f820cb2beb72.elb.us-west-2.amazonaws.com 80:32514/TCP,443:31773/TCP 2d7h   ec2-user:~/environment:$ kubectl get targetgroupbindings -n argocd NAME SERVICE-NAME SERVICE-PORT TARGET-TYPE AGE k8s-argocd-argocdar-34856478fb argo-cd-argocd-server 443 ip 2d7h k8s-argocd-argocdar-bd5df94fc6 argo-cd-argocd-server 80 ip 2d7h |
| # 노드에 taint 정보 확인 ec2-user:~/environment:$ kubectl get nodes -o custom-columns='NODE:.metadata.name,TAINTS:.spec.taints[*].key,VALUES:.spec.taints[*].value,EFFECTS:.spec.taints[*].effect' NODE TAINTS VALUES EFFECTS fargate-ip-10-0-32-234.us-west-2.compute.internal eks.amazonaws.com/compute-type fargate NoSchedule ip-10-0-13-173.us-west-2.compute.internal dedicated CheckoutApp NoSchedule ip-10-0-14-181.us-west-2.compute.internal <none> <none> <none> ip-10-0-16-129.us-west-2.compute.internal <none> <none> <none> ip-10-0-26-237.us-west-2.compute.internal <none> <none> <none> ip-10-0-32-44.us-west-2.compute.internal <none> <none> <none> ip-10-0-6-153.us-west-2.compute.internal dedicated OrdersApp NoSchedule ec2-user:~/environment:$ kubectl get node -L eks.amazonaws.com/nodegroup,karpenter.sh/nodepool NAME STATUS ROLES AGE VERSION NODEGROUP NODEPOOL fargate-ip-10-0-32-234.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-2d5f260 ip-10-0-13-173.us-west-2.compute.internal Ready <none> 18h v1.25.16-eks-59bf375 default ip-10-0-14-181.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-59bf375 ip-10-0-16-129.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-59bf375 ip-10-0-26-237.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-59bf375 initial-20250330054749233900000029 ip-10-0-32-44.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-59bf375 initial-20250330054749233900000029 ip-10-0-6-153.us-west-2.compute.internal Ready <none> 2d7h v1.25.16-eks-59bf375 blue-mng-2025033005474923890000002b # 노드 별 label 확인 ec2-user:~/environment:$ kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, labels: .metadata.labels}' {
"name": "fargate-ip-10-0-32-234.us-west-2.compute.internal",
"labels": {
"beta.kubernetes.io/arch": "amd64",
"beta.kubernetes.io/os": "linux",
"eks.amazonaws.com/compute-type": "fargate",
"failure-domain.beta.kubernetes.io/region": "us-west-2",
"failure-domain.beta.kubernetes.io/zone": "us-west-2c",
"kubernetes.io/arch": "amd64",
"kubernetes.io/hostname": "ip-10-0-32-234.us-west-2.compute.internal",
"kubernetes.io/os": "linux",
"topology.kubernetes.io/region": "us-west-2",
"topology.kubernetes.io/zone": "us-west-2c"
}
}
{
"name": "ip-10-0-13-173.us-west-2.compute.internal",
"labels": {
"beta.kubernetes.io/arch": "amd64",
"beta.kubernetes.io/instance-type": "c4.large",
"beta.kubernetes.io/os": "linux",
"env": "dev",
"failure-domain.beta.kubernetes.io/region": "us-west-2",
"failure-domain.beta.kubernetes.io/zone": "us-west-2a",
"k8s.io/cloud-provider-aws": "a94967527effcefb5f5829f529c0a1b9",
"karpenter.k8s.aws/instance-category": "c",
"karpenter.k8s.aws/instance-cpu": "2",
"karpenter.k8s.aws/instance-cpu-manufacturer": "intel",
"karpenter.k8s.aws/instance-ebs-bandwidth": "500",
"karpenter.k8s.aws/instance-encryption-in-transit-supported": "false",
"karpenter.k8s.aws/instance-family": "c4",
"karpenter.k8s.aws/instance-generation": "4",
"karpenter.k8s.aws/instance-hypervisor": "xen",
"karpenter.k8s.aws/instance-memory": "3840",
"karpenter.k8s.aws/instance-size": "large",
"karpenter.sh/capacity-type": "spot",
"karpenter.sh/initialized": "true",
"karpenter.sh/nodepool": "default",
"karpenter.sh/registered": "true",
"kubernetes.io/arch": "amd64",
"kubernetes.io/hostname": "ip-10-0-13-173.us-west-2.compute.internal",
"kubernetes.io/os": "linux",
"node.kubernetes.io/instance-type": "c4.large",
"team": "checkout",
"topology.ebs.csi.aws.com/zone": "us-west-2a",
"topology.k8s.aws/zone-id": "usw2-az2",
"topology.kubernetes.io/region": "us-west-2",
"topology.kubernetes.io/zone": "us-west-2a"
}
}
{
"name": "ip-10-0-14-181.us-west-2.compute.internal",
"labels": {
"beta.kubernetes.io/arch": "amd64",
"beta.kubernetes.io/instance-type": "m5.large",
"beta.kubernetes.io/os": "linux",
"failure-domain.beta.kubernetes.io/region": "us-west-2",
"failure-domain.beta.kubernetes.io/zone": "us-west-2a",
"k8s.io/cloud-provider-aws": "a94967527effcefb5f5829f529c0a1b9",
"kubernetes.io/arch": "amd64",
"kubernetes.io/hostname": "ip-10-0-14-181.us-west-2.compute.internal",
"kubernetes.io/os": "linux",
"node.kubernetes.io/instance-type": "m5.large",
"node.kubernetes.io/lifecycle": "self-managed",
"team": "carts",
"topology.ebs.csi.aws.com/zone": "us-west-2a",
"topology.kubernetes.io/region": "us-west-2",
"topology.kubernetes.io/zone": "us-west-2a"
}
}
{
"name": "ip-10-0-16-129.us-west-2.compute.internal",
"labels": {
"beta.kubernetes.io/arch": "amd64",
"beta.kubernetes.io/instance-type": "m5.large",
"beta.kubernetes.io/os": "linux",
"failure-domain.beta.kubernetes.io/region": "us-west-2",
"failure-domain.beta.kubernetes.io/zone": "us-west-2b",
"k8s.io/cloud-provider-aws": "a94967527effcefb5f5829f529c0a1b9",
"kubernetes.io/arch": "amd64",
"kubernetes.io/hostname": "ip-10-0-16-129.us-west-2.compute.internal",
"kubernetes.io/os": "linux",
"node.kubernetes.io/instance-type": "m5.large",
"node.kubernetes.io/lifecycle": "self-managed",
"team": "carts",
"topology.ebs.csi.aws.com/zone": "us-west-2b",
"topology.kubernetes.io/region": "us-west-2",
"topology.kubernetes.io/zone": "us-west-2b"
}
}
{
"name": "ip-10-0-26-237.us-west-2.compute.internal",
"labels": {
"beta.kubernetes.io/arch": "amd64",
"beta.kubernetes.io/instance-type": "m5.large",
"beta.kubernetes.io/os": "linux",
"eks.amazonaws.com/capacityType": "ON_DEMAND",
"eks.amazonaws.com/nodegroup": "initial-20250330054749233900000029",
"eks.amazonaws.com/nodegroup-image": "ami-0078a0f78fafda978",
"eks.amazonaws.com/sourceLaunchTemplateId": "lt-08ccee8145fa44688",
"eks.amazonaws.com/sourceLaunchTemplateVersion": "1",
"failure-domain.beta.kubernetes.io/region": "us-west-2",
"failure-domain.beta.kubernetes.io/zone": "us-west-2b",
"k8s.io/cloud-provider-aws": "a94967527effcefb5f5829f529c0a1b9",
"kubernetes.io/arch": "amd64",
"kubernetes.io/hostname": "ip-10-0-26-237.us-west-2.compute.internal",
"kubernetes.io/os": "linux",
"node.kubernetes.io/instance-type": "m5.large",
"topology.ebs.csi.aws.com/zone": "us-west-2b",
"topology.kubernetes.io/region": "us-west-2",
"topology.kubernetes.io/zone": "us-west-2b"
}
}
{
"name": "ip-10-0-32-44.us-west-2.compute.internal",
"labels": {
"beta.kubernetes.io/arch": "amd64",
"beta.kubernetes.io/instance-type": "m5.large",
"beta.kubernetes.io/os": "linux",
"eks.amazonaws.com/capacityType": "ON_DEMAND",
"eks.amazonaws.com/nodegroup": "initial-20250330054749233900000029",
"eks.amazonaws.com/nodegroup-image": "ami-0078a0f78fafda978",
"eks.amazonaws.com/sourceLaunchTemplateId": "lt-08ccee8145fa44688",
"eks.amazonaws.com/sourceLaunchTemplateVersion": "1",
"failure-domain.beta.kubernetes.io/region": "us-west-2",
"failure-domain.beta.kubernetes.io/zone": "us-west-2c",
"k8s.io/cloud-provider-aws": "a94967527effcefb5f5829f529c0a1b9",
"kubernetes.io/arch": "amd64",
"kubernetes.io/hostname": "ip-10-0-32-44.us-west-2.compute.internal",
"kubernetes.io/os": "linux",
"node.kubernetes.io/instance-type": "m5.large",
"topology.ebs.csi.aws.com/zone": "us-west-2c",
"topology.kubernetes.io/region": "us-west-2",
"topology.kubernetes.io/zone": "us-west-2c"
}
}
{
"name": "ip-10-0-6-153.us-west-2.compute.internal",
"labels": {
"beta.kubernetes.io/arch": "amd64",
"beta.kubernetes.io/instance-type": "m5.large",
"beta.kubernetes.io/os": "linux",
"eks.amazonaws.com/capacityType": "ON_DEMAND",
"eks.amazonaws.com/nodegroup": "blue-mng-2025033005474923890000002b",
"eks.amazonaws.com/nodegroup-image": "ami-0078a0f78fafda978",
"eks.amazonaws.com/sourceLaunchTemplateId": "lt-05ef0255d28324859",
"eks.amazonaws.com/sourceLaunchTemplateVersion": "1",
"failure-domain.beta.kubernetes.io/region": "us-west-2",
"failure-domain.beta.kubernetes.io/zone": "us-west-2a",
"k8s.io/cloud-provider-aws": "a94967527effcefb5f5829f529c0a1b9",
"kubernetes.io/arch": "amd64",
"kubernetes.io/hostname": "ip-10-0-6-153.us-west-2.compute.internal",
"kubernetes.io/os": "linux",
"node.kubernetes.io/instance-type": "m5.large",
"topology.ebs.csi.aws.com/zone": "us-west-2a",
"topology.kubernetes.io/region": "us-west-2",
"topology.kubernetes.io/zone": "us-west-2a",
"type": "OrdersMNG"
}
}
# StatefulSet 어플리케이션은 마이그레이션에 주의를 하여야 합니다. 워크샵에는 3개의 STS가 있습니다. ec2-user:~/environment:$ kubectl get sts -A NAMESPACE NAME READY AGE argocd argo-cd-argocd-application-controller 1/1 2d7h catalog catalog-mysql 1/1 2d7h rabbitmq rabbitmq 1/1 2d7h ec2-user:~/environment:$ kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE efs efs.csi.aws.com Delete Immediate true 2d7h gp2 kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 2d7h gp3 (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 2d7h ec2-user:~/environment:$ kubectl describe sc efs Name: efs IsDefaultClass: No Annotations: <none> Provisioner: efs.csi.aws.com Parameters: basePath=/dynamic_provisioning,directoryPerms=755,ensureUniqueDirectory=false,fileSystemId=fs-0422473c6a69c97a1,gidRangeEnd=200,gidRangeStart=100,provisioningMode=efs-ap,reuseAccessPoint=false,subPathPattern=${.PVC.namespace}/${.PVC.name} AllowVolumeExpansion: True MountOptions: iam ReclaimPolicy: Delete VolumeBindingMode: Immediate Events: <none> ec2-user:~/environment:$ kubectl get pv,pvc -A NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE persistentvolume/pvc-39753d3e-3aa2-42ba-8f7c-1e42467e5dd1 4Gi RWO Delete Bound orders/order-mysql-pvc gp3 2d7h persistentvolume/pvc-803336d0-37c2-4ffb-bd2c-bfb8604516dc 4Gi RWO Delete Bound catalog/catalog-mysql-pvc gp3 2d7h persistentvolume/pvc-a0c70b4d-d789-4e0e-8880-527aaccaf8dc 4Gi RWO Delete Bound checkout/checkout-redis-pvc gp3 2d7h NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE catalog persistentvolumeclaim/catalog-mysql-pvc Bound pvc-803336d0-37c2-4ffb-bd2c-bfb8604516dc 4Gi RWO gp3 2d7h checkout persistentvolumeclaim/checkout-redis-pvc Bound pvc-a0c70b4d-d789-4e0e-8880-527aaccaf8dc 4Gi RWO gp3 2d7h orders persistentvolumeclaim/order-mysql-pvc Bound pvc-39753d3e-3aa2-42ba-8f7c-1e42467e5dd1 4Gi RWO gp3 2d7h |
| ec2-user:~/environment:$ aws eks list-access-entries --cluster-name $CLUSTER_NAME { "accessEntries": [ "arn:aws:iam::678866666676:role/aws-service-role/eks.amazonaws.com/AWSServiceRoleForAmazonEKS", "arn:aws:iam::678866666676:role/blue-mng-eks-node-group-2025033005375276890000000e", "arn:aws:iam::678866666676:role/default-selfmng-node-group-20250330053752898800000010", "arn:aws:iam::678866666676:role/fp-profile-2025033005474370920000001f", "arn:aws:iam::678866666676:role/initial-eks-node-group-2025033005375280780000000f", "arn:aws:iam::678866666676:role/karpenter-eksworkshop-eksctl", "arn:aws:iam::678866666676:role/workshop-stack-ekstfcodebuildStackDeployProjectRole-pXvvI9nod8e3" ] } ec2-user:~/environment:$ eksctl get iamidentitymapping --cluster $CLUSTER_NAME ARN USERNAME GROUPS ACCOUNT arn:aws:iam::678866666676:role/WSParticipantRole admin system:masters arn:aws:iam::678866666676:role/blue-mng-eks-node-group-2025033005375276890000000e system:node:{{EC2PrivateDNSName}} system:bootstrappers,system:nodes arn:aws:iam::678866666676:role/fp-profile-2025033005474370920000001f system:node:{{SessionName}} system:bootstrappers,system:nodes,system:node-proxier arn:aws:iam::678866666676:role/initial-eks-node-group-2025033005375280780000000f system:node:{{EC2PrivateDNSName}} system:bootstrappers,system:nodes arn:aws:iam::678866666676:role/workshop-stack-IdeIdeRoleD654ADD4-aVMldSuUNfaa admin system:masters ec2-user:~/environment:$ kubectl describe cm -n kube-system aws-auth Name: aws-auth Namespace: kube-system Labels: <none> Annotations: <none> Data ==== mapRoles: ---- - groups: - system:bootstrappers - system:nodes - system:node-proxier rolearn: arn:aws:iam::678866666676:role/fp-profile-2025033005474370920000001f username: system:node:{{SessionName}} - groups: - system:bootstrappers - system:nodes rolearn: arn:aws:iam::678866666676:role/blue-mng-eks-node-group-2025033005375276890000000e username: system:node:{{EC2PrivateDNSName}} - groups: - system:bootstrappers - system:nodes rolearn: arn:aws:iam::678866666676:role/initial-eks-node-group-2025033005375280780000000f username: system:node:{{EC2PrivateDNSName}} - groups: - system:masters rolearn: arn:aws:iam::678866666676:role/WSParticipantRole username: admin - groups: - system:masters rolearn: arn:aws:iam::678866666676:role/workshop-stack-IdeIdeRoleD654ADD4-aVMldSuUNfaa username: admin mapUsers: ---- [] BinaryData ==== Events: <none> # IRSA : karpenter, alb-controller, ebs-csi-driver, aws-efs-csi-driver 4곳에서 사용 ec2-user:~/environment:$ eksctl get iamserviceaccount --cluster $CLUSTER_NAME No iamserviceaccounts found ec2-user:~/environment:$ kubectl describe sa -A | grep role-arn Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::678866666676:role/karpenter-20250330055227339300000039 Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::678866666676:role/alb-controller-2025033005522735250000003c Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::678866666676:role/eksworkshop-eksctl-ebs-csi-driver-2025033005471464490000001d Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::678866666676:role/aws-efs-csi-driver-2025033005522734720000003a Annotations: eks.amazonaws.com/role-arn: arn:aws:iam::678866666676:role/aws-efs-csi-driver-2025033005522734720000003a # pod identity 없음 ec2-user:~/environment:$ eksctl get podidentityassociation --cluster $CLUSTER_NAME No podidentityassociations found # EKS API Endpoint ec2-user:~/environment:$ aws eks describe-cluster --name $CLUSTER_NAME | jq -r .cluster.endpoint | cut -d '/' -f 3 52C16A04B8574AC4AA8D5815B27D9AF2.gr7.us-west-2.eks.amazonaws.com ec2-user:~/environment:$ APIDNS=$(aws eks describe-cluster --name $CLUSTER_NAME | jq -r .cluster.endpoint | cut -d '/' -f 3) ec2-user:~/environment:$ echo $APIDNS 52C16A04B8574AC4AA8D5815B27D9AF2.gr7.us-west-2.eks.amazonaws.com ec2-user:~/environment:$ dig +short $APIDNS 100.20.157.222 54.201.21.161 ec2-user:~/environment:$ aws eks describe-cluster --name $CLUSTER_NAME --query cluster.identity.oidc.issuer --output text https://oidc.eks.us-west-2.amazonaws.com/id/52C16A04B8574AC4AA8D5815B27D9AF2 ec2-user:~/environment:$ aws iam list-open-id-connect-providers | jq {
"OpenIDConnectProviderList": [
{
"Arn": "arn:aws:iam::678866666676:oidc-provider/oidc.eks.us-west-2.amazonaws.com/id/52C16A04B8574AC4AA8D5815B27D9AF2"
}
]
}
|
EKS 관리콘솔에서 업그레이드 지원
관리콘솔에서도 업그레이드를 지원합니다. 하지만 이번 워크샵에서는 Terraform을 이용하여 진행하겠습니다.
- EC2 - ASG : 3개의 ASG 확인 - blug-mng 관리 노드는 AZ1 곳만 사용 중(us-west-2a)
- STS 어플리케이션에서 PV를 EBS를 사용하는데, EBS는 동일 가용영역에 있는 볼륨만 마운트가 가능하기 때무에 AG를 1곳으로 설정하였슴.
- 이는 워크샵의 목적이며, EBS를 이용하는 PV와 STS에서의 불편함을 체험해보고, kapenter를 이용하면 이런 부분도 관리해주기때문에 관리노력이 덜 드는것을 체험하기 위함임.
kube-ops-view 추가 설치
업그레이드 상황을 시각적으로 확인하기 위해 워큽샵 환경에는 없지만 추가로 설치합니다.
| # helm을 이용하여 kube-ops-view 설치 ec2-user:~/environment:$ helm repo add geek-cookbook https://geek-cookbook.github.io/charts/ "geek-cookbook" has been added to your repositories ec2-user:~/environment:$ helm repo update Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "geek-cookbook" chart repository Update Complete. ⎈Happy Helming!⎈ ec2-user:~/environment:$ helm install kube-ops-view geek-cookbook/kube-ops-view --version 1.2.2 --namespace kube-system NAME: kube-ops-view LAST DEPLOYED: Tue Apr 1 14:39:45 2025 NAMESPACE: kube-system STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: 1. Get the application URL by running these commands: export POD_NAME=$(kubectl get pods --namespace kube-system -l "app.kubernetes.io/name=kube-ops-view,app.kubernetes.io/instance=kube-ops-view" -o jsonpath="{.items[0].metadata.name}") echo "Visit http://127.0.0.1:8080 to use your application" kubectl port-forward $POD_NAME 8080:8080 # 실습 편의를 위핸 NLB 리소스 생성 ec2-user:~/environment:$ cat <<EOF | kubectl apply -f - apiVersion: v1
kind: Service
metadata:
annotations:
service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
service.beta.kubernetes.io/aws-load-balancer-type: external
labels:
app.kubernetes.io/instance: kube-ops-view
app.kubernetes.io/name: kube-ops-view
name: kube-ops-view-nlb
namespace: kube-system
spec:
type: LoadBalancer
ports:
- name: http
port: 80
protocol: TCP
targetPort: 8080
selector:
app.kubernetes.io/instance: kube-ops-view
app.kubernetes.io/name: kube-ops-view
EOF service/kube-ops-view-nlb created # kube-ops-view 접속 URL 확인 (1.5배율) ec2-user:~/environment:$ kubectl get svc -n kube-system kube-ops-view-nlb -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' | awk '{ print "KUBE-OPS-VIEW URL = http://"$1"/#scale=1.5"}' KUBE-OPS-VIEW URL = http://k8s-kubesyst-kubeopsv-5e59100577-8ace3f73927ffe47.elb.us-west-2.amazonaws.com/#scale=1.5  |
(옵션) 실습편의를 위한 추가도구 설치
- krew 설치
| # 설치 ec2-user:~/environment:$ ( set -x; cd "$(mktemp -d)" && OS="$(uname | tr '[:upper:]' '[:lower:]')" && ARCH="$(uname -m | sed -e 's/x86_64/amd64/' -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/')" && KREW="krew-${OS}_${ARCH}" && curl -fsSLO "https://github.com/kubernetes-sigs/krew/releases/latest/download/${KREW}.tar.gz" && tar zxvf "${KREW}.tar.gz" && ./"${KREW}" install krew ) ++ mktemp -d + cd /tmp/tmp.NqeFbbkp1Y ++ uname ++ tr '[:upper:]' '[:lower:]' + OS=linux ++ uname -m ++ sed -e s/x86_64/amd64/ -e 's/\(arm\)\(64\)\?.*/\1\2/' -e 's/aarch64$/arm64/' + ARCH=amd64 + KREW=krew-linux_amd64 + curl -fsSLO https://github.com/kubernetes-sigs/krew/releases/latest/download/krew-linux_amd64.tar.gz + tar zxvf krew-linux_amd64.tar.gz ./._LICENSE tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.provenance' ./LICENSE ./._krew-linux_amd64 tar: Ignoring unknown extended header keyword 'LIBARCHIVE.xattr.com.apple.provenance' ./krew-linux_amd64 + ./krew-linux_amd64 install krew Adding "default" plugin index from https://github.com/kubernetes-sigs/krew-index.git. Updated the local copy of plugin index. Installing plugin: krew Installed plugin: krew \ | Use this plugin: | kubectl krew | Documentation: | https://krew.sigs.k8s.io/ | Caveats: | \ | | krew is now installed! To start using kubectl plugins, you need to add | | krew's installation directory to your PATH: | | | | * macOS/Linux: | | - Add the following to your ~/.bashrc or ~/.zshrc: | | export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH" | | - Restart your shell. | | | | * Windows: Add %USERPROFILE%\.krew\bin to your PATH environment variable | | | | To list krew commands and to get help, run: | | $ kubectl krew | | For a full list of available plugins, run: | | $ kubectl krew search | | | | You can find documentation at | | https://krew.sigs.k8s.io/docs/user-guide/quickstart/. | / / # PATH 설정 ec2-user:~/environment:$ export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH" ec2-user:~/environment:$ vi ~/.bashrc ----------- export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH" ----------- # 플러그인 설치 ec2-user:~/environment:$ kubectl krew install ctx ns df-pv get-all neat stern oomd whoami rbac-tool rolesum Updated the local copy of plugin index. Installing plugin: ctx Installed plugin: ctx \ | Use this plugin: | kubectl ctx | Documentation: | https://github.com/ahmetb/kubectx | Caveats: | \ | | If fzf is installed on your machine, you can interactively choose | | between the entries using the arrow keys, or by fuzzy searching | | as you type. | | See https://github.com/ahmetb/kubectx for customization and details. | / / ...(생략)... ec2-user:~/environment:$ kubectl krew list PLUGIN VERSION ctx v0.9.5 df-pv v0.3.0 get-all v1.3.8 krew v0.4.5 neat v2.0.4 ns v0.9.5 oomd v0.0.7 rbac-tool v1.20.0 rolesum v1.5.5 stern v1.32.0 whoami v0.0.46 ec2-user:~/environment:$ kubectl df-pv PV NAME PVC NAME NAMESPACE NODE NAME POD NAME VOLUME MOUNT NAME SIZE USED AVAILABLE %USED IUSED IFREE %IUSED pvc-a0c70b4d-d789-4e0e-8880-527aaccaf8dc checkout-redis-pvc checkout ip-10-0-13-173.us-west-2.compute.internal checkout-redis-f54bf7cb5-5b2fm data 3Gi 28Ki 3Gi 0.00 12 262132 0.00 pvc-39753d3e-3aa2-42ba-8f7c-1e42467e5dd1 order-mysql-pvc orders ip-10-0-6-153.us-west-2.compute.internal orders-mysql-b9b997d9d-h9xwd data 3Gi 201Mi 3Gi 5.09 199 261945 0.08 pvc-803336d0-37c2-4ffb-bd2c-bfb8604516dc catalog-mysql-pvc catalog ip-10-0-16-129.us-west-2.compute.internal catalog-mysql-0 data 3Gi 201Mi 3Gi 5.09 201 261943 0.08 ec2-user:~/environment:$ kubectl whoami --all User: admin Groups: system:masters system:authenticated ARN: arn:aws:sts::678866666676:assumed-role/workshop-stack-IdeIdeRoleD654ADD4-aVMldSuUNfaa/i-068927555ce25af31 |
- eks-node-view
| # 설치 ec2-user:~/environment:$ wget -O eks-node-viewer https://github.com/awslabs/eks-node-viewer/releases/download/v0.7.1/eks-node-viewer_Linux_x86_64 ec2-user:~/environment:$ chmod +x eks-node-viewer ec2-user:~/environment:$ sudo mv -v eks-node-viewer /usr/local/bin # Standard usage : self-mng 노드는 가격 미출력됨 ec2-user:~/environment:$ eks-node-viewer  # Display both CPU and Memory Usage eks-node-viewer --resources cpu,memory eks-node-viewer --resources cpu,memory --extra-labels eks-node-viewer/node-age # Display extra labels, i.e. AZ : node 에 labels 사용 가능 eks-node-viewer --extra-labels topology.kubernetes.io/zone eks-node-viewer --extra-labels kubernetes.io/arch # Sort by CPU usage in descending order eks-node-viewer --node-sort=eks-node-viewer/node-cpu-usage=dsc # Karenter nodes only eks-node-viewer --node-selector "karpenter.sh/provisioner-name" # Specify a particular AWS profile and region AWS_PROFILE=myprofile AWS_REGION=us-west-2 Computed Labels : --extra-labels # eks-node-viewer/node-age - Age of the node eks-node-viewer --extra-labels eks-node-viewer/node-age eks-node-viewer --extra-labels topology.kubernetes.io/zone,eks-node-viewer/node-age # eks-node-viewer/node-ephemeral-storage-usage - Ephemeral Storage usage (requests) eks-node-viewer --extra-labels eks-node-viewer/node-ephemeral-storage-usage # eks-node-viewer/node-cpu-usage - CPU usage (requests) eks-node-viewer --extra-labels eks-node-viewer/node-cpu-usage # eks-node-viewer/node-memory-usage - Memory usage (requests) eks-node-viewer --extra-labels eks-node-viewer/node-memory-usage # eks-node-viewer/node-pods-usage - Pod usage (requests) eks-node-viewer --extra-labels eks-node-viewer/node-pods-usage |
- k9s
| # curl -sS https://webinstall.dev/k9s | bash k9s version # k9s |
테라폼 코드 확인
- 업그레이드 작업에서 사용 예정
| # 파일 확인 ec2-user:~/environment:$ ls -lrt terraform/ total 40 -rw-r--r--. 1 ec2-user ec2-user 625 Aug 27 2024 vpc.tf -rw-r--r--. 1 ec2-user ec2-user 542 Aug 27 2024 versions.tf -rw-r--r--. 1 ec2-user ec2-user 353 Aug 27 2024 variables.tf -rw-r--r--. 1 ec2-user ec2-user 282 Aug 27 2024 outputs.tf -rw-r--r--. 1 ec2-user ec2-user 1291 Aug 27 2024 gitops-setup.tf -rw-r--r--. 1 ec2-user ec2-user 4512 Feb 5 01:15 base.tf -rw-r--r--. 1 ec2-user ec2-user 5953 Feb 13 19:55 addons.tf -rw-r--r--. 1 ec2-user ec2-user 33 Mar 30 05:43 backend_override.tf # 해당 경로에 tfstate 파일은 없고, S3 에 존재합니다. ec2-user:~/environment:$ cd terraform ec2-user:~/environment/terraform :$ terraform output ╷ │ Warning: No outputs found │ │ The state file either has no outputs defined, or all the defined outputs are empty. Please define an output in your configuration with the │ `output` keyword and run `terraform refresh` for it to become available. If you are using interpolation, please verify the interpolated value is │ not empty. You can use the `terraform console` command to assist. ╵ # 테라폼 백엔드 조정 작업 수행 ec2-user:~/environment/terraform :$ aws s3 ls 2025-03-30 05:34:11 workshop-stack-tfstatebackendbucketf0fc9a9d-qll9it5dhvaw ec2-user:~/environment/terraform :$ aws s3 ls s3://workshop-stack-tfstatebackendbucketf0fc9a9d-qll9it5dhvaw 2025-03-30 05:55:13 933780 terraform.tfstate # (옵션) terraform.tfstate 복사 후 IDE-Server 에서 terraform.tfstate 열어보기 ec2-user:~/environment/terraform :$ aws s3 cp s3://workshop-stack-tfstatebackendbucketf0fc9a9d-qll9it5dhvaw . # backend_override.tf 수정 --> 본인의 S3 버킷으로 수정 terraform {
backend "s3" {
bucket = "workshop-stack-tfstatebackendbucketf0fc9a9d-qll9it5dhvaw"
region = "us-west-2"
key = "terraform.tfstate"
}
}
# 확인 ec2-user:~/environment/terraform:$ terraform state list data.aws_availability_zones.available data.aws_caller_identity.current data.aws_ecrpublic_authorization_token.token data.aws_partition.current aws_codecommit_repository.gitops_repo_cc aws_eks_access_entry.karpenter_node_access_entry aws_iam_service_specific_credential.argocd_codecommit_credential aws_iam_user.argocd_user aws_iam_user_policy.argocd_user_codecommit_rw aws_secretsmanager_secret.argocd_user_creds_secret aws_secretsmanager_secret_version.argocd_user_creds_secret_version kubernetes_annotations.gp2 kubernetes_storage_class_v1.efs kubernetes_storage_class_v1.gp3 time_sleep.wait_60_seconds time_sleep.wait_90_seconds module.ebs_csi_driver_irsa.data.aws_caller_identity.current module.ebs_csi_driver_irsa.data.aws_iam_policy_document.ebs_csi[0] module.ebs_csi_driver_irsa.data.aws_iam_policy_document.this[0] module.ebs_csi_driver_irsa.data.aws_partition.current module.ebs_csi_driver_irsa.data.aws_region.current module.ebs_csi_driver_irsa.aws_iam_policy.ebs_csi[0] module.ebs_csi_driver_irsa.aws_iam_role.this[0] module.ebs_csi_driver_irsa.aws_iam_role_policy_attachment.ebs_csi[0] module.efs.data.aws_iam_policy_document.policy[0] module.efs.aws_efs_backup_policy.this[0] module.efs.aws_efs_file_system.this[0] module.efs.aws_efs_file_system_policy.this[0] module.efs.aws_efs_mount_target.this["us-west-2a"] module.efs.aws_efs_mount_target.this["us-west-2b"] module.efs.aws_efs_mount_target.this["us-west-2c"] module.efs.aws_security_group.this[0] module.efs.aws_security_group_rule.this["vpc"] module.eks.data.aws_caller_identity.current[0] module.eks.data.aws_iam_policy_document.assume_role_policy[0] module.eks.data.aws_iam_policy_document.custom[0] module.eks.data.aws_iam_session_context.current[0] module.eks.data.aws_partition.current[0] module.eks.data.tls_certificate.this[0] module.eks.aws_cloudwatch_log_group.this[0] module.eks.aws_ec2_tag.cluster_primary_security_group["Blueprint"] module.eks.aws_ec2_tag.cluster_primary_security_group["GithubRepo"] module.eks.aws_ec2_tag.cluster_primary_security_group["karpenter.sh/discovery"] module.eks.aws_eks_access_entry.this["cluster_creator"] module.eks.aws_eks_access_policy_association.this["cluster_creator_admin"] module.eks.aws_eks_cluster.this[0] module.eks.aws_iam_openid_connect_provider.oidc_provider[0] module.eks.aws_iam_policy.cluster_encryption[0] module.eks.aws_iam_policy.custom[0] module.eks.aws_iam_role.this[0] module.eks.aws_iam_role_policy_attachment.cluster_encryption[0] module.eks.aws_iam_role_policy_attachment.custom[0] module.eks.aws_iam_role_policy_attachment.this["AmazonEKSClusterPolicy"] module.eks.aws_iam_role_policy_attachment.this["AmazonEKSVPCResourceController"] module.eks.aws_security_group.cluster[0] module.eks.aws_security_group.node[0] module.eks.aws_security_group_rule.cluster["ingress_nodes_443"] module.eks.aws_security_group_rule.node["egress_all"] module.eks.aws_security_group_rule.node["ingress_cluster_443"] module.eks.aws_security_group_rule.node["ingress_cluster_4443_webhook"] module.eks.aws_security_group_rule.node["ingress_cluster_6443_webhook"] module.eks.aws_security_group_rule.node["ingress_cluster_8443_webhook"] module.eks.aws_security_group_rule.node["ingress_cluster_9443_webhook"] module.eks.aws_security_group_rule.node["ingress_cluster_kubelet"] module.eks.aws_security_group_rule.node["ingress_nodes_ephemeral"] module.eks.aws_security_group_rule.node["ingress_self_coredns_tcp"] module.eks.aws_security_group_rule.node["ingress_self_coredns_udp"] module.eks.time_sleep.this[0] module.eks_blueprints_addons.data.aws_caller_identity.current module.eks_blueprints_addons.data.aws_eks_addon_version.this["aws-ebs-csi-driver"] module.eks_blueprints_addons.data.aws_eks_addon_version.this["coredns"] module.eks_blueprints_addons.data.aws_eks_addon_version.this["kube-proxy"] module.eks_blueprints_addons.data.aws_eks_addon_version.this["vpc-cni"] module.eks_blueprints_addons.data.aws_iam_policy_document.aws_efs_csi_driver[0] module.eks_blueprints_addons.data.aws_iam_policy_document.aws_load_balancer_controller[0] module.eks_blueprints_addons.data.aws_iam_policy_document.karpenter[0] module.eks_blueprints_addons.data.aws_iam_policy_document.karpenter_assume_role[0] module.eks_blueprints_addons.data.aws_partition.current module.eks_blueprints_addons.data.aws_region.current module.eks_blueprints_addons.aws_cloudformation_stack.usage_telemetry[0] module.eks_blueprints_addons.aws_cloudwatch_event_rule.karpenter["health_event"] module.eks_blueprints_addons.aws_cloudwatch_event_rule.karpenter["instance_rebalance"] module.eks_blueprints_addons.aws_cloudwatch_event_rule.karpenter["instance_state_change"] module.eks_blueprints_addons.aws_cloudwatch_event_rule.karpenter["spot_interupt"] module.eks_blueprints_addons.aws_cloudwatch_event_target.karpenter["health_event"] module.eks_blueprints_addons.aws_cloudwatch_event_target.karpenter["instance_rebalance"] module.eks_blueprints_addons.aws_cloudwatch_event_target.karpenter["instance_state_change"] module.eks_blueprints_addons.aws_cloudwatch_event_target.karpenter["spot_interupt"] module.eks_blueprints_addons.aws_eks_addon.this["aws-ebs-csi-driver"] module.eks_blueprints_addons.aws_eks_addon.this["coredns"] module.eks_blueprints_addons.aws_eks_addon.this["kube-proxy"] module.eks_blueprints_addons.aws_eks_addon.this["vpc-cni"] module.eks_blueprints_addons.aws_iam_instance_profile.karpenter[0] module.eks_blueprints_addons.aws_iam_role.karpenter[0] module.eks_blueprints_addons.aws_iam_role_policy_attachment.karpenter["AmazonEC2ContainerRegistryReadOnly"] module.eks_blueprints_addons.aws_iam_role_policy_attachment.karpenter["AmazonEKSWorkerNodePolicy"] module.eks_blueprints_addons.aws_iam_role_policy_attachment.karpenter["AmazonEKS_CNI_Policy"] module.eks_blueprints_addons.random_bytes.this module.eks_blueprints_addons.time_sleep.this module.vpc.aws_default_network_acl.this[0] module.vpc.aws_default_route_table.default[0] module.vpc.aws_default_security_group.this[0] module.vpc.aws_eip.nat[0] module.vpc.aws_internet_gateway.this[0] module.vpc.aws_nat_gateway.this[0] module.vpc.aws_route.private_nat_gateway[0] module.vpc.aws_route.public_internet_gateway[0] module.vpc.aws_route_table.private[0] module.vpc.aws_route_table.public[0] module.vpc.aws_route_table_association.private[0] module.vpc.aws_route_table_association.private[1] module.vpc.aws_route_table_association.private[2] module.vpc.aws_route_table_association.public[0] module.vpc.aws_route_table_association.public[1] module.vpc.aws_route_table_association.public[2] module.vpc.aws_subnet.private[0] module.vpc.aws_subnet.private[1] module.vpc.aws_subnet.private[2] module.vpc.aws_subnet.public[0] module.vpc.aws_subnet.public[1] module.vpc.aws_subnet.public[2] module.vpc.aws_vpc.this[0] module.eks.module.eks_managed_node_group["blue-mng"].data.aws_caller_identity.current module.eks.module.eks_managed_node_group["blue-mng"].data.aws_iam_policy_document.assume_role_policy[0] module.eks.module.eks_managed_node_group["blue-mng"].data.aws_partition.current module.eks.module.eks_managed_node_group["blue-mng"].aws_eks_node_group.this[0] module.eks.module.eks_managed_node_group["blue-mng"].aws_iam_role.this[0] module.eks.module.eks_managed_node_group["blue-mng"].aws_iam_role_policy_attachment.this["AmazonEC2ContainerRegistryReadOnly"] module.eks.module.eks_managed_node_group["blue-mng"].aws_iam_role_policy_attachment.this["AmazonEKSWorkerNodePolicy"] module.eks.module.eks_managed_node_group["blue-mng"].aws_iam_role_policy_attachment.this["AmazonEKS_CNI_Policy"] module.eks.module.eks_managed_node_group["blue-mng"].aws_launch_template.this[0] module.eks.module.eks_managed_node_group["initial"].data.aws_caller_identity.current module.eks.module.eks_managed_node_group["initial"].data.aws_iam_policy_document.assume_role_policy[0] module.eks.module.eks_managed_node_group["initial"].data.aws_partition.current module.eks.module.eks_managed_node_group["initial"].aws_eks_node_group.this[0] module.eks.module.eks_managed_node_group["initial"].aws_iam_role.this[0] module.eks.module.eks_managed_node_group["initial"].aws_iam_role_policy_attachment.this["AmazonEC2ContainerRegistryReadOnly"] module.eks.module.eks_managed_node_group["initial"].aws_iam_role_policy_attachment.this["AmazonEKSWorkerNodePolicy"] module.eks.module.eks_managed_node_group["initial"].aws_iam_role_policy_attachment.this["AmazonEKS_CNI_Policy"] module.eks.module.eks_managed_node_group["initial"].aws_launch_template.this[0] module.eks.module.fargate_profile["fp-profile"].data.aws_caller_identity.current module.eks.module.fargate_profile["fp-profile"].data.aws_iam_policy_document.assume_role_policy[0] module.eks.module.fargate_profile["fp-profile"].data.aws_partition.current module.eks.module.fargate_profile["fp-profile"].data.aws_region.current module.eks.module.fargate_profile["fp-profile"].aws_eks_fargate_profile.this[0] module.eks.module.fargate_profile["fp-profile"].aws_iam_role.this[0] module.eks.module.fargate_profile["fp-profile"].aws_iam_role_policy_attachment.this["AmazonEKSFargatePodExecutionRolePolicy"] module.eks.module.fargate_profile["fp-profile"].aws_iam_role_policy_attachment.this["AmazonEKS_CNI_Policy"] module.eks.module.kms.data.aws_caller_identity.current[0] module.eks.module.kms.data.aws_iam_policy_document.this[0] module.eks.module.kms.data.aws_partition.current[0] module.eks.module.kms.aws_kms_alias.this["cluster"] module.eks.module.kms.aws_kms_key.this[0] module.eks.module.self_managed_node_group["default-selfmng"].data.aws_caller_identity.current module.eks.module.self_managed_node_group["default-selfmng"].data.aws_iam_policy_document.assume_role_policy[0] module.eks.module.self_managed_node_group["default-selfmng"].data.aws_partition.current module.eks.module.self_managed_node_group["default-selfmng"].data.aws_ssm_parameter.ami[0] module.eks.module.self_managed_node_group["default-selfmng"].aws_autoscaling_group.this[0] module.eks.module.self_managed_node_group["default-selfmng"].aws_eks_access_entry.this[0] module.eks.module.self_managed_node_group["default-selfmng"].aws_iam_instance_profile.this[0] module.eks.module.self_managed_node_group["default-selfmng"].aws_iam_role.this[0] module.eks.module.self_managed_node_group["default-selfmng"].aws_iam_role_policy_attachment.this["AmazonEC2ContainerRegistryReadOnly"] module.eks.module.self_managed_node_group["default-selfmng"].aws_iam_role_policy_attachment.this["AmazonEKSWorkerNodePolicy"] module.eks.module.self_managed_node_group["default-selfmng"].aws_iam_role_policy_attachment.this["AmazonEKS_CNI_Policy"] module.eks.module.self_managed_node_group["default-selfmng"].aws_launch_template.this[0] module.eks_blueprints_addons.module.argocd.helm_release.this[0] module.eks_blueprints_addons.module.aws_efs_csi_driver.data.aws_caller_identity.current[0] module.eks_blueprints_addons.module.aws_efs_csi_driver.data.aws_iam_policy_document.assume[0] module.eks_blueprints_addons.module.aws_efs_csi_driver.data.aws_iam_policy_document.this[0] module.eks_blueprints_addons.module.aws_efs_csi_driver.data.aws_partition.current[0] module.eks_blueprints_addons.module.aws_efs_csi_driver.aws_iam_policy.this[0] module.eks_blueprints_addons.module.aws_efs_csi_driver.aws_iam_role.this[0] module.eks_blueprints_addons.module.aws_efs_csi_driver.aws_iam_role_policy_attachment.this[0] module.eks_blueprints_addons.module.aws_efs_csi_driver.helm_release.this[0] module.eks_blueprints_addons.module.aws_load_balancer_controller.data.aws_caller_identity.current[0] module.eks_blueprints_addons.module.aws_load_balancer_controller.data.aws_iam_policy_document.assume[0] module.eks_blueprints_addons.module.aws_load_balancer_controller.data.aws_iam_policy_document.this[0] module.eks_blueprints_addons.module.aws_load_balancer_controller.data.aws_partition.current[0] module.eks_blueprints_addons.module.aws_load_balancer_controller.aws_iam_policy.this[0] module.eks_blueprints_addons.module.aws_load_balancer_controller.aws_iam_role.this[0] module.eks_blueprints_addons.module.aws_load_balancer_controller.aws_iam_role_policy_attachment.this[0] module.eks_blueprints_addons.module.aws_load_balancer_controller.helm_release.this[0] module.eks_blueprints_addons.module.karpenter.data.aws_caller_identity.current[0] module.eks_blueprints_addons.module.karpenter.data.aws_iam_policy_document.assume[0] module.eks_blueprints_addons.module.karpenter.data.aws_iam_policy_document.this[0] module.eks_blueprints_addons.module.karpenter.data.aws_partition.current[0] module.eks_blueprints_addons.module.karpenter.aws_iam_policy.this[0] module.eks_blueprints_addons.module.karpenter.aws_iam_role.this[0] module.eks_blueprints_addons.module.karpenter.aws_iam_role_policy_attachment.this[0] module.eks_blueprints_addons.module.karpenter.helm_release.this[0] module.eks_blueprints_addons.module.karpenter_sqs.data.aws_iam_policy_document.this[0] module.eks_blueprints_addons.module.karpenter_sqs.aws_sqs_queue.this[0] module.eks_blueprints_addons.module.karpenter_sqs.aws_sqs_queue_policy.this[0] module.eks_blueprints_addons.module.metrics_server.helm_release.this[0] module.eks.module.eks_managed_node_group["blue-mng"].module.user_data.null_resource.validate_cluster_service_cidr module.eks.module.eks_managed_node_group["initial"].module.user_data.null_resource.validate_cluster_service_cidr module.eks.module.self_managed_node_group["default-selfmng"].module.user_data.null_resource.validate_cluster_service_cidr # 아래 명령이 정상 수행되었다면 테라폼 스테이트 파일이 잘 적용되것임 ec2-user:~/environment/terraform:$ terraform output configure_kubectl = "aws eks --region us-west-2 update-kubeconfig --name eksworkshop-eksctl" |
- vpc.tf
|
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
version = "~> 5.0"
name = local.name
cidr = local.vpc_cidr
azs = local.azs
private_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 4, k)]
public_subnets = [for k, v in local.azs : cidrsubnet(local.vpc_cidr, 8, k + 48)]
enable_nat_gateway = true
single_nat_gateway = true
public_subnet_tags = {
"kubernetes.io/role/elb" = 1
}
private_subnet_tags = {
"kubernetes.io/role/internal-elb" = 1
# Tags subnets for Karpenter auto-discovery
"karpenter.sh/discovery" = local.name
}
tags = local.tags
}
|
- versions.tf
|
terraform {
required_version = ">= 1.3"
required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 5.34"
}
helm = {
source = "hashicorp/helm"
version = ">= 2.9"
}
kubernetes = {
source = "hashicorp/kubernetes"
version = ">= 2.20"
}
}
# ## Used for end-to-end testing on project; update to suit your needs
# backend "s3" {
# bucket = "terraform-ssp-github-actions-state"
# region = "us-west-2"
# key = "e2e/karpenter/terraform.tfstate"
# }
}
|
- variables.tf : eks 버전(1.25), mng(1.25)
|
variable "cluster_version" {
description = "EKS cluster version."
type = string
default = "1.25"
}
variable "mng_cluster_version" {
description = "EKS cluster mng version."
type = string
default = "1.25"
}
variable "ami_id" {
description = "EKS AMI ID for node groups"
type = string
default = ""
}
|
- outputs.tf
|
output "configure_kubectl" {
description = "Configure kubectl: make sure you're logged in with the correct AWS profile and run the following command to update your kubeconfig"
value = "aws eks --region ${local.region} update-kubeconfig --name ${module.eks.cluster_name}"
}
|
- gitops-setup.tf : aws_codecommit (Private Git Repo, 관리형 소스 제어 서비스)
|
resource "aws_codecommit_repository" "gitops_repo_cc" {
repository_name = "eks-gitops-repo"
default_branch = "main"
}
resource "aws_iam_user" "argocd_user" {
name = "argocd-user"
}
resource "aws_iam_service_specific_credential" "argocd_codecommit_credential" {
service_name = "codecommit.amazonaws.com"
user_name = aws_iam_user.argocd_user.name
}
resource "aws_iam_user_policy" "argocd_user_codecommit_rw" {
name = "argocd-user-codecommit-ro"
user = aws_iam_user.argocd_user.name
policy = jsonencode({
Version : "2012-10-17",
Statement : [
{
Effect : "Allow",
Action : "codecommit:*",
Resource : aws_codecommit_repository.gitops_repo_cc.arn
}
]
})
}
resource "aws_secretsmanager_secret" "argocd_user_creds_secret" {
name = "argocd-user-creds"
}
resource "aws_secretsmanager_secret_version" "argocd_user_creds_secret_version" {
secret_id = aws_secretsmanager_secret.argocd_user_creds_secret.id
secret_string = jsonencode({
url = "${aws_codecommit_repository.gitops_repo_cc.clone_url_http}"
username = "${aws_iam_service_specific_credential.argocd_codecommit_credential.service_user_name}"
password = "${aws_iam_service_specific_credential.argocd_codecommit_credential.service_password}"
})
}
|
- base.tf
|
provider "aws" {
region = local.region
}
# Required for public ECR where Karpenter artifacts are hosted
provider "aws" {
region = "us-east-1"
alias = "virginia"
}
provider "kubernetes" {
host = module.eks.cluster_endpoint
cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
exec {
api_version = "client.authentication.k8s.io/v1beta1"
command = "aws"
# This requires the awscli to be installed locally where Terraform is executed
args = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
}
}
provider "helm" {
kubernetes {
host = module.eks.cluster_endpoint
cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
exec {
api_version = "client.authentication.k8s.io/v1beta1"
command = "aws"
# This requires the awscli to be installed locally where Terraform is executed
args = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
}
}
}
data "aws_partition" "current" {}
data "aws_caller_identity" "current" {}
data "aws_ecrpublic_authorization_token" "token" {
provider = aws.virginia
}
data "aws_availability_zones" "available" {}
# tflint-ignore: terraform_unused_declarations
variable "eks_cluster_id" {
description = "EKS cluster name"
type = string
}
variable "aws_region" {
description = "AWS Region"
type = string
}
locals {
name = var.eks_cluster_id
region = var.aws_region
vpc_cidr = "10.0.0.0/16"
azs = slice(data.aws_availability_zones.available.names, 0, 3)
tags = {
Blueprint = local.name
GithubRepo = "github.com/aws-ia/terraform-aws-eks-blueprints"
}
}
################################################################################
# Cluster
################################################################################
module "eks" {
source = "terraform-aws-modules/eks/aws"
version = "~> 20.14"
cluster_name = local.name
cluster_version = var.cluster_version
cluster_endpoint_public_access = true
vpc_id = module.vpc.vpc_id
subnet_ids = module.vpc.private_subnets
# Fargate profiles use the cluster primary security group so these are not utilized
# create_cluster_security_group = false
# create_node_security_group = false
enable_cluster_creator_admin_permissions = true
fargate_profiles = {
fp-profile = {
selectors = [
{ namespace = "assets" }
]
}
}
eks_managed_node_group_defaults = {
cluster_version = var.mng_cluster_version
}
eks_managed_node_groups = {
initial = {
instance_types = ["m5.large", "m6a.large", "m6i.large"]
min_size = 2
max_size = 10
desired_size = 2
update_config = {
max_unavailable_percentage = 35
}
}
blue-mng={
instance_types = ["m5.large", "m6a.large", "m6i.large"]
cluster_version = "1.25"
min_size = 1
max_size = 2
desired_size = 1
update_config = {
max_unavailable_percentage = 35
}
labels = {
type = "OrdersMNG"
}
subnet_ids = [module.vpc.private_subnets[0]]
taints = [
{
key = "dedicated"
value = "OrdersApp"
effect = "NO_SCHEDULE"
}
]
}
}
# For demonstrating node-termination-handler
self_managed_node_groups = {
default-selfmng = {
instance_type = "m5.large"
min_size = 1
max_size = 2
desired_size = 2
# Additional configurations
ami_id = "ami-0ee947a6f4880da75" # Replace with your desired AMI ID
subnet_ids = module.vpc.private_subnets
disk_size = 100
# Optional
bootstrap_extra_args = "--kubelet-extra-args '--node-labels=node.kubernetes.io/lifecycle=self-managed,team=carts'"
# Required for self-managed node groups
create_launch_template = true
launch_template_use_name_prefix = true
}
}
tags = merge(local.tags, {
# NOTE - if creating multiple security groups with this module, only tag the
# security group that Karpenter should utilize with the following tag
# (i.e. - at most, only one security group should have this tag in your account)
"karpenter.sh/discovery" = local.name
})
}
resource "time_sleep" "wait_60_seconds" {
create_duration = "60s"
depends_on = [module.eks]
}
|
- backend_override.tf : 본인 소유의 s3 정보 추가해서 사용 할 것
|
terraform {
backend "s3" {
bucket = "workshop-stack-tfstatebackendbucketf0fc9a9d-qll9it5dhvaw"
region = "us-west-2"
key = "terraform.tfstate"
}
}
|
- addons.tf
|
################################################################################
# EKS Blueprints Addons
################################################################################
module "eks_blueprints_addons" {
depends_on = [ time_sleep.wait_60_seconds ]
source = "aws-ia/eks-blueprints-addons/aws"
version = "~> 1.16"
cluster_name = module.eks.cluster_name
cluster_endpoint = module.eks.cluster_endpoint
cluster_version = module.eks.cluster_version
oidc_provider_arn = module.eks.oidc_provider_arn
# We want to wait for the Fargate profiles to be deployed first
create_delay_dependencies = [for prof in module.eks.fargate_profiles : prof.fargate_profile_arn]
eks_addons = {
coredns = {
addon_version = "v1.8.7-eksbuild.10"
}
kube-proxy = {
addon_version = "v1.25.16-eksbuild.8"
}
vpc-cni = {
most_recent = true
}
aws-ebs-csi-driver = {
service_account_role_arn = module.ebs_csi_driver_irsa.iam_role_arn
}
}
enable_karpenter = true
enable_aws_efs_csi_driver = true
enable_argocd = true
enable_aws_load_balancer_controller = true
enable_metrics_server = true
argocd = {
set = [
{
name = "server.service.type"
value = "LoadBalancer"
},
{
name = "server.service.annotations.service\\.beta\\.kubernetes\\.io/aws-load-balancer-scheme"
value = "internet-facing"
},
{
name = "server.service.annotations.service\\.beta\\.kubernetes\\.io/aws-load-balancer-type"
value = "external"
},
{
name = "server.service.annotations.service\\.beta\\.kubernetes\\.io/aws-load-balancer-nlb-target-type"
value = "ip"
}
]
wait = true
}
aws_load_balancer_controller = {
set = [
{
name = "vpcId"
value = module.vpc.vpc_id
},
{
name = "region"
value = local.region
},
{
name = "podDisruptionBudget.maxUnavailable"
value = 1
},
{
name = "enableServiceMutatorWebhook"
value = "false"
}
]
wait = true
}
karpenter_node = {
# Use static name so that it matches what is defined in `karpenter.yaml` example manifest
iam_role_use_name_prefix = false
}
tags = local.tags
}
resource "time_sleep" "wait_90_seconds" {
create_duration = "90s"
depends_on = [module.eks_blueprints_addons]
}
resource "aws_eks_access_entry" "karpenter_node_access_entry" {
cluster_name = module.eks.cluster_name
principal_arn = module.eks_blueprints_addons.karpenter.node_iam_role_arn
# kubernetes_groups = []
type = "EC2_LINUX"
lifecycle {
ignore_changes = [
user_name
]
}
}
################################################################################
# Supporting Resources
################################################################################
module "ebs_csi_driver_irsa" {
source = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
version = "~> 5.20"
role_name_prefix = "${module.eks.cluster_name}-ebs-csi-driver-"
attach_ebs_csi_policy = true
oidc_providers = {
main = {
provider_arn = module.eks.oidc_provider_arn
namespace_service_accounts = ["kube-system:ebs-csi-controller-sa"]
}
}
tags = local.tags
}
module "efs" {
source = "terraform-aws-modules/efs/aws"
version = "~> 1.1"
creation_token = local.name
name = local.name
# Mount targets / security group
mount_targets = {
for k, v in zipmap(local.azs, module.vpc.private_subnets) : k => { subnet_id = v }
}
security_group_description = "${local.name} EFS security group"
security_group_vpc_id = module.vpc.vpc_id
security_group_rules = {
vpc = {
# relying on the defaults provided for EFS/NFS (2049/TCP + ingress)
description = "NFS ingress from VPC private subnets"
cidr_blocks = module.vpc.private_subnets_cidr_blocks
}
}
tags = local.tags
}
################################################################################
# Storage Classes
################################################################################
resource "kubernetes_annotations" "gp2" {
api_version = "storage.k8s.io/v1"
kind = "StorageClass"
# This is true because the resources was already created by the ebs-csi-driver addon
force = "true"
metadata {
name = "gp2"
}
annotations = {
# Modify annotations to remove gp2 as default storage class still retain the class
"storageclass.kubernetes.io/is-default-class" = "false"
}
depends_on = [
module.eks_blueprints_addons
]
}
resource "kubernetes_storage_class_v1" "gp3" {
metadata {
name = "gp3"
annotations = {
# Annotation to set gp3 as default storage class
"storageclass.kubernetes.io/is-default-class" = "true"
}
}
storage_provisioner = "ebs.csi.aws.com"
allow_volume_expansion = true
reclaim_policy = "Delete"
volume_binding_mode = "WaitForFirstConsumer"
parameters = {
encrypted = true
fsType = "ext4"
type = "gp3"
}
depends_on = [
module.eks_blueprints_addons
]
}
# done: update parameters
resource "kubernetes_storage_class_v1" "efs" {
metadata {
name = "efs"
}
storage_provisioner = "efs.csi.aws.com"
reclaim_policy = "Delete"
parameters = {
provisioningMode = "efs-ap"
fileSystemId = module.efs.id
directoryPerms = "755"
gidRangeStart = "100" # optional
gidRangeEnd = "200" # optional
basePath = "/dynamic_provisioning" # optional
subPathPattern = "$${.PVC.namespace}/$${.PVC.name}" # optional
ensureUniqueDirectory = "false" # optional
reuseAccessPoint = "false" # optional
}
mount_options = [
"iam"
]
depends_on = [
module.eks_blueprints_addons
]
}
|
샘플어플리케이션 구성
- - 샘플 애플리케이션은 고객이 카탈로그를 탐색하고 장바구니에 항목을 추가하며 결제 프로세스를 통해 주문을 완료할 수 있는 간단한 웹 스토어 애플리케이션
- 애플리케이션에는 여러 구성 요소와 종속성이 있습니다:
|
Component
|
Description
|
|
UI
|
Provides the front end user interface and aggregates API calls to the various other services.
|
|
Catalog
|
API for product listings and details
|
|
Cart
|
API for customer shopping carts
|
|
Checkout
|
API to orchestrate the checkout process
|
|
Orders
|
API to receive and process customer orders
|
|
Static assets
|
Serves static assets like images related to the product catalog
|
- 이미지 도커파일과, ECR 공개 저장소 정보 - [Link](https://www.eksworkshop.com/docs/introduction/getting-started/packaging-application)
- 처음에는 로드 밸런서나 관리형 데이터베이스와 같은 AWS 서비스를 사용하지 않고 Amazon EKS 클러스터에 자체적으로 포함된 방식으로 애플리케이션을 배포할 것입니다.
- 실험 과정에서 EKS의 다양한 기능을 활용하여 소매점의 광범위한 AWS 서비스와 기능을 활용할 것입니다.
- 모든 구성 요소는 ArgoCD를 사용하여 EKS 클러스터에 배포됩니다.
- ArgoCD의 Config Repository 는 CodeCommit을 사용합니다. AWS CodeCommit 저장소를 GitOps repo로 사용하여 IDE에 복제할 수 있도록 했습니다.
| ec2-user:~/environment/terraform:$ cd ~/environment ec2-user:~/environment:$ git clone codecommit::${REGION}://eks-gitops-repo Cloning into 'eks-gitops-repo'... hint: Using 'master' as the name for the initial branch. This default branch name hint: is subject to change. To configure the initial branch name to use in all hint: of your new repositories, which will suppress this warning, call: hint: hint: git config --global init.defaultBranch <name> hint: hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and hint: 'development'. The just-created branch can be renamed via this command: hint: hint: git branch -m <name> remote: Counting objects: 77, done. Unpacking objects: 100% (77/77), 14.09 KiB | 262.00 KiB/s, done. ec2-user:~/environment:$ sudo yum install tree -y ec2-user:~/environment:$ tree eks-gitops-repo/ -L 2 eks-gitops-repo/ ├── app-of-apps │ ├── Chart.yaml │ ├── templates │ └── values.yaml └── apps ├── assets ├── carts ├── catalog ├── checkout ├── karpenter ├── kustomization.yaml ├── orders ├── other ├── rabbitmq └── ui 12 directories, 3 files  # ArgoCD는 NLB로 배포해 놓았으니, 접속정보,계정, 암호를 확인하고 웹 접속 ec2-user:~:$ export ARGOCD_SERVER=$(kubectl get svc argo-cd-argocd-server -n argocd -o json | jq --raw-output '.status.loadBalancer.ingress[0].hostname') ec2-user:~:$ echo "ArgoCD URL: http://${ARGOCD_SERVER}" ArgoCD URL: http://k8s-argocd-argocdar-06bc31bc9d-9d58f820cb2beb72.elb.us-west-2.amazonaws.com ec2-user:~:$ export ARGOCD_USER="admin" ec2-user:~:$ export ARGOCD_PWD=$(kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d) ec2-user:~:$ echo "Username: ${ARGOCD_USER}" Username: admin ec2-user:~:$ echo "Password: ${ARGOCD_PWD}" Password: 0Gz7LLeu4gvpP9Tp   # Config Repository는 AWS CodeCommit 주소로 설정이 되어 있습니다.  # 어플리케이션은 app-aof-apps 방식으로 배포되어 있습니다.  |
Argo CD CLI 확인
- argocd를 웹이 아닌 CLI 모드 설정을 하고 정보를 조회해 보겠습니다.
| ec2-user:~:$ argocd login ${ARGOCD_SERVER} --username ${ARGOCD_USER} --password ${ARGOCD_PWD} --insecure --skip-test-tls --grpc-web 'admin:login' logged in successfully Context 'k8s-argocd-argocdar-06bc31bc9d-9d58f820cb2beb72.elb.us-west-2.amazonaws.com' updated ec2-user:~:$ argocd repo list TYPE NAME REPO INSECURE OCI LFS CREDS STATUS MESSAGE PROJECT git https://git-codecommit.us-west-2.amazonaws.com/v1/repos/eks-gitops-repofalse false false true Successful ec2-user:~:$ ec2-user:~:$ argocd app list NAME CLUSTER NAMESPACE PROJECT STATUS HEALTH SYNCPOLICY CONDITIONS REPO PATH TARGET argocd/apps https://kubernetes.default.svcdefault Synced Healthy Auto <none> https://git-codecommit.us-west-2.amazonaws.com/v1/repos/eks-gitops-repoapp-of-apps argocd/assets https://kubernetes.default.svcdefault Synced Healthy Auto-Prune <none> https://git-codecommit.us-west-2.amazonaws.com/v1/repos/eks-gitops-repo apps/assets main argocd/carts https://kubernetes.default.svc default Synced Healthy Auto-Prune <none> https://git-codecommit.us-west-2.amazonaws.com/v1/repos/eks-gitops-repo apps/carts main argocd/catalog https://kubernetes.default.svc default Synced Healthy Auto-Prune <none> https://git-codecommit.us-west-2.amazonaws.com/v1/repos/eks-gitops-repo apps/catalog main argocd/checkout https://kubernetes.default.svc default Synced Healthy Auto-Prune <none> https://git-codecommit.us-west-2.amazonaws.com/v1/repos/eks-gitops-repo apps/checkout main argocd/karpenter https://kubernetes.default.svc default Synced Healthy Auto-Prune <none> https://git-codecommit.us-west-2.amazonaws.com/v1/repos/eks-gitops-repo apps/karpenter main argocd/orders https://kubernetes.default.svc default Synced Healthy Auto-Prune <none> https://git-codecommit.us-west-2.amazonaws.com/v1/repos/eks-gitops-repo apps/orders main argocd/other https://kubernetes.default.svc default Synced Healthy Auto-Prune <none> https://git-codecommit.us-west-2.amazonaws.com/v1/repos/eks-gitops-repo apps/other main argocd/rabbitmq https://kubernetes.default.svc default Synced Healthy Auto-Prune <none> https://git-codecommit.us-west-2.amazonaws.com/v1/repos/eks-gitops-repo apps/rabbitmq main argocd/ui https://kubernetes.default.svc default OutOfSync Healthy Auto-Prune <none> https://git-codecommit.us-west-2.amazonaws.com/v1/repos/eks-gitops-repo apps/ui main ec2-user:~:$ argocd app get apps Name: argocd/apps Project: default Server: https://kubernetes.default.svc Namespace: URL: https://k8s-argocd-argocdar-06bc31bc9d-9d58f820cb2beb72.elb.us-west-2.amazonaws.com/applications/apps Source: - Repo: https://git-codecommit.us-west-2.amazonaws.com/v1/repos/eks-gitops-repo Target: Path: app-of-apps SyncWindow: Sync Allowed Sync Policy: Automated Sync Status: Synced to (d323cec) Health Status: Healthy GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE argoproj.io Application argocd karpenter Synced application.argoproj.io/karpenter created argoproj.io Application argocd assets Synced application.argoproj.io/assets created argoproj.io Application argocd catalog Synced application.argoproj.io/catalog created argoproj.io Application argocd carts Synced application.argoproj.io/carts created argoproj.io Application argocd rabbitmq Synced application.argoproj.io/rabbitmq created argoproj.io Application argocd orders Synced application.argoproj.io/orders created argoproj.io Application argocd other Synced application.argoproj.io/other created argoproj.io Application argocd checkout Synced application.argoproj.io/checkout created argoproj.io Application argocd ui Synced application.argoproj.io/ui created ec2-user:~:$ argocd app get carts Name: argocd/carts Project: default Server: https://kubernetes.default.svc Namespace: URL: https://k8s-argocd-argocdar-06bc31bc9d-9d58f820cb2beb72.elb.us-west-2.amazonaws.com/applications/carts Source: - Repo: https://git-codecommit.us-west-2.amazonaws.com/v1/repos/eks-gitops-repo Target: main Path: apps/carts SyncWindow: Sync Allowed Sync Policy: Automated (Prune) Sync Status: Synced to main (d323cec) Health Status: Healthy GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE Namespace carts Synced namespace/carts created ServiceAccount carts carts Synced serviceaccount/carts created ConfigMap carts carts Synced configmap/carts created Service carts carts-dynamodb Synced Healthy service/carts-dynamodb created Service carts carts Synced Healthy service/carts created apps Deployment carts carts-dynamodb Synced Healthy deployment.apps/carts-dynamodb created apps Deployment carts carts Synced Healthy deployment.apps/carts created ec2-user:~:$ argocd app get carts -o yaml metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"argoproj.io/v1alpha1","kind":"Application","metadata":{"annotations":{},"finalizers":["resources-finalizer.argocd.argoproj.io"],"labels":{"argocd.argoproj.io/instance":"apps"},"name":"carts","namespace":"argocd"},"spec":{"destination":{"server":" https://kubernetes.default.svc "},"ignoreDifferences":[{"group":"apps","jsonPointers":["/spec/replicas","/metadata/annotations/deployment.kubernetes.io/revision"],"kind":"Deployment"},{"group":"autoscaling","jsonPointers":["/status"],"kind":"HorizontalPodAutoscaler"}],"project":"default","source":{"path":"apps/carts","repoURL":" https://git-codecommit.us-west-2.amazonaws.com/v1/repos/eks-gitops-repo ","targetRevision":"main"},"syncPolicy":{"automated":{"prune":true,"selfHeal":true},"syncOptions":["RespectIgnoreDifferences=true"]}}}
creationTimestamp: "2025-03-30T05:57:26Z"
finalizers:
- resources-finalizer.argocd.argoproj.io
generation: 1176
labels:
argocd.argoproj.io/instance: apps
managedFields:
- apiVersion: argoproj.io/v1alpha1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:finalizers:
.: {}
v:"resources-finalizer.argocd.argoproj.io": {}
f:labels:
.: {}
f:argocd.argoproj.io/instance: {}
f:spec:
.: {}
f:destination:
.: {}
f:server: {}
f:ignoreDifferences: {}
f:project: {}
f:source:
.: {}
f:path: {}
f:repoURL: {}
f:targetRevision: {}
f:syncPolicy:
.: {}
f:automated:
.: {}
f:prune: {}
f:selfHeal: {}
f:syncOptions: {}
manager: argocd-controller
operation: Update
time: "2025-03-30T05:57:26Z"
- apiVersion: argoproj.io/v1alpha1
fieldsType: FieldsV1
fieldsV1:
f:status:
.: {}
f:controllerNamespace: {}
f:health:
.: {}
f:status: {}
f:history: {}
f:operationState:
.: {}
f:finishedAt: {}
f:message: {}
f:operation:
.: {}
f:initiatedBy:
.: {}
f:automated: {}
f:retry:
.: {}
f:limit: {}
f:sync:
.: {}
f:prune: {}
f:revision: {}
f:syncOptions: {}
f:phase: {}
f:startedAt: {}
f:syncResult:
.: {}
f:resources: {}
f:revision: {}
f:source:
.: {}
f:path: {}
f:repoURL: {}
f:targetRevision: {}
f:reconciledAt: {}
f:resources: {}
f:sourceType: {}
f:summary:
.: {}
f:images: {}
f:sync:
.: {}
f:comparedTo:
.: {}
f:destination:
.: {}
f:server: {}
f:ignoreDifferences: {}
f:source:
.: {}
f:path: {}
f:repoURL: {}
f:targetRevision: {}
f:revision: {}
f:status: {}
manager: argocd-application-controller
operation: Update
time: "2025-04-01T16:07:17Z"
name: carts
namespace: argocd
resourceVersion: "1523958"
uid: 2c8e8f53-b555-4f9c-a4b2-412affd7b674
spec:
destination:
ignoreDifferences:
- group: apps
jsonPointers:
- /spec/replicas
- /metadata/annotations/deployment.kubernetes.io/revision
kind: Deployment
- group: autoscaling
jsonPointers:
- /status
kind: HorizontalPodAutoscaler
project: default
source:
path: apps/carts
targetRevision: main
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- RespectIgnoreDifferences=true
status:
controllerNamespace: argocd
health:
status: Healthy
history:
- deployStartedAt: "2025-03-30T05:57:27Z"
deployedAt: "2025-03-30T05:57:28Z"
id: 0
initiatedBy: {}
revision: d323cec2bf59360911a6867b7658b15cb21ead40
source:
path: apps/carts
targetRevision: main
operationState:
finishedAt: "2025-03-30T05:57:28Z"
message: successfully synced (all tasks run)
operation:
initiatedBy:
automated: true
retry:
limit: 5
sync:
prune: true
revision: d323cec2bf59360911a6867b7658b15cb21ead40
syncOptions:
- RespectIgnoreDifferences=true
phase: Succeeded
startedAt: "2025-03-30T05:57:27Z"
syncResult:
resources:
- group: ""
hookPhase: Running
kind: Namespace
message: namespace/carts created
name: carts
namespace: ""
status: Synced
syncPhase: Sync
version: v1
- group: ""
hookPhase: Running
kind: ServiceAccount
message: serviceaccount/carts created
name: carts
namespace: carts
status: Synced
syncPhase: Sync
version: v1
- group: ""
hookPhase: Running
kind: ConfigMap
message: configmap/carts created
name: carts
namespace: carts
status: Synced
syncPhase: Sync
version: v1
- group: ""
hookPhase: Running
kind: Service
message: service/carts-dynamodb created
name: carts-dynamodb
namespace: carts
status: Synced
syncPhase: Sync
version: v1
- group: ""
hookPhase: Running
kind: Service
message: service/carts created
name: carts
namespace: carts
status: Synced
syncPhase: Sync
version: v1
- group: apps
hookPhase: Running
kind: Deployment
message: deployment.apps/carts-dynamodb created
name: carts-dynamodb
namespace: carts
status: Synced
syncPhase: Sync
version: v1
- group: apps
hookPhase: Running
kind: Deployment
message: deployment.apps/carts created
name: carts
namespace: carts
status: Synced
syncPhase: Sync
version: v1
revision: d323cec2bf59360911a6867b7658b15cb21ead40
source:
path: apps/carts
targetRevision: main
reconciledAt: "2025-04-01T16:07:17Z"
resources:
- kind: ConfigMap
name: carts
namespace: carts
status: Synced
version: v1
- kind: Namespace
name: carts
status: Synced
version: v1
- health:
status: Healthy
kind: Service
name: carts
namespace: carts
status: Synced
version: v1
- health:
status: Healthy
kind: Service
name: carts-dynamodb
namespace: carts
status: Synced
version: v1
- kind: ServiceAccount
name: carts
namespace: carts
status: Synced
version: v1
- group: apps
health:
status: Healthy
kind: Deployment
name: carts
namespace: carts
status: Synced
version: v1
- group: apps
health:
status: Healthy
kind: Deployment
name: carts-dynamodb
namespace: carts
status: Synced
version: v1
sourceHydrator: {}
sourceType: Kustomize
summary:
images:
- amazon/dynamodb-local:1.13.1
- public.ecr.aws/aws-containers/retail-store-sample-cart:0.7.0
sync:
comparedTo:
destination:
ignoreDifferences:
- group: apps
jsonPointers:
- /spec/replicas
- /metadata/annotations/deployment.kubernetes.io/revision
kind: Deployment
- group: autoscaling
jsonPointers:
- /status
kind: HorizontalPodAutoscaler
source:
path: apps/carts
targetRevision: main
revision: d323cec2bf59360911a6867b7658b15cb21ead40
status: Synced
ec2-user:~:$ argocd app get ui -o yaml metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"argoproj.io/v1alpha1","kind":"Application","metadata":{"annotations":{},"finalizers":["resources-finalizer.argocd.argoproj.io"],"labels":{"argocd.argoproj.io/instance":"apps"},"name":"ui","namespace":"argocd"},"spec":{"destination":{"server":" https://kubernetes.default.svc "},"ignoreDifferences":[{"group":"apps","jsonPointers":["/spec/replicas","/metadata/annotations/deployment.kubernetes.io/revision"],"kind":"Deployment"},{"group":"autoscaling","jsonPointers":["/status"],"kind":"HorizontalPodAutoscaler"}],"project":"default","source":{"path":"apps/ui","repoURL":" https://git-codecommit.us-west-2.amazonaws.com/v1/repos/eks-gitops-repo ","targetRevision":"main"},"syncPolicy":{"automated":{"prune":true,"selfHeal":true},"syncOptions":["RespectIgnoreDifferences=true"]}}}
creationTimestamp: "2025-03-30T05:57:26Z"
finalizers:
- resources-finalizer.argocd.argoproj.io
generation: 167761
labels:
argocd.argoproj.io/instance: apps
managedFields:
- apiVersion: argoproj.io/v1alpha1
fieldsType: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.: {}
f:kubectl.kubernetes.io/last-applied-configuration: {}
f:finalizers:
.: {}
v:"resources-finalizer.argocd.argoproj.io": {}
f:labels:
.: {}
f:argocd.argoproj.io/instance: {}
f:spec:
.: {}
f:destination:
.: {}
f:server: {}
f:ignoreDifferences: {}
f:project: {}
f:source:
.: {}
f:path: {}
f:repoURL: {}
f:targetRevision: {}
f:syncPolicy:
.: {}
f:automated:
.: {}
f:prune: {}
f:selfHeal: {}
f:syncOptions: {}
manager: argocd-controller
operation: Update
time: "2025-03-30T05:57:26Z"
- apiVersion: argoproj.io/v1alpha1
fieldsType: FieldsV1
fieldsV1:
f:status:
.: {}
f:controllerNamespace: {}
f:health:
.: {}
f:status: {}
f:history: {}
f:operationState:
.: {}
f:finishedAt: {}
f:message: {}
f:operation:
.: {}
f:initiatedBy:
.: {}
f:automated: {}
f:retry:
.: {}
f:limit: {}
f:sync:
.: {}
f:prune: {}
f:resources: {}
f:revision: {}
f:syncOptions: {}
f:phase: {}
f:startedAt: {}
f:syncResult:
.: {}
f:resources: {}
f:revision: {}
f:source:
.: {}
f:path: {}
f:repoURL: {}
f:targetRevision: {}
f:reconciledAt: {}
f:resources: {}
f:sourceType: {}
f:summary:
.: {}
f:images: {}
f:sync:
.: {}
f:comparedTo:
.: {}
f:destination:
.: {}
f:server: {}
f:ignoreDifferences: {}
f:source:
.: {}
f:path: {}
f:repoURL: {}
f:targetRevision: {}
f:revision: {}
f:status: {}
manager: argocd-application-controller
operation: Update
time: "2025-04-01T16:08:14Z"
name: ui
namespace: argocd
resourceVersion: "1524370"
uid: 98959ec8-2958-497d-bb55-88f8279350fd
spec:
destination:
ignoreDifferences:
- group: apps
jsonPointers:
- /spec/replicas
- /metadata/annotations/deployment.kubernetes.io/revision
kind: Deployment
- group: autoscaling
jsonPointers:
- /status
kind: HorizontalPodAutoscaler
project: default
source:
path: apps/ui
targetRevision: main
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- RespectIgnoreDifferences=true
status:
controllerNamespace: argocd
health:
status: Healthy
history:
- deployStartedAt: "2025-03-30T05:57:27Z"
deployedAt: "2025-03-30T05:57:28Z"
id: 0
initiatedBy: {}
revision: d323cec2bf59360911a6867b7658b15cb21ead40
source:
path: apps/ui
targetRevision: main
operationState:
finishedAt: "2025-04-01T16:08:14Z"
message: successfully synced (all tasks run)
operation:
initiatedBy:
automated: true
retry:
limit: 5
sync:
prune: true
resources:
- group: autoscaling
kind: HorizontalPodAutoscaler
name: ui
revision: d323cec2bf59360911a6867b7658b15cb21ead40
syncOptions:
- RespectIgnoreDifferences=true
phase: Succeeded
startedAt: "2025-04-01T16:08:14Z"
syncResult:
resources:
- group: autoscaling
hookPhase: Running
kind: HorizontalPodAutoscaler
message: horizontalpodautoscaler.autoscaling/ui configured
name: ui
namespace: ui
status: Synced
syncPhase: Sync
version: v2beta2
revision: d323cec2bf59360911a6867b7658b15cb21ead40
source:
path: apps/ui
targetRevision: main
reconciledAt: "2025-04-01T16:08:14Z"
resources:
- kind: ConfigMap
name: ui
namespace: ui
status: Synced
version: v1
- kind: Namespace
name: ui
status: Synced
version: v1
- health:
status: Healthy
kind: Service
name: ui
namespace: ui
status: Synced
version: v1
- kind: ServiceAccount
name: ui
namespace: ui
status: Synced
version: v1
- group: apps
health:
status: Healthy
kind: Deployment
name: ui
namespace: ui
status: Synced
version: v1
- group: autoscaling
health:
message: recommended size matches current size
status: Healthy
kind: HorizontalPodAutoscaler
name: ui
namespace: ui
status: OutOfSync
version: v2beta2
sourceHydrator: {}
sourceType: Kustomize
summary:
images:
- public.ecr.aws/aws-containers/retail-store-sample-ui:0.4.0
sync:
comparedTo:
destination:
ignoreDifferences:
- group: apps
jsonPointers:
- /spec/replicas
- /metadata/annotations/deployment.kubernetes.io/revision
kind: Deployment
- group: autoscaling
jsonPointers:
- /status
kind: HorizontalPodAutoscaler
source:
path: apps/ui
targetRevision: main
revision: d323cec2bf59360911a6867b7658b15cb21ead40
status: OutOfSync
ec2-user:~:$ # 현재 ui 앱은 Sync가 맞지 않은 상태입니다.  |
(추가) UI 접속을 위한 NLB 설정
- 현재배포된 샘플 앱의 ui 서비스는 CLUSTER-IP 타입이라서 외부 접속이 되지 않는 상태입니다.
| ec2-user:~:$ k -n ui get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ui ClusterIP 172.20.217.17 <none> 80/TCP 2d10h |
- 실습 편의를 위해 NLB를 작성하여 외부에서 접속이 가능하도록 하겟습니다.
| ec2-user:~:$ cat << EOF > ~/environment/eks-gitops-repo/apps/ui/service-nlb.yaml apiVersion: v1 kind: Service metadata: annotations: service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing service.beta.kubernetes.io/aws-load-balancer-type: external labels: app.kubernetes.io/instance: ui app.kubernetes.io/name: ui name: ui-nlb namespace: ui spec: type: LoadBalancer ports: - name: http port: 80 protocol: TCP targetPort: 8080 selector: app.kubernetes.io/instance: ui app.kubernetes.io/name: ui EOF ec2-user:~:$ cat << EOF > ~/environment/eks-gitops-repo/apps/ui/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization namespace: ui resources: - namespace.yaml - configMap.yaml - serviceAccount.yaml - service.yaml - deployment.yaml - hpa.yaml EOF # Config Repository에 PUSH 한 후 Sync 작업 수행 ec2-user:~:$ cd ~/environment/eks-gitops-repo/ ec2-user:~/environment/eks-gitops-repo:$ git add apps/ui/service-nlb.yaml apps/ui/kustomization.yaml ec2-user:~/environment/eks-gitops-repo:$ git commit -m "Add to ui nlb" [main 6a85931] Add to ui nlb Committer: EC2 Default User <ec2-user@ip-192-168-0-227.us-west-2.compute.internal> Your name and email address were configured automatically based on your username and hostname. Please check that they are accurate. You can suppress this message by setting them explicitly. Run the following command and follow the instructions in your editor to edit your configuration file: git config --global --edit After doing this, you may fix the identity used for this commit with: git commit --amend --reset-author 2 files changed, 23 insertions(+) create mode 100644 apps/ui/service-nlb.yaml ec2-user:~/environment/eks-gitops-repo:$ git push Enumerating objects: 10, done. Counting objects: 100% (10/10), done. Delta compression using up to 2 threads Compressing objects: 100% (6/6), done. Writing objects: 100% (6/6), 734 bytes | 122.00 KiB/s, done. Total 6 (delta 3), reused 0 (delta 0), pack-reused 0 (from 0) remote: Validating objects: 100% To codecommit::us-west-2://eks-gitops-repo d323cec..6a85931 main -> main ec2-user:~/environment/eks-gitops-repo:$ argocd app sync ui TIMESTAMP GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE 2025-04-01T16:17:56+00:00 ConfigMap ui ui Synced 2025-04-01T16:17:56+00:00 Namespace ui Synced 2025-04-01T16:17:56+00:00 Service ui ui Synced Healthy 2025-04-01T16:17:56+00:00 ServiceAccount ui ui Synced 2025-04-01T16:17:56+00:00 apps Deployment ui ui Synced Healthy 2025-04-01T16:17:56+00:00 autoscaling HorizontalPodAutoscaler ui ui OutOfSync Healthy 2025-04-01T16:17:58+00:00 Service ui ui-nlb OutOfSync Progressing 2025-04-01T16:17:58+00:00 Namespace ui Synced namespace/ui unchanged 2025-04-01T16:17:58+00:00 ServiceAccount ui ui Synced serviceaccount/ui unchanged 2025-04-01T16:17:58+00:00 ConfigMap ui ui Synced configmap/ui unchanged 2025-04-01T16:17:58+00:00 Service ui ui Synced Healthy service/ui unchanged 2025-04-01T16:17:58+00:00 Service ui ui-nlb OutOfSync Progressing service/ui-nlb created 2025-04-01T16:17:58+00:00 apps Deployment ui ui Synced Healthy deployment.apps/ui configured 2025-04-01T16:17:58+00:00 autoscaling HorizontalPodAutoscaler ui ui OutOfSync Healthy horizontalpodautoscaler.autoscaling/ui configured 2025-04-01T16:17:59+00:00 Service ui ui-nlb OutOfSync Progressing pruned Name: argocd/ui Project: default Server: https://kubernetes.default.svc Namespace: URL: https://k8s-argocd-argocdar-06bc31bc9d-9d58f820cb2beb72.elb.us-west-2.amazonaws.com/applications/ui Source: - Repo: https://git-codecommit.us-west-2.amazonaws.com/v1/repos/eks-gitops-repo Target: main Path: apps/ui SyncWindow: Sync Allowed Sync Policy: Automated (Prune) Sync Status: OutOfSync from main (6a85931) Health Status: Progressing GROUP KIND NAMESPACE NAME STATUS HEALTH HOOK MESSAGE ConfigMap ui ui Synced Namespace ui Synced Service ui ui Synced Healthy Service ui ui-nlb Synced Progressing ServiceAccount ui ui Synced apps Deployment ui ui Synced Healthy autoscaling HorizontalPodAutoscaler ui ui OutOfSync Healthy  ec2-user:~/environment/eks-gitops-repo:$ # UI 접속 URL 확인 (1.5, 1.3 배율) ec2-user:~/environment/eks-gitops-repo:$ kubectl get svc -n ui ui-nlb -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' | awk '{ print "UI URL = http://"$1""}' UI URL = http://k8s-ui-uinlb-5942f84482-4eaa4a50ed46a007.elb.us-west-2.amazonaws.com  |
In-place Cluster Upgrade : 1.25 → 1.26 : 총 1시간 (예상) 소요 이상 실습 포함
[1. 제어부] Control Plane Upgrade - 약 10분 소요 실습 포함
- Amazon EKS에서 클러스터를 실행하는 가장 큰 이점 중 하나는 클러스터 제어 평면의 업그레이드가 단일 완전 자동화된 작업이라는 것입니다. 업그레이드하는 동안 현재 제어 평면 구성 요소는 파란색/녹색 방식으로 업그레이드됩니다. 문제가 발생하면 업그레이드가 롤백되고 애플리케이션은 계속 작동하고 사용 가능합니다.
- 온플레미스에 비해 AWS가 많은 부분을 순차적으로 업데이트 및 롤백을 진행하여 주기 때문에 편하게 작업이 가능합니다.
- 4가지 방법으로 업그레이드가 가능하나, 이 워크샵에서는 테라폼을 이용한 업그레이드로 실습을 진행합니다.
방법 1 : eksctl
이 명령에서 대상 버전을 지정할 수 있지만 --version에 허용되는 값은 클러스터의 현재 버전 또는 한 버전 더 높은 버전입니다. 현재 두 개 이상의 Kubernetes 버전 업그레이드는 지원되지 않습니다.
| eksctl upgrade cluster --name $EKS_CLUSTER_NAME --approve |
방법2 : AWS 관리콘솔 이용
방법3 : AWS CLI 이용
- 클러스터에 관리되는 노드 그룹이 연결되어 있는 경우, 클러스터를 새로운 Kubernetes 버전으로 업데이트하려면 모든 노드 그룹의 Kubernetes 버전이 클러스터의 Kubernetes 버전과 일치해야 합니다.
| aws eks update-cluster-version --region ${AWS_REGION} --name $EKS_CLUSTER_NAME --kubernetes-version 1.26 { "update": { "id": "b5f0ba18-9a87-4450-b5a0-825e6e84496f", "status": "InProgress", "type": "VersionUpdate", "params": [ { "type": "Version", "value": "1.26" }, { "type": "PlatformVersion", "value": "eks.1" } ], [...] "errors": [] } } # 다음 명령으로 클러스터 업데이트 상태를 모니터링할 수 있습니다. 이전 명령에서 반환한 클러스터 이름과 업데이트 ID를 사용합니다. # Successful 상태가 표시되면 업데이트가 완료된 것입니다. aws eks describe-update --region ${AWS_REGION} --name $EKS_CLUSTER_NAME --update-id b5f0ba18-9a87-4450-b5a0-825e6e84496f |
방법4 : Terraform 이용
- 이 랩에서는 terraform을 사용하여 클러스터를 업그레이드합니다. EKS 클러스터는 이미 terraform을 통해 이 랩에 프로비저닝되었습니다. terraform init, plan 및 apply를 수행하여 리소스 상태를 새로 고칩니다.
- 업그레이드 전에 모니터링 작업을 걸어 놓겠습니다.
| ec2-user:~:$ cd ~/environment/terraform # 현재 버전 확인 : 파일로 저장해두기 ec2-user:~/environment/terraform:$ kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].image}" | tr -s '[[:space:]]' '\n' | sort | uniq -c > 1.25.txt ec2-user:~/environment/terraform:$ kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].image}" | tr -s '[[:space:]]' '\n' | sort | uniq -c 6 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.19.3-eksbuild.1 6 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-network-policy-agent:v1.2.0-eksbuild.1 8 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/aws-ebs-csi-driver:v1.41.0 2 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/coredns:v1.8.7-eksbuild.10 2 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-attacher:v4.8.1-eks-1-32-7 6 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-node-driver-registrar:v2.13.0-eks-1-32-7 2 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-provisioner:v5.2.0-eks-1-32-7 2 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-resizer:v1.13.2-eks-1-32-7 2 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-snapshotter:v8.2.1-eks-1-32-7 6 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/kube-proxy:v1.25.16-minimal-eksbuild.8 8 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/livenessprobe:v2.14.0-eks-1-32-7 8 amazon/aws-efs-csi-driver:v1.7.6 1 amazon/dynamodb-local:1.13.1 1 ghcr.io/dexidp/dex:v2.38.0 1 hjacobs/kube-ops-view:20.4.0 1 public.ecr.aws/aws-containers/retail-store-sample-assets:0.4.0 1 public.ecr.aws/aws-containers/retail-store-sample-cart:0.7.0 1 public.ecr.aws/aws-containers/retail-store-sample-catalog:0.4.0 1 public.ecr.aws/aws-containers/retail-store-sample-checkout:0.4.0 1 public.ecr.aws/aws-containers/retail-store-sample-orders:0.4.0 1 public.ecr.aws/aws-containers/retail-store-sample-ui:0.4.0 1 public.ecr.aws/bitnami/rabbitmq:3.11.1-debian-11-r0 2 public.ecr.aws/docker/library/mysql:8.0 1 public.ecr.aws/docker/library/redis:6.0-alpine 1 public.ecr.aws/docker/library/redis:7.0.15-alpine 2 public.ecr.aws/eks-distro/kubernetes-csi/external-provisioner:v3.6.3-eks-1-29-2 8 public.ecr.aws/eks-distro/kubernetes-csi/livenessprobe:v2.11.0-eks-1-29-2 6 public.ecr.aws/eks-distro/kubernetes-csi/node-driver-registrar:v2.9.3-eks-1-29-2 2 public.ecr.aws/eks/aws-load-balancer-controller:v2.7.1 2 public.ecr.aws/karpenter/controller:0.37.0@sha256:157f478f5db1fe999f5e2d27badcc742bf51cc470508b3cebe78224d0947674f 5 quay.io/argoproj/argocd:v2.10.0 1 registry.k8s.io/metrics-server/metrics-server:v0.7.0 # IDE-Server 혹은 자신의 PC에서 반복 접속 해두자! # UI 웹에서는 Spring Boot Actuator 설정이 되어 있슴  ec2-user:~/environment/terraform:$ export UI_WEB=$(kubectl get svc -n ui ui-nlb -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'/actuator/health/liveness) ec2-user:~/environment/terraform:$ curl -s $UI_WEB ; echo {"status":"UP"} ec2-user:~/environment/terraform:$ aws eks describe-cluster --name $EKS_CLUSTER_NAME | egrep 'version|endpoint"|issuer|platformVersion' "version": "1.25", "endpoint": "https://52C16A04B8574AC4AA8D5815B27D9AF2.gr7.us-west-2.eks.amazonaws.com", "issuer": "https://oidc.eks.us-west-2.amazonaws.com/id/52C16A04B8574AC4AA8D5815B27D9AF2" "platformVersion": "eks.44", # 반복 접속 ec2-user:~/environment/terraform:$ while true; do curl -s $UI_WEB; date; aws eks describe-cluster --name eksworkshop-eksctl | egrep 'version|endpoint"|issuer|platformVersion'; echo ; sleep 2; echo; done {"status":"UP"}Tue Apr 1 16:58:18 UTC 2025 "version": "1.25", "endpoint": "https://52C16A04B8574AC4AA8D5815B27D9AF2.gr7.us-west-2.eks.amazonaws.com", "issuer": "https://oidc.eks.us-west-2.amazonaws.com/id/52C16A04B8574AC4AA8D5815B27D9AF2" "platformVersion": "eks.44", {"status":"UP"}Tue Apr 1 16:58:21 UTC 2025 "version": "1.25", "endpoint": "https://52C16A04B8574AC4AA8D5815B27D9AF2.gr7.us-west-2.eks.amazonaws.com", "issuer": "https://oidc.eks.us-west-2.amazonaws.com/id/52C16A04B8574AC4AA8D5815B27D9AF2" "platformVersion": "eks.44", |
variable.tf의 cluster_version 변수를 1.25 에서 로 1.26 으로 변경하고 terraform apply를 수행하여 Control-plane 업그레이드가 진행됩니다.
| # 1.25 --> 1.26 업그레이드시 약 10분 소요 # 클러스터 버전을 변경하면 테라폼 계획에 표시된 것처럼, 관리 노드 그룹에 대한 특정 버전이나 AMI가 테라폼 파일에 정의되지 않은 경우, # eks 클러스터 제어 평면과 관리 노드 그룹 및 애드온과 같은 관련 리소스를 업데이트하게 됩니다. # 이 계획을 적용하여 제어 평면 버전을 업데이트해 보겠습니다. ec2-user:~/environment/terraform:$ terraform apply -auto-approve Apply complete! Resources: 6 added, 2 changed, 6 destroyed. Outputs: configure_kubectl = "aws eks --region us-west-2 update-kubeconfig --name eksworkshop-eksctl" # eks control plane 1.26 업그레이드 확인  aws eks describe-cluster --name $EKS_CLUSTER_NAME | jq {
"cluster": {
"name": "eksworkshop-eksctl",
"arn": "arn:aws:eks:us-west-2:678866666676:cluster/eksworkshop-eksctl",
"createdAt": "2025-03-30T05:38:02.500000+00:00",
"version": "1.26",
"roleArn": "arn:aws:iam::678866666676:role/eksworkshop-eksctl-cluster-20250330053739974300000004",
"resourcesVpcConfig": {
"subnetIds": [
"subnet-042922ea084483364",
"subnet-082eeb7c16623ae93",
"subnet-040fd59c3e2770cb9"
],
"securityGroupIds": [
"sg-034e81419ef24390b"
],
"clusterSecurityGroupId": "sg-0291a3f8e88f9cdae",
"vpcId": "vpc-0b70a20653beed1df",
"endpointPublicAccess": true,
"endpointPrivateAccess": true,
"publicAccessCidrs": [
"0.0.0.0/0"
]
},
"kubernetesNetworkConfig": {
"serviceIpv4Cidr": "172.20.0.0/16",
"ipFamily": "ipv4",
"elasticLoadBalancing": {
"enabled": false
}
},
"logging": {
"clusterLogging": [
{
"types": [
"api",
"audit",
"authenticator"
],
"enabled": true
},
{
"types": [
"controllerManager",
"scheduler"
],
"enabled": false
}
]
},
"identity": {
"oidc": {
}
},
"status": "ACTIVE",
"certificateAuthority": {
"data": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJTHVVZ0R5NGkrM1F3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TlRBek16QXdOVE00TURSYUZ3MHpOVEF6TWpnd05UUXpNRFJhTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUURpQkQwd0JVaDUyRWxlek0rZUh3YVZNVC9sa0xhTXgwN2tneEhTN0cwMkt2MzNHT2p6MDdSSDlzeHcKbk5CZ2ZEUjJwVis1TERzTXgyVktWcm1EQUNHdUs4eC84L2dmd2dIQ25OV0hETklNSC9EM0hLTnlZUEdrVjhwZQo5Y2ZrL3hUYlRHQnhoNTNyVXREcGEyU2lEZ0h3YzRzWUZ0aVJjUXFtOUt0aStJY2JSN2ZNcWdlNDBQRHhHeW1ZCkU5d1V6OUtFeWpjRHlCYloxK1Fna1hoQzBoeTNNb0NWSVlMQmRJOUJ1b1N1NXJtcmJXdU5sQkdxczdPcnpyd2kKdHZNSk9ad3psaWM2QWQ4QXRFTE40OWxEVXZtaVJvY05ROFcyeW10VnVPQXRBRUlLenJVakwxQlc4Rlc5Q1d1NQpiT01ZanlSVlY4TmJsM1JWdnYvS2YvVVNZRW52QWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJTanpveFZoOUhaTW5HK3VXTDlkRWFzakhVOUlEQVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQmY5ZWVTT2x5dQpNYVEvY3l0V1htVXpHWHVUbkZQZG11NGVidTA1VGhQN3YwNnNBNkh0SWZuQ3FVU0NrMHZZUlpRL3M0UW0ySVIxCkFYMS9PWE9LQ0IxRXNDaldDNXRDblpRMHluZkhlMlNWZjJHdmhmYlRCQmdkc2haZDZjaDBtZGl1TzJVUzN3R1gKSDBzZ1g2bm1NY2g0TEc1RnVlNHVldEdVbGxGVEJXeWNBVnZJMEcyMGx4dnZGUUhCbVlzUmNkRWd3TlFOSEdTdwp2SXJsbStjOGtQK0VIcmlMcTh1eU15THhiUTA1Y0RyUlUzY0FpWDQzL0hibnM2WWlPcENZcTRGZlRNZldJOUJFCmpqYjJLaHRuMHJQTmlMNlplMEQyWW9ZWHY5YzNDV24vTVhhRU5weU1zek1CYUNUSDJCMnVYNVNaT3FSbGlUNlUKS2E2MzExNHpkUkhECi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"
},
"platformVersion": "eks.45",
"tags": {
"karpenter.sh/discovery": "eksworkshop-eksctl",
"terraform-aws-modules": "eks",
"Blueprint": "eksworkshop-eksctl",
"GithubRepo": "github.com/aws-ia/terraform-aws-eks-blueprints"
},
"encryptionConfig": [
{
"resources": [
"secrets"
],
"provider": {
"keyArn": "arn:aws:kms:us-west-2:678866666676:key/215f54d9-8a7f-425b-9da4-387f409cc1de"
}
}
],
"health": {
"issues": []
},
"accessConfig": {
"authenticationMode": "API_AND_CONFIG_MAP"
},
"upgradePolicy": {
"supportType": "EXTENDED"
}
}
}
# endpoint, issuer, platformVersion 동일 => IRSA 를 사용하는 4개의 App 동작에 문제 없음! ec2-user:~/environment/terraform:$ aws eks describe-cluster --name $EKS_CLUSTER_NAME | egrep 'version|endpoint"|issuer|platformVersion' # 파드 컨테이너 이미지 버전 모두 동일 kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].image}" | tr -s '[[:space:]]' '\n' | sort | uniq -c > 1.26.txt kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].image}" | tr -s '[[:space:]]' '\n' | sort | uniq -c diff 1.26.txt 1.25.txt # 동일함 # 파드 AGE 정보 확인 : 재생성된 파드 없음! kubectl get pod -A ... |
EKS 제어 평면 업그레이드를 성공적으로 완료했습니다. 이제 애드온 소프트웨어 업그레이드를 진행해 보겠습니다.
[2. 애드온] Upgrading EKS Addons
EKS 업그레이드의 일부로 클러스터에 설치된 애드온을 업그레이드해야 합니다. 애드온은 K8s 애플리케이션에 지원 운영 기능을 제공하는 소프트웨어로, 네트워킹, 컴퓨팅 및 스토리지를 위해 클러스터가 기본 AWS 리소스와 상호 작용할 수 있도록 하는 관찰 에이전트 또는 Kubernetes 드라이버와 같은 소프트웨어가 포함됩니다. 애드온 소프트웨어는 일반적으로 Kubernetes 커뮤니티, AWS와 같은 클라우드 공급자 또는 타사 공급업체에서 빌드하고 유지 관리합니다.
| # 가능한 업그레이드 버전 정보 확인 ec2-user:~/environment/terraform:$ eksctl get addon --cluster $CLUSTER_NAME 2025-04-01 17:27:08 [ℹ] Kubernetes version "1.26" in use by cluster "eksworkshop-eksctl" 2025-04-01 17:27:08 [ℹ] getting all addons 2025-04-01 17:27:09 [ℹ] to see issues for an addon run `eksctl get addon --name <addon-name> --cluster <cluster-name>` NAME VERSION STATUS ISSUES IAMROLE UPDATE AVAILABLE CONFIGURATION VALUES aws-ebs-csi-driver v1.41.0-eksbuild.1 ACTIVE 0 arn:aws:iam::678866666676:role/eksworkshop-eksctl-ebs-csi-driver-2025033005471464490000001d coredns v1.8.7-eksbuild.10 ACTIVE 0 v1.9.3-eksbuild.22,v1.9.3-eksbuild.21,v1.9.3-eksbuild.19,v1.9.3-eksbuild.17,v1.9.3-eksbuild.15,v1.9.3-eksbuild.11,v1.9.3-eksbuild.10,v1.9.3-eksbuild.9,v1.9.3-eksbuild.7,v1.9.3-eksbuild.6,v1.9.3-eksbuild.5,v1.9.3-eksbuild.3,v1.9.3-eksbuild.2 kube-proxy v1.25.16-eksbuild.8 ACTIVE 0 v1.26.15-eksbuild.24,v1.26.15-eksbuild.19,v1.26.15-eksbuild.18,v1.26.15-eksbuild.14,v1.26.15-eksbuild.10,v1.26.15-eksbuild.5,v1.26.15-eksbuild.2,v1.26.13-eksbuild.2,v1.26.11-eksbuild.4,v1.26.11-eksbuild.1,v1.26.9-eksbuild.2,v1.26.7-eksbuild.2,v1.26.6-eksbuild.2,v1.26.6-eksbuild.1,v1.26.4-eksbuild.1,v1.26.2-eksbuild.1 vpc-cni v1.19.3-eksbuild.1 ACTIVE 0 |
terraform을 사용하여 CoreDNS , kube-proxy 및 VPC CNI 의 업그레이드를 살펴보겠습니다.
현재 /home/ec2-user/environment/terraform/addons.tf 파일의 설정은 아래와 같습니다.
|
eks_addons = {
coredns = {
addon_version = "v1.8.7-eksbuild.10"
}
kube-proxy = {
addon_version = "v1.25.16-eksbuild.8"
}
vpc-cni = {
most_recent = true
}
aws-ebs-csi-driver = {
service_account_role_arn = module.ebs_csi_driver_irsa.iam_role_arn
}
}
|
Amazon EKS는 AWS Managed Addons에 대한 주어진 K8s 버전에 대한 호환되는 애드온 버전을 나열하는 API를 제공합니다. 다음 명령을 사용하여 1.26 호환되는 coredns애드온 버전을 찾을 수 있습니다.
| # kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].image}" | tr -s '[[:space:]]' '\n' | sort | uniq -c 6 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.19.3-eksbuild.1 8 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/aws-ebs-csi-driver:v1.41.0 2 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/coredns:v1.8.7-eksbuild.10 2 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-provisioner:v5.2.0-eks-1-32-7 6 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/kube-proxy:v1.25.16-minimal-eksbuild.8 8 amazon/aws-efs-csi-driver:v1.7.6 2 public.ecr.aws/eks/aws-load-balancer-controller:v2.7.1 # aws eks describe-addon-versions --addon-name coredns --kubernetes-version 1.26 --output table \ --query "addons[].addonVersions[:10].{Version:addonVersion,DefaultVersion:compatibilities[0].defaultVersion}" ------------------------------------------ | DescribeAddonVersions | +-----------------+----------------------+ | DefaultVersion | Version | +-----------------+----------------------+ | False | v1.9.3-eksbuild.22 | | False | v1.9.3-eksbuild.21 | | False | v1.9.3-eksbuild.19 | | False | v1.9.3-eksbuild.17 | | False | v1.9.3-eksbuild.15 | | False | v1.9.3-eksbuild.11 | | False | v1.9.3-eksbuild.10 | | False | v1.9.3-eksbuild.9 | | True | v1.9.3-eksbuild.7 | | False | v1.9.3-eksbuild.6 | +-----------------+----------------------+ # aws eks describe-addon-versions --addon-name kube-proxy --kubernetes-version 1.26 --output table \ --query "addons[].addonVersions[:10].{Version:addonVersion,DefaultVersion:compatibilities[0].defaultVersion}" -------------------------------------------- | DescribeAddonVersions | +-----------------+------------------------+ | DefaultVersion | Version | +-----------------+------------------------+ | False | v1.26.15-eksbuild.24 | | False | v1.26.15-eksbuild.19 | | False | v1.26.15-eksbuild.18 | | False | v1.26.15-eksbuild.14 | | False | v1.26.15-eksbuild.10 | | False | v1.26.15-eksbuild.5 | | False | v1.26.15-eksbuild.2 | | False | v1.26.13-eksbuild.2 | | False | v1.26.11-eksbuild.4 | | False | v1.26.11-eksbuild.1 | +-----------------+------------------------+ |
- 참고로 나머지 Addon 인 VPC CNI, EBS CSI Driver 는 현재 최신 버전 사용 중임 - 콘솔에서 addon 에서 Edit 로 버전 정보 확인.
위의 출력에서 최신 버전을 선택하여 addons.tf아래에 표시된 대로 파일에 수정하여 업데이트합니다.
|
eks_addons = {
coredns = {
version = "v1.9.3-eksbuild.22" # Recommended version for EKS 1.26
}
kube_proxy = {
version = "v1.26.15-eksbuild.24" # Recommended version for EKS 1.26
}
vpc-cni = {
most_recent = true
}
aws-ebs-csi-driver = {
service_account_role_arn = module.ebs_csi_driver_irsa.iam_role_arn
}
}
|
terraform apply 명령을 수행하여 롤링 업데이트를 수행합니다.
| # 1분 정도 이내로 롤링 업데이트 완료! ec2-user:~/environment/terraform:$ terraform apply -auto-approve Apply complete! Resources: 0 added, 2 changed, 0 destroyed. Outputs: configure_kubectl = "aws eks --region us-west-2 update-kubeconfig --name eksworkshop-eksctl" # 업그레이드한 add-on POD가 정상수행인지 확인 ec2-user:~/environment/terraform:$ kubectl get pod -n kube-system -l 'k8s-app in (kube-dns, kube-proxy)' NAME READY STATUS RESTARTS AGE coredns-58cc4d964b-br2vr 1/1 Running 0 115s coredns-58cc4d964b-ld6gl 1/1 Running 0 115s kube-proxy-5n6bc 1/1 Running 0 106s kube-proxy-7klz6 1/1 Running 0 101s kube-proxy-lqw9m 1/1 Running 0 112s kube-proxy-qmmb2 1/1 Running 0 115s kube-proxy-qvl7t 1/1 Running 0 109s kube-proxy-xcsdl 1/1 Running 0 104s # 정상적으로 업그레이드되었는지 확인 ec2-user:~/environment/terraform:$ kubectl get pods --all-namespaces -o jsonpath="{.items[*].spec.containers[*].image}" | tr -s '[[:space:]]' '\n' | sort | uniq -c 6 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:v1.19.3-eksbuild.1 6 602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon/aws-network-policy-agent:v1.2.0-eksbuild.1 8 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/aws-ebs-csi-driver:v1.41.0 2 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/coredns:v1.9.3-eksbuild.22 2 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-attacher:v4.8.1-eks-1-32-7 6 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-node-driver-registrar:v2.13.0-eks-1-32-7 2 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-provisioner:v5.2.0-eks-1-32-7 2 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-resizer:v1.13.2-eks-1-32-7 2 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/csi-snapshotter:v8.2.1-eks-1-32-7 6 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/kube-proxy:v1.26.15-minimal-eksbuild.24 8 602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/livenessprobe:v2.14.0-eks-1-32-7 8 amazon/aws-efs-csi-driver:v1.7.6 1 amazon/dynamodb-local:1.13.1 1 ghcr.io/dexidp/dex:v2.38.0 1 hjacobs/kube-ops-view:20.4.0 1 public.ecr.aws/aws-containers/retail-store-sample-assets:0.4.0 1 public.ecr.aws/aws-containers/retail-store-sample-cart:0.7.0 1 public.ecr.aws/aws-containers/retail-store-sample-catalog:0.4.0 1 public.ecr.aws/aws-containers/retail-store-sample-checkout:0.4.0 1 public.ecr.aws/aws-containers/retail-store-sample-orders:0.4.0 1 public.ecr.aws/aws-containers/retail-store-sample-ui:0.4.0 1 public.ecr.aws/bitnami/rabbitmq:3.11.1-debian-11-r0 2 public.ecr.aws/docker/library/mysql:8.0 1 public.ecr.aws/docker/library/redis:6.0-alpine 1 public.ecr.aws/docker/library/redis:7.0.15-alpine 2 public.ecr.aws/eks-distro/kubernetes-csi/external-provisioner:v3.6.3-eks-1-29-2 8 public.ecr.aws/eks-distro/kubernetes-csi/livenessprobe:v2.11.0-eks-1-29-2 6 public.ecr.aws/eks-distro/kubernetes-csi/node-driver-registrar:v2.9.3-eks-1-29-2 2 public.ecr.aws/eks/aws-load-balancer-controller:v2.7.1 2 public.ecr.aws/karpenter/controller:0.37.0@sha256:157f478f5db1fe999f5e2d27badcc742bf51cc470508b3cebe78224d0947674f 5 quay.io/argoproj/argocd:v2.10.0 1 registry.k8s.io/metrics-server/metrics-server:v0.7.0 |
현재 Data-plane에 해당하는 워커노드는 1.25버전입니다.
| ec2-user:~/environment/terraform:$ k get no NAME STATUS ROLES AGE VERSION fargate-ip-10-0-32-234.us-west-2.compute.internal Ready <none> 2d11h v1.25.16-eks-2d5f260 ip-10-0-13-173.us-west-2.compute.internal Ready <none> 22h v1.25.16-eks-59bf375 ip-10-0-14-181.us-west-2.compute.internal Ready <none> 2d12h v1.25.16-eks-59bf375 ip-10-0-16-129.us-west-2.compute.internal Ready <none> 2d12h v1.25.16-eks-59bf375 ip-10-0-26-237.us-west-2.compute.internal Ready <none> 2d12h v1.25.16-eks-59bf375 ip-10-0-32-44.us-west-2.compute.internal Ready <none> 2d12h v1.25.16-eks-59bf375 ip-10-0-6-153.us-west-2.compute.internal Ready <none> 2d12h v1.25.16-eks-59bf375 |
[3.1 데이터부 - 관리형노드그룹] Upgrading EKS Managed Node groups
- 30분 이상 소요
- 3가지 방법으로 업그레이드가 가능하면, 이번 워크샵에서는 Terraform 을 이용합니다.
방법 1 : eksctl
| # 우리는 사용 중인 AMI 유형의 Kubernetes 버전에 대한 최신 AMI 릴리스 버전으로 노드 그룹을 업데이트할 수 있습니다. eksctl upgrade nodegroup --name=managed-ng-1 --cluster=$EKS_CLUSTER_NAME # 노드 그룹이 클러스터의 Kubernetes 버전에서 이전 Kubernetes 버전인 경우, 다음을 통해 노드 그룹을 클러스터의 Kubernetes 버전과 일치하는 최신 AMI 릴리스 버전으로 업데이트할 수 있습니다. eksctl upgrade nodegroup --name=managed-ng-1 --cluster=$EKS_CLUSTER_NAME --kubernetes-version=<control plane version> # 최신 버전 대신 특정 AMI 릴리스 버전으로 업그레이드하려면 다음을 사용합니다. eksctl upgrade nodegroup --name=managed-ng-1 --cluster=$EKS_CLUSTER_NAME --release-version=<specific AMI Release version> |
방법 2 : AWS 관리 콘솔
방법 3 : Terraform
| # 특정 AMI ID 사용하는 노드 업그레이드를 위해 kubernetes 버전 1.25의 ami_id를 가져오고 variable.tf 파일의 변수 ami_id를 결과 값으로 대체합니다. ec2-user:~:$ aws ec2 describe-instances --query "Reservations[*].Instances[*].[Tags[?Key=='Name'].Value | [0], ImageId]" --filters "Name=tag:Blueprint,Values=eksworkshop-eksctl" --output table ---------------------------------------------- | DescribeInstances | +------------------+-------------------------+ | initial | ami-0078a0f78fafda978 | | initial | ami-0078a0f78fafda978 | | default-selfmng | ami-0ee947a6f4880da75 | | blue-mng | ami-0078a0f78fafda978 | | default-selfmng | ami-0ee947a6f4880da75 | +------------------+-------------------------+ ec2-user:~:$ kubectl get node --label-columns=eks.amazonaws.com/capacityType,node.kubernetes.io/lifecycle,karpenter.sh/capacity-type,eks.amazonaws.com/compute-type NAME STATUS ROLES AGE VERSION CAPACITYTYPE LIFECYCLE CAPACITY-TYPE COMPUTE-TYPE fargate-ip-10-0-32-234.us-west-2.compute.internal Ready <none> 2d12h v1.25.16-eks-2d5f260 fargate ip-10-0-13-173.us-west-2.compute.internal Ready <none> 23h v1.25.16-eks-59bf375 spot ip-10-0-14-181.us-west-2.compute.internal Ready <none> 2d12h v1.25.16-eks-59bf375 self-managed ip-10-0-16-129.us-west-2.compute.internal Ready <none> 2d12h v1.25.16-eks-59bf375 self-managed ip-10-0-26-237.us-west-2.compute.internal Ready <none> 2d12h v1.25.16-eks-59bf375 ON_DEMAND ip-10-0-32-44.us-west-2.compute.internal Ready <none> 2d12h v1.25.16-eks-59bf375 ON_DEMAND ip-10-0-6-153.us-west-2.compute.internal Ready <none> 2d12h v1.25.16-eks-59bf375 ON_DEMAND ec2-user:~:$ kubectl get node -L eks.amazonaws.com/nodegroup,karpenter.sh/nodepool NAME STATUS ROLES AGE VERSION NODEGROUP NODEPOOL fargate-ip-10-0-32-234.us-west-2.compute.internal Ready <none> 2d12h v1.25.16-eks-2d5f260 ip-10-0-13-173.us-west-2.compute.internal Ready <none> 23h v1.25.16-eks-59bf375 default ip-10-0-14-181.us-west-2.compute.internal Ready <none> 2d12h v1.25.16-eks-59bf375 ip-10-0-16-129.us-west-2.compute.internal Ready <none> 2d12h v1.25.16-eks-59bf375 ip-10-0-26-237.us-west-2.compute.internal Ready <none> 2d12h v1.25.16-eks-59bf375 initial-20250330054749233900000029 ip-10-0-32-44.us-west-2.compute.internal Ready <none> 2d12h v1.25.16-eks-59bf375 initial-20250330054749233900000029 ip-10-0-6-153.us-west-2.compute.internal Ready <none> 2d12h v1.25.16-eks-59bf375 blue-mng-2025033005474923890000002b ec2-user:~:$ kubectl get node -L eks.amazonaws.com/nodegroup-image | grep ami ip-10-0-26-237.us-west-2.compute.internal Ready <none> 2d12h v1.25.16-eks-59bf375 ami-0078a0f78fafda978 ip-10-0-32-44.us-west-2.compute.internal Ready <none> 2d12h v1.25.16-eks-59bf375 ami-0078a0f78fafda978 ip-10-0-6-153.us-west-2.compute.internal Ready <none> 2d12h v1.25.16-eks-59bf375 ami-0078a0f78fafda978 ec2-user:~:$ aws ssm get-parameter --name /aws/service/eks/optimized-ami/1.25/amazon-linux-2/recommended/image_id \ --region $AWS_REGION --query "Parameter.Value" --output text ami-0078a0f78fafda978 # variable.tf 파일에서 ami_id를 업데이트합니다  # base.tf사용자 정의 관리 노드 그룹을 프로비저닝하려면 다음 코드를 추가합니다 . ...(생략)... custom = {
instance_types = ["t3.medium"]
min_size = 1
max_size = 2
desired_size = 1
update_config = {
max_unavailable_percentage = 35
}
ami_id = try(var.ami_id)
enable_bootstrap_user_data = true
}
...(생략)...  |
terraform apply를 수행합니다.
| ec2-user:~/environment/terraform:$ terraform apply -auto-approve Apply complete! Resources: 10 added, 0 changed, 3 destroyed. Outputs: configure_kubectl = "aws eks --region us-west-2 update-kubeconfig --name eksworkshop-eksctl" # 추가된 custome 노드 그룹 확인   # kubernetes 버전 1.26에 대해 ami_id를 검색합니다. ec2-user:~/environment/terraform:$ aws ssm get-parameter --name /aws/service/eks/optimized-ami/1.26/amazon-linux-2/recommended/image_id \ --region $AWS_REGION --query "Parameter.Value" --output text ami-086414611b43bb691 # variables.tf 파일을 아래와 같이 수정합니다.  # terraform apply를 수행 ec2-user:~/environment/terraform:$ terraform apply -auto-approve Apply complete! Resources: 3 added, 3 changed, 3 destroyed. Outputs: configure_kubectl = "aws eks --region us-west-2 update-kubeconfig --name eksworkshop-eksctl" # inplace 업그레이드가 진행되는 동안 노드목록을 확인하면 아래와 같은 진행상황을 볼수 있습니다.  #업그레이드가 완료되면 initial managed nodegroup의 노드는 1.26으로 업그레이드 되었습니다. ec2-user:~/environment/terraform:$ k get no NAME STATUS ROLES AGE VERSION fargate-ip-10-0-32-234.us-west-2.compute.internal Ready <none> 2d13h v1.25.16-eks-2d5f260 ip-10-0-13-173.us-west-2.compute.internal Ready <none> 23h v1.25.16-eks-59bf375 ip-10-0-14-181.us-west-2.compute.internal Ready <none> 2d13h v1.25.16-eks-59bf375 ip-10-0-16-129.us-west-2.compute.internal Ready <none> 2d13h v1.25.16-eks-59bf375 ip-10-0-18-246.us-west-2.compute.internal Ready <none> 17m v1.26.15-eks-59bf375 ip-10-0-2-145.us-west-2.compute.internal Ready <none> 17m v1.26.15-eks-59bf375 ip-10-0-36-83.us-west-2.compute.internal Ready <none> 16m v1.26.15-eks-59bf375 ip-10-0-6-153.us-west-2.compute.internal Ready <none> 2d13h v1.25.16-eks-59bf375 # custom 은 실습 동작 확인했으니, 다음 실습을 위해서 base.tf 파일에서 삭제하고, terraform apply 를 수행합니다.  ec2-user:~/environment/terraform:$ terraform apply -auto-approve Apply complete! Resources: 3 added, 0 changed, 10 destroyed. Outputs: configure_kubectl = "aws eks --region us-west-2 update-kubeconfig --name eksworkshop-eksctl" |
[3.1-2 데이터부 - 관리형노드그룹 : 블루-그린 업그레이드]
관리 노드 그룹을 업데이트하기 위해 Blue / Green 전략을 사용할 수도 있습니다.
우리는 eks 제어 평면 버전을 업데이트한 후 새로운 관리 노드 그룹을 생성함으로써 이를 수행합니다.
지난 실험[3.1-1]에서는 Blue-mng 관리 노드 그룹을 업데이트하지 않았습니다.
Blue-mng 관리 노드 그룹의 구성을 보면, 이 노드 그룹에 할당된 노드들이 특정 테인트와 라벨을 가지고 있으며, 하나의 특정 가용성 구역에 할당되도록 구성되어 있음을 알 수 있습니다.
Here we have followed the recommended best practice to provision managed node group for stateful workloads, i.e.
상태 저장 워크로드를 위한 관리 노드 그룹은 가용 영역(AZ)별로 프로비저닝되어야 합니다. Managed node groups for stateful workloads should be provisioned per availability zone (AZ).
이 실험실에서는 stateful 워크로드의 마이그레이션을 시연하고 있습니다.
Blue-mng 관리 노드 그룹의 노드에서 실행되는 my-sql 포드로 구성된 Orders 애플리케이션 포드가 있습니다.
우리는 blue-mng와 유사한 구성을 가진 새로운 관리 노드 그룹 green-mng을 만들 것입니다.
Green-mng는 eks_managed_node_group_defaults에 제공된 클러스터 버전(즉, 1.26)과 Blue-mng과 동일한 가용성 영역에서 프로비저닝됩니다.
그 후에, 우리는 blue-mng을 삭제할 것입니다.
블루밍 노드가 자동으로 코드화되고 배수되며, 주문 앱 포드가 그린밍에서 프로비저닝되는 것을 볼 수 있습니다.
| # base.tf 파일에 green-mng 노드그룹을 추가하고 저장합니다. green-mng={
instance_types = ["m5.large", "m6a.large", "m6i.large"]
subnet_ids = [module.vpc.private_subnets[0]]
min_size = 1
max_size = 2
desired_size = 1
update_config = {
max_unavailable_percentage = 35
}
labels = {
type = "OrdersMNG"
}
taints = [
{
key = "dedicated"
value = "OrdersApp"
effect = "NO_SCHEDULE"
}
]
}
 # terraform apply를 수행합니다. ec2-user:~/environment/terraform:$ terraform apply -auto-approve Apply complete! Resources: 10 added, 0 changed, 3 destroyed. Outputs: configure_kubectl = "aws eks --region us-west-2 update-kubeconfig --name eksworkshop-eksctl" # green-mng 노드그룹에 의해 1.26 버전의 node가 생성되었습니다.  # 이제 1.25버전의 blue-mng 노드그룹은 불필요하므로, base.tf 파일에서 제거하고, terraform apply를 수행합니다.  ec2-user:~/environment/terraform:$ terraform apply -auto-approve Apply complete! Resources: 3 added, 0 changed, 10 destroyed. Outputs: configure_kubectl = "aws eks --region us-west-2 update-kubeconfig --name eksworkshop-eksctl" # 이제 managed nodegroup의 모습은 아래와 같습니다.  |
[3.2 데이터부 - 카펜터노드] Upgrading Karpenter managed nodes
Karpenter는 집계된 CPU, 메모리, 볼륨 요청 및 기타 Kubernetes 스케줄링 제약(예: 친화도 및 포드 토폴로지 확산 제약)을 기반으로 스케줄링 불가능한 포드에 대응하여 적절한 크기의 노드를 제공하는 오픈 소스 클러스터 오토스케일러로, 인프라 관리를 간소화합니다.
또한 Karpenter는 노드 그룹 및 Amazon EC2 자동 확장 그룹과 같은 외부 인프라와 용량 관리를 조정하지 않기 때문에 운영 프로세스에 대한 다른 관점을 도입하여 작업자 노드 구성 요소와 운영 체제를 최신 보안 패치 및 기능으로 최신 상태로 유지합니다.
AWS는 새로운 Kubernetes 버전뿐만 아니라 패치 및 CVE(일반 취약점 및 노출)를 위한 AMI를 출시합니다. 다양한 Amazon EKS에 최적화된 AMI 중에서 선택할 수 있습니다. 또는 자신만의 맞춤형 AMI를 사용할 수도 있습니다.
현재 Karpenter에서는 EC2NodeClass 리소스가 amiFamily 값 AL2, AL2023, Bottlerocket, Ubuntu, Windows2019, Windows2022 및 Custom을 지원합니다. amiFamily of Custom을 선택하면 어떤 사용자 지정 AMI를 사용할지 Karpenter에 알려주는 amiSelectorTerms를 지정해야 합니다.
Karpenter는 Karpenter를 통해 프로비저닝된 Kubernetes 작업자 노드를 패치하는 데 Drift 또는 타이머(expireAfter)와 같은 기능을 사용합니다.
- 노드풀에 대해 카펜터를 통해 노드를 프로비저닝해 보겠습니다
| ec2-user:~:$ aws ec2 describe-instances --query "Reservations[*].Instances[*].[Tags[?Key=='Name'].Value | [0], ImageId]" --filters "Name=tag:Blueprint,Values=eksworkshop-eksctl" --output table ---------------------------------------------- | DescribeInstances | +------------------+-------------------------+ | initial | ami-086414611b43bb691 | | initial | ami-0078a0f78fafda978 | | custom | ami-086414611b43bb691 | | initial | ami-0078a0f78fafda978 | | initial | ami-086414611b43bb691 | | custom | ami-086414611b43bb691 | | default-selfmng | ami-0ee947a6f4880da75 | | blue-mng | ami-0078a0f78fafda978 | | custom | ami-086414611b43bb691 | | initial | ami-086414611b43bb691 | | initial | ami-086414611b43bb691 | | custom | ami-0078a0f78fafda978 | | green-mng | ami-086414611b43bb691 | | default-selfmng | ami-0ee947a6f4880da75 | +------------------+-------------------------+ # 기본 노드풀을 통해 프로비저닝된 노드가 버전 v1.25.16-eks-59bf375에 있는 것을 확인할 수 있습니다. ec2-user:~:$ kubectl get nodes -l team=checkout NAME STATUS ROLES AGE VERSION ip-10-0-13-173.us-west-2.compute.internal Ready <none> 24h v1.25.16-eks-59bf375 # Check taints applied on the nodes. ec2-user:~:$ kubectl get nodes -l team=checkout -o jsonpath="{range .items[*]}{.metadata.name} {.spec.taints}{\"\n\"}{end}" ip-10-0-13-173.us-west-2.compute.internal [{"effect":"NoSchedule","key":"dedicated","value":"CheckoutApp"}] ec2-user:~:$ kubectl get nodes -l team=checkout -o jsonpath="{range .items[*]}{.metadata.name} {.spec.taints[?(@.effect=='NoSchedule')]}{\"\n\"}{end}" ip-10-0-13-173.us-west-2.compute.internal {"effect":"NoSchedule","key":"dedicated","value":"CheckoutApp"} ec2-user:~:$ kubectl get pods -n checkout -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES checkout-558f7777c-d6gcv 1/1 Running 0 24h 10.0.13.62 ip-10-0-13-173.us-west-2.compute.internal <none> <none> checkout-redis-f54bf7cb5-5b2fm 1/1 Running 0 24h 10.0.14.94 ip-10-0-13-173.us-west-2.compute.internal <none> <none> # 체크아웃 애플리케이션을 확장해 보겠습니다. 이를 위해 아래와 같이 eks-gitops-repo/apps/checkout 폴더의 deployment.yaml 파일에서 복제본을 1에서 10으로 변경합니다.  # 변경이 완료되면 저장소에 변경 사항을 커밋하고 푸시한 후 Argo-cd가 변경 사항을 동기화할 때까지 기다립니다. ec2-user:~:$ cd ~/environment/eks-gitops-repo ec2-user:~/environment/eks-gitops-repo:$ git add apps/checkout/deployment.yaml ec2-user:~/environment/eks-gitops-repo:$ git commit -m "scale checkout app" [main 7456dc3] scale checkout app Committer: EC2 Default User <ec2-user@ip-192-168-0-227.us-west-2.compute.internal> Your name and email address were configured automatically based on your username and hostname. Please check that they are accurate. You can suppress this message by setting them explicitly. Run the following command and follow the instructions in your editor to edit your configuration file: git config --global --edit After doing this, you may fix the identity used for this commit with: git commit --amend --reset-author 1 file changed, 1 insertion(+), 1 deletion(-) ec2-user:~/environment/eks-gitops-repo:$ git push --set-upstream origin main Enumerating objects: 9, done. Counting objects: 100% (9/9), done. Delta compression using up to 2 threads Compressing objects: 100% (5/5), done. Writing objects: 100% (5/5), 460 bytes | 460.00 KiB/s, done. Total 5 (delta 3), reused 0 (delta 0), pack-reused 0 (from 0) remote: Validating objects: 100% To codecommit::us-west-2://eks-gitops-repo 6a85931..7456dc3 main -> main branch 'main' set up to track 'origin/main'. # LIVE(k8s)에 직접 scale 실행 ec2-user:~/environment/eks-gitops-repo:$ kubectl scale deploy checkout -n checkout --replicas 10 deployment.apps/checkout scaled  # Lets get the ami id for the AMI build for kubernetes version 1.26. ec2-user:~/environment/eks-gitops-repo:$ aws ssm get-parameter --name /aws/service/eks/optimized-ami/1.26/amazon-linux-2/recommended/image_id \ --region ${AWS_REGION} --query "Parameter.Value" --output text ami-086414611b43bb691 # default-ec2nc.yaml 파일의 ami 부분을 1.26 ami 로 수정  # default-np.yaml 파일을 수정 --> node 1대씩 수행하도록 budgents 추가  # 이제 Config Repository 에 PUSH 하면 GitOps를 지원하는 ArgoCD가 배포 --> 약 10분 소요 cd ~/environment/eks-gitops-repo git add apps/karpenter/default-ec2nc.yaml apps/karpenter/default-np.yaml git commit -m "disruption changes" git push --set-upstream origin main argocd app sync karpenter ## |
[3.3 데이터부 - 셀프노드]
- 기본 정보 확인
| # Lets explore the self-managed nodes. ec2-user:~:$ kubectl get nodes --show-labels | grep self-managed ip-10-0-14-181.us-west-2.compute.internal Ready <none> 2d14h v1.25.16-eks-59bf375 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m5.large,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-west-2,failure-domain.beta.kubernetes.io/zone=us-west-2a,k8s.io/cloud-provider-aws=a94967527effcefb5f5829f529c0a1b9,kubernetes.io/arch=amd64,kubernetes.io/hostname=ip-10-0-14-181.us-west-2.compute.internal,kubernetes.io/os=linux,node.kubernetes.io/instance-type=m5.large,node.kubernetes.io/lifecycle=self-managed,team=carts,topology.ebs.csi.aws.com/zone=us-west-2a,topology.kubernetes.io/region=us-west-2,topology.kubernetes.io/zone=us-west-2a ip-10-0-16-129.us-west-2.compute.internal Ready <none> 2d14h v1.25.16-eks-59bf375 beta.kubernetes.io/arch=amd64,beta.kubernetes.io/instance-type=m5.large,beta.kubernetes.io/os=linux,failure-domain.beta.kubernetes.io/region=us-west-2,failure-domain.beta.kubernetes.io/zone=us-west-2b,k8s.io/cloud-provider-aws=a94967527effcefb5f5829f529c0a1b9,kubernetes.io/arch=amd64,kubernetes.io/hostname=ip-10-0-16-129.us-west-2.compute.internal,kubernetes.io/os=linux,node.kubernetes.io/instance-type=m5.large,node.kubernetes.io/lifecycle=self-managed,team=carts,topology.ebs.csi.aws.com/zone=us-west-2b,topology.kubernetes.io/region=us-west-2,topology.kubernetes.io/zone=us-west-2b # Verify if the pods are scheduled on self-managed nodes. ec2-user:~:$ kubectl get pods -n carts -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES carts-7ddbc698d8-q7vjh 1/1 Running 0 2d14h 10.0.7.103 ip-10-0-14-181.us-west-2.compute.internal <none> <none> carts-dynamodb-6594f86bb9-x6785 1/1 Running 0 2d14h 10.0.18.28 ip-10-0-16-129.us-west-2.compute.internal <none> <none> # lets perform in-place upgrade on the self-managed nodes. 자가 관리 노드를 업그레이드하려면 실행 템플릿에서 AMI를 업데이트해야 합니다. 이 워크숍에서는 테라폼을 사용하여 자가 관리 노드를 생성했습니다. 이를 통해 /home/ec2-user/Environment/terraform/base.tf 파일에서 AMI를 업데이트하고 변경 사항을 적용할 수 있습니다. 하지만 먼저 Kubernetes 버전 1.26에 해당하는 최신 AMI를 확인해 보겠습니다. ec2-user:~:$ aws ssm get-parameter --name /aws/service/eks/optimized-ami/1.26/amazon-linux-2/recommended/image_id --region $AWS_REGION --query "Parameter.Value" --output text ami-086414611b43bb691 |
- base.tf 파일의 self_managed_node_groups의 ami_id 값 수정
| self_managed_node_groups = { self-managed-group = { instance_type = "m5.large" . . . # Additional configurations ami_id = "ami-086414611b43bb691" # Replaced the latest AMI ID for EKS 1.26 subnet_ids = module.vpc.private_subnets . . . launch_template_use_name_prefix = true } } |
- terraform apply 실행
| ec2-user:~/environment/terraform:$ terraform apply -auto-approve Apply complete! Resources: 3 added, 2 changed, 3 destroyed. Outputs: configure_kubectl = "aws eks --region us-west-2 update-kubeconfig --name eksworkshop-eksctl" # 신규 버전 EC2 1대가 Ready되고 나서 5분 Refresh 시간 이후에 기존 버전 EC2 1대를 종료 실행하면서, 동시에 추가 신규 버전 EC2 1대 생성되면서 업그레이드 완료 ec2-user:~:$ kubectl get nodes -l node.kubernetes.io/lifecycle=self-managed NAME STATUS ROLES AGE VERSION ip-10-0-22-126.us-west-2.compute.internal Ready <none> 94s v1.26.15-eks-59bf375 ip-10-0-47-22.us-west-2.compute.internal Ready <none> 9m18s v1.26.15-eks-59bf375 |
[3.4 데이터부 - 파게이트노드]
Fargate는 컨테이너에 필요에 따라 적절한 크기의 컴퓨팅 용량을 제공하는 기술입니다.
Fargate를 사용하면 컨테이너를 실행하기 위해 가상 머신 그룹을 직접 프로비저닝, 구성 또는 확장할 필요가 없습니다.
서버 유형을 선택하거나 노드 그룹을 확장할 시기를 결정하거나 클러스터 패킹을 최적화할 필요도 없습니다.
Fargate에서 실행되는 기준을 충족하는 Pod를 시작하면 클러스터에서 실행 중인 Fargate 컨트롤러가 Pod를 Fargate로 인식, 업데이트 및 예약합니다.
AWS Fargate 노드를 업그레이드하려면 K8s 배포를 다시 시작하여 새로운 포드가 최신 Kubernetes 버전에서 자동으로 예약되도록 할 수 있습니다.
- 기본 정보 확인
| ec2-user:~:$ kubectl get pods -n assets -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES assets-7ccc84cb4d-mqs5h 1/1 Running 0 2d14h 10.0.32.234 fargate-ip-10-0-32-234.us-west-2.compute.internal <none> <none> ec2-user:~:$ ec2-user:~:$ kubectl get node $(kubectl get pods -n assets -o jsonpath='{.items[0].spec.nodeName}') -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME fargate-ip-10-0-32-234.us-west-2.compute.internal Ready <none> 2d14h v1.25.16-eks-2d5f260 10.0.32.234 <none> Amazon Linux 2 5.10.234-225.910.amzn2.x86_64 containerd://1.7.25 |
- 디플로이먼트 재시작 Restart 으로 신규 버전 마이그레이션!
| # 디플로이먼트 재시작 Restart --> 아래 명령으로 업그레이드가 완료됨 ec2-user:~:$ kubectl rollout restart deployment assets -n assets deployment.apps/assets restarted # Lets wait for the new pods to become ready. ec2-user:~:$ kubectl wait --for=condition=Ready pods --all -n assets --timeout=180s pod/assets-7ccc84cb4d-mqs5h condition met pod/assets-84f95b8fb4-kl472 condition met # Once the new pods reach ready state, check the version of the new fargate node. ec2-user:~:$ kubectl get pods -n assets -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES assets-84f95b8fb4-kl472 1/1 Running 0 116s 10.0.44.133 fargate-ip-10-0-44-133.us-west-2.compute.internal <none> <none> ec2-user:~:$ kubectl get node $(kubectl get pods -n assets -o jsonpath='{.items[0].spec.nodeName}') -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME fargate-ip-10-0-44-133.us-west-2.compute.internal Ready <none> 82s v1.26.15-eks-2d5f260 10.0.44.133 <none> Amazon Linux 2 5.10.234-225.910.amzn2.x86_64 containerd://1.7.25 |
최종 실습이 완료된 모습
- 최종 업그레이드가 완료되면 Data-plane 의 모든 노드가 1.26으로 업그레이드 완료됨
| ec2-user:~:$ kubectl get no NAME STATUS ROLES AGE VERSION fargate-ip-10-0-44-133.us-west-2.compute.internal Ready <none> 60m v1.26.15-eks-2d5f260 ip-10-0-1-219.us-west-2.compute.internal Ready <none> 2m8s v1.26.15-eks-59bf375 ip-10-0-18-252.us-west-2.compute.internal Ready <none> 3m11s v1.26.15-eks-59bf375 ip-10-0-2-145.us-west-2.compute.internal Ready <none> 141m v1.26.15-eks-59bf375 ip-10-0-22-126.us-west-2.compute.internal Ready <none> 26m v1.26.15-eks-59bf375 ip-10-0-36-83.us-west-2.compute.internal Ready <none> 140m v1.26.15-eks-59bf375 ip-10-0-47-22.us-west-2.compute.internal Ready <none> 34m v1.26.15-eks-59bf375 ip-10-0-8-184.us-west-2.compute.internal Ready <none> 100m v1.26.15-eks-59bf375 |
'Kubernetes' 카테고리의 다른 글
| AEWS [3기] 12주차 - Amazon VPC Lattice for Amazon EKS (0) | 2025.04.26 |
|---|---|
| AEWS [3기] 10주차 - K8S 시크릿 관리 (0) | 2025.04.10 |
| AEWS [3기] 8주차 - K8S CI/CD 와 kustomize (1) | 2025.03.29 |
| AEWS [3기] 7주차 - EKS Mode - Fargate (0) | 2025.03.22 |
| AEWS [3기] 6주차 - K8S API Server Client Certificate 인증 (0) | 2025.03.15 |