EKS 에서 사용하는 스토리지에 대해 알아보기전에 온프레미스 환경에서 구축한 쿠버네티스에서 K8S 스토리지 관련 리소스인 PV(Persistent Volume), PVC(Persistenet Volume Cliaim), SC(Storage Class)에 대해 먼저 알아보겠습니다.
1. 쿠버네티스 클러스터의 PV, PVC, SC 리소스
1.1. K8S v1.31 클러스터 구성
Terraform을 이용하여 아래와 같이 EC2 3대를 이용하여 바닐라 쿠버네티스를 구성하겠습니다.
첨부한 lab-setup.zip 파일을 압축하고 명령프롬프트 또는 쉘을 열고 terraform 실행파일을 OS 환경에 맞게 다운로드 합니다.
| C:\eks-workshop\lab-setup> curl -sO https://releases.hashicorp.com/terraform/1.10.5/terraform_1.10.5_windows_amd64.zip C:\eks-workshop\lab-setup> tar -xf terraform_1.10.5_windows_amd64.zip |
variables.tf 파일을 열고 AWS "access_key" 및 "secret_key"를 본인 소유의 키로 변경합니다.
EC2 접속을 위한 SSH Key-Pair 생성
| C:\eks-workshop\lab-setup> ssh-keygen -t rsa -f edukey -N "" Generating public/private rsa key pair. Your identification has been saved in edukey Your public key has been saved in edukey.pub The key fingerprint is: SHA256:mwFKs5R2KdWdtSR3qq2FC8YpwSiFBg4mHhodZKVLTho administrator@JHY |
“terraform init” 명령을 수행하여 AWS Provider 초기화하고 “terraform apply” 명령을 수행하여 AWS 리소스 생성
| C:\eks-workshop\lab-setup> terraform apply --auto-approve ...(생략) Apply complete! Resources: 5 added, 0 changed, 0 destroyed. Outputs: k8s-cp_public-ip = "13.209.150.124" # “k8s-cp” 인스턴스에 Private Key 복사 C:\eks-workshop\lab-setup> scp -i edukey edukey root@13.209.150.124:~/.ssh/id_rsa The authenticity of host '13.209.150.124 (13.209.150.124)' can't be established. ED25519 key fingerprint is SHA256:NpdHkxNyzqWJjVLVNjW8kC+Vl7iqlH9u5kqR4aiatb8. This key is not known by any other names. Are you sure you want to continue connecting (yes/no/[fingerprint])? Warning: Permanently added '13.209.150.124' (ED25519) to the list of known hosts. edukey 100% 2602 363.0KB/s 00:00 |
k8s-cp_public-ip 와 SSH Private Key를 이용하여 k8s-cp EC2에 root 계정으로 SSH 로그인
k8s-cp 노드에서 k8s-w1, k8s-w2에 SSH Key를 이용한 로그인 설정
| [root@k8s-cp ~]# chmod 400 ~/.ssh/id_rsa [root@k8s-cp ~]# ssh k8s-w1 [root@k8s-cp ~]# ssh k8s-w2 |
현재 K8S는 Control-plane 노드만 구성되어 있습니다.
| [root@k8s-cp ~]# kubectl get no NAME STATUS ROLES AGE VERSION k8s-cp Ready control-plane 4m47s v1.31.4 |
join-command를 생성한 후 k8s-w1, k8s-w2 노드를 클러스터에 조인합니다.
| [root@k8s-cp ~]# kubeadm token create --print-join-command kubeadm join 172.31.1.10:6443 --token njseik.wna0odfa83txmr2l --discovery-token-ca-cert-hash sha256:ce6b8504ea710f82aeea8bf3acadd82df870544134de6de1339f19a446c03a1f [root@k8s-cp ~]# ssh k8s-w1 [root@k8s-w1 ~]# kubeadm join 172.31.1.10:6443 --token njseik.wna0odfa83txmr2l --discovery-token-ca-cert-hash sha256:ce6b8504ea710f82aeea8bf3acadd82df870544134de6de1339f19a446c03a1f ...(생략)... Run 'kubectl get nodes' on the control-plane to see this node join the cluster. [root@k8s-w1 ~]# exit [root@k8s-cp ~]# ssh k8s-w2 [root@k8s-w2 ~]# kubeadm join 172.31.1.10:6443 --token njseik.wna0odfa83txmr2l --discovery-token-ca-cert-hash sha256:ce6b8504ea710f82aeea8bf3acadd82df870544134de6de1339f19a446c03a1f ...(생략)... Run 'kubectl get nodes' on the control-plane to see this node join the cluster. [root@k8s-w2 ~]# exit [root@k8s-cp ~]# kubectl get no NAME STATUS ROLES AGE VERSION k8s-cp Ready control-plane 8m48s v1.31.4 k8s-w1 Ready <none> 56s v1.31.4 k8s-w2 Ready <none> 21s v1.31.4 |
1.2. 방명록 샘플어플리케이션 소개
실습에서 사용할 간단한 방명록 웹플리케이션입니다.
게시글 저장을 위한 MySQL DBMS는 아래와 같이 도커 컨테이너로 k8s-cp 머신(172.31.1.10) 에서 실행해 놓겠습니다.
| [root@k8s-cp ~]# yum install -y docker-ce-27.4.1 [root@k8s-cp ~]# systemctl enable --now docker [root@k8s-cp ~]# docker container run -d \ --name=guestbookdb \ --restart=always \ --network=host \ -e MYSQL_ROOT_PASSWORD=jadecross \ -e MYSQL_DATABASE=guestbook \ mysql:8 |
방명록 웹어플리케이션은 게시글을 등록하면서 파일을 첨부할 수 있는 기능이 있는데, 파일을 첨부하게 되면 컨테이너 파일시스템에 업로드가 됩니다.
쿠버네티스 환경에서 컨테이너는 POD라는 논리적인 컴퓨터 안에서 실행되게 되는데, POD는 일시적인 리소스입니다. POD에 문제가 있으면 지워버리고, 그러면 Self-Healing 되면서 레플리카 개수를 맞추게 됩니다.
이러한 상황에서 방명록 POD를 삭제하면 첨부했던 파일들도 모두 사라지게 되어 첨부한 파일의 영속성을 보장하지 못합니다.
POD가 삭제되거나 비정상일 때도
업로드한 첨부파일을 안전하게
영구히 보존하는 방법은?
이러한 문제점을 해결하기 위해서 K8S 에서는 POD 에 Volume 을 선언하고, Container는 선언된 Volume을 레퍼런스 하면서 컨테이너 파일 시스템에 마운트하는 방식을 이용합니다.
쿠버네티스가 지원하는 볼륨을 타입으로 구분해 보면 아래와 같이 3가지 타입으로 구분해 볼 수 있습니다.
위의 3가지 타입중 emptDir 과 HostPath 타입의 방식은 기본적으로 공유스토리지로 사용할수가 없어서, Network을 지원하는 Storage를 사용할 수 밖에 없습니다.
실습환경에서 사용가능한 NFS 를 이용한 볼륨 실습을 해 보겠습니다.
아래와 같이 k8s-cp 머신에서 NFS 공유폴더 설정을 합니다.
| [root@k8s-cp ~]# dnf install -y nfs-utils ◀━ NFS 패키지 설치 [root@k8s-cp ~]# systemctl enable --now nfs-server [root@k8s-cp ~]# mkdir -p /nfs-share/guestbook-data ◀━ 공유 디렉토리 생성 [root@k8s-cp ~]# chmod 777 /nfs-share/guestbook-data [root@k8s-cp ~]# vi /etc/exports ◀━ 공유 디렉토리 설정파일 편집 /nfs-share/guestbook-data 172.31.0.0/20(rw,sync,no_root_squash) [root@k8s-cp ~]# systemctl restart nfs-server ◀━ nfs 서비스 시작 [root@k8s-cp ~]# exportfs -v ◀━ nfs 공유정보 확인 /nfs-share/guestbook-data 172.31.0.0/20(sync,...) |
NFS 스토리지를 사용하는 방명록 POD를 생성합니다.
# guestbook-pod-nfs.yaml
apiVersion: v1
kind: Pod
metadata:
name: guestbook-nfs
spec:
volumes:
- name: guestbook-data
nfs:
server: 172.31.1.10
path: /nfs-share/guestbook-data
containers:
- image: yu3papa/k8s_guestbook:1.0
name: guestbook
volumeMounts:
- name: guestbook-data
mountPath: /app/upload
ports:
- containerPort: 8080
protocol: TCP
| [root@k8s-cp ~]# k apply -f guestbook-pod-nfs.yaml pod/guestbook-nfs created |
생성된 POD의 IP를 확인하고 웹화면에서 파일을 첨부하면 컨테이너 파일시스템이 첨부가 되지만 최종적으로 NFS 공유폴더를 제공하는 k8s-cp 머신에 파일 저장됩니다.
이런 상태에서 POD를 삭제하여도 첨부한 파일은 NFS 공유폴더에 보존되어 있습니다.
Network 타입을 지원하는 스토리지는 정말 많습니다.
위와 같은 상황에서 POD YAML파일을 작성하는 개발자는 NFS 스토리지에 대한 기술을 알고 있어야 합니다. 또는 AWS 환경을 이용한다면 EBS에 대한 지식도 있어야 합니다.
이러한 상황에서 쿠버네티스는 "스토리지 기술에서 POD 를 분리"하는 방법을 고안하게 됩니다.
이러한 맥락에서 PersistentVolume과 PersistentVolumeClaim 리소스를 이용합니다.
전체적인 사용흐름은 아래와 같으며, 핵심은 "개발자가 스토리지 인프라 세부 사항을 처리하지 않고도 쿠버네티스 클러스터의 스토리지를 요청할 수 있게 하는것"입니다.
PV, PVC를 사용하는 방명록을 구성해 보겠습니다.
| guestbook-pv-nfs.yaml | guestbook-pvc.yaml | guestbook-pod-pvc.yaml |
| apiVersion: v1 kind: PersistentVolume metadata: name: guestbook-pv labels: disk: ssd spec: capacity: storage: 1Gi accessModes: - ReadWriteOnce - ReadOnlyMany persistentVolumeReclaimPolicy: Retain nfs: path: /nfs-share/guestbook-pv server: 172.31.1.10 |
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: guestbook-pvc spec: resources: requests: storage: 1Gi selector: matchLabels: disk: ssd accessModes: - ReadWriteOnce storageClassName: "" |
apiVersion: v1 kind: Pod metadata: name: guestbook-pod-pvc spec: containers: - image: yu3papa/k8s_guestbook:1.0 name: guestbook volumeMounts: - name: guestbook-data mountPath: /app/upload ports: - containerPort: 8080 protocol: TCP volumes: - name: guestbook-data persistentVolumeClaim: claimName: guestbook-pvc |
여기에서 또 해결해야할 이슈가 있습니다. 클러스터가 수 많은 스토리지 기술의 전문가 일 수가 없습니다. 스토리지 전문가는 해당 스토리지 벤더의 엔지니어가 전문가입니다.
쿠버네티스는 이러한 문제를 해결하기 위해 CSI (Contaier Storage Interface) 를 제안합니다.
- CSI 를 사용하면, K8S 의 공통화된 CSI 인터페이스를 통해 다양한 프로바이더를 사용할 수 있다.
해당 스토리지 벤더들이 CSI 규격에 맞는 Provisioner를 쿠버네티스에 배포하고 클러스터 관리자는 SC(Storage Class) 리소스에 Provisioner를 매핑해 놓으면 PVC 리소스만 만들면 자동으로 PV 및 실제 스토리지도 만들어지게 되어 "Dynamic PersistentVolume 프로비저닝" 이 가능해 집니다. 도식화해보면 아래와 같습니다.
온프레미스환경에서 NFS를 이용하여 "Dynamic PersistentVolume 프로비저닝"을 할 수 있도록 K8S Sig그룹에서 "nfs-subdir-external-provisioner"를 제공합니다.
GitHub - kubernetes-sigs/nfs-subdir-external-provisioner: Dynamic sub-dir volume provisioner on a remote NFS server.
Dynamic sub-dir volume provisioner on a remote NFS server. - kubernetes-sigs/nfs-subdir-external-provisioner
github.com
1.2. K8S v1.31 클러스터 삭제
테라폼을 이용하여 지금까지 실습한 환경을 삭제합니다.
| C:\eks-workshop\lab-setup> terraform destroy --auto-approve ...(생략)... aws_instance.k8s-cp: Still destroying... [id=i-0311028a5562d2fc2, 1m1s elapsed] aws_instance.k8s-cp: Destruction complete after 1m1s aws_key_pair.edukey-pair: Destroying... [id=edukey] aws_key_pair.edukey-pair: Destruction complete after 0s Destroy complete! Resources: 5 destroyed. |
2. EKS 실습환경 구성
아래와 같은 환경으로 EKS 실습환경을 구성합니다.
2개의 VPC(EKS 배포, 운영용 구분), myeks-vpc의 public 에 EFS 추가
- myeks-vpc 에 각기 AZ를 사용하는 퍼블릭/프라이빗 서브넷 배치
- operator-vpc 에 AZ1를 사용하는 퍼블릭/프라이빗 서브넷 배치 : 172.20.1.100 운영서버 EC2 배포
- 내부 통신을 위한 VPC Peering 배치
2.1. AWS CloudFormation 을 통해 기본 실습 환경 배포
제 실습환경은 Rocky Linux 9.5 환경에서 진행합니다.
aws CLI 도구를 설정하고 configure 작업을 수행합니다.
| [yu3papa@iworks ~]$ curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" [yu3papa@iworks ~]$ unzip awscliv2.zip [yu3papa@iworks ~]$ sudo ./aws/install [yu3papa@iworks ~]$ aws --version aws-cli/2.24.10 Python/3.12.9 Linux/5.14.0-503.23.1.el9_5.x86_64 exe/x86_64.rocky.9 [yu3papa@iworks ~]$ aws configure AWS Access Key ID [None]: AKIAVVZO********OVO56J AWS Secret Access Key [None]: wDT3QQyYiljs2i***********9zIGTiIH Default region name [None]: ap-northeast-2 Default output format [None]: json |
① SSH 키페어를 생성하고 Private Key를 로컬에 잘 보관합니다.
② 보안을 위해 작업 EC2 에 본인의 집에서만 접속이 되되록 "내 IP"를 확인합니다.
| C:\> curl -s ipinfo.io/ip 221.149.127.57 |
③ CloudFormation 을 이용하여 실습환경 배포
| # yaml 파일 다운로드 [yu3papa@iworks ~]$ curl -O https://s3.ap-northeast-2.amazonaws.com/cloudformation.cloudneta.net/K8S/myeks-3week.yaml # 배포 # aws cloudformation deploy --template-file myeks-1week.yaml --stack-name mykops --parameter-overrides KeyName=<My SSH Keyname> SgIngressSshCidr=<My Home Public IP Address>/32 --region <리전> [yu3papa@iworks ~]$ aws cloudformation deploy \ --template-file ./myeks-3week.yaml \ --stack-name myeks \ --parameter-overrides KeyName=aews-yu3papa SgIngressSshCidr=$(curl -s ipinfo.io/ip)/32 \ --region ap-northeast-2 Waiting for changeset to be created.. Waiting for stack create/update to complete Successfully created/updated stack - myeks # CloudFormation 스택 배포 완료 후 운영서버 EC2 IP 출력 [yu3papa@iworks ~]$ aws cloudformation describe-stacks --stack-name myeks --query 'Stacks[*].Outputs[*].OutputValue' --output text 15.164.244.197 # 운영서버 EC2 에 SSH 접속 [yu3papa@iworks ~]$ ssh -i aews-yu3papa.pem ec2-user@15.164.244.197 , #_ ~\_ ####_ Amazon Linux 2 ~~ \_#####\ ~~ \###| AL2 End of Life is 2026-06-30. ~~ \#/ ___ ~~ V~' '-> ~~~ / A newer version of Amazon Linux is available! ~~._. _/ _/ _/ Amazon Linux 2023, GA and supported until 2028-03-15. _/m/' https://aws.amazon.com/linux/amazon-linux-2023/ # 운영서버에서도 aws configure 작업을 수행 [root@operator-host ~]# aws configure AWS Access Key ID [None]: AKIAVVZO********OVO56J AWS Secret Access Key [None]: wDT3QQyYiljs2i***********9zIGTiIH Default region name [None]: ap-northeast-2 Default output format [None]: json |
위 CloudFormation 을 이용하면 EFS 가 하나 만들어지며, 향후 EKS 에서 볼륨으로 사용할 예정입니다.
EFS 상세정보에서 "네트워크" 탭을 확인하면 향후 EKS 워커노드에서 연결할때 사용할 IP를 확인할 수 있습니다.
2.2. eksctl 을 통해 EKS 배포
Child로 실행한 쉘에서 EKS 관련 환경을 공유하기 위해 실습에 필요한 환경변수를 설정하고 .bashrc 에 추가하여 실습을 좀 더 원할하게 진행하겠습니다.
| [root@operator-host ~]# export CLUSTER_NAME=myeks # myeks-VPC/Subnet 정보 확인 및 변수 지정 [root@operator-host ~]# export VPCID=$(aws ec2 describe-vpcs --filters "Name=tag:Name,Values=$CLUSTER_NAME-VPC" --query 'Vpcs[*].VpcId' --output text) [root@operator-host ~]# echo $VPCID vpc-040caab837037a18c [root@operator-host ~]# export PubSubnet1=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet1" --query "Subnets[0].[SubnetId]" --output text) [root@operator-host ~]# export PubSubnet2=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet2" --query "Subnets[0].[SubnetId]" --output text) [root@operator-host ~]# export PubSubnet3=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet3" --query "Subnets[0].[SubnetId]" --output text) [root@operator-host ~]# echo $PubSubnet1 $PubSubnet2 $PubSubnet3 subnet-08b83e58f1cad6098 subnet-0ad3d1ce95ad8a965 subnet-0f643cf03e72dc812 [root@operator-host ~]# SSHKEYNAME=aews-yu3papa |
myeks.yaml 파일 작성
cat << EOF > myeks.yaml
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: myeks
region: ap-northeast-2
version: "1.31"
iam:
withOIDC: true # enables the IAM OIDC provider as well as IRSA for the Amazon CNI plugin
serviceAccounts: # service accounts to create in the cluster. See IAM Service Accounts
- metadata:
name: aws-load-balancer-controller
namespace: kube-system
wellKnownPolicies:
awsLoadBalancerController: true
vpc:
cidr: 192.168.0.0/16
clusterEndpoints:
privateAccess: true # if you only want to allow private access to the cluster
publicAccess: true # if you want to allow public access to the cluster
id: $VPCID
subnets:
public:
ap-northeast-2a:
az: ap-northeast-2a
cidr: 192.168.1.0/24
id: $PubSubnet1
ap-northeast-2b:
az: ap-northeast-2b
cidr: 192.168.2.0/24
id: $PubSubnet2
ap-northeast-2c:
az: ap-northeast-2c
cidr: 192.168.3.0/24
id: $PubSubnet3
addons:
- name: vpc-cni # no version is specified so it deploys the default version
version: latest # auto discovers the latest available
attachPolicyARNs: # attach IAM policies to the add-on's service account
- arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
configurationValues: |-
enableNetworkPolicy: "true"
- name: kube-proxy
version: latest
- name: coredns
version: latest
- name: metrics-server
version: latest
managedNodeGroups:
- amiFamily: AmazonLinux2023
desiredCapacity: 3
iam:
withAddonPolicies:
certManager: true # Enable cert-manager
externalDNS: true # Enable ExternalDNS
instanceType: t3.medium
preBootstrapCommands:
# install additional packages
- "dnf install nvme-cli links tree tcpdump sysstat ipvsadm ipset bind-utils htop -y"
labels:
alpha.eksctl.io/cluster-name: myeks
alpha.eksctl.io/nodegroup-name: ng1
maxPodsPerNode: 100
maxSize: 3
minSize: 3
name: ng1
ssh:
allow: true
publicKeyName: $SSHKEYNAME
tags:
alpha.eksctl.io/nodegroup-name: ng1
alpha.eksctl.io/nodegroup-type: managed
volumeIOPS: 3000
volumeSize: 120
volumeThroughput: 125
volumeType: gp3
EOF
최종 yaml 로 eks 를 배포합니다.
| [root@operator-host ~]# eksctl create cluster -f myeks.yaml --verbose 4 |
| # 실행 로그 2025-02-22 19:57:03 [▶] Setting credentials expiry window to 30 minutes 2025-02-22 19:57:03 [▶] role ARN for the current session is "arn:aws:iam::390402552337:root" 2025-02-22 19:57:03 [ℹ] eksctl version 0.204.0 2025-02-22 19:57:03 [ℹ] using region ap-northeast-2 2025-02-22 19:57:03 [✔] using existing VPC (vpc-040caab837037a18c) and subnets (private:map[] public:map[ap-northeast-2a:{subnet-08b83e58f1cad6098 ap-northeast-2a 192.168.1.0/24 0 } ap-northeast-2b:{subnet-0ad3d1ce95ad8a965 ap-northeast-2b 192.168.2.0/24 0 } ap-northeast-2c:{subnet-0f643cf03e72dc812 ap-northeast-2c 192.168.3.0/24 0 }]) 2025-02-22 19:57:03 [!] custom VPC/subnets will be used; if resulting cluster doesn't function as expected, make sure to review the configuration of VPC/subnets 2025-02-22 19:57:03 [ℹ] nodegroup "ng1" will use "" [AmazonLinux2023/1.31] 2025-02-22 19:57:03 [ℹ] using EC2 key pair "aews-yu3papa" 2025-02-22 19:57:03 [ℹ] using Kubernetes version 1.31 2025-02-22 19:57:03 [ℹ] creating EKS cluster "myeks" in "ap-northeast-2" region with managed nodes 2025-02-22 19:57:03 [▶] cfg.json = \ { "kind": "ClusterConfig", "apiVersion": "eksctl.io/v1alpha5", "metadata": { "name": "myeks", "region": "ap-northeast-2", "version": "1.31" }, "iam": { "withOIDC": true, "serviceAccounts": [ { "metadata": { "name": "aws-load-balancer-controller", "namespace": "kube-system" }, "wellKnownPolicies": { "imageBuilder": false, "autoScaler": false, "awsLoadBalancerController": true, "externalDNS": false, "certManager": false, "ebsCSIController": false, "efsCSIController": false } } ], "vpcResourceControllerPolicy": true }, "accessConfig": { "authenticationMode": "API_AND_CONFIG_MAP" }, "vpc": { "id": "vpc-040caab837037a18c", "cidr": "192.168.0.0/16", "subnets": { "public": { "ap-northeast-2a": { "id": "subnet-08b83e58f1cad6098", "az": "ap-northeast-2a", "cidr": "192.168.1.0/24" }, "ap-northeast-2b": { "id": "subnet-0ad3d1ce95ad8a965", "az": "ap-northeast-2b", "cidr": "192.168.2.0/24" }, "ap-northeast-2c": { "id": "subnet-0f643cf03e72dc812", "az": "ap-northeast-2c", "cidr": "192.168.3.0/24" } } }, "manageSharedNodeSecurityGroupRules": true, "nat": { "gateway": "Single" }, "clusterEndpoints": { "privateAccess": true, "publicAccess": true } }, "addons": [ { "name": "vpc-cni", "version": "latest", "attachPolicyARNs": [ "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy" ], "wellKnownPolicies": { "imageBuilder": false, "autoScaler": false, "awsLoadBalancerController": false, "externalDNS": false, "certManager": false, "ebsCSIController": false, "efsCSIController": false }, "configurationValues": "enableNetworkPolicy: \"true\"" }, { "name": "kube-proxy", "version": "latest", "wellKnownPolicies": { "imageBuilder": false, "autoScaler": false, "awsLoadBalancerController": false, "externalDNS": false, "certManager": false, "ebsCSIController": false, "efsCSIController": false } }, { "name": "coredns", "version": "latest", "wellKnownPolicies": { "imageBuilder": false, "autoScaler": false, "awsLoadBalancerController": false, "externalDNS": false, "certManager": false, "ebsCSIController": false, "efsCSIController": false } }, { "name": "metrics-server", "version": "latest", "wellKnownPolicies": { "imageBuilder": false, "autoScaler": false, "awsLoadBalancerController": false, "externalDNS": false, "certManager": false, "ebsCSIController": false, "efsCSIController": false } } ], "addonsConfig": {}, "privateCluster": { "enabled": false, "skipEndpointCreation": false }, "managedNodeGroups": [ { "name": "ng1", "amiFamily": "AmazonLinux2023", "instanceType": "t3.medium", "desiredCapacity": 3, "minSize": 3, "maxSize": 3, "volumeSize": 120, "ssh": { "allow": true, "publicKeyName": "aews-yu3papa" }, "labels": { "alpha.eksctl.io/cluster-name": "myeks", "alpha.eksctl.io/nodegroup-name": "ng1" }, "privateNetworking": false, "tags": { "alpha.eksctl.io/nodegroup-name": "ng1", "alpha.eksctl.io/nodegroup-type": "managed" }, "iam": { "withAddonPolicies": { "imageBuilder": false, "autoScaler": false, "externalDNS": true, "certManager": true, "appMesh": null, "appMeshPreview": null, "ebs": false, "fsx": false, "efs": false, "awsLoadBalancerController": false, "albIngress": false, "xRay": false, "cloudWatch": false } }, "securityGroups": { "withShared": null, "withLocal": null }, "maxPodsPerNode": 100, "volumeType": "gp3", "volumeIOPS": 3000, "volumeThroughput": 125, "preBootstrapCommands": [ "dnf install nvme-cli links tree tcpdump sysstat ipvsadm ipset bind-utils htop -y" ], "disableIMDSv1": true, "disablePodIMDS": false, "instanceSelector": {}, "releaseVersion": "" } ], "availabilityZones": [ "ap-northeast-2a", "ap-northeast-2c", "ap-northeast-2b" ] } 2025-02-22 19:57:03 [ℹ] 1 nodegroup (ng1) was included (based on the include/exclude rules) 2025-02-22 19:57:03 [ℹ] will create a CloudFormation stack for cluster itself and 1 managed nodegroup stack(s) 2025-02-22 19:57:03 [ℹ] if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=ap-northeast-2 --cluster=myeks' 2025-02-22 19:57:03 [ℹ] Kubernetes API endpoint access will use provided values {publicAccess=true, privateAccess=true} for cluster "myeks" in "ap-northeast-2" 2025-02-22 19:57:03 [ℹ] CloudWatch logging will not be enabled for cluster "myeks" in "ap-northeast-2" 2025-02-22 19:57:03 [ℹ] you can enable it with 'eksctl utils update-cluster-logging --enable-types={SPECIFY-YOUR-LOG-TYPES-HERE (e.g. all)} --region=ap-northeast-2 --cluster=myeks' 2025-02-22 19:57:03 [ℹ] 2 sequential tasks: { create cluster control plane "myeks", 2 sequential sub-tasks: { 5 sequential sub-tasks: { 1 task: { create addons }, wait for control plane to become ready, associate IAM OIDC provider, 2 sequential sub-tasks: { create IAM role for serviceaccount "kube-system/aws-load-balancer-controller", create serviceaccount "kube-system/aws-load-balancer-controller", }, update VPC CNI to use IRSA if required, }, create managed nodegroup "ng1", } } 2025-02-22 19:57:03 [▶] started task: create cluster control plane "myeks" 2025-02-22 19:57:03 [ℹ] building cluster stack "eksctl-myeks-cluster" 2025-02-22 19:57:03 [▶] CreateStackInput = &cloudformation.CreateStackInput{StackName:(*string)(0xc000f06a40), Capabilities:[]types.Capability{"CAPABILITY_IAM"}, ClientRequestToken:(*string)(nil), DisableRollback:(*bool)(0xc000f12b28), EnableTerminationProtection:(*bool)(nil), NotificationARNs:[]string(nil), OnFailure:"", Parameters:[]types.Parameter(nil), ResourceTypes:[]string(nil), RetainExceptOnCreate:(*bool)(nil), RoleARN:(*string)(nil), RollbackConfiguration:(*types.RollbackConfiguration)(nil), StackPolicyBody:(*string)(nil), StackPolicyURL:(*string)(nil), Tags:[]types.Tag{types.Tag{Key:(*string)(0xc000ee4e30), Value:(*string)(0xc000ee4e40), noSmithyDocumentSerde:document.NoSerde{}}, types.Tag{Key:(*string)(0xc000ee4e50), Value:(*string)(0xc000ee4e60), noSmithyDocumentSerde:document.NoSerde{}}, types.Tag{Key:(*string)(0xc000ee4e70), Value:(*string)(0xc000ee4e80), noSmithyDocumentSerde:document.NoSerde{}}, types.Tag{Key:(*string)(0xc000f07920), Value:(*string)(0xc000f07930), noSmithyDocumentSerde:document.NoSerde{}}}, TemplateBody:(*string)(0xc000f07940), TemplateURL:(*string)(nil), TimeoutInMinutes:(*int32)(nil), noSmithyDocumentSerde:document.NoSerde{}} 2025-02-22 19:57:03 [ℹ] deploying stack "eksctl-myeks-cluster" 2025-02-22 19:57:33 [ℹ] waiting for CloudFormation stack "eksctl-myeks-cluster" 2025-02-22 19:58:03 [ℹ] waiting for CloudFormation stack "eksctl-myeks-cluster" 2025-02-22 19:59:03 [ℹ] waiting for CloudFormation stack "eksctl-myeks-cluster" 2025-02-22 20:00:04 [ℹ] waiting for CloudFormation stack "eksctl-myeks-cluster" 2025-02-22 20:01:04 [ℹ] waiting for CloudFormation stack "eksctl-myeks-cluster" 2025-02-22 20:02:04 [ℹ] waiting for CloudFormation stack "eksctl-myeks-cluster" 2025-02-22 20:03:04 [ℹ] waiting for CloudFormation stack "eksctl-myeks-cluster" 2025-02-22 20:04:04 [ℹ] waiting for CloudFormation stack "eksctl-myeks-cluster" 2025-02-22 20:05:04 [ℹ] waiting for CloudFormation stack "eksctl-myeks-cluster" 2025-02-22 20:06:04 [ℹ] waiting for CloudFormation stack "eksctl-myeks-cluster" 2025-02-22 20:06:04 [▶] processing stack outputs 2025-02-22 20:06:04 [▶] completed task: create cluster control plane "myeks" 2025-02-22 20:06:04 [▶] started task: 2 sequential sub-tasks: { 5 sequential sub-tasks: { 1 task: { create addons }, wait for control plane to become ready, associate IAM OIDC provider, 2 sequential sub-tasks: { create IAM role for serviceaccount "kube-system/aws-load-balancer-controller", create serviceaccount "kube-system/aws-load-balancer-controller", }, update VPC CNI to use IRSA if required, }, create managed nodegroup "ng1", } 2025-02-22 20:06:04 [▶] started task: 5 sequential sub-tasks: { 1 task: { create addons }, wait for control plane to become ready, associate IAM OIDC provider, 2 sequential sub-tasks: { create IAM role for serviceaccount "kube-system/aws-load-balancer-controller", create serviceaccount "kube-system/aws-load-balancer-controller", }, update VPC CNI to use IRSA if required, } 2025-02-22 20:06:04 [▶] started task: 1 task: { create addons } 2025-02-22 20:06:04 [▶] started task: create addons 2025-02-22 20:06:04 [▶] cluster = &types.Cluster{AccessConfig:(*types.AccessConfigResponse)(0xc000571620), Arn:(*string)(0xc000705890), CertificateAuthority:(*types.Certificate)(0xc0007058e0), ClientRequestToken:(*string)(nil), ComputeConfig:(*types.ComputeConfigResponse)(nil), ConnectorConfig:(*types.ConnectorConfigResponse)(nil), CreatedAt:time.Date(2025, time.February, 22, 10, 57, 28, 280000000, time.UTC), EncryptionConfig:[]types.EncryptionConfig(nil), Endpoint:(*string)(0xc000705800), Health:(*types.ClusterHealth)(0xc000571640), Id:(*string)(nil), Identity:(*types.Identity)(0xc0007058b0), KubernetesNetworkConfig:(*types.KubernetesNetworkConfigResponse)(0xc000905470), Logging:(*types.Logging)(0xc0005716e0), Name:(*string)(0xc000705810), OutpostConfig:(*types.OutpostConfigResponse)(nil), PlatformVersion:(*string)(0xc000705900), RemoteNetworkConfig:(*types.RemoteNetworkConfigResponse)(nil), ResourcesVpcConfig:(*types.VpcConfigResponse)(0xc0008100e0), RoleArn:(*string)(0xc0007058a0), Status:"ACTIVE", StorageConfig:(*types.StorageConfigResponse)(nil), Tags:map[string]string{"Name":"eksctl-myeks-cluster/ControlPlane", "alpha.eksctl.io/cluster-name":"myeks", "alpha.eksctl.io/cluster-oidc-enabled":"true", "alpha.eksctl.io/eksctl-version":"0.204.0", "aws:cloudformation:logical-id":"ControlPlane", "aws:cloudformation:stack-id":"arn:aws:cloudformation:ap-northeast-2:390402552337:stack/eksctl-myeks-cluster/bfda6530-f10b-11ef-b7e5-0a7559f07843", "aws:cloudformation:stack-name":"eksctl-myeks-cluster", "eksctl.cluster.k8s.io/v1alpha1/cluster-name":"myeks"}, UpgradePolicy:(*types.UpgradePolicyResponse)(0xc0007d35c0), Version:(*string)(0xc000705820), ZonalShiftConfig:(*types.ZonalShiftConfigResponse)(nil), noSmithyDocumentSerde:document.NoSerde{}} 2025-02-22 20:06:05 [▶] resolve conflicts set to OVERWRITE 2025-02-22 20:06:05 [▶] addon: &{vpc-cni v1.19.2-eksbuild.5 [arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy] map[] {false false false false false false false} map[] <nil> false enableNetworkPolicy: "true" true [] [] []} 2025-02-22 20:06:06 [!] IRSA config is set for "vpc-cni" addon, but since OIDC is disabled on the cluster, eksctl cannot configure the requested permissions; the recommended way to provide IAM permissions for "vpc-cni" addon is via pod identity associations; after addon creation is completed, add all recommended policies to the config file, under `addon.PodIdentityAssociations`, and run `eksctl update addon` 2025-02-22 20:06:06 [ℹ] creating addon: vpc-cni 2025-02-22 20:06:06 [▶] EKS Create Addon output: {%!s(*string=0xc0005248b0) %!s(*string=0xc0005248c0) %!s(*string=0xc0005248a0) %!s(*string=0xc0005248d0) %!s(*string=0xc000524890) 2025-02-22 11:06:06.513 +0000 UTC %!s(*types.AddonHealth=&{[] {}}) %!s(*types.MarketplaceInformation=<nil>) 2025-02-22 11:06:06.528 +0000 UTC %!s(*string=<nil>) [] %!s(*string=<nil>) %!s(*string=<nil>) CREATING map[] {}} 2025-02-22 20:06:06 [ℹ] successfully created addon: vpc-cni 2025-02-22 20:06:06 [▶] resolve conflicts set to OVERWRITE 2025-02-22 20:06:06 [▶] addon: &{kube-proxy v1.31.3-eksbuild.2 [] map[] {false false false false false false false} map[] <nil> false true [] [] []} 2025-02-22 20:06:06 [ℹ] creating addon: kube-proxy 2025-02-22 20:06:07 [▶] EKS Create Addon output: {%!s(*string=0xc0006530f0) %!s(*string=0xc0006530c0) %!s(*string=0xc0006530e0) %!s(*string=0xc0006530d0) %!s(*string=<nil>) 2025-02-22 11:06:07.128 +0000 UTC %!s(*types.AddonHealth=&{[] {}}) %!s(*types.MarketplaceInformation=<nil>) 2025-02-22 11:06:07.142 +0000 UTC %!s(*string=<nil>) [] %!s(*string=<nil>) %!s(*string=<nil>) CREATING map[] {}} 2025-02-22 20:06:07 [ℹ] successfully created addon: kube-proxy 2025-02-22 20:06:07 [▶] resolve conflicts set to OVERWRITE 2025-02-22 20:06:07 [▶] addon: &{coredns v1.11.4-eksbuild.2 [] map[] {false false false false false false false} map[] <nil> false true [] [] []} 2025-02-22 20:06:07 [ℹ] creating addon: coredns 2025-02-22 20:06:08 [▶] EKS Create Addon output: {%!s(*string=0xc000a7da60) %!s(*string=0xc000a7da40) %!s(*string=0xc000a7da30) %!s(*string=0xc000a7da50) %!s(*string=<nil>) 2025-02-22 11:06:08.086 +0000 UTC %!s(*types.AddonHealth=&{[] {}}) %!s(*types.MarketplaceInformation=<nil>) 2025-02-22 11:06:08.104 +0000 UTC %!s(*string=<nil>) [] %!s(*string=<nil>) %!s(*string=<nil>) CREATING map[] {}} 2025-02-22 20:06:08 [ℹ] successfully created addon: coredns 2025-02-22 20:06:08 [▶] resolve conflicts set to OVERWRITE 2025-02-22 20:06:08 [▶] addon: &{metrics-server v0.7.2-eksbuild.2 [] map[] {false false false false false false false} map[] <nil> false true [] [] []} 2025-02-22 20:06:08 [ℹ] creating addon: metrics-server 2025-02-22 20:06:08 [▶] EKS Create Addon output: {%!s(*string=0xc000db5050) %!s(*string=0xc000db5040) %!s(*string=0xc000db5030) %!s(*string=0xc000db5020) %!s(*string=<nil>) 2025-02-22 11:06:08.682 +0000 UTC %!s(*types.AddonHealth=&{[] {}}) %!s(*types.MarketplaceInformation=<nil>) 2025-02-22 11:06:08.698 +0000 UTC %!s(*string=<nil>) [] %!s(*string=<nil>) %!s(*string=<nil>) CREATING map[] {}} 2025-02-22 20:06:08 [ℹ] successfully created addon: metrics-server 2025-02-22 20:06:08 [▶] completed task: create addons 2025-02-22 20:06:08 [▶] completed task: 1 task: { create addons } 2025-02-22 20:06:08 [▶] started task: wait for control plane to become ready 2025-02-22 20:08:09 [▶] cluster = &types.Cluster{AccessConfig:(*types.AccessConfigResponse)(0xc000bf95c0), Arn:(*string)(0xc000ee5050), CertificateAuthority:(*types.Certificate)(0xc000ee4fc0), ClientRequestToken:(*string)(nil), ComputeConfig:(*types.ComputeConfigResponse)(nil), ConnectorConfig:(*types.ConnectorConfigResponse)(nil), CreatedAt:time.Date(2025, time.February, 22, 10, 57, 28, 280000000, time.UTC), EncryptionConfig:[]types.EncryptionConfig(nil), Endpoint:(*string)(0xc000ee4f90), Health:(*types.ClusterHealth)(0xc000bf9600), Id:(*string)(nil), Identity:(*types.Identity)(0xc000ee5020), KubernetesNetworkConfig:(*types.KubernetesNetworkConfigResponse)(0xc000f3fd10), Logging:(*types.Logging)(0xc000bf9620), Name:(*string)(0xc000ee4eb0), OutpostConfig:(*types.OutpostConfigResponse)(nil), PlatformVersion:(*string)(0xc000ee4fe0), RemoteNetworkConfig:(*types.RemoteNetworkConfigResponse)(nil), ResourcesVpcConfig:(*types.VpcConfigResponse)(0xc0004a3ea0), RoleArn:(*string)(0xc000ee4ff0), Status:"ACTIVE", StorageConfig:(*types.StorageConfigResponse)(nil), Tags:map[string]string{"Name":"eksctl-myeks-cluster/ControlPlane", "alpha.eksctl.io/cluster-name":"myeks", "alpha.eksctl.io/cluster-oidc-enabled":"true", "alpha.eksctl.io/eksctl-version":"0.204.0", "aws:cloudformation:logical-id":"ControlPlane", "aws:cloudformation:stack-id":"arn:aws:cloudformation:ap-northeast-2:390402552337:stack/eksctl-myeks-cluster/bfda6530-f10b-11ef-b7e5-0a7559f07843", "aws:cloudformation:stack-name":"eksctl-myeks-cluster", "eksctl.cluster.k8s.io/v1alpha1/cluster-name":"myeks"}, UpgradePolicy:(*types.UpgradePolicyResponse)(0xc000ecca68), Version:(*string)(0xc000ee4ec0), ZonalShiftConfig:(*types.ZonalShiftConfigResponse)(nil), noSmithyDocumentSerde:document.NoSerde{}} 2025-02-22 20:08:09 [▶] completed task: wait for control plane to become ready 2025-02-22 20:08:09 [▶] started task: associate IAM OIDC provider 2025-02-22 20:08:09 [▶] completed task: associate IAM OIDC provider 2025-02-22 20:08:09 [▶] started task: 2 sequential sub-tasks: { create IAM role for serviceaccount "kube-system/aws-load-balancer-controller", create serviceaccount "kube-system/aws-load-balancer-controller", } 2025-02-22 20:08:09 [▶] waiting for 1 parallel tasks to complete 2025-02-22 20:08:09 [▶] started task: 2 sequential sub-tasks: { create IAM role for serviceaccount "kube-system/aws-load-balancer-controller", create serviceaccount "kube-system/aws-load-balancer-controller", } 2025-02-22 20:08:09 [▶] started task: create IAM role for serviceaccount "kube-system/aws-load-balancer-controller" 2025-02-22 20:08:09 [ℹ] building iamserviceaccount stack "eksctl-myeks-addon-iamserviceaccount-kube-system-aws-load-balancer-controller" 2025-02-22 20:08:09 [▶] service account location provided: kube-system/aws-node, adding sub condition 2025-02-22 20:08:09 [▶] CreateStackInput = &cloudformation.CreateStackInput{StackName:(*string)(0xc000ac58e0), Capabilities:[]types.Capability{"CAPABILITY_IAM"}, ClientRequestToken:(*string)(nil), DisableRollback:(*bool)(0xc000f13cc4), EnableTerminationProtection:(*bool)(nil), NotificationARNs:[]string(nil), OnFailure:"", Parameters:[]types.Parameter(nil), ResourceTypes:[]string(nil), RetainExceptOnCreate:(*bool)(nil), RoleARN:(*string)(nil), RollbackConfiguration:(*types.RollbackConfiguration)(nil), StackPolicyBody:(*string)(nil), StackPolicyURL:(*string)(nil), Tags:[]types.Tag{types.Tag{Key:(*string)(0xc000ee54b0), Value:(*string)(0xc000ee54c0), noSmithyDocumentSerde:document.NoSerde{}}, types.Tag{Key:(*string)(0xc000ee54d0), Value:(*string)(0xc000ee54e0), noSmithyDocumentSerde:document.NoSerde{}}, types.Tag{Key:(*string)(0xc000ee54f0), Value:(*string)(0xc000ee5500), noSmithyDocumentSerde:document.NoSerde{}}, types.Tag{Key:(*string)(0xc000f40890), Value:(*string)(0xc000f408a0), noSmithyDocumentSerde:document.NoSerde{}}}, TemplateBody:(*string)(0xc000f408b0), TemplateURL:(*string)(nil), TimeoutInMinutes:(*int32)(nil), noSmithyDocumentSerde:document.NoSerde{}} 2025-02-22 20:08:10 [ℹ] deploying stack "eksctl-myeks-addon-iamserviceaccount-kube-system-aws-load-balancer-controller" 2025-02-22 20:08:10 [ℹ] waiting for CloudFormation stack "eksctl-myeks-addon-iamserviceaccount-kube-system-aws-load-balancer-controller" 2025-02-22 20:08:40 [ℹ] waiting for CloudFormation stack "eksctl-myeks-addon-iamserviceaccount-kube-system-aws-load-balancer-controller" 2025-02-22 20:09:10 [ℹ] waiting for CloudFormation stack "eksctl-myeks-addon-iamserviceaccount-kube-system-aws-load-balancer-controller" 2025-02-22 20:09:10 [▶] completed task: create IAM role for serviceaccount "kube-system/aws-load-balancer-controller" 2025-02-22 20:09:10 [▶] started task: create serviceaccount "kube-system/aws-load-balancer-controller" 2025-02-22 20:09:10 [ℹ] created serviceaccount "kube-system/aws-load-balancer-controller" 2025-02-22 20:09:10 [▶] completed task: create serviceaccount "kube-system/aws-load-balancer-controller" 2025-02-22 20:09:10 [▶] completed task: 2 sequential sub-tasks: { create IAM role for serviceaccount "kube-system/aws-load-balancer-controller", create serviceaccount "kube-system/aws-load-balancer-controller", } 2025-02-22 20:09:10 [▶] completed task: 2 sequential sub-tasks: { create IAM role for serviceaccount "kube-system/aws-load-balancer-controller", create serviceaccount "kube-system/aws-load-balancer-controller", } 2025-02-22 20:09:10 [▶] started task: update VPC CNI to use IRSA if required 2025-02-22 20:09:11 [ℹ] addon "vpc-cni" active 2025-02-22 20:09:11 [▶] addon: &{vpc-cni v1.19.2-eksbuild.5 [arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy] map[] {false false false false false false false} map[] <nil> false enableNetworkPolicy: "true" true [] [] []} 2025-02-22 20:09:11 [▶] resolve conflicts set to OVERWRITE 2025-02-22 20:09:11 [▶] addon: &{vpc-cni v1.19.2-eksbuild.5 [arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy] map[] {false false false false false false false} map[] <nil> false enableNetworkPolicy: "true" true [] [] []} 2025-02-22 20:09:11 [▶] found known service account location kube-system/aws-node 2025-02-22 20:09:11 [▶] service account location provided: kube-system/aws-node, adding sub condition 2025-02-22 20:09:11 [▶] CreateStackInput = &cloudformation.CreateStackInput{StackName:(*string)(0xc00007f660), Capabilities:[]types.Capability{"CAPABILITY_IAM"}, ClientRequestToken:(*string)(nil), DisableRollback:(*bool)(0xc000f13d78), EnableTerminationProtection:(*bool)(nil), NotificationARNs:[]string(nil), OnFailure:"", Parameters:[]types.Parameter(nil), ResourceTypes:[]string(nil), RetainExceptOnCreate:(*bool)(nil), RoleARN:(*string)(nil), RollbackConfiguration:(*types.RollbackConfiguration)(nil), StackPolicyBody:(*string)(nil), StackPolicyURL:(*string)(nil), Tags:[]types.Tag{types.Tag{Key:(*string)(0xc000f40240), Value:(*string)(0xc000f40250), noSmithyDocumentSerde:document.NoSerde{}}, types.Tag{Key:(*string)(0xc000f40260), Value:(*string)(0xc000f40270), noSmithyDocumentSerde:document.NoSerde{}}, types.Tag{Key:(*string)(0xc000f40280), Value:(*string)(0xc000f40290), noSmithyDocumentSerde:document.NoSerde{}}, types.Tag{Key:(*string)(0xc00007fd90), Value:(*string)(0xc00007fda0), noSmithyDocumentSerde:document.NoSerde{}}}, TemplateBody:(*string)(0xc00031ab80), TemplateURL:(*string)(nil), TimeoutInMinutes:(*int32)(nil), noSmithyDocumentSerde:document.NoSerde{}} 2025-02-22 20:09:11 [ℹ] deploying stack "eksctl-myeks-addon-vpc-cni" 2025-02-22 20:09:11 [ℹ] waiting for CloudFormation stack "eksctl-myeks-addon-vpc-cni" 2025-02-22 20:09:41 [ℹ] waiting for CloudFormation stack "eksctl-myeks-addon-vpc-cni" 2025-02-22 20:09:41 [ℹ] updating addon 2025-02-22 20:09:41 [▶] &{AddonName:0xc0009e6960 ClusterName:0xc000a5c910 AddonVersion:0xc000f41f60 ClientRequestToken:<nil> ConfigurationValues:0xc0009e69f0 PodIdentityAssociations:[] ResolveConflicts:OVERWRITE ServiceAccountRoleArn:0xc0004b3160 noSmithyDocumentSerde:{}} 2025-02-22 20:09:42 [▶] &{CreatedAt:2025-02-22 11:09:42.606 +0000 UTC Errors:[] Id:0xc00058e530 Params:[{Type:AddonVersion Value:0xc00058e4d0 noSmithyDocumentSerde:{}} {Type:ServiceAccountRoleArn Value:0xc00058e4e0 noSmithyDocumentSerde:{}} {Type:ResolveConflicts Value:0xc00058e500 noSmithyDocumentSerde:{}} {Type:ConfigurationValues Value:0xc00058e520 noSmithyDocumentSerde:{}}] Status:InProgress Type:AddonUpdate noSmithyDocumentSerde:{}} 2025-02-22 20:09:53 [ℹ] addon "vpc-cni" active 2025-02-22 20:09:53 [▶] completed task: update VPC CNI to use IRSA if required 2025-02-22 20:09:53 [▶] completed task: 5 sequential sub-tasks: { 1 task: { create addons }, wait for control plane to become ready, associate IAM OIDC provider, 2 sequential sub-tasks: { create IAM role for serviceaccount "kube-system/aws-load-balancer-controller", create serviceaccount "kube-system/aws-load-balancer-controller", }, update VPC CNI to use IRSA if required, } 2025-02-22 20:09:53 [▶] started task: create managed nodegroup "ng1" 2025-02-22 20:09:53 [▶] waiting for 1 parallel tasks to complete 2025-02-22 20:09:53 [▶] started task: create managed nodegroup "ng1" 2025-02-22 20:09:53 [▶] started task: create managed nodegroup "ng1" 2025-02-22 20:09:53 [▶] started task: create managed nodegroup "ng1" 2025-02-22 20:09:53 [ℹ] building managed nodegroup stack "eksctl-myeks-nodegroup-ng1" 2025-02-22 20:09:53 [▶] CreateStackInput = &cloudformation.CreateStackInput{StackName:(*string)(0xc000db4ab0), Capabilities:[]types.Capability{"CAPABILITY_IAM"}, ClientRequestToken:(*string)(nil), DisableRollback:(*bool)(0xc00097c5d8), EnableTerminationProtection:(*bool)(nil), NotificationARNs:[]string(nil), OnFailure:"", Parameters:[]types.Parameter(nil), ResourceTypes:[]string(nil), RetainExceptOnCreate:(*bool)(nil), RoleARN:(*string)(nil), RollbackConfiguration:(*types.RollbackConfiguration)(nil), StackPolicyBody:(*string)(nil), StackPolicyURL:(*string)(nil), Tags:[]types.Tag{types.Tag{Key:(*string)(0xc000ee4e30), Value:(*string)(0xc000ee4e40), noSmithyDocumentSerde:document.NoSerde{}}, types.Tag{Key:(*string)(0xc000ee4e50), Value:(*string)(0xc000ee4e60), noSmithyDocumentSerde:document.NoSerde{}}, types.Tag{Key:(*string)(0xc000ee4e70), Value:(*string)(0xc000ee4e80), noSmithyDocumentSerde:document.NoSerde{}}, types.Tag{Key:(*string)(0xc000e8c0d0), Value:(*string)(0xc000e8c0e0), noSmithyDocumentSerde:document.NoSerde{}}, types.Tag{Key:(*string)(0xc000e8c0f0), Value:(*string)(0xc000e8c100), noSmithyDocumentSerde:document.NoSerde{}}}, TemplateBody:(*string)(0xc000e8c110), TemplateURL:(*string)(nil), TimeoutInMinutes:(*int32)(nil), noSmithyDocumentSerde:document.NoSerde{}} 2025-02-22 20:09:53 [ℹ] deploying stack "eksctl-myeks-nodegroup-ng1" 2025-02-22 20:09:53 [ℹ] waiting for CloudFormation stack "eksctl-myeks-nodegroup-ng1" 2025-02-22 20:10:23 [ℹ] waiting for CloudFormation stack "eksctl-myeks-nodegroup-ng1" 2025-02-22 20:11:01 [ℹ] waiting for CloudFormation stack "eksctl-myeks-nodegroup-ng1" 2025-02-22 20:12:10 [ℹ] waiting for CloudFormation stack "eksctl-myeks-nodegroup-ng1" 2025-02-22 20:12:52 [ℹ] waiting for CloudFormation stack "eksctl-myeks-nodegroup-ng1" 2025-02-22 20:12:52 [▶] processing stack outputs 2025-02-22 20:12:52 [▶] completed task: create managed nodegroup "ng1" 2025-02-22 20:12:52 [▶] completed task: create managed nodegroup "ng1" 2025-02-22 20:12:52 [▶] completed task: create managed nodegroup "ng1" 2025-02-22 20:12:52 [▶] completed task: create managed nodegroup "ng1" 2025-02-22 20:12:52 [▶] completed task: 2 sequential sub-tasks: { 5 sequential sub-tasks: { 1 task: { create addons }, wait for control plane to become ready, associate IAM OIDC provider, 2 sequential sub-tasks: { create IAM role for serviceaccount "kube-system/aws-load-balancer-controller", create serviceaccount "kube-system/aws-load-balancer-controller", }, update VPC CNI to use IRSA if required, }, create managed nodegroup "ng1", } 2025-02-22 20:12:52 [ℹ] waiting for the control plane to become ready 2025-02-22 20:12:53 [▶] merging kubeconfig files 2025-02-22 20:12:53 [▶] setting current-context to iam-root-account@myeks.ap-northeast-2.eksctl.io 2025-02-22 20:12:53 [✔] saved kubeconfig as "/root/.kube/config" 2025-02-22 20:12:53 [ℹ] 1 task: { install Nvidia device plugin } 2025-02-22 20:12:53 [▶] waiting for 1 parallel tasks to complete 2025-02-22 20:12:53 [▶] started task: install Nvidia device plugin 2025-02-22 20:12:53 [ℹ] created "kube-system:DaemonSet.apps/nvidia-device-plugin-daemonset" 2025-02-22 20:12:53 [ℹ] as you are using the EKS-Optimized Accelerated AMI with a GPU-enabled instance type, the Nvidia Kubernetes device plugin was automatically installed. to skip installing it, use --install-nvidia-plugin=false. 2025-02-22 20:12:53 [▶] completed task: install Nvidia device plugin 2025-02-22 20:12:53 [✔] all EKS cluster resources for "myeks" have been created 2025-02-22 20:12:53 [ℹ] nodegroup "ng1" has 3 node(s) 2025-02-22 20:12:53 [ℹ] node "ip-192-168-1-193.ap-northeast-2.compute.internal" is ready 2025-02-22 20:12:53 [ℹ] node "ip-192-168-2-200.ap-northeast-2.compute.internal" is ready 2025-02-22 20:12:53 [ℹ] node "ip-192-168-3-188.ap-northeast-2.compute.internal" is ready 2025-02-22 20:12:53 [ℹ] waiting for at least 3 node(s) to become ready in "ng1" 2025-02-22 20:12:53 [▶] event = watch.Event{Type:"ADDED", Object:(*v1.Node)(0xc0007d7508)} 2025-02-22 20:12:53 [▶] node "ip-192-168-1-193.ap-northeast-2.compute.internal" is ready in "ng1" 2025-02-22 20:12:53 [▶] event = watch.Event{Type:"ADDED", Object:(*v1.Node)(0xc0007d7b08)} 2025-02-22 20:12:53 [▶] node "ip-192-168-2-200.ap-northeast-2.compute.internal" is ready in "ng1" 2025-02-22 20:12:53 [▶] event = watch.Event{Type:"ADDED", Object:(*v1.Node)(0xc000cd6008)} 2025-02-22 20:12:53 [▶] node "ip-192-168-3-188.ap-northeast-2.compute.internal" is ready in "ng1" 2025-02-22 20:12:53 [ℹ] nodegroup "ng1" has 3 node(s) 2025-02-22 20:12:53 [ℹ] node "ip-192-168-1-193.ap-northeast-2.compute.internal" is ready 2025-02-22 20:12:53 [ℹ] node "ip-192-168-2-200.ap-northeast-2.compute.internal" is ready 2025-02-22 20:12:53 [ℹ] node "ip-192-168-3-188.ap-northeast-2.compute.internal" is ready 2025-02-22 20:12:53 [✔] created 1 managed nodegroup(s) in cluster "myeks" 2025-02-22 20:12:53 [▶] found authenticator: aws 2025-02-22 20:12:53 [▶] kubectl: "/usr/local/bin/kubectl" 2025-02-22 20:12:53 [▶] kubectl version: v1.31.2-eks-94953ac 2025-02-22 20:12:54 [ℹ] kubectl command should work with "/root/.kube/config", try 'kubectl get nodes' 2025-02-22 20:12:54 [✔] EKS cluster "myeks" in "ap-northeast-2" region is ready 2025-02-22 20:12:54 [▶] cfg.json = \ { "kind": "ClusterConfig", "apiVersion": "eksctl.io/v1alpha5", "metadata": { "name": "myeks", "region": "ap-northeast-2", "version": "1.31" }, "kubernetesNetworkConfig": { "ipFamily": "IPv4", "serviceIPv4CIDR": "10.100.0.0/16" }, "iam": { "serviceRoleARN": "arn:aws:iam::390402552337:role/eksctl-myeks-cluster-ServiceRole-yzipC6igslPD", "withOIDC": true, "serviceAccounts": [ { "metadata": { "name": "aws-load-balancer-controller", "namespace": "kube-system", "annotations": { "eks.amazonaws.com/role-arn": "arn:aws:iam::390402552337:role/eksctl-myeks-addon-iamserviceaccount-kube-sys-Role1-zzbfj7a4hBaW" } }, "wellKnownPolicies": { "imageBuilder": false, "autoScaler": false, "awsLoadBalancerController": true, "externalDNS": false, "certManager": false, "ebsCSIController": false, "efsCSIController": false }, "status": { "roleARN": "arn:aws:iam::390402552337:role/eksctl-myeks-addon-iamserviceaccount-kube-sys-Role1-zzbfj7a4hBaW" }, "tags": { "alpha.eksctl.io/iamserviceaccount-name": "kube-system/aws-load-balancer-controller" } } ], "vpcResourceControllerPolicy": true }, "accessConfig": { "authenticationMode": "API_AND_CONFIG_MAP" }, "vpc": { "id": "vpc-040caab837037a18c", "cidr": "192.168.0.0/16", "securityGroup": "sg-0c3354bd19a8894b4", "subnets": { "public": { "ap-northeast-2a": { "id": "subnet-08b83e58f1cad6098", "az": "ap-northeast-2a", "cidr": "192.168.1.0/24" }, "ap-northeast-2b": { "id": "subnet-0ad3d1ce95ad8a965", "az": "ap-northeast-2b", "cidr": "192.168.2.0/24" }, "ap-northeast-2c": { "id": "subnet-0f643cf03e72dc812", "az": "ap-northeast-2c", "cidr": "192.168.3.0/24" } } }, "sharedNodeSecurityGroup": "sg-044ca4465e589b793", "manageSharedNodeSecurityGroupRules": true, "nat": { "gateway": "Single" }, "clusterEndpoints": { "privateAccess": true, "publicAccess": true } }, "addons": [ { "name": "vpc-cni", "version": "v1.19.2-eksbuild.5", "attachPolicyARNs": [ "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy" ], "wellKnownPolicies": { "imageBuilder": false, "autoScaler": false, "awsLoadBalancerController": false, "externalDNS": false, "certManager": false, "ebsCSIController": false, "efsCSIController": false }, "configurationValues": "enableNetworkPolicy: \"true\"" }, { "name": "kube-proxy", "version": "v1.31.3-eksbuild.2", "wellKnownPolicies": { "imageBuilder": false, "autoScaler": false, "awsLoadBalancerController": false, "externalDNS": false, "certManager": false, "ebsCSIController": false, "efsCSIController": false } }, { "name": "coredns", "version": "v1.11.4-eksbuild.2", "wellKnownPolicies": { "imageBuilder": false, "autoScaler": false, "awsLoadBalancerController": false, "externalDNS": false, "certManager": false, "ebsCSIController": false, "efsCSIController": false } }, { "name": "metrics-server", "version": "v0.7.2-eksbuild.2", "wellKnownPolicies": { "imageBuilder": false, "autoScaler": false, "awsLoadBalancerController": false, "externalDNS": false, "certManager": false, "ebsCSIController": false, "efsCSIController": false } } ], "addonsConfig": {}, "privateCluster": { "enabled": false, "skipEndpointCreation": false }, "managedNodeGroups": [ { "name": "ng1", "amiFamily": "AmazonLinux2023", "instanceType": "t3.medium", "desiredCapacity": 3, "minSize": 3, "maxSize": 3, "volumeSize": 120, "ssh": { "allow": true, "publicKeyName": "aews-yu3papa" }, "labels": { "alpha.eksctl.io/cluster-name": "myeks", "alpha.eksctl.io/nodegroup-name": "ng1" }, "privateNetworking": false, "tags": { "alpha.eksctl.io/nodegroup-name": "ng1", "alpha.eksctl.io/nodegroup-type": "managed" }, "iam": { "withAddonPolicies": { "imageBuilder": false, "autoScaler": false, "externalDNS": true, "certManager": true, "appMesh": null, "appMeshPreview": null, "ebs": false, "fsx": false, "efs": false, "awsLoadBalancerController": false, "albIngress": false, "xRay": false, "cloudWatch": false } }, "securityGroups": { "withShared": null, "withLocal": null }, "maxPodsPerNode": 100, "volumeType": "gp3", "volumeIOPS": 3000, "volumeThroughput": 125, "preBootstrapCommands": [ "dnf install nvme-cli links tree tcpdump sysstat ipvsadm ipset bind-utils htop -y" ], "disableIMDSv1": true, "disablePodIMDS": false, "instanceSelector": {}, "releaseVersion": "" } ], "availabilityZones": [ "ap-northeast-2a", "ap-northeast-2c", "ap-northeast-2b" ] } |
EKS 배보후 기본정보를 확인해 보겠습니다.
| [root@operator-host ~]# kubectl cluster-info Kubernetes control plane is running at https://67FF602331201F8B161177C2E32D332A.gr7.ap-northeast-2.eks.amazonaws.com CoreDNS is running at https://67FF602331201F8B161177C2E32D332A.gr7.ap-northeast-2.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. # 네임스페이스 default 변경 적용 [root@operator-host ~]# kubectl config set-context --current --namespace=default # eksworkshop 으로 context 변경 [root@operator-host ~]# kubectl config rename-context "iam-root-account@myeks.ap-northeast-2.eksctl.io" "eksworkshop" Context "iam-root-account@myeks.ap-northeast-2.eksctl.io" renamed to "eksworkshop". (eksworkshop:default) [root@operator-host ~]# # 노드 정보 확인 [root@operator-host ~]# kubectl get node --label-columns=node.kubernetes.io/instance-type,eks.amazonaws.com/capacityType,topology.kubernetes.io/zone NAME STATUS ROLES AGE VERSION INSTANCE-TYPE CAPACITYTYPE ZONE ip-192-168-1-193.ap-northeast-2.compute.internal Ready <none> 9m34s v1.31.5-eks-5d632ec t3.medium ON_DEMAND ap-northeast-2a ip-192-168-2-200.ap-northeast-2.compute.internal Ready <none> 9m41s v1.31.5-eks-5d632ec t3.medium ON_DEMAND ap-northeast-2b ip-192-168-3-188.ap-northeast-2.compute.internal Ready <none> 9m38s v1.31.5-eks-5d632ec t3.medium ON_DEMAND ap-northeast-2c [root@operator-host ~]# kubectl get node -v=6 I0222 20:21:28.872244 3254 loader.go:395] Config loaded from file: /root/.kube/config I0222 20:21:29.672931 3254 round_trippers.go:553] GET https://67FF602331201F8B161177C2E32D332A.gr7.ap-northeast-2.eks.amazonaws.com/api/v1/nodes?limit=500 200 OK in 794 milliseconds NAME STATUS ROLES AGE VERSION ip-192-168-1-193.ap-northeast-2.compute.internal Ready <none> 9m42s v1.31.5-eks-5d632ec ip-192-168-2-200.ap-northeast-2.compute.internal Ready <none> 9m49s v1.31.5-eks-5d632ec ip-192-168-3-188.ap-northeast-2.compute.internal Ready <none> 9m46s v1.31.5-eks-5d632ec # POD 정보 확인 [root@operator-host ~]# kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system aws-node-bblcc 2/2 Running 0 10m kube-system aws-node-lhw2m 2/2 Running 0 11m kube-system aws-node-x24kb 2/2 Running 0 11m kube-system coredns-86f5954566-8n62v 1/1 Running 0 16m kube-system coredns-86f5954566-bv27z 1/1 Running 0 16m kube-system kube-proxy-gtcvw 1/1 Running 0 10m kube-system kube-proxy-rw2fb 1/1 Running 0 11m kube-system kube-proxy-vrngn 1/1 Running 0 11m kube-system metrics-server-6bf5998d9c-bqzjj 1/1 Running 0 16m kube-system metrics-server-6bf5998d9c-w2x7t 1/1 Running 0 16m kube-system nvidia-device-plugin-daemonset-7wpg4 1/1 Running 0 9m51s kube-system nvidia-device-plugin-daemonset-bc7bf 1/1 Running 0 9m51s kube-system nvidia-device-plugin-daemonset-s4v47 1/1 Running 0 9m51s [root@operator-host ~]# kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE aws-node-bblcc 2/2 Running 0 11m aws-node-lhw2m 2/2 Running 0 11m aws-node-x24kb 2/2 Running 0 11m coredns-86f5954566-8n62v 1/1 Running 0 16m coredns-86f5954566-bv27z 1/1 Running 0 16m kube-proxy-gtcvw 1/1 Running 0 11m kube-proxy-rw2fb 1/1 Running 0 11m kube-proxy-vrngn 1/1 Running 0 11m metrics-server-6bf5998d9c-bqzjj 1/1 Running 0 16m metrics-server-6bf5998d9c-w2x7t 1/1 Running 0 16m nvidia-device-plugin-daemonset-7wpg4 1/1 Running 0 10m nvidia-device-plugin-daemonset-bc7bf 1/1 Running 0 10m nvidia-device-plugin-daemonset-s4v47 1/1 Running 0 10m # 관리형 노드 그룹 확인 [root@operator-host ~]# eksctl get nodegroup --cluster $CLUSTER_NAME CLUSTER NODEGROUP STATUS CREATED MIN SIZE MAX SIZE DESIRED CAPACITY INSTANCE TYPE IMAGE ID ASG NAME TYPE myeks ng1 ACTIVE 2025-02-22T11:10:19Z 3 3 3 t3.medium AL2023_x86_64_STANDARD eks-ng1-26ca96ac-e8f7-d4d5-ab4d-a806e337b5ec managed [root@operator-host ~]# aws eks describe-nodegroup --cluster-name $CLUSTER_NAME --nodegroup-name ng1 | jq { "nodegroup": { "nodegroupName": "ng1", "nodegroupArn": "arn:aws:eks:ap-northeast-2:390402552337:nodegroup/myeks/ng1/26ca96ac-e8f7-d4d5-ab4d-a806e337b5ec", "clusterName": "myeks", "version": "1.31", "releaseVersion": "1.31.5-20250212", "createdAt": "2025-02-22T20:10:19.037000+09:00", "modifiedAt": "2025-02-22T20:21:24.169000+09:00", "status": "ACTIVE", "capacityType": "ON_DEMAND", "scalingConfig": { "minSize": 3, "maxSize": 3, "desiredSize": 3 }, "instanceTypes": [ "t3.medium" ], "subnets": [ "subnet-08b83e58f1cad6098", "subnet-0ad3d1ce95ad8a965", "subnet-0f643cf03e72dc812" ], "amiType": "AL2023_x86_64_STANDARD", "nodeRole": "arn:aws:iam::390402552337:role/eksctl-myeks-nodegroup-ng1-NodeInstanceRole-1etJJTz6hRou", "labels": { "alpha.eksctl.io/cluster-name": "myeks", "alpha.eksctl.io/nodegroup-name": "ng1" }, "resources": { "autoScalingGroups": [ { "name": "eks-ng1-26ca96ac-e8f7-d4d5-ab4d-a806e337b5ec" } ] }, "health": { "issues": [] }, "updateConfig": { "maxUnavailable": 1 }, "launchTemplate": { "name": "eksctl-myeks-nodegroup-ng1", "version": "1", "id": "lt-039071ae3f622f1c0" }, "tags": { "aws:cloudformation:stack-name": "eksctl-myeks-nodegroup-ng1", "alpha.eksctl.io/cluster-name": "myeks", "alpha.eksctl.io/nodegroup-name": "ng1", "aws:cloudformation:stack-id": "arn:aws:cloudformation:ap-northeast-2:390402552337:stack/eksctl-myeks-nodegroup-ng1/8a7d96d0-f10d-11ef-876d-06d9644ecb71", "eksctl.cluster.k8s.io/v1alpha1/cluster-name": "myeks", "aws:cloudformation:logical-id": "ManagedNodeGroup", "alpha.eksctl.io/nodegroup-type": "managed", "alpha.eksctl.io/eksctl-version": "0.204.0" } } } # eks addon 확인 [root@operator-host ~]# eksctl get addon --cluster $CLUSTER_NAME 2025-02-22 20:30:34 [ℹ] Kubernetes version "1.31" in use by cluster "myeks" 2025-02-22 20:30:34 [ℹ] getting all addons 2025-02-22 20:30:35 [ℹ] to see issues for an addon run `eksctl get addon --name <addon-name> --cluster <cluster-name>` NAME VERSION STATUS ISSUES IAMROLE UPDATE AVAILABLE CONFIGURATION VALUES POD IDENTITY ASSOCIATION ROLES coredns v1.11.4-eksbuild.2 ACTIVE 0 kube-proxy v1.31.3-eksbuild.2 ACTIVE 0 metrics-server v0.7.2-eksbuild.2 ACTIVE 0 vpc-cni v1.19.2-eksbuild.5 ACTIVE 0 arn:aws:iam::390402552337:role/eksctl-myeks-addon-vpc-cni-Role1-fr0GM10fLspD enableNetworkPolicy: "true" # aws-load-balancer-controller를 위한 iam service account 생성 확인 : AWS IAM role bound to a Kubernetes service account [root@operator-host ~]# eksctl get iamserviceaccount --cluster $CLUSTER_NAME NAMESPACE NAME ROLE ARN kube-system aws-load-balancer-controller arn:aws:iam::390402552337:role/eksctl-myeks-addon-iamserviceaccount-kube-sys-Role1-zzbfj7a4hBaW |
EKS 워커노드에 할당된 IAM Role 확인
관리형 노드 그룹(EC2) 접속 및 노드 정보 확인 : max-pods
| # 인스턴스 정보 확인 1 [root@operator-host ~]# aws ec2 describe-instances --query "Reservations[*].Instances[*].{InstanceID:InstanceId, PublicIPAdd:PublicIpAddress, PrivateIPAdd:PrivateIpAddress, InstanceName:Tags[?Key=='Name']|[0].Value, Status:State.Name}" --filters Name=instance-state-name,Values=running --output table ----------------------------------------------------------------------------------------- | DescribeInstances | +----------------------+-----------------+----------------+-----------------+-----------+ | InstanceID | InstanceName | PrivateIPAdd | PublicIPAdd | Status | +----------------------+-----------------+----------------+-----------------+-----------+ | i-0fabeb5dd693c0671 | myeks-ng1-Node | 192.168.3.188 | 3.36.132.80 | running | | i-03d19e4196f4c68aa | operator-host | 172.20.1.100 | 15.164.244.197 | running | | i-02c24086e5ef7bd2f | myeks-ng1-Node | 192.168.1.193 | 3.36.125.37 | running | | i-064da1db46bbd4d0e | myeks-ng1-Node | 192.168.2.200 | 43.203.149.39 | running | +----------------------+-----------------+----------------+-----------------+-----------+ # 인스턴스 정보 확인 2 : AZ, ID, 공인IP [root@operator-host ~]# aws ec2 describe-instances \ --filters "Name=tag:Name,Values=myeks-ng1-Node" \ --query "Reservations[].Instances[].{InstanceID:InstanceId, PublicIP:PublicIpAddress, AZ:Placement.AvailabilityZone}" \ --output table ------------------------------------------------------------- | DescribeInstances | +-----------------+-----------------------+-----------------+ | AZ | InstanceID | PublicIP | +-----------------+-----------------------+-----------------+ | ap-northeast-2c| i-0fabeb5dd693c0671 | 3.36.132.80 | | ap-northeast-2a| i-02c24086e5ef7bd2f | 3.36.125.37 | | ap-northeast-2b| i-064da1db46bbd4d0e | 43.203.149.39 | +-----------------+-----------------------+-----------------+ # AZ1 배치된 EC2 공인 IP [root@operator-host ~]# aws ec2 describe-instances \ --filters "Name=tag:Name,Values=myeks-ng1-Node" "Name=availability-zone,Values=ap-northeast-2a" \ --query 'Reservations[*].Instances[*].PublicIpAddress' \ --output text 3.36.125.37 # AZ2 배치된 EC2 공인 IP [root@operator-host ~]# aws ec2 describe-instances \ --filters "Name=tag:Name,Values=myeks-ng1-Node" "Name=availability-zone,Values=ap-northeast-2b" \ --query 'Reservations[*].Instances[*].PublicIpAddress' \ --output text 43.203.149.39 # AZ3 배치된 EC2 공인 IP [root@operator-host ~]# aws ec2 describe-instances \ --filters "Name=tag:Name,Values=myeks-ng1-Node" "Name=availability-zone,Values=ap-northeast-2c" \ --query 'Reservations[*].Instances[*].PublicIpAddress' \ --output text 3.36.132.80 # EC2 공인 IP 변수 지정 [root@operator-host ~]# export N1=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=myeks-ng1-Node" "Name=availability-zone,Values=ap-northeast-2a" --query 'Reservations[*].Instances[*].PublicIpAddress' --output text) [root@operator-host ~]# export N2=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=myeks-ng1-Node" "Name=availability-zone,Values=ap-northeast-2b" --query 'Reservations[*].Instances[*].PublicIpAddress' --output text) [root@operator-host ~]# export N3=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=myeks-ng1-Node" "Name=availability-zone,Values=ap-northeast-2c" --query 'Reservations[*].Instances[*].PublicIpAddress' --output text) [root@operator-host ~]# echo $N1, $N2, $N3 3.36.125.37, 43.203.149.39, 3.36.132.80 # *remoteAccess* 포함된 보안그룹 ID [root@operator-host ~]# aws ec2 describe-security-groups --filters "Name=group-name,Values=*remoteAccess*" | jq { "SecurityGroups": [ { "GroupId": "sg-09c76d5ef9c8d8091", "IpPermissionsEgress": [ { "IpProtocol": "-1", "UserIdGroupPairs": [], "IpRanges": [ { "CidrIp": "0.0.0.0/0" } ], "Ipv6Ranges": [], "PrefixListIds": [] } ], "Tags": [ { "Key": "aws:cloudformation:logical-id", "Value": "SSH" }, { "Key": "aws:cloudformation:stack-name", "Value": "eksctl-myeks-nodegroup-ng1" }, { "Key": "Name", "Value": "eksctl-myeks-nodegroup-ng1/SSH" }, { "Key": "aws:cloudformation:stack-id", "Value": "arn:aws:cloudformation:ap-northeast-2:390402552337:stack/eksctl-myeks-nodegroup-ng1/8a7d96d0-f10d-11ef-876d-06d9644ecb71" }, { "Key": "alpha.eksctl.io/nodegroup-type", "Value": "managed" }, { "Key": "alpha.eksctl.io/cluster-name", "Value": "myeks" }, { "Key": "eksctl.cluster.k8s.io/v1alpha1/cluster-name", "Value": "myeks" }, { "Key": "alpha.eksctl.io/eksctl-version", "Value": "0.204.0" }, { "Key": "alpha.eksctl.io/nodegroup-name", "Value": "ng1" } ], "VpcId": "vpc-040caab837037a18c", "SecurityGroupArn": "arn:aws:ec2:ap-northeast-2:390402552337:security-group/sg-09c76d5ef9c8d8091", "OwnerId": "390402552337", "GroupName": "eksctl-myeks-nodegroup-ng1-remoteAccess", "Description": "Allow SSH access", "IpPermissions": [ { "IpProtocol": "tcp", "FromPort": 22, "ToPort": 22, "UserIdGroupPairs": [], "IpRanges": [ { "Description": "Allow SSH access to managed worker nodes in group ng1", "CidrIp": "0.0.0.0/0" } ], "Ipv6Ranges": [ { "Description": "Allow SSH access to managed worker nodes in group ng1", "CidrIpv6": "::/0" } ], "PrefixListIds": [] } ] } ] } [root@operator-host ~]# export MNSGID=$(aws ec2 describe-security-groups --filters "Name=group-name,Values=*remoteAccess*" --query 'SecurityGroups[*].GroupId' --output text) # 해당 보안그룹 inbound 에 자신의 집 공인 IP 룰 추가 [root@operator-host ~]# aws ec2 authorize-security-group-ingress --group-id $MNSGID --protocol '-1' --cidr $(curl -s ipinfo.io/ip)/32 { "Return": true, "SecurityGroupRules": [ { "SecurityGroupRuleId": "sgr-097f02c88d155093e", "GroupId": "sg-09c76d5ef9c8d8091", "GroupOwnerId": "390402552337", "IsEgress": false, "IpProtocol": "-1", "FromPort": -1, "ToPort": -1, "CidrIpv4": "15.164.244.197/32", "SecurityGroupRuleArn": "arn:aws:ec2:ap-northeast-2:390402552337:security-group-rule/sgr-097f02c88d155093e" } ] } # 해당 보안그룹 inbound 에 운영서버 내부 IP 룰 추가 [root@operator-host ~]# aws ec2 authorize-security-group-ingress --group-id $MNSGID --protocol '-1' --cidr 172.20.1.100/32 { "Return": true, "SecurityGroupRules": [ { "SecurityGroupRuleId": "sgr-0b7b05294bd0d65e6", "GroupId": "sg-09c76d5ef9c8d8091", "GroupOwnerId": "390402552337", "IsEgress": false, "IpProtocol": "-1", "FromPort": -1, "ToPort": -1, "CidrIpv4": "172.20.1.100/32", "SecurityGroupRuleArn": "arn:aws:ec2:ap-northeast-2:390402552337:security-group-rule/sgr-0b7b05294bd0d65e6" } ] } # ping 테스트 [root@operator-host ~]# ping -c 2 $N1 PING 3.36.125.37 (3.36.125.37) 56(84) bytes of data. 64 bytes from 3.36.125.37: icmp_seq=1 ttl=126 time=0.742 ms 64 bytes from 3.36.125.37: icmp_seq=2 ttl=126 time=1.05 ms --- 3.36.125.37 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1007ms rtt min/avg/max/mdev = 0.742/0.896/1.051/0.157 ms [root@operator-host ~]# ping -c 2 $N2 PING 43.203.149.39 (43.203.149.39) 56(84) bytes of data. 64 bytes from 43.203.149.39: icmp_seq=1 ttl=126 time=0.861 ms 64 bytes from 43.203.149.39: icmp_seq=2 ttl=126 time=1.31 ms --- 43.203.149.39 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1008ms rtt min/avg/max/mdev = 0.861/1.085/1.310/0.226 ms [root@operator-host ~]# ping -c 2 $N3 PING 3.36.132.80 (3.36.132.80) 56(84) bytes of data. 64 bytes from 3.36.132.80: icmp_seq=1 ttl=126 time=1.24 ms 64 bytes from 3.36.132.80: icmp_seq=2 ttl=126 time=1.25 ms --- 3.36.132.80 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1001ms rtt min/avg/max/mdev = 1.245/1.250/1.255/0.005 ms |
myeks.yaml 설정파일에 ssh 설정을 allow: true 로 하게 되면 워커노드에 remoteAccess 보안그룹이 자동으로 추가됩니다.
| [root@operator-host ~]# cat myeks.yaml | grep ssh -A 1 ssh: allow: true |
워커노드에 NodePort를 바로 접속할 수 있도롤 아래와 같이 설정합니다.
| # 워커 노드 SSH 접속 [root@operator-host ~]# ssh -i aews-yu3papa.pem -o StrictHostKeyChecking=no ec2-user@$N1 hostname ip-192-168-1-193.ap-northeast-2.compute.internal [root@operator-host ~]# for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh -i aews-yu3papa.pem -o StrictHostKeyChecking=no ec2-user@$i hostname; echo; done >> node 3.36.125.37 << ip-192-168-1-193.ap-northeast-2.compute.internal >> node 43.203.149.39 << ip-192-168-2-200.ap-northeast-2.compute.internal >> node 3.36.132.80 << ip-192-168-3-188.ap-northeast-2.compute.internal |
이제 노드의 기본정보를 확인해 보겠습니다.
| # 노드 기본 정보 확인 [root@operator-host ~]# for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh -i aews-yu3papa.pem ec2-user@$i hostnamectl; echo; done >> node 3.36.125.37 << Static hostname: ip-192-168-1-193.ap-northeast-2.compute.internal Icon name: computer-vm Chassis: vm 🖴 Machine ID: ec29aa23ed9e53f7254d15a6afa0b19b Boot ID: 84cb77b76f36400e95686ad7ba0e2fa7 Virtualization: amazon Operating System: Amazon Linux 2023.6.20250203 CPE OS Name: cpe:2.3:o:amazon:amazon_linux:2023 Kernel: Linux 6.1.127-135.201.amzn2023.x86_64 Architecture: x86-64 Hardware Vendor: Amazon EC2 Hardware Model: t3.medium Firmware Version: 1.0 >> node 43.203.149.39 << Static hostname: ip-192-168-2-200.ap-northeast-2.compute.internal Icon name: computer-vm Chassis: vm 🖴 Machine ID: ec288cecc66f4a88c4c6a2431bd6cb50 Boot ID: 76c4da2e77f44a8385e5dd4c9d834e6c Virtualization: amazon Operating System: Amazon Linux 2023.6.20250203 CPE OS Name: cpe:2.3:o:amazon:amazon_linux:2023 Kernel: Linux 6.1.127-135.201.amzn2023.x86_64 Architecture: x86-64 Hardware Vendor: Amazon EC2 Hardware Model: t3.medium Firmware Version: 1.0 >> node 3.36.132.80 << Static hostname: ip-192-168-3-188.ap-northeast-2.compute.internal Icon name: computer-vm Chassis: vm 🖴 Machine ID: ec2cd3803c17e46194c1af15a865f00a Boot ID: c889707c7b3f4a4682b78fba66e634d9 Virtualization: amazon Operating System: Amazon Linux 2023.6.20250203 CPE OS Name: cpe:2.3:o:amazon:amazon_linux:2023 Kernel: Linux 6.1.127-135.201.amzn2023.x86_64 Architecture: x86-64 Hardware Vendor: Amazon EC2 Hardware Model: t3.medium Firmware Version: 1.0 [root@operator-host ~]# for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh -i aews-yu3papa.pem ec2-user@$i sudo ip -c addr; echo; done >> node 3.36.125.37 << 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host noprefixroute valid_lft forever preferred_lft forever 2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000 link/ether 02:b0:9b:41:8e:e3 brd ff:ff:ff:ff:ff:ff altname enp0s5 inet 192.168.1.193/24 metric 1024 brd 192.168.1.255 scope global dynamic ens5 valid_lft 3590sec preferred_lft 3590sec inet6 fe80::b0:9bff:fe41:8ee3/64 scope link proto kernel_ll valid_lft forever preferred_lft forever 3: enifae84e9e26b@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default link/ether 8e:1d:bf:a3:11:2b brd ff:ff:ff:ff:ff:ff link-netns cni-8832b236-62e5-04bc-db6a-6badb52ccc0f inet6 fe80::8c1d:bfff:fea3:112b/64 scope link proto kernel_ll valid_lft forever preferred_lft forever 4: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000 link/ether 02:2f:4d:ba:4a:a3 brd ff:ff:ff:ff:ff:ff altname enp0s6 inet 192.168.1.89/24 brd 192.168.1.255 scope global ens6 valid_lft forever preferred_lft forever inet6 fe80::2f:4dff:feba:4aa3/64 scope link proto kernel_ll valid_lft forever preferred_lft forever >> node 43.203.149.39 << 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host noprefixroute valid_lft forever preferred_lft forever 2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000 link/ether 06:87:62:db:d5:2f brd ff:ff:ff:ff:ff:ff altname enp0s5 inet 192.168.2.200/24 metric 1024 brd 192.168.2.255 scope global dynamic ens5 valid_lft 3582sec preferred_lft 3582sec inet6 fe80::487:62ff:fedb:d52f/64 scope link proto kernel_ll valid_lft forever preferred_lft forever 3: enif92a418c609@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default link/ether 4a:c6:8d:25:ac:ad brd ff:ff:ff:ff:ff:ff link-netns cni-d55e793c-17c4-3f2b-b514-18ca15e776b3 inet6 fe80::48c6:8dff:fe25:acad/64 scope link proto kernel_ll valid_lft forever preferred_lft forever 4: eni98f6ca7773f@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default link/ether 72:34:d6:a7:9b:7a brd ff:ff:ff:ff:ff:ff link-netns cni-3ae4ec85-681f-d28c-6072-df506911ca44 inet6 fe80::7034:d6ff:fea7:9b7a/64 scope link proto kernel_ll valid_lft forever preferred_lft forever 5: eni72b14925119@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default link/ether 02:98:13:84:05:76 brd ff:ff:ff:ff:ff:ff link-netns cni-5ea5e6af-faa1-7316-f0f4-609f36b37609 inet6 fe80::98:13ff:fe84:576/64 scope link proto kernel_ll valid_lft forever preferred_lft forever 6: eni1a9fabe96b3@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default link/ether 3a:41:34:dc:62:d1 brd ff:ff:ff:ff:ff:ff link-netns cni-6479f9c4-8ffc-1c68-60a4-28667c9eb8ca inet6 fe80::3841:34ff:fedc:62d1/64 scope link proto kernel_ll valid_lft forever preferred_lft forever 7: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000 link/ether 06:dd:cf:cd:23:99 brd ff:ff:ff:ff:ff:ff altname enp0s6 inet 192.168.2.235/24 brd 192.168.2.255 scope global ens6 valid_lft forever preferred_lft forever inet6 fe80::4dd:cfff:fecd:2399/64 scope link proto kernel_ll valid_lft forever preferred_lft forever 8: eni5d39fccd7d1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default link/ether 02:e0:8e:9e:f1:c0 brd ff:ff:ff:ff:ff:ff link-netns cni-72e58089-9fe7-cb48-66d3-b0d193f1df0d inet6 fe80::e0:8eff:fe9e:f1c0/64 scope link proto kernel_ll valid_lft forever preferred_lft forever >> node 3.36.132.80 << 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host noprefixroute valid_lft forever preferred_lft forever 2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000 link/ether 0a:be:1f:e2:43:bf brd ff:ff:ff:ff:ff:ff altname enp0s5 inet 192.168.3.188/24 metric 1024 brd 192.168.3.255 scope global dynamic ens5 valid_lft 3585sec preferred_lft 3585sec inet6 fe80::8be:1fff:fee2:43bf/64 scope link proto kernel_ll valid_lft forever preferred_lft forever 3: eni7877746a959@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UP group default link/ether 9e:24:bf:8d:d2:45 brd ff:ff:ff:ff:ff:ff link-netns cni-1b14fc63-2dbd-59af-676d-e728e21dc2c9 inet6 fe80::9c24:bfff:fe8d:d245/64 scope link proto kernel_ll valid_lft forever preferred_lft forever 4: ens6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000 link/ether 0a:1d:ff:8c:b0:41 brd ff:ff:ff:ff:ff:ff altname enp0s6 inet 192.168.3.88/24 brd 192.168.3.255 scope global ens6 valid_lft forever preferred_lft forever inet6 fe80::81d:ffff:fe8c:b041/64 scope link proto kernel_ll valid_lft forever preferred_lft forever [root@operator-host ~]# for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh -i aews-yu3papa.pem ec2-user@$i lsblk; echo; done >> node 3.36.125.37 << NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS nvme0n1 259:0 0 120G 0 disk ├─nvme0n1p1 259:1 0 120G 0 part / ├─nvme0n1p127 259:2 0 1M 0 part └─nvme0n1p128 259:3 0 10M 0 part /boot/efi >> node 43.203.149.39 << NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS nvme0n1 259:0 0 120G 0 disk ├─nvme0n1p1 259:1 0 120G 0 part / ├─nvme0n1p127 259:2 0 1M 0 part └─nvme0n1p128 259:3 0 10M 0 part /boot/efi >> node 3.36.132.80 << NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS nvme0n1 259:0 0 120G 0 disk ├─nvme0n1p1 259:1 0 120G 0 part / ├─nvme0n1p127 259:2 0 1M 0 part └─nvme0n1p128 259:3 0 10M 0 part /boot/efi [root@operator-host ~]# for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh -i aews-yu3papa.pem ec2-user@$i df -hT /; echo; done >> node 3.36.125.37 << Filesystem Type Size Used Avail Use% Mounted on /dev/nvme0n1p1 xfs 120G 3.9G 117G 4% / >> node 43.203.149.39 << Filesystem Type Size Used Avail Use% Mounted on /dev/nvme0n1p1 xfs 120G 4.1G 116G 4% / >> node 3.36.132.80 << Filesystem Type Size Used Avail Use% Mounted on /dev/nvme0n1p1 xfs 120G 3.9G 117G 4% / |
워커노드의 사이즈를 120G로 설정하여 놓았습니다.
| [root@operator-host ~]# for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh -i aews-yu3papa.pem ec2-user@$i lsblk; echo; done >> node 3.36.125.37 << NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS nvme0n1 259:0 0 120G 0 disk ├─nvme0n1p1 259:1 0 120G 0 part / ├─nvme0n1p127 259:2 0 1M 0 part └─nvme0n1p128 259:3 0 10M 0 part /boot/efi >> node 43.203.149.39 << NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS nvme0n1 259:0 0 120G 0 disk ├─nvme0n1p1 259:1 0 120G 0 part / ├─nvme0n1p127 259:2 0 1M 0 part └─nvme0n1p128 259:3 0 10M 0 part /boot/efi >> node 3.36.132.80 << NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS nvme0n1 259:0 0 120G 0 disk ├─nvme0n1p1 259:1 0 120G 0 part / ├─nvme0n1p127 259:2 0 1M 0 part └─nvme0n1p128 259:3 0 10M 0 part /boot/efi [root@operator-host ~]# for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh -i aews-yu3papa.pem ec2-user@$i df -hT /; echo; done >> node 3.36.125.37 << Filesystem Type Size Used Avail Use% Mounted on /dev/nvme0n1p1 xfs 120G 3.9G 117G 4% / >> node 43.203.149.39 << Filesystem Type Size Used Avail Use% Mounted on /dev/nvme0n1p1 xfs 120G 4.1G 116G 4% / >> node 3.36.132.80 << Filesystem Type Size Used Avail Use% Mounted on /dev/nvme0n1p1 xfs 120G 3.9G 117G 4% / |
EKS 는 디폴트로 gp2 SC(Storage Class)가 존재합니다. 추후에 gp3를 사용하는 SC를 만들어서 사용해 보겠습니다.
| # 스토리지클래스 및 CSI 노드 확인 [root@operator-host ~]# kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2 kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 71m [root@operator-host ~]# kubectl describe sc gp2 Name: gp2 IsDefaultClass: No Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"gp2"},"parameters":{"fsType":"ext4","type":"gp2"},"provisioner":"kubernetes.io/aws-ebs","volumeBindingMode":"WaitForFirstConsumer"} Provisioner: kubernetes.io/aws-ebs Parameters: fsType=ext4,type=gp2 AllowVolumeExpansion: MountOptions: ReclaimPolicy: Delete VolumeBindingMode: WaitForFirstConsumer Events: |
현재 CRD는 아래와 같으며 추후 실습에서 추가 컨트롤러를 설치하면 CRD 가 추가되는것을 확인해 보겠습니다.
| [root@operator-host ~]# kubectl get crd NAME CREATED AT cninodes.vpcresources.k8s.aws 2025-02-22T11:02:36Z eniconfigs.crd.k8s.amazonaws.com 2025-02-22T11:06:12Z policyendpoints.networking.k8s.aws 2025-02-22T11:02:36Z securitygrouppolicies.vpcresources.k8s.aws 2025-02-22T11:02:36Z [root@operator-host ~]# kubectl get csinodes NAME DRIVERS AGE ip-192-168-1-193.ap-northeast-2.compute.internal 0 63m ip-192-168-2-200.ap-northeast-2.compute.internal 0 63m ip-192-168-3-188.ap-northeast-2.compute.internal 0 63m |
기본 POD 개수는 워커노드당 25개인데 설정값을 100으로 설정하여 놓았습니다.
| # max-pods 정보 확인 [root@operator-host ~]# kubectl describe node | grep Capacity: -A13 Capacity: cpu: 2 ephemeral-storage: 125751276Ki hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 3919536Ki pods: 100 Allocatable: cpu: 1930m ephemeral-storage: 114818633946 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 3364528Ki pods: 100 -- Capacity: cpu: 2 ephemeral-storage: 125751276Ki hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 3919544Ki pods: 100 Allocatable: cpu: 1930m ephemeral-storage: 114818633946 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 3364536Ki pods: 100 -- Capacity: cpu: 2 ephemeral-storage: 125751276Ki hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 3919544Ki pods: 100 Allocatable: cpu: 1930m ephemeral-storage: 114818633946 hugepages-1Gi: 0 hugepages-2Mi: 0 memory: 3364536Ki pods: 100 [root@operator-host ~]# kubectl get nodes -o custom-columns="NAME:.metadata.name,MAXPODS:.status.capacity.pods" NAME MAXPODS ip-192-168-1-193.ap-northeast-2.compute.internal 100 ip-192-168-2-200.ap-northeast-2.compute.internal 100 ip-192-168-3-188.ap-northeast-2.compute.internal 100 # Amazon Linux 2 타입은 워커노드 초기 설정을 BootStrap 스크립트를 사용하는데, Amazon Linux 2023은 사용하지 않습니다. [root@operator-host ~]# for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh -i aews-yu3papa.pem ec2-user@$i cat /etc/eks/bootstrap.sh; echo; done >> node 3.36.125.37 << #!/usr/bin/env bash echo >&2 ' !!!!!!!!!! !!!!!!!!!! ERROR: bootstrap.sh has been removed from AL2023-based EKS AMIs. !!!!!!!!!! !!!!!!!!!! EKS nodes are now initialized by nodeadm. !!!!!!!!!! !!!!!!!!!! To migrate your user data, see: !!!!!!!!!! !!!!!!!!!! https://awslabs.github.io/amazon-eks-ami/nodeadm/ !!!!!!!!!! ' [root@operator-host ~]# ssh -i aews-yu3papa.pem ec2-user@$N1 sudo cat /etc/kubernetes/kubelet/config.json | jq { "address": "0.0.0.0", "authentication": { "x509": { "clientCAFile": "/etc/kubernetes/pki/ca.crt" }, "webhook": { "enabled": true, "cacheTTL": "2m0s" }, "anonymous": { "enabled": false } }, "authorization": { "mode": "Webhook", "webhook": { "cacheAuthorizedTTL": "5m0s", "cacheUnauthorizedTTL": "30s" } }, "cgroupDriver": "systemd", "cgroupRoot": "/", "clusterDNS": [ "10.100.0.10" ], "clusterDomain": "cluster.local", "containerRuntimeEndpoint": "unix:///run/containerd/containerd.sock", "evictionHard": { "memory.available": "100Mi", "nodefs.available": "10%", "nodefs.inodesFree": "5%" }, "featureGates": { "RotateKubeletServerCertificate": true }, "hairpinMode": "hairpin-veth", "kubeReserved": { "cpu": "70m", "ephemeral-storage": "1Gi", "memory": "442Mi" }, "kubeReservedCgroup": "/runtime", "logging": { "verbosity": 2 }, "maxPods": 17, "protectKernelDefaults": true, "providerID": "aws:///ap-northeast-2a/i-02c24086e5ef7bd2f", "readOnlyPort": 0, "serializeImagePulls": false, "serverTLSBootstrap": true, "systemReservedCgroup": "/system", "tlsCipherSuites": [ "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305", "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256", "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384", "TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305", "TLS_RSA_WITH_AES_128_GCM_SHA256", "TLS_RSA_WITH_AES_256_GCM_SHA384" ], "kind": "KubeletConfiguration", "apiVersion": "kubelet.config.k8s.io/v1beta1" } [root@operator-host ~]# for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh -i aews-yu3papa.pem ec2-user@$i sudo cat /etc/kubernetes/kubelet/config.json | grep maxPods; echo; done >> node 3.36.125.37 << "maxPods": 17, >> node 43.203.149.39 << "maxPods": 17, >> node 3.36.132.80 << "maxPods": 17, [root@operator-host ~]# for i in $N1 $N2 $N3; do echo ">> node $i <<"; ssh -i aews-yu3papa.pem ec2-user@$i sudo cat /etc/kubernetes/kubelet/config.json.d/00-nodeadm.conf | grep maxPods; echo; done >> node 3.36.125.37 << "maxPods": 100 >> node 43.203.149.39 << "maxPods": 100 >> node 3.36.132.80 << "maxPods": 100 |
(참고) If you don’t specify an AMI ID for the bootstrap.sh file included with Amazon EKS optimized Linux or Bottlerocket, managed node groups enforce a maximum number on the value of maxPods. For instances with less than 30 vCPUs, the maximum number is 110. For instances with greater than 30 vCPUs, the maximum number jumps to 250. These numbers are based on Kubernetes scalability thresholds and recommended settings by internal Amazon EKS scalability team testing. For more information, see the Amazon VPC CNI plugin increases pods per node limits blog post.
EKS에서 EFS를 SC로 사용할 예정입니다. 우선 운영EC2에서 CloudFormation으로 생성한 EFS를 마운트 해보겠습니다.
| # 현재 EFS 정보 확인 [root@operator-host ~]# aws efs describe-file-systems | jq { "FileSystems": [ { "OwnerId": "390402552337", "CreationToken": "ElasticFileSystem-0Vg6jie256jJ", "FileSystemId": "fs-04bdbcbb996487caf", "FileSystemArn": "arn:aws:elasticfilesystem:ap-northeast-2:390402552337:file-system/fs-04bdbcbb996487caf", "CreationTime": "2025-02-22T18:18:50+09:00", "LifeCycleState": "available", "Name": "myeks-EFS", "NumberOfMountTargets": 3, "SizeInBytes": { "Value": 6144, "Timestamp": "2025-02-22T21:56:03+09:00", "ValueInIA": 0, "ValueInStandard": 6144, "ValueInArchive": 0 }, "PerformanceMode": "generalPurpose", "Encrypted": false, "ThroughputMode": "bursting", "Tags": [ { "Key": "Name", "Value": "myeks-EFS" } ], "FileSystemProtection": { "ReplicationOverwriteProtection": "ENABLED" } } ] } # 파일 시스템 ID만 출력 [root@operator-host ~]# aws efs describe-file-systems --query "FileSystems[*].FileSystemId" --output text fs-04bdbcbb996487caf # EFS 마운트 대상 정보 확인 [root@operator-host ~]# aws efs describe-mount-targets --file-system-id $(aws efs describe-file-systems --query "FileSystems[*].FileSystemId" --output text) | jq { "MountTargets": [ { "OwnerId": "390402552337", "MountTargetId": "fsmt-024c4230e40771980", "FileSystemId": "fs-04bdbcbb996487caf", "SubnetId": "subnet-08b83e58f1cad6098", "LifeCycleState": "available", "IpAddress": "192.168.1.141", "NetworkInterfaceId": "eni-07792729b19999f22", "AvailabilityZoneId": "apne2-az1", "AvailabilityZoneName": "ap-northeast-2a", "VpcId": "vpc-040caab837037a18c" }, { "OwnerId": "390402552337", "MountTargetId": "fsmt-070fc8743f88c3222", "FileSystemId": "fs-04bdbcbb996487caf", "SubnetId": "subnet-0f643cf03e72dc812", "LifeCycleState": "available", "IpAddress": "192.168.3.183", "NetworkInterfaceId": "eni-04a4f6f78e77ef232", "AvailabilityZoneId": "apne2-az3", "AvailabilityZoneName": "ap-northeast-2c", "VpcId": "vpc-040caab837037a18c" }, { "OwnerId": "390402552337", "MountTargetId": "fsmt-072c118536be30869", "FileSystemId": "fs-04bdbcbb996487caf", "SubnetId": "subnet-0ad3d1ce95ad8a965", "LifeCycleState": "available", "IpAddress": "192.168.2.238", "NetworkInterfaceId": "eni-06a16b25394c36135", "AvailabilityZoneId": "apne2-az2", "AvailabilityZoneName": "ap-northeast-2b", "VpcId": "vpc-040caab837037a18c" } ] } # IP만 출력 : [root@operator-host ~]# aws efs describe-mount-targets --file-system-id $(aws efs describe-file-systems --query "FileSystems[*].FileSystemId" --output text) --query "MountTargets[*].IpAddress" --output text 192.168.1.141 192.168.3.183 192.168.2.238 # EFS 도메인 이름(예시) : fs-040469b8fab273469.efs.ap-northeast-2.amazonaws.com # EFS 마운트 테스트 -> EFSIP1=<IP만 출력에서 아무 IP나 지정> [root@operator-host ~]# EFSIP1=192.168.1.141 [root@operator-host ~]# df -hT Filesystem Type Size Used Avail Use% Mounted on devtmpfs devtmpfs 981M 0 981M 0% /dev tmpfs tmpfs 990M 0 990M 0% /dev/shm tmpfs tmpfs 990M 428K 989M 1% /run tmpfs tmpfs 990M 0 990M 0% /sys/fs/cgroup /dev/xvda1 xfs 30G 3.0G 28G 10% / tmpfs tmpfs 198M 0 198M 0% /run/user/1000 # 마운트 포인트 폴더 생성 + 마운트 [root@operator-host ~]# mkdir /mnt/myefs [root@operator-host ~]# mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport $EFSIP1:/ /mnt/myefs [root@operator-host ~]# findmnt -t nfs4 TARGET SOURCE FSTYPE OPTIONS /mnt/myefs 192.168.1.141:/ nfs4 rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,noresvport,proto=tcp,timeo=600,retr [root@operator-host ~]# df -hT --type nfs4 Filesystem Type Size Used Avail Use% Mounted on 192.168.1.141:/ nfs4 8.0E 0 8.0E 0% /mnt/myefs # 파일 작성 테스트 # 현재 NFS 통계 확인 [root@operator-host ~]# nfsstat Client rpc stats: calls retrans authrefrsh 17 0 17 Client nfs v4: null read write commit open open_conf 1 5% 0 0% 0 0% 0 0% 0 0% 0 0% open_noat open_dgrd close setattr fsinfo renew 0 0% 0 0% 0 0% 0 0% 2 11% 0 0% setclntid confirm lock lockt locku access 0 0% 0 0% 0 0% 0 0% 0 0% 0 0% getattr lookup lookup_root remove rename link 1 5% 0 0% 1 5% 0 0% 0 0% 0 0% symlink create pathconf statfs readlink readdir 0 0% 0 0% 1 5% 1 5% 0 0% 0 0% server_caps delegreturn getacl setacl fs_locations rel_lkowner 3 17% 0 0% 0 0% 0 0% 0 0% 0 0% secinfo exchange_id create_ses destroy_ses sequence get_lease_t 0 0% 0 0% 2 11% 1 5% 0 0% 2 11% reclaim_comp layoutget getdevinfo layoutcommit layoutreturn getdevlist 0 0% 1 5% 0 0% 0 0% 0 0% 0 0% (null) 1 5% [root@operator-host ~]# echo "EKS Workshop" > /mnt/myefs/memo.txt # 파일 1개를 작성하고 NFS 통계를 보고 증가된 값 확인 [root@operator-host ~]# nfsstat Client rpc stats: calls retrans authrefrsh 21 0 21 Client nfs v4: null read write commit open open_conf 1 4% 0 0% 1 4% 0 0% 1 4% 0 0% open_noat open_dgrd close setattr fsinfo renew 0 0% 0 0% 1 4% 0 0% 2 9% 0 0% setclntid confirm lock lockt locku access 0 0% 0 0% 0 0% 0 0% 0 0% 1 4% getattr lookup lookup_root remove rename link 1 4% 0 0% 1 4% 0 0% 0 0% 0 0% symlink create pathconf statfs readlink readdir 0 0% 0 0% 1 4% 1 4% 0 0% 0 0% server_caps delegreturn getacl setacl fs_locations rel_lkowner 3 14% 0 0% 0 0% 0 0% 0 0% 0 0% secinfo exchange_id create_ses destroy_ses sequence get_lease_t 0 0% 0 0% 2 9% 1 4% 0 0% 2 9% reclaim_comp layoutget getdevinfo layoutcommit layoutreturn getdevlist 0 0% 1 4% 0 0% 0 0% 0 0% 0 0% (null) 1 4% [root@operator-host ~]# ls -l /mnt/myefs total 4 -rw-r--r-- 1 root root 13 Feb 22 22:04 memo.txt [root@operator-host ~]# cat /mnt/myefs/memo.txt EKS Workshop |
# EC2 재부팅 이후에도 mount 탑재가 될 수 있게 설정
vi /etc/fstab
# EFS Mount fs-04bdbcbb996487caf /mnt/myefs nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport 0 0
EKS 배포 후 실습 편의를 위한 설정
- Child 쉘에서도 똑같은 환경변수를 사용하기 위함
cat << EOF >> ~/.bashrc
export CLUSTER_NAME=myeks
export VPCID=$(aws ec2 describe-vpcs --filters "Name=tag:Name,Values=$CLUSTER_NAME-VPC" --query 'Vpcs[*].VpcId' --output text)
export PubSubnet1=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet1" --query "Subnets[0].[SubnetId]" --output text)
export PubSubnet2=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet2" --query "Subnets[0].[SubnetId]" --output text)
export PubSubnet3=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet3" --query "Subnets[0].[SubnetId]" --output text)
export N1=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=$CLUSTER_NAME-ng1-Node" "Name=availability-zone,Values=ap-northeast-2a" --query 'Reservations[*].Instances[*].PublicIpAddress' --output text)
export N2=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=$CLUSTER_NAME-ng1-Node" "Name=availability-zone,Values=ap-northeast-2b" --query 'Reservations[*].Instances[*].PublicIpAddress' --output text)
export N3=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=$CLUSTER_NAME-ng1-Node" "Name=availability-zone,Values=ap-northeast-2c" --query 'Reservations[*].Instances[*].PublicIpAddress' --output text)
MyDomain=yu3papa.shop
MyDnzHostedZoneId=$(aws route53 list-hosted-zones-by-name --dns-name "$MyDomain." --query "HostedZones[0].Id" --output text)
EOF
~/.bashrc 설정 파일을 확인해 보겠습니다.
| [yu3papa@iworks ~]$ cat ~/.bashrc ...(생략)... export CLUSTER_NAME=myeks export VPCID=vpc-040caab837037a18c export PubSubnet1=subnet-08b83e58f1cad6098 export PubSubnet2=subnet-0ad3d1ce95ad8a965 export PubSubnet3=subnet-0f643cf03e72dc812 export N1=3.36.125.37 export N2=43.203.149.39 export N3=3.36.132.80 MyDomain=yu3papa.shop MyDnzHostedZoneId=/hostedzone/Z0888869ZZNUHC5WOC28 # 신규터미널을 열고 환경 확인 Last login: Sat Feb 22 20:51:49 2025 from 192.168.10.5 [yu3papa@iworks ~]$ echo $CLUSTER_NAME $VPCID $PubSubnet1 $PubSubnet2 $PubSubnet3 myeks vpc-040caab837037a18c subnet-08b83e58f1cad6098 subnet-0ad3d1ce95ad8a965 subnet-0f643cf03e72dc812 [yu3papa@iworks ~]$ echo $N1 $N2 $N3 $MyDomain $MyDnzHostedZoneId 3.36.125.37 43.203.149.39 3.36.132.80 yu3papa.shop /hostedzone/Z0888869ZZNUHC5WOC28 [yu3papa@iworks ~]$ tail -n 12 ~/.bashrc unset rc export CLUSTER_NAME=myeks export VPCID=vpc-040caab837037a18c export PubSubnet1=subnet-08b83e58f1cad6098 export PubSubnet2=subnet-0ad3d1ce95ad8a965 export PubSubnet3=subnet-0f643cf03e72dc812 export N1=3.36.125.37 export N2=43.203.149.39 export N3=3.36.132.80 MyDomain=yu3papa.shop MyDnzHostedZoneId=/hostedzone/Z0888869ZZNUHC5WOC28 |
AWS LoadBalancerController, ExternalDNS, kube-ops-view 설치
| # kube-ops-view [yu3papa@iworks ~]$ helm repo add geek-cookbook https://geek-cookbook.github.io/charts/ [yu3papa@iworks ~]$ helm repo list NAME URL geek-cookbook https://geek-cookbook.github.io/charts/ [yu3papa@iworks ~]$ helm install kube-ops-view geek-cookbook/kube-ops-view --version 1.2.2 --set service.main.type=ClusterIP --set env.TZ="Asia/Seoul" --namespace kube-system NAME: kube-ops-view LAST DEPLOYED: Sat Feb 22 22:40:01 2025 NAMESPACE: kube-system STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: 1. Get the application URL by running these commands: export POD_NAME=$(kubectl get pods --namespace kube-system -l "app.kubernetes.io/name=kube-ops-view,app.kubernetes.io/instance=kube-ops-view" -o jsonpath="{.items[0].metadata.name}") echo "Visit http://127.0.0.1:8080 to use your application" kubectl port-forward $POD_NAME 8080:8080 # AWS LoadBalancerController [yu3papa@iworks ~]$ helm repo add eks https://aws.github.io/eks-charts "eks" has been added to your repositories [yu3papa@iworks ~]$ helm repo update Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "eks" chart repository ...Successfully got an update from the "geek-cookbook" chart repository Update Complete. ⎈Happy Helming!⎈ [yu3papa@iworks ~]$ kubectl get sa -n kube-system aws-load-balancer-controller NAME SECRETS AGE aws-load-balancer-controller 0 151m [yu3papa@iworks ~]$ helm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=$CLUSTER_NAME \ --set serviceAccount.create=false --set serviceAccount.name=aws-load-balancer-controller NAME: aws-load-balancer-controller LAST DEPLOYED: Sat Feb 22 22:41:05 2025 NAMESPACE: kube-system STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: AWS Load Balancer controller installed! # ExternalDNS [yu3papa@iworks ~]$ MyDomain=yu3papa.shop [yu3papa@iworks ~]$ MyDnzHostedZoneId=$(aws route53 list-hosted-zones-by-name --dns-name "$MyDomain." --query "HostedZones[0].Id" --output text) [yu3papa@iworks ~]$ echo $MyDnzHostedZoneId /hostedzone/Z0888869ZZNUHC5WOC28 [yu3papa@iworks ~]$ curl -s https://raw.githubusercontent.com/gasida/PKOS/main/aews/externaldns.yaml | MyDomain=$MyDomain MyDnzHostedZoneId=$MyDnzHostedZoneId envsubst | kubectl apply -f - serviceaccount/external-dns created clusterrole.rbac.authorization.k8s.io/external-dns created clusterrolebinding.rbac.authorization.k8s.io/external-dns-viewer created deployment.apps/external-dns created # 사용 리전의 인증서 ARN 확인 : 정상 상태 확인(만료 상태면 에러 발생!) [yu3papa@iworks ~]$ aws acm list-certificates --query 'CertificateSummaryList[].CertificateArn[]' --output text arn:aws:acm:ap-northeast-2:390402552337:certificate/55e87490-f59e-4099-ac6b-d796ab5a1ed8 [yu3papa@iworks ~]$ CERT_ARN=$(aws acm list-certificates --query 'CertificateSummaryList[].CertificateArn[]' --output text) [yu3papa@iworks ~]$ echo $CERT_ARN arn:aws:acm:ap-northeast-2:390402552337:certificate/55e87490-f59e-4099-ac6b-d796ab5a1ed8 # kubeopsview 용 Ingress 설정 : group 설정으로 1대의 ALB를 여러개의 ingress 에서 공용 사용 cat <<EOF | kubectl apply -f - apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: alb.ingress.kubernetes.io/certificate-arn: $CERT_ARN alb.ingress.kubernetes.io/group.name: study alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]' alb.ingress.kubernetes.io/load-balancer-name: myeks-ingress-alb alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/ssl-redirect: "443" alb.ingress.kubernetes.io/success-codes: 200-399 alb.ingress.kubernetes.io/target-type: ip labels: app.kubernetes.io/name: kubeopsview name: kubeopsview namespace: kube-system spec: ingressClassName: alb rules: - host: kubeopsview.$MyDomain http: paths: - backend: service: name: kube-ops-view port: number: 8080 path: / pathType: Prefix EOF ingress.networking.k8s.io/kubeopsview created [yu3papa@iworks ~]$ kubectl -n kube-system get ing NAME CLASS HOSTS ADDRESS PORTS AGE kubeopsview alb kubeopsview.yu3papa.shop myeks-ingress-alb-833395103.ap-northeast-2.elb.amazonaws.com 80 21s |
설치한 AWS LoadBalancerController, ExternalDNS, kube-ops-view 를 확인해 보겠습니다.
| # 설치된 파드 정보 확인 [yu3papa@iworks ~]$ kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE aws-load-balancer-controller-554fbd9d-8kxrh 1/1 Running 0 24m aws-load-balancer-controller-554fbd9d-r4bjd 1/1 Running 0 24m aws-node-bblcc 2/2 Running 0 173m aws-node-lhw2m 2/2 Running 0 173m aws-node-x24kb 2/2 Running 0 173m coredns-86f5954566-8n62v 1/1 Running 0 179m coredns-86f5954566-bv27z 1/1 Running 0 179m external-dns-57c44fb644-r2vjn 1/1 Running 0 23m kube-ops-view-657dbc6cd8-kv9qm 1/1 Running 0 25m kube-proxy-gtcvw 1/1 Running 0 173m kube-proxy-rw2fb 1/1 Running 0 173m kube-proxy-vrngn 1/1 Running 0 173m metrics-server-6bf5998d9c-bqzjj 1/1 Running 0 179m metrics-server-6bf5998d9c-w2x7t 1/1 Running 0 179m nvidia-device-plugin-daemonset-7wpg4 1/1 Running 0 172m nvidia-device-plugin-daemonset-bc7bf 1/1 Running 0 172m nvidia-device-plugin-daemonset-s4v47 1/1 Running 0 172m # service, ep, ingress 확인 [yu3papa@iworks ~]$ kubectl get ingress,svc,ep -n kube-system NAME CLASS HOSTS ADDRESS PORTS AGE ingress.networking.k8s.io/kubeopsview alb kubeopsview.yu3papa.shop myeks-ingress-alb-833395103.ap-northeast-2.elb.amazonaws.com 80 2m21s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/aws-load-balancer-webhook-service ClusterIP 10.100.142.171 <none> 443/TCP 24m service/eks-extension-metrics-api ClusterIP 10.100.247.125 <none> 443/TCP 3h3m service/kube-dns ClusterIP 10.100.0.10 <none> 53/UDP,53/TCP,9153/TCP 179m service/kube-ops-view ClusterIP 10.100.1.128 <none> 8080/TCP 25m service/metrics-server ClusterIP 10.100.245.240 <none> 443/TCP 179m NAME ENDPOINTS AGE endpoints/aws-load-balancer-webhook-service 192.168.1.113:9443,192.168.3.72:9443 24m endpoints/eks-extension-metrics-api 172.0.32.0:10443 3h3m endpoints/kube-dns 192.168.2.169:53,192.168.2.24:53,192.168.2.169:53 + 3 more... 179m endpoints/kube-ops-view 192.168.1.114:8080 25m endpoints/metrics-server 192.168.2.154:10251,192.168.2.213:10251 179m # Kube Ops View 접속 정보 확인 [yu3papa@iworks ~]$ echo -e "Kube Ops View URL = https://kubeopsview.$MyDomain/#scale=1.5" Kube Ops View URL = https://kubeopsview.yu3papa.shop/#scale=1.5 |
3. EKS 스토리지 이해
EKS 는 바닐라 쿠버네티스의 업스트림 버전이기 때문에 블로그 상단에 설명드렸던 PV, PVC, SC 리소스를 동일한 개념으로 사용합니다.
배경 소개
파드 내부의 데이터는 파드가 삭제되면 모두 삭제됨 → 즉, 파드가 모두 상태가 없는(Stateless) 애플리케이션이였음! : Temporary filesystem, Volume
데이터베이스(파드)처럼 데이터 보존이 필요 == 상태가 있는(Stateful) 애플리케이션 : PV & PVC
- 로컬 볼륨(hostPath) ⇒ 퍼시스턴트 볼륨(Persistent Volume, PV) - 어느 노드에서도 연결하여 사용 가능, 예시) NFS, AWS EBS, Ceph 등
파드가 생성될 때 자동으로 볼륨을 마운트하여 파드에 연결하는 기능을 동적 프로비저닝(Dynamic Provisioning)이라고 함
- 퍼시스턴트 볼륨의 사용이 끝났을 때 해당 볼륨은 어떻게 초기화할 것인지 별도로 설정할 수 있는데, 쿠버네티스는 이를 Reclaim Policy 라고 부릅니다.
- Reclaim Policy 에는 크게 Retain(보존), Delete(삭제, 즉 EBS 볼륨도 삭제됨), ~~Recycle~~ 방식이 있습니다.
CSI (Contaier Storage Interface) 소개
- CSI Driver 배경
- Kubernetes source code 내부에 존재하는 AWS EBS provisioner는 당연히 Kubernetes release lifecycle을 따라서 배포되므로, provisioner 신규 기능을 사용하기 위해서는 Kubernetes version을 업그레이드해야 하는 제약 사항이 있습니다. 따라서, Kubernetes 개발자는 Kubernetes 내부에 내장된 provisioner (in-tree)를 모두 삭제하고, 별도의 controller Pod을 통해 동적 provisioning을 사용할 수 있도록 만들었습니다. 이것이 바로 CSI (Container Storage Interface) driver 입니다
- CSI 를 사용하면, K8S 의 공통화된 CSI 인터페이스를 통해 다양한 프로바이더를 사용할 수 있다.
- 아래는 일반적인 CSI driver의 구조입니다. AWS EBS CSI driver 역시 아래와 같은 구조를 가지는데,
오른쪽 StatefulSet 또는 Deployment로 배포된 controller Pod이 AWS API를 사용하여 실제 EBS volume을 생성하는 역할을 합니다.
왼쪽 DaemonSet으로 배포된 node Pod은 AWS API를 사용하여 Kubernetes node (EC2 instance)에 EBS volume을 attach 해줍니다.
파드 기본 저장소 동작 확인
- POD 안의 컨테이너는 기본적으로 임시 파일 시스템입니다.
| # redis 파드를 한개 생성 [yu3papa@iworks ~]$ cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: redis spec: terminationGracePeriodSeconds: 0 containers: - name: redis image: redis EOF pod/redis created # default 네임스페이스에 POD가 1개 생성되어 RUNNING 중입니다. [yu3papa@iworks ~]$ kubectl get po NAME READY STATUS RESTARTS AGE redis 1/1 Running 0 82s # redis 파드 내에 파일 작성 kubectl exec -it redis -- pwd kubectl exec -it redis -- sh -c "echo hello > /data/hello.txt" kubectl exec -it redis -- cat /data/hello.txt # ps 설치 kubectl exec -it redis -- sh -c "apt update && apt install procps -y" kubectl exec -it redis -- ps aux # redis 프로세스 강제 종료 : 파드가 어떻게 되나요? hint) restartPolicy kubectl exec -it redis -- kill 1 kubectl get pod # redis 파드 내에 파일 확인 kubectl exec -it redis -- cat /data/hello.txt kubectl exec -it redis -- ls -l /data # 파드 삭제 kubectl delete pod redis |
위 실습내용에서 중요한 점은 POD 안의 컨테이너 파일시스템은 일시적이라서, POD가 재시작되면 컨테이너 파일시스템은 사라지게 됩니다.
emptyDir 볼륨
- POD와 라이프사이클이 동일한 볼륨
- POD를 삭제하기전까지는 파일이 유지
| # 모니터링 kubectl get pod -w # redis 파드 생성 cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: redis spec: terminationGracePeriodSeconds: 0 containers: - name: redis image: redis volumeMounts: - name: redis-storage mountPath: /data/redis volumes: - name: redis-storage emptyDir: {} EOF # redis 파드 내에 파일 작성 kubectl exec -it redis -- pwd kubectl exec -it redis -- sh -c "echo hello > /data/redis/hello.txt" kubectl exec -it redis -- cat /data/redis/hello.txt # ps 설치 kubectl exec -it redis -- sh -c "apt update && apt install procps -y" kubectl exec -it redis -- ps aux # redis 프로세스 강제 종료 : 파드가 어떻게 되나요? hint) restartPolicy kubectl exec -it redis -- kill 1 kubectl get pod # redis 파드 내에 파일 확인 kubectl exec -it redis -- cat /data/redis/hello.txt kubectl exec -it redis -- ls -l /data/redis # 파드 삭제 후 파일 확인 kubectl delete pod redis cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: redis spec: terminationGracePeriodSeconds: 0 containers: - name: redis image: redis volumeMounts: - name: redis-storage mountPath: /data/redis volumes: - name: redis-storage emptyDir: {} EOF # redis 파드 내에 파일 확인 kubectl exec -it redis -- cat /data/redis/hello.txt kubectl exec -it redis -- ls -l /data/redis # 파드 삭제 kubectl delete pod redis |
emptyDir 볼륨은 POD가 생성될 때 생성되고, POD가 삭제 될 때 같이 삭제되는 임시 볼륨입니다.
POD 내의 컨테이너가 삭제/재시작 되더라도 생명주기는 컨테이너 단위가 아니라 POD 단위이므로 emptyDir 볼륨은 삭제 되지 않고 계속 사용 가능
hostPath 볼륨
- 노드의 파일 시스템에 있는 특정 파일 또는 디렉터리를 마운트
- 시스템 수준 POD(일반적으로 DaemonSet에서 관리됨)는 노드의 파일을 읽거나, 노드의 파일 시스템을 통해 노드의 장치에 액세스해야 한다. 쿠버네티스는 hostPath 볼륨을 통해 이것을 가능하게 한다
- 영구 스토리지 타입이지만 POD가 스케쥴된 노드에 Dependency가 존재하므로, 영구적으로 사용 가능하게 하려면 NAS형태(NFS, ...)의 Storage를 사용해야 함
| # hostPath 볼륨을을 사용하는 시스템 POD 검사 # kubectl describe po etcd-k8s-cp --namespace=kube-system ~~~ Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: etcd-certs: Type: HostPath (bare host directory volume) Path: /etc/kubernetes/pki/etcd HostPathType: DirectoryOrCreate etcd-data: Type: HostPath (bare host directory volume) Path: /var/lib/etcd HostPathType: DirectoryOrCreate QoS Class: BestEffort Node-Selectors: <none> Tolerations: :NoExecute Events: <none> |
hostPath 타입의 볼륨은 emptyDir과는 다르게 POD가 스케쥴링된 노드의 파일시스템을 사용하므로 파일의 영속성은 보장되나 공유스토리지로 사용할 수는 없습니다.
호스트 Path 를 사용하는 PV/PVC
- hostPath 타입의 볼륨이긴 하지만 local-path-provisioner 를 이용하여 PV, PVC 기반의 볼륨을 제공
- POD가 스케쥴링된 노드의 특정 폴더를 사용 : /opt/local-path-provisioner
- https://github.com/rancher/local-path-provisioner
| # 배포 [yu3papa@iworks ~]$ kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.31/deploy/local-path-storage.yaml namespace/local-path-storage created serviceaccount/local-path-provisioner-service-account created role.rbac.authorization.k8s.io/local-path-provisioner-role created clusterrole.rbac.authorization.k8s.io/local-path-provisioner-role created rolebinding.rbac.authorization.k8s.io/local-path-provisioner-bind created clusterrolebinding.rbac.authorization.k8s.io/local-path-provisioner-bind created deployment.apps/local-path-provisioner created storageclass.storage.k8s.io/local-path created configmap/local-path-config created ... apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: local-path provisioner: rancher.io/local-path volumeBindingMode: WaitForFirstConsumer reclaimPolicy: Delete --- apiVersion: v1 kind: ConfigMap metadata: name: local-path-config namespace: local-path-storage data: config.json: |- { "nodePathMap":[ { "node":"DEFAULT_PATH_FOR_NON_LISTED_NODES", "paths":["/opt/local-path-provisioner"] } ] } setup: |- #!/bin/sh set -eu mkdir -m 0777 -p "$VOL_DIR" teardown: |- #!/bin/sh set -eu rm -rf "$VOL_DIR" ... # 확인 [yu3papa@iworks ~]$ kubectl -n local-path-storage get all NAME READY STATUS RESTARTS AGE pod/local-path-provisioner-84967477f-x948v 1/1 Running 0 19s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/local-path-provisioner 1/1 1 1 19s NAME DESIRED CURRENT READY AGE replicaset.apps/local-path-provisioner-84967477f 1 1 1 19s [yu3papa@iworks ~]$ kubectl get pod -n local-path-storage -owide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES local-path-provisioner-84967477f-x948v 1/1 Running 0 44s 192.168.3.232 ip-192-168-3-188.ap-northeast-2.compute.internal <none> <none> [yu3papa@iworks ~]$ kubectl describe cm -n local-path-storage local-path-config Name: local-path-config Namespace: local-path-storage Labels: <none> Annotations: <none> Data ==== config.json: ---- { "nodePathMap":[ { "node":"DEFAULT_PATH_FOR_NON_LISTED_NODES", "paths":["/opt/local-path-provisioner"] } ] } helperPod.yaml: ---- apiVersion: v1 kind: Pod metadata: name: helper-pod spec: priorityClassName: system-node-critical tolerations: - key: node.kubernetes.io/disk-pressure operator: Exists effect: NoSchedule containers: - name: helper-pod image: busybox imagePullPolicy: IfNotPresent setup: ---- #!/bin/sh set -eu mkdir -m 0777 -p "$VOL_DIR" teardown: ---- #!/bin/sh set -eu rm -rf "$VOL_DIR" BinaryData ==== Events: <none> [yu3papa@iworks ~]$ kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2 kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 4h11m local-path rancher.io/local-path Delete WaitForFirstConsumer false 60s [yu3papa@iworks ~]$ kubectl get sc local-path NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE local-path rancher.io/local-path Delete WaitForFirstConsumer false 67s |
- 동작 확인
- PV/PVC 를 사용하는 파드 생성
| # PVC 생성 [yu3papa@iworks ~]$ cat <<EOF | kubectl apply -f - apiVersion: v1 kind: PersistentVolumeClaim metadata: name: localpath-claim spec: accessModes: - ReadWriteOnce storageClassName: local-path resources: requests: storage: 1Gi EOF persistentvolumeclaim/localpath-claim created # PVC 확인 [yu3papa@iworks ~]$ kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE localpath-claim Pending local-path <unset> 13s [yu3papa@iworks ~]$ kubectl describe pvc Name: localpath-claim Namespace: default StorageClass: local-path Status: Pending Volume: Labels: <none> Annotations: <none> Finalizers: [kubernetes.io/pvc-protection] Capacity: Access Modes: VolumeMode: Filesystem Used By: <none> Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal WaitForFirstConsumer 11s (x2 over 21s) persistentvolume-controller waiting for first consumer to be created before binding # 파드 생성 [yu3papa@iworks ~]$ cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: app spec: terminationGracePeriodSeconds: 3 containers: - name: app image: centos command: ["/bin/sh"] args: ["-c", "while true; do echo \$(date -u) >> /data/out.txt; sleep 5; done"] volumeMounts: - name: persistent-storage mountPath: /data volumes: - name: persistent-storage persistentVolumeClaim: claimName: localpath-claim EOF pod/app created # 파드 확인 [yu3papa@iworks ~]$ kubectl get pod,pv,pvc NAME READY STATUS RESTARTS AGE pod/app 1/1 Running 0 22s NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE persistentvolume/pvc-d8465f8f-a572-4d77-9ce9-3d587f3df05c 1Gi RWO Delete Bound default/localpath-claim local-path <unset> 14s NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE persistentvolumeclaim/localpath-claim Bound pvc-d8465f8f-a572-4d77-9ce9-3d587f3df05c 1Gi RWO local-path <unset> 60s # Node Affinity 확인 [yu3papa@iworks ~]$ kubectl describe pv Name: pvc-d8465f8f-a572-4d77-9ce9-3d587f3df05c Labels: <none> Annotations: local.path.provisioner/selected-node: ip-192-168-1-193.ap-northeast-2.compute.internal pv.kubernetes.io/provisioned-by: rancher.io/local-path Finalizers: [kubernetes.io/pv-protection] StorageClass: local-path Status: Bound Claim: default/localpath-claim Reclaim Policy: Delete Access Modes: RWO VolumeMode: Filesystem Capacity: 1Gi Node Affinity: Required Terms: Term 0: kubernetes.io/hostname in [ip-192-168-1-193.ap-northeast-2.compute.internal] Message: Source: Type: HostPath (bare host directory volume) Path: /opt/local-path-provisioner/pvc-d8465f8f-a572-4d77-9ce9-3d587f3df05c_default_localpath-claim HostPathType: DirectoryOrCreate Events: <none> [yu3papa@iworks ~]$ kubectl exec -it app -- tail -f /data/out.txt Sat Feb 22 15:18:45 UTC 2025 Sat Feb 22 15:18:50 UTC 2025 ...(생략)... ^Ccommand terminated with exit code 130 # 워커노드 중 현재 파드가 배포되어 있다만, 아래 경로에 out.txt 파일 존재 확인 [yu3papa@iworks ~]$ for node in $N1 $N2 $N3; do ssh -i aews-yu3papa.pem ec2-user@$node tree /opt/local-path-provisioner; done /opt/local-path-provisioner └── pvc-d8465f8f-a572-4d77-9ce9-3d587f3df05c_default_localpath-claim └── out.txt 1 directory, 1 file /opt/local-path-provisioner [error opening dir] 0 directories, 0 files /opt/local-path-provisioner [error opening dir] 0 directories, 0 files # 해당 워커노드 자체에서 out.txt 파일 확인 : 아래 굵은 부분은 각자 실습 환경에 따라 다름 [yu3papa@iworks ~]$ ssh -i aews-yu3papa.pem ec2-user@$N1 tail -f /opt/local-path-provisioner/pvc-d8465f8f-a572-4d77-9ce9-3d587f3df05c_default_localpath-claim/out.txt Sat Feb 22 15:20:35 UTC 2025 Sat Feb 22 15:20:40 UTC 2025 Sat Feb 22 15:20:45 UTC 2025 Sat Feb 22 15:20:50 UTC 2025 Sat Feb 22 15:20:55 UTC 2025 ...(생략)... ^Ccommand terminated with exit code 130 |
- 파드 삭제 후 파드 재생성해서 데이터 유지 되는지 확인
| # 파드 삭제 후 PV/PVC 확인 [yu3papa@iworks ~]$ kubectl delete pod app pod "app" deleted # POD가 삭제되어도 파일은 영속성을 유지함 [yu3papa@iworks ~]$ for node in $N1 $N2 $N3; do ssh -i aews-yu3papa.pem ec2-user@$node tree /opt/local-path-provisioner; done /opt/local-path-provisioner └── pvc-d8465f8f-a572-4d77-9ce9-3d587f3df05c_default_localpath-claim └── out.txt 1 directory, 1 file /opt/local-path-provisioner [error opening dir] 0 directories, 0 files /opt/local-path-provisioner [error opening dir] 0 directories, 0 files # 파드 다시 실행 [yu3papa@iworks ~]$ cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: app spec: terminationGracePeriodSeconds: 3 containers: - name: app image: centos command: ["/bin/sh"] args: ["-c", "while true; do echo \$(date -u) >> /data/out.txt; sleep 5; done"] volumeMounts: - name: persistent-storage mountPath: /data volumes: - name: persistent-storage persistentVolumeClaim: claimName: localpath-claim EOF pod/app created [yu3papa@iworks ~]$ kubectl get po NAME READY STATUS RESTARTS AGE app 1/1 Running 0 11s # 파일 영속성 확인 [yu3papa@iworks ~]$ kubectl exec -it app -- head /data/out.txt Sat Feb 22 15:18:45 UTC 2025 Sat Feb 22 15:18:50 UTC 2025 Sat Feb 22 15:18:55 UTC 2025 Sat Feb 22 15:19:00 UTC 2025 Sat Feb 22 15:19:05 UTC 2025 Sat Feb 22 15:19:10 UTC 2025 Sat Feb 22 15:19:15 UTC 2025 Sat Feb 22 15:19:20 UTC 2025 Sat Feb 22 15:19:25 UTC 2025 Sat Feb 22 15:19:30 UTC 2025 [yu3papa@iworks ~]$ kubectl exec -it app -- tail -f /data/out.txt Sat Feb 22 15:28:11 UTC 2025 Sat Feb 22 15:28:16 UTC 2025 Sat Feb 22 15:28:21 UTC 2025 <<<<< POD가 삭제되고 다시 만들어지기까지 시간정보가 비어 있는 구간 >>>>> Sat Feb 22 15:29:00 UTC 2025 Sat Feb 22 15:29:05 UTC 2025 Sat Feb 22 15:29:10 UTC 2025 Sat Feb 22 15:29:15 UTC 2025 Sat Feb 22 15:29:20 UTC 2025 Sat Feb 22 15:29:25 UTC 2025 Sat Feb 22 15:29:30 UTC 2025 Sat Feb 22 15:29:35 UTC 2025 Sat Feb 22 15:29:40 UTC 2025 ^Ccommand terminated with exit code 130 |
- 다음 실습을 위해서 파드와 PVC 삭제
| # 파드 삭제 [yu3papa@iworks ~]$ kubectl delete pod app pod "app" deleted # POD를 삭제해도 PVC, PV는 지워지지 않음 [yu3papa@iworks ~]$ kubectl get pv,pvc NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE persistentvolume/pvc-d8465f8f-a572-4d77-9ce9-3d587f3df05c 1Gi RWO Delete Bound default/localpath-claim local-path <unset> 16m NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE persistentvolumeclaim/localpath-claim Bound pvc-d8465f8f-a572-4d77-9ce9-3d587f3df05c 1Gi RWO local-path <unset> 17m # PVC 삭제하면 기본 ReclaimPolicy가 DELETE 여서 PV 도 삭제되고 /opt/local-path-provisioner 밑의 디렉토리도 삭제됨 [yu3papa@iworks ~]$ kubectl delete pvc localpath-claim persistentvolumeclaim "localpath-claim" deleted # 확인 [yu3papa@iworks ~]$ kubectl get pv No resources found [yu3papa@iworks ~]$ for node in $N1 $N2 $N3; do ssh -i aews-yu3papa.pem ec2-user@$node tree /opt/local-path-provisioner; done /opt/local-path-provisioner 0 directories, 0 files /opt/local-path-provisioner [error opening dir] 0 directories, 0 files /opt/local-path-provisioner [error opening dir] 0 directories, 0 files |
4. AWS EBS Controller
ebs-csi-controller
- AWS CSI 드라이버는 크게 2개 구성요소가 있습니다. AWS API를 호출하면서 AWS 스토리지를 관리하는 CSI-Controller와 kubelet과 상호작용하면서 AWS스토리지를 pod에 마운트하는 CSI-Node가 있습니다.
- persistentvolume, persistentvolumeclaim의 accessModes는 ReadWriteOnce로 설정해야 합니다.
- EBS스토리지 기본 설정이 동일 AZ에 있는 EC2 인스턴스(에 배포된 파드)에 연결해야 합니다.
설치 : Amazon EBS CSI driver as an Amazon EKS add-on
| # 아래는 aws-ebs-csi-driver 전체 버전 정보와 기본 설치 버전(True) 정보 확인 [yu3papa@iworks ~]$ aws eks describe-addon-versions \ --addon-name aws-ebs-csi-driver \ --kubernetes-version 1.31 \ --query "addons[].addonVersions[].[addonVersion, compatibilities[].defaultVersion]" \ --output text v1.39.0-eksbuild.1 True v1.38.1-eksbuild.2 False v1.38.1-eksbuild.1 False v1.37.0-eksbuild.2 False v1.37.0-eksbuild.1 False v1.36.0-eksbuild.2 False v1.36.0-eksbuild.1 False v1.35.0-eksbuild.2 False v1.35.0-eksbuild.1 False v1.34.0-eksbuild.1 False v1.33.0-eksbuild.1 False v1.32.0-eksbuild.1 False v1.31.0-eksbuild.1 False v1.30.0-eksbuild.1 False v1.29.1-eksbuild.1 False # ISRA 설정 : AWS관리형 정책 AmazonEBSCSIDriverPolicy 사용 [yu3papa@iworks ~]$ eksctl create iamserviceaccount \ --name ebs-csi-controller-sa \ --namespace kube-system \ --cluster ${CLUSTER_NAME} \ --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \ --approve \ --role-only \ --role-name AmazonEKS_EBS_CSI_DriverRole 2025-02-23 00:56:47 [ℹ] 1 existing iamserviceaccount(s) (kube-system/aws-load-balancer-controller) will be excluded 2025-02-23 00:56:47 [ℹ] 1 iamserviceaccount (kube-system/ebs-csi-controller-sa) was included (based on the include/exclude rules) 2025-02-23 00:56:47 [!] serviceaccounts in Kubernetes will not be created or modified, since the option --role-only is used 2025-02-23 00:56:47 [ℹ] 1 task: { create IAM role for serviceaccount "kube-system/ebs-csi-controller-sa" } 2025-02-23 00:56:47 [ℹ] building iamserviceaccount stack "eksctl-myeks-addon-iamserviceaccount-kube-system-ebs-csi-controller-sa" 2025-02-23 00:56:47 [ℹ] deploying stack "eksctl-myeks-addon-iamserviceaccount-kube-system-ebs-csi-controller-sa" 2025-02-23 00:56:47 [ℹ] waiting for CloudFormation stack "eksctl-myeks-addon-iamserviceaccount-kube-system-ebs-csi-controller-sa" 2025-02-23 00:57:17 [ℹ] waiting for CloudFormation stack "eksctl-myeks-addon-iamserviceaccount-kube-system-ebs-csi-controller-sa" # ISRA 확인 [yu3papa@iworks ~]$ eksctl get iamserviceaccount --cluster ${CLUSTER_NAME} NAMESPACE NAME ROLE ARN kube-system aws-load-balancer-controller arn:aws:iam::390402552337:role/eksctl-myeks-addon-iamserviceaccount-kube-sys-Role1-zzbfj7a4hBaW kube-system ebs-csi-controller-sa arn:aws:iam::390402552337:role/AmazonEKS_EBS_CSI_DriverRole # Amazon EBS CSI driver addon 배포(설치) [yu3papa@iworks ~]$ export ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text) [yu3papa@iworks ~]$ eksctl create addon --name aws-ebs-csi-driver --cluster ${CLUSTER_NAME} --service-account-role-arn arn:aws:iam::${ACCOUNT_ID}:role/AmazonEKS_EBS_CSI_DriverRole --force 2025-02-23 00:57:59 [ℹ] Kubernetes version "1.31" in use by cluster "myeks" 2025-02-23 00:58:00 [ℹ] IRSA is set for "aws-ebs-csi-driver" addon; will use this to configure IAM permissions 2025-02-23 00:58:00 [!] the recommended way to provide IAM permissions for "aws-ebs-csi-driver" addon is via pod identity associations; after addon creation is completed, run `eksctl utils migrate-to-pod-identity` 2025-02-23 00:58:00 [ℹ] using provided ServiceAccountRoleARN "arn:aws:iam::390402552337:role/AmazonEKS_EBS_CSI_DriverRole" 2025-02-23 00:58:00 [ℹ] creating addon: aws-ebs-csi-driver [yu3papa@iworks ~]$ kubectl get sa -n kube-system ebs-csi-controller-sa -o yaml | head -5 apiVersion: v1 automountServiceAccountToken: true kind: ServiceAccount metadata: annotations: # 확인 [yu3papa@iworks ~]$ eksctl get addon --cluster ${CLUSTER_NAME} 2025-02-23 00:58:24 [ℹ] Kubernetes version "1.31" in use by cluster "myeks" 2025-02-23 00:58:24 [ℹ] getting all addons 2025-02-23 00:58:26 [ℹ] to see issues for an addon run `eksctl get addon --name <addon-name> --cluster <cluster-name>` NAME VERSION STATUS ISSUES IAMROLE UPDATE AVAILABLE CONFIGURATION VALUES POD IDENTITY ASSOCIATION ROLES aws-ebs-csi-driver v1.39.0-eksbuild.1 CREATING 0 arn:aws:iam::390402552337:role/AmazonEKS_EBS_CSI_DriverRole coredns v1.11.4-eksbuild.2 ACTIVE 0 kube-proxy v1.31.3-eksbuild.2 ACTIVE 0 metrics-server v0.7.2-eksbuild.2 ACTIVE 0 vpc-cni v1.19.2-eksbuild.5 ACTIVE 0 arn:aws:iam::390402552337:role/eksctl-myeks-addon-vpc-cni-Role1-fr0GM10fLspD enableNetworkPolicy: "true" [yu3papa@iworks ~]$ kubectl get deploy,ds -l=app.kubernetes.io/name=aws-ebs-csi-driver -n kube-system NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/ebs-csi-controller 2/2 2 2 24s NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/ebs-csi-node 3 3 3 3 3 kubernetes.io/os=linux 24s daemonset.apps/ebs-csi-node-windows 0 0 0 0 0 kubernetes.io/os=windows 24s [yu3papa@iworks ~]$ kubectl get pod -n kube-system -l 'app in (ebs-csi-controller,ebs-csi-node)' NAME READY STATUS RESTARTS AGE ebs-csi-controller-7f8f8cb84-c44bx 6/6 Running 0 30s ebs-csi-controller-7f8f8cb84-xp72g 6/6 Running 0 30s ebs-csi-node-lx7gk 3/3 Running 0 30s ebs-csi-node-t9gjr 3/3 Running 0 30s ebs-csi-node-z55h4 3/3 Running 0 30s [yu3papa@iworks ~]$ kubectl get pod -n kube-system -l app.kubernetes.io/component=csi-driver NAME READY STATUS RESTARTS AGE ebs-csi-controller-7f8f8cb84-c44bx 6/6 Running 0 35s ebs-csi-controller-7f8f8cb84-xp72g 6/6 Running 0 35s ebs-csi-node-lx7gk 3/3 Running 0 35s ebs-csi-node-t9gjr 3/3 Running 0 35s ebs-csi-node-z55h4 3/3 Running 0 35s # ebs-csi-controller 파드에 6개 컨테이너 확인 [yu3papa@iworks ~]$ kubectl get pod -n kube-system -l app=ebs-csi-controller -o jsonpath='{.items[0].spec.containers[*].name}' ; echo ebs-plugin csi-provisioner csi-attacher csi-snapshotter csi-resizer liveness-probe # csinodes 확인 [yu3papa@iworks ~]$ kubectl api-resources | grep -i csi csidrivers storage.k8s.io/v1 false CSIDriver csinodes storage.k8s.io/v1 false CSINode csistoragecapacities storage.k8s.io/v1 true CSIStorageCapacity [yu3papa@iworks ~]$ kubectl get csinodes NAME DRIVERS AGE ip-192-168-1-193.ap-northeast-2.compute.internal 1 4h47m ip-192-168-2-200.ap-northeast-2.compute.internal 1 4h47m ip-192-168-3-188.ap-northeast-2.compute.internal 1 4h47m [yu3papa@iworks ~]$ kubectl describe csinodes Name: ip-192-168-1-193.ap-northeast-2.compute.internal Labels: <none> Annotations: storage.alpha.kubernetes.io/migrated-plugins: kubernetes.io/aws-ebs,kubernetes.io/azure-disk,kubernetes.io/azure-file,kubernetes.io/cinder,kubernetes.io/gce-pd,kubernetes.io/portworx-v... CreationTimestamp: Sat, 22 Feb 2025 20:11:47 +0900 Spec: Drivers: ebs.csi.aws.com: Node ID: i-02c24086e5ef7bd2f Allocatables: Count: 25 Topology Keys: [kubernetes.io/os topology.ebs.csi.aws.com/zone topology.kubernetes.io/zone] Events: <none> Name: ip-192-168-2-200.ap-northeast-2.compute.internal Labels: <none> Annotations: storage.alpha.kubernetes.io/migrated-plugins: kubernetes.io/aws-ebs,kubernetes.io/azure-disk,kubernetes.io/azure-file,kubernetes.io/cinder,kubernetes.io/gce-pd,kubernetes.io/portworx-v... CreationTimestamp: Sat, 22 Feb 2025 20:11:40 +0900 Spec: Drivers: ebs.csi.aws.com: Node ID: i-064da1db46bbd4d0e Allocatables: Count: 25 Topology Keys: [kubernetes.io/os topology.ebs.csi.aws.com/zone topology.kubernetes.io/zone] Events: <none> Name: ip-192-168-3-188.ap-northeast-2.compute.internal Labels: <none> Annotations: storage.alpha.kubernetes.io/migrated-plugins: kubernetes.io/aws-ebs,kubernetes.io/azure-disk,kubernetes.io/azure-file,kubernetes.io/cinder,kubernetes.io/gce-pd,kubernetes.io/portworx-v... CreationTimestamp: Sat, 22 Feb 2025 20:11:43 +0900 Spec: Drivers: ebs.csi.aws.com: Node ID: i-0fabeb5dd693c0671 Allocatables: Count: 25 Topology Keys: [kubernetes.io/os topology.ebs.csi.aws.com/zone topology.kubernetes.io/zone] Events: <none> [yu3papa@iworks ~]$ kubectl get csidrivers NAME ATTACHREQUIRED PODINFOONMOUNT STORAGECAPACITY TOKENREQUESTS REQUIRESREPUBLISH MODES AGE ebs.csi.aws.com true false false <unset> false Persistent 81s efs.csi.aws.com false false false <unset> false Persistent 4h56m [yu3papa@iworks ~]$ kubectl describe csidrivers ebs.csi.aws.com Name: ebs.csi.aws.com Namespace: Labels: app.kubernetes.io/component=csi-driver app.kubernetes.io/managed-by=EKS app.kubernetes.io/name=aws-ebs-csi-driver app.kubernetes.io/version=1.39.0 Annotations: <none> API Version: storage.k8s.io/v1 Kind: CSIDriver Metadata: Creation Timestamp: 2025-02-22T15:58:08Z Resource Version: 64188 UID: 114d32aa-bdac-4620-b619-273548eb63c9 Spec: Attach Required: true Fs Group Policy: ReadWriteOnceWithFSType Pod Info On Mount: false Requires Republish: false Se Linux Mount: false Storage Capacity: false Volume Lifecycle Modes: Persistent Events: <none> # (참고) 노드에 최대 EBS 부착 수량 변경 [yu3papa@iworks ~]$ aws eks update-addon --cluster-name ${CLUSTER_NAME} --addon-name aws-ebs-csi-driver \ --addon-version v1.39.0-eksbuild.1 --configuration-values '{ "node": { "volumeAttachLimit": 31, "enableMetrics": true } }' { "update": { "id": "c763589d-e192-3cc4-8049-3db99aeb9909", "status": "InProgress", "type": "AddonUpdate", "params": [ { "type": "AddonVersion", "value": "v1.39.0-eksbuild.1" }, { "type": "ConfigurationValues", "value": "{\n \"node\": {\n \"volumeAttachLimit\": 31,\n \"enableMetrics\": true\n }\n }" } ], "createdAt": "2025-02-23T01:00:04.223000+09:00", "errors": [] } } # 확인 [yu3papa@iworks ~]$ kubectl get ds -n kube-system ebs-csi-node -o yaml apiVersion: apps/v1 kind: DaemonSet metadata: annotations: deprecated.daemonset.template.generation: "2" creationTimestamp: "2025-02-22T15:58:08Z" generation: 2 labels: app.kubernetes.io/component: csi-driver app.kubernetes.io/managed-by: EKS app.kubernetes.io/name: aws-ebs-csi-driver app.kubernetes.io/version: 1.39.0 name: ebs-csi-node namespace: kube-system resourceVersion: "65016" uid: fc40a2f9-e582-4238-a962-a0259b87c22a spec: revisionHistoryLimit: 10 selector: matchLabels: app: ebs-csi-node app.kubernetes.io/name: aws-ebs-csi-driver template: metadata: creationTimestamp: null labels: app: ebs-csi-node app.kubernetes.io/component: csi-driver app.kubernetes.io/managed-by: EKS app.kubernetes.io/name: aws-ebs-csi-driver app.kubernetes.io/version: 1.39.0 spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: eks.amazonaws.com/compute-type operator: NotIn values: - fargate - auto - hybrid - key: node.kubernetes.io/instance-type operator: NotIn values: - a1.medium - a1.large - a1.xlarge - a1.2xlarge - a1.4xlarge containers: - args: - node - --endpoint=$(CSI_ENDPOINT) - --http-endpoint=0.0.0.0:3302 - --csi-mount-point-prefix=/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/ - --volume-attach-limit=31 - --logging-format=text - --v=2 env: - name: CSI_ENDPOINT value: unix:/csi/csi.sock - name: CSI_NODE_NAME valueFrom: fieldRef: apiVersion: v1 fieldPath: spec.nodeName image: 602401143452.dkr.ecr.ap-northeast-2.amazonaws.com/eks/aws-ebs-csi-driver:v1.39.0 imagePullPolicy: IfNotPresent lifecycle: preStop: exec: command: - /bin/aws-ebs-csi-driver - pre-stop-hook livenessProbe: failureThreshold: 5 httpGet: path: /healthz port: healthz scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 3 name: ebs-plugin ports: - containerPort: 9808 name: healthz protocol: TCP resources: limits: memory: 256Mi requests: cpu: 10m memory: 40Mi securityContext: privileged: true readOnlyRootFilesystem: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /var/lib/kubelet mountPropagation: Bidirectional name: kubelet-dir - mountPath: /csi name: plugin-dir - mountPath: /dev name: device-dir - args: - --csi-address=$(ADDRESS) - --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH) - --v=2 env: - name: ADDRESS value: /csi/csi.sock - name: DRIVER_REG_SOCK_PATH value: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock image: 602401143452.dkr.ecr.ap-northeast-2.amazonaws.com/eks/csi-node-driver-registrar:v2.13.0-eks-1-31-12 imagePullPolicy: IfNotPresent livenessProbe: exec: command: - /csi-node-driver-registrar - --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH) - --mode=kubelet-registration-probe failureThreshold: 3 initialDelaySeconds: 30 periodSeconds: 90 successThreshold: 1 timeoutSeconds: 15 name: node-driver-registrar resources: limits: memory: 256Mi requests: cpu: 10m memory: 40Mi securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /csi name: plugin-dir - mountPath: /registration name: registration-dir - mountPath: /var/lib/kubelet/plugins/ebs.csi.aws.com/ name: probe-dir - args: - --csi-address=/csi/csi.sock image: 602401143452.dkr.ecr.ap-northeast-2.amazonaws.com/eks/livenessprobe:v2.14.0-eks-1-31-12 imagePullPolicy: IfNotPresent name: liveness-probe resources: limits: memory: 256Mi requests: cpu: 10m memory: 40Mi securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /csi name: plugin-dir dnsPolicy: ClusterFirst nodeSelector: kubernetes.io/os: linux priorityClassName: system-node-critical restartPolicy: Always schedulerName: default-scheduler securityContext: fsGroup: 0 runAsGroup: 0 runAsNonRoot: false runAsUser: 0 serviceAccount: ebs-csi-node-sa serviceAccountName: ebs-csi-node-sa terminationGracePeriodSeconds: 30 tolerations: - operator: Exists volumes: - hostPath: path: /var/lib/kubelet type: Directory name: kubelet-dir - hostPath: path: /var/lib/kubelet/plugins/ebs.csi.aws.com/ type: DirectoryOrCreate name: plugin-dir - hostPath: path: /var/lib/kubelet/plugins_registry/ type: Directory name: registration-dir - hostPath: path: /dev type: Directory name: device-dir - emptyDir: {} name: probe-dir updateStrategy: rollingUpdate: maxSurge: 0 maxUnavailable: 10% type: RollingUpdate status: currentNumberScheduled: 3 desiredNumberScheduled: 3 numberAvailable: 3 numberMisscheduled: 0 numberReady: 3 observedGeneration: 2 updatedNumberScheduled: 3 [yu3papa@iworks ~]$ kubectl describe csinodes Name: ip-192-168-1-193.ap-northeast-2.compute.internal Labels: <none> Annotations: storage.alpha.kubernetes.io/migrated-plugins: kubernetes.io/aws-ebs,kubernetes.io/azure-disk,kubernetes.io/azure-file,kubernetes.io/cinder,kubernetes.io/gce-pd,kubernetes.io/portworx-v... CreationTimestamp: Sat, 22 Feb 2025 20:11:47 +0900 Spec: Drivers: ebs.csi.aws.com: Node ID: i-02c24086e5ef7bd2f Allocatables: Count: 31 Topology Keys: [kubernetes.io/os topology.ebs.csi.aws.com/zone topology.kubernetes.io/zone] Events: <none> Name: ip-192-168-2-200.ap-northeast-2.compute.internal Labels: <none> Annotations: storage.alpha.kubernetes.io/migrated-plugins: kubernetes.io/aws-ebs,kubernetes.io/azure-disk,kubernetes.io/azure-file,kubernetes.io/cinder,kubernetes.io/gce-pd,kubernetes.io/portworx-v... CreationTimestamp: Sat, 22 Feb 2025 20:11:40 +0900 Spec: Drivers: ebs.csi.aws.com: Node ID: i-064da1db46bbd4d0e Allocatables: Count: 31 Topology Keys: [kubernetes.io/os topology.ebs.csi.aws.com/zone topology.kubernetes.io/zone] Events: <none> Name: ip-192-168-3-188.ap-northeast-2.compute.internal Labels: <none> Annotations: storage.alpha.kubernetes.io/migrated-plugins: kubernetes.io/aws-ebs,kubernetes.io/azure-disk,kubernetes.io/azure-file,kubernetes.io/cinder,kubernetes.io/gce-pd,kubernetes.io/portworx-v... CreationTimestamp: Sat, 22 Feb 2025 20:11:43 +0900 Spec: Drivers: ebs.csi.aws.com: Node ID: i-0fabeb5dd693c0671 Allocatables: Count: 31 Topology Keys: [kubernetes.io/os topology.ebs.csi.aws.com/zone topology.kubernetes.io/zone] Events: <none> |
gp3 스토리지 클래스 생성
- AWS EBS 스토리지 클래스 파라미터
스토리지 클래스
이 문서는 쿠버네티스의 스토리지클래스의 개념을 설명한다. 볼륨과 퍼시스턴트 볼륨에 익숙해지는 것을 권장한다. 소개 스토리지클래스는 관리자가 제공하는 스토리지의 "classes"를 설명할 수
kubernetes.io
aws-ebs-csi-driver/docs/parameters.md at master · kubernetes-sigs/aws-ebs-csi-driver
CSI driver for Amazon EBS https://aws.amazon.com/ebs/ - kubernetes-sigs/aws-ebs-csi-driver
github.com
| # gp3 스토리지 클래스 생성 [yu3papa@iworks ~]$ kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2 kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 5h11m local-path rancher.io/local-path Delete WaitForFirstConsumer false 61m [yu3papa@iworks ~]$ cat <<EOF | kubectl apply -f - kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: gp3 annotations: storageclass.kubernetes.io/is-default-class: "true" allowVolumeExpansion: true provisioner: ebs.csi.aws.com volumeBindingMode: WaitForFirstConsumer parameters: type: gp3 #iops: "5000" #throughput: "250" allowAutoIOPSPerGBIncrease: 'true' encrypted: 'true' fsType: xfs # 기본값이 ext4 EOF storageclass.storage.k8s.io/gp3 created [yu3papa@iworks ~]$ kubectl get sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE gp2 kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 5h11m gp3 (default) ebs.csi.aws.com Delete WaitForFirstConsumer true 5s local-path rancher.io/local-path Delete WaitForFirstConsumer false 61m [yu3papa@iworks ~]$ kubectl describe sc gp3 | grep Parameters Parameters: allowAutoIOPSPerGBIncrease=true,encrypted=true,fsType=xfs,type=gp3 |
- volumeBindingMode 필드는 [볼륨 바인딩과 동적 프로비저닝] 의 시작 시기를 제어합니다. 설정되어 있지 않으면, Immediate 모드가 기본으로 사용된다.
- Immediate 모드는 퍼시스턴트볼륨클레임이 생성되면 볼륨 바인딩과 동적 프로비저닝이 즉시 발생하는 것을 나타냅니다. 토폴로지 제약이 있고 클러스터의 모든 노드에서 전역적으로 접근할 수 없는 스토리지 백엔드의 경우, 파드의 스케줄링 요구 사항에 대한 파악없이 퍼시스턴트볼륨이 바인딩되거나 프로비저닝되며, 이로 인해 스케줄되지 않은 파드가 발생할 수 있습니다.
- WaitForFirstConsumer 모드를 지정해서 이 문제를 해결할 수 있는데 이 모드는 퍼시스턴트볼륨클레임을 사용하는 파드가 생성될 때까지 퍼시스턴트볼륨의 바인딩과 프로비저닝을 지연시킵니다. 퍼시스턴트볼륨은 파드의 스케줄링 제약 조건에 의해 지정된 토폴로지에 따라 선택되거나 프로비저닝 됩니다. 여기에는 [리소스 요구 사항], [노드 셀렉터](, [파드 어피니티(affinity)와 안티-어피니티(anti-affinity)] 그리고 [테인트(taint)와 톨러레이션(toleration)] 포함됩니다.
PVC/PV 파드 테스트
| # 워커노드의 EBS 볼륨 확인 : tag(키/값) 필터링 [yu3papa@iworks ~]$ aws ec2 describe-volumes --filters Name=tag:Name,Values=$CLUSTER_NAME-ng1-Node --output table -------------------------------------------------------------- | DescribeVolumes | +------------------------------------------------------------+ || Volumes || |+---------------------+------------------------------------+| || AvailabilityZone | ap-northeast-2b || || CreateTime | 2025-02-22T11:10:59.623000+00:00 || || Encrypted | False || || Iops | 3000 || || MultiAttachEnabled | False || || Size | 120 || || SnapshotId | snap-09234f0993e1a70d3 || || State | in-use || || Throughput | 125 || || VolumeId | vol-0a83cf8d8a74eb723 || || VolumeType | gp3 || |+---------------------+------------------------------------+| ||| Attachments ||| ||+------------------------+-------------------------------+|| ||| AttachTime | 2025-02-22T11:10:59+00:00 ||| ||| DeleteOnTermination | True ||| ||| Device | /dev/xvda ||| ||| InstanceId | i-064da1db46bbd4d0e ||| ||| State | attached ||| ||| VolumeId | vol-0a83cf8d8a74eb723 ||| ||+------------------------+-------------------------------+|| ||| Operator ||| ||+------------------------------+-------------------------+|| ||| Managed | False ||| ||+------------------------------+-------------------------+|| ||| Tags ||| ||+------------------------------------+-------------------+|| ||| Key | Value ||| ||+------------------------------------+-------------------+|| [yu3papa@iworks ~]$ aws ec2 describe-volumes --filters Name=tag:Name,Values=$CLUSTER_NAME-ng1-Node --query "Volumes[*].Attachments" | jq [ [ { "DeleteOnTermination": true, "VolumeId": "vol-0a83cf8d8a74eb723", "InstanceId": "i-064da1db46bbd4d0e", "Device": "/dev/xvda", "State": "attached", "AttachTime": "2025-02-22T11:10:59+00:00" } ], [ { "DeleteOnTermination": true, "VolumeId": "vol-0b72a2124a83fa5a1", "InstanceId": "i-02c24086e5ef7bd2f", "Device": "/dev/xvda", "State": "attached", "AttachTime": "2025-02-22T11:10:59+00:00" } ], [ { "DeleteOnTermination": true, "VolumeId": "vol-0f01e60b9b8617f5f", "InstanceId": "i-0fabeb5dd693c0671", "Device": "/dev/xvda", "State": "attached", "AttachTime": "2025-02-22T11:10:59+00:00" } ] ] [yu3papa@iworks ~]$ aws ec2 describe-volumes --filters Name=tag:Name,Values=$CLUSTER_NAME-ng1-Node --query "Volumes[*].{ID:VolumeId,Tag:Tags}" | jq [ { "ID": "vol-0a83cf8d8a74eb723", "Tag": [ { "Key": "alpha.eksctl.io/nodegroup-type", "Value": "managed" }, { "Key": "Name", "Value": "myeks-ng1-Node" }, { "Key": "alpha.eksctl.io/nodegroup-name", "Value": "ng1" }, { "Key": "eks:nodegroup-name", "Value": "ng1" }, { "Key": "eks:cluster-name", "Value": "myeks" } ] }, { "ID": "vol-0b72a2124a83fa5a1", "Tag": [ { "Key": "eks:nodegroup-name", "Value": "ng1" }, { "Key": "alpha.eksctl.io/nodegroup-type", "Value": "managed" }, { "Key": "Name", "Value": "myeks-ng1-Node" }, { "Key": "eks:cluster-name", "Value": "myeks" }, { "Key": "alpha.eksctl.io/nodegroup-name", "Value": "ng1" } ] }, { "ID": "vol-0f01e60b9b8617f5f", "Tag": [ { "Key": "eks:nodegroup-name", "Value": "ng1" }, { "Key": "Name", "Value": "myeks-ng1-Node" }, { "Key": "alpha.eksctl.io/nodegroup-name", "Value": "ng1" }, { "Key": "alpha.eksctl.io/nodegroup-type", "Value": "managed" }, { "Key": "eks:cluster-name", "Value": "myeks" } ] } ] [yu3papa@iworks ~]$ aws ec2 describe-volumes --filters Name=tag:Name,Values=$CLUSTER_NAME-ng1-Node --query "Volumes[].[VolumeId, VolumeType, Attachments[].[InstanceId, State][]][]" | jq [ "vol-0a83cf8d8a74eb723", "gp3", [ "i-064da1db46bbd4d0e", "attached" ], "vol-0b72a2124a83fa5a1", "gp3", [ "i-02c24086e5ef7bd2f", "attached" ], "vol-0f01e60b9b8617f5f", "gp3", [ "i-0fabeb5dd693c0671", "attached" ] ] [yu3papa@iworks ~]$ aws ec2 describe-volumes --filters Name=tag:Name,Values=$CLUSTER_NAME-ng1-Node --query "Volumes[].{VolumeId: VolumeId, VolumeType: VolumeType, InstanceId: Attachments[0].InstanceId, State: Attachments[0].State}" | jq [ { "VolumeId": "vol-0a83cf8d8a74eb723", "VolumeType": "gp3", "InstanceId": "i-064da1db46bbd4d0e", "State": "attached" }, { "VolumeId": "vol-0b72a2124a83fa5a1", "VolumeType": "gp3", "InstanceId": "i-02c24086e5ef7bd2f", "State": "attached" }, { "VolumeId": "vol-0f01e60b9b8617f5f", "VolumeType": "gp3", "InstanceId": "i-0fabeb5dd693c0671", "State": "attached" } ] # 워커노드에서 파드에 추가한 EBS 볼륨 확인 [yu3papa@iworks ~]$ aws ec2 describe-volumes --filters Name=tag:ebs.csi.aws.com/cluster,Values=true --output table ----------------- |DescribeVolumes| +---------------+ [yu3papa@iworks ~]$ aws ec2 describe-volumes --filters Name=tag:ebs.csi.aws.com/cluster,Values=true --query "Volumes[*].{ID:VolumeId,Tag:Tags}" | jq [] [yu3papa@iworks ~]$ aws ec2 describe-volumes --filters Name=tag:ebs.csi.aws.com/cluster,Values=true --query "Volumes[].{VolumeId: VolumeId, VolumeType: VolumeType, InstanceId: Attachments[0].InstanceId, State: Attachments[0].State}" | jq [] # 워커노드에서 파드에 추가한 EBS 볼륨 모니터링 [yu3papa@iworks ~]$ while true; do aws ec2 describe-volumes --filters Name=tag:ebs.csi.aws.com/cluster,Values=true --query "Volumes[].{VolumeId: VolumeId, VolumeType: VolumeType, InstanceId: Attachments[0].InstanceId, State: Attachments[0].State}" --output text; date; sleep 1; done Sun Feb 23 01:22:41 AM KST 2025 Sun Feb 23 01:22:43 AM KST 2025 Sun Feb 23 01:22:45 AM KST 2025 Sun Feb 23 01:22:46 AM KST 2025 Sun Feb 23 01:22:48 AM KST 2025 Sun Feb 23 01:22:50 AM KST 2025 ^C # PVC 생성 [yu3papa@iworks ~]$ cat <<EOF | kubectl apply -f - apiVersion: v1 kind: PersistentVolumeClaim metadata: name: ebs-claim spec: accessModes: - ReadWriteOnce resources: requests: storage: 4Gi storageClassName: gp3 EOF persistentvolumeclaim/ebs-claim created [yu3papa@iworks ~]$ kubectl get pvc,pv NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE persistentvolumeclaim/ebs-claim Pending gp3 <unset> 8s # 파드 생성 [yu3papa@iworks ~]$ cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: app spec: terminationGracePeriodSeconds: 3 containers: - name: app image: centos command: ["/bin/sh"] args: ["-c", "while true; do echo \$(date -u) >> /data/out.txt; sleep 5; done"] volumeMounts: - name: persistent-storage mountPath: /data volumes: - name: persistent-storage persistentVolumeClaim: claimName: ebs-claim EOF pod/app created # PVC, 파드 확인 [yu3papa@iworks ~]$ kubectl get pvc,pv,pod NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE persistentvolumeclaim/ebs-claim Bound pvc-de01bd9c-66c2-4f57-bb56-932b42ff8707 4Gi RWO gp3 <unset> 42s NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE persistentvolume/pvc-de01bd9c-66c2-4f57-bb56-932b42ff8707 4Gi RWO Delete Bound default/ebs-claim gp3 <unset> 14s NAME READY STATUS RESTARTS AGE pod/app 1/1 Running 0 17s [yu3papa@iworks ~]$ kubectl get VolumeAttachment NAME ATTACHER PV NODE ATTACHED AGE csi-dee1d07b3e8042f3e2ff3d303663acb042aa31bbe632c5a43e3ac84bbcf78066 ebs.csi.aws.com pvc-de01bd9c-66c2-4f57-bb56-932b42ff8707 ip-192-168-1-193.ap-northeast-2.compute.internal true 21s [yu3papa@iworks ~]$ kubectl df-pv error: unknown command "df-pv" for "kubectl" # 추가된 EBS 볼륨 상세 정보 확인 : AWS 관리콘솔 EC2(EBS)에서 확인 [yu3papa@iworks ~]$ aws ec2 describe-volumes --volume-ids $(kubectl get pv -o jsonpath="{.items[0].spec.csi.volumeHandle}") | jq { "Volumes": [ { "Iops": 3000, "Tags": [ { "Key": "kubernetes.io/created-for/pvc/namespace", "Value": "default" }, { "Key": "KubernetesCluster", "Value": "myeks" }, { "Key": "ebs.csi.aws.com/cluster", "Value": "true" }, { "Key": "Name", "Value": "myeks-dynamic-pvc-de01bd9c-66c2-4f57-bb56-932b42ff8707" }, { "Key": "kubernetes.io/created-for/pv/name", "Value": "pvc-de01bd9c-66c2-4f57-bb56-932b42ff8707" }, { "Key": "kubernetes.io/cluster/myeks", "Value": "owned" }, { "Key": "CSIVolumeName", "Value": "pvc-de01bd9c-66c2-4f57-bb56-932b42ff8707" }, { "Key": "kubernetes.io/created-for/pvc/name", "Value": "ebs-claim" } ], "VolumeType": "gp3", "MultiAttachEnabled": false, "Throughput": 125, "Operator": { "Managed": false }, "VolumeId": "vol-0f429b1f1d2588ede", "Size": 4, "SnapshotId": "", "AvailabilityZone": "ap-northeast-2a", "State": "in-use", "CreateTime": "2025-02-22T16:23:37.220000+00:00", "Attachments": [ { "DeleteOnTermination": false, "VolumeId": "vol-0f429b1f1d2588ede", "InstanceId": "i-02c24086e5ef7bd2f", "Device": "/dev/xvdaa", "State": "attached", "AttachTime": "2025-02-22T16:23:40+00:00" } ], "Encrypted": true, "KmsKeyId": "arn:aws:kms:ap-northeast-2:390402552337:key/d5aee190-b097-4f00-a466-0f13e41a2435" } ] } # PV 상세 확인 : nodeAffinity 내용확인 [yu3papa@iworks ~]$ kubectl get pv -o yaml apiVersion: v1 items: - apiVersion: v1 kind: PersistentVolume metadata: annotations: pv.kubernetes.io/provisioned-by: ebs.csi.aws.com volume.kubernetes.io/provisioner-deletion-secret-name: "" volume.kubernetes.io/provisioner-deletion-secret-namespace: "" creationTimestamp: "2025-02-22T16:23:39Z" finalizers: - external-provisioner.volume.kubernetes.io/finalizer - kubernetes.io/pv-protection - external-attacher/ebs-csi-aws-com name: pvc-de01bd9c-66c2-4f57-bb56-932b42ff8707 resourceVersion: "71534" uid: d047b59d-c61e-4ff3-bd8d-06f67436bd93 spec: accessModes: - ReadWriteOnce capacity: storage: 4Gi claimRef: apiVersion: v1 kind: PersistentVolumeClaim name: ebs-claim namespace: default resourceVersion: "71507" uid: de01bd9c-66c2-4f57-bb56-932b42ff8707 csi: driver: ebs.csi.aws.com fsType: xfs volumeAttributes: storage.kubernetes.io/csiProvisionerIdentity: 1740239893145-1296-ebs.csi.aws.com volumeHandle: vol-0f429b1f1d2588ede nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: topology.kubernetes.io/zone operator: In values: - ap-northeast-2a persistentVolumeReclaimPolicy: Delete storageClassName: gp3 volumeMode: Filesystem status: lastPhaseTransitionTime: "2025-02-22T16:23:39Z" phase: Bound kind: List metadata: resourceVersion: "" [yu3papa@iworks ~]$ kubectl get node --label-columns=topology.ebs.csi.aws.com/zone,topology.k8s.aws/zone-id NAME STATUS ROLES AGE VERSION ZONE ZONE-ID ip-192-168-1-193.ap-northeast-2.compute.internal Ready <none> 5h13m v1.31.5-eks-5d632ec ap-northeast-2a apne2-az1 ip-192-168-2-200.ap-northeast-2.compute.internal Ready <none> 5h13m v1.31.5-eks-5d632ec ap-northeast-2b apne2-az2 ip-192-168-3-188.ap-northeast-2.compute.internal Ready <none> 5h13m v1.31.5-eks-5d632ec ap-northeast-2c apne2-az3 # 파일 내용 추가 저장 확인 [yu3papa@iworks ~]$ kubectl exec app -- tail -f /data/out.txt Sat Feb 22 16:24:25 UTC 2025 Sat Feb 22 16:24:30 UTC 2025 Sat Feb 22 16:24:35 UTC 2025 Sat Feb 22 16:24:40 UTC 2025 Sat Feb 22 16:24:45 UTC 2025 Sat Feb 22 16:24:50 UTC 2025 Sat Feb 22 16:24:55 UTC 2025 Sat Feb 22 16:25:00 UTC 2025 Sat Feb 22 16:25:05 UTC 2025 Sat Feb 22 16:25:10 UTC 2025 ^C # 파드 내에서 볼륨 정보 확인 [yu3papa@iworks ~]$ kubectl exec -it app -- sh -c 'df -hT --type=overlay' Filesystem Type Size Used Avail Use% Mounted on overlay overlay 120G 5.0G 115G 5% / [yu3papa@iworks ~]$ kubectl exec -it app -- sh -c 'df -hT --type=xfs' Filesystem Type Size Used Avail Use% Mounted on /dev/nvme1n1 xfs 4.0G 61M 3.9G 2% /data /dev/nvme0n1p1 xfs 120G 5.0G 115G 5% /etc/hosts |
Resizing Persistent Volumes using Kubernetes
Editor’s note: this post is part of a series of in-depth articles on what’s new in Kubernetes 1.11 In Kubernetes v1.11 the persistent volume expansion feature is being promoted to beta. This feature allows users to easily resize an existing volume by e
kubernetes.io
| # 현재 pv 의 이름을 기준하여 4G > 10G 로 증가 : .spec.resources.requests.storage의 4Gi 를 10Gi로 변경 [yu3papa@iworks ~]$ kubectl get pvc ebs-claim -o jsonpath={.spec.resources.requests.storage} ; echo 4Gi [yu3papa@iworks ~]$ kubectl get pvc ebs-claim -o jsonpath={.status.capacity.storage} ; echo 4Gi [yu3papa@iworks ~]$ kubectl patch pvc ebs-claim -p '{"spec":{"resources":{"requests":{"storage":"10Gi"}}}}' persistentvolumeclaim/ebs-claim patched # 확인 : 볼륨 용량 수정 반영이 되어야 되니, 수치 반영이 조금 느릴수 있다 [yu3papa@iworks ~]$ kubectl exec -it app -- sh -c 'df -hT --type=xfs' Filesystem Type Size Used Avail Use% Mounted on /dev/nvme1n1 xfs 4.0G 61M 3.9G 2% /data /dev/nvme0n1p1 xfs 120G 5.0G 115G 5% /etc/hosts [yu3papa@iworks ~]$ kubectl exec -it app -- sh -c 'df -hT --type=xfs' Filesystem Type Size Used Avail Use% Mounted on /dev/nvme1n1 xfs 10G 105M 9.9G 2% /data /dev/nvme0n1p1 xfs 120G 5.0G 115G 5% /etc/hosts [yu3papa@iworks ~]$ aws ec2 describe-volumes --volume-ids $(kubectl get pv -o jsonpath="{.items[0].spec.csi.volumeHandle}") | jq { "Volumes": [ { "Iops": 3000, "Tags": [ { "Key": "kubernetes.io/created-for/pvc/namespace", "Value": "default" }, { "Key": "KubernetesCluster", "Value": "myeks" }, { "Key": "ebs.csi.aws.com/cluster", "Value": "true" }, { "Key": "Name", "Value": "myeks-dynamic-pvc-de01bd9c-66c2-4f57-bb56-932b42ff8707" }, { "Key": "kubernetes.io/created-for/pv/name", "Value": "pvc-de01bd9c-66c2-4f57-bb56-932b42ff8707" }, { "Key": "kubernetes.io/cluster/myeks", "Value": "owned" }, { "Key": "CSIVolumeName", "Value": "pvc-de01bd9c-66c2-4f57-bb56-932b42ff8707" }, { "Key": "kubernetes.io/created-for/pvc/name", "Value": "ebs-claim" } ], "VolumeType": "gp3", "MultiAttachEnabled": false, "Throughput": 125, "Operator": { "Managed": false }, "VolumeId": "vol-0f429b1f1d2588ede", "Size": 10, "SnapshotId": "", "AvailabilityZone": "ap-northeast-2a", "State": "in-use", "CreateTime": "2025-02-22T16:23:37.220000+00:00", "Attachments": [ { "DeleteOnTermination": false, "VolumeId": "vol-0f429b1f1d2588ede", "InstanceId": "i-02c24086e5ef7bd2f", "Device": "/dev/xvdaa", "State": "attached", "AttachTime": "2025-02-22T16:23:40+00:00" } ], "Encrypted": true, "KmsKeyId": "arn:aws:kms:ap-northeast-2:390402552337:key/d5aee190-b097-4f00-a466-0f13e41a2435" } ] } |
- 삭제
| [yu3papa@iworks ~]$ kubectl delete pod app pod "app" deleted [yu3papa@iworks ~]$ kubectl delete pvc ebs-claim persistentvolumeclaim "ebs-claim" deleted |
5. AWS EFS Controller
EFS 파일시스템 확인 및 EFS Controller Addon 설치
- 구성 아키텍처
| # EFS 정보 확인 [yu3papa@iworks ~]$ aws efs describe-file-systems --query "FileSystems[*].FileSystemId" --output text fs-04bdbcbb996487caf # 아래는 aws-efs-csi-driver 전체 버전 정보와 기본 설치 버전(True) 정보 확인 [yu3papa@iworks ~]$ aws eks describe-addon-versions \ --addon-name aws-efs-csi-driver \ --kubernetes-version 1.31 \ --query "addons[].addonVersions[].[addonVersion, compatibilities[].defaultVersion]" \ --output text v2.1.4-eksbuild.1 True v2.1.3-eksbuild.1 False v2.1.2-eksbuild.1 False v2.1.1-eksbuild.1 False v2.1.0-eksbuild.1 False v2.0.9-eksbuild.1 False v2.0.8-eksbuild.1 False v2.0.7-eksbuild.1 False v2.0.6-eksbuild.2 False v2.0.6-eksbuild.1 False v2.0.5-eksbuild.1 False v2.0.4-eksbuild.1 False v2.0.3-eksbuild.1 False v2.0.2-eksbuild.1 False v2.0.1-eksbuild.1 False v2.0.0-eksbuild.1 False v1.7.7-eksbuild.1 False # ISRA 설정 : 고객관리형 정책 AmazonEKS_EFS_CSI_Driver_Policy 사용 [yu3papa@iworks ~]$ eksctl create iamserviceaccount \ --name efs-csi-controller-sa \ --namespace kube-system \ --cluster ${CLUSTER_NAME} \ --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEFSCSIDriverPolicy \ --approve \ --role-only \ --role-name AmazonEKS_EFS_CSI_DriverRole 2025-02-23 01:43:48 [ℹ] 2 existing iamserviceaccount(s) (kube-system/aws-load-balancer-controller,kube-system/ebs-csi-controller-sa) will be excluded 2025-02-23 01:43:48 [ℹ] 1 iamserviceaccount (kube-system/efs-csi-controller-sa) was included (based on the include/exclude rules) 2025-02-23 01:43:48 [!] serviceaccounts in Kubernetes will not be created or modified, since the option --role-only is used 2025-02-23 01:43:48 [ℹ] 1 task: { create IAM role for serviceaccount "kube-system/efs-csi-controller-sa" } 2025-02-23 01:43:48 [ℹ] building iamserviceaccount stack "eksctl-myeks-addon-iamserviceaccount-kube-system-efs-csi-controller-sa" 2025-02-23 01:43:48 [ℹ] deploying stack "eksctl-myeks-addon-iamserviceaccount-kube-system-efs-csi-controller-sa" 2025-02-23 01:43:48 [ℹ] waiting for CloudFormation stack "eksctl-myeks-addon-iamserviceaccount-kube-system-efs-csi-controller-sa" 2025-02-23 01:44:18 [ℹ] waiting for CloudFormation stack "eksctl-myeks-addon-iamserviceaccount-kube-system-efs-csi-controller-sa" # ISRA 확인 [yu3papa@iworks ~]$ eksctl get iamserviceaccount --cluster ${CLUSTER_NAME} NAMESPACE NAME ROLE ARN kube-system aws-load-balancer-controller arn:aws:iam::390402552337:role/eksctl-myeks-addon-iamserviceaccount-kube-sys-Role1-zzbfj7a4hBaW kube-system ebs-csi-controller-sa arn:aws:iam::390402552337:role/AmazonEKS_EBS_CSI_DriverRole kube-system efs-csi-controller-sa arn:aws:iam::390402552337:role/AmazonEKS_EFS_CSI_DriverRole # Amazon EFS CSI driver addon 배포(설치) [yu3papa@iworks ~]$ export ACCOUNT_ID=$(aws sts get-caller-identity --query 'Account' --output text) [yu3papa@iworks ~]$ eksctl create addon --name aws-efs-csi-driver --cluster ${CLUSTER_NAME} --service-account-role-arn arn:aws:iam::${ACCOUNT_ID}:role/AmazonEKS_EFS_CSI_DriverRole --force 2025-02-23 01:48:15 [ℹ] Kubernetes version "1.31" in use by cluster "myeks" 2025-02-23 01:48:16 [ℹ] IRSA is set for "aws-efs-csi-driver" addon; will use this to configure IAM permissions 2025-02-23 01:48:16 [!] the recommended way to provide IAM permissions for "aws-efs-csi-driver" addon is via pod identity associations; after addon creation is completed, run `eksctl utils migrate-to-pod-identity` 2025-02-23 01:48:16 [ℹ] using provided ServiceAccountRoleARN "arn:aws:iam::390402552337:role/AmazonEKS_EFS_CSI_DriverRole" 2025-02-23 01:48:16 [ℹ] creating addon: aws-efs-csi-driver [yu3papa@iworks ~]$ kubectl get sa -n kube-system efs-csi-controller-sa -o yaml | head -5 apiVersion: v1 kind: ServiceAccount metadata: annotations: eks.amazonaws.com/role-arn: arn:aws:iam::390402552337:role/AmazonEKS_EFS_CSI_DriverRole # 확인 [yu3papa@iworks ~]$ eksctl get addon --cluster ${CLUSTER_NAME} 2025-02-23 01:48:43 [ℹ] Kubernetes version "1.31" in use by cluster "myeks" 2025-02-23 01:48:43 [ℹ] getting all addons 2025-02-23 01:48:44 [ℹ] to see issues for an addon run `eksctl get addon --name <addon-name> --cluster <cluster-name>` NAME VERSION STATUS ISSUES IAMROLE UPDATE AVAILABLE CONFIGURATION VALUES POD IDENTITY ASSOCIATION ROLES aws-ebs-csi-driver v1.39.0-eksbuild.1 ACTIVE 0 { "node": { "volumeAttachLimit": 31, "enableMetrics": true } } aws-efs-csi-driver v2.1.4-eksbuild.1 CREATING 0 arn:aws:iam::390402552337:role/AmazonEKS_EFS_CSI_DriverRole coredns v1.11.4-eksbuild.2 ACTIVE 0 kube-proxy v1.31.3-eksbuild.2 ACTIVE 0 metrics-server v0.7.2-eksbuild.2 ACTIVE 0 vpc-cni v1.19.2-eksbuild.5 ACTIVE 0 arn:aws:iam::390402552337:role/eksctl-myeks-addon-vpc-cni-Role1-fr0GM10fLspD enableNetworkPolicy: "true" [yu3papa@iworks ~]$ kubectl get pod -n kube-system -l "app.kubernetes.io/name=aws-efs-csi-driver,app.kubernetes.io/instance=aws-efs-csi-driver" NAME READY STATUS RESTARTS AGE efs-csi-controller-64fc4bc65d-84bq4 3/3 Running 0 31s efs-csi-controller-64fc4bc65d-dg8h9 3/3 Running 0 31s efs-csi-node-2q79t 3/3 Running 0 31s efs-csi-node-jwwh6 3/3 Running 0 31s efs-csi-node-qt24t 3/3 Running 0 31s [yu3papa@iworks ~]$ kubectl get pod -n kube-system -l app=efs-csi-controller -o jsonpath='{.items[0].spec.containers[*].name}' ; echo efs-plugin csi-provisioner liveness-probe [yu3papa@iworks ~]$ kubectl get csidrivers efs.csi.aws.com -o yaml apiVersion: storage.k8s.io/v1 kind: CSIDriver metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | {"apiVersion":"storage.k8s.io/v1","kind":"CSIDriver","metadata":{"annotations":{},"name":"efs.csi.aws.com"},"spec":{"attachRequired":false}} creationTimestamp: "2025-02-22T11:02:35Z" name: efs.csi.aws.com resourceVersion: "78511" uid: 4446fc0c-7d71-455b-bedd-9ac6716bf4d9 spec: attachRequired: false fsGroupPolicy: ReadWriteOnceWithFSType podInfoOnMount: false requiresRepublish: false seLinuxMount: false storageCapacity: false volumeLifecycleModes: - Persistent |
- AWS → EFS → 파일 시스템 : 네트워크 확인
- EFS 파일시스템을 파드가 사용하게 설정 : Add empty StorageClasses from static example
| # 모니터링 [yu3papa@iworks ~]$ watch 'kubectl get sc efs-sc; echo; kubectl get pv,pvc,pod'^C # [운영 서버 EC2] # 실습 코드 clone [yu3papa@iworks ~]$ ssh -i aews-yu3papa.pem ec2-user@15.164.244.197 Last login: Sat Feb 22 22:38:31 2025 from 221.149.127.57 , #_ ~\_ ####_ Amazon Linux 2 ~~ \_#####\ ~~ \###| AL2 End of Life is 2026-06-30. ~~ \#/ ___ ~~ V~' '-> ~~~ / A newer version of Amazon Linux is available! ~~._. _/ _/ _/ Amazon Linux 2023, GA and supported until 2028-03-15. _/m/' https://aws.amazon.com/linux/amazon-linux-2023/ Last login: Sat Feb 22 22:38:31 KST 2025 on pts/0 [root@operator-host ~]# git clone https://github.com/kubernetes-sigs/aws-efs-csi-driver.git /root/efs-csi Cloning into '/root/efs-csi'... remote: Enumerating objects: 30760, done. remote: Counting objects: 100% (5451/5451), done. remote: Compressing objects: 100% (1512/1512), done. remote: Total 30760 (delta 4230), reused 4145 (delta 3902), pack-reused 25309 (from 3) Receiving objects: 100% (30760/30760), 28.56 MiB | 11.57 MiB/s, done. Resolving deltas: 100% (16621/16621), done. Updating files: 100% (7926/7926), done. [root@operator-host ~]# cd /root/efs-csi/examples/kubernetes/multiple_pods/specs && tree . ├── claim.yaml ├── pod1.yaml ├── pod2.yaml ├── pv.yaml └── storageclass.yaml 0 directories, 5 files # EFS 스토리지클래스 생성 및 확인 [root@operator-host specs]# cat storageclass.yaml kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: efs-sc provisioner: efs.csi.aws.com [root@operator-host specs]# kubectl apply -f storageclass.yaml storageclass.storage.k8s.io/efs-sc created [root@operator-host specs]# kubectl get sc efs-sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE efs-sc efs.csi.aws.com Delete Immediate false 8s # PV 생성 및 확인 : volumeHandle을 자신의 EFS 파일시스템ID로 변경 [root@operator-host specs]# EfsFsId=$(aws efs describe-file-systems --query "FileSystems[*].FileSystemId" --output text) [root@operator-host specs]# echo $EfsFsId fs-04bdbcbb996487caf [root@operator-host specs]# sed -i "s/fs-4af69aab/$EfsFsId/g" pv.yaml [root@operator-host specs]# cat pv.yaml apiVersion: v1 kind: PersistentVolume metadata: name: efs-pv spec: capacity: storage: 5Gi volumeMode: Filesystem accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Retain storageClassName: efs-sc csi: driver: efs.csi.aws.com volumeHandle: fs-04bdbcbb996487caf [root@operator-host specs]# kubectl apply -f pv.yaml persistentvolume/efs-pv created [root@operator-host specs]# kubectl get pv; kubectl describe pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE efs-pv 5Gi RWX Retain Available efs-sc <unset> 7s Name: efs-pv Labels: <none> Annotations: <none> Finalizers: [kubernetes.io/pv-protection] StorageClass: efs-sc Status: Available Claim: Reclaim Policy: Retain Access Modes: RWX VolumeMode: Filesystem Capacity: 5Gi Node Affinity: <none> Message: Source: Type: CSI (a Container Storage Interface (CSI) volume source) Driver: efs.csi.aws.com FSType: VolumeHandle: fs-04bdbcbb996487caf ReadOnly: false VolumeAttributes: <none> Events: <none> # PVC 생성 및 확인 [root@operator-host specs]# cat claim.yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: efs-claim spec: accessModes: - ReadWriteMany storageClassName: efs-sc resources: requests: storage: 5Gi [root@operator-host specs]# kubectl apply -f claim.yaml persistentvolumeclaim/efs-claim created [root@operator-host specs]# kubectl get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE efs-claim Bound efs-pv 5Gi RWX efs-sc <unset> 8s # 파드 생성 및 연동 : 파드 내에 /data 데이터는 EFS를 사용 # 추후에 파드1,2가 각기 다른 노드에 배포되게 추가해두자! [root@operator-host specs]# cat pod1.yaml pod2.yaml apiVersion: v1 kind: Pod metadata: name: app1 spec: containers: - name: app1 image: busybox command: ["/bin/sh"] args: ["-c", "while true; do echo $(date -u) >> /data/out1.txt; sleep 5; done"] volumeMounts: - name: persistent-storage mountPath: /data volumes: - name: persistent-storage persistentVolumeClaim: claimName: efs-claim apiVersion: v1 kind: Pod metadata: name: app2 spec: containers: - name: app2 image: busybox command: ["/bin/sh"] args: ["-c", "while true; do echo $(date -u) >> /data/out2.txt; sleep 5; done"] volumeMounts: - name: persistent-storage mountPath: /data volumes: - name: persistent-storage persistentVolumeClaim: claimName: efs-claim [root@operator-host specs]# kubectl apply -f pod1.yaml,pod2.yaml pod/app1 created pod/app2 created # 파드 정보 확인 : PV에 5Gi 와 파드 내에서 확인한 NFS4 볼륨 크리 8.0E의 차이는 무엇? [root@operator-host specs]# kubectl get pods NAME READY STATUS RESTARTS AGE app1 1/1 Running 0 25s app2 1/1 Running 0 25s [root@operator-host specs]# kubectl exec -ti app1 -- sh -c "df -hT -t nfs4" Filesystem Type Size Used Available Use% Mounted on 127.0.0.1:/ nfs4 8.0E 0 8.0E 0% /data [root@operator-host specs]# kubectl exec -ti app2 -- sh -c "df -hT -t nfs4" Filesystem Type Size Used Available Use% Mounted on 127.0.0.1:/ nfs4 8.0E 0 8.0E 0% /data # 공유 저장소 저장 동작 확인 [root@operator-host specs]# tree /mnt/myefs /mnt/myefs ├── memo.txt ├── out1.txt └── out2.txt 0 directories, 3 files [root@operator-host specs]# tail -f /mnt/myefs/out1.txt Sat Feb 22 17:00:57 UTC 2025 Sat Feb 22 17:01:02 UTC 2025 Sat Feb 22 17:01:07 UTC 2025 Sat Feb 22 17:01:12 UTC 2025 Sat Feb 22 17:01:17 UTC 2025 Sat Feb 22 17:01:22 UTC 2025 Sat Feb 22 17:01:27 UTC 2025 Sat Feb 22 17:01:32 UTC 2025 Sat Feb 22 17:01:37 UTC 2025 Sat Feb 22 17:01:42 UTC 2025 Sat Feb 22 17:01:47 UTC 2025 Sat Feb 22 17:01:52 UTC 2025 ^C [root@operator-host specs]# tail -f /mnt/myefs/out2.txt Sat Feb 22 17:01:12 UTC 2025 Sat Feb 22 17:01:17 UTC 2025 Sat Feb 22 17:01:22 UTC 2025 Sat Feb 22 17:01:27 UTC 2025 Sat Feb 22 17:01:32 UTC 2025 Sat Feb 22 17:01:37 UTC 2025 Sat Feb 22 17:01:42 UTC 2025 Sat Feb 22 17:01:47 UTC 2025 Sat Feb 22 17:01:52 UTC 2025 Sat Feb 22 17:01:57 UTC 2025 ^C [root@operator-host specs]# kubectl exec -ti app1 -- tail -f /data/out1.txt Sat Feb 22 17:01:22 UTC 2025 Sat Feb 22 17:01:27 UTC 2025 Sat Feb 22 17:01:32 UTC 2025 Sat Feb 22 17:01:37 UTC 2025 Sat Feb 22 17:01:42 UTC 2025 Sat Feb 22 17:01:47 UTC 2025 Sat Feb 22 17:01:52 UTC 2025 Sat Feb 22 17:01:57 UTC 2025 Sat Feb 22 17:02:02 UTC 2025 Sat Feb 22 17:02:07 UTC 2025 ^Ccommand terminated with exit code 130 [root@operator-host specs]# kubectl exec -ti app2 -- tail -f /data/out2.txt Sat Feb 22 17:01:32 UTC 2025 Sat Feb 22 17:01:37 UTC 2025 Sat Feb 22 17:01:42 UTC 2025 Sat Feb 22 17:01:47 UTC 2025 Sat Feb 22 17:01:52 UTC 2025 Sat Feb 22 17:01:57 UTC 2025 Sat Feb 22 17:02:02 UTC 2025 Sat Feb 22 17:02:07 UTC 2025 Sat Feb 22 17:02:12 UTC 2025 Sat Feb 22 17:02:17 UTC 2025 ^Ccommand terminated with exit code 130 |
- 실습 완료 후 삭제
| # 쿠버네티스 리소스 삭제 [root@operator-host specs]# kubectl delete pod app1 app2 pod "app1" deleted pod "app2" deleted [root@operator-host specs]# kubectl delete pvc efs-claim persistentvolumeclaim "efs-claim" deleted [root@operator-host specs]# kubectl delete pv efs-pv persistentvolume "efs-pv" deleted [root@operator-host specs]# kubectl delete sc efs-sc storageclass.storage.k8s.io "efs-sc" deleted |
- EFS 파일시스템을 다수의 파드가 사용하게 설정 : Dynamic provisioning using EFS
| # 모니터링 [yu3papa@iworks ~]$ watch 'kubectl get sc efs-sc; echo; kubectl get pv,pvc,pod'^C # [운영 서버 EC2] [yu3papa@iworks ~]$ ssh -i aews-yu3papa.pem ec2-user@15.164.244.197 Last login: Sat Feb 22 22:38:31 2025 from 221.149.127.57 , #_ ~\_ ####_ Amazon Linux 2 ~~ \_#####\ ~~ \###| AL2 End of Life is 2026-06-30. ~~ \#/ ___ ~~ V~' '-> ~~~ / A newer version of Amazon Linux is available! ~~._. _/ _/ _/ Amazon Linux 2023, GA and supported until 2028-03-15. _/m/' https://aws.amazon.com/linux/amazon-linux-2023/ Last login: Sun Feb 23 01:57:44 KST 2025 on pts/1 # EFS 스토리지클래스 생성 및 확인 [root@operator-host ~]# curl -s -O https://raw.githubusercontent.com/kubernetes-sigs/aws-efs-csi-driver/master/examples/kubernetes/dynamic_provisioning/specs/storageclass.yaml [root@operator-host ~]# cat storageclass.yaml kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: efs-sc provisioner: efs.csi.aws.com parameters: provisioningMode: efs-ap fileSystemId: fs-92107410 directoryPerms: "700" gidRangeStart: "1000" # optional gidRangeEnd: "2000" # optional basePath: "/dynamic_provisioning" # optional subPathPattern: "${.PVC.namespace}/${.PVC.name}" # optional ensureUniqueDirectory: "true" # optional reuseAccessPoint: "false" # optional[root@operator-host ~]# [root@operator-host ~]# EfsFsId=$(aws efs describe-file-systems --query "FileSystems[*].FileSystemId" --output text) [root@operator-host ~]# sed -i "s/fs-92107410/$EfsFsId/g" storageclass.yaml [root@operator-host ~]# kubectl apply -f storageclass.yaml storageclass.storage.k8s.io/efs-sc created [root@operator-host ~]# kubectl get sc efs-sc NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE efs-sc efs.csi.aws.com Delete Immediate false 12s # PVC/파드 생성 및 확인 [root@operator-host ~]# curl -s -O https://raw.githubusercontent.com/kubernetes-sigs/aws-efs-csi-driver/master/examples/kubernetes/dynamic_provisioning/specs/pod.yaml [root@operator-host ~]# cat pod.yaml --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: efs-claim spec: accessModes: - ReadWriteMany storageClassName: efs-sc resources: requests: storage: 5Gi --- apiVersion: v1 kind: Pod metadata: name: efs-app spec: containers: - name: app image: centos command: ["/bin/sh"] args: ["-c", "while true; do echo $(date -u) >> /data/out; sleep 5; done"] volumeMounts: - name: persistent-storage mountPath: /data volumes: - name: persistent-storage persistentVolumeClaim: claimName: efs-claim[root@operator-host ~]# [root@operator-host ~]# kubectl apply -f pod.yaml persistentvolumeclaim/efs-claim created pod/efs-app created [root@operator-host ~]# kubectl get pvc,pv,pod NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE persistentvolumeclaim/efs-claim Bound pvc-9899279b-2764-4935-8ee6-8b1985c61678 5Gi RWX efs-sc <unset> 17s NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS VOLUMEATTRIBUTESCLASS REASON AGE persistentvolume/pvc-9899279b-2764-4935-8ee6-8b1985c61678 5Gi RWX Delete Bound default/efs-claim efs-sc <unset> 16s NAME READY STATUS RESTARTS AGE pod/efs-app 1/1 Running 0 17s # PVC/PV 생성 로그 확인 [root@operator-host ~]# kubectl logs -n kube-system -l app=efs-csi-controller -c csi-provisioner -f I0222 16:48:29.363008 1 reflector.go:368] Caches populated for *v1.PersistentVolumeClaim from k8s.io/client-go@v0.31.0/tools/cache/reflector.go:243 I0222 16:48:29.363282 1 reflector.go:368] Caches populated for *v1.StorageClass from k8s.io/client-go@v0.31.0/tools/cache/reflector.go:243 I0222 16:48:29.453866 1 controller.go:824] "Starting provisioner controller" component="efs.csi.aws.com_efs-csi-controller-64fc4bc65d-84bq4_9955ae9f-a6f6-47a2-8915-49480f88ebb3" I0222 16:48:29.453911 1 volume_store.go:98] "Starting save volume queue" I0222 16:48:29.456593 1 reflector.go:368] Caches populated for *v1.StorageClass from k8s.io/client-go@v0.31.0/tools/cache/reflector.go:243 I0222 16:48:29.457321 1 reflector.go:368] Caches populated for *v1.PersistentVolume from k8s.io/client-go@v0.31.0/tools/cache/reflector.go:243 I0222 16:48:29.555566 1 controller.go:873] "Started provisioner controller" component="efs.csi.aws.com_efs-csi-controller-64fc4bc65d-84bq4_9955ae9f-a6f6-47a2-8915-49480f88ebb3" I0222 17:19:02.828324 1 event.go:389] "Event occurred" object="default/efs-claim" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="Provisioning" message="External provisioner is provisioning volume for claim \"default/efs-claim\"" I0222 17:19:03.122066 1 controller.go:955] successfully created PV pvc-9899279b-2764-4935-8ee6-8b1985c61678 for PVC efs-claim and csi volume name fs-04bdbcbb996487caf::fsap-0151b9bbfd7ee62b3 I0222 17:19:03.136203 1 event.go:389] "Event occurred" object="default/efs-claim" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Normal" reason="ProvisioningSucceeded" message="Successfully provisioned volume pvc-9899279b-2764-4935-8ee6-8b1985c61678" W0222 16:48:30.436187 1 feature_gate.go:354] Setting GA feature gate Topology=true. It will be removed in a future release. I0222 16:48:30.437080 1 feature_gate.go:387] feature gates: {map[Topology:true]} I0222 16:48:30.437479 1 csi-provisioner.go:154] Version: v5.1.0 I0222 16:48:30.437572 1 csi-provisioner.go:177] Building kube configs for running in cluster... I0222 16:48:30.441128 1 common.go:143] "Probing CSI driver for readiness" I0222 16:48:30.448649 1 csi-provisioner.go:230] Detected CSI driver efs.csi.aws.com I0222 16:48:30.452688 1 csi-provisioner.go:302] CSI driver does not support PUBLISH_UNPUBLISH_VOLUME, not watching VolumeAttachments I0222 16:48:30.453202 1 controller.go:744] "Using saving PVs to API server in background" I0222 16:48:30.453494 1 leaderelection.go:254] attempting to acquire leader lease kube-system/efs-csi-aws-com... ^C # 파드 정보 확인 [root@operator-host ~]# kubectl exec -it efs-app -- sh -c "df -hT -t nfs4" Filesystem Type Size Used Avail Use% Mounted on 127.0.0.1:/ nfs4 8.0E 0 8.0E 0% /data # 공유 저장소 저장 동작 확인 [root@operator-host ~]# tree /mnt/myefs /mnt/myefs ├── dynamic_provisioning │ └── default │ └── efs-claim-3afddc0f-c72b-4d23-ab27-8302c45c8713 │ └── out ├── memo.txt ├── out1.txt └── out2.txt 3 directories, 4 files [root@operator-host ~]# kubectl exec efs-app -- bash -c "cat /data/out" Sat Feb 22 17:19:13 UTC 2025 Sat Feb 22 17:19:18 UTC 2025 Sat Feb 22 17:19:23 UTC 2025 Sat Feb 22 17:19:28 UTC 2025 Sat Feb 22 17:19:33 UTC 2025 Sat Feb 22 17:19:38 UTC 2025 Sat Feb 22 17:19:43 UTC 2025 Sat Feb 22 17:19:48 UTC 2025 Sat Feb 22 17:19:53 UTC 2025 Sat Feb 22 17:19:58 UTC 2025 Sat Feb 22 17:20:03 UTC 2025 [root@operator-host ~]# kubectl exec efs-app -- bash -c "ls -l /data/out" -rw-r--r--. 1 1000 1000 377 Feb 22 17:20 /data/out [root@operator-host ~]# kubectl exec efs-app -- bash -c "stat /data/" File: /data/ Size: 6144 Blocks: 8 IO Block: 1048576 directory Device: 100007h/1048583d Inode: 2689823224459023729 Links: 2 Access: (0700/drwx------) Uid: ( 1000/ UNKNOWN) Gid: ( 1000/ UNKNOWN) Access: 2025-02-22 17:19:13.598000000 +0000 Modify: 2025-02-22 17:19:13.598000000 +0000 Change: 2025-02-22 17:19:13.598000000 +0000 Birth: - |
- EFS → Access Point 확인
- EFS Access Point는 EFS의 특정 부분을 격리하고, UID/GID를 강제하여 보안성을 높임.
- 여러 팀, 여러 애플리케이션이 같은 EFS를 사용할 때 Access Point를 활용하면 보안과 관리가 용이.
- 실습 완료 후 삭제
| # 쿠버네티스 리소스 삭제 [root@operator-host ~]# kubectl delete -f pod.yaml persistentvolumeclaim "efs-claim" deleted pod "efs-app" deleted [root@operator-host ~]# kubectl delete -f storageclass.yaml storageclass.storage.k8s.io "efs-sc" deleted |
6. AWS 리소스 삭제
- Amazon EKS 클러스터 삭제(10분 정도 소요)
| [yu3papa@iworks ~]$ eksctl delete cluster --name $CLUSTER_NAME |
- (클러스터 삭제 완료 확인 후) AWS CloudFormation 스택 삭제
| [yu3papa@iworks ~]$ aws cloudformation delete-stack --stack-name myeks |
- EKS 배포 후 실습 편의를 위한 변수 설정 삭제
| [yu3papa@iworks ~]$ vi ~/.bashrc |
'Kubernetes' 카테고리의 다른 글
| AEWS [3기] 7주차 - EKS Mode - Fargate (0) | 2025.03.22 |
|---|---|
| AEWS [3기] 6주차 - K8S API Server Client Certificate 인증 (0) | 2025.03.15 |
| AEWS [3기] 5주차 - EKS Autoscaling (0) | 2025.03.08 |
| AEWS [3기] 4주차 - OpenTelemetry (0) | 2025.03.01 |
| AEWS [3기] 2주차 - EKS Networking (iptables 와 NetFilter) (1) | 2025.02.13 |