AEWS [3기] 5주차 - EKS Autoscaling

Kubernetes

AEWS [3기] 5주차 - EKS Autoscaling

yu3papa 2025. 3. 8. 10:37

0. AutoScailing 개요

쿠버네티스에서 운영중인 시스템에 대해 리소스가 부족해지면 정상운영이 힘들어 집니다.

쿠버네티스는 리소스가 부족할 경우 다양한 방법으로 AutoScailing을 지원합니다.

AWS 장표를 보면 아래와 같습니다.

위의 HPA, VPA 는 POD 단위입니다.

POD를 이용하여 AutoScail이 안되어지는 상황이 오면 워커노드를 스케일 아웃해야 합니다.

CAS는 워커노드를 자동으로 Scale In/Out 이 가능합니다.

하지만 기능이 추가된 Karpenter (고성능 노드 오토스케일러)를 실무에서는 많이 사용합니다.

AutoScaling을 이해하기 전에 POD안의 Container가 CPU, Memory 리소스를 어떻게 사용하는지가 중요합니다.

requests : 쿠버네티스 스케쥴러가 POD를 어떤 워커노드에 스케쥴링할때 기준값
limits : 워커노드에 스케쥴링된 후에 해당 워커노드의 자원 사용을 제한 --> Control Group이 동작

클러스터에 자원이 부족해지게 되면 POD에서 설정된 request, limit 값을 확인하여 Qos 클래스로 구분한 후 BestEffort 로 분류된 POD를 제거해 버립니다.

이러한 문제를 해결하기 위해 AutoScailing은 클러스터를 운영하는데 필수적으로 적용되어야 합니다.

수평 확장 Horizontal Scaling
- 운영 환경에 더 많은 워크로드(VM, Task, Pod)를 추가하는 방식
- 트래픽을 여러 워크로드에 분산시키는 방법
- 확장성이 높고 유연한 방법
- 비용 효율적
- Stateless 워크로드에 적합 (데이터 일관성 유지 필요)
수직 확장 Vertical Scaling
- 운영 환경의 기존 워크로드의 성능(CPU, Memory)을 향상하는 방법
- 하드웨어는 확장에 한계가 존재함
- 확장 과정에서의 장애 위험이 존재함
- 수평적 확장과 비교하면 유연성이 부족함
AWS Auto Scaling 정책
- Simple/Step scaling : Manual Reactive, Dynamic scaling
  - 고객이 정의한 단계에 따라 메트릭을 모니터링하고 인스턴스를 추가 또는 제거합니다.
- Target tracking : Automated Reactive, Dynamic scaling
  - 고객이 정의한 목표 메트릭을 유지하기 위해 자동으로 인스턴스를 추가 또는 제거합니다.
- Scheduled scaling : Manual Proactive
  - 고객이 정의한 일정에 따라 인스턴스를 시작하거나 종료합니다.
- Predictive scaling : Automated Proactive
  - 과거 트렌드를 기반으로 용량을 선제적으로 시작합니다.
K8S Auto Scaling 정책
- 확장 방법 : 컨테이너(파드) vs 노드(서버)
  - 컨테이너 수평적 확장
  - 컨테이너 수직적 확장
  - 노드 수평적 확장
  - 노드 수직적 확장
- 확장 기준
  - 컨테이너 메트릭 기반
  - 애플리케이션 메트릭 기반
  - 이벤트(일정, 대기열 등) 기반
- 확장 정책
  - 단순 확장 정책
  - 단계 확장 정책
  - 목표 추적 확장 정책
EKS Auto Scaling - Link
- HPA : 서비스를 처리할 파드 자원이 부족한 경우 신규 파드 Provisioning → 파드 Scale Out
- VPA : 서비스를 처리할 파드 자원이 부족한 경우 파드 교체(자동 or 수동) → 파드 Scale Up
- CAS : 파드를 배포할 노드가 부족한 경우 신규 노드 Provisioning → 노드 Scale Out
- Karpenter : Unscheduled 파드가 있는 경우 새로운 노드 및 파드 Provisioning → 노드 Scale Up/Out

AWS Auto Scaling 한계 → 관리 복잡성 증가, 최적화 설정 어려움, 비용 관리 어려움

https://www.youtube.com/watch?v=jLuVZX6WQsw

오토스케일링 전략은 EC2의 오토스케일링 그룹 사용을 중심으로 동작합니다
노드 그룹에서 인스턴스 타입이 동일하다고 가정합니다.
혼합 인스턴스 타입은 가능한 CPU와 메모리가 균등하게 되어야 합니다.
다양한 인스턴스 타입을 지원하기 위해서는 여러 노드 그룹이 필요합니다.
모범사례는 AZ당 노드 그룹을 가지는 것입니다.

카펜터

유연한 노드 프로비저닝, 빠른 스케일링 및 비용 최적화, 똑똑한 통합 관리 및 간소화된 설정, 향상된 리소스 활용 및 확장성

EC2와 긴밀히 통합 : EC2 Fleet API, ASGs 가 없음
쿠버네티스 네이티브 : Watch API, Labels, Finalizers
자동화된 인스턴스 선택 : 워크로드 요구사항을 인스턴스 타입과 일치
기본 동작 흐름 : 카펜터가 ‘파드 unschedulable’ 감지 후 Pod 스펙 평가 후 ‘EC2 Fleet’를 통해 노드 프로비저닝
Over Provisioning : 스파이크 트래픽 대비하기 - 여유 노드를 항상 유지하기, 더미파드 사용(낮은 우선순위 적용)
Custom Metric 활용하기 : Resource Metric API, Custom Metric API, External Metric API
- Resource Metric API : Node 또는 Pod의 CPU, 메모리 사용량 등의 metric 기반으로한 파드 스케일링, metrics.k8s.io
- Custom Metric API : 사용자가 정의한 클러스터 내부의 metric을 기반으로 파드 스케일링, custom.metrics.k8s.io
- External Metric API : 클러스터 외부에서 수집된 metric 을 기반으로 파드 스케일링, external.metrics.k8s.io

KEDA 활용하기 : 다양한 Event로 Scaling 수행
- 이벤트 기반 파드 오토스케일링 (HPA 이용)
- 트리거 : cron, ..
- ScaledObject 오브젝트에 명세 정의
- 카펜터와 통합되어 클러스터 수준 오토스케일링 지원 : 예) Cron으로 특정 시간에 노드 증설

1. 실습을 위한 EKS 클러스터 구성

AutoScaling를 실습을 위한 EKS 클러스터를 CloudFormation을 이용하여 구성하겠습니다.

myeks-5week.yaml

0.02MB

# 변수 지정

CLUSTER_NAME=myeks

SSHKEYNAME=<SSH 키 페이 이름>

MYACCESSKEY=<IAM Uesr 액세스 키>

MYSECRETKEY=<IAM Uesr 시크릿 키>

# CloudFormation 스택 배포

[yu3papa@iworks ~]$ aws cloudformation deploy --template-file myeks-5week.yaml --stack-name $CLUSTER_NAME --parameter-overrides KeyName=$SSHKEYNAME SgIngressSshCidr=$(curl -s ipinfo.io/ip)/32 MyIamUserAccessKeyID=$MYACCESSKEY MyIamUserSecretAccessKey=$MYSECRETKEY ClusterBaseName=$CLUSTER_NAME --region ap-northeast-2

Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - myeks

# CloudFormation 스택 배포 완료 후 작업용 EC2 IP 출력

[yu3papa@iworks ~]$ aws cloudformation describe-stacks --stack-name myeks --query 'Stacks[*].Outputs[0].OutputValue' --output text

13.124.218.189

운영서버 EC2-IP와 SSH Private-Key를 이용하여 SSH 접속을 합니다.

kubeconfig 작업을 수행하여 kubectl 명령을 수행할 수 있도록 설정합니다.

# 변수 지정

CLUSTER_NAME=myeks

SSHKEYNAME=yu3papakey

eksctl get cluster

# kubeconfig 생성

aws sts get-caller-identity --query Arn

# aws eks update-kubeconfig --name myeks --user-alias <위 출력된 자격증명 사용자>

aws eks update-kubeconfig --name myeks --user-alias admin

kubectl ns default

kubectl get node --label-columns=node.kubernetes.io/instance-type,eks.amazonaws.com/capacityType,topology.kubernetes.io/zone

kubectl get pod -A

kubectl get pdb -n kube-system

운영서버 EC2에서 EKS 설치확인

# default 네임스페이스 적용

kubectl ns default

# 환경변수 정보 확인

# krew 플러그인 확인

kubectl krew list

# 인스턴스 정보 확인

aws ec2 describe-instances --query "Reservations[*].Instances[*].{InstanceID:InstanceId, PublicIPAdd:PublicIpAddress, PrivateIPAdd:PrivateIpAddress, InstanceName:Tags[?Key=='Name']|[0].Value, Status:State.Name}" --filters Name=instance-state-name,Values=running --output table

# 노드 IP 확인 및 PrivateIP 변수 지정

aws ec2 describe-instances --query "Reservations[*].Instances[*].{PublicIPAdd:PublicIpAddress,PrivateIPAdd:PrivateIpAddress,InstanceName:Tags[?Key=='Name']|[0].Value,Status:State.Name}" --filters Name=instance-state-name,Values=running --output table

N1=$(kubectl get node --label-columns=topology.kubernetes.io/zone --selector=topology.kubernetes.io/zone=ap-northeast-2a -o jsonpath={.items[0].status.addresses[0].address})

N2=$(kubectl get node --label-columns=topology.kubernetes.io/zone --selector=topology.kubernetes.io/zone=ap-northeast-2b -o jsonpath={.items[0].status.addresses[0].address})

N3=$(kubectl get node --label-columns=topology.kubernetes.io/zone --selector=topology.kubernetes.io/zone=ap-northeast-2c -o jsonpath={.items[0].status.addresses[0].address})

echo "export N1=$N1" >> /etc/profile

echo "export N2=$N2" >> /etc/profile

echo "export N3=$N3" >> /etc/profile

echo $N1, $N2, $N3

# 노드 IP 로 ping 테스트

for i in $N1 $N2 $N3; do echo ">> node $i <<"; ping -c 1 $i ; echo; done

EKS 배포후 실습편의를 위해 각종 정보를 환경변수 설정파일에 등록하겠습니다.

# 변수 지정

export CLUSTER_NAME=myeks

export VPCID=$(aws ec2 describe-vpcs --filters "Name=tag:Name,Values=$CLUSTER_NAME-VPC" --query 'Vpcs[*].VpcId' --output text)

export PubSubnet1=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet1" --query "Subnets[0].[SubnetId]" --output text)

export PubSubnet2=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet2" --query "Subnets[0].[SubnetId]" --output text)

export PubSubnet3=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet3" --query "Subnets[0].[SubnetId]" --output text)

export N1=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=$CLUSTER_NAME-ng1-Node" "Name=availability-zone,Values=ap-northeast-2a" --query 'Reservations[*].Instances[*].PublicIpAddress' --output text)

export N2=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=$CLUSTER_NAME-ng1-Node" "Name=availability-zone,Values=ap-northeast-2b" --query 'Reservations[*].Instances[*].PublicIpAddress' --output text)

export N3=$(aws ec2 describe-instances --filters "Name=tag:Name,Values=$CLUSTER_NAME-ng1-Node" "Name=availability-zone,Values=ap-northeast-2c" --query 'Reservations[*].Instances[*].PublicIpAddress' --output text)

export CERT_ARN=$(aws acm list-certificates --query 'CertificateSummaryList[].CertificateArn[]' --output text) #사용 리전의 인증서 ARN 확인

MyDomain=yu3papa.shop # 각자 자신의 도메인 이름 입력

MyDnzHostedZoneId=$(aws route53 list-hosted-zones-by-name --dns-name "$MyDomain." --query "HostedZones[0].Id" --output text)

# 실습 완료 후 삭제 할 것!

cat << EOF >> ~/.bash_profile

# eksworkshop

export CLUSTER_NAME=myeks

export VPCID=$(aws ec2 describe-vpcs --filters "Name=tag:Name,Values=$CLUSTER_NAME-VPC" --query 'Vpcs[*].VpcId' --output text)

export PubSubnet1=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet1" --query "Subnets[0].[SubnetId]" --output text)

export PubSubnet2=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet2" --query "Subnets[0].[SubnetId]" --output text)

export PubSubnet3=$(aws ec2 describe-subnets --filters Name=tag:Name,Values="$CLUSTER_NAME-Vpc1PublicSubnet3" --query "Subnets[0].[SubnetId]" --output text)

export CERT_ARN=$(aws acm list-certificates --query 'CertificateSummaryList[].CertificateArn[]' --output text)

MyDomain=yu3papa.shop # 각자 자신의 도메인 이름 입력

MyDnzHostedZoneId=$(aws route53 list-hosted-zones-by-name --dns-name "$MyDomain." --query "HostedZones[0].Id" --output text)

EOF

# [신규 터미널] 확인

echo $CLUSTER_NAME $VPCID $PubSubnet1 $PubSubnet2 $PubSubnet3

echo $N1 $N2 $N3 $MyDomain $MyDnzHostedZoneId

tail -n 15 ~/.bash_profile

AWS LoadBalancer Controller, ExternalDNS, gp3 storageclass, kube-ops-view(Ingress) 설치합니다.

# AWS LoadBalancerController

helm repo add eks https://aws.github.io/eks-charts

helm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=$CLUSTER_NAME \

--set serviceAccount.create=false --set serviceAccount.name=aws-load-balancer-controller

# ExternalDNS

echo $MyDomain

curl -s https://raw.githubusercontent.com/gasida/PKOS/main/aews/externaldns.yaml | MyDomain=$MyDomain MyDnzHostedZoneId=$MyDnzHostedZoneId envsubst | kubectl apply -f -

# gp3 스토리지 클래스 생성

cat <<EOF | kubectl apply -f -

kind: StorageClass

apiVersion: storage.k8s.io/v1

metadata:

annotations:

storageclass.kubernetes.io/is-default-class: "true"

allowVolumeExpansion: true

provisioner: ebs.csi.aws.com

volumeBindingMode: WaitForFirstConsumer

parameters:

type: gp3

allowAutoIOPSPerGBIncrease: 'true'

encrypted: 'true'

fsType: xfs # 기본값이 ext4

EOF

kubectl get sc

# kube-ops-view

helm repo add geek-cookbook https://geek-cookbook.github.io/charts/

helm install kube-ops-view geek-cookbook/kube-ops-view --version 1.2.2 --set service.main.type=ClusterIP --set env.TZ="Asia/Seoul" --namespace kube-system

# kubeopsview 용 Ingress 설정 : group 설정으로 1대의 ALB를 여러개의 ingress 에서 공용 사용

echo $CERT_ARN

cat <<EOF | kubectl apply -f -

apiVersion: networking.k8s.io/v1

kind: Ingress

metadata:

annotations:

alb.ingress.kubernetes.io/certificate-arn: $CERT_ARN

alb.ingress.kubernetes.io/group.name: study

alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'

alb.ingress.kubernetes.io/load-balancer-name: $CLUSTER_NAME-ingress-alb

alb.ingress.kubernetes.io/scheme: internet-facing

alb.ingress.kubernetes.io/ssl-redirect: "443"

alb.ingress.kubernetes.io/success-codes: 200-399

alb.ingress.kubernetes.io/target-type: ip

labels:

app.kubernetes.io/name: kubeopsview

namespace: kube-system

spec:

ingressClassName: alb

rules:

- host: kubeopsview.$MyDomain

http:

paths:

- backend:

service:

port:

number: 8080 # name: http

path: /

pathType: Prefix

EOF

잘 설치되었는지 확인합니다.

# 설치된 파드 정보 확인
[root@operator-host ~]# kubectl get pods -n kube-system
NAME                                          READY   STATUS    RESTARTS   AGE
aws-load-balancer-controller-554fbd9d-tt6hc   1/1     Running   0          6m28s
aws-load-balancer-controller-554fbd9d-vqpls   1/1     Running   0          6m28s
aws-node-9m9jq                                2/2     Running   0          20m
aws-node-dzj26                                2/2     Running   0          20m
aws-node-glzc4                                2/2     Running   0          20m
coredns-86f5954566-q2dqk                      1/1     Running   0          26m
coredns-86f5954566-v7twp                      1/1     Running   0          26m
ebs-csi-controller-549bf6879f-dvg9x           6/6     Running   0          17m
ebs-csi-controller-549bf6879f-g6zr9           6/6     Running   0          17m
ebs-csi-node-jvzmz                            3/3     Running   0          17m
ebs-csi-node-wb7mn                            3/3     Running   0          17m
ebs-csi-node-zh9fz                            3/3     Running   0          17m
external-dns-57c44fb644-8lr98                 1/1     Running   0          5m34s
kube-ops-view-657dbc6cd8-9zpwt                1/1     Running   0          5m5s
kube-proxy-5lhtp                              1/1     Running   0          20m
kube-proxy-ltfvh                              1/1     Running   0          20m
kube-proxy-x5c4c                              1/1     Running   0          20m
metrics-server-6bf5998d9c-5rnzh               1/1     Running   0          26m
metrics-server-6bf5998d9c-rvxwz               1/1     Running   0          26m

# service, ep, ingress 확인
[root@operator-host ~]# kubectl get ingress,svc,ep -n kube-system
NAME                                    CLASS   HOSTS                      ADDRESS                                                         PORTS   AGE
ingress.networking.k8s.io/kubeopsview   alb     kubeopsview.yu3papa.shop   myeks-ingress-alb-1265817985.ap-northeast-2.elb.amazonaws.com   80      5m26s

NAME                                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                  AGE
service/aws-load-balancer-webhook-service   ClusterIP   10.100.170.40    <none>        443/TCP                  7m15s
service/eks-extension-metrics-api           ClusterIP   10.100.128.212   <none>        443/TCP                  31m
service/kube-dns                            ClusterIP   10.100.0.10      <none>        53/UDP,53/TCP,9153/TCP   27m
service/kube-ops-view                       ClusterIP   10.100.132.200   <none>        8080/TCP                 5m52s
service/metrics-server                      ClusterIP   10.100.53.163    <none>        443/TCP                  27m

NAME                                          ENDPOINTS                                                       AGE
endpoints/aws-load-balancer-webhook-service   192.168.2.33:9443,192.168.3.79:9443                             7m15s
endpoints/eks-extension-metrics-api           172.0.32.0:10443                                                31m
endpoints/kube-dns                            192.168.1.228:53,192.168.1.96:53,192.168.1.228:53 + 3 more...   27m
endpoints/kube-ops-view                       192.168.2.150:8080                                              5m52s
endpoints/metrics-server                      192.168.1.28:10251,192.168.1.51:10251                           27m

# Kube Ops View 접속 정보 확인 : 조금 오래 기다리면 접속됨...
[root@operator-host ~]# echo -e "Kube Ops View URL = https://kubeopsview.$MyDomain/#scale=1.5"
Kube Ops View URL = https://kubeopsview.yu3papa.shop/#scale=1.5

프로메테우스 & 그라파나(admin / prom-operator) 설치

대시보드 Import 17900

# repo 추가

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

# 파라미터 파일 생성 : PV/PVC(AWS EBS) 삭제에 불편하니, 4주차 실습과 다르게 PV/PVC 미사용

cat <<EOT > monitor-values.yaml

prometheus:

prometheusSpec:

scrapeInterval: "15s"

evaluationInterval: "15s"

podMonitorSelectorNilUsesHelmValues: false

serviceMonitorSelectorNilUsesHelmValues: false

retention: 5d

retentionSize: "10GiB"

# Enable vertical pod autoscaler support for prometheus-operator

verticalPodAutoscaler:

enabled: true

ingress:

enabled: true

ingressClassName: alb

hosts:

- prometheus.$MyDomain

paths:

- /*

annotations:

alb.ingress.kubernetes.io/scheme: internet-facing

alb.ingress.kubernetes.io/target-type: ip

alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'

alb.ingress.kubernetes.io/certificate-arn: $CERT_ARN

alb.ingress.kubernetes.io/success-codes: 200-399

alb.ingress.kubernetes.io/load-balancer-name: myeks-ingress-alb

alb.ingress.kubernetes.io/group.name: study

alb.ingress.kubernetes.io/ssl-redirect: '443'

grafana:

defaultDashboardsTimezone: Asia/Seoul

adminPassword: prom-operator

defaultDashboardsEnabled: false

ingress:

enabled: true

ingressClassName: alb

hosts:

- grafana.$MyDomain

paths:

- /*

annotations:

alb.ingress.kubernetes.io/scheme: internet-facing

alb.ingress.kubernetes.io/target-type: ip

alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]'

alb.ingress.kubernetes.io/certificate-arn: $CERT_ARN

alb.ingress.kubernetes.io/success-codes: 200-399

alb.ingress.kubernetes.io/load-balancer-name: myeks-ingress-alb

alb.ingress.kubernetes.io/group.name: study

alb.ingress.kubernetes.io/ssl-redirect: '443'

kube-state-metrics:

rbac:

extraRules:

- apiGroups: ["autoscaling.k8s.io"]

resources: ["verticalpodautoscalers"]

verbs: ["list", "watch"]

customResourceState:

enabled: true

config:

kind: CustomResourceStateMetrics

spec:

resources:

- groupVersionKind:

group: autoscaling.k8s.io

kind: "VerticalPodAutoscaler"

version: "v1"

labelsFromPath:

verticalpodautoscaler: [metadata, name]

namespace: [metadata, namespace]

target_api_version: [apiVersion]

target_kind: [spec, targetRef, kind]

target_name: [spec, targetRef, name]

metrics:

- name: "vpa_containerrecommendations_target"

help: "VPA container recommendations for memory."

each:

type: Gauge

gauge:

path: [status, recommendation, containerRecommendations]

valueFrom: [target, memory]

labelsFromPath:

container: [containerName]

commonLabels:

resource: "memory"

unit: "byte"

- name: "vpa_containerrecommendations_target"

help: "VPA container recommendations for cpu."

each:

type: Gauge

gauge:

path: [status, recommendation, containerRecommendations]

valueFrom: [target, cpu]

labelsFromPath:

container: [containerName]

commonLabels:

resource: "cpu"

unit: "core"

selfMonitor:

enabled: true

alertmanager:

enabled: false

defaultRules:

create: false

kubeControllerManager:

enabled: false

kubeEtcd:

enabled: false

kubeScheduler:

enabled: false

prometheus-windows-exporter:

prometheus:

monitor:

enabled: false

EOT

cat monitor-values.yaml

# helm 배포

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack --version 69.3.1 \

-f monitor-values.yaml --create-namespace --namespace monitoring

# helm 확인

helm get values -n monitoring kube-prometheus-stack

# PV 사용하지 않음

[root@operator-host ~]# kubectl get pv,pvc -A
No resources found

[root@operator-host ~]# kubectl df-pv
INFO[2025-03-08T17:15:31+09:00] Either no volumes found in namespace/s: 'all' or the storage provisioner used for the volumes does not publish metrics to kubelet

# 프로메테우스 웹 접속

echo -e "https://prometheus.$MyDomain"

# 그라파나 웹 접속 : admin / prom-operator

echo -e "https://grafana.$MyDomain"

[root@operator-host ~]# kubectl get targetgroupbindings.elbv2.k8s.aws -A
NAMESPACE     NAME                               SERVICE-NAME                       SERVICE-PORT   TARGET-TYPE   AGE
kube-system   k8s-kubesyst-kubeopsv-f59a7f60c9   kube-ops-view                      8080           ip            17m
monitoring    k8s-monitori-kubeprom-5c3d6ee03d   kube-prometheus-stack-grafana      80             ip            9m11s
monitoring    k8s-monitori-kubeprom-ad3e3e2518   kube-prometheus-stack-prometheus   9090           ip            9m9s

# 상세 확인

[root@operator-host ~]# kubectl get pod -n monitoring -l app.kubernetes.io/name=kube-state-metrics
NAME READY STATUS RESTARTS AGE
kube-prometheus-stack-kube-state-metrics-5674c7ddd8-g8qwz 1/1 Running 0 9m50s

[root@operator-host ~]# kubectl describe pod -n monitoring -l app.kubernetes.io/name=kube-state-metrics
...

Service Account: kube-prometheus-stack-kube-state-metrics

...

Args:

--port=8080

--resources=certificatesigningrequests,configmaps,cronjobs,daemonsets,deployments,endpoints,horizontalpodautoscalers,ingresses,jobs,leases,limitranges,mutatingwebhookconfigurations,namespaces,networkpolicies,nodes,persistentvolumeclaims,persistentvolumes,poddisruptionbudgets,pods,replicasets,replicationcontrollers,resourcequotas,secrets,services,statefulsets,storageclasses,validatingwebhookconfigurations,volumeattachments

--custom-resource-state-config-file=/etc/customresourcestate/config.yaml

...

Volumes:

customresourcestate-config:

Type: ConfigMap (a volume populated by a ConfigMap)

Name: kube-prometheus-stack-kube-state-metrics-customresourcestate-config

Optional: false

...

[root@operator-host ~]# kubectl describe cm -n monitoring kube-prometheus-stack-kube-state-metrics-customresourcestate-config
Name:         kube-prometheus-stack-kube-state-metrics-customresourcestate-config
Namespace:    monitoring
Labels:       app.kubernetes.io/component=metrics
              app.kubernetes.io/instance=kube-prometheus-stack
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=kube-state-metrics
              app.kubernetes.io/part-of=kube-state-metrics
              app.kubernetes.io/version=2.14.0
              helm.sh/chart=kube-state-metrics-5.29.0
              release=kube-prometheus-stack
Annotations:  meta.helm.sh/release-name: kube-prometheus-stack
              meta.helm.sh/release-namespace: monitoring

Data
====
config.yaml:
----
kind: CustomResourceStateMetrics
spec:
  resources:
  - groupVersionKind:
      group: autoscaling.k8s.io
      kind: VerticalPodAutoscaler
      version: v1
    labelsFromPath:
      namespace:
      - metadata
      - namespace
      target_api_version:
      - apiVersion
      target_kind:
      - spec
      - targetRef
      - kind
      target_name:
      - spec
      - targetRef
      - name
      verticalpodautoscaler:
      - metadata
      - name
    metrics:
    - commonLabels:
        resource: memory
        unit: byte
      each:
        gauge:
          labelsFromPath:
            container:
            - containerName
          path:
          - status
          - recommendation
          - containerRecommendations
          valueFrom:
          - target
          - memory
        type: Gauge
      help: VPA container recommendations for memory.
      name: vpa_containerrecommendations_target
    - commonLabels:
        resource: cpu
        unit: core
      each:
        gauge:
          labelsFromPath:
            container:
            - containerName
          path:
          - status
          - recommendation
          - containerRecommendations
          valueFrom:
          - target
          - cpu
        type: Gauge
      help: VPA container recommendations for cpu.
      name: vpa_containerrecommendations_target

BinaryData
====

Events:  <none>

...

[root@operator-host ~]# kubectl get clusterrole kube-prometheus-stack-kube-state-metrics
NAME                                       CREATED AT
kube-prometheus-stack-kube-state-metrics   2025-03-08T08:08:16Z

[root@operator-host ~]# kubectl describe clusterrole kube-prometheus-stack-kube-state-metrics
Name:         kube-prometheus-stack-kube-state-metrics
Labels:       app.kubernetes.io/component=metrics
              app.kubernetes.io/instance=kube-prometheus-stack
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=kube-state-metrics
              app.kubernetes.io/part-of=kube-state-metrics
              app.kubernetes.io/version=2.14.0
              helm.sh/chart=kube-state-metrics-5.29.0
              release=kube-prometheus-stack
Annotations:  meta.helm.sh/release-name: kube-prometheus-stack
              meta.helm.sh/release-namespace: monitoring
PolicyRule:
  Resources                                                     Non-Resource URLs  Resource Names  Verbs
  ---------                                                     -----------------  --------------  -----
  configmaps                                                    []                 []              [list watch]
  endpoints                                                     []                 []              [list watch]
  limitranges                                                   []                 []              [list watch]
  namespaces                                                    []                 []              [list watch]
  nodes                                                         []                 []              [list watch]
  persistentvolumeclaims                                        []                 []              [list watch]
  persistentvolumes                                             []                 []              [list watch]
  pods                                                          []                 []              [list watch]
  replicationcontrollers                                        []                 []              [list watch]
  resourcequotas                                                []                 []              [list watch]
  secrets                                                       []                 []              [list watch]
  services                                                      []                 []              [list watch]
  mutatingwebhookconfigurations.admissionregistration.k8s.io    []                 []              [list watch]
  validatingwebhookconfigurations.admissionregistration.k8s.io  []                 []              [list watch]
  customresourcedefinitions.apiextensions.k8s.io                []                 []              [list watch]
  daemonsets.apps                                               []                 []              [list watch]
  deployments.apps                                              []                 []              [list watch]
  replicasets.apps                                              []                 []              [list watch]
  statefulsets.apps                                             []                 []              [list watch]
  verticalpodautoscalers.autoscaling.k8s.io                     []                 []              [list watch]
  horizontalpodautoscalers.autoscaling                          []                 []              [list watch]
  cronjobs.batch                                                []                 []              [list watch]
  jobs.batch                                                    []                 []              [list watch]
  certificatesigningrequests.certificates.k8s.io                []                 []              [list watch]
  leases.coordination.k8s.io                                    []                 []              [list watch]
  daemonsets.extensions                                         []                 []              [list watch]
  deployments.extensions                                        []                 []              [list watch]
  ingresses.extensions                                          []                 []              [list watch]
  replicasets.extensions                                        []                 []              [list watch]
  ingresses.networking.k8s.io                                   []                 []              [list watch]
  networkpolicies.networking.k8s.io                             []                 []              [list watch]
  poddisruptionbudgets.policy                                   []                 []              [list watch]
  storageclasses.storage.k8s.io                                 []                 []              [list watch]
  volumeattachments.storage.k8s.io                              []                 []              [list watch]

[root@operator-host ~]# kubectl describe clusterrole kube-prometheus-stack-kube-state-metrics | grep verticalpodautoscalers
  verticalpodautoscalers.autoscaling.k8s.io                     []                 []              [list watch]

EKS Node Viewer

노드 할당 가능 용량과 요청 request 리소스 표시, 실제 파드 리소스 사용량 X - 링크

동작
- It displays the scheduled pod resource requests vs the allocatable capacity on the node.
- It does not look at the actual pod resource usage.
- Node마다 할당 가능한 용량과 스케줄링된 POD(컨테이너)의 Resource 중 request 값을 표시한다.
- 실제 POD(컨테이너) 리소스 사용량은 아니다. /pkg/model/pod.go 파일을 보면 컨테이너의 request 합을 반환하며, init containers는 미포함
- https://github.com/awslabs/eks-node-viewer/blob/main/pkg/model/pod.go#L82

// Requested returns the sum of the resources requested by the pod. This doesn't include any init containers as we

// are interested in the steady state usage of the pod

func (p *Pod) Requested() v1.ResourceList {

p.mu.RLock()

defer p.mu.RUnlock()

requested := v1.ResourceList{}

for _, c := range p.pod.Spec.Containers {

for rn, q := range c.Resources.Requests {

existing := requested[rn]

existing.Add(q)

requested[rn] = existing

}

requested[v1.ResourcePods] = resource.MustParse("1")

return requested

}

설치

# 운영서버 EC2에 설치 : userdata 통해 이미 설치 되어 있음

yum install golang -y

go install github.com/awslabs/eks-node-viewer/cmd/eks-node-viewer@latest # 설치 시 2~3분 정도 소요

사용

# Standard usage
[root@operator-host ~]# eks-node-viewer

# Display both CPU and Memory Usage
[root@operator-host ~]# eks-node-viewer --resources cpu,memory

[root@operator-host ~]# eks-node-viewer --resources cpu,memory --extra-labels eks-node-viewer/node-age

# Display extra labels, i.e. AZ : node 에 labels 사용 가능
[root@operator-host ~]# eks-node-viewer --extra-labels topology.kubernetes.io/zone

[root@operator-host ~]# eks-node-viewer --extra-labels kubernetes.io/arch

# Sort by CPU usage in descending order
[root@operator-host ~]# eks-node-viewer --node-sort=eks-node-viewer/node-cpu-usage=dsc

# Karenter nodes only
[root@operator-host ~]# eks-node-viewer --node-selector "karpenter.sh/provisioner-name"

# Specify a particular AWS profile and region

AWS_PROFILE=myprofile AWS_REGION=us-west-2

Computed Labels : --extra-labels

# eks-node-viewer/node-age - Age of the node

eks-node-viewer --extra-labels eks-node-viewer/node-age

eks-node-viewer --extra-labels topology.kubernetes.io/zone,eks-node-viewer/node-age

# eks-node-viewer/node-ephemeral-storage-usage - Ephemeral Storage usage (requests)

eks-node-viewer --extra-labels eks-node-viewer/node-ephemeral-storage-usage

# eks-node-viewer/node-cpu-usage - CPU usage (requests)

eks-node-viewer --extra-labels eks-node-viewer/node-cpu-usage

# eks-node-viewer/node-memory-usage - Memory usage (requests)

eks-node-viewer --extra-labels eks-node-viewer/node-memory-usage

# eks-node-viewer/node-pods-usage - Pod usage (requests)

eks-node-viewer --extra-labels eks-node-viewer/node-pods-usage

HPA - Horizontal Pod Autoscaler

CPU 부하량을 기준으로 자동으로 POD 개수를 Scale In/Out 실습을 진행하겠습니다.
그라파나(22128 , 22251) 대시보드 Import

샘플 애플리케이션 배포
- (참고) hpa-example : Dockerfile , index.php (CPU 과부하 연산 수행 , 100만번 덧셈 수행)
- Dockerfile

FROM php:5-apache

COPY index.php /var/www/html/index.php

RUN chmod a+rx index.php

index.php

<?php

$x = 0.0001;

for ($i = 0; $i <= 1000000; $i++) {

$x += sqrt($x);

}

echo "OK!";

# Run and expose php-apache server
cat << EOF > php-apache.yaml

apiVersion: apps/v1

kind: Deployment

metadata:

spec:

selector:

matchLabels:

run: php-apache

template:

metadata:

labels:

run: php-apache

spec:

containers:

- name: php-apache

image: registry.k8s.io/hpa-example

ports:

- containerPort: 80

resources:

limits:

cpu: 500m

requests:

cpu: 200m

---

apiVersion: v1

kind: Service

metadata:

labels:

run: php-apache

spec:

ports:

- port: 80

selector:

run: php-apache

EOF
kubectl apply -f php-apache.yaml

# 확인
[root@operator-host ~]# kubectl exec -it deploy/php-apache -- cat /var/www/html/index.php
<?php
$x = 0.0001;
for ($i = 0; $i <= 1000000; $i++) {
$x += sqrt($x);
}
echo "OK!";
?>

# 모니터링 : 터미널2개 사용
[root@operator-host ~]# watch -d 'kubectl get hpa,pod;echo;kubectl top pod;echo;kubectl top node'

--> 현재 1개의 POD가 사용중입니다.

HPA 정책 생성 및 부하 발생 후 오토 스케일링 테스트 : 증가 시 기본 대기 시간(30초), 감소 시 기본 대기 시간(5분) → 조정 가능

# Create the HorizontalPodAutoscaler : requests.cpu=200m - 알고리즘
# Since each pod requests 200 milli-cores by kubectl run, this means an average CPU usage of 100 milli-cores.
cat <<EOF | kubectl apply -f -

apiVersion: autoscaling/v2

kind: HorizontalPodAutoscaler

metadata:

spec:

scaleTargetRef:

apiVersion: apps/v1

kind: Deployment

minReplicas: 1

maxReplicas: 10

metrics:

- type: Resource

resource:

target:

averageUtilization: 50

type: Utilization

EOF
혹은
kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10

# 확인
[root@operator-host ~]# kubectl get hpa
NAME         REFERENCE               TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   cpu: 0%/50%   1         10        1          82s

[root@operator-host ~]# kubectl describe hpa
Name:                                                  php-apache
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Sat, 08 Mar 2025 22:08:04 +0900
Reference:                                             Deployment/php-apache
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  0% (1m) / 50%
Min replicas:                                          1
Max replicas:                                          10
Deployment pods:                                       1 current / 1 desired
Conditions:
  Type            Status  Reason               Message
  ----            ------  ------               -------
  AbleToScale     True    ScaleDownStabilized  recent recommendations were higher than current one, applying the highest recent recommendation
  ScalingActive   True    ValidMetricFound     the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  False   DesiredWithinRange   the desired count is within the acceptable range
Events:           <none>

# HPA 리소스가 생성된 후에 그라파나에서 모니터링이 가능합니다.

# HPA 설정 확인
kubectl get hpa php-apache -o yaml | kubectl neat

apiVersion: autoscaling/v2

kind: HorizontalPodAutoscaler

spec:

minReplicas: 1 # [4] 또는 최소 1개까지 줄어들 수도 있습니다

maxReplicas: 10 # [3] 포드를 최대 10개까지 늘립니다

scaleTargetRef:

apiVersion: apps/v1

kind: Deployment

metrics:

- type: Resource

resource:

target:

type: Utilization

averageUtilization: 50 # [2] CPU 활용률이 50% 이상인 경우

# --> 이제 부하를 발생시키겠습니다.

# 반복 접속 1 (파드1 IP로 접속) >> 증가 확인 후 중지

while true;do curl -s $PODIP; sleep 0.5; done

# 반복 접속 2 (서비스명 도메인으로 파드들 분산 접속) >> 증가 확인(몇개까지 증가되는가? 그 이유는?) 후 중지

## >> [scale back down] 중지 5분 후 파드 갯수 감소 확인

# Run this in a separate terminal

# so that the load generation continues and you can carry on with the rest of the steps

kubectl run -i --tty load-generator --rm --image=busybox:1.28 --restart=Never -- /bin/sh -c "while sleep 0.01; do wget -q -O- http://php-apache; done"

# --> CPU 사용률이 증가하면서 POD 가 7개까지 Scale Out이 되었습니다.

# Horizontal Pod Autoscaler Status Conditions
kubectl describe hpa
...
Events:
  Type    Reason             Age    From                       Message
  ----    ------             ----   ----                       -------
  Normal  SuccessfulRescale  13m    horizontal-pod-autoscaler  New size: 2; reason: cpu resource utilization (percentage of request) above target
  Normal  SuccessfulRescale  11m    horizontal-pod-autoscaler  New size: 3; reason: cpu resource utilization (percentage of request) above target
  Normal  SuccessfulRescale  11m    horizontal-pod-autoscaler  New size: 6; reason: cpu resource utilization (percentage of request) above target
  Normal  SuccessfulRescale  10m    horizontal-pod-autoscaler  New size: 8; reason: cpu resource utilization (percentage of request) above target
  Normal  SuccessfulRescale  5m35s  horizontal-pod-autoscaler  New size: 7; reason: All metrics below target
  Normal  SuccessfulRescale  4m35s  horizontal-pod-autoscaler  New size: 5; reason: All metrics below target
  Normal  SuccessfulRescale  4m5s   horizontal-pod-autoscaler  New size: 2; reason: All metrics below target
  Normal  SuccessfulRescale  3m50s  horizontal-pod-autoscaler  New size: 1; reason: All metrics below target

(참고) HPA 프로메테우스 메트릭

kube_horizontalpodautoscaler_status_current_replicas
kube_horizontalpodautoscaler_status_desired_replicas
kube_horizontalpodautoscaler_status_target_metric
kube_horizontalpodautoscaler_status_condition

kube_horizontalpodautoscaler_spec_target_metric
kube_horizontalpodautoscaler_spec_min_replicas
kube_horizontalpodautoscaler_spec_max_replicas

# [운영서버 EC2]
kubectl get pod -n monitoring -l app.kubernetes.io/name=kube-state-metrics -owide
kubectl get pod -n monitoring -l app.kubernetes.io/name=kube-state-metrics -o jsonpath="{.items[*].status.podIP}"
PODIP=$(kubectl get pod -n monitoring -l app.kubernetes.io/name=kube-state-metrics -o jsonpath="{.items[*].status.podIP}")
curl -s http://$PODIP:8080/metrics | grep -i horizontalpodautoscaler | grep HELP
# HELP kube_horizontalpodautoscaler_info Information about this autoscaler.
# HELP kube_horizontalpodautoscaler_metadata_generation [STABLE] The generation observed by the HorizontalPodAutoscaler controller.
# HELP kube_horizontalpodautoscaler_spec_max_replicas [STABLE] Upper limit for the number of pods that can be set by the autoscaler; cannot be smaller than MinReplicas.
# HELP kube_horizontalpodautoscaler_spec_min_replicas [STABLE] Lower limit for the number of pods that can be set by the autoscaler, default 1.
# HELP kube_horizontalpodautoscaler_spec_target_metric The metric specifications used by this autoscaler when calculating the desired replica count.
# HELP kube_horizontalpodautoscaler_status_target_metric The current metric status used by this autoscaler when calculating the desired replica count.
# HELP kube_horizontalpodautoscaler_status_current_replicas [STABLE] Current number of replicas of pods managed by this autoscaler.
# HELP kube_horizontalpodautoscaler_status_desired_replicas [STABLE] Desired number of replicas of pods managed by this autoscaler.
# HELP kube_horizontalpodautoscaler_annotations Kubernetes annotations converted to Prometheus labels.
# HELP kube_horizontalpodautoscaler_labels [STABLE] Kubernetes labels converted to Prometheus labels.
# HELP kube_horizontalpodautoscaler_status_condition [STABLE] The condition of this autoscaler.

curl -s http://$PODIP:8080/metrics | grep -i horizontalpodautoscaler
...

실습 리소스 삭제

[root@operator-host ~]# kubectl delete deploy,svc,hpa,pod --all
deployment.apps "php-apache" deleted
service "kubernetes" deleted
service "php-apache" deleted
horizontalpodautoscaler.autoscaling "php-apache" deleted
pod "php-apache-d87b7ff46-62z46" deleted
pod "php-apache-d87b7ff46-dqb22" deleted
pod "php-apache-d87b7ff46-gtvsd" deleted
pod "php-apache-d87b7ff46-jd6ls" deleted
pod "php-apache-d87b7ff46-tt858" deleted
pod "php-apache-d87b7ff46-wtg6n" deleted
pod "php-apache-d87b7ff46-zwrtz" deleted

KEDA - Kubernetes based Event Driven Autoscaler

KEDA AutoScaler 소개 - Docs , DevOcean반면에 KEDA는 특정 이벤트를 기반으로 스케일 여부를 결정할 수 있습니다.이러한 이벤트를 활용하여 worker의 scale을 결정한다면 queue에 task가 많이 추가되는 시점에 더 빠르게 확장할 수 있습니다.
예를 들어 airflow는 metadb를 통해 현재 실행 중이거나 대기 중인 task가 얼마나 존재하는지 알 수 있습니다.
기존의 HPA(Horizontal Pod Autoscaler)는 리소스(CPU, Memory) 메트릭을 기반으로 스케일 여부를 결정하게 됩니다.

Agent — KEDA activates and deactivates Kubernetes Deployments to scale to and from zero on no events. This is one of the primary roles of the keda-operator container that runs when you install KEDA.
Metrics — KEDA acts as a Kubernetes metrics server that exposes rich event data like queue length or stream lag to the Horizontal Pod Autoscaler to drive scale out. It is up to the Deployment to consume the events directly from the source. This preserves rich event integration and enables gestures like completing or abandoning queue messages to work out of the box. The metric serving is the primary role of the keda-operator-metrics-apiserver container that runs when you install KEDA.
Admission Webhooks - Automatically validate resource changes to prevent misconfiguration and enforce best practices by using an admission controller. As an example, it will prevent multiple ScaledObjects to target the same scale target. keda-admission-webhooks

kubectl get pod -n keda
NAME                                               READY   STATUS    RESTARTS      AGE
keda-operator-6bdffdc78-5rqnp                      1/1     Running   1 (11m ago)   11m
keda-operator-metrics-apiserver-74d844d769-2vrcq   1/1     Running   0             11m
keda-admission-webhooks-86cffccbf5-kmb7v           1/1     Running   0             11m

예) KEDA Scalers : kafka trigger for an Apache Kafka topic - Link

KEDA | Apache Kafka

Scale applications based on an Apache Kafka topic or other services that support Kafka protocol.

keda.sh

triggers:

- type: kafka

metadata:

bootstrapServers: kafka.svc:9092

consumerGroup: my-group

topic: test-topic

lagThreshold: '5' # Average target value to trigger scaling actions. (Default: 5, Optional)

activationLagThreshold: '3' # Target value for activating the scaler. Learn more about activation here.

offsetResetPolicy: latest

allowIdleConsumers: false

scaleToZeroOnInvalidOffset: false

excludePersistentLag: false

limitToPartitionsWithLag: false

version: 1.0.0

partitionLimitation: '1,2,10-20,31'

sasl: plaintext

tls: enable

unsafeSsl: 'false'

KEDA with Helm : 특정 **이벤트(cron 등)**기반의 파드 오토 스케일링 - Chart , Grafana , Cron , SQS_Scale , aws-sqs-queue
- KEDA 대시보드 Import : https://github.com/kedacore/keda/blob/main/config/grafana/keda-dashboard.json

{

"annotations": {

"list": [

{

"builtIn": 1,

"datasource": {

"type": "grafana",

"uid": "-- Grafana --"

"enable": true,

"hide": true,

"iconColor": "rgba(0, 211, 255, 1)",

"name": "Annotations & Alerts",

"target": {

"limit": 100,

"matchAny": false,

"tags": [],

"type": "dashboard"

}

]

"description": "Visualize metrics provided by KEDA",

"editable": true,

"fiscalYearStartMonth": 0,

"graphTooltip": 0,

"id": 1653,

"links": [],

"liveNow": false,

"panels": [

{

"collapsed": false,

"gridPos": {

"h": 1,

"w": 24,

"x": 0,

"y": 0

"id": 8,

"panels": [],

"title": "Metric Server",

"type": "row"

{

"datasource": {

"type": "prometheus",

"uid": "${datasource}"

"description": "The total number of errors encountered for all scalers.",

"fieldConfig": {

"defaults": {

"color": {

"mode": "palette-classic"

"custom": {

"axisCenteredZero": false,

"axisColorMode": "text",

"axisLabel": "",

"axisPlacement": "auto",

"barAlignment": 0,

"drawStyle": "line",

"fillOpacity": 25,

"gradientMode": "opacity",

"hideFrom": {

"legend": false,

"tooltip": false,

"viz": false

"lineInterpolation": "linear",

"lineWidth": 2,

"pointSize": 5,

"scaleDistribution": {

"type": "linear"

"showPoints": "never",

"spanNulls": true,

"stacking": {

"group": "A",

"mode": "none"

"thresholdsStyle": {

"mode": "off"

}

"mappings": [],

"thresholds": {

"mode": "absolute",

"steps": [

{

"color": "green",

"value": null

{

"color": "red",

"value": 80

}

]

"unit": "Errors/sec"

"overrides": [

{

"matcher": {

"id": "byName",

"options": "http-demo"

"properties": [

{

"id": "color",

"value": {

"fixedColor": "red",

"mode": "fixed"

}

]

{

"matcher": {

"id": "byName",

"options": "scaledObject"

"properties": [

{

"id": "color",

"value": {

"fixedColor": "red",

"mode": "fixed"

}

]

{

"matcher": {

"id": "byName",

"options": "keda-system/keda-operator-metrics-apiserver"

"properties": [

{

"id": "color",

"value": {

"fixedColor": "red",

"mode": "fixed"

}

]

}

]

"gridPos": {

"h": 9,

"w": 8,

"x": 0,

"y": 1

"id": 4,

"options": {

"legend": {

"calcs": [],

"displayMode": "list",

"placement": "bottom",

"showLegend": true

"tooltip": {

"mode": "single",

"sort": "none"

}

"targets": [

{

"datasource": {

"type": "prometheus",

"uid": "${datasource}"

"editorMode": "code",

"expr": "sum by(job) (rate(keda_scaler_detail_errors_total{}[5m]))",

"legendFormat": "{{ job }}",

"range": true,

"refId": "A"

}

"title": "Scaler Total Errors",

"type": "timeseries"

{

"datasource": {

"type": "prometheus",

"uid": "${datasource}"

"description": "The number of errors that have occurred for each scaler.",

"fieldConfig": {

"defaults": {

"color": {

"mode": "palette-classic"

"custom": {

"axisCenteredZero": false,

"axisColorMode": "text",

"axisLabel": "",

"axisPlacement": "auto",

"barAlignment": 0,

"drawStyle": "line",

"fillOpacity": 25,

"gradientMode": "opacity",

"hideFrom": {

"legend": false,

"tooltip": false,

"viz": false

"lineInterpolation": "linear",

"lineWidth": 2,

"pointSize": 5,

"scaleDistribution": {

"type": "linear"

"showPoints": "never",

"spanNulls": true,

"stacking": {

"group": "A",

"mode": "none"

"thresholdsStyle": {

"mode": "off"

}

"mappings": [],

"thresholds": {

"mode": "absolute",

"steps": [

{

"color": "green",

"value": null

{

"color": "red",

"value": 80

}

]

"unit": "Errors/sec"

"overrides": [

{

"matcher": {

"id": "byName",

"options": "http-demo"

"properties": [

{

"id": "color",

"value": {

"fixedColor": "red",

"mode": "fixed"

}

]

{

"matcher": {

"id": "byName",

"options": "scaler"

"properties": [

{

"id": "color",

"value": {

"fixedColor": "red",

"mode": "fixed"

}

]

{

"matcher": {

"id": "byName",

"options": "prometheusScaler"

"properties": [

{

"id": "color",

"value": {

"fixedColor": "red",

"mode": "fixed"

}

]

}

]

"gridPos": {

"h": 9,

"w": 8,

"x": 8,

"y": 1

"id": 3,

"options": {

"legend": {

"calcs": [],

"displayMode": "list",

"placement": "bottom",

"showLegend": true

"tooltip": {

"mode": "single",

"sort": "none"

}

"targets": [

{

"datasource": {

"type": "prometheus",

"uid": "${datasource}"

"editorMode": "code",

"expr": "sum by(scaler) (rate(keda_scaler_detail_errors_total{exported_namespace=~\"$namespace\", scaledObject=~\"$scaledObject\", scaler=~\"$scaler\"}[5m]))",

"legendFormat": "{{ scaler }}",

"range": true,

"refId": "A"

}

"title": "Scaler Errors",

"type": "timeseries"

{

"datasource": {

"type": "prometheus",

"uid": "${datasource}"

"description": "The number of errors that have occurred for each scaled object.",

"fieldConfig": {

"defaults": {

"color": {

"mode": "palette-classic"

"custom": {

"axisCenteredZero": false,

"axisColorMode": "text",

"axisLabel": "",

"axisPlacement": "auto",

"barAlignment": 0,

"drawStyle": "line",

"fillOpacity": 25,

"gradientMode": "opacity",

"hideFrom": {

"legend": false,

"tooltip": false,

"viz": false

"lineInterpolation": "linear",

"lineWidth": 2,

"pointSize": 5,

"scaleDistribution": {

"type": "linear"

"showPoints": "never",

"spanNulls": true,

"stacking": {

"group": "A",

"mode": "none"

"thresholdsStyle": {

"mode": "off"

}

"mappings": [],

"thresholds": {

"mode": "absolute",

"steps": [

{

"color": "green",

"value": null

{

"color": "red",

"value": 80

}

]

"unit": "Errors/sec"

"overrides": [

{

"matcher": {

"id": "byName",

"options": "http-demo"

"properties": [

{

"id": "color",

"value": {

"fixedColor": "red",

"mode": "fixed"

}

]

}

]

"gridPos": {

"h": 9,

"w": 8,

"x": 16,

"y": 1

"id": 2,

"options": {

"legend": {

"calcs": [],

"displayMode": "list",

"placement": "bottom",

"showLegend": true

"tooltip": {

"mode": "single",

"sort": "none"

}

"targets": [

{

"datasource": {

"type": "prometheus",

"uid": "${datasource}"

"editorMode": "code",

"expr": "sum by(scaledObject) (rate(keda_scaled_object_errors_total{exported_namespace=~\"$namespace\", scaledObject=~\"$scaledObject\"}[5m]))",

"legendFormat": "{{ scaledObject }}",

"range": true,

"refId": "A"

}

"title": "Scaled Object Errors",

"type": "timeseries"

{

"collapsed": false,

"gridPos": {

"h": 1,

"w": 24,

"x": 0,

"y": 10

"id": 10,

"panels": [],

"title": "Scale Target",

"type": "row"

{

"datasource": {

"type": "prometheus",

"uid": "${datasource}"

"description": "The current value for each scaler’s metric that would be used by the HPA in computing the target average.",

"fieldConfig": {

"defaults": {

"color": {

"mode": "palette-classic"

"custom": {

"axisCenteredZero": false,

"axisColorMode": "text",

"axisLabel": "",

"axisPlacement": "auto",

"barAlignment": 0,

"drawStyle": "line",

"fillOpacity": 25,

"gradientMode": "opacity",

"hideFrom": {

"legend": false,

"tooltip": false,

"viz": false

"lineInterpolation": "linear",

"lineWidth": 2,

"pointSize": 5,

"scaleDistribution": {

"type": "linear"

"showPoints": "never",

"spanNulls": true,

"stacking": {

"group": "A",

"mode": "none"

"thresholdsStyle": {

"mode": "off"

}

"mappings": [],

"thresholds": {

"mode": "absolute",

"steps": [

{

"color": "green",

"value": null

{

"color": "red",

"value": 80

}

]

"unit": "none"

"overrides": [

{

"matcher": {

"id": "byName",

"options": "http-demo"

"properties": [

{

"id": "color",

"value": {

"fixedColor": "blue",

"mode": "fixed"

}

]

}

]

"gridPos": {

"h": 9,

"w": 24,

"x": 0,

"y": 11

"id": 5,

"options": {

"legend": {

"calcs": [],

"displayMode": "list",

"placement": "bottom",

"showLegend": true

"tooltip": {

"mode": "single",

"sort": "none"

}

"targets": [

{

"datasource": {

"type": "prometheus",

"uid": "${datasource}"

"editorMode": "code",

"expr": "sum by(metric) (keda_scaler_metrics_value{exported_namespace=~\"$namespace\", metric=~\"$metric\", scaledObject=~\"$scaledObject\"})",

"legendFormat": "{{ metric }}",

"range": true,

"refId": "A"

}

"title": "Scaler Metric Value",

"type": "timeseries"

{

"datasource": {

"type": "prometheus",

"uid": "${datasource}"

"description": "shows current replicas against max ones based on time difference",

"fieldConfig": {

"defaults": {

"color": {

"mode": "palette-classic"

"custom": {

"axisCenteredZero": false,

"axisColorMode": "text",

"axisLabel": "",

"axisPlacement": "auto",

"barAlignment": 0,

"drawStyle": "line",

"fillOpacity": 21,

"gradientMode": "opacity",

"hideFrom": {

"legend": false,

"tooltip": false,

"viz": false

"lineInterpolation": "linear",

"lineStyle": {

"fill": "solid"

"lineWidth": 1,

"pointSize": 5,

"scaleDistribution": {

"type": "linear"

"showPoints": "auto",

"spanNulls": false,

"stacking": {

"group": "A",

"mode": "none"

"thresholdsStyle": {

"mode": "off"

}

"mappings": [],

"thresholds": {

"mode": "absolute",

"steps": [

{

"color": "green",

"value": null

}

]

"unit": "short"

"overrides": []

"gridPos": {

"h": 8,

"w": 24,

"x": 0,

"y": 20

"id": 13,

"options": {

"legend": {

"calcs": [],

"displayMode": "list",

"placement": "bottom",

"showLegend": true

"tooltip": {

"mode": "single",

"sort": "none"

}

"targets": [

{

"datasource": {

"type": "prometheus",

"uid": "${datasource}"

"editorMode": "code",

"exemplar": false,

"expr": "kube_horizontalpodautoscaler_status_current_replicas{namespace=\"$namespace\",horizontalpodautoscaler=~\"keda-hpa-$scaledObject\"}",

"format": "time_series",

"instant": false,

"interval": "",

"legendFormat": "current_replicas",

"range": true,

"refId": "A"

{

"datasource": {

"type": "prometheus",

"uid": "${datasource}"

"editorMode": "code",

"exemplar": false,

"expr": "kube_horizontalpodautoscaler_spec_max_replicas{namespace=\"$namespace\",horizontalpodautoscaler=~\"keda-hpa-$scaledObject\"}",

"format": "time_series",

"hide": false,

"instant": false,

"legendFormat": "max_replicas",

"range": true,

"refId": "B"

}

"title": "Current/max replicas (time based)",

"type": "timeseries"

{

"datasource": {

"type": "prometheus",

"uid": "${datasource}"

"description": "shows current replicas against max ones based on time difference",

"fieldConfig": {

"defaults": {

"color": {

"mode": "continuous-GrYlRd"

"custom": {

"fillOpacity": 70,

"lineWidth": 0,

"spanNulls": false

"mappings": [

{

"options": {

"0": {

"color": "green",

"index": 0,

"text": "No scaling"

}

"type": "value"

{

"options": {

"from": -200,

"result": {

"color": "light-red",

"index": 1,

"text": "Scaling down"

"to": 0

"type": "range"

{

"options": {

"from": 0,

"result": {

"color": "semi-dark-red",

"index": 2,

"text": "Scaling up"

"to": 200

"type": "range"

}

"thresholds": {

"mode": "absolute",

"steps": [

{

"color": "green",

"value": null

}

]

"unit": "none"

"overrides": []

"gridPos": {

"h": 8,

"w": 24,

"x": 0,

"y": 28

"id": 16,

"options": {

"alignValue": "left",

"legend": {

"displayMode": "list",

"placement": "bottom",

"showLegend": false,

"width": 0

"mergeValues": true,

"rowHeight": 1,

"showValue": "never",

"tooltip": {

"mode": "single",

"sort": "none"

}

"targets": [

{

"datasource": {

"type": "prometheus",

"uid": "${datasource}"

"editorMode": "code",

"exemplar": false,

"expr": "delta(kube_horizontalpodautoscaler_status_current_replicas{namespace=\"$namespace\",horizontalpodautoscaler=~\"keda-hpa-$scaledObject\"}[1m])",

"format": "time_series",

"instant": false,

"interval": "",

"legendFormat": ".",

"range": true,

"refId": "A"

}

"title": "Changes in replicas",

"type": "state-timeline"

{

"datasource": {

"type": "prometheus",

"uid": "${datasource}"

"description": "shows current replicas against max ones",

"fieldConfig": {

"defaults": {

"color": {

"mode": "thresholds"

"mappings": [],

"min": 0,

"thresholds": {

"mode": "percentage",

"steps": [

{

"color": "green"

{

"color": "red",

"value": 80

}

]

"unit": "short"

"overrides": []

"gridPos": {

"h": 8,

"w": 12,

"x": 0,

"y": 36

"id": 15,

"options": {

"orientation": "auto",

"reduceOptions": {

"calcs": [

"lastNotNull"

"fields": "/^current_replicas$/",

"values": false

"showThresholdLabels": false,

"showThresholdMarkers": true

"pluginVersion": "9.5.2",

"targets": [

{

"datasource": {

"type": "prometheus",

"uid": "${datasource}"

"editorMode": "code",

"exemplar": false,

"expr": "kube_horizontalpodautoscaler_status_current_replicas{namespace=\"$namespace\",horizontalpodautoscaler=~\"keda-hpa-$scaledObject\"}",

"instant": true,

"legendFormat": "current_replicas",

"range": false,

"refId": "A"

{

"datasource": {

"type": "prometheus",

"uid": "${datasource}"

"editorMode": "code",

"exemplar": false,

"expr": "kube_horizontalpodautoscaler_spec_max_replicas{namespace=\"$namespace\",horizontalpodautoscaler=~\"keda-hpa-$scaledObject\"}",

"hide": false,

"instant": true,

"legendFormat": "max_replicas",

"range": false,

"refId": "B"

}

"title": "Current/max replicas",

"type": "gauge"

}

"refresh": "1m",

"schemaVersion": 38,

"style": "dark",

"tags": [],

"templating": {

"list": [

{

"current": {

"selected": false,

"text": "Prometheus",

"value": "Prometheus"

"hide": 0,

"includeAll": false,

"multi": false,

"name": "datasource",

"options": [],

"query": "prometheus",

"queryValue": "",

"refresh": 1,

"regex": "",

"skipUrlSync": false,

"type": "datasource"

{

"current": {

"selected": false,

"text": "bhe-test",

"value": "bhe-test"

"datasource": {

"type": "prometheus",

"uid": "${datasource}"

"definition": "label_values(keda_scaler_active,exported_namespace)",

"hide": 0,

"includeAll": false,

"multi": false,

"name": "namespace",

"options": [],

"query": {

"query": "label_values(keda_scaler_active,exported_namespace)",

"refId": "PrometheusVariableQueryEditor-VariableQuery"

"refresh": 1,

"regex": "",

"skipUrlSync": false,

"sort": 1,

"type": "query"

{

"current": {

"selected": false,

"text": "All",

"value": "$__all"

"datasource": {

"type": "prometheus",

"uid": "${datasource}"

"definition": "label_values(keda_scaler_active{exported_namespace=\"$namespace\"},scaledObject)",

"hide": 0,

"includeAll": true,

"multi": true,

"name": "scaledObject",

"options": [],

"query": {

"query": "label_values(keda_scaler_active{exported_namespace=\"$namespace\"},scaledObject)",

"refId": "PrometheusVariableQueryEditor-VariableQuery"

"refresh": 2,

"regex": "",

"skipUrlSync": false,

"sort": 0,

"type": "query"

{

"current": {

"selected": false,

"text": "cronScaler",

"value": "cronScaler"

"datasource": {

"type": "prometheus",

"uid": "${datasource}"

"definition": "label_values(keda_scaler_active{exported_namespace=\"$namespace\"},scaler)",

"hide": 0,

"includeAll": false,

"multi": false,

"name": "scaler",

"options": [],

"query": {

"query": "label_values(keda_scaler_active{exported_namespace=\"$namespace\"},scaler)",

"refId": "PrometheusVariableQueryEditor-VariableQuery"

"refresh": 2,

"regex": "",

"skipUrlSync": false,

"sort": 0,

"type": "query"

{

"current": {

"selected": false,

"text": "s0-cron-Etc-UTC-40xxxx-55xxxx",

"value": "s0-cron-Etc-UTC-40xxxx-55xxxx"

"datasource": {

"type": "prometheus",

"uid": "${datasource}"

"definition": "label_values(keda_scaler_active{exported_namespace=\"$namespace\"},metric)",

"hide": 0,

"includeAll": false,

"multi": false,

"name": "metric",

"options": [],

"query": {

"query": "label_values(keda_scaler_active{exported_namespace=\"$namespace\"},metric)",

"refId": "PrometheusVariableQueryEditor-VariableQuery"

"refresh": 2,

"regex": "",

"skipUrlSync": false,

"sort": 0,

"type": "query"

}

]

"time": {

"from": "now-24h",

"to": "now"

"timepicker": {},

"timezone": "",

"title": "KEDA",

"uid": "asdasd8rvmMxdVk",

"version": 8,

"weekStart": ""

}

# 설치 전 기존 metrics-server 제공 Metris API 확인
kubectl get --raw "/apis/metrics.k8s.io" -v=6 | jq
kubectl get --raw "/apis/metrics.k8s.io" | jq
{
  "kind": "APIGroup",
  "apiVersion": "v1",
  "name": "metrics.k8s.io",
  ...

# KEDA 설치 : serviceMonitor 만으로도 충분할듯..
cat <<EOT > keda-values.yaml

metricsServer:

useHostNetwork: true

prometheus:

metricServer:

enabled: true

port: 9022

portName: metrics

path: /metrics

serviceMonitor:

# Enables ServiceMonitor creation for the Prometheus Operator

enabled: true

podMonitor:

# Enables PodMonitor creation for the Prometheus Operator

enabled: true

operator:

enabled: true

port: 8080

serviceMonitor:

# Enables ServiceMonitor creation for the Prometheus Operator

enabled: true

podMonitor:

# Enables PodMonitor creation for the Prometheus Operator

enabled: true

webhooks:

enabled: true

port: 8020

serviceMonitor:

# Enables ServiceMonitor creation for the Prometheus webhooks

enabled: true

EOT

helm repo add kedacore https://kedacore.github.io/charts
helm repo update
helm install keda kedacore/keda --version 2.16.0 --namespace keda --create-namespace -f keda-values.yaml

# KEDA 설치 확인
[root@operator-host ~]# kubectl get crd | grep keda
cloudeventsources.eventing.keda.sh           2025-03-08T13:40:25Z
clustercloudeventsources.eventing.keda.sh    2025-03-08T13:40:25Z
clustertriggerauthentications.keda.sh        2025-03-08T13:40:25Z
scaledjobs.keda.sh                           2025-03-08T13:40:25Z
scaledobjects.keda.sh                        2025-03-08T13:40:25Z
triggerauthentications.keda.sh               2025-03-08T13:40:25Z

[root@operator-host ~]# kubectl get all -n keda
NAME                                                   READY   STATUS    RESTARTS       AGE
pod/keda-admission-webhooks-86cffccbf5-84c5m           1/1     Running   0              3m17s
pod/keda-operator-6bdffdc78-9v76f                      1/1     Running   1 (3m4s ago)   3m17s
pod/keda-operator-metrics-apiserver-74d844d769-vdvcr   1/1     Running   0              3m17s

NAME                                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/keda-admission-webhooks           ClusterIP   10.100.214.210   <none>        443/TCP,8020/TCP    3m18s
service/keda-operator                     ClusterIP   10.100.2.93      <none>        9666/TCP,8080/TCP   3m18s
service/keda-operator-metrics-apiserver   ClusterIP   10.100.7.86      <none>        443/TCP,9022/TCP    3m18s

NAME                                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/keda-admission-webhooks           1/1     1            1           3m18s
deployment.apps/keda-operator                     1/1     1            1           3m18s
deployment.apps/keda-operator-metrics-apiserver   1/1     1            1           3m18s

NAME                                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/keda-admission-webhooks-86cffccbf5           1         1         1       3m17s
replicaset.apps/keda-operator-6bdffdc78                      1         1         1       3m17s
replicaset.apps/keda-operator-metrics-apiserver-74d844d769   1         1         1       3m17s

[root@operator-host ~]# kubectl get validatingwebhookconfigurations keda-admission -o yaml

apiVersion: admissionregistration.k8s.io/v1

kind: ValidatingWebhookConfiguration

metadata:

annotations:

meta.helm.sh/release-name: keda

meta.helm.sh/release-namespace: keda

creationTimestamp: "2025-03-08T13:40:26Z"

generation: 2

labels:

app.kubernetes.io/component: operator

app.kubernetes.io/instance: keda

app.kubernetes.io/managed-by: Helm

app.kubernetes.io/name: keda-admission-webhooks

app.kubernetes.io/part-of: keda-operator

app.kubernetes.io/version: 2.16.0

helm.sh/chart: keda-2.16.0

resourceVersion: "102153"

uid: 662907eb-01fa-4544-9778-3d16f160adb1

webhooks:

- admissionReviewVersions:

- v1

clientConfig:

caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURFRENDQWZpZ0F3SUJBZ0lCQURBTkJna3Foa2lHOXcwQkFRc0ZBREFoTVJBd0RnWURWUVFLRXdkTFJVUkIKVDFKSE1RMHdDd1lEVlFRREV3UkxSVVJCTUI0WERUSTFNRE13T0RFeU5EQXpPRm9YRFRNMU1ETXdOakV6TkRBegpPRm93SVRFUU1BNEdBMVVFQ2hNSFMwVkVRVTlTUnpFTk1Bc0dBMVVFQXhNRVMwVkVRVENDQVNJd0RRWUpLb1pJCmh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTVdkcnlGWW5FMndaTDU2bG95dmdDc0c5ZmoxcUhtRld2QXQKM25KeVVibnhPYnQ4Wi9yR29pU2tkTGpQRHVlUGNoMUdGeE11OXNUeEljUlhCRDR1cGZKYW5za1VncHZYRDE1dgo0R2Fob24wekJCcmF4bkRFREVydjZ1ZXhpbmMyaXhCY0UwcEVHMjUvRnZHYi9yRVh6eUlPOGlnYVg1eXUyNWtOCk5VRERFV0RONzZ4QVZMbytIaUlranJWSERHMHhYZitSTXVuaVAzYktpb1lxZWVuLzF5ZURBNWwweTBGVERIYzcKWm1qdFRQam5zdFVHSnQxaVlPZm9oM2Nqbm04SnRQWlBydVQyeTk0MmxJMElxZHZGRkkwK0ExVFg5WE1BOExuNQpXRVNIQTF5VllVVjF5WjJaZUovc3RCRGJJVGRsbGxRbEFXS2ZCOHgyZ0ZjaEVDL01aeFVDQXdFQUFhTlRNRkV3CkRnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME9CQllFRkZrWHhwSWcKdkhwTlYrTVNpbm9RcUNxUzlISVVNQThHQTFVZEVRUUlNQWFDQkV0RlJFRXdEUVlKS29aSWh2Y05BUUVMQlFBRApnZ0VCQUx5cjVFVlExWEdkMXYzdDROT25mMWE1NnQ5QWhvM3NlcG15UlMrODQrZ2VsN1FUT3ZtdnYxcDNXNURVCkx5eHRGSTNYR0p6S21tdWV2NFU3K2hsSENBU0FQNkRoRktDTFdrWmhJUVJLV0d0SGIzWXFlU09hNE4rTkdhdHkKQVNJTGM1K3htb2NvQzdWaU90aDVtVjYyQWcwdmhIdXJQRWV1TUpDeitoZlFJZmFHQXU0MUdMSDhDVXlnRVNKWQpGZVRTUUdrcCsrazJwQmszcVlzY1czSWNobDV0VWE1MTU2TDAyZlhEQUJLeDJHSDdKa0RETzRwcU4zL2RCV2VhCmo0SVdzcGtJUloyOERWc1EzakZ2SEM1R0dZa3duMVFRRU9jeVlnSjhGS2Q4VTFGbU1VckNMSGhZOUtNRWwvODkKUEk5VUIzbmVmMTAwV3UyQUc5WTdWWkNJRGdnPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==

service:

namespace: keda

path: /validate-keda-sh-v1alpha1-scaledobject

port: 443

failurePolicy: Ignore

matchPolicy: Equivalent

namespaceSelector: {}

objectSelector: {}

rules:

- apiGroups:

- keda.sh

apiVersions:

- v1alpha1

operations:

- CREATE

- UPDATE

resources:

- scaledobjects

scope: '*'

sideEffects: None

timeoutSeconds: 10

- admissionReviewVersions:

- v1

clientConfig:

service:

namespace: keda

path: /validate-keda-sh-v1alpha1-triggerauthentication

port: 443

failurePolicy: Ignore

matchPolicy: Equivalent

namespaceSelector: {}

objectSelector: {}

rules:

- apiGroups:

- keda.sh

apiVersions:

- v1alpha1

operations:

- CREATE

- UPDATE

resources:

- triggerauthentications

scope: '*'

sideEffects: None

timeoutSeconds: 10

- admissionReviewVersions:

- v1

clientConfig:

service:

namespace: keda

path: /validate-keda-sh-v1alpha1-clustertriggerauthentication

port: 443

failurePolicy: Ignore

matchPolicy: Equivalent

namespaceSelector: {}

objectSelector: {}

rules:

- apiGroups:

- keda.sh

apiVersions:

- v1alpha1

operations:

- CREATE

- UPDATE

resources:

- clustertriggerauthentications

scope: '*'

sideEffects: None

timeoutSeconds: 10

[root@operator-host ~]# kubectl get podmonitor,servicemonitors -n keda
NAME                                                               AGE
podmonitor.monitoring.coreos.com/keda-operator                     4m16s
podmonitor.monitoring.coreos.com/keda-operator-metrics-apiserver   4m16s

NAME                                                                   AGE
servicemonitor.monitoring.coreos.com/keda-admission-webhooks           4m16s
servicemonitor.monitoring.coreos.com/keda-operator                     4m16s
servicemonitor.monitoring.coreos.com/keda-operator-metrics-apiserver   4m16s

[root@operator-host ~]# kubectl get apiservice v1beta1.external.metrics.k8s.io -o yaml

apiVersion: apiregistration.k8s.io/v1

kind: APIService

metadata:

annotations:

meta.helm.sh/release-name: keda

meta.helm.sh/release-namespace: keda

creationTimestamp: "2025-03-08T13:40:25Z"

labels:

app.kubernetes.io/component: operator

app.kubernetes.io/instance: keda

app.kubernetes.io/managed-by: Helm

app.kubernetes.io/name: v1beta1.external.metrics.k8s.io

app.kubernetes.io/part-of: keda-operator

app.kubernetes.io/version: 2.16.0

helm.sh/chart: keda-2.16.0

resourceVersion: "102154"

uid: 76882ec1-33a8-4fce-a758-76738614b9ae

spec:

group: external.metrics.k8s.io

groupPriorityMinimum: 100

service:

namespace: keda

port: 443

version: v1beta1

versionPriority: 100

status:

conditions:

- lastTransitionTime: "2025-03-08T13:40:54Z"

message: all checks passed

reason: Passed

status: "True"

type: Available

# CPU/Mem은 기존 metrics-server 의존하여, KEDA metrics-server는 외부 이벤트 소스(Scaler) 메트릭을 노출
## https://keda.sh/docs/2.16/operate/metrics-server/
[root@operator-host ~]# kubectl get pod -n keda -l app=keda-operator-metrics-apiserver
NAME READY STATUS RESTARTS AGE
keda-operator-metrics-apiserver-74d844d769-vdvcr 1/1 Running 0 8m9s

# Querying metrics exposed by KEDA Metrics Server
[root@operator-host ~]# kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1" | jq

{

"kind": "APIResourceList",

"apiVersion": "v1",

"groupVersion": "external.metrics.k8s.io/v1beta1",

"resources": [

{

"name": "externalmetrics",

"singularName": "",

"namespaced": true,

"kind": "ExternalMetricValueList",

"verbs": [

"get"

]

}

]

}

# keda 네임스페이스에 디플로이먼트 생성
[root@operator-host ~]# kubectl apply -f php-apache.yaml -n keda
deployment.apps/php-apache created
service/php-apache created
[root@operator-host ~]# kubectl get pod -n keda
NAME                                               READY   STATUS    RESTARTS      AGE
keda-admission-webhooks-86cffccbf5-84c5m           1/1     Running   0             13m
keda-operator-6bdffdc78-9v76f                      1/1     Running   1 (13m ago)   13m
keda-operator-metrics-apiserver-74d844d769-vdvcr   1/1     Running   0             13m
php-apache-d87b7ff46-tv2r6                         1/1     Running   0             61s

# ScaledObject 정책 생성 : cron
cat <<EOT > keda-cron.yaml

apiVersion: keda.sh/v1alpha1

kind: ScaledObject

metadata:

spec:

minReplicaCount: 0

maxReplicaCount: 2 # Specifies the maximum number of replicas to scale up to (defaults to 100).

pollingInterval: 30 # Specifies how often KEDA should check for scaling events

cooldownPeriod: 300 # Specifies the cool-down period in seconds after a scaling event

scaleTargetRef: # Identifies the Kubernetes deployment or other resource that should be scaled.

apiVersion: apps/v1

kind: Deployment

triggers: # Defines the specific configuration for your chosen scaler, including any required parameters or settings

- type: cron

metadata:

timezone: Asia/Seoul

start: 00,15,30,45 * * * *

end: 05,20,35,50 * * * *

desiredReplicas: "1"

EOT

[root@operator-host ~]# kubectl apply -f keda-cron.yaml -n keda
scaledobject.keda.sh/php-apache-cron-scaled created

# 그라파나 대시보드 추가 : 대시보드 상단에 namespace : keda 로 변경하기!

# 모니터링
watch -d 'kubectl get ScaledObject,hpa,pod -n keda'

# 확인
[root@operator-host ~]# kubectl get ScaledObject,hpa,pod -n keda
NAME                                          SCALETARGETKIND      SCALETARGETNAME   MIN   MAX   READY   ACTIVE   FALLBACK   PAUSED    TRIGGERS   AUTHENTICATIONS   AGE
scaledobject.keda.sh/php-apache-cron-scaled   apps/v1.Deployment   php-apache        0     2     True    True     False      Unknown                                6m26s

NAME                                                                  REFERENCE               TARGETS             MINPODS   MAXPODS   REPLICAS   AGE
horizontalpodautoscaler.autoscaling/keda-hpa-php-apache-cron-scaled   Deployment/php-apache   <unknown>/1 (avg)   1         2         1          6m26s

NAME                                                   READY   STATUS    RESTARTS      AGE
pod/keda-admission-webhooks-86cffccbf5-84c5m           1/1     Running   0             21m
pod/keda-operator-6bdffdc78-9v76f                      1/1     Running   1 (21m ago)   21m
pod/keda-operator-metrics-apiserver-74d844d769-vdvcr   1/1     Running   0             21m
pod/php-apache-d87b7ff46-bbbnb                         1/1     Running   0             115s

kubectl get hpa -o jsonpath="{.items[0].spec}" -n keda | jq
...
"metrics": [
    {
      "external": {
        "metric": {
          "name": "s0-cron-Asia-Seoul-00,15,30,45xxxx-05,20,35,50xxxx",
          "selector": {
            "matchLabels": {
              "scaledobject.keda.sh/name": "php-apache-cron-scaled"
            }
          }
        },
        "target": {
          "averageValue": "1",
          "type": "AverageValue"
        }
      },
      "type": "External"
    }

# KEDA 및 deployment 등 삭제
[root@operator-host ~]# kubectl delete ScaledObject -n keda php-apache-cron-scaled && kubectl delete deploy php-apache -n keda && helm uninstall keda -n keda
scaledobject.keda.sh "php-apache-cron-scaled" deleted
deployment.apps "php-apache" deleted
release "keda" uninstalled
[root@operator-host ~]# kubectl delete namespace keda
namespace "keda" deleted

VPA - Vertical Pod Autoscaler

VPA 소개 : pod resources.request을 최대한 최적값으로 수정

VPA는 HPA와 같이 사용할 수 없습니다.
VPA는 pod자원을 최적값으로 수정하기 위해 pod를 재실행(기존 pod를 종료하고 새로운 pod실행)합니다.
계산 방식 : ‘기준값(파드가 동작하는데 필요한 최소한의 값)’ 결정 → ‘마진(약간의 적절한 버퍼)’ 추가 → 상세정리 Link

Pod CPU/Memory 리소스 최적화하기 (VPA 및 Kubecost 추천로직 분석)

devocean.sk.com

# [운영서버 EC2] 코드 다운로드
# git clone https://github.com/kubernetes/autoscaler.git # userdata 로 설치 되어 있음
[root@operator-host ~]# cd ~/autoscaler/vertical-pod-autoscaler/
[root@operator-host vertical-pod-autoscaler]# tree hack
hack
├── api-docs
│   └── config.yaml
├── boilerplate.go.txt
├── convert-alpha-objects.sh
├── deploy-for-e2e-locally.sh
├── deploy-for-e2e.sh
├── dev-deploy-locally.sh
├── e2e
│   ├── Dockerfile.externalmetrics-writer
│   ├── k8s-metrics-server.yaml
│   ├── metrics-pump.yaml
│   ├── prometheus-adapter.yaml
│   ├── prometheus.yaml
│   ├── recommender-externalmetrics-deployment.yaml
│   └── vpa-rbac.diff
├── emit-metrics.py
├── generate-api-docs.sh
├── generate-crd-yaml.sh
├── generate-flags.sh
├── lib
│   └── util.sh
├── local-cluster.md
├── run-e2e-locally.sh
├── run-e2e.sh
├── run-e2e-tests.sh
├── tools.go
├── update-codegen.sh
├── update-kubernetes-deps-in-e2e.sh
├── update-kubernetes-deps.sh
├── verify-codegen.sh
├── verify-vpa-flags.sh
├── vpa-apply-upgrade.sh
├── vpa-down.sh
├── vpa-process-yaml.sh
├── vpa-process-yamls.sh
├── vpa-up.sh
└── warn-obsolete-vpa-objects.sh

3 directories, 34 files

# openssl 버전 확인
[root@operator-host vertical-pod-autoscaler]# openssl version
OpenSSL 1.0.2k-fips  26 Jan 2017

# 1.0 제거
[root@operator-host vertical-pod-autoscaler]# yum remove openssl -y

# openssl 1.1.1 이상 버전 확인
[root@operator-host vertical-pod-autoscaler]# yum install openssl11 -y
[root@operator-host vertical-pod-autoscaler]# openssl11 version
OpenSSL 1.1.1zb  11 Feb 2025

# 스크립트파일내에 openssl11 수정
[root@operator-host vertical-pod-autoscaler]# sed -i 's/openssl/openssl11/g' ~/autoscaler/vertical-pod-autoscaler/pkg/admission-controller/gencerts.sh

# git commit
git status
git config --global user.email "yu3papa.j@gmail.com"
git config --global user.name "yu3papa"
git add .
git commit -m "openssl version modify"

# Deploy the Vertical Pod Autoscaler to your cluster with the following command.
watch -d kubectl get pod -n kube-system

[root@operator-host vertical-pod-autoscaler]# cat hack/vpa-up.sh

#!/bin/bash

# Licensed under the Apache License, Version 2.0 (the "License");

# you may not use this file except in compliance with the License.

# You may obtain a copy of the License at

# http://www.apache.org/licenses/LICENSE-2.0

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

set -o errexit

set -o nounset

set -o pipefail

SCRIPT_ROOT=$(dirname ${BASH_SOURCE})/..

DEFAULT_TAG="1.3.0"

TAG_TO_APPLY=${TAG-$DEFAULT_TAG}

if [ "${TAG_TO_APPLY}" == "${DEFAULT_TAG}" ]; then

git switch --detach vertical-pod-autoscaler-${DEFAULT_TAG}

$SCRIPT_ROOT/hack/vpa-process-yamls.sh apply $*

[root@operator-host vertical-pod-autoscaler]# ./hack/vpa-up.sh
HEAD is now at 200a292f8 Merge pull request #7774 from jm-franc/vpa-finalize-branch
customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalercheckpoints.autoscaling.k8s.io created
customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalers.autoscaling.k8s.io created
clusterrole.rbac.authorization.k8s.io/system:metrics-reader created
clusterrole.rbac.authorization.k8s.io/system:vpa-actor created
clusterrole.rbac.authorization.k8s.io/system:vpa-status-actor created
clusterrole.rbac.authorization.k8s.io/system:vpa-checkpoint-actor created
clusterrole.rbac.authorization.k8s.io/system:evictioner created
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-actor created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-status-actor created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-checkpoint-actor created
clusterrole.rbac.authorization.k8s.io/system:vpa-target-reader created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-target-reader-binding created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-evictioner-binding created
serviceaccount/vpa-admission-controller created
serviceaccount/vpa-recommender created
serviceaccount/vpa-updater created
clusterrole.rbac.authorization.k8s.io/system:vpa-admission-controller created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-admission-controller created
clusterrole.rbac.authorization.k8s.io/system:vpa-status-reader created
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-status-reader-binding created
role.rbac.authorization.k8s.io/system:leader-locking-vpa-updater created
rolebinding.rbac.authorization.k8s.io/system:leader-locking-vpa-updater created
role.rbac.authorization.k8s.io/system:leader-locking-vpa-recommender created
rolebinding.rbac.authorization.k8s.io/system:leader-locking-vpa-recommender created
deployment.apps/vpa-updater created
deployment.apps/vpa-recommender created
Generating certs for the VPA Admission Controller in /tmp/vpa-certs.
./hack/../hack/../pkg/admission-controller/gencerts.sh: line 42: openssl: command not found
deployment.apps/vpa-admission-controller created
service/vpa-webhook created

# 재실행!
[root@operator-host vertical-pod-autoscaler]# sed -i 's/openssl/openssl11/g' ~/autoscaler/vertical-pod-autoscaler/pkg/admission-controller/gencerts.sh
[root@operator-host vertical-pod-autoscaler]# ./hack/vpa-up.sh
M       vertical-pod-autoscaler/pkg/admission-controller/gencerts.sh
HEAD is now at 200a292f8 Merge pull request #7774 from jm-franc/vpa-finalize-branch
customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalercheckpoints.autoscaling.k8s.io unchanged
customresourcedefinition.apiextensions.k8s.io/verticalpodautoscalers.autoscaling.k8s.io unchanged
clusterrole.rbac.authorization.k8s.io/system:metrics-reader unchanged
clusterrole.rbac.authorization.k8s.io/system:vpa-actor unchanged
clusterrole.rbac.authorization.k8s.io/system:vpa-status-actor unchanged
clusterrole.rbac.authorization.k8s.io/system:vpa-checkpoint-actor unchanged
clusterrole.rbac.authorization.k8s.io/system:evictioner unchanged
clusterrolebinding.rbac.authorization.k8s.io/system:metrics-reader unchanged
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-actor unchanged
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-status-actor unchanged
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-checkpoint-actor unchanged
clusterrole.rbac.authorization.k8s.io/system:vpa-target-reader unchanged
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-target-reader-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-evictioner-binding unchanged
serviceaccount/vpa-admission-controller unchanged
serviceaccount/vpa-recommender unchanged
serviceaccount/vpa-updater unchanged
clusterrole.rbac.authorization.k8s.io/system:vpa-admission-controller unchanged
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-admission-controller unchanged
clusterrole.rbac.authorization.k8s.io/system:vpa-status-reader unchanged
clusterrolebinding.rbac.authorization.k8s.io/system:vpa-status-reader-binding unchanged
role.rbac.authorization.k8s.io/system:leader-locking-vpa-updater unchanged
rolebinding.rbac.authorization.k8s.io/system:leader-locking-vpa-updater unchanged
role.rbac.authorization.k8s.io/system:leader-locking-vpa-recommender unchanged
rolebinding.rbac.authorization.k8s.io/system:leader-locking-vpa-recommender unchanged
deployment.apps/vpa-updater unchanged
deployment.apps/vpa-recommender unchanged
Generating certs for the VPA Admission Controller in /tmp/vpa-certs.
Generating RSA private key, 2048 bit long modulus (2 primes)
............................................................................+++++
.....+++++
e is 65537 (0x010001)
Can't load /root/.rnd into RNG
140406327777088:error:2406F079:random number generator:RAND_load_file:Cannot open file:crypto/rand/randfile.c:98:Filename=/root/.rnd
Generating RSA private key, 2048 bit long modulus (2 primes)
...........................................................+++++
..........................................................+++++
e is 65537 (0x010001)
Signature ok
subject=CN = vpa-webhook.kube-system.svc
Getting CA Private Key
Uploading certs to the cluster.
secret/vpa-tls-certs created
Deleting /tmp/vpa-certs.
deployment.apps/vpa-admission-controller unchanged
service/vpa-webhook unchanged

[root@operator-host vertical-pod-autoscaler]# kubectl get crd | grep autoscaling
verticalpodautoscalercheckpoints.autoscaling.k8s.io   2025-03-08T14:25:55Z
verticalpodautoscalers.autoscaling.k8s.io             2025-03-08T14:25:55Z

[root@operator-host vertical-pod-autoscaler]# kubectl get mutatingwebhookconfigurations vpa-webhook-config
NAME                 WEBHOOKS   AGE
vpa-webhook-config   1          14s

[root@operator-host vertical-pod-autoscaler]# kubectl get mutatingwebhookconfigurations vpa-webhook-config -o json | jq

{

"apiVersion": "admissionregistration.k8s.io/v1",

"kind": "MutatingWebhookConfiguration",

"metadata": {

"creationTimestamp": "2025-03-08T14:28:34Z",

"generation": 1,

"name": "vpa-webhook-config",

"resourceVersion": "116465",

"uid": "77d4a3aa-53ac-461c-8ff8-acaff602b3f6"

"webhooks": [

{

"admissionReviewVersions": [

"v1"

"clientConfig": {

"caBundle": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURMVENDQWhXZ0F3SUJBZ0lVVTlCNjR4ZEV2T1piVWNmQnZyTkhsZTVxWjdJd0RRWUpLb1pJaHZjTkFRRUwKQlFBd0dURVhNQlVHQTFVRUF3d09kbkJoWDNkbFltaHZiMnRmWTJFd0lCY05NalV3TXpBNE1UUXlOekkzV2hnUApNakk1T0RFeU1qSXhOREkzTWpkYU1Ca3hGekFWQmdOVkJBTU1Eblp3WVY5M1pXSm9iMjlyWDJOaE1JSUJJakFOCkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQXQySXZWbldSUlNaUkYxZHcrMlpNemJ4U0RUT0YKVjNDN25IZkRZcDI1UFpmUmY2WS81UXBLTDFJdWIzRytaUGNjVFJONVkwVlJWTUQ2VG5WRG1MMGJhYlF3U2xiRwptWGZyY3FzUnl4bmdmWHBobFJaYnNiWEtoZGdKaU5qRmVSYkpVZlBPZ3VLMk13YmtiaWxSOFF3S055ZCtzdWpaClZyUU5GSmNsRjFDR1BCSERaQVNTbHdVcEZZbERuTW9DZG9BTmpCT2pCRmdMeUNuaUxIZThHNjg1OHpVMVJBUWkKU3hvcGZjOXpFNCsvenhzWHE4YS84NUJuaWsxeTBsb1VuRk9EMDQySm80Z3JzdFFtNkp2SVFpZVY1cHZMeDN1bQpjMGJSM1BwOUhCTTF4Ty8rdGdYY29rVkUzb1hNSzlKQ3prWnYwa08rSHVGNnIzdFhjclNXWDd2dHV3SURBUUFCCm8yc3dhVEFkQmdOVkhRNEVGZ1FVK2FJYU9PLzljcVhMNWRzL0dCUEFJU3NnQ0d3d0h3WURWUjBqQkJnd0ZvQVUKK2FJYU9PLzljcVhMNWRzL0dCUEFJU3NnQ0d3d0RBWURWUjBUQkFVd0F3RUIvekFaQmdOVkhSRUVFakFRZ2c1MgpjR0ZmZDJWaWFHOXZhMTlqWVRBTkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQUNNenppREZQUGNyV0hzMnVBejNHCmdJL3czaXdKdmxPODhzWTNGYXZ0eXJLdGZMbGtTVVJ1MVhXQTVIYk9TUnB3Y1IyNEZlRFFFU2YwcS9NMlp5SnAKZDhRSWtSeHY4TzJBLzdpU0IrQ2krNTlVNXVEaUluY09EeUd4NTZXUktQUjJqVUJESTkzbVp1aUhFZHJ4ZHF4Ygp5SWJjZ2FKUHZKNDE4RndTamlVaHc2aytFQzlRVHI0S1U1dmdZVyt4NDhZN2VxaDZqMUQvMUxzODlrUkxMZGtMCnlycE1uMlNYNzhLMVRtWklZdnlvd3dtZ0VHaVllbk91N3M0K2RPYnZQTm5XQ3gyS253SFd3UjA1MDFZNDI0VkIKMlRYV0hwQ002SXk2OEJIcmNmeEwzK2MwOS9XWVVLTWlXNXg5d2hobERQNnAyRm51MUdreGxLYUNNbU1kR2tvUgp0dz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K",

"service": {

"name": "vpa-webhook",

"namespace": "kube-system",

"port": 443

}

"failurePolicy": "Ignore",

"matchPolicy": "Equivalent",

"name": "vpa.k8s.io",

"namespaceSelector": {

"matchExpressions": [

{

"key": "kubernetes.io/metadata.name",

"operator": "NotIn",

"values": [

]

}

]

"objectSelector": {},

"reinvocationPolicy": "Never",

"rules": [

{

"apiGroups": [

"apiVersions": [

"v1"

"operations": [

"CREATE"

"resources": [

"pods"

"scope": "*"

{

"apiGroups": [

"autoscaling.k8s.io"

"apiVersions": [

"*"

"operations": [

"CREATE",

"UPDATE"

"resources": [

"verticalpodautoscalers"

"scope": "*"

}

"sideEffects": "None",

"timeoutSeconds": 30

}

]

}

공식 예제 : pod가 실행되면 약 2~3분 뒤에 pod resource.reqeust가 VPA에 의해 수정 - 링크
- vpa에 spec.updatePolicy.updateMode를 Off 로 변경 시 파드에 Spec을 자동으로 변경 재실행 하지 않습니다. 기본값(Auto)

# 모니터링
watch -d "kubectl top pod;echo "----------------------";kubectl describe pod | grep Requests: -A2"

# 공식 예제 배포
[root@operator-host vertical-pod-autoscaler]# cat examples/hamster.yaml

# This config creates a deployment with two pods, each requesting 100 millicores

# and trying to utilize slightly above 500 millicores (repeatedly using CPU for

# 0.5s and sleeping 0.5s).

# It also creates a corresponding Vertical Pod Autoscaler that adjusts the

# requests.

# Note that the update mode is left unset, so it defaults to "Auto" mode.

---

apiVersion: "autoscaling.k8s.io/v1"

kind: VerticalPodAutoscaler

metadata:

spec:

# recommenders field can be unset when using the default recommender.

# When using an alternative recommender, the alternative recommender's name

# can be specified as the following in a list.

# recommenders:

# - name: 'alternative'

targetRef:

apiVersion: "apps/v1"

kind: Deployment

resourcePolicy:

containerPolicies:

- containerName: '*'

minAllowed:

cpu: 100m

memory: 50Mi

maxAllowed:

cpu: 1

memory: 500Mi

controlledResources: ["cpu", "memory"]

---

apiVersion: apps/v1

kind: Deployment

metadata:

spec:

selector:

matchLabels:

app: hamster

replicas: 2

template:

metadata:

labels:

app: hamster

spec:

securityContext:

runAsNonRoot: true

runAsUser: 65534 # nobody

containers:

- name: hamster

image: registry.k8s.io/ubuntu-slim:0.14

resources:

requests:

cpu: 100m

memory: 50Mi

command: ["/bin/sh"]

args:

- "-c"

- "while true; do timeout 0.5s yes >/dev/null; sleep 0.5s; done"

[root@operator-host vertical-pod-autoscaler]# kubectl apply -f examples/hamster.yaml && kubectl get vpa -w
verticalpodautoscaler.autoscaling.k8s.io/hamster-vpa created
deployment.apps/hamster created
NAME          MODE   CPU   MEM   PROVIDED   AGE
hamster-vpa   Auto                          2s

# 파드 리소스 Requestes 확인
kubectl describe pod | grep Requests: -A2
    Requests:
      cpu:        100m
      memory:     50Mi
--
    Requests:
      cpu:        587m
      memory:     262144k
--
    Requests:
      cpu:        587m
      memory:     262144k

# VPA에 의해 기존 파드 삭제되고 신규 파드가 생성됨
kubectl get events --sort-by=".metadata.creationTimestamp" | grep VPA
2m16s       Normal    EvictedByVPA             pod/hamster-5bccbb88c6-s6jkp         Pod was evicted by VPA Updater to apply resource recommendation.
76s         Normal    EvictedByVPA             pod/hamster-5bccbb88c6-jc6gq         Pod was evicted by VPA Updater to apply resource recommendation.

# 삭제
[root@operator-host vertical-pod-autoscaler]# kubectl delete -f examples/hamster.yaml && cd ~/autoscaler/vertical-pod-autoscaler/ && ./hack/vpa-down.sh
verticalpodautoscaler.autoscaling.k8s.io "hamster-vpa" deleted
deployment.apps "hamster" deleted
customresourcedefinition.apiextensions.k8s.io "verticalpodautoscalercheckpoints.autoscaling.k8s.io" deleted
customresourcedefinition.apiextensions.k8s.io "verticalpodautoscalers.autoscaling.k8s.io" deleted
clusterrole.rbac.authorization.k8s.io "system:metrics-reader" deleted
clusterrole.rbac.authorization.k8s.io "system:vpa-actor" deleted
clusterrole.rbac.authorization.k8s.io "system:vpa-status-actor" deleted
clusterrole.rbac.authorization.k8s.io "system:vpa-checkpoint-actor" deleted
clusterrole.rbac.authorization.k8s.io "system:evictioner" deleted
clusterrolebinding.rbac.authorization.k8s.io "system:metrics-reader" deleted
clusterrolebinding.rbac.authorization.k8s.io "system:vpa-actor" deleted
clusterrolebinding.rbac.authorization.k8s.io "system:vpa-status-actor" deleted
clusterrolebinding.rbac.authorization.k8s.io "system:vpa-checkpoint-actor" deleted
clusterrole.rbac.authorization.k8s.io "system:vpa-target-reader" deleted
clusterrolebinding.rbac.authorization.k8s.io "system:vpa-target-reader-binding" deleted
clusterrolebinding.rbac.authorization.k8s.io "system:vpa-evictioner-binding" deleted
serviceaccount "vpa-admission-controller" deleted
serviceaccount "vpa-recommender" deleted
serviceaccount "vpa-updater" deleted
clusterrole.rbac.authorization.k8s.io "system:vpa-admission-controller" deleted
clusterrolebinding.rbac.authorization.k8s.io "system:vpa-admission-controller" deleted
clusterrole.rbac.authorization.k8s.io "system:vpa-status-reader" deleted
clusterrolebinding.rbac.authorization.k8s.io "system:vpa-status-reader-binding" deleted
role.rbac.authorization.k8s.io "system:leader-locking-vpa-updater" deleted
rolebinding.rbac.authorization.k8s.io "system:leader-locking-vpa-updater" deleted
role.rbac.authorization.k8s.io "system:leader-locking-vpa-recommender" deleted
rolebinding.rbac.authorization.k8s.io "system:leader-locking-vpa-recommender" deleted
deployment.apps "vpa-updater" deleted
deployment.apps "vpa-recommender" deleted
Deleting VPA Admission Controller certs.
secret "vpa-tls-certs" deleted
Unregistering VPA admission controller webhook
Warning: deleting cluster-scoped resources, not scoped to the provided namespace
mutatingwebhookconfiguration.admissionregistration.k8s.io "vpa-webhook-config" deleted
deployment.apps "vpa-admission-controller" deleted
service "vpa-webhook" deleted
resource mapping not found for name: "verticalpodautoscalers.autoscaling.k8s.io" namespace: "" from "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first
resource mapping not found for name: "verticalpodautoscalercheckpoints.autoscaling.k8s.io" namespace: "" from "STDIN": no matches for kind "CustomResourceDefinition" in version "apiextensions.k8s.io/v1beta1"
ensure CRDs are installed first

EKS 삭제 (실습 리소스 정리)

# 클러스터 삭제
[root@operator-host ~]# eksctl delete cluster --name "${CLUSTER_NAME}"

# CloudFormation에서 myeks 스택 삭제

'Kubernetes' 카테고리의 다른 글

AEWS [3기] 7주차 - EKS Mode - Fargate (0)	2025.03.22
AEWS [3기] 6주차 - K8S API Server Client Certificate 인증 (0)	2025.03.15
AEWS [3기] 4주차 - OpenTelemetry (0)	2025.03.01
AEWS [3기] 3주차 - EKS Storage, Managed Node Groups (1)	2025.02.22
AEWS [3기] 2주차 - EKS Networking (iptables 와 NetFilter) (1)	2025.02.13

현재글AEWS [3기] 5주차 - EKS Autoscaling

yu3papa 님의 블로그

yu3papa 님의 블로그 입니다.

Today :
Yesterday :

yu3papa 님의 블로그