RKE2
Introduction | RKE2
RKE2 is Rancher's enterprise-ready next-generation Kubernetes distribution. It has also been known as RKE Government.
docs.rke2.io
RKE2 는 공식 문서상에 아래와 같이 정의하고 있습니다.
RKE2 is Rancher's enterprise-ready next-generation Kubernetes distribution. It has also been known as RKE Government.
위 정의에서 Goverment란 용어가 핵심이라고 생각합니다. 미국 연방정부 부분의 보안 및 규정 을 만족하는 K8S 배포판입니다.
특징
- Rancher 에서 개발한 엔터프라이즈 K8S 배포판, 미국 연방 정부 부문의 보안 및 규정 준수
- 운영자의 개입을 최소화하면서 클러스터가 CIS Kubernetes Benchmark v1.7 또는 v1.8 을 통과할 수 있도록 설정 및 구성 옵션을 제공합니다 .
- RKE2는 컨트롤 플레인 구성 요소를 kubelet이 관리하는 정적 Pod로 실행합니다. 내장 컨테이너 런타임은 containerd 입니다.
- 이 배포판은 독립 실행형으로 실행되거나 Rancher 에 통합되어 실행될 수 있습니다.
- RKE2는 경량 Kubernetes 배포판인 K3S를 개발하고 유지 관리하면서 얻은 교훈을 바탕으로, K3s의 사용 편의성을 유지하면서도 엔터프라이즈 환경에 적합한 배포판을 구축한 제품입니다.
- RKE2는 가장 간단하게 말하면 Kubernetes 클러스터에 참여할 모든 노드에 설치 및 구성해야 하는 단일 바이너리 파일입니다.
- 시작되면 RKE2는 각 노드의 역할에 맞는 에이전트를 부트스트랩하고 관리하는 동시에 네트워크에서 필요한 콘텐츠를 가져올 수 있습니다.
RKE2 구성요소는 아래와 같으며, K8S 상의 이미 검증된 오픈소스를 최대한 활용합니다.
| k3s | ||
| Helm Controller | GitOps 스타일 경량 배포기 - RKE2 클러스터가 올라올 때 필수 Helm 차트를 자동으로 설치·유지하는 내장 배포 컨트롤러 /var/lib/rancher/rke2/server/manifests 디렉토리에 (HelmChart) YAML 파일만 떨어뜨리면 Helm Controller에 의해 자동으로 Helm chart를 다운로드/설치/업데이트. |
|
| K8s | API Server Controller Manager Scheduler Proxy Kubelet |
|
| etcd runc containerd |
||
| CNI | Canal (Calico & Flannel) | 네트워크 연결은 Flannel, 보안 정책은 Calico가 담당하는 하이브리드 CNI canal DaemonSet 파드 내 컨테이너 : flannel (vxlan overlay 담당) , calico-node (network policy, iptables) |
| CoreDNS Ingress NGINX Controller and/or Traefik Metrics Server Helm |
실습환경 구성
Rocky Linux 9 버전의 2대의 머신을 Vagrant를 이용하여 구성합니다.
- Vagrant 관련 파일
더보기
- Vagrantfile
# Base Image https://portal.cloud.hashicorp.com/vagrant/discover/bento/rockylinux-9
BOX_IMAGE = "bento/rockylinux-9" # "bento/rockylinux-10.0"
BOX_VERSION = "202510.26.0"
N = 2 # max number of Node
Vagrant.configure("2") do |config|
# Nodes
(1..N).each do |i|
config.vm.define "k8s-node#{i}" do |subconfig|
subconfig.vm.box = BOX_IMAGE
subconfig.vm.box_version = BOX_VERSION
subconfig.vm.provider "virtualbox" do |vb|
vb.customize ["modifyvm", :id, "--groups", "/RKE2-Lab"]
vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
vb.name = "k8s-node#{i}"
vb.cpus = 4
vb.memory = 4096
vb.linked_clone = true
end
subconfig.vm.host_name = "k8s-node#{i}"
subconfig.vm.network "private_network", ip: "192.168.10.1#{i}"
subconfig.vm.network "forwarded_port", guest: 22, host: "6000#{i}", auto_correct: true, id: "ssh"
subconfig.vm.synced_folder "./", "/vagrant", disabled: true
subconfig.vm.provision "shell", path: "init_cfg.sh" , args: [ N ]
end
end
end
- init_cfg.sh
#!/usr/bin/env bash
echo ">>>> Initial Config Start <<<<"
echo "[TASK 1] Change Timezone and Enable NTP"
timedatectl set-local-rtc 0
timedatectl set-timezone Asia/Seoul
echo "[TASK 2] Disable firewalld and selinux"
systemctl disable --now firewalld >/dev/null 2>&1
setenforce 0
sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config
echo "[TASK 3] Disable and turn off SWAP & Delete swap partitions"
swapoff -a
sed -i '/swap/d' /etc/fstab
sfdisk --delete /dev/sda 2 >/dev/null 2>&1
partprobe /dev/sda >/dev/null 2>&1
echo "[TASK 4] Config kernel & module"
cat << EOF > /etc/modules-load.d/k8s.conf
overlay
br_netfilter
#vxlan
EOF
modprobe overlay >/dev/null 2>&1
modprobe br_netfilter >/dev/null 2>&1
#modprobe vxlan >/dev/null 2>&1
cat << EOF >/etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
sysctl --system >/dev/null 2>&1
echo "[TASK 5] Setting Local DNS Using Hosts file"
sed -i '/^127\.0\.\(1\|2\)\.1/d' /etc/hosts
for (( i=1; i<=$1; i++ )); do echo "192.168.10.1$i k8s-node$i" >> /etc/hosts; done
echo "[TASK 6] Install Helm"
curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | DESIRED_VERSION=v3.20.0 bash >/dev/null 2>&1
# echo "[TASK 6] Delete default routing - enp0s9 NIC" # setenforce 0 설정 필요
# nmcli connection modify enp0s9 ipv4.never-default yes
# nmcli connection up enp0s9 >/dev/null 2>&1
echo "[TASK 7] Setting SSHD"
#echo "root:qwe123" | chpasswd
cat << EOF >> /etc/ssh/sshd_config
PermitRootLogin yes
PasswordAuthentication yes
EOF
systemctl restart sshd >/dev/null 2>&1
echo "[TASK 8] Install packages"
dnf install -y conntrack python3-pip git >/dev/null 2>&1
echo "[TASK 9] NetworkManager to ignore calico/flannel related network interfaces"
# https://docs.rke2.io/known_issues#networkmanager
cat << EOF > /etc/NetworkManager/conf.d/k8s.conf
[keyfile]
unmanaged-devices=interface-name:flannel*;interface-name:cali*;interface-name:tunl*;interface-name:vxlan.calico;interface-name:vxlan-v6.calico;interface-name:wireguard.cali;interface-name:wg-v6.cali
EOF
systemctl reload NetworkManager
echo "[TASK 11] Install K9s"
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
wget -P /tmp https://github.com/derailed/k9s/releases/latest/download/k9s_linux_${CLI_ARCH}.tar.gz >/dev/null 2>&1
tar -xzf /tmp/k9s_linux_${CLI_ARCH}.tar.gz -C /tmp
chown root:root /tmp/k9s
mv /tmp/k9s /usr/local/bin/
chmod +x /usr/local/bin/k9s
echo "[TASK 12] ETC"
echo "sudo su -" >> /home/vagrant/.bashrc
echo ">>>> Initial Config End <<<<"
RKE2 설치
RKE2 서버 노드 설치
RKE2 는 systemd 기반 시스템에 서비스로 편리하게 설치할 수 있는 설치 스크립트를 제공(https://get.rke2.io/) 합니다.
설치에 적용할 수 있는 아래 환경변수를 이용하면 커스터마이징이 가능합니다. 환경변수 설정이 없으며 디폴트 값으로 설치됩니다.
| INSTALL_RKE2_VERSION | GitHub에서 다운로드할 RKE2 버전 - stable(기본값) |
| INSTALL_RKE2_TYPE | 생성할 systemd 서비스의 유형 - server(기본값), agent |
| NSTALL_RKE2_CHANNEL_URL | RKE2 다운로드 URL을 가져오기 위한 채널 URL - https://update.rke2.io/v1-release/channels (기본값) |
| INSTALL_RKE2_CHANNEL | RKE2 다운로드 URL을 가져오는 데 사용할 채널 - stable(기본값), latest, testing |
| INSTALL_RKE2_METHOD | rpm(RPM 기반 시스템 경우 기본값), tar(그외 경우 기본값) |
- k8s-node1 에서 RKE2 서버노드 설치를 진행합니다.
| # systemd 기반 시스템에 서비스로 편리하게 설치할 수 있는 설치 스크립트를 제공 : 서비스와 바이너리 파일이 컴퓨터에 설치 # https://docs.rke2.io/install/methods curl -sfL https://get.rke2.io --output install.sh chmod +x install.sh # 실제 K8S 클러스터를 올리는게 아니라, 필요한 실행파일, 디렉토리 구조, Repo 구성을 진행합니다. INSTALL_RKE2_CHANNEL=v1.33 ./install.sh [INFO] using stable RPM repositories [INFO] using 1.33 series from channel stable ...(생략)... Installed: rke2-common-1.33.7~rke2r3-0.el9.aarch64 rke2-selinux-0.22-1.el9.noarch rke2-server-1.33.7~rke2r3-0.el9.aa # rke2 버전 확인 rke2 --version rke2 version v1.33.7+rke2r3 (7e4fd1a82edf497cab91c220144619bbad659cf4) go version go1.24.11 X:boringcrypto # repo 추가 확인 dnf repolist repo id repo name appstream Rocky Linux 9 - AppStream baseos Rocky Linux 9 - BaseOS extras Rocky Linux 9 - Extras rancher-rke2-1.33-stable Rancher RKE2 1.33 (v1.33) rancher-rke2-common-stable Rancher RKE2 Common (v1.33) tree /etc/yum.repos.d/ /etc/yum.repos.d/ ├── rancher-rke2.repo ├── rocky-addons.repo ├── rocky-devel.repo ├── rocky-extras.repo └── rocky.repo cat /etc/yum.repos.d/rancher-rke2.repo [rancher-rke2-common-stable] name=Rancher RKE2 Common (v1.33) baseurl=https://rpm.rancher.io/rke2/stable/common/centos/9/noarch enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://rpm.rancher.io/public.key [rancher-rke2-1.33-stable] name=Rancher RKE2 1.33 (v1.33) baseurl=https://rpm.rancher.io/rke2/stable/1.33/centos/9/aarch64 enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://rpm.rancher.io/public.key # 디렉터리 생성 확인 tree /etc/rancher/ /etc/rancher/ └── rke2 tree /var/lib/rancher/ /var/lib/rancher/ └── rke2 ├── agent │ ├── containerd │ │ └── io.containerd.snapshotter.v1.overlayfs │ │ └── snapshots │ └── logs ├── data └── server # rke2 명령 확인 # https://docs.rke2.io/install/configuration#running-the-binary-directly # rke2 server : Run the RKE2 management server, which will also launch the Kubernetes control plane components such as the API server, controller-manager, and scheduler. Only Supported on Linux. # rke2 agent : Run the RKE2 node agent. This will cause RKE2 to run as a worker node, launching the Kubernetes node services kubelet and kube-proxy. Supported on Linux and Windows. rke2 --h COMMANDS: server Run management server agent Run node agent # RKE2 설정 : cni 플러그인(canal) 등 # https://docs.rke2.io/install/configuration # https://docs.rke2.io/advanced cat << EOF > /etc/rancher/rke2/config.yaml write-kubeconfig-mode: "0644"
debug: true
cni: canal
bind-address: 192.168.10.11
advertise-address: 192.168.10.11
node-ip: 192.168.10.11
disable-cloud-controller: true
disable:
- servicelb
- rke2-coredns-autoscaler
- rke2-ingress-nginx
- rke2-snapshot-controller
- rke2-snapshot-controller-crd
- rke2-snapshot-validation-webhook
# canal cni 플러그인 helm chart values 파일 작성 # https://docs.rke2.io/networking/basic_network_options # https://github.com/rancher/rke2-charts/blob/main-source/packages/rke2-canal/charts/values.yaml mkdir -p /var/lib/rancher/rke2/server/manifests/ cat << EOF > /var/lib/rancher/rke2/server/manifests/rke2-canal-config.yaml kind: HelmChartConfig
metadata:
name: rke2-canal
namespace: kube-system
spec:
valuesContent: |-
flannel:
iface: "enp0s8"
# coredns 의 autoscaler 미설치를 위한 helm chart values 파일 작성 # https://docs.rke2.io/add-ons/helm#customizing-packaged-components-with-helmchartconfig # https://github.com/rancher/rke2-charts/tree/main/charts/rke2-coredns/rke2-coredns/1.45.200 cat << EOF > /var/lib/rancher/rke2/server/manifests/rke2-coredns-config.yaml kind: HelmChartConfig
metadata:
name: rke2-coredns
namespace: kube-system
spec:
valuesContent: |-
autoscaler:
enabled: false
# RKE2 시작 : 2분 정도 소요 -> coredns 파드까지 정상화 대략 1~2분 추가 소요 systemctl enable --now rke2-server.service systemctl status rke2-server --no-pager ● rke2-server.service - Rancher Kubernetes Engine v2 (server) Loaded: loaded (/usr/lib/systemd/system/rke2-server.service; enabled; preset: disabled) Active: active (running) since Wed 2026-02-18 13:06:23 KST; 25min ago Docs: https://github.com/rancher/rke2#readme Process: 6931 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS) Process: 6932 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS) Main PID: 6933 (rke2) Tasks: 162 Memory: 2.1G CPU: 5min 17.531s CGroup: /system.slice/rke2-server.service ├─ 6933 "/usr/bin/rke2 server" ├─ 6954 containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml ├─ 7018 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --config-dir=/var/lib/rancher/rke2/ag… ├─ 7072 /var/lib/rancher/rke2/data/v1.33.8-rke2r1-1b2872361ec5/bin/containerd-shim-runc-v2 -namespace k8s.io -id c8191abea8c642f41e03e6a5afd743157a2b62… ├─ 7077 /var/lib/rancher/rke2/data/v1.33.8-rke2r1-1b2872361ec5/bin/containerd-shim-runc-v2 -namespace k8s.io -id 7a6b2e479dd5125c2398ad241abc1d669e4723… ├─ 7292 /var/lib/rancher/rke2/data/v1.33.8-rke2r1-1b2872361ec5/bin/containerd-shim-runc-v2 -namespace k8s.io -id 64a8a94a463ec77ae7e80b6a18f82514362e3e… ├─ 7376 /var/lib/rancher/rke2/data/v1.33.8-rke2r1-1b2872361ec5/bin/containerd-shim-runc-v2 -namespace k8s.io -id 393d8367a845629f737d5728a98d179094ed95… ├─ 7394 /var/lib/rancher/rke2/data/v1.33.8-rke2r1-1b2872361ec5/bin/containerd-shim-runc-v2 -namespace k8s.io -id c9ee6d3c4d425f649f999b33c7707c9e947d61… ├─ 8071 /var/lib/rancher/rke2/data/v1.33.8-rke2r1-1b2872361ec5/bin/containerd-shim-runc-v2 -namespace k8s.io -id c6816d83b0816a22324fa1dd4afedc4acfb23e… ├─ 9625 /var/lib/rancher/rke2/data/v1.33.8-rke2r1-1b2872361ec5/bin/containerd-shim-runc-v2 -namespace k8s.io -id fb6e36730fc3e84d053418e300161adde2dd6e… ├─ 9652 /var/lib/rancher/rke2/data/v1.33.8-rke2r1-1b2872361ec5/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4d8761f0eff2e77e9de3dacb19d03f95e757d9… └─10039 /var/lib/rancher/rke2/data/v1.33.8-rke2r1-1b2872361ec5/bin/containerd-shim-runc-v2 -namespace k8s.io -id aea63f507d68b9942bc602ddabe0dbe35ad4c3… #(참고) RKE는 하위 프로세스들이 실행되면, systemd가 Parent로 되게 설정함 https://tech-recipe.tistory.com/52  # 프로세스 확인 pstree -a | grep -v color | grep 'rke2$' -A5 |-rke2 | |-containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml | | `-14*[{containerd}] | |-kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s... | | `-16*[{kubelet}] | `-11*[{rke2}] pstree -a | grep -v color | grep 'containerd-shim ' -A2 |-containerd-shim -namespace k8s.io -idc8191abea8c642f41e03e6a5afd | |-kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s... | | `-7*[{kube-proxy}] -- |-containerd-shim -namespace k8s.io -id7a6b2e479dd5125c2398ad241ab | |-etcd --config-file=/var/lib/rancher/rke2/server/db/etcd/config | | `-11*[{etcd}] -- |-containerd-shim -namespace k8s.io -id64a8a94a463ec77ae7e80b6a18f | |-kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --advertise-address=192.168.10.11... | | `-11*[{kube-apiserver}] -- |-containerd-shim -namespace k8s.io -id393d8367a845629f737d5728a98 | |-kube-controller --permit-port-sharing=true --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins--terminated-pod-gc-thres | | `-6*[{kube-controller}] -- |-containerd-shim -namespace k8s.io -idc9ee6d3c4d425f649f999b33c77 | |-kube-scheduler --permit-port-sharing=true ... | | `-8*[{kube-scheduler}] -- |-containerd-shim -namespace k8s.io -idc6816d83b0816a22324fa1dd4af | |-flanneld --ip-masq --kube-subnet-mgr --iptables-forward-rules=false --ip-blackhole-route | | |-(timeout) -- |-containerd-shim -namespace k8s.io -idfb6e36730fc3e84d053418e3001 | |-coredns -conf /etc/coredns/Corefile | | `-7*[{coredns}] -- |-containerd-shim -namespace k8s.io -id4d8761f0eff2e77e9de3dacb19d | |-cluster-proport --namespace=kube-system --configmap=rke2-coredns-rke2-coredns-autoscaler--target=Deployment/rke2-c | | `-6*[{cluster-proport}] -- |-containerd-shim -namespace k8s.io -idaea63f507d68b9942bc602ddabe | |-metrics-server --secure-port=10250 --cert-dir=/tmp --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname... | | `-8*[{metrics-server}] # 자격증명 파일 복사 mkdir ~/.kube cp /etc/rancher/rke2/rke2.yaml ~/.kube/config # /etc/rancher 디렉터리 확인 tree /etc/rancher/ /etc/rancher/ ├── node │ └── password └── rke2 ├── config.yaml ├── rke2-pss.yaml └── rke2.yaml 2 directories, 4 files cat /etc/rancher/node/password 08ce9be15deb8c6c10726f30c20781e8 cat /etc/rancher/rke2/config.yaml write-kubeconfig-mode: "0644" debug: true cni: canal bind-address: 192.168.10.11 advertise-address: 192.168.10.11 node-ip: 192.168.10.11 disable-cloud-controller: true disable: - servicelb - rke2-coredns-autoscaler - rke2-ingress-nginx - rke2-snapshot-controller - rke2-snapshot-controller-crd - rke2-snapshot-validation-webhook cat /etc/rancher/rke2/rke2-pss.yaml apiVersion: apiserver.config.k8s.io/v1 kind: AdmissionConfiguration plugins: - name: PodSecurity configuration: apiVersion: pod-security.admission.config.k8s.io/v1beta1 kind: PodSecurityConfiguration defaults: enforce: "privileged" enforce-version: "latest" exemptions: usernames: [] runtimeClasses: [] namespaces: [] # 바이너리 파일 확인 tree /var/lib/rancher/rke2/bin/ /var/lib/rancher/rke2/bin/ ├── containerd ├── containerd-shim-runc-v2 ├── crictl ├── ctr ├── kubectl ├── kubelet └── runc # PATH 안 건드리고 표준 위치로 바이너리 노출 설정 : 심볼릭 링크 방식 ln -s /var/lib/rancher/rke2/bin/containerd /usr/local/bin/containerd ln -s /var/lib/rancher/rke2/bin/kubectl /usr/local/bin/kubectl ln -s /var/lib/rancher/rke2/bin/crictl /usr/local/bin/crictl ln -s /var/lib/rancher/rke2/bin/runc /usr/local/bin/runc ln -s /var/lib/rancher/rke2/bin/ctr /usr/local/bin/ctr ln -s /var/lib/rancher/rke2/agent/etc/crictl.yaml /etc/crictl.yaml runc --version runc version 1.4.0 commit: v1.4.0-0-g8bd78a99 spec: 1.3.0 go: go1.24.11 X:boringcrypto libseccomp: 2.5.4 containerd --version containerd github.com/k3s-io/containerd v2.1.5-k3s1 e77c15f30e5162d6abab671b0d74ca2243e2916e kubectl version Client Version: v1.33.8+rke2r1 Kustomize Version: v5.6.0 Server Version: v1.33.8+rke2r1 # 편의성 설정 source <(kubectl completion bash) alias k=kubectl complete -F __start_kubectl k echo 'source <(kubectl completion bash)' >> /etc/profile echo 'alias k=kubectl' >> /etc/profile echo 'complete -F __start_kubectl k' >> /etc/profile # 확인 kubectl cluster-info -v=6 I0218 13:12:41.348931 14784 loader.go:402] Config loaded from file: /root/.kube/config I0218 13:12:41.350585 14784 envvar.go:172] "Feature gate default state" feature="WatchListClient" enabled=false I0218 13:12:41.350835 14784 envvar.go:172] "Feature gate default state" feature="ClientsAllowCBOR" enabled=false I0218 13:12:41.352096 14784 envvar.go:172] "Feature gate default state" feature="ClientsPreferCBOR" enabled=false I0218 13:12:41.352123 14784 envvar.go:172] "Feature gate default state" feature="InformerResourceVersion" enabled=false I0218 13:12:41.352131 14784 envvar.go:172] "Feature gate default state" feature="InOrderInformers" enabled=true I0218 13:12:41.364128 14784 round_trippers.go:632] "Response" verb="GET" url="https://192.168.10.11:6443/api?timeout=32s" status="200 OK" milliseconds=11 I0218 13:12:41.369158 14784 round_trippers.go:632] "Response" verb="GET" url="https://192.168.10.11:6443/apis?timeout=32s" status="200 OK" milliseconds=2 I0218 13:12:41.393121 14784 round_trippers.go:632] "Response" verb="GET" url="https://192.168.10.11:6443/api/v1/namespaces/kube-system/services?labelSelector=kubernetes.io%2Fcluster-service%3Dtrue" status="200 OK" milliseconds=7 Kubernetes control plane is running at https://192.168.10.11:6443 CoreDNS is running at https://192.168.10.11:6443/api/v1/namespaces/kube-system/services/rke2-coredns-rke2-coredns:udp-53/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. # 노드, 파드 정보 확인 kubectl get node -owide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME k8s-node1 Ready control-plane,etcd,master 6m50s v1.33.8+rke2r1 192.168.10.11 <none> Rocky Linux 9.6 (Blue Onyx) 5.14.0-570.52.1.el9_6.x86_64 containerd://2.1.5-k3s1 helm list -A NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION rke2-canal kube-system 1 2026-02-18 04:06:42.787680481 +0000 UTC deployed rke2-canal-v3.31.3-build2026020600 v3.31.3 rke2-coredns kube-system 1 2026-02-18 04:06:42.83878489 +0000 UTC deployed rke2-coredns-1.45.201 1.13.1 rke2-metrics-server kube-system 1 2026-02-18 04:07:35.768630923 +0000 UTC deployed rke2-metrics-server-3.13.007 0.8.0 rke2-runtimeclasses kube-system 1 2026-02-18 04:07:35.775136792 +0000 UTC deployed rke2-runtimeclasses-0.1.000 0.1.0 kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system etcd-k8s-node1 1/1 Running 0 7m26s kube-system helm-install-rke2-canal-7nlz9 0/1 Completed 0 7m23s kube-system helm-install-rke2-coredns-2769x 0/1 Completed 0 7m23s kube-system helm-install-rke2-metrics-server-9lcss 0/1 Completed 0 7m23s kube-system helm-install-rke2-runtimeclasses-5xrzr 0/1 Completed 0 7m23s kube-system kube-apiserver-k8s-node1 1/1 Running 0 7m26s kube-system kube-controller-manager-k8s-node1 1/1 Running 0 7m26s kube-system kube-proxy-k8s-node1 1/1 Running 0 7m26s kube-system kube-scheduler-k8s-node1 1/1 Running 0 7m26s kube-system rke2-canal-ptx7b 2/2 Running 0 7m3s kube-system rke2-coredns-rke2-coredns-559595db99-fs27b 1/1 Running 0 7m6s kube-system rke2-coredns-rke2-coredns-autoscaler-8c8d5f4f4-ll2f9 1/1 Running 0 7m6s kube-system rke2-metrics-server-fdcdf575d-42nmb 1/1 Running 0 6m13s |
워커 노드 추가
k8s-node2를 클러스터에 워커노드로 Join 하는 실습을 진행합니다.
| [k8s-node1] # control-plane인 k8s-node1 에서 TOKEN 값을 확인 cat /var/lib/rancher/rke2/server/node-token K10d37597051b8b6a68d072f42350af53c2ea977b253ca435263c87a517002ca756::server:d7bb934b9778853e373e8bf78fcf1aa9 # 노드(서버/에이전트)가 RKE2 클러스터에 조인할 때 사용하는 전용 관리/부트스트랩 API 포트 확인 ss -tnlp | grep 9345 LISTEN 0 4096 192.168.10.11:9345 0.0.0.0:* users:(("rke2",pid=6348,fd=6)) # 아래는 k8s-node2 에서 수행 [k8s-node2] # Run the installer curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE="agent" INSTALL_RKE2_CHANNEL=v1.33 sh - # Configure the rke2-agent service : rke-agent 설정 파일 작성 ## The rke2 server process listens on port 9345 for new nodes to register. TOKEN=K10d37597051b8b6a68d072f42350af53c2ea977b253ca435263c87a517002ca756::server:d7bb934b9778853e373e8bf78fcf1aa9 mkdir -p /etc/rancher/rke2/ cat << EOF > /etc/rancher/rke2/config.yaml server: https://192.168.10.11:9345 token: $TOKEN EOF # Enabled/Start the service systemctl enable --now rke2-agent.service [k8s-node1] # JOIN 이 완료되면 control-plane인 k8s-node1 에서 확인 kubectl get no NAME STATUS ROLES AGE VERSION k8s-node1 Ready control-plane,etcd,master 63m v1.33.8+rke2r1 k8s-node2 Ready <none> 2m9s v1.33.8+rke2r1 kubectl get po -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system etcd-k8s-node1 1/1 Running 0 63m 192.168.10.11 k8s-node1 <none> <none> kube-system helm-install-rke2-canal-7nlz9 0/1 Completed 0 63m 192.168.10.11 k8s-node1 <none> <none> kube-system helm-install-rke2-coredns-2769x 0/1 Completed 0 63m 192.168.10.11 k8s-node1 <none> <none> kube-system helm-install-rke2-metrics-server-9lcss 0/1 Completed 0 63m 10.42.0.4 k8s-node1 <none> <none> kube-system helm-install-rke2-runtimeclasses-5xrzr 0/1 Completed 0 63m 10.42.0.3 k8s-node1 <none> <none> kube-system kube-apiserver-k8s-node1 1/1 Running 0 63m 192.168.10.11 k8s-node1 <none> <none> kube-system kube-controller-manager-k8s-node1 1/1 Running 0 63m 192.168.10.11 k8s-node1 <none> <none> kube-system kube-proxy-k8s-node1 1/1 Running 0 63m 192.168.10.11 k8s-node1 <none> <none> kube-system kube-proxy-k8s-node2 1/1 Running 0 2m26s 192.168.10.12 k8s-node2 <none> <none> kube-system kube-scheduler-k8s-node1 1/1 Running 0 63m 192.168.10.11 k8s-node1 <none> <none> kube-system rke2-canal-7pjs5 2/2 Running 0 2m26s 192.168.10.12 k8s-node2 <none> <none> kube-system rke2-canal-ptx7b 2/2 Running 0 63m 192.168.10.11 k8s-node1 <none> <none> kube-system rke2-coredns-rke2-coredns-559595db99-fs27b 1/1 Running 0 63m 10.42.0.2 k8s-node1 <none> <none> kube-system rke2-coredns-rke2-coredns-559595db99-vmtv2 1/1 Running 0 89s 10.42.1.2 k8s-node2 <none> <none> kube-system rke2-coredns-rke2-coredns-autoscaler-8c8d5f4f4-ll2f9 1/1 Running 0 63m 10.42.0.5 k8s-node1 <none> <none> kube-system rke2-metrics-server-fdcdf575d-42nmb 1/1 Running 0 62m 10.42.0.6 k8s-node1 <none> <none> |
샘플 애플리케이션 배포
RKE2 에 샘플 어플리케이션을 배포하고 동작 확인
| # 샘플 애플리케이션 배포 cat << EOF | kubectl apply -f - apiVersion: apps/v1 kind: Deployment metadata: name: webpod spec: replicas: 2 selector: matchLabels: app: webpod template: metadata: labels: app: webpod spec: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - sample-app topologyKey: "kubernetes.io/hostname" containers: - name: webpod image: traefik/whoami ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: webpod labels: app: webpod spec: selector: app: webpod ports: - protocol: TCP port: 80 targetPort: 80 nodePort: 30000 type: NodePort EOF # 확인 kubectl get deploy,pod,svc,ep -owide NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR deployment.apps/webpod 2/2 2 2 13s webpod traefik/whoami app=webpod NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES pod/webpod-697b545f57-cvkwx 1/1 Running 0 13s 10.42.1.3 k8s-node2 <none> <none> pod/webpod-697b545f57-j58n2 1/1 Running 0 13s 10.42.0.7 k8s-node1 <none> <none> NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR service/kubernetes ClusterIP 10.43.0.1 <none> 443/TCP 66m <none> service/webpod NodePort 10.43.77.240 <none> 80:30000/TCP 13s app=webpod NAME ENDPOINTS AGE endpoints/kubernetes 192.168.10.11:6443 66m endpoints/webpod 10.42.0.7:80,10.42.1.3:80 12s # [HostPC] 반복 호출 : IP는 node 작업에 따라 변경 while true; do curl -s http://192.168.10.12:30000 | grep Hostname; date; sleep 1; done Hostname: webpod-697b545f57-cvkwx Wed Feb 18 14:14:32 2026 Hostname: webpod-697b545f57-cvkwx Wed Feb 18 14:14:33 2026 ...(생략)... |
실습 리소스 삭제
| # 실습 종료후 Vagrant 로 생성한 vm 삭제 vagrant destroy -f |
Cluster API
Cluster API는 Kubernetes 클러스터를 프로비저닝, 업그레이드 및 운영하기 위한 선언적 API 및 도구를 제공하는 Kubernetes 하위 프로젝트입니다.
https://cluster-api.sigs.k8s.io/introduction
Introduction - The Cluster API Book
Cluster API is a Kubernetes sub-project focused on providing declarative APIs and tooling to simplify provisioning, upgrading, and operating multiple Kubernetes clusters. Started by the Kubernetes Special Interest Group (SIG) Cluster Lifecycle, the Cluster
cluster-api.sigs.k8s.io
- Kubernetes SIG(Special Interest Group) 클러스터 라이프사이클 에서 시작된 클러스터 API 프로젝트는 Kubernetes 스타일의 API와 패턴을 사용하여 플랫폼 운영자를 위한 클러스터 라이프사이클 관리를 자동화합니다.
- 가상 머신, 네트워크, 로드 밸런서, VPC와 같은 지원 인프라와 Kubernetes 클러스터 구성은 애플리케이션 개발자가 워크로드를 배포하고 관리하는 방식과 동일하게 정의됩니다. 이를 통해 다양한 인프라 환경에서 일관되고 반복 가능한 클러스터 배포가 가능합니다.
위 그림을 보면 k8s 클러스터를 찍어내듯이 새롭게 구성하는 모습을 형상화하였습니다. 일반적인 환경의 업체보다는 "K8S as a Service" 가 필요한 플랫폼 기업에서 사용하면 효과적이라고 생각됩니다.
Cluster API 프로젝트 목표
- 선언적 API를 사용하여 Kubernetes 표준을 준수하는 클러스터의 수명 주기(생성, 확장, 업그레이드, 삭제)를 관리합니다.
- 온프레미스 환경과 클라우드 환경 모두에서 작업할 수 있어야 합니다.
- 공통 연산을 정의하고, 기본 구현을 제공하며, 다른 구현으로 교체할 수 있는 기능을 제공합니다.
- 기존 생태계 구성 요소의 기능을 중복해서 개발하는 대신 재사용 및 통합합니다(예: 노드 문제 감지기, 클러스터 자동 확장기, SIG-Multi-cluster).
- Kubernetes 라이프사이클 제품이 Cluster API를 점진적으로 채택할 수 있는 전환 경로를 제공합니다.
- 특히, 기존의 클러스터 라이프사이클 관리 도구는 여러 릴리스에 걸쳐 단계적으로 Cluster API를 채택하거나 클러스터 API의 하위 집합을 채택할 수 있어야 합니다.
Cluster API 실습
① kind k8s에 관리용 Management 클러스터 설치 init
https://cluster-api.sigs.k8s.io/clusterctl/commands/init
| # 작업 디렉터리 생성 mkdir capi-docker && cd capi-docker docker context ls NAME DESCRIPTION DOCKER ENDPOINT ERROR default * Current DOCKER_HOST based configuration unix:///var/run/docker.sock # kind를 이용하여 k8s 설치 kind create cluster --name myk8s --image kindest/node:v1.35.0 --config - <<EOF kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
extraMounts:
- hostPath: /var/run/docker.sock
containerPath: /var/run/docker.sock
extraPortMappings:
- containerPort: 30000 # sample
hostPort: 30000
- containerPort: 30001 # kube-ops-view
hostPort: 30001
# (옵션) kube-ops-view helm repo add geek-cookbook https://geek-cookbook.github.io/charts/ helm install kube-ops-view geek-cookbook/kube-ops-view --version 1.2.2 \ --set service.main.type=NodePort,service.main.ports.http.nodePort=30001 \ --set env.TZ="Asia/Seoul" --namespace kube-system # open "http://127.0.0.1:30001/#scale=1.5"  # Install clusterctl curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.12.2/clusterctl-linux-amd64 -o clusterctl sudo install -o root -g root -m 0755 clusterctl /usr/local/bin/clusterctl ## 버전 확인 clusterctl version -o json | jq {
"clusterctl": {
"major": "1",
"minor": "12",
"gitVersion": "v1.12.2",
"gitCommit": "9ee80d1ff529c48ae0b8e022ec01d70ac496e8e5",
"gitTreeState": "clean",
"buildDate": "2026-01-20T16:48:19Z",
"goVersion": "go1.24.12",
"compiler": "gc",
"platform": "linux/amd64"
}
}
# [Docker 프로바이더] Initialize the management cluster : 현재 k8s 를 관리 클러스터로 변환 ## Docker 프로바이더는 프로덕션 환경에 사용하도록 설계되지 않았으며 개발 환경 전용 ## ClusterTopology관리형 토폴로지 및 ClusterClass 지원을 활성화하는 데 필요한 기능은 다음과 같이 활성화 ## https://cluster-api.sigs.k8s.io/tasks/experimental-features/experimental-features export CLUSTER_TOPOLOGY=true # Enable the experimental Cluster topology feature clusterctl init --infrastructure docker # Initialize the management cluster Fetching providers Installing cert-manager version="v1.19.1" Waiting for cert-manager to be available... spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default value changed from `Never` to `Always`. Installing provider="cluster-api" version="v1.12.3" targetNamespace="capi-system" spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default value changed from `Never` to `Always`. Installing provider="bootstrap-kubeadm" version="v1.12.3" targetNamespace="capi-kubeadm-bootstrap-system" spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default value changed from `Never` to `Always`. Installing provider="control-plane-kubeadm" version="v1.12.3" targetNamespace="capi-kubeadm-control-plane-system" spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default value changed from `Never` to `Always`. Installing provider="infrastructure-docker" version="v1.12.3" targetNamespace="capd-system" spec.privateKey.rotationPolicy: In cert-manager >= v1.18.0, the default value changed from `Never` to `Always`. Your management cluster has been initialized successfully! You can now create your first workload cluster by running the following: clusterctl generate cluster [name] --kubernetes-version [version] | kubectl apply -f - New clusterctl version available: v1.12.2 -> v1.12.3 sigs.k8s.io/cluster-api # 확인 ## 관련 crd 설치 kubectl get crd | grep x-k8s clusterclasses.cluster.x-k8s.io 2026-02-18T06:53:00Z clusterresourcesetbindings.addons.cluster.x-k8s.io 2026-02-18T06:53:00Z clusterresourcesets.addons.cluster.x-k8s.io 2026-02-18T06:53:00Z clusters.cluster.x-k8s.io 2026-02-18T06:53:01Z devclusters.infrastructure.cluster.x-k8s.io 2026-02-18T06:53:04Z devclustertemplates.infrastructure.cluster.x-k8s.io 2026-02-18T06:53:04Z devmachines.infrastructure.cluster.x-k8s.io 2026-02-18T06:53:04Z devmachinetemplates.infrastructure.cluster.x-k8s.io 2026-02-18T06:53:04Z dockerclusters.infrastructure.cluster.x-k8s.io 2026-02-18T06:53:04Z dockerclustertemplates.infrastructure.cluster.x-k8s.io 2026-02-18T06:53:04Z dockermachinepools.infrastructure.cluster.x-k8s.io 2026-02-18T06:53:04Z dockermachinepooltemplates.infrastructure.cluster.x-k8s.io 2026-02-18T06:53:04Z dockermachines.infrastructure.cluster.x-k8s.io 2026-02-18T06:53:04Z dockermachinetemplates.infrastructure.cluster.x-k8s.io 2026-02-18T06:53:04Z extensionconfigs.runtime.cluster.x-k8s.io 2026-02-18T06:53:01Z ipaddressclaims.ipahttp://m.cluster.x-k8s.io 2026-02-18T06:53:01Z ipaddresses.ipahttp://m.cluster.x-k8s.io 2026-02-18T06:53:01Z kubeadmconfigs.bootstrap.cluster.x-k8s.io 2026-02-18T06:53:02Z kubeadmconfigtemplates.bootstrap.cluster.x-k8s.io 2026-02-18T06:53:02Z kubeadmcontrolplanes.controlplane.cluster.x-k8s.io 2026-02-18T06:53:03Z kubeadmcontrolplanetemplates.controlplane.cluster.x-k8s.io 2026-02-18T06:53:03Z machinedeployments.cluster.x-k8s.io 2026-02-18T06:53:01Z machinedrainrules.cluster.x-k8s.io 2026-02-18T06:53:01Z machinehealthchecks.cluster.x-k8s.io 2026-02-18T06:53:01Z machinepools.cluster.x-k8s.io 2026-02-18T06:53:01Z machines.cluster.x-k8s.io 2026-02-18T06:53:01Z machinesets.cluster.x-k8s.io 2026-02-18T06:53:01Z providers.clusterctl.cluster.x-k8s.io 2026-02-18T06:52:20Z ## capd-system, capi-(kueadm-X/Y, system), cert-manager 네임스페이스가 생성 kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE capd-system capd-controller-manager-54755cdd6-cjmsf 1/1 Running 0 89s capi-kubeadm-bootstrap-system capi-kubeadm-bootstrap-controller-manager-94d8964d9-v6hs2 1/1 Running 0 91s capi-kubeadm-control-plane-system capi-kubeadm-control-plane-controller-manager-6796744c76-42d8c 1/1 Running 0 90s capi-system capi-controller-manager-59c5798655-m9bjr 1/1 Running 0 92s cert-manager cert-manager-598d877b78-pvt4k 1/1 Running 0 2m8s cert-manager cert-manager-cainjector-6b5777d564-4gsnt 1/1 Running 0 2m8s cert-manager cert-manager-webhook-5d9fc6b4ff-v4gk5 1/1 Running 0 2m8s kube-system coredns-7d764666f9-4rn5m 1/1 Running 0 5m33s kube-system coredns-7d764666f9-hsr92 1/1 Running 0 5m33s kube-system etcd-myk8s-control-plane 1/1 Running 0 5m39s kube-system kindnet-59gbn 1/1 Running 0 5m33s kube-system kube-apiserver-myk8s-control-plane 1/1 Running 0 5m39s kube-system kube-controller-manager-myk8s-control-plane 1/1 Running 0 5m39s kube-system kube-ops-view-5c64986f74-5lmd2 1/1 Running 0 3m24s kube-system kube-proxy-49jsl 1/1 Running 0 5m33s kube-system kube-scheduler-myk8s-control-plane 1/1 Running 0 5m39s local-path-storage local-path-provisioner-67b8995b4b-9jzvv 1/1 Running 0 5m33s |
② 관리용 Management K8S 클러스터 정보 확인
| ## Enabling Experimental Features 정보 확인 : ClusterTopology=true , InPlaceUpdates=false 등 kubectl describe -n capi-system deployment.apps/capi-controller-manager | grep feature-gates --feature-gates=MachinePool=true,ClusterTopology=true,RuntimeSDK=false,MachineSetPreflightChecks=true,MachineWaitForVolumeDetachConsiderVolumeAttachments=true,PriorityQueue=false,ReconcilerRateLimiting=false,InPlaceUpdates=false,MachineTaintPropagation=false # 프로바이더 (타입별)확인 : CAPI 구성요소가 설치된 상태 kubectl get providers.clusterctl.cluster.x-k8s.io -A NAMESPACE NAME AGE TYPE PROVIDER VERSION capd-system infrastructure-docker 4m56s InfrastructureProvider docker v1.12.3 capi-kubeadm-bootstrap-system bootstrap-kubeadm 4m58s BootstrapProvider kubeadm v1.12.3 capi-kubeadm-control-plane-system control-plane-kubeadm 4m57s ControlPlaneProvider kubeadm v1.12.3 capi-system cluster-api 4m59s CoreProvider cluster-api v1.12.3 # CAPI의 핵심 컨트롤러 집합 : Cluster / MachineDeployment / MachineSet / Machine CRD 관리, 전체 reconcile orchestration 담당 kubectl get providers -n capi-system cluster-api -o yaml apiVersion: clusterctl.cluster.x-k8s.io/v1alpha3
kind: Provider
metadata:
creationTimestamp: "2026-02-18T06:53:02Z"
generation: 1
labels:
cluster.x-k8s.io/provider: cluster-api
clusterctl.cluster.x-k8s.io: ""
clusterctl.cluster.x-k8s.io/core: inventory
name: cluster-api
namespace: capi-system
resourceVersion: "1064"
uid: 34380fef-226b-413e-8c27-39fda6b19964
providerName: cluster-api
type: CoreProvider
version: v1.12.3
# 노드를 Kubernetes로 부팅시키는 역할 : cloud-init user-data 생성, kubeadm join/init config 생성 kubectl get providers -n capi-kubeadm-bootstrap-system bootstrap-kubeadm -o yaml apiVersion: clusterctl.cluster.x-k8s.io/v1alpha3
kind: Provider
metadata:
creationTimestamp: "2026-02-18T06:53:03Z"
generation: 1
labels:
cluster.x-k8s.io/provider: bootstrap-kubeadm
clusterctl.cluster.x-k8s.io: ""
clusterctl.cluster.x-k8s.io/core: inventory
name: bootstrap-kubeadm
namespace: capi-kubeadm-bootstrap-system
resourceVersion: "1126"
uid: a2b58ebb-f52f-47fb-9849-95227a160b1d
providerName: kubeadm
type: BootstrapProvider
version: v1.12.3
# Control Plane 전용 Machine 관리 : KubeadmControlPlane 리소스 관리, Control Plane 노드 스케일링, etcd 포함 업그레이드 관리 kubectl get providers -n capi-kubeadm-control-plane-system control-plane-kubeadm -o yaml apiVersion: clusterctl.cluster.x-k8s.io/v1alpha3
kind: Provider
metadata:
creationTimestamp: "2026-02-18T06:53:04Z"
generation: 1
labels:
cluster.x-k8s.io/provider: control-plane-kubeadm
clusterctl.cluster.x-k8s.io: ""
clusterctl.cluster.x-k8s.io/core: inventory
name: control-plane-kubeadm
namespace: capi-kubeadm-control-plane-system
resourceVersion: "1200"
uid: c65b2a94-7eec-499c-8672-9ab6ba2c1f52
providerName: kubeadm
type: ControlPlaneProvider
version: v1.12.3
# 실제 인프라 리소스 생성 담당 : 실제 Docker 컨테이너를 VM처럼 생성, Dev/Test 용도 (CAPD) kubectl get providers -n capd-system infrastructure-docker -o yaml apiVersion: clusterctl.cluster.x-k8s.io/v1alpha3
kind: Provider
metadata:
creationTimestamp: "2026-02-18T06:53:05Z"
generation: 1
labels:
cluster.x-k8s.io/provider: infrastructure-docker
clusterctl.cluster.x-k8s.io: ""
clusterctl.cluster.x-k8s.io/core: inventory
name: infrastructure-docker
namespace: capd-system
resourceVersion: "1303"
uid: a59b4f02-5ec2-4387-b587-4ed288a648b4
providerName: docker
type: InfrastructureProvider
version: v1.12.3
|
③ cert-manger 확인
| kubectl get crd | grep cert certificaterequests.cert-manager.io 2026-02-18T06:52:25Z certificates.cert-manager.io 2026-02-18T06:52:25Z challenges.acme.cert-manager.io 2026-02-18T06:52:25Z clusterissuers.cert-manager.io 2026-02-18T06:52:25Z issuers.cert-manager.io 2026-02-18T06:52:26Z orders.acme.cert-manager.io 2026-02-18T06:52:25Z kubectl get deploy,pod,svc,ep,cm,secret,sa -n cert-manager NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/cert-manager 1/1 1 1 9m deployment.apps/cert-manager-cainjector 1/1 1 1 9m deployment.apps/cert-manager-webhook 1/1 1 1 9m NAME READY STATUS RESTARTS AGE pod/cert-manager-598d877b78-pvt4k 1/1 Running 0 9m pod/cert-manager-cainjector-6b5777d564-4gsnt 1/1 Running 0 9m pod/cert-manager-webhook-5d9fc6b4ff-v4gk5 1/1 Running 0 9m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/cert-manager ClusterIP 10.96.98.8 <none> 9402/TCP 9m service/cert-manager-cainjector ClusterIP 10.96.16.30 <none> 9402/TCP 9m service/cert-manager-webhook ClusterIP 10.96.98.31 <none> 443/TCP,9402/TCP 9m NAME ENDPOINTS AGE endpoints/cert-manager 10.244.0.7:9402 9m endpoints/cert-manager-cainjector 10.244.0.6:9402 9m endpoints/cert-manager-webhook 10.244.0.8:10250,10.244.0.8:9402 9m NAME DATA AGE configmap/kube-root-ca.crt 1 9m1s NAME TYPE DATA AGE secret/cert-manager-webhook-ca Opaque 3 8m33s NAME AGE serviceaccount/cert-manager 9m serviceaccount/cert-manager-cainjector 9m serviceaccount/cert-manager-webhook 9m serviceaccount/default 9m1s kubectl get issuers.cert-manager.io -A NAMESPACE NAME READY AGE capd-system capd-selfsigned-issuer True 8m50s capi-kubeadm-bootstrap-system capi-kubeadm-bootstrap-selfsigned-issuer True 8m52s capi-kubeadm-control-plane-system capi-kubeadm-control-plane-selfsigned-issuer True 8m51s capi-system capi-selfsigned-issuer True 8m54s kubectl get certificaterequests.cert-manager.io -A -owide NAMESPACE NAME APPROVED DENIED READY ISSUER REQUESTER STATUS AGE capd-system capd-serving-cert-1 True True capd-selfsigned-issuer system:serviceaccount:cert-manager:cert-manager Certificate fetched from issuer successfully 9m9s capi-kubeadm-bootstrap-system capi-kubeadm-bootstrap-serving-cert-1 True True capi-kubeadm-bootstrap-selfsigned-issuer system:serviceaccount:cert-manager:cert-manager Certificate fetched from issuer successfully 9m11s capi-kubeadm-control-plane-system capi-kubeadm-control-plane-serving-cert-1 True True capi-kubeadm-control-plane-selfsigned-issuer system:serviceaccount:cert-manager:cert-manager Certificate fetched from issuer successfully 9m10s capi-system capi-serving-cert-1 True True capi-selfsigned-issuer system:serviceaccount:cert-manager:cert-manager Certificate fetched from issuer successfully 9m12s kubectl get certificates.cert-manager.io -A -owide NAMESPACE NAME READY SECRET ISSUER STATUS AGE capd-system capd-serving-cert True capd-webhook-service-cert capd-selfsigned-issuer Certificate is up to date and has not expired 9m27s capi-kubeadm-bootstrap-system capi-kubeadm-bootstrap-serving-cert True capi-kubeadm-bootstrap-webhook-service-cert capi-kubeadm-bootstrap-selfsigned-issuer Certificate is up to date and has not expired 9m29s capi-kubeadm-control-plane-system capi-kubeadm-control-plane-serving-cert True capi-kubeadm-control-plane-webhook-service-cert capi-kubeadm-control-plane-selfsigned-issuer Certificate is up to date and has not expired 9m28s capi-system capi-serving-cert True capi-webhook-service-cert capi-selfsigned-issuer Certificate is up to date and has not expired 9m31s |
④ 첫 번째 워크로드 Workload 클러스터 생성 및 확인
| # 첫 번째 워크로드 구성을 위한 환경 변수 설정 : 필요에 맞게 수정. ## The list of service CIDR, default ["10.128.0.0/12"] export SERVICE_CIDR=["10.20.0.0/16"] ## The list of pod CIDR, default ["192.168.0.0/16"] export POD_CIDR=["10.10.0.0/16"] ## The service domain, default "cluster.local" export SERVICE_DOMAIN="myk8s-1.local" ## PSS Disable export POD_SECURITY_STANDARD_ENABLED="false" # Generating the cluster configuration : 워크로드 클러스터 생성 --dry-run ## https://cluster-api.sigs.k8s.io/clusterctl/commands/generate-cluster clusterctl generate cluster capi-quickstart --flavor development \ --kubernetes-version v1.34.3 \ --control-plane-machine-count=3 \ --worker-machine-count=3 \ > capi-quickstart.yaml # Cluster, Machines, Machine Deployments 등과 같은 Cluster API 객체의 미리 정의된 목록이 포함된 YAML 파일이 생성됨. cat capi-quickstart.yaml cat capi-quickstart.yaml | grep -E '^apiVersion:|^kind:' apiVersion: cluster.x-k8s.io/v1beta2 kind: ClusterClass # ClusterClass 정의 (설계도) apiVersion: infrastructure.cluster.x-k8s.io/v1beta2 kind: DockerClusterTemplate apiVersion: controlplane.cluster.x-k8s.io/v1beta2 kind: KubeadmControlPlaneTemplate apiVersion: infrastructure.cluster.x-k8s.io/v1beta2 kind: DockerMachineTemplate apiVersion: infrastructure.cluster.x-k8s.io/v1beta2 kind: DockerMachineTemplate apiVersion: infrastructure.cluster.x-k8s.io/v1beta2 kind: DockerMachinePoolTemplate apiVersion: bootstrap.cluster.x-k8s.io/v1beta2 kind: KubeadmConfigTemplate apiVersion: cluster.x-k8s.io/v1beta2 kind: Cluster # Cluster (Topology 기반 실제 클러스터) # Apply the workload cluster kubectl apply -f capi-quickstart.yaml clusterclass.cluster.x-k8s.io/quick-start created dockerclustertemplate.infrastructure.cluster.x-k8s.io/quick-start-cluster created kubeadmcontrolplanetemplate.controlplane.cluster.x-k8s.io/quick-start-control-plane created dockermachinetemplate.infrastructure.cluster.x-k8s.io/quick-start-control-plane created dockermachinetemplate.infrastructure.cluster.x-k8s.io/quick-start-default-worker-machinetemplate created dockermachinepooltemplate.infrastructure.cluster.x-k8s.io/quick-start-default-worker-machinepooltemplate created kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io/quick-start-default-worker-bootstraptemplate created cluster.cluster.x-k8s.io/capi-quickstart created ##### 생성 확인 & kubeconfig 자격 증명 & CNI 플러그인 설치 후 확인 # Accessing the workload cluster ## The cluster will now start provisioning. You can check status with kubectl get cluster -o wide NAME CLUSTERCLASS AVAILABLE CP DESIRED CP CURRENT CP READY CP AVAILABLE CP UP-TO-DATE W DESIRED W CURRENT W READY W AVAILABLE W UP-TO-DATE PAUSED PHASE AGE VERSION capi-quickstart quick-start False 3 2 0 0 2 3 3 0 0 3 False Provisioned 119s v1.34.3 ## You can also get an “at glance” view of the cluster and its resources by running: clusterctl describe cluster capi-quickstart NAME REPLICAS AVAILABLE READY UP TO DATE STATUS REASON SINCE MESSAGE Cluster/capi-quickstart 5/6 0 0 5 False NotAvailable 2m32s * WorkersAvailable: │ * MachineDeployment capi-quickstart-md-0-nggp5: 0 available replicas, at least 3 required │ (spec.strategy.rollout.maxUnavailable is 0, spec.replicas is 3) ├─ClusterInfrastructure - DockerCluster/capi-quickstart-sxndn True Ready 2m26s ├─ControlPlane - KubeadmControlPlane/capi-quickstart-t85tp 2/3 0 0 2 True Available 69s │ └─2 Machines... 0 0 2 False NotReady 76s See capi-quickstart-t85tp-q68rk, capi-quickstart-t85tp-wks7g └─Workers └─MachineDeployment/capi-quickstart-md-0-nggp5 3/3 0 0 3 False NotAvailable 2m32s 0 available replicas, at least 3 required (spec.strategy.rollout.maxUnavailable is 0, spec.replicas │ is 3) └─3 Machines... 0 0 3 False NotReady 2m15s See capi-quickstart-md-0-nggp5-ktbr7-qwf9k, capi-quickstart-md-0-nggp5-ktbr7-tfbdr, ... # docker 프로바이더에 의해서, 워크로드 클러스터에 머신이 컨테이너로 기동 docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 15455de22dd1 kindest/node:v1.34.3 "/usr/local/bin/entr…" About a minute ago Up About a minute capi-quickstart-md-0-nggp5-ktbr7-tfbdr 6a0ba5492f8f kindest/node:v1.34.3 "/usr/local/bin/entr…" About a minute ago Up About a minute capi-quickstart-md-0-nggp5-ktbr7-v9wp9 dc5ed6880fcb kindest/node:v1.34.3 "/usr/local/bin/entr…" About a minute ago Up About a minute capi-quickstart-md-0-nggp5-ktbr7-qwf9k 6b0064291f54 kindest/node:v1.34.3 "/usr/local/bin/entr…" 2 minutes ago Up 2 minutes 127.0.0.1:32770->6443/tcp capi-quickstart-t85tp-q68rk 13a26eaa4503 kindest/haproxy:v20230606-42a2262b "haproxy -W -db -f /…" 3 minutes ago Up 2 minutes 0.0.0.0:32768->6443/tcp, 0.0.0.0:32769->8404/tcp capi-quickstart-lb 834590feb6e7 kindest/node:v1.35.0 "/usr/local/bin/entr…" 28 minutes ago Up 28 minutes 0.0.0.0:30000-30001->30000-30001/tcp, 127.0.0.1:45647->6443/tcp myk8s-control-plane # 컨트롤 플레인 역할 컨테이너 로그 1대 확인 해보기 docker logs -f capi-quickstart-t85tp-q68rk ... Welcome to Debian GNU/Linux 12 (bookworm)! Queued start job for default target graphical.target. [ OK ] Created slice kubelet.slic… used to run Kubernetes / Kubelet. [ OK ] Created slice system-modpr…lice - Slice /system/modprobe. ... # 워크로드 클러스터 자격증명 clusterctl get kubeconfig capi-quickstart > capi-quickstart.kubeconfig kubectl --kubeconfig=capi-quickstart.kubeconfig get nodes -owide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME capi-quickstart-t85tp-q68rk NotReady control-plane 5m16s v1.34.3 172.18.0.4 <none> Debian GNU/Linux 12 (bookworm) 6.12.0-124.31.1.el10_1.x86_64 containerd://2.2.0 # 노드 NotReady 상태 해결 : CNI 플러그인 설치 kubectl --kubeconfig=capi-quickstart.kubeconfig apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/calico.yaml kubectl --kubeconfig=capi-quickstart.kubeconfig get nodes -owide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME capi-quickstart-md-0-lt794-jfjvx-9wrql Ready <none> 113s v1.35.0 192.168.97.6 <none> Debian GNU/Linux 12 (bookworm) 6.17.4-orbstack-00308-g195e9689a04f containerd://2.2.0 capi-quickstart-md-0-lt794-jfjvx-f4pr4 Ready <none> 113s v1.35.0 192.168.97.5 <none> Debian GNU/Linux 12 (bookworm) 6.17.4-orbstack-00308-g195e9689a04f containerd://2.2.0 capi-quickstart-md-0-lt794-jfjvx-rjfz9 Ready <none> 113s v1.35.0 192.168.97.7 <none> Debian GNU/Linux 12 (bookworm) 6.17.4-orbstack-00308-g195e9689a04f containerd://2.2.0 capi-quickstart-sn62k-6n9sd Ready control-plane 2m14s v1.35.0 192.168.97.4 <none> Debian GNU/Linux 12 (bookworm) 6.17.4-orbstack-00308-g195e9689a04f containerd://2.2.0 capi-quickstart-sn62k-pr4kp Ready control-plane 46s v1.35.0 192.168.97.9 <none> Debian GNU/Linux 12 (bookworm) 6.17.4-orbstack-00308-g195e9689a04f containerd://2.2.0 capi-quickstart-sn62k-xhmtw Ready control-plane 75s v1.35.0 192.168.97.8 <none> Debian GNU/Linux 12 (bookworm) 6.17.4-orbstack-00308-g195e9689a04f containerd://2.2.0 |
⑤ 첫 번째 워크로드 클러스터 관련 Cluster API CRD 확인
| # clusterclasses 확인 : 클러스터의 기본 템플릿 정보 제공, 예) 아래 처럼 etcd 컨테이너 이미지 tag 3.5.3-0 kubectl get clusterclasses quick-start -o yaml ... patches: - definitions: - jsonPatches: ... - name: etcdImageTag required: true schema: openAPIV3Schema: default: "" description: etcdImageTag sets the tag for the etcd image. example: 3.5.3-0 type: string ... kubectl get clusterclasses -owide NAME PAUSED VARIABLES READY AGE quick-start False True 10m # cluster 확인 : 아래 Varibles 중 Name에 값이 없는 경우, clusterclasses 템플릿에 variables 값이 반영됨 kubectl get cluster -owide NAME CLUSTERCLASS AVAILABLE CP DESIRED CP CURRENT CP READY CP AVAILABLE CP UP-TO-DATE W DESIRED W CURRENT W READY W AVAILABLE W UP-TO-DATE PAUSED PHASE AGE VERSION capi-quickstart quick-start True 3 3 3 3 3 3 3 3 3 3 False Provisioned 4m37s v1.35.0 kubectl describe cluster capi-quickstart ... Spec: Cluster Network: Pods: Cidr Blocks: 10.10.0.0/16 Service Domain: myk8s-1.local Services: Cidr Blocks: 10.20.0.0/16 Control Plane Endpoint: Host: 192.168.97.3 Port: 6443 ... Topology: Class Ref: Name: quick-start Control Plane: Replicas: 3 Variables: Name: imageRepository Value: Name: etcdImageTag Value: Name: coreDNSImageTag Value: Name: podSecurityStandard Value: Audit: restricted Enabled: false Enforce: baseline Warn: restricted Version: v1.34.3 Workers: Machine Deployments: Class: default-worker Name: md-0 Replicas: 3 # 워크로드 클러스터에서 etcd 이미지 태그 정보 확인 kubectl --kubeconfig=capi-quickstart.kubeconfig get pod -n kube-system -l component=etcd -o yaml | grep image: | uniq image: registry.k8s.io/etcd:3.6.5-0 # To verify the first control plane is up: kubectl get kubeadmcontrolplane -owide NAME CLUSTER AVAILABLE DESIRED CURRENT READY AVAILABLE UP-TO-DATE PAUSED INITIALIZED AGE VERSION capi-quickstart-j9fdm capi-quickstart True 3 3 3 3 3 False true 48m v1.34.3 ## kubeadm 에 의해 생성된 컨트롤 플레인 정보 kubectl get kubeadmcontrolplane -o yaml ... spec: kubeadmConfigSpec: clusterConfiguration: apiServer: certSANs: - localhost - 127.0.0.1 - 0.0.0.0 - host.docker.internal initConfiguration: nodeRegistration: kubeletExtraArgs: - name: eviction-hard value: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0% joinConfiguration: nodeRegistration: kubeletExtraArgs: - name: eviction-hard value: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0% machineTemplate: metadata: labels: cluster.x-k8s.io/cluster-name: capi-quickstart topology.cluster.x-k8s.io/owned: "" spec: infrastructureRef: apiGroup: infrastructure.cluster.x-k8s.io kind: DockerMachineTemplate name: capi-quickstart-tdwjt replicas: 3 rollout: strategy: rollingUpdate: maxSurge: 1 type: RollingUpdate version: v1.34.3 status: availableReplicas: 3 ... # kubeadmconfigtemplate 확인 kubectl get kubeadmconfigtemplate NAME CLUSTERCLASS CLUSTER AGE capi-quickstart-md-0-ln5zl capi-quickstart 37m # 실제 Cluster 생성 시 자동으로 만들어진 복사본 quick-start-default-worker-bootstraptemplate quick-start 37m # ClusterClass에 속함, 실제 클러스터에 직접 사용되지 않음, '설계도용 템플릿' ## kubeadm 작업 시 템플릿 : 아래는 워커 노드 join 시, kubelet args 설정 kubectl get kubeadmconfigtemplate -l cluster.x-k8s.io/cluster-name=capi-quickstart # 실제 Cluster 생성 시 자동으로 만들어진 복사본 kubectl get kubeadmconfigtemplate -l cluster.x-k8s.io/cluster-name=capi-quickstart -o yaml # 아래 spec 내용과 동일 ... spec: template: spec: joinConfiguration: nodeRegistration: kubeletExtraArgs: - name: eviction-hard value: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0% kubectl get kubeadmconfigtemplate quick-start-default-worker-bootstraptemplate -o yaml # ClusterClass에 속함, 실제 클러스터에 직접 사용되지 않음, '설계도용 템플릿' ... spec: template: spec: joinConfiguration: nodeRegistration: kubeletExtraArgs: - name: eviction-hard value: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0% |
실습 정리
| # 첫 번째 워크로드 클러스터(머신 포함) 삭제 kubectl delete cluster capi-quickstart # 관리 클러스터 삭제 kind delete cluster --name myk8s |