[K8S Deploy 3주차] kubeadm 을 이용한 K8S 클러스터 구성

이번 블로그에서는 kubeadm 을 이용하여 아래와 같은 K8S 클러스터 구성을 진행합니다.

kubeadm 의 공식정의(https://github.com/kubernetes/kubeadm)는 아래와 같습니다.

Kubeadm is a tool built to provide best-practice "fast paths" for creating Kubernetes clusters. It performs the actions necessary to get a minimum viable, secure cluster up and running in a user-friendly way. Kubeadm's scope is limited to the local node filesystem and the Kubernetes API, and it is intended to be a composable building block of higher level tools.

 

kubeadm 을 이용하여 아래 요소를 Deploy  합니다.

• Controlplane Component 를 Static Pod 로 구성
• "cri(containerd) 및 kubelet" 에 대한 설정 구성
  • ‘cri(containerd) 및 kubelet’ 설치는 사전에 되어 있어야 함

0. 실습 환경 구성

3개의 VM을 Vagrant를 이용하여 생성합니다. OS 는 "Rocky Linux 10" 버전을 사용합니다.

• Vagrantfile

# Base Image  https://portal.cloud.hashicorp.com/vagrant/discover/bento/rockylinux-10.0
BOX_IMAGE = "bento/rockylinux-10.0" # "bento/rockylinux-9"
BOX_VERSION = "202510.26.0"
N = 2 # max number of Worker Nodes

Vagrant.configure("2") do |config|
# ControlPlane Nodes 
    config.vm.define "k8s-ctr" do |subconfig|
      subconfig.vm.box = BOX_IMAGE
      subconfig.vm.box_version = BOX_VERSION
      subconfig.vm.provider "virtualbox" do |vb|
        vb.customize ["modifyvm", :id, "--groups", "/K8S-Upgrade-Lab"]
        vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
        vb.name = "k8s-ctr"
        vb.cpus = 4
        vb.memory = 3072 # 2048 2560 3072 4096
        vb.linked_clone = true
      end
      subconfig.vm.host_name = "k8s-ctr"
      subconfig.vm.network "private_network", ip: "192.168.56.100"
      subconfig.vm.network "forwarded_port", guest: 22, host: "60000", auto_correct: true, id: "ssh"
      subconfig.vm.synced_folder "./", "/vagrant", disabled: true
    end

# Worker Nodes
  (1..N).each do |i|
    config.vm.define "k8s-w#{i}" do |subconfig|
      subconfig.vm.box = BOX_IMAGE
      subconfig.vm.box_version = BOX_VERSION
      subconfig.vm.provider "virtualbox" do |vb|
        vb.customize ["modifyvm", :id, "--groups", "/K8S-Upgrade-Lab"]
        vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
        vb.name = "k8s-w#{i}"
        vb.cpus = 2
        vb.memory = 2048
        vb.linked_clone = true
      end
      subconfig.vm.host_name = "k8s-w#{i}"
      subconfig.vm.network "private_network", ip: "192.168.56.10#{i}"
      subconfig.vm.network "forwarded_port", guest: 22, host: "6000#{i}", auto_correct: true, id: "ssh"
      subconfig.vm.synced_folder "./", "/vagrant", disabled: true
    end
  end

end

# VM 시작 vagrant up # k8s-ctr 접속 vagrant ssh k8s-ctr This system is built by the Bento project by Chef Software More information can be found at https://github.com/chef/bento Use of this system is acceptance of the OS vendor EULA and License Agreements. vagrant@k8s-ctr:~$ # root 계정으로 전환 vagrant@k8s-ctr:~$ sudo su - root@k8s-ctr:~#

 

kubeadm 으로 k8s 클러스터 구성 절차는 아래 단계를 따릅니다.

1. [공통] 사전 설정
2. [공통] CRI 설치 : containerd
3. [공통] kubeadm, kubelet 및 kubectl 설치
4. [Controlplane node] kubeadm 으로 k8s 클러스터 구성 → Flannel CNI 설치 → 편의성 설치 및 확인
5. [Worker nodes] kubeadm 으로 k8s 클러스터 join → 확인

이번 실습에서는 K8S v1.32.11 기준으로 클러스터를 구성하며, 주요 구성요소 버전은 아래와 같습니다.

항목	버전	k8s 버전 호환성
Rocky Linux	10.0-1.6	RHEL 10 소스 기반 배포판으로 RHEL 정보 참고
containerd	v2.1.5	CRI Version(v1), k8s 1.32~1.35 지원 - Link
runc	v1.3.3	정보 조사 필요 https://github.com/opencontainers/runc
kubelet	v1.32.11	k8s 버전 정책 문서 참고 - Docs
kubeadm	v1.32.11	상동
kubectl	v1.32.11	상동
helm	v3.18.6	k8s 1.30.x ~ 1.33.x 지원 - Docs
flannel cni	v0.27.3	k8s 1.28~ 이후 - Release

 

1. [공통] 사전 설정 - 모든 노드에서 작업

Vagrant 를 이용한 Rocky Linux 10 에서는 2가지 문제가 있으며, 뒤쪽에서 원인과 트러블슈팅을 진행합니다.

• 첫번째는 default route 문제입니다.

ip route default via 10.0.2.2 dev enp0s3 proto dhcp src 10.0.2.15 metric 100 default via 192.168.56.1 dev enp0s8 proto static metric 101 10.0.2.0/24 dev enp0s3 proto kernel scope link src 10.0.2.15 metric 100 192.168.56.0/24 dev enp0s8 proto kernel scope link src 192.168.56.100 metric 101

• 2번째는 SWAP 메모리 disable 부분입니다.

lsblk NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS sda      8:0    0  64G  0 disk ├─sda1   8:1    0   1M  0 part ├─sda2   8:2    0   3G  0 part [SWAP] └─sda3   8:3    0  61G  0 part /

• Time, NTP 설정 : 인증서 만료 시간, 로그 타임스탬프 등 모든 노드에 동기화된 시간 필요

# 시스템의 하드웨어 시계(RTC, Real-Time Clock)를 UTC(협정 세계시) 기준으로 유지하도록 설정 timedatectl set-local-rtc 0 timedatectl status                Local time: Thu 2026-01-22 14:27:00 UTC            Universal time: Thu 2026-01-22 14:27:00 UTC                  RTC time: Thu 2026-01-22 14:27:00                 Time zone: UTC (UTC, +0000) System clock synchronized: yes               NTP service: active           RTC in local TZ: no # 시스템 타임존(Timezone)을 한국(KST, UTC+9) 으로 설정 : 시스템 시간은 UTC 기준 유지, 표시만 KST로 변환 timedatectl set-timezone Asia/Seoul date Thu Jan 22 11:29:01 PM KST 2026 # systemd가 시간 동기화 서비스(chronyd) 를 관리하도록 설정되어 있음 : ntpd 대신 chrony 사용 (Rocky 9/10 기본) # 네트워크 시간 프로토콜(NTP, Network Time Protocol)을 사용하여 시스템 시간을 인터넷상의 시간 서버와 자동으로 동기화하도록 설정 # Rocky Linux 10을 포함한 RHEL 계열 운영체제에서는 아래 명령을 실행하면 백그라운드에서 chronyd 라는 서비스(데몬)가 활성화됩니다. timedatectl set-ntp true  # chronyc 확인 # chrony가 어떤 NTP 서버들을 알고 있고, 그중 어떤 서버를 기준으로 시간을 맞추는지를 보여줍니다. ## Stratum 2: 매우 신뢰도 높은 서버 ## Reach 377: 최근 8회 연속 응답 성공 (최대값) chronyc sources -v   .-- Source mode  '^' = server, '=' = peer, '#' = local clock.  / .- Source state '*' = current best, '+' = combined, '-' = not combined, | /             'x' = may be in error, '~' = too variable, '?' = unusable. ||                                                 .- xxxx [ yyyy ] +/- zzzz ||      Reachability register (octal) -.           |  xxxx = adjusted offset, ||      Log2(Polling interval) --.      |          |  yyyy = measured offset, ||                                \     |          |  zzzz = estimated error. ||                                 |    |           \ MS Name/IP address         Stratum Poll Reach LastRx Last sample =============================================================================== ^+ 211.108.117.211               2  10   377   443   -358us[ -358us] +/- 2602us ^- mail.innotab.com              3   9   377   507   -497us[ -497us] +/-   17ms ^* mail.pyeongga.com             2  10   377   513   -819us[ -743us] +/- 2853us ^- 175.210.18.47                 2  10   377   188    -43us[  -43us] +/-   15ms # 현재 시스템 시간이 얼마나 정확한지 종합 성적표 chronyc tracking Reference ID    : DD97764E (mail.pyeongga.com) Stratum         : 3 Ref time (UTC)  : Thu Jan 22 14:26:38 2026 System time     : 0.000023923 seconds fast of NTP time Last offset     : +0.000076193 seconds RMS offset      : 0.005207670 seconds Frequency       : 496.915 ppm slow Residual freq   : +0.001 ppm Skew            : 0.102 ppm Root delay      : 0.003935313 seconds Root dispersion : 0.000818134 seconds Update interval : 513.7 seconds Leap status     : Normal

 

• SELinux 설정
  • 공식 문서에 SELinux 기능은 Permissive 권장

# 런타임에 selinux 기능 off setenforce 0 # 재부팅 후에도 selinux 기능을 permisive  설정되도록 /etc/selinux/config 파일 수정 sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config

• 방화벽(firewalld) 서비스 disable

systemctl disable --now firewalld Removed '/etc/systemd/system/multi-user.target.wants/firewalld.service'. Removed '/etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service'. systemctl status firewalld ○ firewalld.service - firewalld - dynamic firewall daemon      Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; preset: enabled)      Active: inactive (dead)        Docs: man:firewalld(1) Jan 22 22:13:09 localhost systemd[1]: Starting firewalld.service - firewalld - dynamic firewall daemon... Jan 22 22:13:12 localhost systemd[1]: Started firewalld.service - firewalld - dynamic firewall daemon. Jan 22 23:41:23 k8s-ctr systemd[1]: Stopping firewalld.service - firewalld - dynamic firewall daemon... Jan 22 23:41:24 k8s-ctr systemd[1]: firewalld.service: Deactivated successfully. Jan 22 23:41:24 k8s-ctr systemd[1]: Stopped firewalld.service - firewalld - dynamic firewall daemon. Jan 22 23:41:24 k8s-ctr systemd[1]: firewalld.service: Consumed 1.172s CPU time, 69.9M memory peak.

• Swap 비활성화

# 현재 swap 메모리 사용중 free -h                total        used        free      shared  buff/cache   available Mem:           2.9Gi       419Mi       2.5Gi        15Mi       197Mi       2.5Gi Swap:          3.0Gi          0B       3.0Gi # 런타임에 swap 메모리 사용 중지 swapoff -a free -h                total        used        free      shared  buff/cache   available Mem:           2.9Gi       418Mi       2.5Gi        15Mi       197Mi       2.5Gi Swap:             0B          0B          0B # 재부팅 시에도 'Swap 비활성화' 적용되도록 /etc/fstab에서 swap 라인 삭제 sed -i '/swap/d' /etc/fstab

• 커널 모듈 및 커널 파라미터 설정(네트워크 관련)

# 런타임에 커널 모듈 로드 modprobe overlay modprobe br_netfilter lsmod | grep -iE 'overlay|br_netfilter' br_netfilter           36864  0 bridge                417792  1 br_netfilter overlay               245760  0 # 재시작 후에도 위의 모듈이 로드 될 수 있도록 모듈 설정 파일 추가 cat <<EOF | tee /etc/modules-load.d/k8s.conf overlay br_netfilter EOF tree /etc/modules-load.d/ /etc/modules-load.d/ └── k8s.conf # 커널 파라미터 설정 : 네트워크 설정 - 브릿지 트래픽이 iptables를 거치도록 함 cat <<EOF | tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-iptables  = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.ipv4.ip_forward                 = 1 EOF tree /etc/sysctl.d/ /etc/sysctl.d/ ├── 99-sysctl.conf -> ../sysctl.conf └── k8s.conf # 커널 파라미터 설정을 런타임에 적용 sysctl --system

• hosts 파일 설정

sed -i '/^127\.0\.\(1\|2\)\.1/d' /etc/hosts cat << EOF >> /etc/hosts 192.168.56.100 k8s-ctr 192.168.56.101 k8s-w1 192.168.56.102 k8s-w2 EOF cat /etc/hosts

 

2. [공통] CRI 설치 : containerd(runc) v2.1.5

• https://github.com/containerd/containerd/tree/main
• https://v1-32.docs.kubernetes.io/ko/docs/setup/production-environment/container-runtimes/
• Containerd 와 K8S(1.32~1.34) 호환 버전 확인(2.1.0+, 2.0.1+, 1.7.24+, 1.6.36+…)
  • https://containerd.io/releases/#kubernetes-support

Kubernetes Version	containerd Version	CRI Version
1.32*	2.1.0+, 2.0.1+, 1.7.24+, 1.6.36+	v1
1.33	2.1.0+, 2.0.4+, 1.7.24+, 1.6.36+	v1
1.34	2.1.3+, 2.0.6+, 1.7.28+, 1.6.39+	v1
1.35	2.2.0+, 2.1.5+, 1.7.28+	v1

 

# Docker Communit Edition 저장소 추가 : dockerd 설치 X, containerd 설치 OK dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo Adding repo from: https://download.docker.com/linux/centos/docker-ce.repo dnf repolist repo id                                        repo name appstream                                  Rocky Linux 10 - AppStream baseos                                       Rocky Linux 10 - BaseOS docker-ce-stable                        Docker CE Stable - x86_64 extras                                         Rocky Linux 10 - Extras # Repository Metadata 갱신 dnf makecache # 설치 가능한 모든 containerd.io 버전 확인 dnf list --showduplicates containerd.io Available Packages containerd.io.x86_64              1.7.23-3.1.el10               docker-ce-stable containerd.io.x86_64              1.7.24-3.1.el10               docker-ce-stable containerd.io.x86_64              1.7.25-3.1.el10               docker-ce-stable containerd.io.x86_64              1.7.26-3.1.el10               docker-ce-stable containerd.io.x86_64              1.7.27-3.1.el10               docker-ce-stable containerd.io.x86_64              1.7.28-1.el10                 docker-ce-stable containerd.io.x86_64              1.7.28-2.el10                 docker-ce-stable containerd.io.x86_64              1.7.29-1.el10                 docker-ce-stable containerd.io.x86_64              2.1.5-1.el10                  docker-ce-stable containerd.io.x86_64              2.2.0-2.el10                  docker-ce-stable containerd.io.x86_64              2.2.1-1.el10                  docker-ce-stable # containerd 설치 dnf install -y containerd.io-2.1.5-1.el10 ... Installed:   containerd.io-2.1.5-1.el10.x86_64 # 설치된 파일 확인 which runc /usr/bin/runc runc --version runc version 1.3.3 commit: v1.3.3-0-gd842d771 spec: 1.2.1 go: go1.24.9 libseccomp: 2.5.3 which containerd /usr/bin/containerd containerd --version containerd containerd.io v2.1.5 fcd43222d6b07379a4be9786bda52438f0dd16a1 which containerd-shim-runc-v2 /usr/bin/containerd-shim-runc-v2 containerd-shim-runc-v2 -v containerd-shim-runc-v2:   Version:  v2.1.5   Revision: fcd43222d6b07379a4be9786bda52438f0dd16a1   Go version: go1.24.9 which ctr /usr/bin/ctr ctr --version ctr containerd.io v2.1.5 cat /etc/containerd/config.toml #   Copyright 2018-2022 Docker Inc. #   Licensed under the Apache License, Version 2.0 (the "License"); #   you may not use this file except in compliance with the License. #   You may obtain a copy of the License at #       http://www.apache.org/licenses/LICENSE-2.0 #   Unless required by applicable law or agreed to in writing, software #   distributed under the License is distributed on an "AS IS" BASIS, #   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. #   See the License for the specific language governing permissions and #   limitations under the License. disabled_plugins = ["cri"] #root = "/var/lib/containerd" #state = "/run/containerd" #subreaper = true #oom_score = 0 #[grpc] #  address = "/run/containerd/containerd.sock" #  uid = 0 #  gid = 0 #[debug] #  address = "/run/containerd/debug.sock" #  uid = 0 #  gid = 0 #  level = "info" cat /usr/lib/systemd/system/containerd.service # Copyright The containerd Authors. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # #     http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. [Unit] Description=containerd container runtime Documentation=https://containerd.io After=network.target dbus.service [Service] ExecStartPre=-/sbin/modprobe overlay ExecStart=/usr/bin/containerd Type=notify Delegate=yes KillMode=process Restart=always RestartSec=5 # Having non-zero Limit*s causes performance problems due to accounting overhead # in the kernel. We recommend using cgroups to do container-local accounting. LimitNPROC=infinity LimitCORE=infinity # Comment TasksMax if your systemd version does not supports it. # Only systemd 226 and above support this version. TasksMax=infinity OOMScoreAdjust=-999 [Install] WantedBy=multi-user.target # 기본 설정 생성 및 SystemdCgroup 활성화 (매우 중요) containerd config default | tee /etc/containerd/config.toml version = 3                    # containerd version 2.0 이상 시 root = '/var/lib/containerd' state = '/run/containerd' ...(생략) # https://v1-32.docs.kubernetes.io/ko/docs/setup/production-environment/container-runtimes/#cgroupfs-cgroup-driver # cgroupfs 드라이버는 kubelet의 기본 cgroup 드라이버이다.  # cgroupfs 드라이버가 사용될 때, kubelet과 컨테이너 런타임은 직접적으로 cgroup 파일시스템과 상호작용하여 cgroup들을 설정한다. # cgroupfs 드라이버가 권장되지 않는 때가 있는데, systemd가 init 시스템인 경우이다.  # 이것은 systemd가 시스템에 단 하나의 cgroup 관리자만 있을 것으로 기대하기 때문이다.  # 또한, cgroup v2를 사용할 경우에도 cgroupfs 대신 systemd cgroup 드라이버를 사용한다. # ----------------------------------------------------------- # https://github.com/containerd/containerd/blob/main/docs/cri/config.md ## In containerd 2.x version = 3 [plugins.'io.containerd.cri.v1.images']   snapshotter = "overlayfs" sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml # systemd unit 파일 최신 상태 읽기 systemctl daemon-reload # containerd start 와 enabled systemctl enable --now containerd # containerd의 유닉스 도메인 소켓 확인 : kubelet에서 사용 , containerd client 3종(ctr, nerdctr, crictl)도 사용 containerd config dump | grep -n containerd.sock 11:  address = '/run/containerd/containerd.sock' ls -l /run/containerd/containerd.sock srw-rw----. 1 root root 0 Jan 23 00:22 /run/containerd/containerd.sock ss -xl | grep containerd u_str LISTEN 0      4096        /run/containerd/containerd.sock.ttrpc 26657            * 0 u_str LISTEN 0      4096              /run/containerd/containerd.sock 26658            * 0 ss -xnp | grep containerd u_str ESTAB 0      0             * 26648            * 25779 users:(("containerd",pid=5786,fd=2),("containerd",pid=5786,fd=1))    # 플러그인 확인 ctr --address /run/containerd/containerd.sock version Client:   Version:  v2.1.5   Revision: fcd43222d6b07379a4be9786bda52438f0dd16a1   Go version: go1.24.9 Server:   Version:  v2.1.5   Revision: fcd43222d6b07379a4be9786bda52438f0dd16a1   UUID: a36b0a8b-f2d2-4ca4-816e-db92ba8d6edd ctr plugins ls TYPE                                       ID                       PLATFORMS         STATUS     io.containerd.content.v1           content               -                             ok     # 이미지 레이어 저장 ... io.containerd.snapshotter.v1    native                 linux/arm64/v8       ok         io.containerd.snapshotter.v1    overlayfs            linux/arm64/v8       ok     # Kubernetes 기본 snapshotter io.containerd.snapshotter.v1    zfs                      linux/arm64/v8       skip       ...    io.containerd.metadata.v1        bolt                     -                            ok     # 메타데이터 DB (bolt)

 

3. [공통] kubeadm, kubelet 및 kubectl 설치 v1.32.11

• https://v1-32.docs.kubernetes.io/ko/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
• containerd 를 사용하는 클라이언트 도구는 3가지가 있음 - ctr, nerdctl, crictl

# repo 추가 ## exclude=... : 실수로 dnf update 시 kubelet 자동 업그레이드 방지 ## k8s 버전에 맞게 아래 노랑색 부분 치환하여야 함 cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/ enabled=1 gpgcheck=1 gpgkey=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/repodata/repomd.xml.key exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni EOF dnf makecache # 설치 ## --disableexcludes=... kubernetes repo에 설정된 exclude 규칙을 이번 설치에서만 무시(1회성 옵션 처럼 사용) ## 버전 정보 미지정 시, 제공 가능 최신 버전 설치됨. dnf install -y kubelet kubeadm kubectl --disableexcludes=kubernetes ...(생략)... Installed:   cri-tools-1.32.0-150500.1.1.x86_64       kubeadm-1.32.11-150500.1.1.x86_64            kubectl-1.32.11-150500.1.1.x86_64   kubelet-1.32.11-150500.1.1.x86_64        kubernetes-cni-1.6.0-150500.1.1.x86_64 # kubelet 활성화 (실제 기동은 kubeadm init 후에 시작됨) systemctl enable --now kubelet # cri-tools 버전 확인하면 설정파일인  /etc/crictl.yaml 이 없어서 WARN 발생하므로 /etc/crictl.yaml 파일 생성  which crictl && crictl version WARN[0000] Config "/etc/crictl.yaml" does not exist, trying next: "/usr/bin/crictl.yaml"  # /etc/crictl.yaml 파일 작성 cat << EOF > /etc/crictl.yaml runtime-endpoint: unix:///run/containerd/containerd.sock image-endpoint: unix:///run/containerd/containerd.sock EOF # containerd  에 대한 기본 정보 확인 crictl info | jq {   "cniconfig": {     "Networks": [       {         "Config": {           "CNIVersion": "0.3.1",           "Name": "cni-loopback",           "Plugins": [             {               "Network": {                 "ipam": {},                 "type": "loopback"               },               "Source": "{\"type\":\"loopback\"}"             }           ],           "Source": "{\n\"cniVersion\": \"0.3.1\",\n\"name\": \"cni-loopback\",\n\"plugins\": [{\n  \"type\": \"loopback\"\n}]\n}"         },         "IFName": "lo"       }     ],     "PluginConfDir": "/etc/cni/net.d",     "PluginDirs": [       "/opt/cni/bin"     ],     "PluginMaxConfNum": 1,     "Prefix": "eth"   },   "config": {     "cdiSpecDirs": [       "/etc/cdi",       "/var/run/cdi"     ],     "cni": {       "binDir": "",       "binDirs": [         "/opt/cni/bin"       ],       "confDir": "/etc/cni/net.d",       "confTemplate": "",       "ipPref": "",       "maxConfNum": 1,       "setupSerially": false,       "useInternalLoopback": false     },     "containerd": {       "defaultRuntimeName": "runc",       "ignoreBlockIONotEnabledErrors": false,       "ignoreRdtNotEnabledErrors": false,       "runtimes": {         "runc": {           "ContainerAnnotations": [],           "PodAnnotations": [],           "baseRuntimeSpec": "",           "cgroupWritable": false,           "cniConfDir": "",           "cniMaxConfNum": 0,           "io_type": "",           "options": {             "BinaryName": "",             "CriuImagePath": "",             "CriuWorkPath": "",             "IoGid": 0,             "IoUid": 0,             "NoNewKeyring": false,             "Root": "",             "ShimCgroup": "",             "SystemdCgroup": true           },           "privileged_without_host_devices": false,           "privileged_without_host_devices_all_devices_allowed": false,           "runtimePath": "",           "runtimeType": "io.containerd.runc.v2",           "sandboxer": "podsandbox",           "snapshotter": ""         }       }     },     "containerdEndpoint": "/run/containerd/containerd.sock",     "containerdRootDir": "/var/lib/containerd",     "device_ownership_from_security_context": false,     "disableApparmor": false,     "disableHugetlbController": true,     "disableProcMount": false,     "drainExecSyncIOTimeout": "0s",     "enableCDI": true,     "enableSelinux": false,     "enableUnprivilegedICMP": true,     "enableUnprivilegedPorts": true,     "ignoreDeprecationWarnings": [],     "ignoreImageDefinedVolumes": false,     "maxContainerLogLineSize": 16384,     "netnsMountsUnderStateDir": false,     "restrictOOMScoreAdj": false,     "rootDir": "/var/lib/containerd/io.containerd.grpc.v1.cri",     "selinuxCategoryRange": 1024,     "stateDir": "/run/containerd/io.containerd.grpc.v1.cri",     "tolerateMissingHugetlbController": true,     "unsetSeccompProfile": ""   },   "golang": "go1.24.9",   "lastCNILoadStatus": "cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config",   "lastCNILoadStatus.default": "cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config",   "runtimeHandlers": [     {       "features": {         "recursive_read_only_mounts": true,         "user_namespaces": true       }     },     {       "features": {         "recursive_read_only_mounts": true,         "user_namespaces": true       },       "name": "runc"     }   ],   "status": {     "conditions": [       {         "message": "",         "reason": "",         "status": true,         "type": "RuntimeReady"       },       {         "message": "Network plugin returns error: cni plugin not initialized",         "reason": "NetworkPluginNotReady",         "status": false,         "type": "NetworkReady"       },       {         "message": "",         "reason": "",         "status": true,         "type": "ContainerdHasNoDeprecationWarnings"       }     ]   } } # kubernetes-cni : 파드 네트워크 구성을 위한 CNI 바이너리 파일 확인 tree /opt/cni /opt/cni └── bin     ├── bandwidth     ├── bridge     ├── dhcp     ├── dummy     ├── firewall     ├── host-device     ├── host-local     ├── ipvlan     ├── LICENSE     ├── loopback     ├── macvlan     ├── portmap     ├── ptp     ├── README.md     ├── sbr     ├── static     ├── tap     ├── tuning     ├── vlan     └── vrf tree /etc/cni/ /etc/cni/ └── net.d # 클러스터는 아직 구성이 안되었지만 기본 디렉토리 구조는 생성된 상태 tree /etc/kubernetes /etc/kubernetes └── manifests tree /var/lib/kubelet /var/lib/kubelet 0 directories, 0 files cat /etc/sysconfig/kubelet KUBELET_EXTRA_ARGS=

 

 

4. [k8s-ctr] kubeadm 으로 k8s 클러스터 구성 & Flannel CNI 설치 v0.27.3

k8s 클러스터를 구성하기 위해서 수행하는 "kubeadm init" 은 아래 절차대로 작업을 수행합니다.

https://kubernetes.io/docs/reference/setup-tools/kubeadm/implementation-details/

Implementation details

 

1. 사전 검사 (Preflight Checks)
   • [Error] if the CRI endpoint does not answer ← CRI(Container Runtime Interface) 엔드포인트 연결 확인
   • [Error] if user is not root ← 루트 권한 확인
   • [Error] if kubelet version is lower that the minimum kubelet version supported by kubeadm (current minor -1) ← 버전 확인
2. 인증서 및 키 생성 (Certificates) : Control Plane이 안전하게 통신할 수 있도록 인증서를 /etc/kubernetes/pki 생성
   • CA 인증서 (클러스터 전체 기본 신뢰체계)
   • API Server용 서버 인증서
   • API Server ↔ kubelet 통신용 인증서
   • Front-Proxy CA / Front-Proxy 클라이언트 인증서
   • etcd 관련 인증서 및 키 (local etcd 사용 시)
3. control plane components 를 위한 kubeconfig 파일을 /etc/kubernetes 생성
   • A kubeconfig file for the kubelet to use during TLS bootstrap - /etc/kubernetes/bootstrap-kubelet.conf
   • A kubeconfig file for controller-manager, /etc/kubernetes/controller-manager.conf
   • A kubeconfig file for scheduler, /etc/kubernetes/scheduler.conf
   • /etc/kubernetes/admin.conf , /etc/kubernetes/super-admin.conf
4. Control Plane 구성 요소 생성 (Static Pods)
   • 공통 사항
     • All static Pods are deployed on kube-system namespace
     • All static Pods get tier:control-plane and component:{component-name} labels
     • All static Pods use the system-node-critical priority class
     • hostNetwork: true is set on all static Pods to allow control plane startup
   • kube-apiserver
   • kube-controller-manager
   • kube-scheduler
   • etcd (local etcd 사용하는 경우)
5. kubelet 시작 및 대기
   • kubeadm은 manifest 적용 후 kubelet을 재시작하거나 설정을 적용하고 , Control Plane이 성공적으로 시작될 때까지 기다립니다.
   • API Server가 정상적으로 건강 상태(healthz)를 반환할 때까지 대기합니다 : /healthz or /livez endpoints.
6. Save the kubeadm ClusterConfiguration in a ConfigMap for later reference
   • kubectl get cm -n kube-system **kubeadm-config**
7. control-plane 노드 라벨링 및 태인트 Mark the node as control-plane
   • Labels the node as control-plane with node-role.kubernetes.io/control-plane=""
   • Taints the node with node-role.kubernetes.io/control-plane:NoSchedule ← 일반 워크로드가 Control Plane에 스케줄 X
8. Bootstrap 토큰 생성 및 클러스터 부트스트랩 설정 Configure TLS-Bootstrapping for node joining
   • Control Plane에 부트스트랩 토큰을 생성합니다. Create a bootstrap token
   • 각 노드가 클러스터에 합류할 수 있도록 ConfigMap, RBAC 규칙, CSR 접근 권한 등을 설정합니다.
     • Create the public cluster-info ConfigMap : '신원 확인 전, 최소한의 신뢰 부트스트랩 데이터'
       • This phase creates the cluster-info ConfigMap in the kube-public namespace.
       • Additionally, it creates a Role and a RoleBinding granting access to the ConfigMap for unauthenticated users (i.e. users in RBAC group system:unauthenticated).
     • Allow joining nodes to call CSR API …(생략)…
   • 부트스트랩 토큰은 기본적으로 24시간 유효하며, kubeadm join 시 사용할 수 있습니다.
9. 필수 애드온 설치 (CoreDNS, kube-proxy)
   • kube-proxy
     • A ServiceAccount for kube-proxy is created in the kube-system namespace
     • then kube-proxy is deployed as a DaemonSet
   • CoreDNS
     • The CoreDNS service is named kube-dns for compatibility reasons with the legacy kube-dns addon.
     • A ServiceAccount for CoreDNS is created in the kube-system namespace.
     • The coredns ServiceAccount is bound to the privileges in the system:coredns ClusterRole.
     • In Kubernetes version 1.21, support for using kube-dns with kubeadm was removed.

# kubeadm Configuration 파일 작성 cat << EOF > kubeadm-init.yaml apiVersion: kubeadm.k8s.io/v1beta4 kind: InitConfiguration bootstrapTokens: - token: "123456.1234567890123456"   ttl: "0s"   usages:   - signing   - authentication nodeRegistration:   kubeletExtraArgs:     - name: node-ip       value: "192.168.56.100"   criSocket: "unix:///run/containerd/containerd.sock" localAPIEndpoint:   advertiseAddress: "192.168.56.100" --- apiVersion: kubeadm.k8s.io/v1beta4 kind: ClusterConfiguration kubernetesVersion: "1.32.11" networking:   podSubnet: "10.244.0.0/16"   serviceSubnet: "10.96.0.0/16" EOF # (옵션) 컨테이너 이미지 미리 다운로드 : 특히 업그레이드 작업 시, 작업 시간 단축을 위해서 수행할 것 kubeadm config images pull # k8s controlplane 초기화 설정 수행 kubeadm init --config="kubeadm-init.yaml" --dry-run # kubeadm init 작업 수행 kubeadm init --config="kubeadm-init.yaml" [init] Using Kubernetes version: v1.32.11 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action beforehand using 'kubeadm config images pull' W0123 01:14:00.621003    8189 checks.go:843] detected that the sandbox image "" of the container runtime is inconsistent with that used by kubeadm.It is recommended to use "registry.k8s.io/pause:3.10" as the CRI sandbox image. [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [k8s-ctr kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.56.100] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [k8s-ctr localhost] and IPs [192.168.56.100 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [k8s-ctr localhost] and IPs [192.168.56.100 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "super-admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests" [kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s [kubelet-check] The kubelet is healthy after 508.658846ms [api-check] Waiting for a healthy API server. This can take up to 4m0s [api-check] The API server is healthy after 5.502703086s [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Skipping phase. Please see --upload-certs [mark-control-plane] Marking the node k8s-ctr as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] [mark-control-plane] Marking the node k8s-ctr as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule] [bootstrap-token] Using token: 123456.1234567890123456 [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user:   mkdir -p $HOME/.kube   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config   sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run:   export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:   https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.56.100:6443 --token 123456.1234567890123456 \         --discovery-token-ca-cert-hash sha256:775c3c589ebc24b303439684e3c31a2c4739c977b7e94ef60bc17078da66515a # crictl 확인 crictl images IMAGE                                                    TAG                 IMAGE ID            SIZE registry.k8s.io/coredns/coredns               v1.11.3             2f6c962e7b831       16.9MB registry.k8s.io/etcd                                   3.5.24-0            1211402d28f58       21.9MB registry.k8s.io/kube-apiserver                   v1.32.11            58951ea1a0b5d       26.4MB registry.k8s.io/kube-controller-manager   v1.32.11            82766e5f2d560       24.2MB registry.k8s.io/kube-proxy                         v1.32.11            dcdb790dc2bfe       27.6MB registry.k8s.io/kube-scheduler                  v1.32.11            cfa17ff3d6634       19.2MB registry.k8s.io/pause                                3.10                afb61768ce381       268kB crictl ps CONTAINER           IMAGE               CREATED             STATE               NAME                      ATTEMPT             POD ID              POD                               NAMESPACE a04be00090580       dcdb790dc2bfe       26 seconds ago      Running             kube-proxy                0                   1fd91b0a982bb       kube-proxy-7w44b                  kube-system b005f34739da5       82766e5f2d560       37 seconds ago      Running             kube-controller-manager   0                   555d146c3ec07       kube-controller-manager-k8s-ctr   kube-system eb42b9c47fdce       cfa17ff3d6634       37 seconds ago      Running             kube-scheduler            0                   e649514d0a1b7       kube-scheduler-k8s-ctr            kube-system bbe8495d2a205       58951ea1a0b5d       37 seconds ago      Running             kube-apiserver            0                   be25c00dd555c       kube-apiserver-k8s-ctr            kube-system c00a944599500       1211402d28f58       37 seconds ago      Running             etcd                      0                   ce6b89dea28da       etcd-k8s-ctr                      kube-system # kubeconfig 작성 mkdir -p /root/.kube cp -i /etc/kubernetes/admin.conf /root/.kube/config chown $(id -u):$(id -g) /root/.kube/config # 확인 kubectl cluster-info Kubernetes control plane is running at https://192.168.56.100:6443 CoreDNS is running at https://192.168.56.100:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. kubectl get node -owide NAME      STATUS     ROLES           AGE     VERSION    INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION                 CONTAINER-RUNTIME k8s-ctr   NotReady   control-plane   3m13s   v1.32.11   192.168.56.100   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.x86_64   containerd://2.1.5 kubectl get nodes -o json | jq ".items[] | {name:.metadata.name} + .status.capacity" {   "name": "k8s-ctr",   "cpu": "4",   "ephemeral-storage": "62374Mi",   "hugepages-2Mi": "0",   "memory": "3036936Ki",   "pods": "110" } kubectl get pod -n kube-system -owide NAME                              READY   STATUS    RESTARTS   AGE     IP               NODE      NOMINATED NODE   READINESS GATES coredns-668d6bf9bc-gp9dk          0/1     Pending   0          4m5s    <none>           <none>    <none>           <none> coredns-668d6bf9bc-hgllr          0/1     Pending   0          4m5s    <none>           <none>    <none>           <none> etcd-k8s-ctr                      1/1     Running   0          4m12s   192.168.56.100   k8s-ctr   <none>           <none> kube-apiserver-k8s-ctr            1/1     Running   0          4m11s   192.168.56.100   k8s-ctr   <none>           <none> kube-controller-manager-k8s-ctr   1/1     Running   0          4m11s   192.168.56.100   k8s-ctr   <none>           <none> kube-proxy-hhmmv                  1/1     Running   0          4m5s    192.168.56.100   k8s-ctr   <none>           <none> kube-scheduler-k8s-ctr            1/1     Running   0          4m11s   192.168.56.100   k8s-ctr   <none>           <none> # coredns 의 service name 확인 : kube-dns kubectl get svc -n kube-system NAME       TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   4m40s # cluster-info ConfigMap 공개 : cluster-info는 '신원 확인 전, 최소한의 신뢰 부트스트랩 데이터' kubectl -n kube-public get configmap cluster-info NAME           DATA   AGE cluster-info   2      5m46s kubectl -n kube-public get configmap cluster-info -o yaml apiVersion: v1 data:   jws-kubeconfig-123456: eyJhbGciOiJIUzI1NiIsImtpZCI6IjEyMzQ1NiJ9..IrhWmwnSAjRjS9ESnifd3c-BPnDvm8WkC0iB1mcwqtM   kubeconfig: |     apiVersion: v1     clusters:     - cluster:         certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJVjMyNkN6SEZqK2N3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TmpBeE1qSXhOakE1TURCYUZ3MHpOakF4TWpBeE5qRTBNREJhTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUURObXBxdStaYkpYU1VpOGk4VjYzc2dOZFR1cWVia3lCbjV1L1REVlI0WDJTcGE4RUtsODlLMkhBUnMKUXdCNTVBbXU3ZzlZempPTVdpM3V2NHNqMHk4QVVHcTFxa0RDajhpUHMxcW1JakgzeVdaRTJuMDZHRTQvcUlnLwo5UFl1MGtWNWdoVGM4QnJ2cStIdklsN3VqdnFiczJwSHNBZHJFUnNWOHI1UC8wRWRSdUIzU3V5SHV0ZCtwQnEzCnpZU2dTWWpDaVh6QVRad1RKN2pmdUFXUFhXSG1YOXNYeHZmS1R3MVZBMkVGTkYzdmRML0VhVEF6WUtvUzhISDQKOVp5WFBZZDRvdFZwTHlxWDNUSlpxRWRIam0rQVdleVQwVzhvaWc5VE53UVFCZExWMEQ4emk2STRVV1ozUU9nWQp1L0dTNC92T0d5QWVBdnNJc0c5L2sxK1hTU1U5QWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJSOVhtd1hhT1hCYmlJVFYraU5qK2ppUFBobm1qQVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQ2NhUU85eC9vegpvVjlzM2hKT1VCS1Q2eHVqMWRvM2NWbjFvaG1nWHgvMUhQMlBIS2p5WW9SY3ZtRFJkd3hxeHJKODV0bEVDck1GCk02QldDVnhncmlXa2R0NjJIVTMwc0NlUzl4RE42OFgzWVQ4dFBFM2grNDFGcnliRVJOOVEwZFJZd2ZTekEzRW8KRGE4N0NDYWkyYXIyS1dMUU1JMnV4YWE0c1RKcmFzcHNEYm5RZnZJeEs5TTJWRWFLUXJ4b2tlZTNWYkhwQ0lDWApWaitHdGMvdnJlUzBleW1XWW1YbW5wSEY2d0kzV2NMRVg5azE4Z1pYR3JsWWlXUUJCZXFkTFl0ZFdJeXV5TXB1CmZVeWtkNFZ0Y0Rlcms2eC9RaHRjUHA0YWxzMXgwZG9uNTRoQk5qTUxQM3pHV01ROU1YcXpjaFhtVm9zbDQ4MW8KWEJ6eHpRYmRaaFpwCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K         server: https://192.168.56.100:6443       name: ""     contexts: null     current-context: ""     kind: Config     preferences: {}     users: null kind: ConfigMap metadata:   creationTimestamp: "2026-01-22T16:14:11Z"   name: cluster-info   namespace: kube-public   resourceVersion: "362"   uid: 460f42e0-962b-4353-8443-e17582d70fc5 kubectl -n kube-public get configmap cluster-info -o jsonpath='{.data.kubeconfig}' | grep certificate-authority-data | cut -d ':' -f2 | tr -d ' ' | base64 -d | openssl x509 -text -noout Certificate:     Data:         Version: 3 (0x2)         Serial Number: 6304399610631000039 (0x577dba0b31c58fe7)         Signature Algorithm: sha256WithRSAEncryption         Issuer: CN=kubernetes         Validity             Not Before: Jan 22 16:09:00 2026 GMT             Not After : Jan 20 16:14:00 2036 GMT         Subject: CN=kubernetes         Subject Public Key Info:             Public Key Algorithm: rsaEncryption                 Public-Key: (2048 bit)                 Modulus:                     00:cd:9a:9a:ae:f9:96:c9:5d:25:22:f2:2f:15:eb:                     7b:20:35:d4:ee:a9:e6:e4:c8:19:f9:bb:f4:c3:55:                     1e:17:d9:2a:5a:f0:42:a5:f3:d2:b6:1c:04:6c:43:                     00:79:e4:09:ae:ee:0f:58:ce:33:8c:5a:2d:ee:bf:                     8b:23:d3:2f:00:50:6a:b5:aa:40:c2:8f:c8:8f:b3:                     5a:a6:22:31:f7:c9:66:44:da:7d:3a:18:4e:3f:a8:                     88:3f:f4:f6:2e:d2:45:79:82:14:dc:f0:1a:ef:ab:                     e1:ef:22:5e:ee:8e:fa:9b:b3:6a:47:b0:07:6b:11:                     1b:15:f2:be:4f:ff:41:1d:46:e0:77:4a:ec:87:ba:                     d7:7e:a4:1a:b7:cd:84:a0:49:88:c2:89:7c:c0:4d:                     9c:13:27:b8:df:b8:05:8f:5d:61:e6:5f:db:17:c6:                     f7:ca:4f:0d:55:03:61:05:34:5d:ef:74:bf:c4:69:                     30:33:60:aa:12:f0:71:f8:f5:9c:97:3d:87:78:a2:                     d5:69:2f:2a:97:dd:32:59:a8:47:47:8e:6f:80:59:                     ec:93:d1:6f:28:8a:0f:53:37:04:10:05:d2:d5:d0:                     3f:33:8b:a2:38:51:66:77:40:e8:18:bb:f1:92:e3:                     fb:ce:1b:20:1e:02:fb:08:b0:6f:7f:93:5f:97:49:                     25:3d                 Exponent: 65537 (0x10001)         X509v3 extensions:             X509v3 Key Usage: critical                 Digital Signature, Key Encipherment, Certificate Sign             X509v3 Basic Constraints: critical                 CA:TRUE             X509v3 Subject Key Identifier:                 7D:5E:6C:17:68:E5:C1:6E:22:13:57:E8:8D:8F:E8:E2:3C:F8:67:9A             X509v3 Subject Alternative Name:                 DNS:kubernetes     Signature Algorithm: sha256WithRSAEncryption     Signature Value:         9c:69:03:bd:c7:fa:33:a1:5f:6c:de:12:4e:50:12:93:eb:1b:         a3:d5:da:37:71:59:f5:a2:19:a0:5f:1f:f5:1c:fd:8f:1c:a8:         f2:62:84:5c:be:60:d1:77:0c:6a:c6:b2:7c:e6:d9:44:0a:b3:         05:33:a0:56:09:5c:60:ae:25:a4:76:de:b6:1d:4d:f4:b0:27:         92:f7:10:cd:eb:c5:f7:61:3f:2d:3c:4d:e1:fb:8d:45:af:26:         c4:44:df:50:d1:d4:58:c1:f4:b3:03:71:28:0d:af:3b:08:26:         a2:d9:aa:f6:29:62:d0:30:8d:ae:c5:a6:b8:b1:32:6b:6a:ca:         6c:0d:b9:d0:7e:f2:31:2b:d3:36:54:46:8a:42:bc:68:91:e7:         b7:55:b1:e9:08:80:97:56:3f:86:b5:cf:ef:ad:e4:b4:7b:29:         96:62:65:e6:9e:91:c5:eb:02:37:59:c2:c4:5f:d9:35:f2:06:         57:1a:b9:58:89:64:01:05:ea:9d:2d:8b:5d:58:8c:ae:c8:ca:         6e:7d:4c:a4:77:85:6d:70:37:ab:93:ac:7f:42:1b:5c:3e:9e:         1a:96:cd:71:d1:da:27:e7:88:41:36:33:0b:3f:7c:c6:58:c4:         3d:31:7a:b3:72:15:e6:56:8b:25:e3:cd:68:5c:1c:f1:cd:06:         dd:66:16:69 kubectl -n kube-public get role NAME                                   CREATED AT kubeadm:bootstrap-signer-clusterinfo   2026-01-22T16:14:12Z system:controller:bootstrap-signer     2026-01-22T16:14:11Z kubectl -n kube-public get rolebinding NAME                                   ROLE                                        AGE kubeadm:bootstrap-signer-clusterinfo   Role/kubeadm:bootstrap-signer-clusterinfo   7m35s system:controller:bootstrap-signer     Role/system:controller:bootstrap-signer     7m36s # kubeadm init 시 생성되는 객체 - Namespace: kube-public - ConfigMap: cluster-info - Role + RoleBinding  >> 대상: system:unauthenticated (인증 안 된 사용자) >> 권한: get on configmaps/cluster-info 👉 아직 클러스터 인증서가 없는 노드(worker) 가 (kubeadm join 전) API Server에 처음 접속해서 최소 정보(엔드포인트 + CA)를 얻기 위해 필요

 

• [k8s-ctr] Flannel CNI 설치 v0.27.3
  • https://github.com/flannel-io/flannel

# 현재 클러스터는 CNI 가 구성되어 있지 않아서 NotReady 상태임 kubectl get nodes NAME      STATUS     ROLES           AGE   VERSION k8s-ctr   NotReady   control-plane   10m   v1.32.11 # 현재 k8s 클러스터에 파드 전체 CIDR 확인 kubectl describe pod -n kube-system kube-controller-manager-k8s-ctr ...     Command:       kube-controller-manager       --allocate-node-cidrs=true       --cluster-cidr=10.244.0.0/16       --service-cluster-ip-range=10.96.0.0/16       ... # 노드별 파드 CIDR 확인  kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDR}{"\n"}{end}' k8s-ctr 10.244.0.0/24 # helm 3 설치 : https://helm.sh/docs/intro/install curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | DESIRED_VERSION=v3.18.6 bash helm version version.BuildInfo{Version:"v3.18.6", GitCommit:"b76a950f6835474e0906b96c9ec68a2eff3a6430", GitTreeState:"clean", GoVersion:"go1.24.6"} # Deploying Flannel with Helm # https://github.com/flannel-io/flannel/blob/master/Documentation/configuration.md helm repo add flannel https://flannel-io.github.io/flannel "flannel" has been added to your repositories helm repo update kubectl create namespace kube-flannel namespace/kube-flannel created cat << EOF > flannel.yaml podCidr: "10.244.0.0/16" flannel:   cniBinDir: "/opt/cni/bin"   cniConfDir: "/etc/cni/net.d"   args:   - "--ip-masq"   - "--kube-subnet-mgr"   - "--iface=enp0s8"     backend: "vxlan" EOF helm install flannel flannel/flannel --namespace kube-flannel --version 0.27.3 -f flannel.yaml NAME: flannel LAST DEPLOYED: Fri Jan 23 01:30:35 2026 NAMESPACE: kube-flannel STATUS: deployed REVISION: 1 TEST SUITE: None # 확인 kubectl get no NAME      STATUS   ROLES           AGE   VERSION k8s-ctr   Ready    control-plane   21m   v1.32.11 kubectl get po -A NAMESPACE      NAME                              READY   STATUS    RESTARTS   AGE kube-flannel   kube-flannel-ds-w64qj             1/1     Running   0          72s kube-system    coredns-668d6bf9bc-gp9dk          1/1     Running   0          21m kube-system    coredns-668d6bf9bc-hgllr          1/1     Running   0          21m kube-system    etcd-k8s-ctr                      1/1     Running   0          21m kube-system    kube-apiserver-k8s-ctr            1/1     Running   0          21m kube-system    kube-controller-manager-k8s-ctr   1/1     Running   0          21m kube-system    kube-proxy-hhmmv                  1/1     Running   0          21m kube-system    kube-scheduler-k8s-ctr            1/1     Running   0          21m

 

5. [k8s-w1/w2] kubeadm 으로 k8s join

워커노드의  join 과정에서 수행되는 작업은 아래와 같습니다.

https://kubernetes.io/docs/reference/setup-tools/kubeadm/implementation-details/#kubeadm-join-phases-internal-design

Implementation details

• Kubernetes API 서버를 신뢰하도록 하는 디스커버리 단계 → Kubernetes API 서버가 노드를 신뢰하도록 하는 TLS 부트스트랩 단계

1. Preflight checks : init 과 동일
2. Discovery cluster-info 클러스터 정보 발견
   • 공개 정보 조회: 사용자가 입력한 토큰과 -discovery-token-ca-cert-hash를 사용하여 API 서버의 kube-public/cluster-info ConfigMap을 익명으로 요청합니다.
3. Shared token discovery (--discovery-token) or File/https discovery (--discovery-file)
   • 신뢰 검증: 가져온 ConfigMap 내의 CA 인증서가 사용자가 입력한 해시값(--discovery-token-ca-cert-hash sha256:xxxx)과 일치하는지 확인하여, 접속하려는 마스터 노드가 진짜인지 검증합니다.
4. TLS Bootstrap (인증서 발급 요청)
   • 부트스트랩 인증: 이제 마스터를 신뢰할 수 있으므로, 토큰을 사용해 API 서버에 정식으로 인증을 시도합니다.
   • CSR 생성: 워커 노드는 자신만의 개인키를 만들고, API 서버에 "나를 위한 인증서를 서명해달라"는 CSR(Certificate Signing Request)을 보냅니다.
   • 자동 승인: 마스터의 Certificate 발급자는 이 요청이 유효한 토큰을 통한 것임을 확인하고 자동으로 승인합니다.
5. Kubelet 설정 및 기동
   • Kubeconfig 생성: 발급받은 정식 인증서를 사용하여 /etc/kubernetes/kubelet.conf 파일을 생성합니다.
   • 노드 등록: kubelet이 이 설정을 가지고 실행되면서 API 서버에 자신을 "Node" 리소스로 등록합니다.
   • 최종 합류: 마스터는 이 노드에 kube-proxy 등을 배포하며, 노드 상태가 Ready가 될 준비를 마칩니다.

# kubeadm Configuration 파일 작성 NODEIP=$(ip -4 addr show enp0s8 | grep -oP '(?<=inet\s)\d+(\.\d+){3}') echo $NODEIP 192.168.56.101 cat << EOF > kubeadm-join.yaml apiVersion: kubeadm.k8s.io/v1beta4 kind: JoinConfiguration discovery:   bootstrapToken:     token: "123456.1234567890123456"     apiServerEndpoint: "192.168.56.100:6443"     unsafeSkipCAVerification: true nodeRegistration:   criSocket: "unix:///run/containerd/containerd.sock"   kubeletExtraArgs:     - name: node-ip       value: "$NODEIP" EOF # 클러스터 join kubeadm join --config="kubeadm-join.yaml" [preflight] Running pre-flight checks [preflight] Reading configuration from the "kubeadm-config" ConfigMap in namespace "kube-system"... [preflight] Use 'kubeadm init phase upload-config --config your-config.yaml' to re-upload it. [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-check] Waiting for a healthy kubelet at http://127.0.0.1:10248/healthz. This can take up to 4m0s [kubelet-check] The kubelet is healthy after 502.200642ms [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster. # 클러스터 노드 확인 kubectl get no -o wide NAME      STATUS   ROLES           AGE     VERSION    INTERNAL-IP      EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION                 CONTAINER-RUNTIME k8s-ctr   Ready    control-plane   41m     v1.32.11   192.168.56.100   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.x86_64   containerd://2.1.5 k8s-w1    Ready    <none>          2m26s   v1.32.11   192.168.56.101   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.x86_64   containerd://2.1.5 k8s-w2    Ready    <none>          37s     v1.32.11   192.168.56.102   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.x86_64   containerd://2.1.5 # 전체 POD 확인 kubectl get po -A NAMESPACE      NAME                              READY   STATUS    RESTARTS   AGE kube-flannel   kube-flannel-ds-bdh4j             1/1     Running   0          3m2s kube-flannel   kube-flannel-ds-rnw8b             1/1     Running   0          73s kube-flannel   kube-flannel-ds-w64qj             1/1     Running   0          21m kube-system    coredns-668d6bf9bc-gp9dk          1/1     Running   0          41m kube-system    coredns-668d6bf9bc-hgllr          1/1     Running   0          41m kube-system    etcd-k8s-ctr                      1/1     Running   0          41m kube-system    kube-apiserver-k8s-ctr            1/1     Running   0          41m kube-system    kube-controller-manager-k8s-ctr   1/1     Running   0          41m kube-system    kube-proxy-hhmmv                  1/1     Running   0          41m kube-system    kube-proxy-pmmq4                  1/1     Running   0          3m2s kube-system    kube-proxy-qlnp6                  1/1     Running   0          73s kube-system    kube-scheduler-k8s-ctr            1/1     Running   0          41m

 

이렇게 해서 Rocky Linux 10 버전에 K8S v1.32.11 클러스터를 구성하였습니다.