카테고리 없음

[K8S Deploy 5주차] kubespray를 이용한 K8S HA클러스터 구성

yu3papa 2026. 2. 7. 08:57

이번 블로그에서는 kubespray  를 이용하여 HA 방식의 k8s 클러스터를 아래와 같이 구성합니다.

External LB → HA 컨트플 플레인 노드(3대) + (Worker Client-Side LoadBalancing)

실습환경 구성

vagranat 를 이용하여 Rocky Linux 10  버전의 6대의 가상머신을 아래와 같이 준비합니다.

Vagrantfile

더보기

Vagrantfile

# Base Image  https://portal.cloud.hashicorp.com/vagrant/discover/bento/rockylinux-10.0
BOX_IMAGE = "bento/rockylinux-10.0" # "bento/rockylinux-9"
BOX_VERSION = "202510.26.0"
N = 5 # max number of Node

Vagrant.configure("2") do |config|

# Nodes 
  (1..M).each do |i|
    config.vm.define "k8s-node#{i}" do |subconfig|
      subconfig.vm.box = BOX_IMAGE
      subconfig.vm.box_version = BOX_VERSION
      subconfig.vm.provider "virtualbox" do |vb|
        vb.customize ["modifyvm", :id, "--groups", "/Kubespary-Lab"]
        vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
        vb.name = "k8s-node#{i}"
        vb.cpus = 4
        vb.memory = 2048
        vb.linked_clone = true
      end
      subconfig.vm.host_name = "k8s-node#{i}"
      subconfig.vm.network "private_network", ip: "192.168.10.1#{i}"
      subconfig.vm.network "forwarded_port", guest: 22, host: "6000#{i}", auto_correct: true, id: "ssh"
      subconfig.vm.synced_folder "./", "/vagrant", disabled: true
      subconfig.vm.provision "shell", path: "init_cfg.sh"
    end
  end

# Admin & LoadBalancer Node
    config.vm.define "admin-lb" do |subconfig|
      subconfig.vm.box = BOX_IMAGE
      subconfig.vm.box_version = BOX_VERSION
      subconfig.vm.provider "virtualbox" do |vb|
        vb.customize ["modifyvm", :id, "--groups", "/Kubespary-Lab"]
        vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
        vb.name = "admin-lb"
        vb.cpus = 2
        vb.memory = 1024
        vb.linked_clone = true
      end
      subconfig.vm.host_name = "admin-lb"
      subconfig.vm.network "private_network", ip: "192.168.10.10"
      subconfig.vm.network "forwarded_port", guest: 22, host: "60000", auto_correct: true, id: "ssh"
      subconfig.vm.synced_folder "./", "/vagrant", disabled: true
      subconfig.vm.provision "shell", path: "admin-lb.sh"
    end

end

 

admin-lb.sh

  • HAProxy 설치/설정, NFS Server 설치/설정, ssh 설정, hosts 도메인 설정, kubespray git clone, kubectl/k9s/helm 설치 등
#!/usr/bin/env bash

echo ">>>> Initial Config Start <<<<"


echo "[TASK 1] Change Timezone and Enable NTP"
timedatectl set-local-rtc 0
timedatectl set-timezone Asia/Seoul


echo "[TASK 2] Disable firewalld and selinux"
systemctl disable --now firewalld >/dev/null 2>&1
setenforce 0
sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config


echo "[TASK 3] Setting Local DNS Using Hosts file"
sed -i '/^127\.0\.\(1\|2\)\.1/d' /etc/hosts
echo "192.168.10.10 k8s-api-srv.admin-lb.com admin-lb" >> /etc/hosts
for (( i=1; i<=$1; i++  )); do echo "192.168.10.1$i k8s-node$i" >> /etc/hosts; done


echo "[TASK 4] Delete default routing - enp0s9 NIC" # setenforce 0 설정 필요
nmcli connection modify enp0s9 ipv4.never-default yes
nmcli connection up enp0s9 >/dev/null 2>&1


echo "[TASK 5] Install kubectl"
cat << EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.32/rpm/repodata/repomd.xml.key
exclude=kubectl
EOF
dnf install -y -q kubectl --disableexcludes=kubernetes >/dev/null 2>&1


echo "[TASK 6] Install HAProxy"
dnf install -y haproxy >/dev/null 2>&1

cat << EOF > /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log         127.0.0.1 local2

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

    # utilize system-wide crypto-policies
    ssl-default-bind-ciphers PROFILE=SYSTEM
    ssl-default-server-ciphers PROFILE=SYSTEM

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  tcplog
    option                  dontlognull
    option http-server-close
    #option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

# ---------------------------------------------------------------------
# Kubernetes API Server Load Balancer Configuration
# ---------------------------------------------------------------------
frontend k8s-api
    bind *:6443
    mode tcp
    option tcplog
    default_backend k8s-api-backend

backend k8s-api-backend
    mode tcp
    option tcp-check
    option log-health-checks
    timeout client 3h
    timeout server 3h
    balance roundrobin
    server k8s-node1 192.168.10.11:6443 check check-ssl verify none inter 10000
    server k8s-node2 192.168.10.12:6443 check check-ssl verify none inter 10000
    server k8s-node3 192.168.10.13:6443 check check-ssl verify none inter 10000

# ---------------------------------------------------------------------
# HAProxy Stats Dashboard - http://192.168.10.10:9000/haproxy_stats
# ---------------------------------------------------------------------
listen stats
    bind *:9000
    mode http
    stats enable
    stats uri /haproxy_stats
    stats realm HAProxy\ Statistic
    stats admin if TRUE

# ---------------------------------------------------------------------
# Configure the Prometheus exporter - curl http://192.168.10.10:8405/metrics
# ---------------------------------------------------------------------
frontend prometheus
    bind *:8405
    mode http
    http-request use-service prometheus-exporter if { path /metrics }
    no log
EOF
systemctl enable --now haproxy >/dev/null 2>&1


echo "[TASK 7] Install nfs-utils"
dnf install -y nfs-utils >/dev/null 2>&1
systemctl enable --now nfs-server >/dev/null 2>&1
mkdir -p /srv/nfs/share
chown nobody:nobody /srv/nfs/share
chmod 755 /srv/nfs/share
echo '/srv/nfs/share *(rw,async,no_root_squash,no_subtree_check)' > /etc/exports
exportfs -rav


echo "[TASK 8] Install packages"
dnf install -y python3-pip git sshpass >/dev/null 2>&1


echo "[TASK 9] Setting SSHD"
echo "root:qwe123" | chpasswd

cat << EOF >> /etc/ssh/sshd_config
PermitRootLogin yes
PasswordAuthentication yes
EOF
systemctl restart sshd >/dev/null 2>&1


echo "[TASK 10] Setting SSH Key"
ssh-keygen -t rsa -N "" -f /root/.ssh/id_rsa >/dev/null 2>&1

sshpass -p 'qwe123' ssh-copy-id -o StrictHostKeyChecking=no root@192.168.10.10  >/dev/null 2>&1  # cat /root/.ssh/authorized_keys
for (( i=1; i<=$1; i++  )); do sshpass -p 'qwe123' ssh-copy-id -o StrictHostKeyChecking=no root@192.168.10.1$i >/dev/null 2>&1 ; done

ssh -o StrictHostKeyChecking=no root@admin-lb hostname >/dev/null 2>&1
for (( i=1; i<=$1; i++  )); do sshpass -p 'qwe123' ssh -o StrictHostKeyChecking=no root@k8s-node$i hostname >/dev/null 2>&1 ; done


echo "[TASK 11] Clone Kubespray Repository"
git clone -b v2.29.1 https://github.com/kubernetes-sigs/kubespray.git /root/kubespray >/dev/null 2>&1

cp -rfp /root/kubespray/inventory/sample /root/kubespray/inventory/mycluster
cat << EOF > /root/kubespray/inventory/mycluster/inventory.ini
[kube_control_plane]
k8s-node1 ansible_host=192.168.10.11 ip=192.168.10.11 etcd_member_name=etcd1
k8s-node2 ansible_host=192.168.10.12 ip=192.168.10.12 etcd_member_name=etcd2
k8s-node3 ansible_host=192.168.10.13 ip=192.168.10.13 etcd_member_name=etcd3

[etcd:children]
kube_control_plane

[kube_node]
k8s-node4 ansible_host=192.168.10.14 ip=192.168.10.14
#k8s-node5 ansible_host=192.168.10.15 ip=192.168.10.15
EOF


echo "[TASK 12] Install Python Dependencies"
pip3 install -r /root/kubespray/requirements.txt >/dev/null 2>&1 # pip3 list


echo "[TASK 13] Install K9s"
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
wget -P /tmp https://github.com/derailed/k9s/releases/latest/download/k9s_linux_${CLI_ARCH}.tar.gz  >/dev/null 2>&1
tar -xzf /tmp/k9s_linux_${CLI_ARCH}.tar.gz -C /tmp
chown root:root /tmp/k9s
mv /tmp/k9s /usr/local/bin/
chmod +x /usr/local/bin/k9s


echo "[TASK 14] Install kubecolor"
dnf install -y -q 'dnf-command(config-manager)' >/dev/null 2>&1
dnf config-manager --add-repo https://kubecolor.github.io/packages/rpm/kubecolor.repo >/dev/null 2>&1
dnf install -y -q kubecolor >/dev/null 2>&1


echo "[TASK 15] Install Helm"
curl -fsSL https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | DESIRED_VERSION=v3.18.6 bash >/dev/null 2>&1


echo "[TASK 16] ETC"
echo "sudo su -" >> /home/vagrant/.bashrc


echo ">>>> Initial Config End <<<<"

 

init_cfg.sh

  • ssh 설정, hosts 도메인 설정 등
#!/usr/bin/env bash

echo ">>>> Initial Config Start <<<<"


echo "[TASK 1] Change Timezone and Enable NTP"
timedatectl set-local-rtc 0
timedatectl set-timezone Asia/Seoul


echo "[TASK 2] Disable firewalld and selinux"
systemctl disable --now firewalld >/dev/null 2>&1
setenforce 0
sed -i 's/^SELINUX=enforcing/SELINUX=permissive/' /etc/selinux/config


echo "[TASK 3] Disable and turn off SWAP & Delete swap partitions"
swapoff -a
sed -i '/swap/d' /etc/fstab
sfdisk --delete /dev/sda 2 >/dev/null 2>&1
partprobe /dev/sda >/dev/null 2>&1


echo "[TASK 4] Config kernel & module"
cat << EOF > /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
modprobe overlay >/dev/null 2>&1
modprobe br_netfilter >/dev/null 2>&1

cat << EOF >/etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
sysctl --system >/dev/null 2>&1


echo "[TASK 5] Setting Local DNS Using Hosts file"
sed -i '/^127\.0\.\(1\|2\)\.1/d' /etc/hosts
echo "192.168.10.10 k8s-api-srv.admin-lb.com admin-lb" >> /etc/hosts
for (( i=1; i<=$1; i++  )); do echo "192.168.10.1$i k8s-node$i" >> /etc/hosts; done


echo "[TASK 6] Delete default routing - enp0s9 NIC" # setenforce 0 설정 필요
nmcli connection modify enp0s9 ipv4.never-default yes
nmcli connection up enp0s9 >/dev/null 2>&1


echo "[TASK 7] Setting SSHD"
echo "root:qwe123" | chpasswd

cat << EOF >> /etc/ssh/sshd_config
PermitRootLogin yes
PasswordAuthentication yes
EOF
systemctl restart sshd >/dev/null 2>&1


echo "[TASK 8] Install packages"
dnf install -y git nfs-utils >/dev/null 2>&1


echo "[TASK 9] ETC"
echo "sudo su -" >> /home/vagrant/.bashrc


echo ">>>> Initial Config End <<<<"

 

실습 환경 배포

# 실습용 디렉터리 생성
mkdir k8s-ha-kubespary
cd k8s-ha-kubespary

# 파일 다운로드
curl -O https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/k8s-ha-kubespary/Vagrantfile
curl -O https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/k8s-ha-kubespary/admin-lb.sh
curl -O https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/k8s-ha-kubespary/init_cfg.sh

# 실습 환경 배포
vagrant up

 

실습은 "admin-lb" 머신에서 root 계정으로 진행합니다. "vagrant ssh admin-lb" 를 수행하여 root 계정의 ssh 접속을 수행합니다.

# 관리 대상 노드 통신 확인
cat /etc/hosts
192.168.10.10 k8s-api-srv.admin-lb.com admin-lb
192.168.10.11 k8s-node1
192.168.10.12 k8s-node2
192.168.10.13 k8s-node3
192.168.10.14 k8s-node4
192.168.10.15 k8s-node5

# ssh 접속확인
for i in {0..5}; do echo ">> k8s-node$i <<"; ssh 192.168.10.1$i hostname; echo; done
for i in {1..5}; do echo ">> k8s-node$i <<"; ssh k8s-node$i hostname; echo; done

# 파이썬 버전 정보 확인
python -V && pip -V
Python 3.12.9
pip 23.3.2 from /usr/lib/python3.12/site-packages/pip (python 3.12)

# kubespary 작업 디렉터리 및 파일 확인
tree /root/kubespray/ -L 2

cd /root/kubespray/
cat ansible.cfg
[ssh_connection]
pipelining=True
ssh_args = -o ControlMaster=auto -o ControlPersist=30m -o ConnectionAttempts=100 -o UserKnownHostsFile=/dev/null
#control_path = ~/.ssh/ansible-%%r@%%h:%%p
[defaults]
# https://github.com/ansible/ansible/issues/56930 (to ignore group names with - and .)
force_valid_group_names = ignore

host_key_checking=False
gathering = smart
fact_caching = jsonfile
fact_caching_connection = /tmp
fact_caching_timeout = 86400
timeout = 300
stdout_callback = default
display_skipped_hosts = no
library = ./library
callbacks_enabled = profile_tasks
roles_path = roles:$VIRTUAL_ENV/usr/local/share/kubespray/roles:$VIRTUAL_ENV/usr/local/share/ansible/roles:/usr/share/kubespray/roles
deprecation_warnings=False
inventory_ignore_extensions = ~, .orig, .bak, .ini, .cfg, .retry, .pyc, .pyo, .creds, .gpg
[inventory]
ignore_patterns = artifacts, credentials


cat /root/kubespray/inventory/mycluster/inventory.ini
[kube_control_plane]
k8s-node1 ansible_host=192.168.10.11 ip=192.168.10.11 etcd_member_name=etcd1
k8s-node2 ansible_host=192.168.10.12 ip=192.168.10.12 etcd_member_name=etcd2
k8s-node3 ansible_host=192.168.10.13 ip=192.168.10.13 etcd_member_name=etcd3

[etcd:children]
kube_control_plane

[kube_node]
k8s-node4 ansible_host=192.168.10.14 ip=192.168.10.14
#k8s-node5 ansible_host=192.168.10.15 ip=192.168.10.15


# NFS Server 정보 확인
systemctl status nfs-server --no-pager
tree /srv/nfs/share/
exportfs -rav
exporting *:/srv/nfs/share

cat /etc/exports
/srv/nfs/share *(rw,async,no_root_squash,no_subtree_check)


# admin-lb IP에 TCP 6443 호출(인입)시, 백엔드 대상인 k8s-node1~3에 각각 분산 전달 설정 확인
cat /etc/haproxy/haproxy.cfg
...
# ---------------------------------------------------------------------
# Kubernetes API Server Load Balancer Configuration
# ---------------------------------------------------------------------
frontend k8s-api
    bind *:6443
    mode tcp
    option tcplog
    default_backend k8s-api-backend

backend k8s-api-backend
    mode tcp
    option tcp-check
    option log-health-checks
    timeout client 3h
    timeout server 3h
    balance roundrobin
    server k8s-node1 192.168.10.11:6443 check check-ssl verify none inter 10000
    server k8s-node2 192.168.10.12:6443 check check-ssl verify none inter 10000
    server k8s-node3 192.168.10.13:6443 check check-ssl verify none inter 10000


# HAProxy 상태 확인
systemctl status haproxy.service --no-pager
journalctl -u haproxy.service --no-pager
ss -tnlp | grep haproxy
LISTEN 0      3000         0.0.0.0:6443       0.0.0.0:*    users:(("haproxy",pid=4915,fd=7))  # k8s api loadbalancer
LISTEN 0      3000         0.0.0.0:9000       0.0.0.0:*    users:(("haproxy",pid=4915,fd=8))  # haproxy stat dashbaord
LISTEN 0      3000         0.0.0.0:8405       0.0.0.0:*    users:(("haproxy",pid=4915,fd=9))  # metrics exporter

# 통계 페이지 접속
http://192.168.10.10:9000/haproxy_stats

현재 ‘HAProxy 통계 페이지’에 backend 대상 서버는 모두 DOWN 상태

# (참고) 프로테우스 메트릭 엔드포인트 접속
curl http://192.168.10.10:8405/metrics
~~~
haproxy_resolver_too_big{resolver="default",nameserver="168.126.63.1"} 0
# HELP haproxy_resolver_truncated Truncated
# TYPE haproxy_resolver_truncated gauge
haproxy_resolver_truncated{resolver="default",nameserver="168.126.63.1"} 0
# HELP haproxy_resolver_outdated Outdated
# TYPE haproxy_resolver_outdated gauge
haproxy_resolver_outdated{resolver="default",nameserver="168.126.63.1"} 0

 

Kubespary 를 통한 k8s 배포

cd /root/kubespray/

##### 설정파일을 실습환경에 맞게 수정 #####
# k8s_cluster.yml # for every node in the cluster (not etcd when it's separate)
sed -i 's|kube_owner: kube|kube_owner: root|g' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
sed -i 's|kube_network_plugin: calico|kube_network_plugin: flannel|g' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
sed -i 's|kube_proxy_mode: ipvs|kube_proxy_mode: iptables|g' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
sed -i 's|enable_nodelocaldns: true|enable_nodelocaldns: false|g' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
grep -iE 'kube_owner|kube_network_plugin:|kube_proxy_mode|enable_nodelocaldns:' inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
kube_owner: root
kube_network_plugin: flannel
kube_proxy_mode: iptables
enable_nodelocaldns: false

## coredns autoscaler 미설치 --> 실습환경의 메모리 절약을 위해, 실제 환경에서는 불필요한 작업
echo "enable_dns_autoscaler: false" >> inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml

# flannel 설정 수정
echo "flannel_interface: enp0s9" >> inventory/mycluster/group_vars/k8s_cluster/k8s-net-flannel.yml
grep "^[^#]" inventory/mycluster/group_vars/k8s_cluster/k8s-net-flannel.yml
flannel_interface: enp0s9

# addons
sed -i 's|metrics_server_enabled: false|metrics_server_enabled: true|g' inventory/mycluster/group_vars/k8s_cluster/addons.yml
grep -iE 'metrics_server_enabled:' inventory/mycluster/group_vars/k8s_cluster/addons.yml
## cat roles/kubernetes-apps/metrics_server/defaults/main.yml                            # 메트릭서버 관련 디폴트 변수 참고
## cat roles/kubernetes-apps/metrics_server/templates/metrics-server-deployment.yaml.j2  # jinja2 템플릿 파일 참고
echo "metrics_server_requests_cpu: 25m"     >> inventory/mycluster/group_vars/k8s_cluster/addons.yml
echo "metrics_server_requests_memory: 16Mi" >> inventory/mycluster/group_vars/k8s_cluster/addons.yml


# 지원 버전 정보 확인
cat roles/kubespray_defaults/vars/main/checksums.yml | grep -i kube -A40

# 배포: 아래처럼 반드시 ~/kubespray 디렉토리에서 ansible-playbook 를 실행, 8분 정도 소요
# 배포 전, Task 목록 확인
ansible-playbook -i inventory/mycluster/inventory.ini -v cluster.yml --list-tasks 
...(생략)...
      kubernetes/preinstall : Configure dhclient hooks for resolv.conf (RH-only)        TAGS: [bootstrap_os, resolvconf]
      kubernetes/preinstall : Remove kubespray specific config from dhclient config     TAGS: [bootstrap_os, resolvconf]
      kubernetes/preinstall : Remove kubespray specific dhclient hook   TAGS: [bootstrap_os, resolvconf]
      kubernetes/preinstall : Flush handlers    TAGS: [resolvconf]
      kubernetes/preinstall : Check if we are running inside a Azure VM TAGS: [bootstrap_os, resolvconf]
      Run calico checks TAGS: [resolvconf]


ANSIBLE_FORCE_COLOR=true ansible-playbook -i inventory/mycluster/inventory.ini -v cluster.yml -e kube_version="1.32.9" | tee kubespray_install.log

# 설치후  k8s api 호출 확인 : IP, Domain
for i in {1..3}; do echo ">> k8s-node$i <<"; curl -sk https://192.168.10.1$i:6443/version | grep Version; echo; done
>> k8s-node1 <<
  "gitVersion": "v1.32.9",
  "goVersion": "go1.23.12",

>> k8s-node2 <<
  "gitVersion": "v1.32.9",
  "goVersion": "go1.23.12",

>> k8s-node3 <<
  "gitVersion": "v1.32.9",
  "goVersion": "go1.23.12",

# kubeconfig 작업
mkdir /root/.kube
scp k8s-node1:/root/.kube/config /root/.kube/

# API Server 주소를 localhost에서 컨트롤 플레인 1번 node P로 변경 : 1번 node 장애 시, 직접 수동으로 다른 node IP 변경 필요.
# 현재 상태에서는 haproxy를 이용하여 127.0.0.1도 kube api-server 와 통신이 가능한 상태임
sed -i 's/127.0.0.1/192.168.10.11/g' /root/.kube/config
혹은 
sed -i 's/127.0.0.1/192.168.10.12/g' /root/.kube/config
sed -i 's/127.0.0.1/192.168.10.13/g' /root/.kube/config

# 클러스터 노드 확인
kubectl get node -owide
NAME        STATUS   ROLES           AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                        KERNEL-VERSION                 CONTAINER-RUNTIME
k8s-node1   Ready    control-plane   22m   v1.32.9   192.168.10.11   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.x86_64   containerd://2.1.5
k8s-node2   Ready    control-plane   22m   v1.32.9   192.168.10.12   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.x86_64   containerd://2.1.5
k8s-node3   Ready    control-plane   21m   v1.32.9   192.168.10.13   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.x86_64   containerd://2.1.5
k8s-node4   Ready    <none>          19m   v1.32.9   192.168.10.14   <none>        Rocky Linux 10.0 (Red Quartz)   6.12.0-55.39.1.el10_0.x86_64   containerd://2.1.5

# etcd 정보 확인 : etcd name 확인
ssh k8s-node1 etcdctl.sh member list -w table
+------------------+---------+-------+----------------------------+----------------------------+------------+
|        ID        | STATUS  | NAME  |         PEER ADDRS         |        CLIENT ADDRS        | IS LEARNER |
+------------------+---------+-------+----------------------------+----------------------------+------------+
|  8b0ca30665374b0 | started | etcd3 | https://192.168.10.13:2380 | https://192.168.10.13:2379 |      false |
| 2106626b12a4099f | started | etcd2 | https://192.168.10.12:2380 | https://192.168.10.12:2379 |      false |
| c6702130d82d740f | started | etcd1 | https://192.168.10.11:2380 | https://192.168.10.11:2379 |      false |
+------------------+---------+-------+----------------------------+----------------------------+------------+

# 첫번째 Control-Plane은 kubespray 입장에서 가장 중요함
for i in {1..3}; do echo ">> k8s-node$i <<"; ssh k8s-node$i etcdctl.sh endpoint status -w table; echo; done
>> k8s-node1 <<
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|    ENDPOINT    |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 127.0.0.1:2379 | c6702130d82d740f |  3.5.25 |  6.3 MB |      true |      false |         5 |       9069 |               9069 |        |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

>> k8s-node2 <<
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|    ENDPOINT    |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 127.0.0.1:2379 | 2106626b12a4099f |  3.5.25 |  6.3 MB |     false |      false |         5 |       9074 |               9074 |        |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

>> k8s-node3 <<
+----------------+-----------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|    ENDPOINT    |       ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+-----------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 127.0.0.1:2379 | 8b0ca30665374b0 |  3.5.25 |  6.3 MB |     false |      false |         5 |       9075 |               9075 |        |
+----------------+-----------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

# 자동완성 및 단축키 설정
source <(kubectl completion bash)
alias k=kubectl
alias kc=kubecolor
complete -F __start_kubectl k
echo 'source <(kubectl completion bash)' >> /etc/profile
echo 'alias k=kubectl' >> /etc/profile
echo 'alias kc=kubecolor' >> /etc/profile
echo 'complete -F __start_kubectl k' >> /etc/profile

 

Kubespray 변수 우선순위 구조 및 검색

▼ Command-line values (for example, -u my_user, these are not variables)
▼ Role defaults (as defined in Role directory structure) 1
▼ Inventory file or script group vars 2
▼ Inventory group_vars/all 3
▼ Playbook group_vars/all 3
▼ Inventory group_vars/* 3
▼ Playbook group_vars/* 3
▼ Inventory file or script host vars 2
▼ Inventory host_vars/* 3
▼ Playbook host_vars/* 3
▼ Host facts and cached set_facts 4
▼ Play vars
▼ Play vars_prompt
▼ Play vars_files
▼ Role vars (as defined in Role directory structure)
▼ Block vars (for tasks in block only)
▼ Task vars (for the task only)
▼ include_vars
▼ Registered vars and set_facts
▼ Role (and include_role) params
▼ include params
▼ Extra vars (for example, -e "user=my_user")(always win precedence)

 

  • Kubespray 변수 우선순위 구조 (Override Flow)
[ 낮은 우선순위 ]
┌─────────────────────────────────────────────┐
│ roles/*/defaults/main.yml                   │  ← Kubespray role 기본값
│   (예: bin_dir, kube_version 기본값)        │
└─────────────────────────────────────────────┘
                ⬇ override

┌─────────────────────────────────────────────┐
│ roles/*/vars/main.yml                       │  ← role 내부 강제 변수 (웬만해선 안 건드림)
└─────────────────────────────────────────────┘
                ⬇ override

┌─────────────────────────────────────────────┐
│ inventory/mycluster/group_vars/all/*.yml    │  ← 전체 노드 공통 설정 # 99% 여기서 조절
│ inventory/mycluster/group_vars/k8s_cluster/*.yml
│ inventory/mycluster/group_vars/etcd.yml     │
└─────────────────────────────────────────────┘
                ⬇ override

┌─────────────────────────────────────────────┐
│ inventory/mycluster/host_vars/<node>.yml    │  ← 특정 노드에만 적용 # 특정 노드만 다르게 쓸 때
└─────────────────────────────────────────────┘
                ⬇ override

┌─────────────────────────────────────────────┐
│ playbook vars (vars:, vars_files:)          │  ← reset.yml / cluster.yml 내부 vars # 실행하는 플레이북에 선언된 경우
└─────────────────────────────────────────────┘
                ⬇ override

┌─────────────────────────────────────────────┐
│ --extra-vars (-e)                           │  ← CLI에서 준 값 (최강자)
│   ex) -e kube_version=v1.29.3               │
└─────────────────────────────────────────────┘
[ 높은 우선순위 ]
# 전체 노드 공통 설정(예시) : 99% 여기서 조절
inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
kube_version: v1.29.3
kube_network_plugin: cilium

# 특정 노드에만 적용(예시) : 특정 노드만 다르게 쓸 때
inventory/mycluster/host_vars/k8s-ctr1.yml
node_labels:
  node-role.kubernetes.io/control-plane: "true"

# 실행하는 플레이북에 선언된 경우(예) : 아래 경우 플레이북을 import 할 때만 적용되는 “로컬 변수 override”
cat playbooks/scale.yml | grep 'Install etcd' -A5
- name: Install etcd
  vars:                               # inventory/group_vars 보다 우선순위가 높음, 이 playbook import 범위 안에서만 유효
    etcd_cluster_setup: false         # etcd 신규 클러스터 bootstrap 로직 비활성화
    etcd_events_cluster_setup: false  # 이벤트 전용 etcd 클러스터 구성 안 함
  import_playbook: install_etcd.yml   # install_etcd.yml이라는 별도의 플레이북 파일을 현재 위치에 포함시켜 실행하라는 명령


# 예시) dns autoscaler 미설치 하기 위해 검색
grep -Rni "autoscaler" inventory/mycluster/ playbooks/ roles/ -A2 -B1
grep -Rni "autoscaler" inventory/mycluster/ playbooks/ roles/ --include="*.yml" -A2 -B1
roles/kubespray_defaults/defaults/main/main.yml:130:# Enable dns autoscaler
roles/kubespray_defaults/defaults/main/main.yml:131:enable_dns_autoscaler: true

 

  • control 컴포넌트 확인
# kube-apiserver 정보 확인
ssh k8s-node1 cat /etc/kubernetes/manifests/kube-apiserver.yaml
...
spec:
  containers:
  - command:
    - kube-apiserver
    - --advertise-address=192.168.10.11
    ...
    - --apiserver-count=3
    - --authorization-mode=Node,RBAC
    - '--bind-address=::'
    ...
    - --etcd-servers=https://192.168.10.11:2379,https://192.168.10.12:2379,https://192.168.10.13:2379
    # kubeadm 직접 설치 시 정보와 비교 : --etcd-servers=https://127.0.0.1:2379

# (참고) lease 정보 확인 : apiserver 는 모두 Active 동작, kcm / scheduler 은 1대만 리더 역할
kubectl get lease -n kube-system
NAME                                   HOLDER                                                                      AGE
apiserver-3jsrenrspxlfjr2cvxzde6qwdi   apiserver-3jsrenrspxlfjr2cvxzde6qwdi_25f81820-25e1-4e92-b27d-116b9429c353   5h12m
apiserver-syplgv2uz3ssgciixtnxs4xeza   apiserver-syplgv2uz3ssgciixtnxs4xeza_62b92e03-f014-4b16-9de2-b4f82de0f861   5h12m
apiserver-z2kpjb5k5ch6lznxmv3gnpujmy   apiserver-z2kpjb5k5ch6lznxmv3gnpujmy_c6523dd7-2550-462f-80c1-ab8d16a38781   5h12m
kube-controller-manager                k8s-node2_5d90d703-85ad-4f58-a413-3f484214c4f1                              5h12m
kube-scheduler                         k8s-node2_c3bdf688-9708-4313-8827-a727f2bded21                              5h12m

# kube-controller-manager 정보 확인 : 'bind-address=::' IPv6/IPv4 지원
ssh k8s-node1 cat /etc/kubernetes/manifests/kube-controller-manager.yaml
    - --allocate-node-cidrs=true
    - --cluster-cidr=10.233.64.0/18
    - --node-cidr-mask-size-ipv4=24
    - --service-cluster-ip-range=10.233.0.0/18
    - '--bind-address=::'
    - --leader-elect=true
   
# kube-scheduler 정보 확인
ssh k8s-node1 cat /etc/kubernetes/manifests/kube-scheduler.yaml
    - '--bind-address=::'
    - --leader-elect=true


# 인증서 정보 확인 : ctrl1번 노드만 super-admin.conf 확인
for i in {1..3}; do echo ">> k8s-node$i <<"; ssh k8s-node$i ls -l /etc/kubernetes/super-admin.conf ; echo; done

## 보통 coredns 는 service cide 의 10번째 ip를 사용. 하지만 kubespary 경우 보통 3번째 ip를 사용.
for i in {1..3}; do echo ">> k8s-node$i <<"; ssh k8s-node$i kubeadm certs check-expiration ; echo; done
...
>> k8s-node3 <<
...
W0128 22:47:42.339227  106954 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [10.233.0.3]
CERTIFICATE                  EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                   Jan 28, 2027 08:17 UTC   364d            ca                      no
apiserver                    Jan 28, 2027 08:17 UTC   364d            ca                      no
apiserver-kubelet-client     Jan 28, 2027 08:17 UTC   364d            ca                      no
controller-manager.conf      Jan 28, 2027 08:17 UTC   364d            ca                      no
front-proxy-client           Jan 28, 2027 08:17 UTC   364d            front-proxy-ca          no
scheduler.conf               Jan 28, 2027 08:17 UTC   364d            ca                      no
!MISSING! super-admin.conf # ctrl1번 노드만 super-admin.conf 확인

# 파드에 coredns service (Cluster) IP 확인
kubectl exec -it -n kube-system nginx-proxy-k8s-node4 -- cat /etc/resolv.conf
search kube-systehttp://m.svc.cluster.local svc.cluster.local cluster.local default.svc.cluster.local
nameserver 10.233.0.3
options ndots:5

# kubespray는 coredns의 서비스명이 kube-dns가 아니라 coredns이며, clusterIP 가 10.233.0.3
kubectl get svc -n kube-system coredns
NAME      TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
coredns   ClusterIP   10.233.0.3   <none>        53/UDP,53/TCP,9153/TCP   5h34m

kubectl get cm -n kube-system kubelet-config -o yaml | grep clusterDNS -A2
    clusterDNS:
    - 10.233.0.3
    clusterDomain: cluster.local

# (참고) kubeadm 통한 직접 설정 시 : 10.96.0.0/16 대역 중 10번째 IP 사용 확인
kubectl get svc,ep -n kube-system
NAME               TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
service/kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   4h50m
...

# kubespray task에 의해서 호스트에서도 서비스명 도메인 질의를 위해 ns 최상단 추가 등 확인
ssh k8s-node1 cat /etc/resolv.conf
# Generated by NetworkManager
search default.svc.cluster.local svc.cluster.local
nameserver 10.233.0.3
nameserver 168.126.63.1
nameserver 8.8.8.8
options ndots:2 timeout:2 attempts:2

## 현재 k8s join 되지 않는 노드는 기본 dns 설정 상태
ssh k8s-node5 cat /etc/resolv.conf
# Generated by NetworkManager
nameserver 168.126.63.1
nameserver 8.8.8.8

# kubeadm-config cm 정보 확인
kubectl get cm -n kube-system kubeadm-config -o yaml
...
    apiVersion: kubeadhttp://m.k8s.io/v1beta4
    caCertificateValidityPeriod: 87600h0m0s
    certificateValidityPeriod: 8760h0m0s
    certificatesDir: /etc/kubernetes/ssl
    clusterName: cluster.local
    controlPlaneEndpoint: 192.168.10.11:6443    # << ctrl2~3은??
    ...
    etcd:
      external:                                 # kubeadm 이 파드로 배포한것이 아니며 systemd unit 구성.
        caFile: /etc/ssl/etcd/ssl/ca.pem
        certFile: /etc/ssl/etcd/ssl/node-k8s-node1.pem
        endpoints:
        - https://192.168.10.11:2379
        - https://192.168.10.12:2379
        - https://192.168.10.13:2379
        keyFile: /etc/ssl/etcd/ssl/node-k8s-node1-key.pem
    imageRepository: registry.k8s.io
    kind: ClusterConfiguration
    kubernetesVersion: v1.32.9
    networking:
      dnsDomain: cluster.local
      podSubnet: 10.233.64.0/18
      serviceSubnet: 10.233.0.0/18
...

# kubeadm 과 동일하게 kubelet node 최초 join 시 CSR 사용 확인
kubectl get csr
NAME        AGE   SIGNERNAME                                    REQUESTOR                 REQUESTEDDURATION   CONDITION
csr-4h74m   20m   kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:74gawm   <none>              Approved,Issued
csr-7t6rz   21m   kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:cu2j8d   <none>              Approved,Issued
csr-j6dj7   21m   kubernetes.io/kube-apiserver-client-kubelet   system:node:k8s-node1     <none>              Approved,Issued
csr-mxn26   21m   kubernetes.io/kube-apiserver-client-kubelet   system:bootstrap:d3e4e2   <none>              Approved,Issued

 

K8S API 엔드포인트 따른 클러스터 구성 분류

  • [Case0] 단일 컨트플 플레인 노드 1대 + 워커 노드 1대
  • [Case1] HA 컨트플 플레인 노드(3대) + (Worker Client-Side LoadBalancing)
  • [Case2] External LB → HA 컨트플 플레인 노드(3대) + (Worker Client-Side LoadBalancing)
  • [Case3] HA 컨트롤 플레인 노드(3대) ↔ External LB(단일 엔드포인트) ← 워커 노드

[Case0] 단일 컨트플 플레인 노드 1대 + 워커 노드 1대

[Case1] HA 컨트플 플레인 노드(3대) + (Worker Client-Side LoadBalancing)

https://sirzzang.github.io/kubernetes/Kubernetes-Kubespray-05-00/

  • 이번 블로그의 실습 환경임
  • 워커노드에서 k8s apiserver 를 부하분산 호출을 위해서는 Client-Side LB를 사용!
  • Cilium 과 같은 데몬셋에서 배포된 파드들이 k8s api endpoint 호출을 위해서, 동일한 127.0.0.1::6443 로컬 엔드포인트도 사용 가능!
    • 컨트롤 플레인 노드 컴포넌트 → k8s api endpoint 분석
# apiserver static 파드의 bind-address 에 '::' 확인 
kubectl describe pod -n kube-system kube-apiserver-k8s-node1 | grep -E 'address|secure-port'
Annotations:          kubeadhttp://m.kubernetes.io/kube-apiserver.advertise-address.endpoint: 192.168.10.11:6443
      --advertise-address=192.168.10.11
      --secure-port=6443
      --bind-address=::
      
ssh k8s-node1 ss -tnlp | grep 6443
LISTEN 0      4096               *:6443             *:*    users:(("kube-apiserver",pid=26124,fd=3))

ssh k8s-node1 ip -br -4 addr
ssh k8s-node1 curl -sk https://127.0.0.1:6443/version | grep gitVersion
ssh k8s-node1 curl -sk https://192.168.10.11:6443/version | grep gitVersion
ssh k8s-node1 curl -sk https://10.0.2.15:6443/version | grep gitVersion
  "gitVersion": "v1.32.9",
  

# admin 자격증명(client) -> api-server 호출 시 엔드포인트 정보 확인
ssh k8s-node1 cat /etc/kubernetes/admin.conf | grep server
    server: https://127.0.0.1:6443

# super-admin 자격증명(client) -> api-server 호출 시 엔드포인트 정보 확인
ssh k8s-node1 cat /etc/kubernetes/super-admin.conf | grep server
    server: https://192.168.10.11:6443


# kubelet(client) -> api-server 호출 시 엔드포인트 정보 확인 : https://127.0.0.1:6443
ssh k8s-node1 cat /etc/kubernetes/kubelet.conf
ssh k8s-node1 cat /etc/kubernetes/kubelet.conf | grep server
    server: https://127.0.0.1:6443

# kube-proxy(client) -> api-server 호출 시 엔드포인트 정보 확인
kc get cm -n kube-system kube-proxy -o yaml
k get cm -n kube-system kube-proxy -o yaml | grep server
        server: https://127.0.0.1:6443

# kube-controller-manager(client) -> api-server 호출 시 엔드포인트 정보 확인
ssh k8s-node1 cat /etc/kubernetes/controller-manager.conf | grep server
    server: https://127.0.0.1:6443

# kube-scheduler(client) -> api-server 호출 시 엔드포인트 정보 확인
ssh k8s-node1 cat /etc/kubernetes/scheduler.conf | grep server
    server: https://127.0.0.1:6443
  • 워커노드 컴포넌트 → k8s api endpoint 분석
# worker(kubeclt, kube-proxy) -> k8s api

# 워커노드에서 정보 확인
ssh k8s-node4 crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME                ATTEMPT             POD ID              POD                               NAMESPACE
3c09f930b22b0       5a91d90f47ddf       15 minutes ago      Running             nginx-proxy         0                   81b36842732ba       nginx-proxy-k8s-node4             kube-system
...

ssh k8s-node4 cat /etc/nginx/nginx.conf
error_log stderr notice;

worker_processes 2;
worker_rlimit_nofile 130048;
worker_shutdown_timeout 10s;
...
stream {
  upstream kube_apiserver {
    least_conn;
    server 192.168.10.11:6443;
    server 192.168.10.12:6443;
    server 192.168.10.13:6443;
    }

  server {
    listen        127.0.0.1:6443;
    proxy_pass    kube_apiserver;
    proxy_timeout 10m;
    proxy_connect_timeout 1s;
  }
}

http {
...
  server {
    listen 8081;
    location /healthz {
      access_log off;
      return 200;
...

ssh k8s-node4 curl -s localhost:8081/healthz -I
HTTP/1.1 200 OK
Server: nginx

# 워커노드에서 -> Client-Side LB를 사용해서 k8s api 호출 시도
ssh k8s-node4 curl -sk https://127.0.0.1:6443/version | grep Version
  "gitVersion": "v1.32.9",
  "goVersion": "go1.23.12",

ssh k8s-node4 ss -tnlp | grep nginx
LISTEN 0      511          0.0.0.0:8081       0.0.0.0:*    users:(("nginx",pid=15043,fd=6),("nginx",pid=15042,fd=6),("nginx",pid=15016,fd=6))
LISTEN 0      511        127.0.0.1:6443       0.0.0.0:*    users:(("nginx",pid=15043,fd=5),("nginx",pid=15042,fd=5),("nginx",pid=15016,fd=5))

# kubelet(client) -> api-server 호출 시 엔드포인트 정보 확인 : https://localhost:6443
ssh k8s-node4 cat /etc/kubernetes/kubelet.conf
ssh k8s-node4 cat /etc/kubernetes/kubelet.conf | grep server
    server: https://localhost:6443

# kube-proxy(client) -> api-server 호출 시 엔드포인트 정보 확인
kc get cm -n kube-system kube-proxy -o yaml
kubectl get cm -n kube-system kube-proxy -o yaml | grep 'kubeconfig.conf:' -A18
  kubeconfig.conf: |-
    apiVersion: v1
    kind: Config
    clusters:
    - cluster:
        certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        server: https://127.0.0.1:6443
      name: default
...


# nginx.conf 생성 Task
tree roles/kubernetes/node/tasks/loadbalancer
cat roles/kubernetes/node/tasks/loadbalancer/nginx-proxy.yml
...
- name: Nginx-proxy | Write nginx-proxy configuration
  template:
    src: "loadbalancer/nginx.conf.j2"
    dest: "{{ nginx_config_dir }}/nginx.conf"
    owner: root
    mode: "0755"
    backup: true

# nginx.conf jinja2 템플릿 파일
cat roles/kubernetes/node/templates/loadbalancer/nginx.conf.j2
error_log stderr notice;

worker_processes 2;
worker_rlimit_nofile 130048;
worker_shutdown_timeout 10s;

events {
  multi_accept on;
  use epoll;
  worker_connections 16384;
}

stream {
  upstream kube_apiserver {
    least_conn;                 # RoundRobin 이 아닌 Lease_conn이 기본 설정인 이유는?
    {% for host in groups['kube_control_plane'] -%}
    server {{ hostvars[host]['main_access_ip'] | ansible.utils.ipwrap }}:{{ kube_apiserver_port }};
    {% endfor -%}
  }

  server {
    listen        127.0.0.1:{{ loadbalancer_apiserver_port|default(kube_apiserver_port) }};
    {% if ipv6_stack -%}
    listen        [::1]:{{ loadbalancer_apiserver_port|default(kube_apiserver_port) }};
    {% endif -%}
    proxy_pass    kube_apiserver;
    proxy_timeout 10m;
    proxy_connect_timeout 1s;
  }
}

# nginx static pod 매니페스트 파일 확인
cat roles/kubernetes/node/templates/manifests/nginx-proxy.manifest.j2
apiVersion: v1
kind: Pod
metadata:
  name: {{ loadbalancer_apiserver_pod_name }}
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
    k8s-app: kube-nginx
  annotations:
    nginx-cfg-checksum: "{{ nginx_stat.stat.checksum }}"
...

[Case2] External LB → HA 컨트플 플레인 노드(3대) + (Worker Client-Side LoadBalancing)

https://sirzzang.github.io/kubernetes/Kubernetes-Kubespray-05-00/

실습 준비

  • 샘플 애플리케이션 배포, 반복 호출
# 샘플 애플리케이션 배포
cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webpod
spec:
  replicas: 2
  selector:
    matchLabels:
      app: webpod
  template:
    metadata:
      labels:
        app: webpod
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - sample-app
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: webpod
        image: traefik/whoami
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: webpod
  labels:
    app: webpod
spec:
  selector:
    app: webpod
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
    nodePort: 30003
  type: NodePort
EOF

 

# 배포 확인
kubectl get deploy,svc,ep webpod -owide
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE   CONTAINERS   IMAGES           SELECTOR
deployment.apps/webpod   2/2     2            0           65s   webpod       traefik/whoami   app=webpod

NAME             TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE   SELECTOR
service/webpod   NodePort   10.233.30.166   <none>        80:30003/TCP   65s   app=webpod

NAME               ENDPOINTS   AGE
endpoints/webpod   <none>      64s

[admin-lb] # IP는 node 작업에 따라 변경
while true; do curl -s http://192.168.10.14:30003 | grep Hostname; sleep 1; done

# 성공
ssh k8s-node1 curl -s webpod -I
HTTP/1.1 200 OK

# 성공
ssh k8s-node1 curl -s webpod.default -I
HTTP/1.1 200 OK

# 실패
ssh k8s-node1 curl -s webpod.default.svc -I
ssh k8s-node1 curl -s webpod.default.svc.cluster -I

# 성공
ssh k8s-node1 curl -s webpod.default.svc.cluster.local -I
HTTP/1.1 200 OK
  • (장애 재현) 만약 컨트롤 플레인 1번 노드 장애 발생 시 영향도
# [admin-lb] kubeconfig 자격증명 사용 시 정보 확인
cat /root/.kube/config | grep server
    server: https://192.168.10.11:6443

    
# 모니터링 : 신규 터미널 4개
# ----------------------
## [admin-lb]
while true; do kubectl get node ; echo ; curl -sk https://192.168.10.12:6443/version | grep gitVersion ; sleep 1; echo ; done

## [k8s-node2]
watch -d kubectl get pod -n kube-system
kubectl logs -n kube-system nginx-proxy-k8s-node4 -f

## [k8s-node4] 
while true; do curl -sk https://127.0.0.1:6443/version | grep gitVersion ; date; sleep 1; echo ; done
# ----------------------


# 장애 재현 
[k8s-node1] poweroff


# [k8s-node2]
kubectl logs -n kube-system nginx-proxy-k8s-node4 -f
2026/01/28 12:47:08 [error] 20#20: *3145 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: 127.0.0.1:6443, upstream: "192.168.10.11:6443", bytes from/to client:0/0, bytes from/to upstream:0/0
2026/01/28 12:47:08 [warn] 20#20: *3145 upstream server temporarily disabled while connecting to upstream, client: 127.0.0.1, server: 127.0.0.1:6443, upstream: "192.168.10.11:6443", bytes from/to client:0/0, bytes from/to upstream:0/0

# [k8s-node4] 하지만 백엔드 대상 서버가 나머지 2대가 있으니 아래 요청 처리 정상!
while true; do curl -sk https://127.0.0.1:6443/version | grep gitVersion ; date; sleep 1; echo ; done
  "gitVersion": "v1.32.9",

# [admin-lb] 아래 자격증명 서버 정보 수정 필요
while true; do kubectl get node ; echo ; curl -sk https://192.168.10.12:6443/version | grep gitVersion ; sleep 1; echo ; done
Unable to connect to the server: dial tcp 192.168.10.11:6443: connect: no route to host  # << 요건 실패!
  "gitVersion": "v1.32.9",  # << 요건 성공!

sed -i 's/192.168.10.11/192.168.10.12/g' /root/.kube/config

while true; do kubectl get node ; echo ; curl -sk https://192.168.10.12:6443/version | grep gitVersion ; sleep 1; echo ; done
NAME        STATUS     ROLES           AGE     VERSION
k8s-node1   NotReady   control-plane   4h35m   v1.32.9
k8s-node2   Ready      control-plane   4h35m   v1.32.9
k8s-node3   Ready      control-plane   4h35m   v1.32.9
k8s-node4   Ready      <none>          4h34m   v1.32.9
  "gitVersion": "v1.32.9",
  • External LB → HA 컨트플 플레인 노드(3대) : k8s apiserver 호출 설정
curl -sk https://192.168.10.10:6443/version | grep gitVersion
  "gitVersion": "v1.32.9",

sed -i 's/192.168.10.12/192.168.10.10/g' /root/.kube/config

# 인증서 SAN list 확인
kubectl get node
E0128 23:53:41.079370   70802 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: Get \"https://192.168.10.10:6443/api?timeout=32s\": tls: failed to verify certificate: x509: certificate is valid for 10.233.0.1, 192.168.10.11, 127.0.0.1, ::1, 192.168.10.12, 192.168.10.13, 10.0.2.15, fd17:625c:f037:2:a00:27ff:fe90:eaeb, not 192.168.10.10"

# 인증서에 SAN 정보 확인
ssh k8s-node1 cat /etc/kubernetes/ssl/apiserver.crt | openssl x509 -text -noout
...

ssh k8s-node1 kubectl get cm -n kube-system kubeadm-config -o yaml
    apiServer:
      certSANs:
      - kubernetes
      - kubernetes.default
      - kubernetes.default.svc
      - kubernetes.default.svc.cluster.local
      - 10.233.0.1
      - localhost
      - 127.0.0.1
      - ::1
      - k8s-node1
      - k8s-node2
      - k8s-node3
      - lb-apiserver.kubernetes.local
      - 192.168.10.11
      - 192.168.10.12
      - 192.168.10.13
      - 10.0.2.15
      - fd17:625c:f037:2:a00:27ff:fe90:eaeb

# 인증서 SAN 에 'IP, Domain' 추가
echo "supplementary_addresses_in_ssl_keys: [192.168.10.10, k8s-api-srv.admin-lb.com]" >> inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
grep "^[^#]" inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml
  
# ansible-playbook -i inventory/mycluster/inventory.ini -v cluster.yml --tags "kube-apiserver" --list-tasks
# ansible-playbook -i inventory/mycluster/inventory.ini -v cluster.yml --tags "kubeadm" --list-tasks
# ansible-playbook -i inventory/mycluster/inventory.ini -v cluster.yml --tags "facts" --list-tasks
ansible-playbook -i inventory/mycluster/inventory.ini -v cluster.yml --tags "control-plane" --list-tasks
...
 play #10 (kube_control_plane): Install the control plane TAGS: []
    tasks:
      ...
      kubernetes/control-plane : Kubeadm | aggregate all SANs TAGS: [control-plane, facts]
...


# (신규터미널) 모니터링
[k8s-node4]
while true; do curl -sk https://127.0.0.1:6443/version | grep gitVersion ; date ; sleep 1; echo ; done


# 1분 이내 완료
ansible-playbook -i inventory/mycluster/inventory.ini -v cluster.yml --tags "control-plane" --limit kube_control_plane -e kube_version="1.32.9"
Gather minimal facts ------------------------------------------------------------------------------------------------------- 2.00s
kubernetes/control-plane : Kubeadm | Check apiserver.crt SAN hosts --------------------------------------------------------- 1.57s
kubernetes/control-plane : Kubeadm | Check apiserver.crt SAN IPs ----------------------------------------------------------- 1.33s
kubernetes/control-plane : Backup old certs and keys ----------------------------------------------------------------------- 1.26s
Gather necessary facts (hardware) ------------------------------------------------------------------------------------------ 0.98s
kubernetes/control-plane : Install | Copy kubectl binary from download dir ------------------------------------------------- 0.95s
kubernetes/preinstall : Create other directories of root owner ------------------------------------------------------------- 0.92s
win_nodes/kubernetes_patch : debug ----------------------------------------------------------------------------------------- 0.84s
kubernetes/control-plane : Backup old confs -------------------------------------------------------------------------------- 0.83s
kubernetes/control-plane : Update server field in component kubeconfigs ---------------------------------------------------- 0.78s
kubernetes/control-plane : Kubeadm | Create kubeadm config ----------------------------------------------------------------- 0.76s
kubernetes/preinstall : Create kubernetes directories ---------------------------------------------------------------------- 0.67s
kubernetes/control-plane : Kubeadm | regenerate apiserver cert 2/2 --------------------------------------------------------- 0.50s
kubernetes/control-plane : Renew K8S control plane certificates monthly 2/2 ------------------------------------------------ 0.46s
kubernetes/control-plane : Create kube-scheduler config -------------------------------------------------------------------- 0.41s
Gather necessary facts (network) ------------------------------------------------------------------------------------------- 0.38s
kubernetes/control-plane : Install script to renew K8S control plane certificates ------------------------------------------ 0.37s
kubernetes/control-plane : Kubeadm | regenerate apiserver cert 1/2 --------------------------------------------------------- 0.34s
kubernetes/control-plane : Kubeadm | aggregate all SANs -------------------------------------------------------------------- 0.29s
kubernetes/control-plane : Check which kube-control nodes are already members of the cluster ------------------------------- 0.28s


# 192.168.10.10 엔드포인트 요청 성공!
kubectl get node -v=6
...
I0129 00:17:13.825729   81610 round_trippers.go:560] GET https://192.168.10.10:6443/api/v1/nodes?limit=500 200 OK in 8 milliseconds
NAME        STATUS   ROLES           AGE     VERSION
k8s-node1   Ready    control-plane   7h      v1.32.9
k8s-node2   Ready    control-plane   7h      v1.32.9
k8s-node3   Ready    control-plane   7h      v1.32.9
k8s-node4   Ready    <none>          6h59m   v1.32.9

# ip, domain 둘 다 확인
sed -i 's/192.168.10.10/k8s-api-srv.admin-lb.com/g' /root/.kube/config

# 추가 확인
ssh k8s-node1 cat /etc/kubernetes/ssl/apiserver.crt | openssl x509 -text -noout
X509v3 Subject Alternative Name:
  DNS:k8s-api-srv.admin-lb.com, DNS:k8s-node1, DNS:k8s-node2, DNS:k8s-node3, DNS:kubernetes, DNS:kubernetes.default,
  DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, DNS:lb-apiserver.kubernetes.local, 
  DNS:localhost, IP Address:10.233.0.1, IP Address:192.168.10.11, IP Address:127.0.0.1, 
  IP Address:0:0:0:0:0:0:0:1, IP Address:192.168.10.10, IP Address:192.168.10.12, IP Address:192.168.10.13, 
  IP Address:10.0.2.15, IP Address:FD17:625C:F037:2:A00:27FF:FE90:EAEB

# 해당 cm은 최초 설치 후 자동 업데이트 X, 업그레이드에 활용된다고 하니, 위 처럼 kubeadm config 변경 시 직접 cm도 같이 변경해두자.
kubectl get cm -n kube-system kubeadm-config -o yaml
...

kubectl edit cm -n kube-system kubeadm-config # or k9s -> cm kube-system
...


[Case3] HA 컨트롤 플레인 노드(3대) ↔ External LB(단일 엔드포인트) ← 워커 노드

https://sirzzang.github.io/kubernetes/Kubernetes-Kubespray-05-00/

 

 

'카테고리 없음'의 다른글

  • 현재글[K8S Deploy 5주차] kubespray를 이용한 K8S HA클러스터 구성

티스토리툴바