Kubernetes

Cilium Study [1기] (8주차) - Cilium Security

yu3papa 2025. 9. 4. 09:07

0. 실습환경 구성

  • Vagrantfile : 가상머신 정의, 부팅 시 초기 프로비저닝 설정
# Variables
K8SV = '1.33.4-1.1' # Kubernetes Version : apt list -a kubelet , ex) 1.32.5-1.1
CONTAINERDV = '1.7.27-1' # Containerd Version : apt list -a containerd.io , ex) 1.6.33-1
CILIUMV = '1.18.1' # Cilium CNI Version : https://github.com/cilium/cilium/tags
N = 2 # max number of worker nodes

# Base Image  https://portal.cloud.hashicorp.com/vagrant/discover/bento/ubuntu-24.04
BOX_IMAGE = "bento/ubuntu-24.04"
BOX_VERSION = "202508.03.0"

Vagrant.configure("2") do |config|
#-ControlPlane Node
    config.vm.define "k8s-ctr" do |subconfig|
      subconfig.vm.box = BOX_IMAGE
      
      subconfig.vm.box_version = BOX_VERSION
      subconfig.vm.provider "virtualbox" do |vb|
        vb.customize ["modifyvm", :id, "--groups", "/Cilium-Lab"]
        vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
        vb.name = "k8s-ctr"
        vb.cpus = 4
        vb.memory = 2560
        vb.linked_clone = true
      end
      subconfig.vm.host_name = "k8s-ctr"
      subconfig.vm.network "private_network", ip: "192.168.10.100"
      subconfig.vm.network "forwarded_port", guest: 22, host: 60000, auto_correct: true, id: "ssh"
      subconfig.vm.synced_folder "./", "/vagrant", disabled: true
      subconfig.vm.provision "shell", path: "https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/cilium-study/8w/init_cfg.sh", args: [ K8SV, CONTAINERDV ]
      subconfig.vm.provision "shell", path: "https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/cilium-study/8w/k8s-ctr.sh", args: [ N, CILIUMV, K8SV ]
    end

#-Worker Nodes Subnet1
  (1..N).each do |i|
    config.vm.define "k8s-w#{i}" do |subconfig|
      subconfig.vm.box = BOX_IMAGE
      subconfig.vm.box_version = BOX_VERSION
      subconfig.vm.provider "virtualbox" do |vb|
        vb.customize ["modifyvm", :id, "--groups", "/Cilium-Lab"]
        vb.customize ["modifyvm", :id, "--nicpromisc2", "allow-all"]
        vb.name = "k8s-w#{i}"
        vb.cpus = 4
        vb.memory = 2048
        vb.linked_clone = true
      end
      subconfig.vm.host_name = "k8s-w#{i}"
      subconfig.vm.network "private_network", ip: "192.168.10.10#{i}"
      subconfig.vm.network "forwarded_port", guest: 22, host: "6000#{i}", auto_correct: true, id: "ssh"
      subconfig.vm.synced_folder "./", "/vagrant", disabled: true
      subconfig.vm.provision "shell", path: "https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/cilium-study/8w/init_cfg.sh", args: [ K8SV, CONTAINERDV]
      subconfig.vm.provision "shell", path: "https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/cilium-study/8w/k8s-w.sh"
    end
  end

end

 

  • init_cfg.sh
#!/usr/bin/env bash

echo ">>>> Initial Config Start <<<<"

echo "[TASK 1] Setting Profile & Bashrc"
echo 'alias vi=vim' >> /etc/profile
echo "sudo su -" >> /home/vagrant/.bashrc
ln -sf /usr/share/zoneinfo/Asia/Seoul /etc/localtime # Change Timezone


echo "[TASK 2] Disable AppArmor"
systemctl stop ufw && systemctl disable ufw >/dev/null 2>&1
systemctl stop apparmor && systemctl disable apparmor >/dev/null 2>&1


echo "[TASK 3] Disable and turn off SWAP"
swapoff -a && sed -i '/swap/s/^/#/' /etc/fstab


echo "[TASK 4] Install Packages"
apt update -qq >/dev/null 2>&1
apt-get install apt-transport-https ca-certificates curl gpg -y -qq >/dev/null 2>&1

# Download the public signing key for the Kubernetes package repositories.
mkdir -p -m 755 /etc/apt/keyrings
K8SMMV=$(echo $1 | sed -En 's/^([0-9]+\.[0-9]+)\..*/\1/p')
curl -fsSL https://pkgs.k8s.io/core:/stable:/v$K8SMMV/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v$K8SMMV/deb/ /" >> /etc/apt/sources.list.d/kubernetes.list
curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null

# packets traversing the bridge are processed by iptables for filtering
echo 1 > /proc/sys/net/ipv4/ip_forward
echo "net.ipv4.ip_forward = 1" >> /etc/sysctl.d/k8s.conf

# enable br_netfilter for iptables 
modprobe br_netfilter
modprobe overlay
echo "br_netfilter" >> /etc/modules-load.d/k8s.conf
echo "overlay" >> /etc/modules-load.d/k8s.conf


echo "[TASK 5] Install Kubernetes components (kubeadm, kubelet and kubectl)"
# Update the apt package index, install kubelet, kubeadm and kubectl, and pin their version
apt update >/dev/null 2>&1

# apt list -a kubelet ; apt list -a containerd.io
apt-get install -y kubelet=$1 kubectl=$1 kubeadm=$1 containerd.io=$2 >/dev/null 2>&1
apt-mark hold kubelet kubeadm kubectl >/dev/null 2>&1

# containerd configure to default and cgroup managed by systemd
containerd config default > /etc/containerd/config.toml
sed -i 's/SystemdCgroup = false/SystemdCgroup = true/g' /etc/containerd/config.toml

# avoid WARN&ERRO(default endpoints) when crictl run  
cat <<EOF > /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
image-endpoint: unix:///run/containerd/containerd.sock
EOF

# ready to install for k8s 
systemctl restart containerd && systemctl enable containerd
systemctl enable --now kubelet


echo "[TASK 6] Install Packages & Helm"
export DEBIAN_FRONTEND=noninteractive
apt-get install -y bridge-utils sshpass net-tools conntrack ngrep tcpdump ipset arping wireguard jq yq tree bash-completion unzip kubecolor termshark >/dev/null 2>&1
curl -s https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash >/dev/null 2>&1


echo "[TASK 7] Install pwru"
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
wget https://github.com/cilium/pwru/releases/download/v1.0.10/pwru-linux-${CLI_ARCH}.tar.gz >/dev/null 2>&1
tar -xvzf pwru-linux-${CLI_ARCH}.tar.gz >/dev/null 2>&1
mv pwru /usr/local/bin/pwru >/dev/null 2>&1


echo ">>>> Initial Config End <<<<"
  • k8s-ctr.sh
#!/usr/bin/env bash

echo ">>>> K8S Controlplane config Start <<<<"

echo "[TASK 1] Initial Kubernetes"
curl --silent -o /root/kubeadm-init-ctr-config.yaml https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/cilium-study/kubeadm-init-ctr-config.yaml
K8SMMV=$(echo $3 | sed -En 's/^([0-9]+\.[0-9]+\.[0-9]+).*/\1/p')
sed -i "s/K8S_VERSION_PLACEHOLDER/v${K8SMMV}/g" /root/kubeadm-init-ctr-config.yaml
kubeadm init --config="/root/kubeadm-init-ctr-config.yaml"  >/dev/null 2>&1


echo "[TASK 2] Setting kube config file"
mkdir -p /root/.kube
cp -i /etc/kubernetes/admin.conf /root/.kube/config
chown $(id -u):$(id -g) /root/.kube/config


echo "[TASK 3] Source the completion"
echo 'source <(kubectl completion bash)' >> /etc/profile
echo 'source <(kubeadm completion bash)' >> /etc/profile


echo "[TASK 4] Alias kubectl to k"
echo 'alias k=kubectl' >> /etc/profile
echo 'alias kc=kubecolor' >> /etc/profile
echo 'complete -F __start_kubectl k' >> /etc/profile


echo "[TASK 5] Install Kubectx & Kubens"
git clone https://github.com/ahmetb/kubectx /opt/kubectx >/dev/null 2>&1
ln -s /opt/kubectx/kubens /usr/local/bin/kubens
ln -s /opt/kubectx/kubectx /usr/local/bin/kubectx


echo "[TASK 6] Install Kubeps & Setting PS1"
git clone https://github.com/jonmosco/kube-ps1.git /root/kube-ps1 >/dev/null 2>&1
cat <<"EOT" >> /root/.bash_profile
source /root/kube-ps1/kube-ps1.sh
KUBE_PS1_SYMBOL_ENABLE=true
function get_cluster_short() {
  echo "$1" | cut -d . -f1
}
KUBE_PS1_CLUSTER_FUNCTION=get_cluster_short
KUBE_PS1_SUFFIX=') '
PS1='$(kube_ps1)'$PS1
EOT
kubectl config rename-context "kubernetes-admin@kubernetes" "HomeLab" >/dev/null 2>&1


echo "[TASK 7] Install Cilium CNI"
NODEIP=$(ip -4 addr show eth1 | grep -oP '(?<=inet\s)\d+(\.\d+){3}')
helm repo add cilium https://helm.cilium.io/ >/dev/null 2>&1
helm repo update >/dev/null 2>&1
helm install cilium cilium/cilium --version $2 --namespace kube-system \
--set k8sServiceHost=192.168.10.100 --set k8sServicePort=6443 \
--set ipam.mode="cluster-pool" --set ipam.operator.clusterPoolIPv4PodCIDRList={"172.20.0.0/16"} --set ipv4NativeRoutingCIDR=172.20.0.0/16 \
--set routingMode=native --set autoDirectNodeRoutes=true --set endpointRoutes.enabled=true --set directRoutingSkipUnreachable=true \
--set kubeProxyReplacement=true --set bpf.masquerade=true --set installNoConntrackIptablesRules=true \
--set endpointHealthChecking.enabled=false --set healthChecking=false \
--set hubble.enabled=true --set hubble.relay.enabled=true --set hubble.ui.enabled=true \
--set hubble.ui.service.type=NodePort --set hubble.ui.service.nodePort=30003 \
--set prometheus.enabled=true --set operator.prometheus.enabled=true --set hubble.metrics.enableOpenMetrics=true \
--set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,httpV2:exemplars=true;labelsContext=source_ip\,source_namespace\,source_workload\,destination_ip\,destination_namespace\,destination_workload\,traffic_direction}" \
--set ingressController.enabled=true --set ingressController.loadbalancerMode=shared --set loadBalancer.l7.backend=envoy \
--set localRedirectPolicy=true --set l2announcements.enabled=true \
--set operator.replicas=1 --set ciliumEndpointSlice.enabled=true --set debug.enabled=true >/dev/null 2>&1


echo "[TASK 8] Install Cilium / Hubble CLI"
CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt)
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-${CLI_ARCH}.tar.gz >/dev/null 2>&1
tar xzvfC cilium-linux-${CLI_ARCH}.tar.gz /usr/local/bin
rm cilium-linux-${CLI_ARCH}.tar.gz

HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
HUBBLE_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then HUBBLE_ARCH=arm64; fi
curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-${HUBBLE_ARCH}.tar.gz >/dev/null 2>&1
tar xzvfC hubble-linux-${HUBBLE_ARCH}.tar.gz /usr/local/bin
rm hubble-linux-${HUBBLE_ARCH}.tar.gz


echo "[TASK 9] Remove node taint"
kubectl taint nodes k8s-ctr node-role.kubernetes.io/control-plane-


echo "[TASK 10] local DNS with hosts file"
echo "192.168.10.100 k8s-ctr" >> /etc/hosts
for (( i=1; i<=$1; i++  )); do echo "192.168.10.10$i k8s-w$i" >> /etc/hosts; done


echo "[TASK 11] Dynamically provisioning persistent local storage with Kubernetes"
kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/v0.0.31/deploy/local-path-storage.yaml >/dev/null 2>&1
kubectl patch storageclass local-path -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' >/dev/null 2>&1


echo "[TASK 12] Install Prometheus & Grafana"
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.18.1/examples/kubernetes/addons/prometheus/monitoring-example.yaml >/dev/null 2>&1
kubectl patch svc -n cilium-monitoring prometheus -p '{"spec": {"type": "NodePort", "ports": [{"port": 9090, "targetPort": 9090, "nodePort": 30001}]}}' >/dev/null 2>&1
kubectl patch svc -n cilium-monitoring grafana -p '{"spec": {"type": "NodePort", "ports": [{"port": 3000, "targetPort": 3000, "nodePort": 30002}]}}' >/dev/null 2>&1


echo "[TASK 13] Install Metrics-server"
helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/ >/dev/null 2>&1
helm upgrade --install metrics-server metrics-server/metrics-server --set 'args[0]=--kubelet-insecure-tls' -n kube-system  >/dev/null 2>&1


echo "[TASK 14] Install k9s"
CLI_ARCH=amd64
if [ "$(uname -m)" = "aarch64" ]; then CLI_ARCH=arm64; fi
wget https://github.com/derailed/k9s/releases/latest/download/k9s_linux_${CLI_ARCH}.deb -O /tmp/k9s_linux_${CLI_ARCH}.deb  >/dev/null 2>&1
apt install /tmp/k9s_linux_${CLI_ARCH}.deb  >/dev/null 2>&1


echo ">>>> K8S Controlplane Config End <<<<"
  • k8s-w.sh
#!/usr/bin/env bash

echo ">>>> K8S Node config Start <<<<"


echo "[TASK 1] K8S Controlplane Join"
curl --silent -o /root/kubeadm-join-worker-config.yaml https://raw.githubusercontent.com/gasida/vagrant-lab/refs/heads/main/cilium-study/2w/kubeadm-join-worker-config.yaml
NODEIP=$(ip -4 addr show eth1 | grep -oP '(?<=inet\s)\d+(\.\d+){3}')
sed -i "s/NODE_IP_PLACEHOLDER/${NODEIP}/g" /root/kubeadm-join-worker-config.yaml
kubeadm join --config="/root/kubeadm-join-worker-config.yaml" > /dev/null 2>&1


echo ">>>> K8S Node config End <<<<"
# 프로메테우스 접속
http://192.168.10.100:30001

# 그라파나 접속
http://192.168.10.100:30002

# 허블 UI 접속
http://192.168.10.100:30003

 

1. Cilium Security

cilium은 Layer 3, 4, 5 계층에 대해 보안을 제공합니다.

https://docs.cilium.io/en/stable/security/network/identity/

    • 모든 엔드포인트에 Identity 가 할당되며, Identity 는 Labels 과 클러스터 내에 유일한 ID로 구성됩니다.
    • 같은 보안정책을 사용하면 Identity 가 같습니다.
    • 그래서 Identiy 를 기반으로 Network 정책을 적용하게 됩니다.
  • 보안 정책은 세션 기반 프로토콜에 대해 상태 저장 정책 적용되어, 응답 패킷은 자동으로 허용됨(https://docs.cilium.io/en/stable/security/network/policyenforcement/#policy-enforcement)
  • 보안 정책은 수신 or 송신 시 적용됨
  • 기본 보안 정책
    • https://docs.cilium.io/en/stable/security/network/policyenforcement/#default-security-policy)
    • 정책이 로드되지 않은 경우, 정책 적용이 명시적으로 활성화되지 않은 한 모든 통신을 허용하는 것이 기본 동작입니다.
    • 첫 번째 정책 규칙이 로드되는 즉시 정책 적용이 자동으로 활성화되며, 모든 통신은 허용 목록에 추가되어야 하며, 그렇지 않으면 관련 패킷이 삭제됩니다.
    • 마찬가지로, 엔드포인트에 L4 정책이 적용되지 않으면 모든 포트와의 통신이 허용됩니다.
    • 엔드포인트에 하나 이상의 L4 정책을 연결하면 명시적으로 허용하지 않는 한 포트에 대한 모든 연결이 차단됩니다.

Application  레벨의 Layer 7 정책은 Envoy Proxy를 이용합니다.

Cilium은 POD간의 통신에 대해 암호화 통신(IpSec, WireGuard)을 투명하게 Enable/Disable 기능을 제공합니다.

 

Identity

  • 모든 엔드포인트에 Identity 가 할당되며, Identity 는 Labels 과 클러스터 내에 유일한 ID로 구성됩니다.
  • 엔드포인트에는 Security Relevant Labels에 일치하는 ID가 할당.
  • 엔드포인트들이 동일한 Security Relevant Labels 사용 시 동일한 ID를 공유.
kubectl get ciliumendpoints.cilium.io -n kube-system
NAME                              SECURITY IDENTITY   ENDPOINT STATE   IPV4           IPV6
coredns-674b8bbfcf-gn28v          63630               ready            172.20.0.35
coredns-674b8bbfcf-t7w4r          63630               ready            172.20.0.126
hubble-relay-fdd49b976-vzdww      4648                ready            172.20.0.221
hubble-ui-655f947f96-rzrb6        31044               ready            172.20.0.230
metrics-server-5dd7b49d79-6lccw   39864               ready            172.20.0.104

kubectl get ciliumidentities.cilium.io
NAME    NAMESPACE            AGE
31044   kube-system          51m
3187    local-path-storage   51m
39864   kube-system          51m
4648    kube-system          51m
50494   cilium-monitoring    51m
63630   kube-system          51m
666     cilium-monitoring    51m

 

  • 용어 정리
  • 엔드포인트의 ID는 엔드포인트에서 파생된 포드 또는 컨테이너와 연관된 레이블을기반으로 결정됩니다.
  • 파드 또는 컨테이너가 시작되면 Cilium은 컨테이너 런타임에서 수신한 이벤트를 기반으로 네트워크에서 포드 또는 컨테이너를 나타내는 엔드포인트를 생성합니다 .
  • 다음 단계로 Cilium은 생성된 엔드포인트의 ID를 확인합니다 . 포드 또는 컨테이너의 레이블이변경될 때마다 ID가 재확인되고 필요에 따라 자동으로 수정됩니다.
kubectl get ciliumidentities.cilium.io 39864 -o yaml | yq
{
  "apiVersion": "cilium.io/v2",
  "kind": "CiliumIdentity",
  "metadata": {
    "creationTimestamp": "2025-09-04T00:19:19Z",
    "generation": 1,
    "labels": {
      "io.kubernetes.pod.namespace": "kube-system"
    },
    "name": "39864",
    "resourceVersion": "944",
    "uid": "e017cec6-71b3-46e9-950e-c4b0a148d611"
  },
  "security-labels": {
    "k8s:app.kubernetes.io/instance": "metrics-server",
    "k8s:app.kubernetes.io/name": "metrics-server",
    "k8s:io.ciliuhttp://m.k8s.namespace.labels.kubernetes.io/metadata.name": "kube-system",
    "k8s:io.ciliuhttp://m.k8s.policy.cluster": "default",
    "k8s:io.ciliuhttp://m.k8s.policy.serviceaccount": "metrics-server",  <-- Cilium 이 실행될때 관리하는 레이블
    "k8s:io.kubernetes.pod.namespace": "kube-system"
  }
}

# 아래 결과에서 1~10번까지는 예약된 레이블임
kubectl exec -it -n kube-system ds/cilium -- cilium identity list
ID      LABELS
1       reserved:host
        reserved:kube-apiserver
2       reserved:world
3       reserved:unmanaged
4       reserved:health
5       reserved:init
6       reserved:remote-node
7       reserved:kube-apiserver
        reserved:remote-node
8       reserved:ingress
9       reserved:world-ipv4
10      reserved:world-ipv6
666     k8s:app=prometheus
        k8s:io.ciliuhttp://m.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring
        k8s:io.ciliuhttp://m.k8s.policy.cluster=default
        k8s:io.ciliuhttp://m.k8s.policy.serviceaccount=prometheus-k8s
        k8s:io.kubernetes.pod.namespace=cilium-monitoring
3187    k8s:app=local-path-provisioner
        k8s:io.ciliuhttp://m.k8s.namespace.labels.kubernetes.io/metadata.name=local-path-storage
        k8s:io.ciliuhttp://m.k8s.policy.cluster=default
        k8s:io.ciliuhttp://m.k8s.policy.serviceaccount=local-path-provisioner-service-account
        k8s:io.kubernetes.pod.namespace=local-path-storage
4648    k8s:app.kubernetes.io/name=hubble-relay
        k8s:app.kubernetes.io/part-of=cilium
        k8s:io.ciliuhttp://m.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system
        k8s:io.ciliuhttp://m.k8s.policy.cluster=default
        k8s:io.ciliuhttp://m.k8s.policy.serviceaccount=hubble-relay
        k8s:io.kubernetes.pod.namespace=kube-system
        k8s:k8s-app=hubble-relay
31044   k8s:app.kubernetes.io/name=hubble-ui
        k8s:app.kubernetes.io/part-of=cilium
        k8s:io.ciliuhttp://m.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system
        k8s:io.ciliuhttp://m.k8s.policy.cluster=default
        k8s:io.ciliuhttp://m.k8s.policy.serviceaccount=hubble-ui
        k8s:io.kubernetes.pod.namespace=kube-system
        k8s:k8s-app=hubble-ui
39864   k8s:app.kubernetes.io/instance=metrics-server
        k8s:app.kubernetes.io/name=metrics-server
        k8s:io.ciliuhttp://m.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system
        k8s:io.ciliuhttp://m.k8s.policy.cluster=default
        k8s:io.ciliuhttp://m.k8s.policy.serviceaccount=metrics-server
        k8s:io.kubernetes.pod.namespace=kube-system
50494   k8s:app=grafana
        k8s:io.ciliuhttp://m.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring
        k8s:io.ciliuhttp://m.k8s.policy.cluster=default
        k8s:io.ciliuhttp://m.k8s.policy.serviceaccount=default
        k8s:io.kubernetes.pod.namespace=cilium-monitoring
63630   k8s:io.ciliuhttp://m.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system
        k8s:io.ciliuhttp://m.k8s.policy.cluster=default
        k8s:io.ciliuhttp://m.k8s.policy.serviceaccount=coredns
        k8s:io.kubernetes.pod.namespace=kube-system
        k8s:k8s-app=kube-dns

kubectl get pod -n kube-system -l k8s-app=kube-dns --show-labels
NAME                       READY   STATUS    RESTARTS   AGE   LABELS
coredns-674b8bbfcf-gn28v   1/1     Running   0          60m   k8s-app=kube-dns,pod-template-hash=674b8bbfcf
coredns-674b8bbfcf-t7w4r   1/1     Running   0          60m   k8s-app=kube-dns,pod-template-hash=674b8bbfcf

 

Special Identities

  • Cilium에서 관리하는 모든 엔드포인트에는 ID가 할당됩니다.
  • Cilium에서 관리하지 않는 네트워크 엔드포인트와의 통신을 허용하기 위해 이러한 엔드포인트를 나타내는 특수 ID가 존재합니다.
  • 특별히 예약된 ID에는 reserved 문자열 접두사가 붙습니다
# 아래 결과에서 1~10번까지는 예약된 레이블임
kubectl exec -it -n kube-system ds/cilium -- cilium identity list
ID      LABELS
1       reserved:host
        reserved:kube-apiserver
2       reserved:world
3       reserved:unmanaged
4       reserved:health
5       reserved:init
6       reserved:remote-node
7       reserved:kube-apiserver
        reserved:remote-node
8       reserved:ingress
9       reserved:world-ipv4
10      reserved:world-ipv6
666     k8s:app=prometheus
        k8s:io.ciliuhttp://m.k8s.namespace.labels.kubernetes.io/metadata.name=cilium-monitoring
        k8s:io.ciliuhttp://m.k8s.policy.cluster=default
        k8s:io.ciliuhttp://m.k8s.policy.serviceaccount=prometheus-k8s
        k8s:io.kubernetes.pod.namespace=cilium-monitoring

 

Identity Management in the Cluster

  • ID는 전체 클러스터에서 유효합니다.
  • 즉, 여러 클러스터 노드에서 여러 개의 포드 또는 컨테이너가 시작되더라도 ID 관련 레이블을 공유하는 경우 모든 포드 또는 컨테이너가 단일 ID를 확인하고 공유합니다.
  • 이를 위해서는 클러스터 노드 간의 조정이 필요합니다.
  • 엔드포인트 ID를 확인하는 작업은 분산 키-값 저장소를 통해 수행됩니다.
  • 분산 키-값 저장소는 다음 값이 이전에 확인되지 않은 경우 새로운 고유 식별자를 생성하는 형태의 원자적 연산을 수행할 수 있도록 합니다.
  • 이를 통해 각 클러스터 노드는 ID 관련 레이블 하위 집합을 생성한 다음 키-값 저장소를 쿼리하여 ID를 도출할 수 있습니다.
  • 레이블 집합이 이전에 쿼리되었는지 여부에 따라 새 ID가 생성되거나 초기 쿼리의 ID가 반환됩니다.

 

NetworkPolicy

아래 그림은 Kubernetes가 기본적으로 지원하는 NetworkPolicy와, Cilium이 추가로 풍부하게 지원하는 NetworkPolicy 다이어그램입니다.

https://isovalent.com/blog/post/intro-to-cilium-network-policies/

Policy Enforcement Modes by Cilium Network Policy

(https://docs.cilium.io/en/stable/security/policy/intro/)

  • 3 policy enforcement modes
    • default
    • always
    • never
  • Endpoint default policy
    • 기본적으로 모든 엔드포인트에 대해 모든 송신 및 수신 트래픽이 허용됩니다. 네트워크 정책에 따라 엔드포인트가 선택되면 명시적으로 허용된 트래픽 만 허용되는 기본 거부 상태로 전환됩니다 . 이 상태는 방향별로 적용됩니다.
      • 규칙이 엔드포인트를 선택 하고 해당 규칙에 수신 섹션이 있는 경우, 엔드포인트는 수신에 대해 기본 거부 모드로 전환됩니다.
      • 규칙이 엔드포인트를 선택 하고 규칙에 송신 섹션이 있는 경우, 엔드포인트는 송신에 대해 기본 거부 모드로 전환됩니다.
    • EnableDefaultDeny 7계층 정책(https://docs.cilium.io/en/stable/security/policy/language/#l7-policy)에는 적용되지 않습니다.
      • 7계층 모두 허용이 포함되지 않은 7계층 규칙을 추가하면 default-deny가 명시적으로 비활성화된 경우에도 삭제가 발생합니다.
  • Rule Basics
    • 엔드포인트 선택기 / 노드 선택기
      • 정책 규칙이 적용될 엔드포인트 또는 노드를 선택합니다. 정책 규칙은 선택기에 지정된 레이블과 일치하는 모든 엔드포인트에 적용.
    • 입구
      • 엔드포인트의 유입 시, 즉 엔드포인트에 들어오는 모든 네트워크 패킷에 적용해야 하는 규칙 목록
    • 출구
      • 엔드포인트의 출구에서 적용되어야 하는 규칙 목록, 즉 엔드포인트를 떠나는 모든 네트워크 패킷에 적용되어야 하는 규칙 목록
    • 라벨
      • 레이블은 규칙을 식별하는 데 사용됩니다.
      • 레이블을 사용하여 규칙을 나열하고 삭제할 수 있습니다.
      • Kubernetes를 통해 가져온 정책 규칙에는 NetworkPolicy 또는 CiliumNetworkPolicy 리소스 에 지정된 이름에 해당하는 레이블이 자동으로 io.cilium.k8s.policy.name=NAME지정됨
  • 정책 에디터 온라인 사이트

 

정책 예제 실습 참고

Cilium Security 실습

[LAB1] DNS 기반 보안 정책

L7에서 처리하는 DNS 실습이 아니고, Cilium Agent가 처리하는 실습입니다.

  • CIDR 또는 IP 기반 정책은 외부 서비스와 연결된 IP가 자주 변경될 수 있으므로 관리가 어렵고 번거롭습니다.
  • Cilium의 DNS 기반 정책은 DNS-IP 매핑 추적과 같은 복잡한 측면을 관리하는 동시에 액세스 제어를 쉽게 지정할 수 있는 메커니즘을 제공합니다.
  • 이 가이드에서는 다음 사항에 대해 알아봅니다.
    • DNS 기반 정책을 사용하여 클러스터 외부 서비스에 대한 이탈 액세스 제어
    • 패턴(또는 와일드카드)을 사용하여 DNS 도메인 하위 집합을 허용 목록에 추가
    • 외부 서비스 접근 제한을 위한 DNS, 포트 및 L7 규칙 결합.

데모 애플리케이션 배포

  • Empire의 mediabot pod가 Empire의 git 저장소 관리를 위해 GitHub에 접근해야 하는 간단한 시나리오를 사용하겠습니다.
  • pod는 다른 외부 서비스에 접근할 수 없어야 합니다.
cat << EOF > dns-sw-app.yaml
apiVersion: v1
kind: Pod
metadata:
  name: mediabot
  labels:
    org: empire
    class: mediabot
    app: mediabot <-- Hubble UI 에 표시되는 이름으로 추가 (app, k8s-app 레이블을 이용하면 Hubble UI 에서 쉽게 식별 가능)
spec:
  containers:
  - name: mediabot
    image: quay.io/cilium/json-mock:v1.3.8@sha256:5aad04835eda9025fe4561ad31be77fd55309af8158ca8663a72f6abb78c2603
EOF

kubectl apply -f dns-sw-app.yaml
pod/mediabot created

kubectl wait pod/mediabot --for=condition=Ready
pod/mediabot condition met

# 확인
kubectl exec -it -n kube-system ds/cilium -- cilium identity list
kubectl get pods
ID      LABELS
1       reserved:host
...(생략)...
9222    k8s:app=mediabot
        k8s:class=mediabot
        k8s:io.ciliuhttp://m.k8s.namespace.labels.kubernetes.io/metadata.name=default
        k8s:io.ciliuhttp://m.k8s.policy.cluster=default
        k8s:io.ciliuhttp://m.k8s.policy.serviceaccount=default
        k8s:io.kubernetes.pod.namespace=default
        k8s:org=empire

kubectl get pods
NAME       READY   STATUS    RESTARTS   AGE
mediabot   1/1     Running   0          2m34s

# 외부 통신 확인 : hubble ui 에서 확인 --> 현재는 외부통신이 가능한 상태임
kubectl exec mediabot -- curl -I -s https://api.github.com | head -1
HTTP/2 200

kubectl exec mediabot -- curl -I -s --max-time 5 https://support.github.com | head -1
HTTP/2 302

 

DNS Egress 정책 적용  1

  • mediabot포드가 api.github.com에만 액세스하도록 허용
# api.github.com 도메인에 대해 egress 로 빠져나갈때 정책 설정
cat << EOF | kubectl apply -f -
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "fqdn"
spec:
  endpointSelector:
    matchLabels:
      org: empire
      class: mediabot
  egress:
  - toFQDNs:
    - matchName: "api.github.com"
  - toEndpoints:
    - matchLabels:
        "k8s:io.kubernetes.pod.namespace": kube-system
        "k8s:k8s-app": kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: "*"
EOF
ciliumnetworkpolicy.cilium.io/fqdn created

# 확인
kubectl get cnp
NAME   AGE   VALID
fqdn   18s   True

kubectl exec -it -n kube-system ds/cilium -- cilium policy selectors
SELECTOR                                                                                                                                                                      LABELS         USERS   IDENTITIES
&LabelSelector{MatchLabels:map[string]string{any.class: mediabot,any.org: empire,k8s.io.kubernetes.pod.namespace: default,},MatchExpressions:[]LabelSelectorRequirement{},}   default/fqdn   1       9222

# Cilium의 DNS Proxy 동작하게 되는데, DNS 설정 확인
cilium config view | grep -i dns
dnsproxy-enable-transparent-mode                  true
dnsproxy-socket-linger-timeout                    10
hubble-metrics                                    dns drop tcp flow port-distribution icmp httpV2:exemplars=true;labelsContext=source_ip,source_namespace,source_workload,destination_ip,destination_namespace,destination_workload,traffic_direction
tofqdns-dns-reject-response-code                  refused
tofqdns-enable-dns-compression                    true
tofqdns-endpoint-max-ip-per-hostname              1000
tofqdns-idle-connection-grace-period              0s
tofqdns-max-deferred-connection-deletes           10000
tofqdns-preallocate-identities                    true
tofqdns-proxy-response-max-delay                  100ms

# 외부 통신 확인 : hubble ui 에서 확인
kubectl exec mediabot -- curl -I -s https://api.github.com | head -1
HTTP/2 200


# api.github.com 만 도메인질의가 가능하게 하였기 때문에 아래 명령은 실패함
kubectl exec mediabot -- curl -I -s --max-time 5 https://support.github.com | head -1
command terminated with exit code 28


# cilium-agent 내에 go 로 구현된 lightweight proxy 가 DNS 쿼리/응답 감시
cilium hubble port-forward&
hubble observe --pod mediabot
Aug 31 03:59:53.209: default/mediabot:47883 (ID:5190) <- kube-system/coredns-674b8bbfcf-p6pbn:53 (ID:1363) dns-response proxy FORWARDED (DNS Answer "20.200.245.245" TTL: 30 (Proxy api.github.com. A))
...
Aug 31 03:59:53.212: default/mediabot:38212 (ID:5190) -> api.github.com:443 (ID:16777217) policy-verdict:L3-Only EGRESS ALLOWED (TCP Flags: SYN)
...

# 개별 Cilium Agent 의 dns caching 확인
# cilium 파드 이름 지정
export CILIUMPOD0=$(kubectl get -l k8s-app=cilium pods -n kube-system --field-selector spec.nodeName=k8s-ctr -o jsonpath='{.items[0].metadata.name}')
export CILIUMPOD1=$(kubectl get -l k8s-app=cilium pods -n kube-system --field-selector spec.nodeName=k8s-w1  -o jsonpath='{.items[0].metadata.name}')
export CILIUMPOD2=$(kubectl get -l k8s-app=cilium pods -n kube-system --field-selector spec.nodeName=k8s-w2  -o jsonpath='{.items[0].metadata.name}')
echo $CILIUMPOD0 $CILIUMPOD1 $CILIUMPOD2
cilium-c46v5 cilium-kzlfh cilium-hjgtd

# 단축키(alias) 지정
alias c0="kubectl exec -it $CILIUMPOD0 -n kube-system -c cilium-agent -- cilium"
alias c1="kubectl exec -it $CILIUMPOD1 -n kube-system -c cilium-agent -- cilium"
alias c2="kubectl exec -it $CILIUMPOD2 -n kube-system -c cilium-agent -- cilium"

c0 fqdn cache list
Endpoint   Source   FQDN   TTL   ExpirationTime   IPs

c1 fqdn cache list
Endpoint   Source   FQDN   TTL   ExpirationTime   IPs

c2 fqdn cache list
Endpoint   Source       FQDN                  TTL   ExpirationTime             IPs
520        connection   support.github.com.   0     2025-09-04T02:48:38.187Z   185.199.111.133
520        connection   support.github.com.   0     2025-09-04T02:48:38.187Z   185.199.110.133
520        connection   api.github.com.       0     2025-09-04T02:48:38.187Z   20.200.245.245
520        connection   support.github.com.   0     2025-09-04T02:48:38.187Z   185.199.109.133
520        connection   support.github.com.   0     2025-09-04T02:48:38.187Z   185.199.108.133

c0 fqdn names
{
  "DNSPollNames": null,
  "FQDNPolicySelectors": []
}

c1 fqdn names
{
  "DNSPollNames": null,
  "FQDNPolicySelectors": []
}

c2 fqdn names
{
  "DNSPollNames": null,
  "FQDNPolicySelectors": [
    {
      "regexString": "^api[.]github[.]com[.]$",
      "selectorString": "MatchName: api.github.com, MatchPattern: "
    }
  ]
}

 

DNS Egress 정책 적용  2

  • 모든 GitHub 하위 도메인(예: 패턴)에 액세스.
# fqdn 캐시 초기화 및 정책 삭제
kubectl delete cnp fqdn
ciliumnetworkpolicy.cilium.io "fqdn" deleted
c1 fqdn cache clean -f
FQDN proxy cache cleared
c2 fqdn cache clean -f
FQDN proxy cache cleared

# dns-pattern.yaml 내용
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "fqdn"
spec:
  endpointSelector:
    matchLabels:
      org: empire
      class: mediabot
  egress:
  - toFQDNs:
    - matchName: "*.github.com"
  - toEndpoints:
    - matchLabels:
        "k8s:io.kubernetes.pod.namespace": kube-system
        "k8s:k8s-app": kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: "*"
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.18.1/examples/kubernetes-dns/dns-pattern.yaml
ciliumnetworkpolicy.cilium.io/fqdn created

c1 fqdn names
{
  "DNSPollNames": null,
  "FQDNPolicySelectors": []
}


c2 fqdn names
{
  "DNSPollNames": null,
  "FQDNPolicySelectors": [
    {
      "regexString": "^[-a-zA-Z0-9_]*[.]github[.]com[.]$",
      "selectorString": "MatchName: , MatchPattern: *.github.com"
    }
  ]
}

c1 fqdn cache list
Endpoint   Source   FQDN   TTL   ExpirationTime   IPs

c2 fqdn cache list
Endpoint   Source   FQDN   TTL   ExpirationTime   IPs

# 확인
kubectl get cnp
NAME   AGE   VALID
fqdn   37s   True

# 외부 통신 확인 : hubble ui 에서 확인 >> github.com 은 공식 문서 설명대로라면 안되야됨..
##  It is important to note and test that this doesn’t allow access to github.com because the *. 
## in the pattern requires one subdomain to be present in the DNS name
kubectl exec mediabot -- curl -I -s https://support.github.com | head -1
HTTP/2 302

 kubectl exec mediabot -- curl -I -s https://gist.github.com | head -1
HTTP/2 302

kubectl exec mediabot -- curl -I -s --max-time 5 https://github.com | head -1
HTTP/2 200 <-- 공식문서에서는 안되어야 함 --> 최신 Cilium  버전에서 적용되었는지 확인 필요!!!

kubectl exec mediabot -- curl -I -s --max-time 5 https://cilium.io| head -1
command terminated with exit code 28

 

DNS Egress 정책 적용  3

  • DNS, Port 조합 적용
# dns-port.yaml 내용
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "fqdn"
spec:
  endpointSelector:
    matchLabels:
      org: empire
      class: mediabot
  egress:
  - toFQDNs:
    - matchPattern: "*.github.com"
    toPorts:
    - ports:
      - port: "443"
        protocol: TCP
  - toEndpoints:
    - matchLabels:
        "k8s:io.kubernetes.pod.namespace": kube-system
        "k8s:k8s-app": kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: ANY
      rules:
        dns:
        - matchPattern: "*"

kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.18.1/examples/kubernetes-dns/dns-port.yaml
ciliumnetworkpolicy.cilium.io/fqdn configured

c1 fqdn names
{
  "DNSPollNames": null,
  "FQDNPolicySelectors": []
}

c2 fqdn names
{
  "DNSPollNames": null,
  "FQDNPolicySelectors": [
    {
      "regexString": "^[-a-zA-Z0-9_]*[.]github[.]com[.]$",
      "selectorString": "MatchName: , MatchPattern: *.github.com"
    }
  ]
}

c1 fqdn cache list
c2 fqdn cache list
c1 fqdn cache clean -f
c2 fqdn cache clean -f

# 외부 통신 확인 : hubble ui 에서 확인
# http는 접근안되고, https 만 접근가능한지 확인
kubectl exec mediabot -- curl -I -s https://support.github.com | head -1
HTTP/2 302

kubectl exec mediabot -- curl -I -s --max-time 5 http://support.github.com | head -1
command terminated with exit code 28

 

실습 리소스 삭제

kubectl delete -f https://raw.githubusercontent.com/cilium/cilium/1.18.1/examples/kubernetes-dns/dns-sw-app.yaml
pod "mediabot" deleted

kubectl delete cnp fqdn
ciliumnetworkpolicy.cilium.io "fqdn" deleted

 

 

[LAB2] 네트워크 기반 보안 정책

샘플 애플리케이션 배포

# 샘플 애플리케이션 배포
cat << EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: webpod
spec:
  replicas: 2
  selector:
    matchLabels:
      app: webpod
  template:
    metadata:
      labels:
        app: webpod
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - sample-app
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: webpod
        image: traefik/whoami
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: webpod
  labels:
    app: webpod
spec:
  selector:
    app: webpod
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80
  type: ClusterIP
EOF

deployment.apps/webpod created
service/webpod created

# k8s-ctr 노드에 curl-pod 파드 배포
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: curl-pod
  labels:
    app: curl
spec:
  nodeName: k8s-ctr
  containers:
  - name: curl
    image: nicolaka/netshoot
    command: ["tail"]
    args: ["-f", "/dev/null"]
  terminationGracePeriodSeconds: 0
EOF
pod/curl-pod created

# 배포 확인
kubectl get deploy,svc,ep webpod -owide
kubectl get endpointslices -l app=webpod
kubectl get ciliumendpoints # IP 확인

# 통신 문제 확인
kubectl exec -it curl-pod -- curl -s --connect-timeout 1 webpod | grep Hostname
kubectl exec -it curl-pod -- sh -c 'while true; do curl -s --connect-timeout 1 webpod | grep Hostname; echo "---" ; sleep 1; done'

# cilium-dbg, map
kubectl exec -n kube-system ds/cilium -- cilium-dbg ip list
kubectl exec -n kube-system ds/cilium -- cilium-dbg endpoint list
kubectl exec -n kube-system ds/cilium -- cilium-dbg service list
kubectl exec -n kube-system ds/cilium -- cilium-dbg bpf lb list
kubectl exec -n kube-system ds/cilium -- cilium-dbg bpf nat list
kubectl exec -n kube-system ds/cilium -- cilium-dbg map list | grep -v '0             0'
kubectl exec -n kube-system ds/cilium -- cilium-dbg map get cilium_lb4_services_v2
kubectl exec -n kube-system ds/cilium -- cilium-dbg map get cilium_lb4_backends_v3
kubectl exec -n kube-system ds/cilium -- cilium-dbg map get cilium_lb4_reverse_nat
kubectl exec -n kube-system ds/cilium -- cilium-dbg map get cilium_ipcache_v2

 

Transparent Encryption with WireGuard 소개

  • Each node automatically creates its own encryption key-pair and distributes its public key via the io.ciliuhttp://m.network.wg-pub-key annotation in the Kubernetes CiliumNode custom resource object.
  • Each node’s public key is then used by other nodes to decrypt and encrypt traffic from and to Cilium-managed endpoints running on that node.
  • 한 노드 내에서 파드(엔드포인트)간 통신 시에는 암호화 되지 않습니다.
  • The WireGuard tunnel endpoint is exposed on UDP port 51871 on each node.
  • Limitations : L7 policy enforcement and visibility , eBPF-based host routing

 

[Lab2] WireGuard 설정 및 실습 : 터널 모드는 두 번 캡슐화됨

  • WireGuard 터널 엔드포인트는 51871각 노드의 UDP 포트에 노출
# [커널 구성 옵션] CONFIG_WIREGUARD=m on Linux 5.6 and newe
uname -ar
Linux k8s-ctr 6.8.0-64-generic #67-Ubuntu SMP PREEMPT_DYNAMIC Sun Jun 15 20:23:31 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

grep -E 'CONFIG_WIREGUARD=m' /boot/config-$(uname -r)
CONFIG_WIREGUARD=m

# 설정 전 기본 정보 확인
ip -c addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 08:00:27:6d:e2:c4 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    inet 10.0.2.15/24 metric 100 brd 10.0.2.255 scope global dynamic eth0
       valid_lft 72776sec preferred_lft 72776sec
    inet6 fd17:625c:f037:2:a00:27ff:fe6d:e2c4/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 86277sec preferred_lft 14277sec
    inet6 fe80::a00:27ff:fe6d:e2c4/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 08:00:27:d8:af:3f brd ff:ff:ff:ff:ff:ff
    altname enp0s8
    inet 192.168.10.100/24 brd 192.168.10.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fed8:af3f/64 scope link
       valid_lft forever preferred_lft forever
4: cilium_net@cilium_host: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether f6:88:d8:a1:05:ae brd ff:ff:ff:ff:ff:ff
    inet6 fe80::f488:d8ff:fea1:5ae/64 scope link
       valid_lft forever preferred_lft forever
5: cilium_host@cilium_net: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether a6:15:11:d5:44:e7 brd ff:ff:ff:ff:ff:ff
    inet 172.20.0.246/32 scope global cilium_host
       valid_lft forever preferred_lft forever
    inet6 fe80::a415:11ff:fed5:44e7/64 scope link
       valid_lft forever preferred_lft forever
7: lxcf8ea143ff8eb@if6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 36:95:57:54:35:39 brd ff:ff:ff:ff:ff:ff link-netns cni-97ba907f-61fc-2a74-023b-d289fc14c266
    inet6 fe80::3495:57ff:fe54:3539/64 scope link
       valid_lft forever preferred_lft forever
9: lxccab69e99ac2a@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 32:ff:44:4a:e7:01 brd ff:ff:ff:ff:ff:ff link-netns cni-50b73b9f-b34b-f6e2-c04c-6f0707c87d82
    inet6 fe80::30ff:44ff:fe4a:e701/64 scope link
       valid_lft forever preferred_lft forever
11: lxc02adeea6d742@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 86:b9:01:1f:ee:a5 brd ff:ff:ff:ff:ff:ff link-netns cni-193ea5c3-e3e6-2989-284d-d5f0b3bd72e9
    inet6 fe80::84b9:1ff:fe1f:eea5/64 scope link
       valid_lft forever preferred_lft forever
13: lxc722d7cafecaf@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 72:19:58:24:97:4b brd ff:ff:ff:ff:ff:ff link-netns cni-7806a104-036b-d794-d00d-df1c74b91f6b
    inet6 fe80::7019:58ff:fe24:974b/64 scope link
       valid_lft forever preferred_lft forever
15: lxc0a58a862f7da@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether d6:80:b4:72:26:3b brd ff:ff:ff:ff:ff:ff link-netns cni-71bc2cbb-a9cd-4c04-5a27-21a265f1dbce
    inet6 fe80::d480:b4ff:fe72:263b/64 scope link
       valid_lft forever preferred_lft forever
17: lxc0e18eab0cc57@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether f6:9d:25:0e:a4:b6 brd ff:ff:ff:ff:ff:ff link-netns cni-47bbe76e-04f2-a764-69d1-ad312519338f
    inet6 fe80::f49d:25ff:fe0e:a4b6/64 scope link
       valid_lft forever preferred_lft forever
19: lxccf7bcac36875@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 1e:78:26:40:45:78 brd ff:ff:ff:ff:ff:ff link-netns cni-659154b4-7f0a-b3b1-2723-bdbfb9e9557d
    inet6 fe80::1c78:26ff:fe40:4578/64 scope link
       valid_lft forever preferred_lft forever
21: lxceb07c71d3053@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether e2:28:cd:c6:e3:a9 brd ff:ff:ff:ff:ff:ff link-netns cni-fc407f9a-a665-3947-80c9-640098a90bbd
    inet6 fe80::e028:cdff:fec6:e3a9/64 scope link
       valid_lft forever preferred_lft forever
23: lxcd0fb5db9c698@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 7a:34:b6:01:14:50 brd ff:ff:ff:ff:ff:ff link-netns cni-48fc2f38-d95c-d154-8297-6e5ad36020c4
    inet6 fe80::7834:b6ff:fe01:1450/64 scope link
       valid_lft forever preferred_lft forever

ip -c route
default via 10.0.2.2 dev eth0 proto dhcp src 10.0.2.15 metric 100
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15 metric 100
10.0.2.2 dev eth0 proto dhcp scope link src 10.0.2.15 metric 100
10.0.2.3 dev eth0 proto dhcp scope link src 10.0.2.15 metric 100
172.20.0.28 dev lxccf7bcac36875 proto kernel scope link
172.20.0.35 dev lxc02adeea6d742 proto kernel scope link
172.20.0.104 dev lxceb07c71d3053 proto kernel scope link
172.20.0.108 dev lxc0a58a862f7da proto kernel scope link
172.20.0.126 dev lxc722d7cafecaf proto kernel scope link
172.20.0.138 dev lxcd0fb5db9c698 proto kernel scope link
172.20.0.221 dev lxccab69e99ac2a proto kernel scope link
172.20.0.230 dev lxc0e18eab0cc57 proto kernel scope link
172.20.0.243 dev lxcf8ea143ff8eb proto kernel scope link
172.20.1.0/24 via 192.168.10.101 dev eth1 proto kernel
172.20.2.0/24 via 192.168.10.102 dev eth1 proto kernel
192.168.10.0/24 dev eth1 proto kernel scope link src 192.168.10.100

ip rule show
9:      from all fwmark 0x200/0xf00 lookup 2004
10:     from all fwmark 0xa00/0xf00 lookup 2005
100:    from all lookup local
32766:  from all lookup main
32767:  from all lookup default

# 설정
helm upgrade cilium cilium/cilium --version 1.18.1 --namespace kube-system --reuse-values \
  --set encryption.enabled=true --set encryption.type=wireguard
I0904 13:04:00.466958   12905 warnings.go:110] "Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice"
I0904 13:04:00.474536   12905 warnings.go:110] "Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice"
I0904 13:04:00.497161   12905 warnings.go:110] "Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice"
Release "cilium" has been upgraded. Happy Helming!
NAME: cilium
LAST DEPLOYED: Thu Sep  4 13:03:53 2025
NAMESPACE: kube-system
STATUS: deployed
REVISION: 2
TEST SUITE: None
NOTES:
You have successfully installed Cilium with Hubble Relay and Hubble UI.

Your release version is 1.18.1.

For any further help, visit https://docs.cilium.io/en/v1.18/gettinghelp

kubectl -n kube-system rollout restart ds/cilium
daemonset.apps/cilium restarted

# 확인
cilium config view | grep -i wireguard
enable-wireguard                                  true
wireguard-persistent-keepalive                    0s
root@k8s-ctr:~# kubectl exec -it -n kube-system ds/cilium -- cilium encrypt status
Encryption: Disabled
root@k8s-ctr:~# kubectl exec -it -n kube-system ds/cilium -- cilium encrypt status
Encryption: Wireguard
Interface: cilium_wg0
        Public key: PcQBKph7U042nYB43g9c9Pyu45vrzUqHEgavClw8kiw=
        Number of peers: 1

kubectl exec -it -n kube-system ds/cilium -- cilium status | grep Encryption
Encryption:              Wireguard   [NodeEncryption: Disabled, cilium_wg0 (Pubkey: PcQBKph7U042nYB43g9c9Pyu45vrzUqHEgavClw8kiw=, Port: 51871, Peers: 2)]

kubectl exec -it -n kube-system ds/cilium -- cilium debuginfo --output json

kubectl exec -it -n kube-system ds/cilium -- cilium debuginfo --output json | jq .encryption
{
  "wireguard": {
    "interfaces": [
      {
        "listen-port": 51871,
        "name": "cilium_wg0",
        "peer-count": 2,
        "peers": [
          {
            "allowed-ips": [
              "172.20.0.138/32",
              "172.20.0.0/24",
              "172.20.0.35/32",
              "172.20.0.221/32",
              "172.20.0.230/32",
              "172.20.0.189/32",
              "172.20.0.243/32",
              "172.20.0.126/32",
              "172.20.0.108/32",
              "172.20.0.104/32",
              "192.168.10.100/32",
              "172.20.0.246/32",
              "172.20.0.28/32"
            ],
            "endpoint": "192.168.10.100:51871",
            "last-handshake-time": "0001-01-01T00:00:00.000Z",
            "public-key": "HnWK2O/diHP7OZdAol9NM7f7AoT+MON/H+hU3+s2pQE="
          },
          {
            "allowed-ips": [
              "172.20.1.109/32",
              "172.20.1.0/24",
              "172.20.1.221/32",
              "172.20.1.19/32",
              "192.168.10.101/32"
            ],
            "endpoint": "192.168.10.101:51871",
            "last-handshake-time": "0001-01-01T00:00:00.000Z",
            "public-key": "q+FUbVl2gKV7xYoxiTAExUBPt3XsaKvaPcLt6qxsKAY="
          }
        ],
        "public-key": "PcQBKph7U042nYB43g9c9Pyu45vrzUqHEgavClw8kiw="
      }
    ],
    "node-encryption": "Disabled"
  }
}

ip -d -c addr show cilium_wg0
24: cilium_wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default
    link/none  promiscuity 0  allmulti 0 minmtu 0 maxmtu 2147483552
    wireguard numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 gro_max_size 65536

ip rule show
9:      from all fwmark 0x200/0xf00 lookup 2004
10:     from all fwmark 0xa00/0xf00 lookup 2005
100:    from all lookup local
32766:  from all lookup main
32767:  from all lookup default

# wireguard 정보 확인
wg -h
Usage: wg <cmd> [<args>]

Available subcommands:
  show: Shows the current configuration and device information
  showconf: Shows the current configuration of a given WireGuard interface, for use with `setconf'
  set: Change the current configuration, add peers, remove peers, or change peers
  setconf: Applies a configuration file to a WireGuard interface
  addconf: Appends a configuration file to a WireGuard interface
  syncconf: Synchronizes a configuration file to a WireGuard interface
  genkey: Generates a new private key and writes it to stdout
  genpsk: Generates a new preshared key and writes it to stdout
  pubkey: Reads a private key from stdin and writes a public key to stdout
You may pass `--help' to any of these subcommands to view usage.

wg show
interface: cilium_wg0
  public key: HnWK2O/diHP7OZdAol9NM7f7AoT+MON/H+hU3+s2pQE=
  private key: (hidden)
  listening port: 51871
  fwmark: 0xe00

peer: PcQBKph7U042nYB43g9c9Pyu45vrzUqHEgavClw8kiw=
  endpoint: 192.168.10.102:51871
  allowed ips: 192.168.10.102/32, 172.20.2.7/32, 172.20.2.100/32, 172.20.2.0/24, 172.20.2.35/32

peer: q+FUbVl2gKV7xYoxiTAExUBPt3XsaKvaPcLt6qxsKAY=
  endpoint: 192.168.10.101:51871
  allowed ips: 172.20.1.221/32, 172.20.1.19/32, 172.20.1.0/24, 172.20.1.109/32, 192.168.10.101/32

wg show all public-key
cilium_wg0      HnWK2O/diHP7OZdAol9NM7f7AoT+MON/H+hU3+s2pQE=

wg show all private-key
cilium_wg0      yD75TIFreDp+ckbtf6Al4CapQQaQN8XXM1xWgsW3xkY=

wg show all preshared-keys
cilium_wg0      PcQBKph7U042nYB43g9c9Pyu45vrzUqHEgavClw8kiw=    (none)
cilium_wg0      q+FUbVl2gKV7xYoxiTAExUBPt3XsaKvaPcLt6qxsKAY=    (none)

wg show all endpoints
cilium_wg0      PcQBKph7U042nYB43g9c9Pyu45vrzUqHEgavClw8kiw=    192.168.10.102:51871
q+FUbVl2gKV7xYoxiTAExUBPt3XsaKvaPcLt6qxsKAY=    192.168.10.101:51871
root@k8s-ctr:~# wg show all transfer
cilium_wg0      PcQBKph7U042nYB43g9c9Pyu45vrzUqHEgavClw8kiw=    0       0
cilium_wg0      q+FUbVl2gKV7xYoxiTAExUBPt3XsaKvaPcLt6qxsKAY=    0       0

# 퍼블릭 키 확인
kubectl get cn -o yaml | grep annotations -A1
    annotations:
      network.cilium.io/wg-pub-key: HnWK2O/diHP7OZdAol9NM7f7AoT+MON/H+hU3+s2pQE=
--
    annotations:
      network.cilium.io/wg-pub-key: q+FUbVl2gKV7xYoxiTAExUBPt3XsaKvaPcLt6qxsKAY=
--
    annotations:
      network.cilium.io/wg-pub-key: PcQBKph7U042nYB43g9c9Pyu45vrzUqHEgavClw8kiw=

 

  • 통신 확인
# curl 호출
kubectl exec -it curl-pod -- curl webpod
kubectl exec -it curl-pod -- curl webpod


# tcpdump
tcpdump -i cilium_wg0 -n
tcpdump -eni any udp port 51871
tcpdump -eni any udp port 51871 -w /tmp/wg.pcap  # vagrant scp k8s-ctr:/tmp/wg.pcap . > wireshark 로 확인

# query the flow API and look for flows
hubble observe --pod curl-pod

'Kubernetes' 카테고리의 다른 글

MinIO - DirectPV & Performance  (1) 2025.09.20
MinIO 소개  (0) 2025.09.13
Cilium Study [1기] (7주차) - Jmeter를 이용한 K8S 부하테스트  (0) 2025.08.29
Cilium Study [1기] (6주차) - Cilium ServiceMesh  (0) 2025.08.19
Cilium Study [1기] (5주차) - BGP Control Plane  (4) 2025.08.13

'Kubernetes'의 다른글

  • 현재글Cilium Study [1기] (8주차) - Cilium Security

관련글

티스토리툴바