MinIO - DirectPV & Performance

Kubernetes

MinIO - DirectPV & Performance

yu3papa 2025. 9. 20. 10:38

실습 환경 구성

AWS 서울리전에 EC2 리눅스를 1대 생성하고 4개의 EBS를 연결하여 DirectPV 실습

경량의 쿠버네티스인 k3s 이용하여 MinIO를 이용한 DirectPV 실습

(참고) CloudFormation을 EC2 생성시 아래 스크립트를 이용하여 k3s 및 관련 툴이 설치 됩니다.

hostnamectl --static set-hostname k3s-s

# Config convenience
echo 'alias vi=vim' >> /etc/profile
echo "sudo su -" >> /home/ubuntu/.bashrc
ln -sf /usr/share/zoneinfo/Asia/Seoul /etc/localtime

# Disable ufw & apparmor
systemctl stop ufw && systemctl disable ufw
systemctl stop apparmor && systemctl disable apparmor

# Install packages
apt update && apt-get install bridge-utils net-tools conntrack ngrep jq yq tree unzip kubecolor fio tuned -y

# local dns - hosts file
echo "192.168.10.10 k3s-s" >> /etc/hosts

# Install k3s-server
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.33.4+k3s1 INSTALL_K3S_EXEC=" --disable=traefik" K3S_KUBECONFIG_MODE="644" sh -s - server --token miniotoken

# Change kubeconfig
echo 'export KUBECONFIG=/etc/rancher/k3s/k3s.yaml' >> /etc/profile

# Install Helm
curl -s https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash

# Alias kubectl to k
echo 'alias kc=kubecolor' >> /etc/profile
echo 'alias k=kubectl' >> /etc/profile
echo 'complete -o default -F __start_kubectl k' >> /etc/profile

# kubectl Source the completion
source <(kubectl completion bash)
echo 'source <(kubectl completion bash)' >> /etc/profile

# Install Kubectx & Kubens
git clone https://github.com/ahmetb/kubectx /opt/kubectx
ln -s /opt/kubectx/kubens /usr/local/bin/kubens
ln -s /opt/kubectx/kubectx /usr/local/bin/kubectx

# Install Kubeps & Setting PS1
git clone https://github.com/jonmosco/kube-ps1.git /root/kube-ps1
cat <<"EOT" >> ~/.bash_profile
source /root/kube-ps1/kube-ps1.sh
KUBE_PS1_SYMBOL_ENABLE=true
function get_cluster_short() {
echo "$1" | cut -d . -f1
}
KUBE_PS1_CLUSTER_FUNCTION=get_cluster_short
KUBE_PS1_SUFFIX=') '
PS1='$(kube_ps1)'$PS1
EOT

# Install MC : https://docs.min.io/community/minio-object-store/reference/minio-mc.html
curl -O https://dl.min.io/client/mc/release/linux-amd64/mc
chmod +x mc
cp mc /usr/bin

minio-ec2-1node.yaml

0.01MB

VPC 1개, EC2 인스턴스 1대 (Ubuntu 24.04 LTS, t3.xlarge - vCPU 4 , Mem 16, Data용 EBS 볼륨 4개)
CloudFormation 을 이용할 실습환경 배포

# CloudFormation 스택 배포
aws cloudformation deploy --template-file minio-ec2-1node.yaml --stack-name miniolab --parameter-overrides KeyName=yu3papa SgIngressSshCidr=125.176.27.182/32 --region ap-northeast-2
Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - miniolab

# CloudFormation 스택 배포 완료 후 작업용 EC2 IP 출력

# 생성된 우분투 ec2 리눅스에 SSH 접속
ssh -i yu3papa.pem ubuntu@3.36.72.189
...(생략)...
Last login: Sat Sep 20 10:20:14 2025 from 125.176.27.182
(⎈|default:N/A) root@k3s-s:~#

EC2 접속후 krew 플러그인을 설치합니다.

kc get node -o wide
NAME    STATUS   ROLES                  AGE   VERSION        INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION    CONTAINER-RUNTIME
k3s-s   Ready    control-plane,master   20m   v1.33.4+k3s1   192.168.10.10   <none>        Ubuntu 24.04.3 LTS   6.14.0-1012-aws   containerd://2.0.5-k3s2

hostnamectl
Static hostname: k3s-s
       Icon name: computer-vm
         Chassis: vm 🖴
      Machine ID: ec29fa949c1835c935138ba971cbb155
         Boot ID: 90ef457b6b4c4b14ab8f500a0c3afbcc
  Virtualization: amazon
Operating System: Ubuntu 24.04.3 LTS
          Kernel: Linux 6.14.0-1012-aws
    Architecture: x86-64
Hardware Vendor: Amazon EC2
  Hardware Model: t3.xlarge
Firmware Version: 1.0
   Firmware Date: Mon 2017-10-16
    Firmware Age: 7y 11month 4d

# Install Krew
wget -P /root "https://github.com/kubernetes-sigs/krew/releases/latest/download/krew-linux_amd64.tar.gz"
tar zxvf "/root/krew-linux_amd64.tar.gz" --warning=no-unknown-keyword
./krew-linux_amd64 install krew
export PATH="${KREW_ROOT:-$HOME/.krew}/bin:$PATH" # export PATH="$PATH:/root/.krew/bin"
echo 'export PATH="$PATH:/root/.krew/bin:/root/go/bin"' >> /etc/profile
kubectl krew install get-all neat rolesum pexec stern
kubectl krew list
PLUGIN   VERSION
get-all  v1.4.2
krew     v0.4.5
neat     v2.0.4
pexec    v0.4.1
rolesum  v1.5.5
stern    v1.33.0

위와 같이 설정한 EC2에는 총 5개의 EBS가 장착되어 있고 sda는 운영체제가 설치되어 있고, sdb~sde 까지 DirectPV에서 사용할 예정입니다.

DirectPV 실습전에 AWS의 EBS 볼륨에 대한 성능을 확인해 보겠습니다.

fio 라는 대표적인 디스크 성능측정도구를 이용하겠습니다.

lsblk -a -o NAME,KNAME,MAJ:MIN,SIZE,TYPE,MOUNTPOINT,FSTYPE,UUID,MODEL,SERIAL
NAME         KNAME      MAJ:MIN  SIZE TYPE MOUNTPOINT                   FSTYPE   UUID                                 MODEL                      SERIAL
loop0        loop0        7:0   27.6M loop /snap/amazon-ssm-agent/11797 squashfs
loop1        loop1        7:1   73.9M loop /snap/core22/2111            squashfs
loop2        loop2        7:2   50.8M loop /snap/snapd/25202            squashfs
loop3        loop3        7:3      0B loop
loop4        loop4        7:4      0B loop
loop5        loop5        7:5      0B loop
loop6        loop6        7:6      0B loop
loop7        loop7        7:7      0B loop
nvme2n1      nvme2n1    259:0     30G disk                                                                            Amazon Elastic Block Store vol0d1b5e3a4b519fd5c
nvme1n1      nvme1n1    259:1     30G disk                                                                            Amazon Elastic Block Store vol0ce96fb18da9930ca
nvme0n1      nvme0n1    259:2     30G disk                                                                            Amazon Elastic Block Store vol0f8084134c5b34fb5
├─nvme0n1p1  nvme0n1p1  259:5     29G part /                            ext4     04a692e8-fb29-4bab-b64d-906697110bb7
├─nvme0n1p14 nvme0n1p14 259:6      4M part
├─nvme0n1p15 nvme0n1p15 259:7    106M part /boot/efi                    vfat     F567-A352
└─nvme0n1p16 nvme0n1p16 259:8    913M part /boot                        ext4     13015197-4a14-4c28-bbd2-85341b3dff09
nvme3n1      nvme3n1    259:3     30G disk                                                                            Amazon Elastic Block Store vol037a27b57f3caaf4c
nvme4n1      nvme4n1    259:4     30G disk                                                                            Amazon Elastic Block Store vol0102cfd2ed23c6fb5

# 측정 기본
## IOPS: 초당 입출력 횟수
## Throughput: 초당 MB 전송량
## Block size, Queue depth, RW 패턴 (랜덤/순차, 읽기/쓰기) 조합에 따라 결과가 달라집니다.
## AWS gp3는 기본 IOPS 3000/Throughput 125MB/s

# 4k 랜덤 읽기/쓰기, iodepth=16, numjobs=4
## --rw=randrw : 랜덤 읽기/쓰기 혼합
## --rwmixread=70 : 70% 읽기 / 30% 쓰기
## --bs=4k : 4KB 블록 → IOPS 측정용
## --iodepth=16 : 큐 깊이 16
## --numjobs=4 : 4개의 병렬 job
## --time_based --runtime=60 : 60초 동안 측정
## --group_reporting : 그룹 단위 결과 요약

# Random Read/Write 성능측정
fio --name=randrw_test \
  --filename=/mnt/testfile \
  --size=4G \
  --rw=randrw \
  --rwmixread=70 \
  --bs=4k \
  --iodepth=16 \
  --numjobs=4 \
  --time_based \
  --runtime=60 \
  --group_reporting

## 읽기 평균 IOPS : 2388
  read: IOPS=2388, BW=9553KiB/s (9782kB/s)(560MiB/60003msec)

## 쓰기 평균 IOPS : 1030
  write: IOPS=1030, BW=4121KiB/s (4220kB/s)(241MiB/60003msec); 0 zone resets

## 디스크 활용률 : 읽기/쓰기 IO 수량(132k/49k) , 디스크 활용률 87%
Disk stats (read/write):
  nvme0n1: ios=132972/49709, sectors=1063776/420736, merge=0/8, ticks=232695/154017, in_queue=386713, util=87.09%

# 측정 결과 EBS의 3000 IOPS 를 지원하고 있음

minio 권장 커널 파라미터

MinIO의 최적 성능을 사용하려면 아래 주소에서 확인할 수 있는 권장 설정이 확인되어야 합니다.

https://docs.min.io/community/minio-object-store/operations/checklists/software.html

MinIO 는 디스크의 최적의 성능을 사용하는게 목적인데, 어플리케이션 레벨에서 수행되는 파일 인덱스 처리, 안티바이러스 프로그램은 오히려 성능을 저하시키니 Disable 하는것을 권장합니다.

(참고로 AWS EC2 에는 아래 기능 자체가 설치되어 있지는 않습니다.)

# mlocate or plocate
systemctl disable --now plocate-updatedb
systemctl list-timers | grep locate

# updatedb
systemctl disable --now updatedb.timer
systemctl list-timers | grep updatedb

# auditd
systemctl disable --now auditd
systemctl list-timers | grep audit

# Crowdstrike Falcon & Antivirus software (clamav)

DirectPV

https://docs.min.io/community/minio-directpv/

쿠버네티스 CSI (Container Storage Interface) 표준을 따르는 DAS (direct attached storage) 로컬 볼륨

구성요소 : CSI Driver, Plugin
- 로컬 볼륨을 프로비저닝하는 Kubernetes 클러스터에 직접 설치된 DirectPV CSI 드라이버
- 명령줄 인터페이스를 통해 DirectPV CSI 드라이버를 관리하기 위해 로컬 머신에 설치된 DirectPV 플러그인
- 기본적으로 DirectPV는 분산형 영구 볼륨 관리자입니다. DirectPV는 SAN(Storage Area Network)이나 NAS(Network Attached Storage)와 같은 스토리지 시스템이 아닙니다. DirectPV를 사용하면 분산 환경의 여러 서버에서 드라이브를 검색, 포맷, 마운트, 예약 및 모니터링할 수 있습니다.
- DirectPV는 마운트된 드라이브를 사용하여 Kubernetes가 PVC를 이행하는 데 사용하는 PV를 생성합니다.(https://kubernetes.io/docs/concepts/storage/persistent-volumes/)

DirectPV 설치 & Disk 구성

# DirectPV를 사용하기 위한 도구를 krew 플러그인에서 제공합니다.
k krew install directpv
Updated the local copy of plugin index.
Installing plugin: directpv
Installed plugin: directpv
\
| Use this plugin:
|      kubectl directpv
| Documentation:
|      https://github.com/minio/directpv
/
WARNING: You installed plugin "directpv" from the krew-index plugin repository.
   These plugins are not audited for security by the Krew maintainers.
   Run them at your own risk.

# 도움말 확인
k directpv -h

# 설치
k directpv install
Installing on unsupported Kubernetes v1.33

███████████████████████████████████████████████████████████████████████████ 100%

┌──────────────────────────────────────┬──────────────────────────┐
│ NAME                                 │ KIND                     │
├──────────────────────────────────────┼──────────────────────────┤
│ directpv                             │ Namespace                │
│ directpv-min-io                      │ ServiceAccount           │
│ directpv-min-io                      │ ClusterRole              │
│ directpv-min-io                      │ ClusterRoleBinding       │
│ directpv-min-io                      │ Role                     │
│ directpv-min-io                      │ RoleBinding              │
│ directpvdrives.directpv.min.io       │ CustomResourceDefinition │
│ directpvvolumes.directpv.min.io      │ CustomResourceDefinition │
│ directpvnodes.directpv.min.io        │ CustomResourceDefinition │
│ directpvinitrequests.directpv.min.io │ CustomResourceDefinition │
│ directpv-min-io                      │ CSIDriver                │
│ directpv-min-io                      │ StorageClass             │
│ node-server                          │ Daemonset                │
│ controller                           │ Deployment               │
└──────────────────────────────────────┴──────────────────────────┘

DirectPV installed successfully

# CRD 확인
k get crd | grep min
directpvdrives.directpv.min.io         2025-09-20T02:33:01Z
directpvinitrequests.directpv.min.io   2025-09-20T02:33:01Z
directpvnodes.directpv.min.io          2025-09-20T02:33:01Z
directpvvolumes.directpv.min.io        2025-09-20T02:33:01Z

k get sc
NAME                   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
directpv-min-io        directpv-min-io         Delete          WaitForFirstConsumer   true                   2m10s
local-path (default)   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  87m

k get sc directpv-min-io -o yaml | yq
{
  "allowVolumeExpansion": true,
  "allowedTopologies": [
    {
      "matchLabelExpressions": [
        {
          "key": "directpv.min.io/identity",
          "values": [
            "directpv-min-io"
          ]
        }
      ]
    }
  ],
  "apiVersion": "storage.k8s.io/v1",
  "kind": "StorageClass",
  "metadata": {
    "annotations": {
      "directpv.min.io/plugin-version": "v4.1.4"
    },
    "creationTimestamp": "2025-09-20T02:33:01Z",
    "finalizers": [
      "foregroundDeletion"
    ],
    "labels": {
      "application-name": "directpv.min.io",
      "application-type": "CSIDriver",
      "directpv.min.io/created-by": "kubectl-directpv",
      "directpv.min.io/version": "v1beta1"
    },
    "name": "directpv-min-io",
    "resourceVersion": "2102",
    "uid": "37c78747-d7cd-47e0-89ef-6ca2def20162"
  },
  "parameters": {
    "csi.storage.k8s.io/fstype": "xfs"
  },
  "provisioner": "directpv-min-io",
  "reclaimPolicy": "Delete",
  "volumeBindingMode": "WaitForFirstConsumer"
}

k get-all -n directpv
I0920 11:35:40.914210    9503 warnings.go:110] "Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice"
NAME                                                        NAMESPACE  AGE
configmap/kube-root-ca.crt                                  directpv   2m39s
pod/controller-77b4dd6494-7nhs7                             directpv   2m39s
pod/controller-77b4dd6494-pntfl                             directpv   2m39s
pod/controller-77b4dd6494-sqfsn                             directpv   2m39s
pod/node-server-l792h                                       directpv   2m39s
serviceaccount/default                                      directpv   2m39s
serviceaccount/directpv-min-io                              directpv   2m39s
controllerrevision.apps/node-server-76c6d7d5b9              directpv   2m39s
daemonset.apps/node-server                                  directpv   2m39s
deployment.apps/controller                                  directpv   2m39s
replicaset.apps/controller-77b4dd6494                       directpv   2m39s
lease.coordination.k8s.io/directpv-min-io                   directpv   2m29s
lease.coordination.k8s.io/external-resizer-directpv-min-io  directpv   2m28s
rolebinding.rbac.authorization.k8s.io/directpv-min-io       directpv   2m39s
role.rbac.authorization.k8s.io/directpv-min-io              directpv   2m39s

k get deploy,ds,pod -n directpv
NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/controller   3/3     3            3           3m12s

NAME                         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/node-server   1         1         1       1            1           <none>          3m12s

NAME                              READY   STATUS    RESTARTS   AGE
pod/controller-77b4dd6494-7nhs7   3/3     Running   0          3m12s
pod/controller-77b4dd6494-pntfl   3/3     Running   0          3m12s
pod/controller-77b4dd6494-sqfsn   3/3     Running   0          3m12s
pod/node-server-l792h             4/4     Running   0          3m12s

# 권한 확인
k rolesum directpv-min-io -n directpv
ServiceAccount: directpv/directpv-min-io
Secrets:

Policies:
• [RB] directpv/directpv-min-io ⟶  [R] directpv/directpv-min-io
  Resource                    Name  Exclude  Verbs  G L W C U P D DC
  leases.coordination.k8s.io  [*]     [-]     [-]   ✔ ✔ ✔ ✔ ✔ ✖ ✔ ✖

• [CRB] */directpv-min-io ⟶  [CR] */directpv-min-io
  Resource                                                          Name  Exclude  Verbs  G L W C U P D DC
  csinodes.storage.k8s.io                                           [*]     [-]     [-]   ✔ ✔ ✔ ✖ ✖ ✖ ✖ ✖
  customresourcedefinition.[apiextensions.k8s.io,directpv.min.io]   [*]     [-]     [-]   ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✖
  customresourcedefinitions.[apiextensions.k8s.io,directpv.min.io]  [*]     [-]     [-]   ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✖
  directpvdrives.directpv.min.io                                    [*]     [-]     [-]   ✔ ✔ ✔ ✔ ✔ ✖ ✔ ✖
  directpvinitrequests.directpv.min.io                              [*]     [-]     [-]   ✔ ✔ ✔ ✔ ✔ ✖ ✔ ✖
  directpvnodes.directpv.min.io                                     [*]     [-]     [-]   ✔ ✔ ✔ ✔ ✔ ✖ ✔ ✖
  directpvvolumes.directpv.min.io                                   [*]     [-]     [-]   ✔ ✔ ✔ ✔ ✔ ✖ ✔ ✖
  endpoints                                                         [*]     [-]     [-]   ✔ ✔ ✔ ✔ ✔ ✖ ✔ ✖
  events                                                            [*]     [-]     [-]   ✖ ✔ ✔ ✔ ✔ ✔ ✖ ✖
  leases.coordination.k8s.io                                        [*]     [-]     [-]   ✔ ✔ ✔ ✔ ✔ ✖ ✔ ✖
  nodes                                                             [*]     [-]     [-]   ✔ ✔ ✔ ✖ ✖ ✖ ✖ ✖
  persistentvolumeclaims                                            [*]     [-]     [-]   ✔ ✔ ✔ ✖ ✔ ✖ ✖ ✖
  persistentvolumeclaims/status                                     [*]     [-]     [-]   ✖ ✖ ✖ ✖ ✖ ✔ ✖ ✖
  persistentvolumes                                                 [*]     [-]     [-]   ✔ ✔ ✔ ✔ ✖ ✔ ✔ ✖
  pod                                                               [*]     [-]     [-]   ✔ ✔ ✔ ✖ ✖ ✖ ✖ ✖
  pods                                                              [*]     [-]     [-]   ✔ ✔ ✔ ✖ ✖ ✖ ✖ ✖
  secret                                                            [*]     [-]     [-]   ✔ ✔ ✔ ✖ ✖ ✖ ✖ ✖
  secrets                                                           [*]     [-]     [-]   ✔ ✔ ✔ ✖ ✖ ✖ ✖ ✖
  storageclasses.storage.k8s.io                                     [*]     [-]     [-]   ✔ ✔ ✔ ✖ ✖ ✖ ✖ ✖
  volumeattachments.storage.k8s.io                                  [*]     [-]     [-]   ✔ ✔ ✔ ✖ ✖ ✖ ✖ ✖
  volumesnapshotcontents.snapshot.storage.k8s.io                    [*]     [-]     [-]   ✔ ✔ ✖ ✖ ✖ ✖ ✖ ✖
  volumesnapshots.snapshot.storage.k8s.io                           [*]     [-]     [-]   ✔ ✔ ✖ ✖ ✖ ✖ ✖ ✖

k get directpvnodes.directpv.min.io
NAME    AGE
k3s-s   3m59s

k get directpvnodes.directpv.min.io -o yaml | yq
{
  "apiVersion": "v1",
  "items": [
    {
      "apiVersion": "directpv.min.io/v1beta1",
      "kind": "DirectPVNode",
      "metadata": {
        "creationTimestamp": "2025-09-20T02:33:11Z",
        "generation": 1,
        "labels": {
          "directpv.min.io/created-by": "node-controller",
          "directpv.min.io/node": "k3s-s",
          "directpv.min.io/version": "v1beta1"
        },
        "name": "k3s-s",
        "resourceVersion": "2188",
        "uid": "1cc32f4b-deee-4a39-82a8-3814b07339c9"
      },
      "spec": {},
      "status": {
        "devices": [
          {
            "deniedReason": "Too small",
            "id": "259:6$lVCOx26hZO2s872NOjNJra4S5ZaX9+Q56CQObs9o5+w=",
            "majorMinor": "259:6",
            "make": "Amazon Elastic Block Store (Part 14)",
            "name": "nvme0n1p14",
            "size": 4194304
          },
          {
            "deniedReason": "Mounted",
            "fsType": "ext4",
            "fsuuid": "13015197-4a14-4c28-bbd2-85341b3dff09",
            "id": "259:8$QUneuLFDTeOsXC2uD/VKWrKx/gv975cbvXzIYrtxX5Q=",
            "majorMinor": "259:8",
            "make": "Amazon Elastic Block Store (Part 16)",
            "name": "nvme0n1p16",
            "size": 957350400
          },
          {
            "deniedReason": "Mounted",
            "fsType": "ext4",
            "fsuuid": "04a692e8-fb29-4bab-b64d-906697110bb7",
            "id": "259:5$oDeIgERHCSJIXrymn/bbYWBp7fQGVXvfqoHG05mL1dE=",
            "majorMinor": "259:5",
            "make": "Amazon Elastic Block Store (Part 1)",
            "name": "nvme0n1p1",
            "size": 31137447424
          },
          {
            "id": "259:0$xi6Ajz0HPzT6wr/uMMViQ3psRBdWHeOwQOySleQps3o=",
            "majorMinor": "259:0",
            "make": "Amazon Elastic Block Store",
            "name": "nvme2n1",
            "size": 32212254720
          },
          {
            "deniedReason": "Too small; Mounted",
            "fsType": "vfat",
            "fsuuid": "F567-A352",
            "id": "259:7$4Ghr/DjNZtwvOsBElnLkwg4okZ2vh00OJifip6QiZfc=",
            "majorMinor": "259:7",
            "make": "Amazon Elastic Block Store (Part 15)",
            "name": "nvme0n1p15",
            "size": 111149056
          },
          {
            "deniedReason": "Partitioned",
            "id": "259:2$jc81R04kRB2F2ve1IxDxP8tYwx31Sh++HnpXlpnEmkk=",
            "majorMinor": "259:2",
            "make": "Amazon Elastic Block Store",
            "name": "nvme0n1",
            "size": 32212254720
          },
          {
            "id": "259:4$r2C4ZMPnjz1wv/ZB7XNgDtEcvApI3aLSJN1f8ejPQbw=",
            "majorMinor": "259:4",
            "make": "Amazon Elastic Block Store",
            "name": "nvme4n1",
            "size": 32212254720
          },
          {
            "id": "259:1$fBOgXnM6DRfwqOJX4IkNNw2u0MEwzvBRjAIxhootiIs=",
            "majorMinor": "259:1",
            "make": "Amazon Elastic Block Store",
            "name": "nvme1n1",
            "size": 32212254720
          },
          {
            "id": "259:3$ezOPU9Oa/SQcVhrmeyDJ4lxcPQjEgYqYrZwOfzgsiVY=",
            "majorMinor": "259:3",
            "make": "Amazon Elastic Block Store",
            "name": "nvme3n1",
            "size": 32212254720
          }
        ]
      }
    }
  ],
  "kind": "List",
  "metadata": {
    "resourceVersion": ""
  }
}

# node-server 로그
kubectl stern -n directpv -l selector.directpv.min.io.service=enabled
+ node-server-jm5bk › node-controller
+ node-server-jm5bk › liveness-probe
+ node-server-jm5bk › node-driver-registrar
+ node-server-jm5bk › node-server
...

# controller 로그
kubectl stern -n directpv -l selector.directpv.min.io=controller-4h5ww
+ controller-6d678bbb-8tt8z › csi-provisioner
+ controller-6d678bbb-8tt8z › controller
+ controller-6d678bbb-8tt8z › csi-resizer
...

# controller 파드 1대 정보 확인
kc describe pod -n directpv controller-75d5595d5b-bqwbs
...
Containers:
  csi-provisioner:
    ...
    Args:
      --v=3
      --timeout=300s
      --csi-address=$(CSI_ENDPOINT)
      --leader-election
      --feature-gates=Topology=true
      --strict-topology
    ...
    Environment:
      CSI_ENDPOINT:  unix:///csi/csi.sock
    Mounts:
      /csi from socket-dir (rw)
  ...
  csi-resizer:
    ...
    Args:
      --v=3
      --timeout=300s
      --csi-address=$(CSI_ENDPOINT)
      --leader-election
    ...
    Environment:
      CSI_ENDPOINT:  unix:///csi/csi.sock
    Mounts:
      /csi from socket-dir (rw)
  ...
  controller:
    ...
    Args:
      controller
      --identity=directpv-min-io
      -v=3
      --csi-endpoint=$(CSI_ENDPOINT)
      --kube-node-name=$(KUBE_NODE_NAME)
      --readiness-port=30443
    ...
    Environment:
      KUBE_NODE_NAME:   (v1:spec.nodeName)
      CSI_ENDPOINT:    unix:///csi/csi.sock
    Mounts:
      /csi from socket-dir (rw)
...
Volumes:
  socket-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins/controller-controller
    HostPathType:  DirectoryOrCreate
...

ls -l /var/lib/kubelet/plugins/controller-controller
total 0
srwxr-xr-x 1 root root 0 Sep 20 11:33 csi.sock

# 누가 leader 인지 확인
k get lease -n directpv
NAME                               HOLDER                               AGE
directpv-min-io                    1758335591796-9374-directpv-min-io   8m5s
external-resizer-directpv-min-io   controller-77b4dd6494-pntfl          8m4s

k get lease -n directpv -o yaml | yq
{
  "apiVersion": "v1",
  "items": [
    {
      "apiVersion": "coordination.k8s.io/v1",
      "kind": "Lease",
      "metadata": {
        "creationTimestamp": "2025-09-20T02:33:11Z",
        "name": "directpv-min-io",
        "namespace": "directpv",
        "resourceVersion": "2568",
        "uid": "587b607f-1a66-4352-9cbc-da3878c8e3cf"
      },
      "spec": {
        "acquireTime": "2025-09-20T02:33:11.797196Z",
        "holderIdentity": "1758335591796-9374-directpv-min-io",
        "leaseDurationSeconds": 15,
        "leaseTransitions": 0,
        "renewTime": "2025-09-20T02:41:37.290618Z"
      }
    },
    {
      "apiVersion": "coordination.k8s.io/v1",
      "kind": "Lease",
      "metadata": {
        "creationTimestamp": "2025-09-20T02:33:12Z",
        "name": "external-resizer-directpv-min-io",
        "namespace": "directpv",
        "resourceVersion": "2569",
        "uid": "ed2e8598-976e-43b8-ab96-69738511891b"
      },
      "spec": {
        "acquireTime": "2025-09-20T02:33:12.244361Z",
        "holderIdentity": "controller-77b4dd6494-pntfl",
        "leaseDurationSeconds": 15,
        "leaseTransitions": 0,
        "renewTime": "2025-09-20T02:41:37.777013Z"
      }
    }
  ],
  "kind": "List",
  "metadata": {
    "resourceVersion": ""
  }
}

k get pod -n directpv -l selector.directpv.min.io.service=enabled
NAME                READY   STATUS    RESTARTS   AGE
node-server-l792h   4/4     Running   0          9m9s

kc describe pod -n directpv -l selector.directpv.min.io.service=enabled
Name:             node-server-l792h
Namespace:        directpv
Priority:         0
Service Account:  directpv-min-io
Node:             k3s-s/192.168.10.10
Start Time:       Sat, 20 Sep 2025 11:33:01 +0900
Labels:           controller-revision-hash=76c6d7d5b9
                  pod-template-generation=1
                  selector.directpv.min.io=directpv-min-io-d3an7
                  selector.directpv.min.io.service=enabled
Annotations:      created-by: kubectl-directpv
Status:           Running
IP:               10.42.0.5
IPs:
  IP:           10.42.0.5
Controlled By:  DaemonSet/node-server
Containers:
  node-driver-registrar:
    Container ID:  containerd://e547d9fed659baf8866b56e627ecde175fd572a377d60cba04ca48ea948f9f57
    Image:         quay.io/minio/csi-node-driver-registrar@sha256:dafc7f667aa2e20d7f059c20db02dd6987c2624d64d8f166cd5930721be98ea9
    Image ID:      quay.io/minio/csi-node-driver-registrar@sha256:dafc7f667aa2e20d7f059c20db02dd6987c2624d64d8f166cd5930721be98ea9
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=3
      --csi-address=unix:///csi/csi.sock
      --kubelet-registration-path=/var/lib/kubelet/plugins/directpv-min-io/csi.sock
    State:          Running
      Started:      Sat, 20 Sep 2025 11:33:05 +0900
    Ready:          True
    Restart Count:  0
    Environment:
      KUBE_NODE_NAME:   (v1:spec.nodeName)
    Mounts:
      /csi from socket-dir (rw)
      /registration from registration-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z6b2w (ro)
  node-server:
    Container ID:  containerd://bdf91ce5b4cf531b605c46d863cbf19187a893bcd6c8e95fdd795e57f5e52500
    Image:         quay.io/minio/directpv:v4.1.4
    Image ID:      quay.io/minio/directpv@sha256:83fd05fe114ed15c3975333c90cbe18c782d9c4d5c7ad6fdb8cc835e380ba505
    Ports:         30443/TCP, 9898/TCP, 10443/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP
    Args:
      node-server
      -v=3
      --identity=directpv-min-io
      --csi-endpoint=$(CSI_ENDPOINT)
      --kube-node-name=$(KUBE_NODE_NAME)
      --readiness-port=30443
      --metrics-port=10443
    State:          Running
      Started:      Sat, 20 Sep 2025 11:33:11 +0900
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:healthz/healthz delay=60s timeout=10s period=10s #success=1 #failure=5
    Readiness:      http-get http://:readinessport/ready delay=60s timeout=10s period=10s #success=1 #failure=5
    Environment:
      KUBE_NODE_NAME:   (v1:spec.nodeName)
      CSI_ENDPOINT:    unix:///csi/csi.sock
    Mounts:
      /csi from socket-dir (rw)
      /dev from devfs (ro)
      /run/udev/data from run-udev-data-dir (ro)
      /sys from sysfs (rw)
      /var/lib/direct-csi/ from direct-csi-common-root (rw)
      /var/lib/directpv/ from directpv-common-root (rw)
      /var/lib/kubelet/plugins from plugins-dir (rw)
      /var/lib/kubelet/pods from mountpoint-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z6b2w (ro)
  node-controller:
    Container ID:  containerd://3030445af8cac15c284cca614b81ad01c4c463f83d3e1081ddfe8d559d85f5bb
    Image:         quay.io/minio/directpv:v4.1.4
    Image ID:      quay.io/minio/directpv@sha256:83fd05fe114ed15c3975333c90cbe18c782d9c4d5c7ad6fdb8cc835e380ba505
    Port:          <none>
    Host Port:     <none>
    Args:
      node-controller
      -v=3
      --kube-node-name=$(KUBE_NODE_NAME)
    State:          Running
      Started:      Sat, 20 Sep 2025 11:33:11 +0900
    Ready:          True
    Restart Count:  0
    Environment:
      KUBE_NODE_NAME:   (v1:spec.nodeName)
    Mounts:
      /csi from socket-dir (rw)
      /dev from devfs (ro)
      /run/udev/data from run-udev-data-dir (ro)
      /sys from sysfs (rw)
      /var/lib/direct-csi/ from direct-csi-common-root (rw)
      /var/lib/directpv/ from directpv-common-root (rw)
      /var/lib/kubelet/plugins from plugins-dir (rw)
      /var/lib/kubelet/pods from mountpoint-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z6b2w (ro)
  liveness-probe:
    Container ID:  containerd://8ca4ce18d00186a9390e1a29ce5d077ea96950bf36d4efb52156d55c3996669b
    Image:         quay.io/minio/livenessprobe@sha256:783010e10e4d74b6b2b157a4b52772c5a264fd76bb2ad671054b8c3f706c8324
    Image ID:      quay.io/minio/livenessprobe@sha256:783010e10e4d74b6b2b157a4b52772c5a264fd76bb2ad671054b8c3f706c8324
    Port:          <none>
    Host Port:     <none>
    Args:
      --csi-address=/csi/csi.sock
      --health-port=9898
    State:          Running
      Started:      Sat, 20 Sep 2025 11:33:15 +0900
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /csi from socket-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-z6b2w (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  socket-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins/directpv-min-io
    HostPathType:  DirectoryOrCreate
  mountpoint-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/pods
    HostPathType:  DirectoryOrCreate
  registration-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins_registry
    HostPathType:  DirectoryOrCreate
  plugins-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins
    HostPathType:  DirectoryOrCreate
  directpv-common-root:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/directpv/
    HostPathType:  DirectoryOrCreate
  direct-csi-common-root:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/direct-csi/
    HostPathType:  DirectoryOrCreate
  sysfs:
    Type:          HostPath (bare host directory volume)
    Path:          /sys
    HostPathType:  DirectoryOrCreate
  devfs:
    Type:          HostPath (bare host directory volume)
    Path:          /dev
    HostPathType:  DirectoryOrCreate
  run-udev-data-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /run/udev/data
    HostPathType:  DirectoryOrCreate
  kube-api-access-z6b2w:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    Optional:                false
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  9m34s  default-scheduler  Successfully assigned directpv/node-server-l792h to k3s-s
  Normal  Pulling    9m35s  kubelet            Pulling image "quay.io/minio/csi-node-driver-registrar@sha256:dafc7f667aa2e20d7f059c20db02dd6987c2624d64d8f166cd5930721be98ea9"
  Normal  Pulled     9m31s  kubelet            Successfully pulled image "quay.io/minio/csi-node-driver-registrar@sha256:dafc7f667aa2e20d7f059c20db02dd6987c2624d64d8f166cd5930721be98ea9" in 3.589s (3.589s including waiting). Image size: 17235263 bytes.
  Normal  Created    9m31s  kubelet            Created container: node-driver-registrar
  Normal  Started    9m31s  kubelet            Started container node-driver-registrar
  Normal  Pulling    9m31s  kubelet            Pulling image "quay.io/minio/directpv:v4.1.4"
  Normal  Pulled     9m25s  kubelet            Successfully pulled image "quay.io/minio/directpv:v4.1.4" in 5.842s (5.842s including waiting). Image size: 94515090 bytes.
  Normal  Created    9m25s  kubelet            Created container: node-server
  Normal  Started    9m25s  kubelet            Started container node-server
  Normal  Pulled     9m25s  kubelet            Container image "quay.io/minio/directpv:v4.1.4" already present on machine
  Normal  Created    9m25s  kubelet            Created container: node-controller
  Normal  Started    9m25s  kubelet            Started container node-controller
  Normal  Pulling    9m25s  kubelet            Pulling image "quay.io/minio/livenessprobe@sha256:783010e10e4d74b6b2b157a4b52772c5a264fd76bb2ad671054b8c3f706c8324"
  Normal  Pulled     9m22s  kubelet            Successfully pulled image "quay.io/minio/livenessprobe@sha256:783010e10e4d74b6b2b157a4b52772c5a264fd76bb2ad671054b8c3f706c8324" in 3.294s (3.294s including waiting). Image size: 17370202 bytes.
  Normal  Created    9m22s  kubelet            Created container: liveness-probe
  Normal  Started    9m21s  kubelet            Started container liveness-probe

discover → init

DirectPV를 이용하여 물리디스크 관리

# 현재 노드의 모든 Disk가 관리가 되고 있지 않습니다.
k directpv info
┌─────────┬──────────┬───────────┬─────────┬────────┐
│ NODE    │ CAPACITY │ ALLOCATED │ VOLUMES │ DRIVES │
├─────────┼──────────┼───────────┼─────────┼────────┤
│ • k3s-s │ -        │ -         │ -       │ -      │
└─────────┴──────────┴───────────┴─────────┴────────┘

0 B/0 B used, 0 volumes, 0 drives

# 사용가능한 Disk 를 찾기
k directpv discover

Discovered node 'k3s-s' ✔

┌─────────────────────┬───────┬─────────┬────────┬────────────┬────────────────────────────┬───────────┬─────────────┐
│ ID                  │ NODE  │ DRIVE   │ SIZE   │ FILESYSTEM │ MAKE                       │ AVAILABLE │ DESCRIPTION │
├─────────────────────┼───────┼─────────┼────────┼────────────┼────────────────────────────┼───────────┼─────────────┤
│ 259:1$fBOgXnM6DR... │ k3s-s │ nvme1n1 │ 30 GiB │ -          │ Amazon Elastic Block Store │ YES       │ -           │
│ 259:0$xi6Ajz0HPz... │ k3s-s │ nvme2n1 │ 30 GiB │ -          │ Amazon Elastic Block Store │ YES       │ -           │
│ 259:3$ezOPU9Oa/S... │ k3s-s │ nvme3n1 │ 30 GiB │ -          │ Amazon Elastic Block Store │ YES       │ -           │
│ 259:4$r2C4ZMPnjz... │ k3s-s │ nvme4n1 │ 30 GiB │ -          │ Amazon Elastic Block Store │ YES       │ -           │
└─────────────────────┴───────┴─────────┴────────┴────────────┴────────────────────────────┴───────────┴─────────────┘

Generated 'drives.yaml' successfully.

# (참고) 적용 예외 설정 시 select: "no" 설정
cat drives.yaml | yq
{
  "version": "v1",
  "nodes": [
    {
      "name": "k3s-s",
      "drives": [
        {
          "id": "259:3$ezOPU9Oa/SQcVhrmeyDJ4lxcPQjEgYqYrZwOfzgsiVY=",
          "name": "nvme3n1",
          "size": 32212254720,
          "make": "Amazon Elastic Block Store",
          "select": "yes"
        },
        {
          "id": "259:4$r2C4ZMPnjz1wv/ZB7XNgDtEcvApI3aLSJN1f8ejPQbw=",
          "name": "nvme4n1",
          "size": 32212254720,
          "make": "Amazon Elastic Block Store",
          "select": "yes"
        },
        {
          "id": "259:1$fBOgXnM6DRfwqOJX4IkNNw2u0MEwzvBRjAIxhootiIs=",
          "name": "nvme1n1",
          "size": 32212254720,
          "make": "Amazon Elastic Block Store",
          "select": "yes"
        },
        {
          "id": "259:0$xi6Ajz0HPzT6wr/uMMViQ3psRBdWHeOwQOySleQps3o=",
          "name": "nvme2n1",
          "size": 32212254720,
          "make": "Amazon Elastic Block Store",
          "select": "yes"
        }
      ]
    }
  ]
}

# 초기화를 수행하면 기존 Disk의 내용이 삭제되므로 주의사항이 뜸
k directpv init drives.yaml
ERROR Initializing the drives will permanently erase existing data. Please review carefully before performing this *DANGEROUS* operation and retry this command with --dangerous flag.

# Perform initialization of drives which will permanently erase existing data
k directpv init drives.yaml --dangerous

███████████████████████████████████████████████████████████████████████████ 100%

Processed initialization request '810a7d56-1a42-4875-bdb7-cfecbe8dc668' for node 'k3s-s' ✔

┌──────────────────────────────────────┬───────┬─────────┬─────────┐
│ REQUEST_ID                           │ NODE  │ DRIVE   │ MESSAGE │
├──────────────────────────────────────┼───────┼─────────┼─────────┤
│ 810a7d56-1a42-4875-bdb7-cfecbe8dc668 │ k3s-s │ nvme1n1 │ Success │
│ 810a7d56-1a42-4875-bdb7-cfecbe8dc668 │ k3s-s │ nvme2n1 │ Success │
│ 810a7d56-1a42-4875-bdb7-cfecbe8dc668 │ k3s-s │ nvme3n1 │ Success │
│ 810a7d56-1a42-4875-bdb7-cfecbe8dc668 │ k3s-s │ nvme4n1 │ Success │
└──────────────────────────────────────┴───────┴─────────┴─────────┘

k directpv list drives
┌───────┬─────────┬────────────────────────────┬────────┬────────┬─────────┬────────┐
│ NODE  │ NAME    │ MAKE                       │ SIZE   │ FREE   │ VOLUMES │ STATUS │
├───────┼─────────┼────────────────────────────┼────────┼────────┼─────────┼────────┤
│ k3s-s │ nvme1n1 │ Amazon Elastic Block Store │ 30 GiB │ 30 GiB │ -       │ Ready  │
│ k3s-s │ nvme2n1 │ Amazon Elastic Block Store │ 30 GiB │ 30 GiB │ -       │ Ready  │
│ k3s-s │ nvme3n1 │ Amazon Elastic Block Store │ 30 GiB │ 30 GiB │ -       │ Ready  │
│ k3s-s │ nvme4n1 │ Amazon Elastic Block Store │ 30 GiB │ 30 GiB │ -       │ Ready  │
└───────┴─────────┴────────────────────────────┴────────┴────────┴─────────┴────────┘

# 현재 120G 를 사용할 수 있음
k directpv info
┌─────────┬──────────┬───────────┬─────────┬────────┐
│ NODE    │ CAPACITY │ ALLOCATED │ VOLUMES │ DRIVES │
├─────────┼──────────┼───────────┼─────────┼────────┤
│ • k3s-s │ 120 GiB  │ 0 B       │ 0       │ 4      │
└─────────┴──────────┴───────────┴─────────┴────────┘

0 B/120 GiB used, 0 volumes, 4 drives

# Disk 초기화가 되었기 때문에 Mount Point가 설정되었음
lsblk
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop0          7:0    0 27.6M  1 loop /snap/amazon-ssm-agent/11797
loop1          7:1    0 73.9M  1 loop /snap/core22/2111
loop2          7:2    0 50.8M  1 loop /snap/snapd/25202
loop3          7:3    0   16M  0 loop
nvme2n1      259:0    0   30G  0 disk /var/lib/directpv/mnt/94dca655-fc05-4e5f-a7d8-a8b1e953f220
nvme1n1      259:1    0   30G  0 disk /var/lib/directpv/mnt/afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445
nvme0n1      259:2    0   30G  0 disk
├─nvme0n1p1  259:5    0   29G  0 part /
├─nvme0n1p14 259:6    0    4M  0 part
├─nvme0n1p15 259:7    0  106M  0 part /boot/efi
└─nvme0n1p16 259:8    0  913M  0 part /boot
nvme3n1      259:3    0   30G  0 disk /var/lib/directpv/mnt/94ca8715-bbb0-47ad-93f9-43af207e0240
nvme4n1      259:4    0   30G  0 disk /var/lib/directpv/mnt/668267e9-fdd2-46c9-b854-1ac1de70132f
(⎈|default:N/A) root@k3s-s:~# df -hT --type xfs
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/nvme3n1   xfs    30G  248M   30G   1% /var/lib/directpv/mnt/94ca8715-bbb0-47ad-93f9-43af207e0240
/dev/nvme2n1   xfs    30G  248M   30G   1% /var/lib/directpv/mnt/94dca655-fc05-4e5f-a7d8-a8b1e953f220
/dev/nvme4n1   xfs    30G  248M   30G   1% /var/lib/directpv/mnt/668267e9-fdd2-46c9-b854-1ac1de70132f
/dev/nvme1n1   xfs    30G  248M   30G   1% /var/lib/directpv/mnt/afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445

tree -h /var/lib/directpv/
[4.0K]  /var/lib/directpv/
├── [4.0K]  mnt
│   ├── [  75]  668267e9-fdd2-46c9-b854-1ac1de70132f
│   ├── [  75]  94ca8715-bbb0-47ad-93f9-43af207e0240
│   ├── [  75]  94dca655-fc05-4e5f-a7d8-a8b1e953f220
│   └── [  75]  afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445
└── [  40]  tmp

7 directories, 0 files

# CR 확인
k get directpvdrives.directpv.min.io -o yaml | yq
{
  "apiVersion": "v1",
  "items": [
    {
      "apiVersion": "directpv.min.io/v1beta1",
      "kind": "DirectPVDrive",
      "metadata": {
        "creationTimestamp": "2025-09-20T02:56:01Z",
        "finalizers": [
          "directpv.min.io/data-protection"
        ],
        "generation": 1,
        "labels": {
          "directpv.min.io/access-tier": "Default",
          "directpv.min.io/created-by": "directpv-driver",
          "directpv.min.io/drive-name": "nvme4n1",
          "directpv.min.io/node": "k3s-s",
          "directpv.min.io/version": "v1beta1"
        },
        "name": "668267e9-fdd2-46c9-b854-1ac1de70132f",
        "resourceVersion": "3180",
        "uid": "c58a2447-5406-4068-98b7-0fd2421ecc7f"
      },
      "spec": {},
      "status": {
        "allocatedCapacity": 0,
        "freeCapacity": 32196100096,
        "fsuuid": "668267e9-fdd2-46c9-b854-1ac1de70132f",
        "make": "Amazon Elastic Block Store",
        "status": "Ready",
        "topology": {
          "directpv.min.io/identity": "directpv-min-io",
          "directpv.min.io/node": "k3s-s",
          "directpv.min.io/rack": "default",
          "directpv.min.io/region": "default",
          "directpv.min.io/zone": "default"
        },
        "totalCapacity": 32212254720
      }
    },
    {
      "apiVersion": "directpv.min.io/v1beta1",
      "kind": "DirectPVDrive",
      "metadata": {
        "creationTimestamp": "2025-09-20T02:56:01Z",
        "finalizers": [
          "directpv.min.io/data-protection"
        ],
        "generation": 1,
        "labels": {
          "directpv.min.io/access-tier": "Default",
          "directpv.min.io/created-by": "directpv-driver",
          "directpv.min.io/drive-name": "nvme3n1",
          "directpv.min.io/node": "k3s-s",
          "directpv.min.io/version": "v1beta1"
        },
        "name": "94ca8715-bbb0-47ad-93f9-43af207e0240",
        "resourceVersion": "3178",
        "uid": "bbf8a31e-4e26-4742-a9db-941c60064ee0"
      },
      "spec": {},
      "status": {
        "allocatedCapacity": 0,
        "freeCapacity": 32196100096,
        "fsuuid": "94ca8715-bbb0-47ad-93f9-43af207e0240",
        "make": "Amazon Elastic Block Store",
        "status": "Ready",
        "topology": {
          "directpv.min.io/identity": "directpv-min-io",
          "directpv.min.io/node": "k3s-s",
          "directpv.min.io/rack": "default",
          "directpv.min.io/region": "default",
          "directpv.min.io/zone": "default"
        },
        "totalCapacity": 32212254720
      }
    },
    {
      "apiVersion": "directpv.min.io/v1beta1",
      "kind": "DirectPVDrive",
      "metadata": {
        "creationTimestamp": "2025-09-20T02:56:01Z",
        "finalizers": [
          "directpv.min.io/data-protection"
        ],
        "generation": 1,
        "labels": {
          "directpv.min.io/access-tier": "Default",
          "directpv.min.io/created-by": "directpv-driver",
          "directpv.min.io/drive-name": "nvme2n1",
          "directpv.min.io/node": "k3s-s",
          "directpv.min.io/version": "v1beta1"
        },
        "name": "94dca655-fc05-4e5f-a7d8-a8b1e953f220",
        "resourceVersion": "3179",
        "uid": "16b03e2e-ddfd-4de6-9af9-f0e70a2a4f32"
      },
      "spec": {},
      "status": {
        "allocatedCapacity": 0,
        "freeCapacity": 32196100096,
        "fsuuid": "94dca655-fc05-4e5f-a7d8-a8b1e953f220",
        "make": "Amazon Elastic Block Store",
        "status": "Ready",
        "topology": {
          "directpv.min.io/identity": "directpv-min-io",
          "directpv.min.io/node": "k3s-s",
          "directpv.min.io/rack": "default",
          "directpv.min.io/region": "default",
          "directpv.min.io/zone": "default"
        },
        "totalCapacity": 32212254720
      }
    },
    {
      "apiVersion": "directpv.min.io/v1beta1",
      "kind": "DirectPVDrive",
      "metadata": {
        "creationTimestamp": "2025-09-20T02:56:01Z",
        "finalizers": [
          "directpv.min.io/data-protection"
        ],
        "generation": 1,
        "labels": {
          "directpv.min.io/access-tier": "Default",
          "directpv.min.io/created-by": "directpv-driver",
          "directpv.min.io/drive-name": "nvme1n1",
          "directpv.min.io/node": "k3s-s",
          "directpv.min.io/version": "v1beta1"
        },
        "name": "afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445",
        "resourceVersion": "3181",
        "uid": "158b3114-7960-4d06-af89-8b9b97c8d40e"
      },
      "spec": {},
      "status": {
        "allocatedCapacity": 0,
        "freeCapacity": 32196100096,
        "fsuuid": "afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445",
        "make": "Amazon Elastic Block Store",
        "status": "Ready",
        "topology": {
          "directpv.min.io/identity": "directpv-min-io",
          "directpv.min.io/node": "k3s-s",
          "directpv.min.io/rack": "default",
          "directpv.min.io/region": "default",
          "directpv.min.io/zone": "default"
        },
        "totalCapacity": 32212254720
      }
    }
  ],
  "kind": "List",
  "metadata": {
    "resourceVersion": ""
  }
}

k get directpvdrives.directpv.min.io
NAME                                   AGE
668267e9-fdd2-46c9-b854-1ac1de70132f   58s
94ca8715-bbb0-47ad-93f9-43af207e0240   58s
94dca655-fc05-4e5f-a7d8-a8b1e953f220   58s
afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445   58s

# OS 가 재기동되면 Mount 정보가 사라지므로, OS 재기동 후 유지하려면 /etc/fstab 파일 수정할 것
cat /etc/fstab
LABEL=cloudimg-rootfs   /        ext4   discard,commit=30,errors=remount-ro     0 1
LABEL=BOOT      /boot   ext4    defaults        0 2
LABEL=UEFI      /boot/efi       vfat    umask=0077      0 1

DirectPV 간단 테스트

# 현재 Drive만 만들었고, Volume은 아직 만들지 않은 상태입니다.
df -hT --type xfs
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/nvme3n1   xfs    30G  248M   30G   1% /var/lib/directpv/mnt/94ca8715-bbb0-47ad-93f9-43af207e0240
/dev/nvme2n1   xfs    30G  248M   30G   1% /var/lib/directpv/mnt/94dca655-fc05-4e5f-a7d8-a8b1e953f220
/dev/nvme4n1   xfs    30G  248M   30G   1% /var/lib/directpv/mnt/668267e9-fdd2-46c9-b854-1ac1de70132f
/dev/nvme1n1   xfs    30G  248M   30G   1% /var/lib/directpv/mnt/afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445

k get directpvdrives,directpvvolumes
NAME                                                                 AGE
directpvdrive.directpv.min.io/668267e9-fdd2-46c9-b854-1ac1de70132f   23m
directpvdrive.directpv.min.io/94ca8715-bbb0-47ad-93f9-43af207e0240   23m
directpvdrive.directpv.min.io/94dca655-fc05-4e5f-a7d8-a8b1e953f220   23m
directpvdrive.directpv.min.io/afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445   23m

# PVC를 만들때 SC를 directpv-min-io 를 지정하면 PV가 만들어지고 실제 볼륨도 생성됩니다.
#
cat << EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nginx-pvc
spec:
  volumeMode: Filesystem
  storageClassName: directpv-min-io
  accessModes: [ "ReadWriteOnce" ]
  resources:
    requests:
      storage: 8Mi
---
apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
spec:
  volumes:
    - name: nginx-volume
      persistentVolumeClaim:
        claimName: nginx-pvc
  containers:
    - name: nginx-container
      image: nginx:alpine
      volumeMounts:
        - mountPath: "/mnt"
          name: nginx-volume
EOF
persistentvolumeclaim/nginx-pvc unchanged
pod/nginx-pod created

k get pod,pvc,pv
NAME            READY   STATUS    RESTARTS   AGE
pod/nginx-pod   1/1     Running   0          93s

NAME                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      VOLUMEATTRIBUTESCLASS   AGE
persistentvolumeclaim/nginx-pvc   Bound    pvc-5d47ac1c-7d95-4e4c-8cef-357987f64e82   8Mi        RWO            directpv-min-io   <unset>                 2m27s

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS      VOLUMEATTRIBUTESCLASS   REASON   AGE
persistentvolume/pvc-5d47ac1c-7d95-4e4c-8cef-357987f64e82   8Mi        RWO            Delete           Bound    default/nginx-pvc   directpv-min-io   <unset>                          93s

# nginx POD에서 mount 정보를 확인해보면 8M 사이즈의 볼륨이 마운트 되었습니다.
k exec -it nginx-pod -- df -hT -t xfs
Filesystem           Type            Size      Used Available Use% Mounted on
/dev/nvme4n1         xfs             8.0M         0      8.0M   0% /mnt

# hello.txt 파일 생성
k exec -it nginx-pod -- sh -c 'echo hello > /mnt/hello.txt'
k exec -it nginx-pod -- sh -c 'cat /mnt/hello.txt'
hello

# HOST 에서 Block 정보 확인
lsblk
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop0          7:0    0 27.6M  1 loop /snap/amazon-ssm-agent/11797
loop1          7:1    0 73.9M  1 loop /snap/core22/2111
loop2          7:2    0 50.8M  1 loop /snap/snapd/25202
loop3          7:3    0   16M  0 loop
nvme2n1      259:0    0   30G  0 disk /var/lib/directpv/mnt/94dca655-fc05-4e5f-a7d8-a8b1e953f220
nvme1n1      259:1    0   30G  0 disk /var/lib/directpv/mnt/afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445
nvme0n1      259:2    0   30G  0 disk
├─nvme0n1p1  259:5    0   29G  0 part /
├─nvme0n1p14 259:6    0    4M  0 part
├─nvme0n1p15 259:7    0  106M  0 part /boot/efi
└─nvme0n1p16 259:8    0  913M  0 part /boot
nvme3n1      259:3    0   30G  0 disk /var/lib/directpv/mnt/94ca8715-bbb0-47ad-93f9-43af207e0240
nvme4n1      259:4    0   30G  0 disk /var/lib/kubelet/pods/476727a6-6ab9-4862-b4df-4036b2b3791f/volumes/kubernetes.io~csi/pvc-5d47ac1c-7d95-4e4c-8cef-357987f64e82/mount
                                      /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/ece2c81c42117a6e6a095e54ba4ac74caa9f3fde023d6544fc4adf56513fbf27/globalmount
                                      /var/lib/directpv/mnt/668267e9-fdd2-46c9-b854-1ac1de70132f

tree -a /var/lib/directpv/mnt
/var/lib/directpv/mnt
├── 668267e9-fdd2-46c9-b854-1ac1de70132f
│   ├── .FSUUID.668267e9-fdd2-46c9-b854-1ac1de70132f -> .
│   ├── .directpv
│   │   └── meta.info
│   └── pvc-5d47ac1c-7d95-4e4c-8cef-357987f64e82
│       └── hello.txt
├── 94ca8715-bbb0-47ad-93f9-43af207e0240
│   ├── .FSUUID.94ca8715-bbb0-47ad-93f9-43af207e0240 -> .
│   └── .directpv
│       └── meta.info
├── 94dca655-fc05-4e5f-a7d8-a8b1e953f220
│   ├── .FSUUID.94dca655-fc05-4e5f-a7d8-a8b1e953f220 -> .
│   └── .directpv
│       └── meta.info
└── afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445
    ├── .FSUUID.afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445 -> .
    └── .directpv
        └── meta.info

cat /var/lib/directpv/mnt/*/pvc*/hello.txt
hello

k get directpvvolumes
NAME                                       AGE
pvc-5d47ac1c-7d95-4e4c-8cef-357987f64e82   5m49s

k get directpvvolumes -o yaml | yq
{
  "apiVersion": "v1",
  "items": [
    {
      "apiVersion": "directpv.min.io/v1beta1",
      "kind": "DirectPVVolume",
      "metadata": {
        "creationTimestamp": "2025-09-20T03:23:28Z",
        "finalizers": [
          "directpv.min.io/pv-protection",
          "directpv.min.io/purge-protection"
        ],
        "generation": 3,
        "labels": {
          "directpv.min.io/created-by": "directpv-controller",
          "directpv.min.io/drive": "668267e9-fdd2-46c9-b854-1ac1de70132f",
          "directpv.min.io/drive-name": "nvme4n1",
          "directpv.min.io/node": "k3s-s",
          "directpv.min.io/pod.name": "nginx-pod",
          "directpv.min.io/pod.namespace": "default",
          "directpv.min.io/version": "v1beta1"
        },
        "name": "pvc-5d47ac1c-7d95-4e4c-8cef-357987f64e82",
        "resourceVersion": "4366",
        "uid": "94d5c75e-b7cc-4d7d-ae0c-d35211c1085b"
      },
      "status": {
        "availableCapacity": 8388608,
        "dataPath": "/var/lib/directpv/mnt/668267e9-fdd2-46c9-b854-1ac1de70132f/.FSUUID.668267e9-fdd2-46c9-b854-1ac1de70132f/pvc-5d47ac1c-7d95-4e4c-8cef-357987f64e82",
        "fsuuid": "668267e9-fdd2-46c9-b854-1ac1de70132f",
        "stagingTargetPath": "/var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/ece2c81c42117a6e6a095e54ba4ac74caa9f3fde023d6544fc4adf56513fbf27/globalmount",
        "status": "Ready",
        "targetPath": "/var/lib/kubelet/pods/476727a6-6ab9-4862-b4df-4036b2b3791f/volumes/kubernetes.io~csi/pvc-5d47ac1c-7d95-4e4c-8cef-357987f64e82/mount",
        "totalCapacity": 8388608,
        "usedCapacity": 0
      }
    }
  ],
  "kind": "List",
  "metadata": {
    "resourceVersion": ""
  }
}

# NGINX POD를 삭제해도 PVC, PV는 삭제되지 않음
k delete pod nginx-pod
pod "nginx-pod" deleted

k get pvc,pv
NAME                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      VOLUMEATTRIBUTESCLASS   AGE
persistentvolumeclaim/nginx-pvc   Bound    pvc-5d47ac1c-7d95-4e4c-8cef-357987f64e82   8Mi        RWO            directpv-min-io   <unset>                 7m23s

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS      VOLUMEATTRIBUTESCLASS   REASON   AGE
persistentvolume/pvc-5d47ac1c-7d95-4e4c-8cef-357987f64e82   8Mi        RWO            Delete           Bound    default/nginx-pvc   directpv-min-io   <unset>                          6m29s

# PVC를 삭제하면 PV도 삭제되고 실제 볼륨도 삭제됨
k delete pvc nginx-pvc
persistentvolumeclaim "nginx-pvc" deleted

k get pvc,pv
No resources found

lsblk
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop0          7:0    0 27.6M  1 loop /snap/amazon-ssm-agent/11797
loop1          7:1    0 73.9M  1 loop /snap/core22/2111
loop2          7:2    0 50.8M  1 loop /snap/snapd/25202
loop3          7:3    0   16M  0 loop
nvme2n1      259:0    0   30G  0 disk /var/lib/directpv/mnt/94dca655-fc05-4e5f-a7d8-a8b1e953f220
nvme1n1      259:1    0   30G  0 disk /var/lib/directpv/mnt/afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445
nvme0n1      259:2    0   30G  0 disk
├─nvme0n1p1  259:5    0   29G  0 part /
├─nvme0n1p14 259:6    0    4M  0 part
├─nvme0n1p15 259:7    0  106M  0 part /boot/efi
└─nvme0n1p16 259:8    0  913M  0 part /boot
nvme3n1      259:3    0   30G  0 disk /var/lib/directpv/mnt/94ca8715-bbb0-47ad-93f9-43af207e0240
nvme4n1      259:4    0   30G  0 disk /var/lib/directpv/mnt/668267e9-fdd2-46c9-b854-1ac1de70132f

tree -a /var/lib/directpv/mnt
/var/lib/directpv/mnt
├── 668267e9-fdd2-46c9-b854-1ac1de70132f
│   ├── .FSUUID.668267e9-fdd2-46c9-b854-1ac1de70132f -> .
│   └── .directpv
│       └── meta.info
├── 94ca8715-bbb0-47ad-93f9-43af207e0240
│   ├── .FSUUID.94ca8715-bbb0-47ad-93f9-43af207e0240 -> .
│   └── .directpv
│       └── meta.info
├── 94dca655-fc05-4e5f-a7d8-a8b1e953f220
│   ├── .FSUUID.94dca655-fc05-4e5f-a7d8-a8b1e953f220 -> .
│   └── .directpv
│       └── meta.info
└── afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445
    ├── .FSUUID.afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445 -> .
    └── .directpv
        └── meta.info

13 directories, 4 files

특정 Drive 에 볼륨 사용 테스트

쿠버네티스 환경에서 API 리소스에 사용하는 Label은 아키텍처적으로 중요한 역할을 합니다. 특정 Label 이 있는 리소스를 필터링해서 Self Healing을 하거나, 네트워크 패킷을 포워딩하거나, 특정 Drive를 선택하게 할 수 있습니다.

vi create-storage-class.sh

# 첫번째 Disk에 tier=fast 레이블을 적용
# Label the 'nvme1n1' drive in all nodes as 'fast' with the 'tier' key.
kubectl directpv label drives --drives=nvme1n1 tier=fast
Label 'directpv.min.io/tier:fast' successfully set on k3s-s/nvme1n1

# Verify that the labels are properly set by using the list command
k directpv list drives --show-labels
┌───────┬─────────┬────────────────────────────┬────────┬────────┬─────────┬────────┬───────────┐
│ NODE  │ NAME    │ MAKE                       │ SIZE   │ FREE   │ VOLUMES │ STATUS │ LABELS    │
├───────┼─────────┼────────────────────────────┼────────┼────────┼─────────┼────────┼───────────┤
│ k3s-s │ nvme1n1 │ Amazon Elastic Block Store │ 30 GiB │ 30 GiB │ -       │ Ready  │ tier=fast │
│ k3s-s │ nvme2n1 │ Amazon Elastic Block Store │ 30 GiB │ 30 GiB │ -       │ Ready  │ -         │
│ k3s-s │ nvme3n1 │ Amazon Elastic Block Store │ 30 GiB │ 30 GiB │ -       │ Ready  │ -         │
│ k3s-s │ nvme4n1 │ Amazon Elastic Block Store │ 30 GiB │ 30 GiB │ -       │ Ready  │ -         │
└───────┴─────────┴────────────────────────────┴────────┴────────┴─────────┴────────┴───────────┘

k get sc
NAME                   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
directpv-min-io        directpv-min-io         Delete          WaitForFirstConsumer   true                   119m
local-path (default)   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  3h24m

# DirectPV를 이용하여 새로운 스토리지 클래스를 생성하는 스크립트 생성
# https://docs.min.io/community/minio-directpv/resource-management/scripts/#create-storage-classsh
vi create-storage-class.sh
chmod +x create-storage-class.sh

# 도움말 확인
./create-storage-class.sh -h
USAGE:
  create-storage-class.sh <NAME> <DRIVE-LABELS> ...
ARGUMENTS:
  NAME           new storage class name.
  DRIVE-LABELS   drive labels to be attached.
EXAMPLE:
  # Create new storage class 'fast-tier-storage' with drive labels 'directpv.min.io/tier: fast'
  $ create-storage-class.sh fast-tier-storage 'directpv.min.io/tier: fast'
  # Create new storage class with more than one drive label
  $ create-storage-class.sh fast-tier-unique 'directpv.min.io/tier: fast' 'directpv.min.io/volume-claim-id: bcea279a-df70-4d23-be41-9490f9933004'

# directpv.min.io/tier: fast 레이블을 이용하여 SC 생성
./create-storage-class.sh fast-tier-storage 'directpv.min.io/tier: fast'
storageclass.storage.k8s.io/fast-tier-storage created

k get sc
NAME                   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
directpv-min-io        directpv-min-io         Delete          WaitForFirstConsumer   true                   120m
fast-tier-storage      directpv-min-io         Delete          WaitForFirstConsumer   true                   7s
local-path (default)   rancher.io/local-path   Delete          WaitForFirstConsumer   false                  3h25m

kc describe sc fast-tier-storag
Name:            fast-tier-storage
IsDefaultClass:  No
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"allowVolumeExpansion":true,"allowedTopologies":[{"matchLabelExpressions":[{"key":"directpv.min.io/identity","values":["directpv-min-io"]}]}],"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"finalizers":["foregroundDeletion"],"labels":{"application-name":"directpv.min.io","application-type":"CSIDriver","directpv.min.io/created-by":"kubectl-directpv","directpv.min.io/version":"v1beta1"},"name":"fast-tier-storage"},"parameters":{"directpv.min.io/tier":"fast","fstype":"xfs"},"provisioner":"directpv-min-io","reclaimPolicy":"Delete","volumeBindingMode":"WaitForFirstConsumer"}

Provisioner:           directpv-min-io
Parameters:            directpv.min.io/tier=fast,fstype=xfs
AllowVolumeExpansion:  True
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     WaitForFirstConsumer
AllowedTopologies:
  Term 0:              directpv.min.io/identity in [directpv-min-io]
Events:                <none>

# 특정레이블이 붙어 있는 DirectPV를 사용하는 PVC,PV, POD 생성
cat << EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nginx-pvc
spec:
  volumeMode: Filesystem
  storageClassName: fast-tier-storage
  accessModes: [ "ReadWriteOnce" ]
  resources:
    requests:
      storage: 8Mi
---
apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod
spec:
  volumes:
    - name: nginx-volume
      persistentVolumeClaim:
        claimName: nginx-pvc
  containers:
    - name: nginx-container
      image: nginx:alpine
      volumeMounts:
        - mountPath: "/mnt"
          name: nginx-volume
EOF
persistentvolumeclaim/nginx-pvc created
pod/nginx-pod created

# 해당 drive가 있는 node 에 pv 가 만들어지고, 파드 역시 해당 node 에 기동되었음
k get pod,pvc,pv -owide
NAME            READY   STATUS    RESTARTS   AGE   IP           NODE    NOMINATED NODE   READINESS GATES
pod/nginx-pod   1/1     Running   0          15s   10.42.0.10   k3s-s   <none>           <none>

NAME                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS        VOLUMEATTRIBUTESCLASS   AGE   VOLUMEMODE
persistentvolumeclaim/nginx-pvc   Bound    pvc-825cf353-c880-449f-8482-15770a1ec908   8Mi        RWO            fast-tier-storage   <unset>                 15s   Filesystem

NAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS        VOLUMEATTRIBUTESCLASS   REASON   AGE   VOLUMEMODE
persistentvolume/pvc-825cf353-c880-449f-8482-15770a1ec908   8Mi        RWO            Delete           Bound    default/nginx-pvc   fast-tier-storage   <unset>                          15s   Filesystem

kc describe pv
Name:              pvc-825cf353-c880-449f-8482-15770a1ec908
Labels:            <none>
Annotations:       pv.kubernetes.io/provisioned-by: directpv-min-io
                   volume.kubernetes.io/provisioner-deletion-secret-name:
                   volume.kubernetes.io/provisioner-deletion-secret-namespace:
Finalizers:        [external-provisioner.volume.kubernetes.io/finalizer kubernetes.io/pv-protection]
StorageClass:      fast-tier-storage
Status:            Bound
Claim:             default/nginx-pvc
Reclaim Policy:    Delete
Access Modes:      RWO
VolumeMode:        Filesystem
Capacity:          8Mi
Node Affinity:
  Required Terms:
    Term 0:        directpv.min.io/region in [default]
                   directpv.min.io/zone in [default]
                   directpv.min.io/identity in [directpv-min-io]
                   directpv.min.io/node in [k3s-s]
                   directpv.min.io/rack in [default]
Message:
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            directpv-min-io
    FSType:            xfs
    VolumeHandle:      pvc-825cf353-c880-449f-8482-15770a1ec908
    ReadOnly:          false
    VolumeAttributes:      directpv.min.io/tier=fast
                           fstype=xfs
                           storage.kubernetes.io/csiProvisionerIdentity=1758335591796-9374-directpv-min-io
Events:                <none>

kc describe node
Name:               k3s-s
Roles:              control-plane,master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=k3s
                    beta.kubernetes.io/os=linux
                    directpv.min.io/identity=directpv-min-io
                    directpv.min.io/node=k3s-s
                    directpv.min.io/rack=default
                    directpv.min.io/region=default
                    directpv.min.io/zone=default
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=k3s-s
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=true
                    node-role.kubernetes.io/master=true
                    node.kubernetes.io/instance-type=k3s
Annotations:        alpha.kubernetes.io/provided-node-ip: 192.168.10.10
                    csi.volume.kubernetes.io/nodeid: {"directpv-min-io":"k3s-s"}
                    flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"a2:6c:e1:80:d1:28"}
                    flannel.alpha.coreos.com/backend-type: vxlan
                    flannel.alpha.coreos.com/kube-subnet-manager: true
                    flannel.alpha.coreos.com/public-ip: 192.168.10.10
                    k3s.io/hostname: k3s-s
                    k3s.io/internal-ip: 192.168.10.10
                    k3s.io/node-args: ["server","--disable","traefik","server","--token","********"]
                    k3s.io/node-config-hash: RE5Q4OCGORC6RD2UFXPER7OWY5V7RM6A3BSJ6UPUELZ3Z3HOPRCQ====
                    k3s.io/node-env: {"K3S_KUBECONFIG_MODE":"644"}
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Sat, 20 Sep 2025 10:07:54 +0900
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  k3s-s
  AcquireTime:     <unset>
  RenewTime:       Sat, 20 Sep 2025 13:34:13 +0900
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Sat, 20 Sep 2025 13:30:20 +0900   Sat, 20 Sep 2025 10:07:54 +0900   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Sat, 20 Sep 2025 13:30:20 +0900   Sat, 20 Sep 2025 10:07:54 +0900   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Sat, 20 Sep 2025 13:30:20 +0900   Sat, 20 Sep 2025 10:07:54 +0900   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Sat, 20 Sep 2025 13:30:20 +0900   Sat, 20 Sep 2025 10:07:54 +0900   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  192.168.10.10
  Hostname:    k3s-s
Capacity:
  cpu:                4
  ephemeral-storage:  29378688Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             16167188Ki
  pods:               110
Allocatable:
  cpu:                4
  ephemeral-storage:  28579587664
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             16167188Ki
  pods:               110
System Info:
  Machine ID:                 ec29fa949c1835c935138ba971cbb155
  System UUID:                ec29fa94-9c18-35c9-3513-8ba971cbb155
  Boot ID:                    90ef457b-6b4c-4b14-ab8f-500a0c3afbcc
  Kernel Version:             6.14.0-1012-aws
  OS Image:                   Ubuntu 24.04.3 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://2.0.5-k3s2
  Kubelet Version:            v1.33.4+k3s1
  Kube-Proxy Version:
PodCIDR:                      10.42.0.0/24
PodCIDRs:                     10.42.0.0/24
ProviderID:                   k3s://k3s-s
Non-terminated Pods:          (8 in total)
  Namespace                   Name                                       CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                       ------------  ----------  ---------------  -------------  ---
  default                     nginx-pod                                  0 (0%)        0 (0%)      0 (0%)           0 (0%)         38s
  directpv                    controller-77b4dd6494-7nhs7                0 (0%)        0 (0%)      0 (0%)           0 (0%)         121m
  directpv                    controller-77b4dd6494-pntfl                0 (0%)        0 (0%)      0 (0%)           0 (0%)         121m
  directpv                    controller-77b4dd6494-sqfsn                0 (0%)        0 (0%)      0 (0%)           0 (0%)         121m
  directpv                    node-server-l792h                          0 (0%)        0 (0%)      0 (0%)           0 (0%)         121m
  kube-system                 coredns-64fd4b4794-ms5h2                   100m (2%)     0 (0%)      70Mi (0%)        170Mi (1%)     3h26m
  kube-system                 local-path-provisioner-774c6665dc-bmshm    0 (0%)        0 (0%)      0 (0%)           0 (0%)         3h26m
  kube-system                 metrics-server-7bfffcd44-45wzq             100m (2%)     0 (0%)      70Mi (0%)        0 (0%)         3h26m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                200m (5%)   0 (0%)
  memory             140Mi (0%)  170Mi (1%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-1Gi      0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
Events:              <none>

k get directpvnodes.directpv.min.io -o yaml | yq
{
  "apiVersion": "v1",
  "items": [
    {
      "apiVersion": "directpv.min.io/v1beta1",
      "kind": "DirectPVNode",
      "metadata": {
        "creationTimestamp": "2025-09-20T02:33:11Z",
        "generation": 3,
        "labels": {
          "directpv.min.io/created-by": "node-controller",
          "directpv.min.io/node": "k3s-s",
          "directpv.min.io/version": "v1beta1"
        },
        "name": "k3s-s",
        "resourceVersion": "3135",
        "uid": "1cc32f4b-deee-4a39-82a8-3814b07339c9"
      },
      "spec": {},
      "status": {
        "devices": [
          {
            "deniedReason": "Too small; Mounted",
            "fsType": "vfat",
            "fsuuid": "F567-A352",
            "id": "259:7$4Ghr/DjNZtwvOsBElnLkwg4okZ2vh00OJifip6QiZfc=",
            "majorMinor": "259:7",
            "make": "Amazon Elastic Block Store (Part 15)",
            "name": "nvme0n1p15",
            "size": 111149056
          },
          {
            "id": "259:3$ezOPU9Oa/SQcVhrmeyDJ4lxcPQjEgYqYrZwOfzgsiVY=",
            "majorMinor": "259:3",
            "make": "Amazon Elastic Block Store",
            "name": "nvme3n1",
            "size": 32212254720
          },
          {
            "id": "259:4$r2C4ZMPnjz1wv/ZB7XNgDtEcvApI3aLSJN1f8ejPQbw=",
            "majorMinor": "259:4",
            "make": "Amazon Elastic Block Store",
            "name": "nvme4n1",
            "size": 32212254720
          },
          {
            "id": "259:1$fBOgXnM6DRfwqOJX4IkNNw2u0MEwzvBRjAIxhootiIs=",
            "majorMinor": "259:1",
            "make": "Amazon Elastic Block Store",
            "name": "nvme1n1",
            "size": 32212254720
          },
          {
            "id": "259:0$xi6Ajz0HPzT6wr/uMMViQ3psRBdWHeOwQOySleQps3o=",
            "majorMinor": "259:0",
            "make": "Amazon Elastic Block Store",
            "name": "nvme2n1",
            "size": 32212254720
          },
          {
            "deniedReason": "Too small",
            "id": "259:6$lVCOx26hZO2s872NOjNJra4S5ZaX9+Q56CQObs9o5+w=",
            "majorMinor": "259:6",
            "make": "Amazon Elastic Block Store (Part 14)",
            "name": "nvme0n1p14",
            "size": 4194304
          },
          {
            "deniedReason": "Mounted",
            "fsType": "ext4",
            "fsuuid": "13015197-4a14-4c28-bbd2-85341b3dff09",
            "id": "259:8$QUneuLFDTeOsXC2uD/VKWrKx/gv975cbvXzIYrtxX5Q=",
            "majorMinor": "259:8",
            "make": "Amazon Elastic Block Store (Part 16)",
            "name": "nvme0n1p16",
            "size": 957350400
          },
          {
            "deniedReason": "Mounted",
            "fsType": "ext4",
            "fsuuid": "04a692e8-fb29-4bab-b64d-906697110bb7",
            "id": "259:5$oDeIgERHCSJIXrymn/bbYWBp7fQGVXvfqoHG05mL1dE=",
            "majorMinor": "259:5",
            "make": "Amazon Elastic Block Store (Part 1)",
            "name": "nvme0n1p1",
            "size": 31137447424
          },
          {
            "deniedReason": "Partitioned",
            "id": "259:2$jc81R04kRB2F2ve1IxDxP8tYwx31Sh++HnpXlpnEmkk=",
            "majorMinor": "259:2",
            "make": "Amazon Elastic Block Store",
            "name": "nvme0n1",
            "size": 32212254720
          }
        ]
      }
    }
  ],
  "kind": "List",
  "metadata": {
    "resourceVersion": ""
  }
}

kc describe directpvdrives.directpv.min.io
Name:         668267e9-fdd2-46c9-b854-1ac1de70132f
Namespace:
Labels:       directpv.min.io/access-tier=Default
              directpv.min.io/created-by=directpv-driver
              directpv.min.io/drive-name=nvme4n1
              directpv.min.io/node=k3s-s
              directpv.min.io/version=v1beta1
Annotations:  <none>
API Version:  directpv.min.io/v1beta1
Kind:         DirectPVDrive
Metadata:
  Creation Timestamp:  2025-09-20T02:56:01Z
  Finalizers:
    directpv.min.io/data-protection
  Generation:        3
  Resource Version:  4662
  UID:               c58a2447-5406-4068-98b7-0fd2421ecc7f
Spec:
Status:
  Allocated Capacity:  16154624
  Free Capacity:       32196100096
  Fsuuid:              668267e9-fdd2-46c9-b854-1ac1de70132f
  Make:                Amazon Elastic Block Store
  Status:              Ready
  Topology:
    directpv.min.io/identity:  directpv-min-io
    directpv.min.io/node:      k3s-s
    directpv.min.io/rack:      default
    directpv.min.io/region:    default
    directpv.min.io/zone:      default
  Total Capacity:              32212254720
Events:                        <none>

Name:         94ca8715-bbb0-47ad-93f9-43af207e0240
Namespace:
Labels:       directpv.min.io/access-tier=Default
              directpv.min.io/created-by=directpv-driver
              directpv.min.io/drive-name=nvme3n1
              directpv.min.io/node=k3s-s
              directpv.min.io/version=v1beta1
Annotations:  <none>
API Version:  directpv.min.io/v1beta1
Kind:         DirectPVDrive
Metadata:
  Creation Timestamp:  2025-09-20T02:56:01Z
  Finalizers:
    directpv.min.io/data-protection
  Generation:        1
  Resource Version:  3178
  UID:               bbf8a31e-4e26-4742-a9db-941c60064ee0
Spec:
Status:
  Allocated Capacity:  0
  Free Capacity:       32196100096
  Fsuuid:              94ca8715-bbb0-47ad-93f9-43af207e0240
  Make:                Amazon Elastic Block Store
  Status:              Ready
  Topology:
    directpv.min.io/identity:  directpv-min-io
    directpv.min.io/node:      k3s-s
    directpv.min.io/rack:      default
    directpv.min.io/region:    default
    directpv.min.io/zone:      default
  Total Capacity:              32212254720
Events:                        <none>

Name:         94dca655-fc05-4e5f-a7d8-a8b1e953f220
Namespace:
Labels:       directpv.min.io/access-tier=Default
              directpv.min.io/created-by=directpv-driver
              directpv.min.io/drive-name=nvme2n1
              directpv.min.io/node=k3s-s
              directpv.min.io/version=v1beta1
Annotations:  <none>
API Version:  directpv.min.io/v1beta1
Kind:         DirectPVDrive
Metadata:
  Creation Timestamp:  2025-09-20T02:56:01Z
  Finalizers:
    directpv.min.io/data-protection
  Generation:        1
  Resource Version:  3179
  UID:               16b03e2e-ddfd-4de6-9af9-f0e70a2a4f32
Spec:
Status:
  Allocated Capacity:  0
  Free Capacity:       32196100096
  Fsuuid:              94dca655-fc05-4e5f-a7d8-a8b1e953f220
  Make:                Amazon Elastic Block Store
  Status:              Ready
  Topology:
    directpv.min.io/identity:  directpv-min-io
    directpv.min.io/node:      k3s-s
    directpv.min.io/rack:      default
    directpv.min.io/region:    default
    directpv.min.io/zone:      default
  Total Capacity:              32212254720
Events:                        <none>

Name:         afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445
Namespace:
Labels:       directpv.min.io/access-tier=Default
              directpv.min.io/created-by=directpv-driver
              directpv.min.io/drive-name=nvme1n1
              directpv.min.io/node=k3s-s
              directpv.min.io/tier=fast
              directpv.min.io/version=v1beta1
Annotations:  <none>
API Version:  directpv.min.io/v1beta1
Kind:         DirectPVDrive
Metadata:
  Creation Timestamp:  2025-09-20T02:56:01Z
  Finalizers:
    directpv.min.io/data-protection
    directpv.min.io.volume/pvc-825cf353-c880-449f-8482-15770a1ec908
  Generation:        2
  Resource Version:  7435
  UID:               158b3114-7960-4d06-af89-8b9b97c8d40e
Spec:
Status:
  Allocated Capacity:  8388608
  Free Capacity:       32187711488
  Fsuuid:              afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445
  Make:                Amazon Elastic Block Store
  Status:              Ready
  Topology:
    directpv.min.io/identity:  directpv-min-io
    directpv.min.io/node:      k3s-s
    directpv.min.io/rack:      default
    directpv.min.io/region:    default
    directpv.min.io/zone:      default
  Total Capacity:              32212254720
Events:
  Type    Reason       Age   From                 Message
  ----    ------       ----  ----                 -------
  Normal  VolumeAdded  61s   directpv-controller  volume pvc-825cf353-c880-449f-8482-15770a1ec908 with size 8,388,608 is added
# 삭제
kubectl delete pod nginx-pod && kubectl delete pvc nginx-pvc
pod "nginx-pod" deleted
persistentvolumeclaim "nginx-pvc" deleted

# 삭제 확인
k get pv,pvc,po
No resources found

minio 설치 & 버킷 생성 mybucket

helm repo add minio-operator https://operator.min.io
"minio-operator" has been added to your repositories

# https://github.com/minio/operator/blob/master/helm/operator/values.yaml
cat << EOF > minio-operator-values.yaml
operator:
  env:
  - name: MINIO_OPERATOR_RUNTIME
    value: "Rancher"
  replicaCount: 1
EOF

helm install --namespace minio-operator --create-namespace minio-operator minio-operator/operator --values minio-operator-values.yaml
NAME: minio-operator
LAST DEPLOYED: Sat Sep 20 13:55:26 2025
NAMESPACE: minio-operator
STATUS: deployed
REVISION: 1
TEST SUITE: None

# 확인 : 참고로 현재는 오퍼레이터 관리 웹 미제공
k get crd
NAME                                   CREATED AT
addons.k3s.cattle.io                   2025-09-20T01:07:53Z
directpvdrives.directpv.min.io         2025-09-20T02:33:01Z
directpvinitrequests.directpv.min.io   2025-09-20T02:33:01Z
directpvnodes.directpv.min.io          2025-09-20T02:33:01Z
directpvvolumes.directpv.min.io        2025-09-20T02:33:01Z
etcdsnapshotfiles.k3s.cattle.io        2025-09-20T01:07:53Z
helmchartconfigs.helhttp://m.cattle.io 2025-09-20T01:07:53Z
helmcharts.helhttp://m.cattle.io 2025-09-20T01:07:53Z
policybindings.sts.min.io              2025-09-20T04:55:27Z
tenants.minio.min.io                   2025-09-20T04:55:27Z

k get-all -n minio-operator
I0920 13:56:03.476883   18612 warnings.go:110] "Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice"
NAME                                           NAMESPACE       AGE
configmap/kube-root-ca.crt                     minio-operator  36s
endpoints/operator                             minio-operator  36s
endpoints/sts                                  minio-operator  36s
pod/minio-operator-75946dc4db-g4qhv            minio-operator  36s
secret/sh.helhttp://m.release.v1.minio-operator.v1 minio-operator  36s
secret/sts-tls                                 minio-operator  26s
serviceaccount/default                         minio-operator  36s
serviceaccount/minio-operator                  minio-operator  36s
service/operator                               minio-operator  36s
service/sts                                    minio-operator  36s
deployment.apps/minio-operator                 minio-operator  36s
replicaset.apps/minio-operator-75946dc4db      minio-operator  36s
lease.coordination.k8s.io/minio-operator-lock  minio-operator  31s
endpointslice.discovery.k8s.io/operator-kwwh9  minio-operator  36s
endpointslice.discovery.k8s.io/sts-jrqkd       minio-operator  36s

k get pod,svc,ep -n minio-operator
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME                                  READY   STATUS    RESTARTS   AGE
pod/minio-operator-75946dc4db-g4qhv   1/1     Running   0          44s

NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/operator   ClusterIP   10.43.36.168    <none>        4221/TCP   44s
service/sts        ClusterIP   10.43.148.194   <none>        4223/TCP   44s

NAME                 ENDPOINTS         AGE
endpoints/operator   10.42.0.11:4221   44s
endpoints/sts        10.42.0.11:4223   44s

k exec -it -n minio-operator deploy/minio-operator -- env | grep MINIO
MINIO_OPERATOR_RUNTIME=Rancher

# If using Amazon Elastic Block Store (EBS) CSI driver : Please make sure to set xfs for "csi.storage.k8s.io/fstype" parameter under StorageClass.parameters.
k get sc directpv-min-io -o yaml | grep -i fstype
  csi.storage.k8s.io/fstype: xfs

# tenant values : https://github.com/minio/operator/blob/master/helm/tenant/values.yaml
cat << EOF > minio-tenant-1-values.yaml
tenant:
  name: tenant1

  configSecret:
    name: tenant1-env-configuration
    accessKey: minio
    secretKey: minio123

  pools:
    - servers: 1
      name: pool-0
      volumesPerServer: 4
      size: 10Gi
      storageClassName: directpv-min-io
  env:
    - name: MINIO_STORAGE_CLASS_STANDARD
      value: "EC:1"

  metrics:
    enabled: true
    port: 9000
    protocol: http
EOF

helm install --namespace tenant1 --create-namespace --values minio-tenant-1-values.yaml tenant1 minio-operator/tenant \
&& kubectl get tenants -A -w
NAME: tenant1
LAST DEPLOYED: Sat Sep 20 13:56:57 2025
NAMESPACE: tenant1
STATUS: deployed
REVISION: 1
TEST SUITE: None
NAMESPACE   NAME      STATE   HEALTH   AGE
tenant1     tenant1                    0s
tenant1     tenant1                    5s
tenant1     tenant1                    5s
tenant1     tenant1   Waiting for MinIO TLS Certificate            5s
tenant1     tenant1   Provisioning MinIO Cluster IP Service            15s
tenant1     tenant1   Provisioning Console Service                     15s
tenant1     tenant1   Provisioning MinIO Headless Service              15s
tenant1     tenant1   Provisioning MinIO Headless Service              16s
tenant1     tenant1   Provisioning MinIO Statefulset                   16s
tenant1     tenant1   Provisioning MinIO Statefulset                   16s
tenant1     tenant1   Provisioning MinIO Statefulset                   16s
tenant1     tenant1   Waiting for Tenant to be healthy                 17s

kubectl get tenants -n tenant1
NAME      STATE         HEALTH   AGE
tenant1   Initialized   green    4m29s

kubectl get tenants -n tenant1 -owide -o yaml | yq
{
  "apiVersion": "v1",
  "items": [
    {
      "apiVersion": "minio.min.io/v2",
      "kind": "Tenant",
      "metadata": {
        "annotations": {
          "meta.helm.sh/release-name": "tenant1",
          "meta.helm.sh/release-namespace": "tenant1",
          "prometheus.io/path": "/minio/v2/metrics/cluster",
          "prometheus.io/port": "9000",
          "prometheus.io/scheme": "http",
          "prometheus.io/scrape": "true"
        },
        "creationTimestamp": "2025-09-20T04:56:57Z",
        "generation": 1,
        "labels": {
          "app": "minio",
          "app.kubernetes.io/managed-by": "Helm"
        },
        "name": "tenant1",
        "namespace": "tenant1",
        "resourceVersion": "8886",
        "uid": "eb27e7f5-830f-4b19-9c76-1ff4a57c9891"
      },
      "spec": {
        "configuration": {
          "name": "tenant1-env-configuration"
        },
        "env": [
          {
            "name": "MINIO_STORAGE_CLASS_STANDARD",
            "value": "EC:1"
          }
        ],
        "features": {
          "bucketDNS": false,
          "enableSFTP": false
        },
        "image": "quay.io/minio/minio:RELEASE.2025-04-08T15-41-24Z",
        "imagePullPolicy": "IfNotPresent",
        "mountPath": "/export",
        "podManagementPolicy": "Parallel",
        "pools": [
          {
            "name": "pool-0",
            "servers": 1,
            "volumeClaimTemplate": {
              "metadata": {
                "name": "data"
              },
              "spec": {
                "accessModes": [
                  "ReadWriteOnce"
                ],
                "resources": {
                  "requests": {
                    "storage": "10Gi"
                  }
                },
                "storageClassName": "directpv-min-io"
              }
            },
            "volumesPerServer": 4
          }
        ],
        "poolsMetadata": {
          "annotations": {},
          "labels": {}
        },
        "prometheusOperator": false,
        "requestAutoCert": true,
        "subPath": "/data"
      },
      "status": {
        "availableReplicas": 1,
        "certificates": {
          "autoCertEnabled": true,
          "customCertificates": {}
        },
        "currentState": "Initialized",
        "drivesOnline": 4,
        "healthStatus": "green",
        "pools": [
          {
            "legacySecurityContext": false,
            "ssName": "tenant1-pool-0",
            "state": "PoolInitialized"
          }
        ],
        "revision": 0,
        "syncVersion": "v6.0.0",
        "usage": {
          "capacity": 32212094976,
          "rawCapacity": 42949459968,
          "rawUsage": 212992,
          "usage": 159744
        },
        "writeQuorum": 3
      }
    }
  ],
  "kind": "List",
  "metadata": {
    "resourceVersion": ""
  }
}

kc describe tenants -n tenant1
Name:         tenant1
Namespace:    tenant1
Labels:       app=minio
              app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: tenant1
              meta.helm.sh/release-namespace: tenant1
              prometheus.io/path: /minio/v2/metrics/cluster
              prometheus.io/port: 9000
              prometheus.io/scheme: http
              prometheus.io/scrape: true
API Version:  minio.min.io/v2
Kind:         Tenant
Metadata:
  Creation Timestamp:  2025-09-20T04:56:57Z
  Generation:          1
  Resource Version:    8886
  UID:                 eb27e7f5-830f-4b19-9c76-1ff4a57c9891
Spec:
  Configuration:
    Name:  tenant1-env-configuration
  Env:
    Name:   MINIO_STORAGE_CLASS_STANDARD
    Value:  EC:1
  Features:
    Bucket DNS:           false
    Enable SFTP:          false
  Image:                  quay.io/minio/minio:RELEASE.2025-04-08T15-41-24Z
  Image Pull Policy:      IfNotPresent
  Mount Path:             /export
  Pod Management Policy:  Parallel
  Pools:
    Name:     pool-0
    Servers:  1
    Volume Claim Template:
      Metadata:
        Name:  data
      Spec:
        Access Modes:
          ReadWriteOnce
        Resources:
          Requests:
            Storage:         10Gi
        Storage Class Name:  directpv-min-io
    Volumes Per Server:      4
  Pools Metadata:
    Annotations:
    Labels:
  Prometheus Operator:  false
  Request Auto Cert:    true
  Sub Path:             /data
Status:
  Available Replicas:  1
  Certificates:
    Auto Cert Enabled:  true
    Custom Certificates:
  Current State:  Initialized
  Drives Online:  4
  Health Status:  green
  Pools:
    Legacy Security Context:  false
    Ss Name:                  tenant1-pool-0
    State:                    PoolInitialized
  Revision:                   0
  Sync Version:               v6.0.0
  Usage:
    Capacity:      32212094976
    Raw Capacity:  42949459968
    Raw Usage:     212992
    Usage:         159744
  Write Quorum:    3
Events:
  Type     Reason                 Age                    From            Message
  ----     ------                 ----                   ----            -------
  Normal   CSRCreated             5m38s                  minio-operator  MinIO CSR Created
  Normal   SvcCreated             5m28s                  minio-operator  MinIO Service Created
  Normal   SvcCreated             5m28s                  minio-operator  Console Service Created
  Normal   SvcCreated             5m28s                  minio-operator  Headless Service created
  Normal   PoolCreated            5m27s                  minio-operator  Tenant pool pool-0 created
  Normal   Updated                5m23s                  minio-operator  Headless Service Updated
  Warning  WaitingMinIOIsHealthy  5m13s (x4 over 5m26s)  minio-operator  Waiting for MinIO to be ready

lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
loop0         7:0    0 27.6M  1 loop /snap/amazon-ssm-agent/11797
loop1         7:1    0 73.9M  1 loop /snap/core22/2111
loop2         7:2    0 50.8M  1 loop /snap/snapd/25202
loop3         7:3    0   16M  0 loop
nvme2n1     259:0    0   30G  0 disk /var/lib/kubelet/pods/96e4e9ba-def5-4051-a89a-8d3bf6efd8b1/volumes/kubernetes.io~csi/pvc-6aa17a8c-3e37-4dbd-a837-32683b6d765f/mount
                                     /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/1303f8614e90f62edef9c5293c7415b999bccfac7d71c9b210b24281e6837828/globalmount
                                     /var/lib/directpv/mnt/94dca655-fc05-4e5f-a7d8-a8b1e953f220
nvme1n1     259:1    0   30G  0 disk /var/lib/kubelet/pods/96e4e9ba-def5-4051-a89a-8d3bf6efd8b1/volumes/kubernetes.io~csi/pvc-a6e3bbf9-ab29-41d8-ac3b-c1d8920e0d98/mount
                                     /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/b1fcff20409de9e419cac67cfada28d24fd259e30f5ea0f294205e4619cff68b/globalmount
                                     /var/lib/directpv/mnt/afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445
nvme0n1     259:2    0   30G  0 disk
├─nvme0n1p1 259:5    0   29G  0 part /
├─nvme0n1p14
│           259:6    0    4M  0 part
├─nvme0n1p15
│           259:7    0  106M  0 part /boot/efi
└─nvme0n1p16
            259:8    0  913M  0 part /boot
nvme3n1     259:3    0   30G  0 disk /var/lib/kubelet/pods/96e4e9ba-def5-4051-a89a-8d3bf6efd8b1/volumes/kubernetes.io~csi/pvc-4a372c6f-6fe1-4ec8-8616-eba5e6b03b57/mount
                                     /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/2f2e1ad824df2e9a63b1c08f2944717c807ba77d59ae6b36686aea080bf56e9d/globalmount
                                     /var/lib/directpv/mnt/94ca8715-bbb0-47ad-93f9-43af207e0240
nvme4n1     259:4    0   30G  0 disk /var/lib/kubelet/pods/96e4e9ba-def5-4051-a89a-8d3bf6efd8b1/volumes/kubernetes.io~csi/pvc-214d8389-2b4c-48b6-b2d4-67fc413e78fd/mount
                                     /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/ff3df50badeadcbe76e413d19beabe12727118b85265d067058892fd39269817/globalmount
                                     /var/lib/directpv/mnt/668267e9-fdd2-46c9-b854-1ac1de70132f

k directpv info
┌─────────┬──────────┬───────────┬─────────┬────────┐
│ NODE    │ CAPACITY │ ALLOCATED │ VOLUMES │ DRIVES │
├─────────┼──────────┼───────────┼─────────┼────────┤
│ • k3s-s │ 120 GiB  │ 40 GiB    │ 4       │ 4      │
└─────────┴──────────┴───────────┴─────────┴────────┘

40 GiB/120 GiB used, 4 volumes, 4 drives

k directpv list drives
┌───────┬─────────┬────────────────────────────┬────────┬────────┬─────────┬────────┐
│ NODE  │ NAME    │ MAKE                       │ SIZE   │ FREE   │ VOLUMES │ STATUS │
├───────┼─────────┼────────────────────────────┼────────┼────────┼─────────┼────────┤
│ k3s-s │ nvme1n1 │ Amazon Elastic Block Store │ 30 GiB │ 20 GiB │ 1       │ Ready  │
│ k3s-s │ nvme2n1 │ Amazon Elastic Block Store │ 30 GiB │ 20 GiB │ 1       │ Ready  │
│ k3s-s │ nvme3n1 │ Amazon Elastic Block Store │ 30 GiB │ 20 GiB │ 1       │ Ready  │
│ k3s-s │ nvme4n1 │ Amazon Elastic Block Store │ 30 GiB │ 20 GiB │ 1       │ Ready  │
└───────┴─────────┴────────────────────────────┴────────┴────────┴─────────┴────────┘

k directpv list volumes
┌──────────────────────────────────────────┬──────────┬───────┬─────────┬──────────────────┬──────────────┬─────────┐
│ VOLUME                                   │ CAPACITY │ NODE  │ DRIVE   │ PODNAME          │ PODNAMESPACE │ STATUS  │
├──────────────────────────────────────────┼──────────┼───────┼─────────┼──────────────────┼──────────────┼─────────┤
│ pvc-a6e3bbf9-ab29-41d8-ac3b-c1d8920e0d98 │ 10 GiB   │ k3s-s │ nvme1n1 │ tenant1-pool-0-0 │ tenant1      │ Bounded │
│ pvc-6aa17a8c-3e37-4dbd-a837-32683b6d765f │ 10 GiB   │ k3s-s │ nvme2n1 │ tenant1-pool-0-0 │ tenant1      │ Bounded │
│ pvc-4a372c6f-6fe1-4ec8-8616-eba5e6b03b57 │ 10 GiB   │ k3s-s │ nvme3n1 │ tenant1-pool-0-0 │ tenant1      │ Bounded │
│ pvc-214d8389-2b4c-48b6-b2d4-67fc413e78fd │ 10 GiB   │ k3s-s │ nvme4n1 │ tenant1-pool-0-0 │ tenant1      │ Bounded │
└──────────────────────────────────────────┴──────────┴───────┴─────────┴──────────────────┴──────────────┴─────────┘

k get directpvvolumes.directpv.min.io
NAME                                       AGE
pvc-214d8389-2b4c-48b6-b2d4-67fc413e78fd   6m23s
pvc-4a372c6f-6fe1-4ec8-8616-eba5e6b03b57   6m23s
pvc-6aa17a8c-3e37-4dbd-a837-32683b6d765f   6m23s
pvc-a6e3bbf9-ab29-41d8-ac3b-c1d8920e0d98   6m23s

k get directpvvolumes.directpv.min.io -o yaml | yq
{
  "apiVersion": "v1",
  "items": [
    {
      "apiVersion": "directpv.min.io/v1beta1",
      "kind": "DirectPVVolume",
      "metadata": {
        "creationTimestamp": "2025-09-20T04:57:13Z",
        "finalizers": [
          "directpv.min.io/pv-protection",
          "directpv.min.io/purge-protection"
        ],
        "generation": 3,
        "labels": {
          "directpv.min.io/created-by": "directpv-controller",
          "directpv.min.io/drive": "668267e9-fdd2-46c9-b854-1ac1de70132f",
          "directpv.min.io/drive-name": "nvme4n1",
          "directpv.min.io/node": "k3s-s",
          "directpv.min.io/pod.name": "tenant1-pool-0-0",
          "directpv.min.io/pod.namespace": "tenant1",
          "directpv.min.io/version": "v1beta1"
        },
        "name": "pvc-214d8389-2b4c-48b6-b2d4-67fc413e78fd",
        "resourceVersion": "8674",
        "uid": "10192b4c-44b1-4355-af69-1abb230d52b1"
      },
      "status": {
        "availableCapacity": 10737418240,
        "dataPath": "/var/lib/directpv/mnt/668267e9-fdd2-46c9-b854-1ac1de70132f/.FSUUID.668267e9-fdd2-46c9-b854-1ac1de70132f/pvc-214d8389-2b4c-48b6-b2d4-67fc413e78fd",
        "fsuuid": "668267e9-fdd2-46c9-b854-1ac1de70132f",
        "stagingTargetPath": "/var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/ff3df50badeadcbe76e413d19beabe12727118b85265d067058892fd39269817/globalmount",
        "status": "Ready",
        "targetPath": "/var/lib/kubelet/pods/96e4e9ba-def5-4051-a89a-8d3bf6efd8b1/volumes/kubernetes.io~csi/pvc-214d8389-2b4c-48b6-b2d4-67fc413e78fd/mount",
        "totalCapacity": 10737418240,
        "usedCapacity": 0
      }
    },
    {
      "apiVersion": "directpv.min.io/v1beta1",
      "kind": "DirectPVVolume",
      "metadata": {
        "creationTimestamp": "2025-09-20T04:57:13Z",
        "finalizers": [
          "directpv.min.io/pv-protection",
          "directpv.min.io/purge-protection"
        ],
        "generation": 4,
        "labels": {
          "directpv.min.io/created-by": "directpv-controller",
          "directpv.min.io/drive": "94ca8715-bbb0-47ad-93f9-43af207e0240",
          "directpv.min.io/drive-name": "nvme3n1",
          "directpv.min.io/node": "k3s-s",
          "directpv.min.io/pod.name": "tenant1-pool-0-0",
          "directpv.min.io/pod.namespace": "tenant1",
          "directpv.min.io/version": "v1beta1"
        },
        "name": "pvc-4a372c6f-6fe1-4ec8-8616-eba5e6b03b57",
        "resourceVersion": "8671",
        "uid": "bfc7feef-d793-466a-bf01-73bb86bbacf3"
      },
      "status": {
        "availableCapacity": 10737418240,
        "dataPath": "/var/lib/directpv/mnt/94ca8715-bbb0-47ad-93f9-43af207e0240/.FSUUID.94ca8715-bbb0-47ad-93f9-43af207e0240/pvc-4a372c6f-6fe1-4ec8-8616-eba5e6b03b57",
        "fsuuid": "94ca8715-bbb0-47ad-93f9-43af207e0240",
        "stagingTargetPath": "/var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/2f2e1ad824df2e9a63b1c08f2944717c807ba77d59ae6b36686aea080bf56e9d/globalmount",
        "status": "Ready",
        "targetPath": "/var/lib/kubelet/pods/96e4e9ba-def5-4051-a89a-8d3bf6efd8b1/volumes/kubernetes.io~csi/pvc-4a372c6f-6fe1-4ec8-8616-eba5e6b03b57/mount",
        "totalCapacity": 10737418240,
        "usedCapacity": 0
      }
    },
    {
      "apiVersion": "directpv.min.io/v1beta1",
      "kind": "DirectPVVolume",
      "metadata": {
        "creationTimestamp": "2025-09-20T04:57:13Z",
        "finalizers": [
          "directpv.min.io/pv-protection",
          "directpv.min.io/purge-protection"
        ],
        "generation": 3,
        "labels": {
          "directpv.min.io/created-by": "directpv-controller",
          "directpv.min.io/drive": "94dca655-fc05-4e5f-a7d8-a8b1e953f220",
          "directpv.min.io/drive-name": "nvme2n1",
          "directpv.min.io/node": "k3s-s",
          "directpv.min.io/pod.name": "tenant1-pool-0-0",
          "directpv.min.io/pod.namespace": "tenant1",
          "directpv.min.io/version": "v1beta1"
        },
        "name": "pvc-6aa17a8c-3e37-4dbd-a837-32683b6d765f",
        "resourceVersion": "8673",
        "uid": "7a87c555-55d2-4397-9e20-095fb47819b1"
      },
      "status": {
        "availableCapacity": 10737418240,
        "dataPath": "/var/lib/directpv/mnt/94dca655-fc05-4e5f-a7d8-a8b1e953f220/.FSUUID.94dca655-fc05-4e5f-a7d8-a8b1e953f220/pvc-6aa17a8c-3e37-4dbd-a837-32683b6d765f",
        "fsuuid": "94dca655-fc05-4e5f-a7d8-a8b1e953f220",
        "stagingTargetPath": "/var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/1303f8614e90f62edef9c5293c7415b999bccfac7d71c9b210b24281e6837828/globalmount",
        "status": "Ready",
        "targetPath": "/var/lib/kubelet/pods/96e4e9ba-def5-4051-a89a-8d3bf6efd8b1/volumes/kubernetes.io~csi/pvc-6aa17a8c-3e37-4dbd-a837-32683b6d765f/mount",
        "totalCapacity": 10737418240,
        "usedCapacity": 0
      }
    },
    {
      "apiVersion": "directpv.min.io/v1beta1",
      "kind": "DirectPVVolume",
      "metadata": {
        "creationTimestamp": "2025-09-20T04:57:13Z",
        "finalizers": [
          "directpv.min.io/pv-protection",
          "directpv.min.io/purge-protection"
        ],
        "generation": 3,
        "labels": {
          "directpv.min.io/created-by": "directpv-controller",
          "directpv.min.io/drive": "afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445",
          "directpv.min.io/drive-name": "nvme1n1",
          "directpv.min.io/node": "k3s-s",
          "directpv.min.io/pod.name": "tenant1-pool-0-0",
          "directpv.min.io/pod.namespace": "tenant1",
          "directpv.min.io/version": "v1beta1"
        },
        "name": "pvc-a6e3bbf9-ab29-41d8-ac3b-c1d8920e0d98",
        "resourceVersion": "8672",
        "uid": "57f4e39b-4334-42ce-9d3e-66411e53a1bb"
      },
      "status": {
        "availableCapacity": 10737418240,
        "dataPath": "/var/lib/directpv/mnt/afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445/.FSUUID.afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445/pvc-a6e3bbf9-ab29-41d8-ac3b-c1d8920e0d98",
        "fsuuid": "afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445",
        "stagingTargetPath": "/var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/b1fcff20409de9e419cac67cfada28d24fd259e30f5ea0f294205e4619cff68b/globalmount",
        "status": "Ready",
        "targetPath": "/var/lib/kubelet/pods/96e4e9ba-def5-4051-a89a-8d3bf6efd8b1/volumes/kubernetes.io~csi/pvc-a6e3bbf9-ab29-41d8-ac3b-c1d8920e0d98/mount",
        "totalCapacity": 10737418240,
        "usedCapacity": 0
      }
    }
  ],
  "kind": "List",
  "metadata": {
    "resourceVersion": ""
  }
}

kc describe directpvvolumes
Name:         pvc-214d8389-2b4c-48b6-b2d4-67fc413e78fd
Namespace:
Labels:       directpv.min.io/created-by=directpv-controller
              directpv.min.io/drive=668267e9-fdd2-46c9-b854-1ac1de70132f
              directpv.min.io/drive-name=nvme4n1
              directpv.min.io/node=k3s-s
              directpv.min.io/pod.name=tenant1-pool-0-0
              directpv.min.io/pod.namespace=tenant1
              directpv.min.io/version=v1beta1
Annotations:  <none>
API Version:  directpv.min.io/v1beta1
Kind:         DirectPVVolume
Metadata:
  Creation Timestamp:  2025-09-20T04:57:13Z
  Finalizers:
    directpv.min.io/pv-protection
    directpv.min.io/purge-protection
  Generation:        3
  Resource Version:  8674
  UID:               10192b4c-44b1-4355-af69-1abb230d52b1
Status:
  Available Capacity:   10737418240
  Data Path:            /var/lib/directpv/mnt/668267e9-fdd2-46c9-b854-1ac1de70132f/.FSUUID.668267e9-fdd2-46c9-b854-1ac1de70132f/pvc-214d8389-2b4c-48b6-b2d4-67fc413e78fd
  Fsuuid:               668267e9-fdd2-46c9-b854-1ac1de70132f
  Staging Target Path:  /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/ff3df50badeadcbe76e413d19beabe12727118b85265d067058892fd39269817/globalmount
  Status:               Ready
  Target Path:          /var/lib/kubelet/pods/96e4e9ba-def5-4051-a89a-8d3bf6efd8b1/volumes/kubernetes.io~csi/pvc-214d8389-2b4c-48b6-b2d4-67fc413e78fd/mount
  Total Capacity:       10737418240
  Used Capacity:        0
Events:                 <none>

Name:         pvc-4a372c6f-6fe1-4ec8-8616-eba5e6b03b57
Namespace:
Labels:       directpv.min.io/created-by=directpv-controller
              directpv.min.io/drive=94ca8715-bbb0-47ad-93f9-43af207e0240
              directpv.min.io/drive-name=nvme3n1
              directpv.min.io/node=k3s-s
              directpv.min.io/pod.name=tenant1-pool-0-0
              directpv.min.io/pod.namespace=tenant1
              directpv.min.io/version=v1beta1
Annotations:  <none>
API Version:  directpv.min.io/v1beta1
Kind:         DirectPVVolume
Metadata:
  Creation Timestamp:  2025-09-20T04:57:13Z
  Finalizers:
    directpv.min.io/pv-protection
    directpv.min.io/purge-protection
  Generation:        4
  Resource Version:  8671
  UID:               bfc7feef-d793-466a-bf01-73bb86bbacf3
Status:
  Available Capacity:   10737418240
  Data Path:            /var/lib/directpv/mnt/94ca8715-bbb0-47ad-93f9-43af207e0240/.FSUUID.94ca8715-bbb0-47ad-93f9-43af207e0240/pvc-4a372c6f-6fe1-4ec8-8616-eba5e6b03b57
  Fsuuid:               94ca8715-bbb0-47ad-93f9-43af207e0240
  Staging Target Path:  /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/2f2e1ad824df2e9a63b1c08f2944717c807ba77d59ae6b36686aea080bf56e9d/globalmount
  Status:               Ready
  Target Path:          /var/lib/kubelet/pods/96e4e9ba-def5-4051-a89a-8d3bf6efd8b1/volumes/kubernetes.io~csi/pvc-4a372c6f-6fe1-4ec8-8616-eba5e6b03b57/mount
  Total Capacity:       10737418240
  Used Capacity:        0
Events:
  Type    Reason             Age    From                 Message
  ----    ------             ----   ----                 -------
  Normal  VolumeProvisioned  6m33s  directpv-controller  volume is reused

Name:         pvc-6aa17a8c-3e37-4dbd-a837-32683b6d765f
Namespace:
Labels:       directpv.min.io/created-by=directpv-controller
              directpv.min.io/drive=94dca655-fc05-4e5f-a7d8-a8b1e953f220
              directpv.min.io/drive-name=nvme2n1
              directpv.min.io/node=k3s-s
              directpv.min.io/pod.name=tenant1-pool-0-0
              directpv.min.io/pod.namespace=tenant1
              directpv.min.io/version=v1beta1
Annotations:  <none>
API Version:  directpv.min.io/v1beta1
Kind:         DirectPVVolume
Metadata:
  Creation Timestamp:  2025-09-20T04:57:13Z
  Finalizers:
    directpv.min.io/pv-protection
    directpv.min.io/purge-protection
  Generation:        3
  Resource Version:  8673
  UID:               7a87c555-55d2-4397-9e20-095fb47819b1
Status:
  Available Capacity:   10737418240
  Data Path:            /var/lib/directpv/mnt/94dca655-fc05-4e5f-a7d8-a8b1e953f220/.FSUUID.94dca655-fc05-4e5f-a7d8-a8b1e953f220/pvc-6aa17a8c-3e37-4dbd-a837-32683b6d765f
  Fsuuid:               94dca655-fc05-4e5f-a7d8-a8b1e953f220
  Staging Target Path:  /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/1303f8614e90f62edef9c5293c7415b999bccfac7d71c9b210b24281e6837828/globalmount
  Status:               Ready
  Target Path:          /var/lib/kubelet/pods/96e4e9ba-def5-4051-a89a-8d3bf6efd8b1/volumes/kubernetes.io~csi/pvc-6aa17a8c-3e37-4dbd-a837-32683b6d765f/mount
  Total Capacity:       10737418240
  Used Capacity:        0
Events:                 <none>

Name:         pvc-a6e3bbf9-ab29-41d8-ac3b-c1d8920e0d98
Namespace:
Labels:       directpv.min.io/created-by=directpv-controller
              directpv.min.io/drive=afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445
              directpv.min.io/drive-name=nvme1n1
              directpv.min.io/node=k3s-s
              directpv.min.io/pod.name=tenant1-pool-0-0
              directpv.min.io/pod.namespace=tenant1
              directpv.min.io/version=v1beta1
Annotations:  <none>
API Version:  directpv.min.io/v1beta1
Kind:         DirectPVVolume
Metadata:
  Creation Timestamp:  2025-09-20T04:57:13Z
  Finalizers:
    directpv.min.io/pv-protection
    directpv.min.io/purge-protection
  Generation:        3
  Resource Version:  8672
  UID:               57f4e39b-4334-42ce-9d3e-66411e53a1bb
Status:
  Available Capacity:   10737418240
  Data Path:            /var/lib/directpv/mnt/afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445/.FSUUID.afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445/pvc-a6e3bbf9-ab29-41d8-ac3b-c1d8920e0d98
  Fsuuid:               afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445
  Staging Target Path:  /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/b1fcff20409de9e419cac67cfada28d24fd259e30f5ea0f294205e4619cff68b/globalmount
  Status:               Ready
  Target Path:          /var/lib/kubelet/pods/96e4e9ba-def5-4051-a89a-8d3bf6efd8b1/volumes/kubernetes.io~csi/pvc-a6e3bbf9-ab29-41d8-ac3b-c1d8920e0d98/mount
  Total Capacity:       10737418240
  Used Capacity:        0
Events:                 <none>

tree -ah /var/lib/kubelet/plugins
[4.0K]  /var/lib/kubelet/plugins
├── [4.0K]  controller-controller
│   └── [   0]  csi.sock
├── [4.0K]  directpv-min-io
│   └── [   0]  csi.sock
└── [4.0K]  kubernetes.io
    └── [4.0K]  csi
        └── [4.0K]  directpv-min-io
            ├── [4.0K]  1303f8614e90f62edef9c5293c7415b999bccfac7d71c9b210b24281e6837828
            │   ├── [  18]  globalmount
            │   │   └── [  24]  data
            │   │       └── [  98]  .minio.sys
            │   │           ├── [  77]  buckets
            │   │           │   ├── [  21]  .background-heal.json
            │   │           │   │   └── [ 466]  xl.meta
            │   │           │   ├── [  21]  .bloomcycle.bin
            │   │           │   │   └── [ 515]  xl.meta
            │   │           │   └── [  21]  .usage.json
            │   │           │       └── [ 534]  xl.meta
            │   │           ├── [  36]  config
            │   │           │   ├── [  21]  config.json
            │   │           │   │   └── [5.1K]  xl.meta
            │   │           │   └── [  25]  iam
            │   │           │       └── [  21]  format.json
            │   │           │           └── [ 431]  xl.meta
            │   │           ├── [ 342]  format.json
            │   │           ├── [   6]  multipart
            │   │           ├── [  21]  pool.bin
            │   │           │   └── [ 464]  xl.meta
            │   │           └── [  64]  tmp
            │   │               ├── [ 182]  .trash
            │   │               │   ├── [  25]  0d56f005-fcad-4df2-8c8d-94ef6aeb7bb5
            │   │               │   │   └── [ 507]  xl.meta.bkp
            │   │               │   ├── [  25]  b1d56ad7-cf29-49d0-b5d9-1a8bacab4457
            │   │               │   │   └── [ 494]  xl.meta.bkp
            │   │               │   ├── [  25]  c92d62ca-87d1-4b6c-912e-ac3cb35adce7
            │   │               │   │   └── [ 534]  xl.meta.bkp
            │   │               │   └── [  25]  d2b50b25-0660-4fae-8191-92d21fce691e
            │   │               │       └── [ 534]  xl.meta.bkp
            │   │               └── [2.0K]  1fc71795-b86c-4d31-a636-64c9d2353ec7
            │   └── [  91]  vol_data.json
            ├── [4.0K]  2f2e1ad824df2e9a63b1c08f2944717c807ba77d59ae6b36686aea080bf56e9d
            │   ├── [  18]  globalmount
            │   │   └── [  24]  data
            │   │       └── [  98]  .minio.sys
            │   │           ├── [  77]  buckets
            │   │           │   ├── [  21]  .background-heal.json
            │   │           │   │   └── [ 466]  xl.meta
            │   │           │   ├── [  21]  .bloomcycle.bin
            │   │           │   │   └── [ 515]  xl.meta
            │   │           │   └── [  21]  .usage.json
            │   │           │       └── [ 534]  xl.meta
            │   │           ├── [  36]  config
            │   │           │   ├── [  21]  config.json
            │   │           │   │   └── [5.1K]  xl.meta
            │   │           │   └── [  25]  iam
            │   │           │       └── [  21]  format.json
            │   │           │           └── [ 431]  xl.meta
            │   │           ├── [ 342]  format.json
            │   │           ├── [   6]  multipart
            │   │           ├── [  21]  pool.bin
            │   │           │   └── [ 464]  xl.meta
            │   │           └── [  64]  tmp
            │   │               ├── [ 182]  .trash
            │   │               │   ├── [  25]  178d3d1f-c9a0-4577-9277-ea9205157477
            │   │               │   │   └── [ 534]  xl.meta.bkp
            │   │               │   ├── [  25]  8bd8f221-14f7-4356-b927-2bd65120dddd
            │   │               │   │   └── [ 507]  xl.meta.bkp
            │   │               │   ├── [  25]  c788b2f7-0af0-481e-8f92-6fbbdafd5897
            │   │               │   │   └── [ 494]  xl.meta.bkp
            │   │               │   └── [  25]  ce051ba8-d500-4d3f-8dd3-6101c78eaac0
            │   │               │       └── [ 534]  xl.meta.bkp
            │   │               └── [2.0K]  49594e7d-93b3-4d93-9a9e-71fe2fe4793b
            │   └── [  91]  vol_data.json
            ├── [4.0K]  b1fcff20409de9e419cac67cfada28d24fd259e30f5ea0f294205e4619cff68b
            │   ├── [  18]  globalmount
            │   │   └── [  24]  data
            │   │       └── [  98]  .minio.sys
            │   │           ├── [  77]  buckets
            │   │           │   ├── [  21]  .background-heal.json
            │   │           │   │   └── [ 466]  xl.meta
            │   │           │   ├── [  21]  .bloomcycle.bin
            │   │           │   │   └── [ 515]  xl.meta
            │   │           │   └── [  21]  .usage.json
            │   │           │       └── [ 534]  xl.meta
            │   │           ├── [  36]  config
            │   │           │   ├── [  21]  config.json
            │   │           │   │   └── [5.1K]  xl.meta
            │   │           │   └── [  25]  iam
            │   │           │       └── [  21]  format.json
            │   │           │           └── [ 431]  xl.meta
            │   │           ├── [ 342]  format.json
            │   │           ├── [   6]  multipart
            │   │           ├── [  21]  pool.bin
            │   │           │   └── [ 464]  xl.meta
            │   │           └── [  64]  tmp
            │   │               ├── [ 182]  .trash
            │   │               │   ├── [  25]  040bd878-2bcf-4b24-a313-c571e04b9836
            │   │               │   │   └── [ 534]  xl.meta.bkp
            │   │               │   ├── [  25]  069ca199-851c-402b-9158-2206a3f53012
            │   │               │   │   └── [ 494]  xl.meta.bkp
            │   │               │   ├── [  25]  155bf173-abf6-49f4-81cd-d8eda8f97b1b
            │   │               │   │   └── [ 534]  xl.meta.bkp
            │   │               │   └── [  25]  1d1e29d1-a4aa-45df-bfcb-8fd00e75d7ad
            │   │               │       └── [ 507]  xl.meta.bkp
            │   │               └── [2.0K]  4eca5a1c-f617-41dc-a50e-47b18b54bf43
            │   └── [  91]  vol_data.json
            └── [4.0K]  ff3df50badeadcbe76e413d19beabe12727118b85265d067058892fd39269817
                ├── [  18]  globalmount
                │   └── [  24]  data
                │       └── [  98]  .minio.sys
                │           ├── [  77]  buckets
                │           │   ├── [  21]  .background-heal.json
                │           │   │   └── [ 466]  xl.meta
                │           │   ├── [  21]  .bloomcycle.bin
                │           │   │   └── [ 515]  xl.meta
                │           │   └── [  21]  .usage.json
                │           │       └── [ 534]  xl.meta
                │           ├── [  36]  config
                │           │   ├── [  21]  config.json
                │           │   │   └── [5.1K]  xl.meta
                │           │   └── [  25]  iam
                │           │       └── [  21]  format.json
                │           │           └── [ 431]  xl.meta
                │           ├── [ 342]  format.json
                │           ├── [   6]  multipart
                │           ├── [  21]  pool.bin
                │           │   └── [ 464]  xl.meta
                │           └── [  64]  tmp
                │               ├── [ 182]  .trash
                │               │   ├── [  25]  58121f85-78dd-4ad0-b5b5-c242e32e6a6c
                │               │   │   └── [ 534]  xl.meta.bkp
                │               │   ├── [  25]  7df339a2-977d-42a4-a335-4195b29bb5db
                │               │   │   └── [ 534]  xl.meta.bkp
                │               │   ├── [  25]  fabbbcac-1149-4894-be28-5871f6a69d44
                │               │   │   └── [ 494]  xl.meta.bkp
                │               │   └── [  25]  fb9950b3-2aaf-4ed2-8419-632cd4ef1c99
                │               │       └── [ 507]  xl.meta.bkp
                │               └── [2.0K]  0f3ba3ea-5262-4c76-804c-46cc321fedcb
                └── [  91]  vol_data.json

86 directories, 54 files

tree -ah /var/lib/directpv/mnt
[4.0K]  /var/lib/directpv/mnt
├── [ 123]  668267e9-fdd2-46c9-b854-1ac1de70132f
│   ├── [   1]  .FSUUID.668267e9-fdd2-46c9-b854-1ac1de70132f -> .
│   ├── [  23]  .directpv
│   │   └── [  82]  meta.info
│   └── [  18]  pvc-214d8389-2b4c-48b6-b2d4-67fc413e78fd
│       └── [  24]  data
│           └── [  98]  .minio.sys
│               ├── [  77]  buckets
│               │   ├── [  21]  .background-heal.json
│               │   │   └── [ 466]  xl.meta
│               │   ├── [  21]  .bloomcycle.bin
│               │   │   └── [ 515]  xl.meta
│               │   └── [  21]  .usage.json
│               │       └── [ 534]  xl.meta
│               ├── [  36]  config
│               │   ├── [  21]  config.json
│               │   │   └── [5.1K]  xl.meta
│               │   └── [  25]  iam
│               │       └── [  21]  format.json
│               │           └── [ 431]  xl.meta
│               ├── [ 342]  format.json
│               ├── [   6]  multipart
│               ├── [  21]  pool.bin
│               │   └── [ 464]  xl.meta
│               └── [  64]  tmp
│                   ├── [ 182]  .trash
│                   │   ├── [  25]  58121f85-78dd-4ad0-b5b5-c242e32e6a6c
│                   │   │   └── [ 534]  xl.meta.bkp
│                   │   ├── [  25]  7df339a2-977d-42a4-a335-4195b29bb5db
│                   │   │   └── [ 534]  xl.meta.bkp
│                   │   ├── [  25]  fabbbcac-1149-4894-be28-5871f6a69d44
│                   │   │   └── [ 494]  xl.meta.bkp
│                   │   └── [  25]  fb9950b3-2aaf-4ed2-8419-632cd4ef1c99
│                   │       └── [ 507]  xl.meta.bkp
│                   └── [2.0K]  0f3ba3ea-5262-4c76-804c-46cc321fedcb
├── [ 123]  94ca8715-bbb0-47ad-93f9-43af207e0240
│   ├── [   1]  .FSUUID.94ca8715-bbb0-47ad-93f9-43af207e0240 -> .
│   ├── [  23]  .directpv
│   │   └── [  82]  meta.info
│   └── [  18]  pvc-4a372c6f-6fe1-4ec8-8616-eba5e6b03b57
│       └── [  24]  data
│           └── [  98]  .minio.sys
│               ├── [  77]  buckets
│               │   ├── [  21]  .background-heal.json
│               │   │   └── [ 466]  xl.meta
│               │   ├── [  21]  .bloomcycle.bin
│               │   │   └── [ 515]  xl.meta
│               │   └── [  21]  .usage.json
│               │       └── [ 534]  xl.meta
│               ├── [  36]  config
│               │   ├── [  21]  config.json
│               │   │   └── [5.1K]  xl.meta
│               │   └── [  25]  iam
│               │       └── [  21]  format.json
│               │           └── [ 431]  xl.meta
│               ├── [ 342]  format.json
│               ├── [   6]  multipart
│               ├── [  21]  pool.bin
│               │   └── [ 464]  xl.meta
│               └── [  64]  tmp
│                   ├── [ 182]  .trash
│                   │   ├── [  25]  178d3d1f-c9a0-4577-9277-ea9205157477
│                   │   │   └── [ 534]  xl.meta.bkp
│                   │   ├── [  25]  8bd8f221-14f7-4356-b927-2bd65120dddd
│                   │   │   └── [ 507]  xl.meta.bkp
│                   │   ├── [  25]  c788b2f7-0af0-481e-8f92-6fbbdafd5897
│                   │   │   └── [ 494]  xl.meta.bkp
│                   │   └── [  25]  ce051ba8-d500-4d3f-8dd3-6101c78eaac0
│                   │       └── [ 534]  xl.meta.bkp
│                   └── [2.0K]  49594e7d-93b3-4d93-9a9e-71fe2fe4793b
├── [ 123]  94dca655-fc05-4e5f-a7d8-a8b1e953f220
│   ├── [   1]  .FSUUID.94dca655-fc05-4e5f-a7d8-a8b1e953f220 -> .
│   ├── [  23]  .directpv
│   │   └── [  82]  meta.info
│   └── [  18]  pvc-6aa17a8c-3e37-4dbd-a837-32683b6d765f
│       └── [  24]  data
│           └── [  98]  .minio.sys
│               ├── [  77]  buckets
│               │   ├── [  21]  .background-heal.json
│               │   │   └── [ 466]  xl.meta
│               │   ├── [  21]  .bloomcycle.bin
│               │   │   └── [ 515]  xl.meta
│               │   └── [  21]  .usage.json
│               │       └── [ 534]  xl.meta
│               ├── [  36]  config
│               │   ├── [  21]  config.json
│               │   │   └── [5.1K]  xl.meta
│               │   └── [  25]  iam
│               │       └── [  21]  format.json
│               │           └── [ 431]  xl.meta
│               ├── [ 342]  format.json
│               ├── [   6]  multipart
│               ├── [  21]  pool.bin
│               │   └── [ 464]  xl.meta
│               └── [  64]  tmp
│                   ├── [ 182]  .trash
│                   │   ├── [  25]  0d56f005-fcad-4df2-8c8d-94ef6aeb7bb5
│                   │   │   └── [ 507]  xl.meta.bkp
│                   │   ├── [  25]  b1d56ad7-cf29-49d0-b5d9-1a8bacab4457
│                   │   │   └── [ 494]  xl.meta.bkp
│                   │   ├── [  25]  c92d62ca-87d1-4b6c-912e-ac3cb35adce7
│                   │   │   └── [ 534]  xl.meta.bkp
│                   │   └── [  25]  d2b50b25-0660-4fae-8191-92d21fce691e
│                   │       └── [ 534]  xl.meta.bkp
│                   └── [2.0K]  1fc71795-b86c-4d31-a636-64c9d2353ec7
└── [ 123]  afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445
    ├── [   1]  .FSUUID.afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445 -> .
    ├── [  23]  .directpv
    │   └── [  82]  meta.info
    └── [  18]  pvc-a6e3bbf9-ab29-41d8-ac3b-c1d8920e0d98
        └── [  24]  data
            └── [  98]  .minio.sys
                ├── [  77]  buckets
                │   ├── [  21]  .background-heal.json
                │   │   └── [ 466]  xl.meta
                │   ├── [  21]  .bloomcycle.bin
                │   │   └── [ 515]  xl.meta
                │   └── [  21]  .usage.json
                │       └── [ 534]  xl.meta
                ├── [  36]  config
                │   ├── [  21]  config.json
                │   │   └── [5.1K]  xl.meta
                │   └── [  25]  iam
                │       └── [  21]  format.json
                │           └── [ 431]  xl.meta
                ├── [ 342]  format.json
                ├── [   6]  multipart
                ├── [  21]  pool.bin
                │   └── [ 464]  xl.meta
                └── [  64]  tmp
                    ├── [ 182]  .trash
                    │   ├── [  25]  040bd878-2bcf-4b24-a313-c571e04b9836
                    │   │   └── [ 534]  xl.meta.bkp
                    │   ├── [  25]  069ca199-851c-402b-9158-2206a3f53012
                    │   │   └── [ 494]  xl.meta.bkp
                    │   ├── [  25]  155bf173-abf6-49f4-81cd-d8eda8f97b1b
                    │   │   └── [ 534]  xl.meta.bkp
                    │   └── [  25]  1d1e29d1-a4aa-45df-bfcb-8fd00e75d7ad
                    │       └── [ 507]  xl.meta.bkp
                    └── [2.0K]  4eca5a1c-f617-41dc-a50e-47b18b54bf43

89 directories, 52 files

cat /var/lib/kubelet/plugins/kubernetes.io/csi/directpv-min-io/*/vol_data.json
{"driverName":"directpv-min-io","volumeHandle":"pvc-6aa17a8c-3e37-4dbd-a837-32683b6d765f"}
{"driverName":"directpv-min-io","volumeHandle":"pvc-4a372c6f-6fe1-4ec8-8616-eba5e6b03b57"}
{"driverName":"directpv-min-io","volumeHandle":"pvc-a6e3bbf9-ab29-41d8-ac3b-c1d8920e0d98"}
{"driverName":"directpv-min-io","volumeHandle":"pvc-214d8389-2b4c-48b6-b2d4-67fc413e78fd"}

k get pvc -n tenant1
NAME                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      VOLUMEATTRIBUTESCLASS   AGE
data0-tenant1-pool-0-0   Bound    pvc-a6e3bbf9-ab29-41d8-ac3b-c1d8920e0d98   10Gi       RWO            directpv-min-io   <unset>                 6m58s
data1-tenant1-pool-0-0   Bound    pvc-214d8389-2b4c-48b6-b2d4-67fc413e78fd   10Gi       RWO            directpv-min-io   <unset>                 6m58s
data2-tenant1-pool-0-0   Bound    pvc-4a372c6f-6fe1-4ec8-8616-eba5e6b03b57   10Gi       RWO            directpv-min-io   <unset>                 6m58s
data3-tenant1-pool-0-0   Bound    pvc-6aa17a8c-3e37-4dbd-a837-32683b6d765f   10Gi       RWO            directpv-min-io   <unset>                 6m58s

k get pvc -n tenant1 -o yaml | yq
{
  "apiVersion": "v1",
  "items": [
    {
      "apiVersion": "v1",
      "kind": "PersistentVolumeClaim",
      "metadata": {
        "annotations": {
          "pv.kubernetes.io/bind-completed": "yes",
          "pv.kubernetes.io/bound-by-controller": "yes",
          "volume.beta.kubernetes.io/storage-provisioner": "directpv-min-io",
          "volume.kubernetes.io/selected-node": "k3s-s",
          "volume.kubernetes.io/storage-provisioner": "directpv-min-io"
        },
        "creationTimestamp": "2025-09-20T04:57:13Z",
        "finalizers": [
          "kubernetes.io/pvc-protection"
        ],
        "labels": {
          "v1.min.io/console": "tenant1-console",
          "v1.min.io/pool": "pool-0",
          "v1.min.io/tenant": "tenant1"
        },
        "name": "data0-tenant1-pool-0-0",
        "namespace": "tenant1",
        "resourceVersion": "8633",
        "uid": "a6e3bbf9-ab29-41d8-ac3b-c1d8920e0d98"
      },
      "spec": {
        "accessModes": [
          "ReadWriteOnce"
        ],
        "resources": {
          "requests": {
            "storage": "10Gi"
          }
        },
        "storageClassName": "directpv-min-io",
        "volumeMode": "Filesystem",
        "volumeName": "pvc-a6e3bbf9-ab29-41d8-ac3b-c1d8920e0d98"
      },
      "status": {
        "accessModes": [
          "ReadWriteOnce"
        ],
        "capacity": {
          "storage": "10Gi"
        },
        "phase": "Bound"
      }
    },
    {
      "apiVersion": "v1",
      "kind": "PersistentVolumeClaim",
      "metadata": {
        "annotations": {
          "pv.kubernetes.io/bind-completed": "yes",
          "pv.kubernetes.io/bound-by-controller": "yes",
          "volume.beta.kubernetes.io/storage-provisioner": "directpv-min-io",
          "volume.kubernetes.io/selected-node": "k3s-s",
          "volume.kubernetes.io/storage-provisioner": "directpv-min-io"
        },
        "creationTimestamp": "2025-09-20T04:57:13Z",
        "finalizers": [
          "kubernetes.io/pvc-protection"
        ],
        "labels": {
          "v1.min.io/console": "tenant1-console",
          "v1.min.io/pool": "pool-0",
          "v1.min.io/tenant": "tenant1"
        },
        "name": "data1-tenant1-pool-0-0",
        "namespace": "tenant1",
        "resourceVersion": "8646",
        "uid": "214d8389-2b4c-48b6-b2d4-67fc413e78fd"
      },
      "spec": {
        "accessModes": [
          "ReadWriteOnce"
        ],
        "resources": {
          "requests": {
            "storage": "10Gi"
          }
        },
        "storageClassName": "directpv-min-io",
        "volumeMode": "Filesystem",
        "volumeName": "pvc-214d8389-2b4c-48b6-b2d4-67fc413e78fd"
      },
      "status": {
        "accessModes": [
          "ReadWriteOnce"
        ],
        "capacity": {
          "storage": "10Gi"
        },
        "phase": "Bound"
      }
    },
    {
      "apiVersion": "v1",
      "kind": "PersistentVolumeClaim",
      "metadata": {
        "annotations": {
          "pv.kubernetes.io/bind-completed": "yes",
          "pv.kubernetes.io/bound-by-controller": "yes",
          "volume.beta.kubernetes.io/storage-provisioner": "directpv-min-io",
          "volume.kubernetes.io/selected-node": "k3s-s",
          "volume.kubernetes.io/storage-provisioner": "directpv-min-io"
        },
        "creationTimestamp": "2025-09-20T04:57:13Z",
        "finalizers": [
          "kubernetes.io/pvc-protection"
        ],
        "labels": {
          "v1.min.io/console": "tenant1-console",
          "v1.min.io/pool": "pool-0",
          "v1.min.io/tenant": "tenant1"
        },
        "name": "data2-tenant1-pool-0-0",
        "namespace": "tenant1",
        "resourceVersion": "8662",
        "uid": "4a372c6f-6fe1-4ec8-8616-eba5e6b03b57"
      },
      "spec": {
        "accessModes": [
          "ReadWriteOnce"
        ],
        "resources": {
          "requests": {
            "storage": "10Gi"
          }
        },
        "storageClassName": "directpv-min-io",
        "volumeMode": "Filesystem",
        "volumeName": "pvc-4a372c6f-6fe1-4ec8-8616-eba5e6b03b57"
      },
      "status": {
        "accessModes": [
          "ReadWriteOnce"
        ],
        "capacity": {
          "storage": "10Gi"
        },
        "phase": "Bound"
      }
    },
    {
      "apiVersion": "v1",
      "kind": "PersistentVolumeClaim",
      "metadata": {
        "annotations": {
          "pv.kubernetes.io/bind-completed": "yes",
          "pv.kubernetes.io/bound-by-controller": "yes",
          "volume.beta.kubernetes.io/storage-provisioner": "directpv-min-io",
          "volume.kubernetes.io/selected-node": "k3s-s",
          "volume.kubernetes.io/storage-provisioner": "directpv-min-io"
        },
        "creationTimestamp": "2025-09-20T04:57:13Z",
        "finalizers": [
          "kubernetes.io/pvc-protection"
        ],
        "labels": {
          "v1.min.io/console": "tenant1-console",
          "v1.min.io/pool": "pool-0",
          "v1.min.io/tenant": "tenant1"
        },
        "name": "data3-tenant1-pool-0-0",
        "namespace": "tenant1",
        "resourceVersion": "8641",
        "uid": "6aa17a8c-3e37-4dbd-a837-32683b6d765f"
      },
      "spec": {
        "accessModes": [
          "ReadWriteOnce"
        ],
        "resources": {
          "requests": {
            "storage": "10Gi"
          }
        },
        "storageClassName": "directpv-min-io",
        "volumeMode": "Filesystem",
        "volumeName": "pvc-6aa17a8c-3e37-4dbd-a837-32683b6d765f"
      },
      "status": {
        "accessModes": [
          "ReadWriteOnce"
        ],
        "capacity": {
          "storage": "10Gi"
        },
        "phase": "Bound"
      }
    }
  ],
  "kind": "List",
  "metadata": {
    "resourceVersion": ""
  }
}

kc describe pvc -n tenant1
Name:          data0-tenant1-pool-0-0
Namespace:     tenant1
StorageClass:  directpv-min-io
Status:        Bound
Volume:        pvc-a6e3bbf9-ab29-41d8-ac3b-c1d8920e0d98
Labels:        v1.min.io/console=tenant1-console
               v1.min.io/pool=pool-0
               v1.min.io/tenant=tenant1
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: directpv-min-io
               volume.kubernetes.io/selected-node: k3s-s
               volume.kubernetes.io/storage-provisioner: directpv-min-io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      10Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       tenant1-pool-0-0
Events:
  Type    Reason                 Age                    From                                                                              Message
  ----    ------                 ----                   ----                                                                              -------
  Normal  WaitForFirstConsumer   7m10s                  persistentvolume-controller                                                       waiting for first consumer to be created before binding
  Normal  ExternalProvisioning   7m10s (x2 over 7m10s)  persistentvolume-controller                                                       Waiting for a volume to be created either by the external provisioner 'directpv-min-io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Normal  Provisioning           7m10s                  directpv-min-io_controller-77b4dd6494-sqfsn_abc55137-7bf2-4ecc-82c5-d801bfd9579c  External provisioner is provisioning volume for claim "tenant1/data0-tenant1-pool-0-0"
  Normal  ProvisioningSucceeded  7m10s                  directpv-min-io_controller-77b4dd6494-sqfsn_abc55137-7bf2-4ecc-82c5-d801bfd9579c  Successfully provisioned volume pvc-a6e3bbf9-ab29-41d8-ac3b-c1d8920e0d98

Name:          data1-tenant1-pool-0-0
Namespace:     tenant1
StorageClass:  directpv-min-io
Status:        Bound
Volume:        pvc-214d8389-2b4c-48b6-b2d4-67fc413e78fd
Labels:        v1.min.io/console=tenant1-console
               v1.min.io/pool=pool-0
               v1.min.io/tenant=tenant1
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: directpv-min-io
               volume.kubernetes.io/selected-node: k3s-s
               volume.kubernetes.io/storage-provisioner: directpv-min-io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      10Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       tenant1-pool-0-0
Events:
  Type    Reason                 Age                    From                                                                              Message
  ----    ------                 ----                   ----                                                                              -------
  Normal  WaitForFirstConsumer   7m10s                  persistentvolume-controller                                                       waiting for first consumer to be created before binding
  Normal  ExternalProvisioning   7m10s (x2 over 7m10s)  persistentvolume-controller                                                       Waiting for a volume to be created either by the external provisioner 'directpv-min-io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Normal  Provisioning           7m10s                  directpv-min-io_controller-77b4dd6494-sqfsn_abc55137-7bf2-4ecc-82c5-d801bfd9579c  External provisioner is provisioning volume for claim "tenant1/data1-tenant1-pool-0-0"
  Normal  ProvisioningSucceeded  7m10s                  directpv-min-io_controller-77b4dd6494-sqfsn_abc55137-7bf2-4ecc-82c5-d801bfd9579c  Successfully provisioned volume pvc-214d8389-2b4c-48b6-b2d4-67fc413e78fd

Name:          data2-tenant1-pool-0-0
Namespace:     tenant1
StorageClass:  directpv-min-io
Status:        Bound
Volume:        pvc-4a372c6f-6fe1-4ec8-8616-eba5e6b03b57
Labels:        v1.min.io/console=tenant1-console
               v1.min.io/pool=pool-0
               v1.min.io/tenant=tenant1
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: directpv-min-io
               volume.kubernetes.io/selected-node: k3s-s
               volume.kubernetes.io/storage-provisioner: directpv-min-io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      10Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       tenant1-pool-0-0
Events:
  Type     Reason                 Age                    From                                                                              Message
  ----     ------                 ----                   ----                                                                              -------
  Normal   WaitForFirstConsumer   7m10s                  persistentvolume-controller                                                       waiting for first consumer to be created before binding
  Normal   ExternalProvisioning   7m10s (x2 over 7m10s)  persistentvolume-controller                                                       Waiting for a volume to be created either by the external provisioner 'directpv-min-io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Warning  ProvisioningFailed     7m10s                  directpv-min-io_controller-77b4dd6494-sqfsn_abc55137-7bf2-4ecc-82c5-d801bfd9579c  failed to provision volume with StorageClass "directpv-min-io": rpc error: code = Internal desc = unable to update reserved drive for volume pvc-4a372c6f-6fe1-4ec8-8616-eba5e6b03b57; drive=668267e9-fdd2-46c9-b854-1ac1de70132f, node=k3s-s, name=nvme4n1; Operation cannot be fulfilled on directpvdrives.directpv.min.io "668267e9-fdd2-46c9-b854-1ac1de70132f": the object has been modified; please apply your changes to the latest version and try again
  Normal   Provisioning           7m9s (x2 over 7m10s)   directpv-min-io_controller-77b4dd6494-sqfsn_abc55137-7bf2-4ecc-82c5-d801bfd9579c  External provisioner is provisioning volume for claim "tenant1/data2-tenant1-pool-0-0"
  Normal   ProvisioningSucceeded  7m9s                   directpv-min-io_controller-77b4dd6494-sqfsn_abc55137-7bf2-4ecc-82c5-d801bfd9579c  Successfully provisioned volume pvc-4a372c6f-6fe1-4ec8-8616-eba5e6b03b57

Name:          data3-tenant1-pool-0-0
Namespace:     tenant1
StorageClass:  directpv-min-io
Status:        Bound
Volume:        pvc-6aa17a8c-3e37-4dbd-a837-32683b6d765f
Labels:        v1.min.io/console=tenant1-console
               v1.min.io/pool=pool-0
               v1.min.io/tenant=tenant1
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: directpv-min-io
               volume.kubernetes.io/selected-node: k3s-s
               volume.kubernetes.io/storage-provisioner: directpv-min-io
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      10Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Used By:       tenant1-pool-0-0
Events:
  Type    Reason                 Age                    From                                                                              Message
  ----    ------                 ----                   ----                                                                              -------
  Normal  WaitForFirstConsumer   7m10s                  persistentvolume-controller                                                       waiting for first consumer to be created before binding
  Normal  ExternalProvisioning   7m10s (x2 over 7m10s)  persistentvolume-controller                                                       Waiting for a volume to be created either by the external provisioner 'directpv-min-io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Normal  Provisioning           7m10s                  directpv-min-io_controller-77b4dd6494-sqfsn_abc55137-7bf2-4ecc-82c5-d801bfd9579c  External provisioner is provisioning volume for claim "tenant1/data3-tenant1-pool-0-0"
  Normal  ProvisioningSucceeded  7m10s                  directpv-min-io_controller-77b4dd6494-sqfsn_abc55137-7bf2-4ecc-82c5-d801bfd9579c  Successfully provisioned volume pvc-6aa17a8c-3e37-4dbd-a837-32683b6d765f
(⎈|default:N/A) root@k3s-s:~# kubectl get sts,pod,svc,ep,pvc,secret -n tenant1
Warning: v1 Endpoints is deprecated in v1.33+; use discovery.k8s.io/v1 EndpointSlice
NAME                              READY   AGE
statefulset.apps/tenant1-pool-0   1/1     7m17s

NAME                   READY   STATUS    RESTARTS   AGE
pod/tenant1-pool-0-0   2/2     Running   0          7m17s

NAME                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/minio             ClusterIP   10.43.170.174   <none>        443/TCP    7m18s
service/tenant1-console   ClusterIP   10.43.98.213    <none>        9443/TCP   7m18s
service/tenant1-hl        ClusterIP   None            <none>        9000/TCP   7m18s

NAME                        ENDPOINTS         AGE
endpoints/minio             10.42.0.12:9000   7m18s
endpoints/tenant1-console   10.42.0.12:9443   7m18s
endpoints/tenant1-hl        10.42.0.12:9000   7m18s

NAME                                           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS      VOLUMEATTRIBUTESCLASS   AGE
persistentvolumeclaim/data0-tenant1-pool-0-0   Bound    pvc-a6e3bbf9-ab29-41d8-ac3b-c1d8920e0d98   10Gi       RWO            directpv-min-io   <unset>                 7m17s
persistentvolumeclaim/data1-tenant1-pool-0-0   Bound    pvc-214d8389-2b4c-48b6-b2d4-67fc413e78fd   10Gi       RWO            directpv-min-io   <unset>                 7m17s
persistentvolumeclaim/data2-tenant1-pool-0-0   Bound    pvc-4a372c6f-6fe1-4ec8-8616-eba5e6b03b57   10Gi       RWO            directpv-min-io   <unset>                 7m17s
persistentvolumeclaim/data3-tenant1-pool-0-0   Bound    pvc-6aa17a8c-3e37-4dbd-a837-32683b6d765f   10Gi       RWO            directpv-min-io   <unset>                 7m17s

NAME                                   TYPE                 DATA   AGE
secret/sh.helm.release.v1.tenant1.v1   helm.sh/release.v1   1      7m33s
secret/tenant1-env-configuration       Opaque               1      7m33s
secret/tenant1-tls                     Opaque               2      7m23s

kubectl get pod -n tenant1 -l v1.min.io/pool=pool-0 -owide
NAME               READY   STATUS    RESTARTS   AGE     IP           NODE    NOMINATED NODE   READINESS GATES
tenant1-pool-0-0   2/2     Running   0          7m23s   10.42.0.12   k3s-s   <none>           <none>

kc describe pod -n tenant1 -l v1.min.io/pool=pool-0
Name:             tenant1-pool-0-0
Namespace:        tenant1
Priority:         0
Service Account:  tenant1-sa
Node:             k3s-s/192.168.10.10
Start Time:       Sat, 20 Sep 2025 13:57:15 +0900
Labels:           apps.kubernetes.io/pod-index=0
                  controller-revision-hash=tenant1-pool-0-b5b7b8c97
                  statefulset.kubernetes.io/pod-name=tenant1-pool-0-0
                  v1.min.io/console=tenant1-console
                  v1.min.io/pool=pool-0
                  v1.min.io/tenant=tenant1
Annotations:      min.io/revision: 0
Status:           Running
IP:               10.42.0.12
IPs:
  IP:           10.42.0.12
Controlled By:  StatefulSet/tenant1-pool-0
Init Containers:
  validate-arguments:
    Container ID:  containerd://98370489d3114970050995891d82ba1c386c7e46261776fd38a45fc7b4b03a88
    Image:         quay.io/minio/operator-sidecar:v7.0.1
    Image ID:      quay.io/minio/operator-sidecar@sha256:ccc4fb79135aed8bf9c73d6f758d8e4f4ff24f51ca1bdd6f5b8ad9a9ad841a2f
    Port:          <none>
    Host Port:     <none>
    Args:
      validate
      --tenant
      tenant1
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Sat, 20 Sep 2025 13:57:19 +0900
      Finished:     Sat, 20 Sep 2025 13:57:19 +0900
    Ready:          True
    Restart Count:  0
    Environment:
      CLUSTER_DOMAIN:  cluster.local
    Mounts:
      /tmp/minio-config from configuration (rw)
      /tmp/minio/ from cfg-vol (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-smh8h (ro)
Containers:
  minio:
    Container ID:  containerd://7c0810a86316e58cc6b8fec6285ef84f095c8ec629aff095d5ab8e452f2335ed
    Image:         quay.io/minio/minio:RELEASE.2025-04-08T15-41-24Z
    Image ID:      quay.io/minio/minio@sha256:8834ae47a2de3509b83e0e70da9369c24bbbc22de42f2a2eddc530eee88acd1b
    Ports:         9000/TCP, 9443/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      server
      --certs-dir
      /tmp/certs
      --console-address
      :9443
    State:          Running
      Started:      Sat, 20 Sep 2025 13:57:24 +0900
    Ready:          True
    Restart Count:  0
    Environment:
      MINIO_CONFIG_ENV_FILE:  /tmp/minio/config.env
    Mounts:
      /export0 from data0 (rw)
      /export1 from data1 (rw)
      /export2 from data2 (rw)
      /export3 from data3 (rw)
      /tmp/certs from tenant1-tls (rw)
      /tmp/minio/ from cfg-vol (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-smh8h (ro)
  sidecar:
    Container ID:  containerd://f22ed43f8774d13c96cf51a460cdb70cba1ba2d366fd43f9e4e850b4921e5e8a
    Image:         quay.io/minio/operator-sidecar:v7.0.1
    Image ID:      quay.io/minio/operator-sidecar@sha256:ccc4fb79135aed8bf9c73d6f758d8e4f4ff24f51ca1bdd6f5b8ad9a9ad841a2f
    Port:          <none>
    Host Port:     <none>
    Args:
      sidecar
      --tenant
      tenant1
      --config-name
      tenant1-env-configuration
    State:          Running
      Started:      Sat, 20 Sep 2025 13:57:24 +0900
    Ready:          True
    Restart Count:  0
    Readiness:      http-get http://:4444/ready delay=5s timeout=1s period=1s #success=1 #failure=1
    Environment:
      CLUSTER_DOMAIN:  cluster.local
    Mounts:
      /tmp/minio-config from configuration (rw)
      /tmp/minio/ from cfg-vol (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-smh8h (ro)
Conditions:
  Type                        Status
  PodReadyToStartContainers   True
  Initialized                 True
  Ready                       True
  ContainersReady             True
  PodScheduled                True
Volumes:
  data0:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data0-tenant1-pool-0-0
    ReadOnly:   false
  data1:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data1-tenant1-pool-0-0
    ReadOnly:   false
  data2:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data2-tenant1-pool-0-0
    ReadOnly:   false
  data3:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data3-tenant1-pool-0-0
    ReadOnly:   false
  cfg-vol:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  tenant1-tls:
    Type:        Projected (a volume that contains injected data from multiple sources)
    SecretName:  tenant1-tls
    Optional:    false
  configuration:
    Type:        Projected (a volume that contains injected data from multiple sources)
    SecretName:  tenant1-env-configuration
    Optional:    false
  kube-api-access-smh8h:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    Optional:                false
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  7m25s  default-scheduler  Successfully assigned tenant1/tenant1-pool-0-0 to k3s-s
  Normal  Pulling    7m25s  kubelet            Pulling image "quay.io/minio/operator-sidecar:v7.0.1"
  Normal  Pulled     7m22s  kubelet            Successfully pulled image "quay.io/minio/operator-sidecar:v7.0.1" in 3.618s (3.618s including waiting). Image size: 22033319 bytes.
  Normal  Created    7m22s  kubelet            Created container: validate-arguments
  Normal  Started    7m22s  kubelet            Started container validate-arguments
  Normal  Pulling    7m21s  kubelet            Pulling image "quay.io/minio/minio:RELEASE.2025-04-08T15-41-24Z"
  Normal  Pulled     7m17s  kubelet            Successfully pulled image "quay.io/minio/minio:RELEASE.2025-04-08T15-41-24Z" in 4.255s (4.255s including waiting). Image size: 63924941 bytes.
  Normal  Created    7m17s  kubelet            Created container: minio
  Normal  Started    7m17s  kubelet            Started container minio
  Normal  Pulled     7m17s  kubelet            Container image "quay.io/minio/operator-sidecar:v7.0.1" already present on machine
  Normal  Created    7m17s  kubelet            Created container: sidecar
  Normal  Started    7m17s  kubelet            Started container sidecar
(⎈|default:N/A) root@k3s-s:~# kubectl stern -n tenant1 -l v1.min.io/pool=pool-0
+ tenant1-pool-0-0 › minio
+ tenant1-pool-0-0 › validate-arguments
+ tenant1-pool-0-0 › sidecar
tenant1-pool-0-0 sidecar 2025/09/20 04:57:24 sidecar_utils.go:50: Starting Sidecar
tenant1-pool-0-0 minio INFO: Formatting 1st pool, 1 set(s), 4 drives per set.
tenant1-pool-0-0 minio INFO: WARNING: Host local has more than 1 drives of set. A host failure will result in data becoming unavailable.
tenant1-pool-0-0 minio MinIO Object Storage Server
tenant1-pool-0-0 minio Copyright: 2015-2025 MinIO, Inc.
tenant1-pool-0-0 minio License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html
tenant1-pool-0-0 minio Version: RELEASE.2025-04-08T15-41-24Z (go1.24.2 linux/amd64)
tenant1-pool-0-0 minio
tenant1-pool-0-0 minio API: https://minio.tenant1.svc.cluster.local
tenant1-pool-0-0 minio WebUI: https://10.42.0.12:9443 https://127.0.0.1:9443
- tenant1-pool-0-0 › validate-arguments
tenant1-pool-0-0 minio
tenant1-pool-0-0 minio Docs: https://docs.min.io
tenant1-pool-0-0 minio INFO:
tenant1-pool-0-0 minio  You are running an older version of MinIO released 5 months before the latest release
tenant1-pool-0-0 minio  Update: Run `mc admin update ALIAS`
tenant1-pool-0-0 minio
tenant1-pool-0-0 minio
^C

kubectl exec -it -n tenant1 sts/tenant1-pool-0 -c minio -- id
uid=1000 gid=1000 groups=1000

kubectl exec -it -n tenant1 sts/tenant1-pool-0 -c minio -- env
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
HOSTNAME=tenant1-pool-0-0
MINIO_ACCESS_KEY_FILE=access_key
MINIO_SECRET_KEY_FILE=secret_key
MINIO_ROOT_USER_FILE=access_key
MINIO_ROOT_PASSWORD_FILE=secret_key
MINIO_KMS_SECRET_KEY_FILE=kms_master_key
MINIO_UPDATE_MINISIGN_PUBKEY=RWTx5Zr1tiHQLwG9keckT0c45M3AGeHD6IvimQHpyRywVWGbP1aVSGav
MINIO_CONFIG_ENV_FILE=/tmp/minio/config.env
MC_CONFIG_DIR=/tmp/.mc
TENANT1_CONSOLE_SERVICE_PORT=9443
KUBERNETES_SERVICE_PORT_HTTPS=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
TENANT1_CONSOLE_SERVICE_HOST=10.43.98.213
TENANT1_CONSOLE_SERVICE_PORT_HTTPS_CONSOLE=9443
TENANT1_CONSOLE_PORT_9443_TCP=tcp://10.43.98.213:9443
TENANT1_CONSOLE_PORT_9443_TCP_PORT=9443
KUBERNETES_SERVICE_HOST=10.43.0.1
KUBERNETES_PORT=tcp://10.43.0.1:443
MINIO_SERVICE_HOST=10.43.170.174
MINIO_PORT_443_TCP_PORT=443
TENANT1_CONSOLE_PORT_9443_TCP_PROTO=tcp
KUBERNETES_SERVICE_PORT=443
KUBERNETES_PORT_443_TCP=tcp://10.43.0.1:443
KUBERNETES_PORT_443_TCP_PORT=443
MINIO_PORT_443_TCP_PROTO=tcp
MINIO_PORT_443_TCP_ADDR=10.43.170.174
TENANT1_CONSOLE_PORT=tcp://10.43.98.213:9443
TENANT1_CONSOLE_PORT_9443_TCP_ADDR=10.43.98.213
KUBERNETES_PORT_443_TCP_ADDR=10.43.0.1
MINIO_SERVICE_PORT=443
MINIO_SERVICE_PORT_HTTPS_MINIO=443
MINIO_PORT=tcp://10.43.170.174:443
MINIO_PORT_443_TCP=tcp://10.43.170.174:443
TERM=xterm
HOME=/

kubectl exec -it -n tenant1 sts/tenant1-pool-0 -c minio -- cat /tmp/minio/config.env
export MINIO_ARGS="/export{0...3}/data"
export MINIO_PROMETHEUS_JOB_ID="minio-job"
export MINIO_ROOT_PASSWORD="minio123"
export MINIO_ROOT_USER="minio"
export MINIO_SERVER_URL="https://minio.tenant1.svc.cluster.local:443"
export MINIO_STORAGE_CLASS_STANDARD="EC:1"
export MINIO_UPDATE="on"
export MINIO_UPDATE_MINISIGN_PUBKEY="RWTx5Zr1tiHQLwG9keckT0c45M3AGeHD6IvimQHpyRywVWGbP1aVSGav"

kubectl get secret -n tenant1 tenant1-env-configuration -o jsonpath='{.data.config\.env}' | base64 -d ; echo
export MINIO_ROOT_USER="minio"
export MINIO_ROOT_PASSWORD="minio123"

kubectl get secret -n tenant1 tenant1-tls -o jsonpath='{.data.public\.crt}' | base64 -d
-----BEGIN CERTIFICATE-----
MIICeDCCAh2gAwIBAgIQVEwEIiO82AwdroJiVOiK0DAKBggqhkjOPQQDAjAjMSEw
HwYDVQQDDBhrM3Mtc2VydmVyLWNhQDE3NTgzMzA0NjMwHhcNMjUwOTIwMDQ1MjAy
WhcNMjYwOTIwMDQ1MjAyWjBUMRUwEwYDVQQKEwxzeXN0ZW06bm9kZXMxOzA5BgNV
BAMMMnN5c3RlbTpub2RlOioudGVuYW50MS1obC50ZW5hbnQxLnN2Yy5jbHVzdGVy
LmxvY2FsMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEQbjj5L1YuN7lpbO7cm+X
oCWi2n7zUZHK7sLp44nALR5ap9SLmvZPlY7DqCvjtWQS47uvJNKYNe2x+SMzWZu3
JKOCAQAwgf0wDgYDVR0PAQH/BAQDAgWgMBMGA1UdJQQMMAoGCCsGAQUFBwMBMAwG
A1UdEwEB/wQCMAAwHwYDVR0jBBgwFoAU50QQ6n95QeBQjoRIIR8gysXkH/cwgaYG
A1UdEQSBnjCBm4I1dGVuYW50MS1wb29sLTAtMC50ZW5hbnQxLWhsLnRlbmFudDEu
c3ZjLmNsdXN0ZXIubG9jYWyCH21pbmlvLnRlbmFudDEuc3ZjLmNsdXN0ZXIubG9j
YWyCDW1pbmlvLnRlbmFudDGCEW1pbmlvLnRlbmFudDEuc3ZjggIqLoIbKi50ZW5h
bnQxLnN2Yy5jbHVzdGVyLmxvY2FsMAoGCCqGSM49BAMCA0kAMEYCIQChMJnhgXLN
gxkHFT81X6gYx6hlDzGM6GbQusUG7JETkQIhAO1coIJQDqny6ATPnAS6DeiBsene
kSgZhA6DNY7/O5vl
-----END CERTIFICATE-----

kubectl get secret -n tenant1 tenant1-tls -o jsonpath='{.data.public\.crt}' | base64 -d | openssl x509 -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            54:4c:04:22:23:bc:d8:0c:1d:ae:82:62:54:e8:8a:d0
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: CN = k3s-server-ca@1758330463
        Validity
            Not Before: Sep 20 04:52:02 2025 GMT
            Not After : Sep 20 04:52:02 2026 GMT
        Subject: O = system:nodes, CN = system:node:*.tenant1-hl.tenant1.svc.cluster.local
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:41:b8:e3:e4:bd:58:b8:de:e5:a5:b3:bb:72:6f:
                    97:a0:25:a2:da:7e:f3:51:91:ca:ee:c2:e9:e3:89:
                    c0:2d:1e:5a:a7:d4:8b:9a:f6:4f:95:8e:c3:a8:2b:
                    e3:b5:64:12:e3:bb:af:24:d2:98:35:ed:b1:f9:23:
                    33:59:9b:b7:24
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier:
                E7:44:10:EA:7F:79:41:E0:50:8E:84:48:21:1F:20:CA:C5:E4:1F:F7
            X509v3 Subject Alternative Name:
                DNS:tenant1-pool-0-0.tenant1-hl.tenant1.svc.cluster.local, DNS:minio.tenant1.svc.cluster.local, DNS:minio.tenant1, DNS:minio.tenant1.svc, DNS:*., DNS:*.tenant1.svc.cluster.local
    Signature Algorithm: ecdsa-with-SHA256
    Signature Value:
        30:46:02:21:00:a1:30:99:e1:81:72:cd:83:19:07:15:3f:35:
        5f:a8:18:c7:a8:65:0f:31:8c:e8:66:d0:ba:c5:06:ec:91:13:
        91:02:21:00:ed:5c:a0:82:50:0e:a9:f2:e8:04:cf:9c:04:ba:
        0d:e8:81:b1:e9:de:91:28:19:84:0e:83:35:8e:ff:3b:9b:e5

kubectl patch svc -n tenant1 tenant1-console -p '{"spec": {"type": "NodePort", "ports": [{"port": 9443, "targetPort": 9443, "nodePort": 30001}]}}'
service/tenant1-console patched

# 기본키(minio , minio123)
echo "https://$(curl -s ipinfo.io/ip):30001"
https://3.36.72.189:30001

kubectl patch svc -n tenant1 minio -p '{"spec": {"type": "NodePort", "ports": [{"port": 443, "targetPort": 9000, "nodePort": 30002}]}}'
service/minio patched

# mc alias
mc alias set k8s-tenant1 https://127.0.0.1:30002 minio minio123 --insecure
mc: Configuration written to `/root/.mc/config.json`. Please update your access credentials.
mc: Successfully created `/root/.mc/share`.
mc: Initialized share uploads `/root/.mc/share/uploads.json` file.
mc: Initialized share downloads `/root/.mc/share/downloads.json` file.
Added `k8s-tenant1` successfully.

mc alias list
gcs
  URL       : https://storage.googleapis.com
  AccessKey : YOUR-ACCESS-KEY-HERE
  SecretKey : YOUR-SECRET-KEY-HERE
  API       : S3v2
  Path      : dns
  Src       : /root/.mc/config.json

k8s-tenant1
  URL       : https://127.0.0.1:30002
  AccessKey : minio
  SecretKey : minio123
  API       : s3v4
  Path      : auto
  Src       : /root/.mc/config.json

local
  URL       : http://localhost:9000
  AccessKey :
  SecretKey :
  API       :
  Path      : auto
  Src       : /root/.mc/config.json

play
  URL       : https://play.min.io
  AccessKey : Q3AM3UQ867SPQQA43P2F
  SecretKey : zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG
  API       : S3v4
  Path      : auto
  Src       : /root/.mc/config.json

s3
  URL       : https://s3.amazonaws.com
  AccessKey : YOUR-ACCESS-KEY-HERE
  SecretKey : YOUR-SECRET-KEY-HERE
  API       : S3v4
  Path      : dns
  Src       : /root/.mc/config.json

mc admin info k8s-tenant1 --insecure
●  127.0.0.1:30002
   Uptime: 10 minutes
   Version: 2025-04-08T15:41:24Z
   Network: 1/1 OK
   Drives: 4/4 OK
   Pool: 1

┌──────┬──────────────────────┬─────────────────────┬──────────────┐
│ Pool │ Drives Usage         │ Erasure stripe size │ Erasure sets │
│ 1st  │ 0.0% (total: 30 GiB) │ 4                   │ 1            │
└──────┴──────────────────────┴─────────────────────┴──────────────┘

4 drives online, 0 drives offline, EC:1

# 버킷 생성
mc mb k8s-tenant1/mybucket --insecure
Bucket created successfully `k8s-tenant1/mybucket`.

mc ls k8s-tenant1 --insecure
[2025-09-20 14:07:55 KST]     0B mybucket/

활용 실습

https://docs.min.io/community/minio-object-store/reference/minio-mc.html

MinIO Client — MinIO Object Storage (AGPLv3)

The following commands add a temporary extension to your system PATH for running the mc utility. Defer to your operating system instructions for making permanent modifications to your system PATH. Alternatively, execute mc by navigating to the parent folde

docs.min.io

# [신규 터미널] 모니터링 Show HTTP call trace for all incoming and internode on MinIO
# https://docs.min.io/community/minio-object-store/reference/minio-mc-admin/mc-admin-trace.html
# mc admin trace --errors -v -a k8s-tenant1 --insecure # 4xx, 5xx 에러만 보기
mc admin trace -v -a k8s-tenant1 --insecure

# https://docs.min.io/community/minio-object-store/reference/minio-mc/mc-ls.html
mc ls --summarize --recursive k8s-tenant1 --insecure

Total Size: 0 B
Total Objects: 0

# https://docs.min.io/community/minio-object-store/reference/minio-mc/mc-cat.html
echo hello > hello.txt
mc cp ./hello.txt k8s-tenant1/mybucket/ --insecure
/root/hello.txt:           6 B / 6 B ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 494 B/s 0s

mc cat k8s-tenant1/mybucket/hello.txt --insecure
hello

# 100MB 파일 생성
</dev/urandom tr -dc 'A-Za-z0-9' | head -c 100M > randtext.txt
ls -alh randtext.txt
-rw-r--r-- 1 root root 100M Sep 20 14:24 randtext.txt
cat randtext.txt | md5sum
e9dacdeefb85d5ab467b39dbd062eb28  -

# https://docs.min.io/community/minio-object-store/reference/minio-mc/mc-cp.html
mc cp ./randtext.txt k8s-tenant1/mybucket/ --insecure
/root/randtext.txt:        100.00 MiB / 100.00 MiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 309.26 MiB/s 0s

mc cat k8s-tenant1/mybucket/randtext.txt --insecure | md5sum
e9dacdeefb85d5ab467b39dbd062eb28  -

# MinIO Multipart Upload는 '큰 파일을 S3 방식으로 업로드할 때 쪼개서 올리고, 다 올리면 서버가 합쳐주는' 기능
[Client] ---CreateMultipartUpload---> [MinIO]
         <--- UploadId ---------------
[Client] ---UploadPart(part1)--------> [MinIO]
[Client] ---UploadPart(part2)--------> [MinIO]
...
[Client] ---CompleteMultipartUpload--> [MinIO]
         <--- OK + Final Object -------

mc admin trace -v -a k8s-tenant1 --insecure
127.0.0.1:30002 [REQUEST s3.CompleteMultipartUpload] [2025-09-14T15:11:04.710] [Client IP: 10.42.0.1]
127.0.0.1:30002 POST /mybucket/randtext.txt?uploadId=ZTMyZjNlMjAtZDkyNi00YjcyLTlhNGItZTI4ODBiNzBmZTM1LmY3N2ZiOTY0LWM5NmItNDE5YS1iNThmLWY3ODAyM2FmY2JjM3gxNzU3ODMwMjY0NDE5MzA0NDQ1
127.0.0.1:30002 Proto: HTTP/1.1
127.0.0.1:30002 Host: 127.0.0.1:30002
127.0.0.1:30002 Content-Type: application/octet-stream
127.0.0.1:30002 User-Agent: MinIO (linux; amd64) minio-go/v7.0.90 mc/RELEASE.2025-08-13T08-35-41Z
127.0.0.1:30002 X-Amz-Content-Sha256: 159a60374060e572e313b089f1d4658285b3676e38c08cdba940c0342f7431a9
127.0.0.1:30002 X-Amz-Date: 20250914T061104Z
127.0.0.1:30002 Accept-Encoding: zstd,gzip
127.0.0.1:30002 Authorization: AWS4-HMAC-SHA256 Credential=minio/20250914/us-east-1/s3/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date, Signature=3006225675b1e83ec73a4acdc49e14912ea547e075be739cc9a580ebde31a858
127.0.0.1:30002 Content-Length: 687
127.0.0.1:30002 <CompleteMultipartUpload xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Part><PartNumber>1</PartNumber><ETag>2e7fda86cc02f78e25480c65efb94a29</ETag></Part><Part><PartNumber>2</PartNumber><ETag>e2b66d6f0368ce9a13de6ca12ed78a90</ETag></Part><Part><PartNumber>3</PartNumber><ETag>7b0ae1958b0f62992b13dfe4a7dddf07</ETag></Part><Part><PartNumber>4</PartNumber><ETag>942130b42c320b33a77ad9812416546b</ETag></Part><Part><PartNumber>5</PartNumber><ETag>81bea9efbeb94cadacb210343698806d</ETag></Part><Part><PartNumber>6</PartNumber><ETag>89c6655445c13b36c83a37b8b8e589c1</ETag></Part><Part><PartNumber>7</PartNumber><ETag>084d78f3c6126f91dc72c1933754ec46</ETag></Part></CompleteMultipartUpload>

127.0.0.1:30002 [RESPONSE] [2025-09-14T15:11:04.717] [ Duration 7.469ms TTFB 7.391942ms ↑ 793 B  ↓ 321 B ]
...
<CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Location>https://127.0.0.1:30002/mybucket/randtext.txt</Location><Bucket>mybucket</Bucket><Key>randtext.txt</Key><ETag>"2ab7293624b88db302add6a2cacdd804-7"</ETag></CompleteMultipartUploadResult>
# 참고) put https://docs.min.io/community/minio-object-store/reference/minio-mc/mc-put.html

cp randtext.txt randtext2.txt
mc cp ./randtext2.txt k8s-tenant1/mybucket/ --disable-multipart --insecure
/root/randtext2.txt:       100.00 MiB / 100.00 MiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 272.52 MiB/s 0s

mc ls --summarize --recursive --versions k8s-tenant1 --insecure
[2025-09-20 14:07:55 KST]     0B mybucket/
[2025-09-20 14:24:08 KST]     6B STANDARD null v1 PUT mybucket/hello.txt
[2025-09-20 14:24:59 KST] 100MiB STANDARD null v1 PUT mybucket/randtext.txt
[2025-09-20 14:25:49 KST] 100MiB STANDARD null v1 PUT mybucket/randtext2.txt

Total Size: 200 MiB
Total Objects: 4

k exec -it -n tenant1 tenant1-pool-0-0 -c minio -- sh -c 'df -hT --type xfs'
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/nvme1n1   xfs    10G   67M   10G   1% /export0
/dev/nvme4n1   xfs    10G   67M   10G   1% /export1
/dev/nvme3n1   xfs    10G   67M   10G   1% /export2
/dev/nvme2n1   xfs    10G   67M   10G   1% /export3

k exec -it -n tenant1 tenant1-pool-0-0 -c minio -- ls -R /export0
/export0:
data

/export0/data:
mybucket

/export0/data/mybucket:
hello.txt  randtext.txt  randtext2.txt

/export0/data/mybucket/hello.txt:
xl.meta

/export0/data/mybucket/randtext.txt:
373427ec-c35a-4cb4-ada6-cfff86ac1d2b  xl.meta

/export0/data/mybucket/randtext.txt/373427ec-c35a-4cb4-ada6-cfff86ac1d2b:
part.1  part.2  part.3  part.4  part.5  part.6  part.7

/export0/data/mybucket/randtext2.txt:
e3d23b43-5fbf-4fa9-8b29-b67e9075ad76  xl.meta

/export0/data/mybucket/randtext2.txt/e3d23b43-5fbf-4fa9-8b29-b67e9075ad76:
part.1

k exec -it -n tenant1 tenant1-pool-0-0 -c minio -- ls -R /export1
/export1:
data

/export1/data:
mybucket

/export1/data/mybucket:
hello.txt  randtext.txt  randtext2.txt

/export1/data/mybucket/hello.txt:
xl.meta

/export1/data/mybucket/randtext.txt:
373427ec-c35a-4cb4-ada6-cfff86ac1d2b  xl.meta

/export1/data/mybucket/randtext.txt/373427ec-c35a-4cb4-ada6-cfff86ac1d2b:
part.1  part.2  part.3  part.4  part.5  part.6  part.7

/export1/data/mybucket/randtext2.txt:
e3d23b43-5fbf-4fa9-8b29-b67e9075ad76  xl.meta

/export1/data/mybucket/randtext2.txt/e3d23b43-5fbf-4fa9-8b29-b67e9075ad76:
part.1
k exec -it -n tenant1 tenant1-pool-0-0 -c minio -- ls -R /export2
/export2:
data

/export2/data:
mybucket

/export2/data/mybucket:
hello.txt  randtext.txt  randtext2.txt

/export2/data/mybucket/hello.txt:
xl.meta

/export2/data/mybucket/randtext.txt:
373427ec-c35a-4cb4-ada6-cfff86ac1d2b  xl.meta

/export2/data/mybucket/randtext.txt/373427ec-c35a-4cb4-ada6-cfff86ac1d2b:
part.1  part.2  part.3  part.4  part.5  part.6  part.7

/export2/data/mybucket/randtext2.txt:
e3d23b43-5fbf-4fa9-8b29-b67e9075ad76  xl.meta

/export2/data/mybucket/randtext2.txt/e3d23b43-5fbf-4fa9-8b29-b67e9075ad76:
part.1
k exec -it -n tenant1 tenant1-pool-0-0 -c minio -- ls -R /export3
/export3:
data

/export3/data:
mybucket

/export3/data/mybucket:
hello.txt  randtext.txt  randtext2.txt

/export3/data/mybucket/hello.txt:
xl.meta

/export3/data/mybucket/randtext.txt:
373427ec-c35a-4cb4-ada6-cfff86ac1d2b  xl.meta

/export3/data/mybucket/randtext.txt/373427ec-c35a-4cb4-ada6-cfff86ac1d2b:
part.1  part.2  part.3  part.4  part.5  part.6  part.7

/export3/data/mybucket/randtext2.txt:
e3d23b43-5fbf-4fa9-8b29-b67e9075ad76  xl.meta

/export3/data/mybucket/randtext2.txt/e3d23b43-5fbf-4fa9-8b29-b67e9075ad76:
part.1

tree -a /var/lib/directpv/mnt
/var/lib/directpv/mnt
├── 668267e9-fdd2-46c9-b854-1ac1de70132f
│   ├── .FSUUID.668267e9-fdd2-46c9-b854-1ac1de70132f -> .
│   ├── .directpv
│   │   └── meta.info
│   └── pvc-214d8389-2b4c-48b6-b2d4-67fc413e78fd
│       └── data
│           ├── .minio.sys
│           │   ├── buckets
│           │   │   ├── .background-heal.json
│           │   │   │   └── xl.meta
│           │   │   ├── .bloomcycle.bin
│           │   │   │   └── xl.meta
│           │   │   ├── .usage-cache.bin
│           │   │   │   └── xl.meta
│           │   │   ├── .usage-cache.bin.bkp
│           │   │   │   └── xl.meta
│           │   │   ├── .usage.json
│           │   │   │   └── xl.meta
│           │   │   └── mybucket
│           │   │       ├── .metadata.bin
│           │   │       │   └── xl.meta
│           │   │       ├── .usage-cache.bin
│           │   │       │   └── xl.meta
│           │   │       └── .usage-cache.bin.bkp
│           │   │           └── xl.meta
│           │   ├── config
│           │   │   ├── config.json
│           │   │   │   └── xl.meta
│           │   │   └── iam
│           │   │       ├── format.json
│           │   │       │   └── xl.meta
│           │   │       └── sts
│           │   │           ├── 7MBQEE4A1RCSD8MXVICS
│           │   │           │   └── identity.json
│           │   │           │       └── xl.meta
│           │   │           └── 7MH3LVEE6GR4R51PRCOF
│           │   │               └── identity.json
│           │   │                   └── xl.meta
│           │   ├── format.json
│           │   ├── multipart
│           │   ├── pool.bin
│           │   │   └── xl.meta
│           │   └── tmp
│           │       ├── .trash
│           │       │   ├── 006fdcfb-80d6-4591-82c0-b961d5e988cd
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 049ee707-f322-4c15-869b-3467630c3547
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 04bd7c26-3182-43e7-b8a7-222b17b8fa48
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 0bd70081-bdd8-42e7-a9a0-c50629cdb913
│           │       │   ├── 100bf5c5-4d43-4acb-b5c5-c03d3d54c0cc
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 11f80acd-8e00-40ae-8f4d-c9a6bd3cc265
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 15414eab-7a3b-4a97-aec0-3bd425784ceb
│           │       │   ├── 21a98aae-b503-4b61-8e04-eb18d3ce840d
│           │       │   ├── 30b97719-61f9-43da-a401-cb8779bb0b9c
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 3f0adbe8-4184-43d2-a187-aedd294ceef3
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 4e77226a-4dbe-4569-9b0e-ce42f124be65
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 4f597774-3c8c-4ea4-aca8-129edd62b530
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 571d2a47-58fe-4113-a578-dbd82ce0b98a
│           │       │   ├── 58742c44-8d0a-4f1e-be39-1b045014c91b
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 5c0da458-5cff-4e8b-93f8-0e3efe388ac9
│           │       │   ├── 6721d170-fcd7-4198-ac26-655f791bf041
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 6b3e1d30-ad3c-4527-98d0-1a430c697674
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 6b477cbf-d77b-4e5b-b478-f28653abff4f
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 707151d5-45f0-4c96-8391-694589c55d91
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 872cbfdd-8d97-427c-b00b-3ecc2394ad7e
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 9471b23a-aace-4759-a894-45db581da583
│           │       │   ├── 948cd323-9cf9-4b1c-aeb8-f8dcefbd3472
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 9564c467-f1b4-4064-a1e6-45116771e308
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 969f27c5-1af3-4048-8520-06087235c86e
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 98d270a7-2bfb-4d2d-8995-0c6d84236b65
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 9d76c024-e1fa-4f45-9323-c779c00ae80a
│           │       │   ├── a17ed609-8165-42dd-9a0f-0f8004d5691d
│           │       │   ├── a54af509-fa05-417b-8961-982f47c53662
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── a5ee46a9-60e4-4cff-861e-cff3fc662cd8
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── cb1e96eb-1cb9-49b6-b8a0-179214287378
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── dcc6fe56-e62c-4d1a-8311-ca1349ae69a7
│           │       │   │   └── xl.meta.bkp
│           │       │   └── e93df4d6-cc56-4bde-822b-ee06d7f1153f
│           │       │       └── xl.meta.bkp
│           │       └── 0f3ba3ea-5262-4c76-804c-46cc321fedcb
│           └── mybucket
│               ├── hello.txt
│               │   └── xl.meta
│               ├── randtext.txt
│               │   ├── 373427ec-c35a-4cb4-ada6-cfff86ac1d2b
│               │   │   ├── part.1
│               │   │   ├── part.2
│               │   │   ├── part.3
│               │   │   ├── part.4
│               │   │   ├── part.5
│               │   │   ├── part.6
│               │   │   └── part.7
│               │   └── xl.meta
│               └── randtext2.txt
│                   ├── e3d23b43-5fbf-4fa9-8b29-b67e9075ad76
│                   │   └── part.1
│                   └── xl.meta
├── 94ca8715-bbb0-47ad-93f9-43af207e0240
│   ├── .FSUUID.94ca8715-bbb0-47ad-93f9-43af207e0240 -> .
│   ├── .directpv
│   │   └── meta.info
│   └── pvc-4a372c6f-6fe1-4ec8-8616-eba5e6b03b57
│       └── data
│           ├── .minio.sys
│           │   ├── buckets
│           │   │   ├── .background-heal.json
│           │   │   │   └── xl.meta
│           │   │   ├── .bloomcycle.bin
│           │   │   │   └── xl.meta
│           │   │   ├── .usage-cache.bin
│           │   │   │   └── xl.meta
│           │   │   ├── .usage-cache.bin.bkp
│           │   │   │   └── xl.meta
│           │   │   ├── .usage.json
│           │   │   │   └── xl.meta
│           │   │   └── mybucket
│           │   │       ├── .metadata.bin
│           │   │       │   └── xl.meta
│           │   │       ├── .usage-cache.bin
│           │   │       │   └── xl.meta
│           │   │       └── .usage-cache.bin.bkp
│           │   │           └── xl.meta
│           │   ├── config
│           │   │   ├── config.json
│           │   │   │   └── xl.meta
│           │   │   └── iam
│           │   │       ├── format.json
│           │   │       │   └── xl.meta
│           │   │       └── sts
│           │   │           ├── 7MBQEE4A1RCSD8MXVICS
│           │   │           │   └── identity.json
│           │   │           │       └── xl.meta
│           │   │           └── 7MH3LVEE6GR4R51PRCOF
│           │   │               └── identity.json
│           │   │                   └── xl.meta
│           │   ├── format.json
│           │   ├── multipart
│           │   ├── pool.bin
│           │   │   └── xl.meta
│           │   └── tmp
│           │       ├── .trash
│           │       │   ├── 00429d98-9994-4ef5-ad5d-da1af22deec8
│           │       │   ├── 00453b3f-9c16-472a-8d1b-a500092addd4
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 05cd70b9-fa7e-4b55-94ac-6ae7527182fd
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 078726d8-849c-4b2a-8061-345e9aad4b57
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 0828a598-2376-45ef-8ef0-13cca73cba1e
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 22a805bd-f4a6-42f5-819b-73116159991f
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 26e28304-1087-4c89-8dcf-6adc5c1f3998
│           │       │   ├── 2eba803e-e18d-4b85-aeeb-0c55c7aced90
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 368dce08-f056-453d-a09c-d56b9cd0512d
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 42095284-2f39-4ad9-899b-7aa026cd7fff
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 44de14b6-033a-46df-ba7c-4caf929913f8
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 4b64ab54-0233-48a1-9130-d71ad13dedfa
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 4f0dbb28-ed3b-49e8-b06c-79fe75639c8c
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 54fd0e5a-db30-4306-b106-691d5002babb
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 57b307cb-02e6-4b40-81c3-b097b657e437
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 5bf1d20a-ef64-4d46-9f3a-966b701c44ca
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 76d95cd5-5f69-45ce-87a5-2a0c155d7871
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 85661bc6-7c93-4a0f-99e1-320b441197d3
│           │       │   ├── 9076fbea-8b8d-4666-8185-1ac076f46b89
│           │       │   ├── beaaa1a1-2b83-433e-b86d-26d3f75f81df
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── c288aa6c-87a3-4227-9183-ea5964051d40
│           │       │   ├── c76aaa46-6fd9-4b50-a928-bc0396670df4
│           │       │   ├── d32eedb8-0f2a-4c85-97a7-1fc9db40c3d9
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── d451e679-5bfa-4b3c-89b6-29c19206f4f1
│           │       │   ├── dbaa1c58-3d47-4e12-9737-bfb56b6e6371
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── e09a53cb-2a1a-4889-bfa9-6d683998a02c
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── e5b6681e-cc28-4fbc-80cd-39a066d0d09b
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── e8f12744-4f79-464c-a7da-ec7245578ecf
│           │       │   ├── f34e5780-c827-4d02-8905-5634db2310e6
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── f433ed4e-450b-4aaa-9d90-49a04e2fe427
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── fbf234bc-a79b-4911-b831-7927c1538dae
│           │       │   │   └── xl.meta.bkp
│           │       │   └── ffce816c-a115-4765-b5ce-2b2174403a89
│           │       │       └── xl.meta.bkp
│           │       └── 49594e7d-93b3-4d93-9a9e-71fe2fe4793b
│           └── mybucket
│               ├── hello.txt
│               │   └── xl.meta
│               ├── randtext.txt
│               │   ├── 373427ec-c35a-4cb4-ada6-cfff86ac1d2b
│               │   │   ├── part.1
│               │   │   ├── part.2
│               │   │   ├── part.3
│               │   │   ├── part.4
│               │   │   ├── part.5
│               │   │   ├── part.6
│               │   │   └── part.7
│               │   └── xl.meta
│               └── randtext2.txt
│                   ├── e3d23b43-5fbf-4fa9-8b29-b67e9075ad76
│                   │   └── part.1
│                   └── xl.meta
├── 94dca655-fc05-4e5f-a7d8-a8b1e953f220
│   ├── .FSUUID.94dca655-fc05-4e5f-a7d8-a8b1e953f220 -> .
│   ├── .directpv
│   │   └── meta.info
│   └── pvc-6aa17a8c-3e37-4dbd-a837-32683b6d765f
│       └── data
│           ├── .minio.sys
│           │   ├── buckets
│           │   │   ├── .background-heal.json
│           │   │   │   └── xl.meta
│           │   │   ├── .bloomcycle.bin
│           │   │   │   └── xl.meta
│           │   │   ├── .usage-cache.bin
│           │   │   │   └── xl.meta
│           │   │   ├── .usage-cache.bin.bkp
│           │   │   │   └── xl.meta
│           │   │   ├── .usage.json
│           │   │   │   └── xl.meta
│           │   │   └── mybucket
│           │   │       ├── .metadata.bin
│           │   │       │   └── xl.meta
│           │   │       ├── .usage-cache.bin
│           │   │       │   └── xl.meta
│           │   │       └── .usage-cache.bin.bkp
│           │   │           └── xl.meta
│           │   ├── config
│           │   │   ├── config.json
│           │   │   │   └── xl.meta
│           │   │   └── iam
│           │   │       ├── format.json
│           │   │       │   └── xl.meta
│           │   │       └── sts
│           │   │           ├── 7MBQEE4A1RCSD8MXVICS
│           │   │           │   └── identity.json
│           │   │           │       └── xl.meta
│           │   │           └── 7MH3LVEE6GR4R51PRCOF
│           │   │               └── identity.json
│           │   │                   └── xl.meta
│           │   ├── format.json
│           │   ├── multipart
│           │   ├── pool.bin
│           │   │   └── xl.meta
│           │   └── tmp
│           │       ├── .trash
│           │       │   ├── 032027e7-b5a6-4064-a53b-02d228ec5f0a
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 034f62b2-744c-4806-b70b-a4f45a19f841
│           │       │   ├── 0949eea7-f714-4975-bab2-414db9b69f6e
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 0f6dd7c4-c48b-4874-aafd-60fc4a84e25a
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 12673790-7565-4fea-84d6-2af512f62f3d
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 1461ae9e-0295-45d4-8600-c417fa387a64
│           │       │   ├── 167dc3ef-3ce8-4e49-85b2-dcc91bbfa015
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 16cb95fc-55ae-4c13-b4f0-76164f2435e1
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 3872b7ac-4f03-49e0-99d5-5976e6a140ff
│           │       │   ├── 3cac4403-d128-4f98-9c6a-baed7e0070b0
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 4993e421-dc05-4ce3-9aed-965eb523387d
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 4f9b40de-af26-4641-83d0-d34c98cf0ad3
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 5483040f-7068-4eb6-911e-8db3aa07c8f1
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 64b2e147-6f85-4204-879f-1b94d7b3bd8b
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 711d1818-731c-4471-b133-e87da045c7e9
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 7508f792-391c-42d3-909f-71499f0ea2f1
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 80c3c43d-64f3-4ff4-b185-e19a60282dd6
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 84df7354-1928-48f8-a6bf-c2ec4698bf19
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 8ac6fef1-8a7e-43fd-805b-1db1c3947d9a
│           │       │   ├── 8ad30567-ef42-40bb-8063-51991b94d0d2
│           │       │   ├── 96f79219-ba78-45b9-80b8-acaad6e3be71
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── 97c1d642-b38a-4ea6-8094-fd7338f153c0
│           │       │   ├── b6cca107-478b-47ec-ba57-61b26709278f
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── bd37c07a-7240-4bdb-ad64-0019edabc3ff
│           │       │   ├── c05f3c4e-7b6f-4532-95fe-5b575665ebbc
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── ca92ad81-f8fc-423e-a03c-8bab94d7134e
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── cbc9252d-0511-43cf-b32d-52ca00850a54
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── d3c710bc-98d0-4902-b3bd-a5f4b2df85d7
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── d54eff25-5850-4b9b-b5cd-09ba3d902b60
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── ec701ed0-6701-4688-b2a4-fb858e90f299
│           │       │   │   └── xl.meta.bkp
│           │       │   ├── efbfa4d6-592c-4495-836e-5c1d968908e1
│           │       │   │   └── xl.meta.bkp
│           │       │   └── f675868f-1053-4683-acab-d784ff035146
│           │       └── 1fc71795-b86c-4d31-a636-64c9d2353ec7
│           └── mybucket
│               ├── hello.txt
│               │   └── xl.meta
│               ├── randtext.txt
│               │   ├── 373427ec-c35a-4cb4-ada6-cfff86ac1d2b
│               │   │   ├── part.1
│               │   │   ├── part.2
│               │   │   ├── part.3
│               │   │   ├── part.4
│               │   │   ├── part.5
│               │   │   ├── part.6
│               │   │   └── part.7
│               │   └── xl.meta
│               └── randtext2.txt
│                   ├── e3d23b43-5fbf-4fa9-8b29-b67e9075ad76
│                   │   └── part.1
│                   └── xl.meta
└── afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445
    ├── .FSUUID.afa0c1ac-a9a4-4cd5-8ea2-3d1cb51a9445 -> .
    ├── .directpv
    │   └── meta.info
    └── pvc-a6e3bbf9-ab29-41d8-ac3b-c1d8920e0d98
        └── data
            ├── .minio.sys
            │   ├── buckets
            │   │   ├── .background-heal.json
            │   │   │   └── xl.meta
            │   │   ├── .bloomcycle.bin
            │   │   │   └── xl.meta
            │   │   ├── .usage-cache.bin
            │   │   │   └── xl.meta
            │   │   ├── .usage-cache.bin.bkp
            │   │   │   └── xl.meta
            │   │   ├── .usage.json
            │   │   │   └── xl.meta
            │   │   └── mybucket
            │   │       ├── .metadata.bin
            │   │       │   └── xl.meta
            │   │       ├── .usage-cache.bin
            │   │       │   └── xl.meta
            │   │       └── .usage-cache.bin.bkp
            │   │           └── xl.meta
            │   ├── config
            │   │   ├── config.json
            │   │   │   └── xl.meta
            │   │   └── iam
            │   │       ├── format.json
            │   │       │   └── xl.meta
            │   │       └── sts
            │   │           ├── 7MBQEE4A1RCSD8MXVICS
            │   │           │   └── identity.json
            │   │           │       └── xl.meta
            │   │           └── 7MH3LVEE6GR4R51PRCOF
            │   │               └── identity.json
            │   │                   └── xl.meta
            │   ├── format.json
            │   ├── multipart
            │   ├── pool.bin
            │   │   └── xl.meta
            │   └── tmp
            │       ├── .trash
            │       │   ├── 0304523e-8f2e-4835-bb34-0aca8df3e9a9
            │       │   ├── 03739d19-d53a-4372-8933-8351e5711aa5
            │       │   │   └── xl.meta.bkp
            │       │   ├── 1055d635-0916-43e9-8e26-7e474ab6cd1a
            │       │   │   └── xl.meta.bkp
            │       │   ├── 14d29fdb-ef2a-445e-976c-648a0762c066
            │       │   │   └── xl.meta.bkp
            │       │   ├── 1ba4f3a7-be5e-463f-b64e-585b94beec85
            │       │   ├── 1deda179-282f-4b3d-9612-f942fed452b2
            │       │   │   └── xl.meta.bkp
            │       │   ├── 1f99bf28-678c-4b5a-965b-8eed3dbf1bb7
            │       │   │   └── xl.meta.bkp
            │       │   ├── 22bb00bd-cc1b-41a1-b60c-14cc8b5d4e91
            │       │   │   └── xl.meta.bkp
            │       │   ├── 246695ca-8a7a-4253-87e9-3ecba6a3a7e4
            │       │   │   └── xl.meta.bkp
            │       │   ├── 2d197490-bd5d-4fbe-9268-a1539156dc26
            │       │   │   └── xl.meta.bkp
            │       │   ├── 351180bb-c1b5-4a15-8641-761e7c1ba6e6
            │       │   │   └── xl.meta.bkp
            │       │   ├── 3eb3ef1e-2d76-45c2-9997-0cb122faab64
            │       │   ├── 4af1135a-d470-423d-ac00-b75279120b83
            │       │   │   └── xl.meta.bkp
            │       │   ├── 518242f7-6b63-4022-beb3-d0f86fac9c2e
            │       │   │   └── xl.meta.bkp
            │       │   ├── 5af48e46-f887-4259-8a99-33fef0ad6915
            │       │   │   └── xl.meta.bkp
            │       │   ├── 68dee719-884d-4290-af20-ab0f38d7cb43
            │       │   ├── 6a149cd9-8583-41cf-abc2-89d08376525c
            │       │   │   └── xl.meta.bkp
            │       │   ├── 7047e8f5-e137-46cc-8ac3-12e223813b6a
            │       │   ├── 8d53a24e-cba6-416d-892b-a5902e0992fd
            │       │   │   └── xl.meta.bkp
            │       │   ├── 9c630f25-19bb-4d6e-9803-1570a5d46780
            │       │   ├── 9f901e89-f1fe-4a5e-827d-038042d40e02
            │       │   │   └── xl.meta.bkp
            │       │   ├── a251fadf-5cf0-4b89-b5d7-934feeea89bd
            │       │   ├── b11d73f3-0352-434e-9944-77f0d9b469e7
            │       │   │   └── xl.meta.bkp
            │       │   ├── c6766137-3249-45f2-81f9-2782603ddb8d
            │       │   │   └── xl.meta.bkp
            │       │   ├── ceb2aa42-f25f-45f6-8e05-648b472773ac
            │       │   │   └── xl.meta.bkp
            │       │   ├── d65ba5b3-ec17-40a7-970c-e57e2f75a6ff
            │       │   │   └── xl.meta.bkp
            │       │   ├── d6dd1d39-dd96-4887-83d7-b1c2b41739ba
            │       │   │   └── xl.meta.bkp
            │       │   ├── d962df64-4be9-4a07-bf22-b5bd3522b91b
            │       │   │   └── xl.meta.bkp
            │       │   ├── e6fd6a44-9636-4f99-9bb8-1266f37baab9
            │       │   │   └── xl.meta.bkp
            │       │   ├── ee036975-a68f-4807-9791-4f0d3e0559d0
            │       │   │   └── xl.meta.bkp
            │       │   ├── fbddbf16-901c-4189-a680-c5dcf7f7f27c
            │       │   │   └── xl.meta.bkp
            │       │   └── fc9cf036-bc12-4128-a3c1-ca886f151cf7
            │       └── 4eca5a1c-f617-41dc-a50e-47b18b54bf43
            └── mybucket
                ├── hello.txt
                │   └── xl.meta
                ├── randtext.txt
                │   ├── 373427ec-c35a-4cb4-ada6-cfff86ac1d2b
                │   │   ├── part.1
                │   │   ├── part.2
                │   │   ├── part.3
                │   │   ├── part.4
                │   │   ├── part.5
                │   │   ├── part.6
                │   │   └── part.7
                │   └── xl.meta
                └── randtext2.txt
                    ├── e3d23b43-5fbf-4fa9-8b29-b67e9075ad76
                    │   └── part.1
                    └── xl.meta

241 directories, 232 files

# rm - https://docs.min.io/community/minio-object-store/reference/minio-mc/mc-rm.html
mc rm k8s-tenant1/mybucket/randtext2.txt --insecure
Removed `k8s-tenant1/mybucket/randtext2.txt`.

mc cp k8s-tenant1/mybucket/randtext.txt ./randtext3.txt  --insecure
.../mybucket/randtext.txt: 100.00 MiB / 100.00 MiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 569.60 MiB/s 0s

(⎈|default:N/A) root@k3s-s:~#

실습 환경 삭제

aws cloudformation delete-stack --stack-name miniolab

'Kubernetes' 카테고리의 다른 글

[K8S Deploy 2주차] Ansible 기초 (0)	2026.01.12
[K8S Deploy 1주차] Kubernetes 손 설치 (1)	2026.01.10
MinIO 소개 (0)	2025.09.13
Cilium Study [1기] (8주차) - Cilium Security (0)	2025.09.04
Cilium Study [1기] (7주차) - Jmeter를 이용한 K8S 부하테스트 (0)	2025.08.29

현재글MinIO - DirectPV & Performance

yu3papa 님의 블로그

yu3papa 님의 블로그 입니다.

Today :
Yesterday :

yu3papa 님의 블로그