Too many open files debug in k8s

ulimit -n

Output : 1048576

kubelet limit

systemctl show kubelet | grep NOFILE

Output :

LimitNOFILE=524288
LimitNOFILESoft=1024

systemd global

systemctl show --property=DefaultLimitNOFILE

Output: DefaultLimitNOFILE=524288

container runtime

systemctl show containerd | grep NOFILE

Output:

LimitNOFILE=infinity
LimitNOFILESoft=infinity

Socket summary

ss -s

Total: 795
TCP:   2101 (estab 22, closed 2066, orphaned 0, timewait 20)

Transport Total     IP        IPv6
RAW       1         0         1        
UDP       8         5         3        
TCP       35        27        8        
INET      44        32        12       
FRAG      0         0         0        

System-wide (all processes)

cat /proc/sys/fs/file-nr Output:

17784   0       2097152

Kubelet FD usage

pidof kubelet ls /proc/$(pidof kubelet)/fd | wc -l

Output: 1953

Edit Container runtime

sudo systemctl edit containerd

Per-pod socket usage (ADVANCED)

crictl ps
crictl inspect <container_id> | grep pid

Correct fix for kubelet (DO THIS)

sudo systemctl edit kubelet

  [Service]
 LimitNOFILE=524288

sudo systemctl daemon-reload
sudo systemctl restart kubelet