Content Comparison

...

CNI 종류 및 구성방식에 따라 트래픽 전송 방식에 차이가 있습니다.
어떤 식으로 트래픽이 전달되는지 확인하는 방법을 설명합니다.

sung-ubuntu01 - Control Plane #1

sung-ubuntu02 - Control Plane #2

sung-ubuntu03 - Control Plane #3

sung-ubuntu04 - Worker Node #1

sung-ubuntu05 - Worker Node #2

192.168.1.10 cp-k8s
192.168.1.101 w1-k8s
192.168.1.102 w2-k8s
192.168.1.103 w3-k8s

3.1 파드 네크워크

...

도커에서 POD 네트워크 설명할 경우와 인터페이스 이름이 다릅니다.
도커의 역할

...

파드 네트워크 구성에 대해 알아봅니다.

코드 블럭

root@sung-ubuntu01:~/tmp#root@cp-k8s# kubectl get pod -o wide
NAME          READY   STATUS    RESTARTS   AGE     IP            NODE            NOMINATED NODE   READINESS GATES
ubuntu-test   1/1     Running   0          6m20s150m   10172.23316.99103.1134   sungw2-ubuntu04k8s   <none>           <none>


###POD 접속
root@sung-ubuntu01:~/tmp#root@cp-k8s## kubectl exec -it ubuntu-test -- bash
# apt update
# apt install -y net-tools iputils-ping

root@ubuntu-test:/# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=54 time=39.6 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=54 time=38.1 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=54 time=38.8 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=54 time=39.3 ms

**# 컨테이너의 네트워크 인터페이스 확인**
# root@ubuntu-test:/# ifconfig -a
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1480
        inet 10172.23316.99103.1134  netmask 255.255.255.255  broadcast 0.0.0.0
        ether 0636:55d7:8404:5ab0:ac6a:6b0b  txqueuelen 01000  (Ethernet)
        RX packets 57189667  bytes 2402641632353517 (2432.03 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 36908976  bytes 250168492421 (250492.14 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        
#노드와 파드 사이를 연결하는 인터페이스입니다.
tunl0: flags=128<NOARP>  mtu 1480
        tunnel   txqueuelen 1000  (IPIP Tunnel)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0ons 0

* loopback ip : host 자신을 가리키는 논리적인 인터페이스
**#노드 네트워크 확인**
root@sung-ubuntu04:~# ifconfig -a
...
tunl0: flags=193<UP,RUNNING,NOARP>  mtu 1480
        inet 10172.23316.99103.0128  netmask 255.255.255.255
        tunnel   txqueuelen 1000  (IPIP Tunnel)
        RX packets 60111  bytes 852815872 (15.8.5 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 6691  bytes 447614596 (414.45 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

*mtu 1480인 이유?
IPIP 터널을 사용하고 있기 때문에, 캡슐화된 패킷의 크기는 원래 패킷보다 더 크기 때문에 MTU 조절이 필요하다
1480인 이유는 캡슐화된 패킷이 Ethernet 패킷에 포함될 때 전체 크기가 1500을 초과하지 않도록 하기 위해서이다.

...

코드 블럭

apiVersion: v1
kind: Pod
metadata:
  name: multi-container
spec:
  containers:
  - name: ubuntu
    image: ubuntu:20.04
    command: ["/bin/sleep", "3650d"]
    imagePullPolicy: IfNotPresent
    volumeMounts:
    - mountPath: /cache
      name: cache-volume
  - name: nginx
    image: nginx
    ports:
    - containerPort: 80
  volumes:
  - name: cache-volume
    emptyDir: {}
  restartPolicy: Always
  dnsConfig:
    nameservers:
      - 8.8.8.8

root@sungroot@cp-ubuntu01k8s:~/tmp#~# kubectlk get pod -o wide
NAME                 READY   STATUS    RESTARTS   AGE   IP        READY   STATUS NODE   RESTARTS       AGE  NOMINATED NODE IP  READINESS GATES multi-container   2/2     Running   0NODE     NOMINATED NODE   READINESS 25mGATES
multi-container  10.233.78.3   sung-ubuntu05   <none>           <none> ubuntu-test       12/12     Running   0          57m    15m    10172.23316.99132.16     sungw3-ubuntu04k8s   <none>           <none>

컨테이너 내부에서 네트워크 흐름을 알아봅니다.

코드 블럭

**#ubuntu 컨테이너 접속**
root@sung-ubuntu01:~/tmp# kubectl exec -it multi-container -c ubuntu -- bash

### POD 안에서
# apt update
# apt install -y net-tools iputils-ping

root@multi-container:/# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1480
        inet 10.233.78.3  netmask 255.255.255.255  broadcast 0.0.0.0
nfs-client-provisioner-5cf87f6995-vg6fq   1/1     Running   28 (10m ago)   7h4m   172.16.221.133   w1-k8s   <none>           <none>
ubuntu-test                    ether ce:de:b3:90:c1:a7  txqueuelen 0  (Ethernet)     1/1    RX packetsRunning 5206  bytes0 23989810 (23.9 MB)         RX errors 036m  dropped 0 172.16.103.134 overruns 0 w2-k8s frame 0 <none>        TX packets 3160  bytes 213900 (213.9 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536 <none>

컨테이너 내부에서 네트워크 흐름을 알아봅니다.

코드 블럭

**#ubuntu 컨테이너 접속**
root@cp-k8s:~# kubectl exec -it multi-container -c ubuntu -- bash

### POD 안에서
# apt update
# apt install -y net-tools iputils-ping

root@multi-container:/# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1480
        inet 127172.016.0132.16  netmask 255.255.255.255  broadcast 0.0.0.0
         loopether da:6c:e7:bc:8b:f9  txqueuelen 1000  (Local LoopbackEthernet)
        RX packets 04920  bytes 041607447 (041.06 BMB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 03877  bytes 0217509 (0217.05 BKB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

**# nginx 컨테이너 접속**
root@sung-ubuntu01:~/tmp# kubectl exec -it multi-container -c nginx -- bash

### POD 안에서 실행
# apt update
# apt install -y net-tools iputils-ping

root@multi-container:/# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1480
        inet 10.233.78.3  netmask 255.255.255.255  broadcast 0.0.0.0
        ether ce:de:b3:90:c1:a7  txqueuelen 0  (Ethernetlo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 62870  bytes 330130140 (310.40 MiBB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 39350  bytes 2675910 (2610.30 KiBB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1**# nginx 컨테이너 접속**
root@cp-k8s:~# kubectl exec -it multi-container -c nginx -- bash

### POD 안에서 실행
# apt update
# apt install -y net-tools iputils-ping

root@multi-container:/# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1480
        inet 172.16.132.6  netmask 255.255.255.255  broadcast 0.0.0.0
         loopether da:6c:e7:bc:8b:f9  txqueuelen 1000  (Local LoopbackEthernet)
        RX packets 0590  bytes 09590376 (09.01 BMiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0355  bytes 021258 (020.07 BKiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

노드 네트워크 확인

코드 블럭

root@sung-ubuntu05:~#
ifconfig
-a
calib4cfe5eb958lo: flags=4163<UP73<UP,BROADCASTLOOPBACK,RUNNING,MULTICAST>RUNNING>  mtu 148065536
        inet6 fe80::ecee:eeff:feee:eeee  prefixlen 64  scopeid 0x20<link>inet 127.0.0.1  netmask 255.0.0.0
          loop ether ee:ee:ee:ee:ee:ee  txqueuelen 01000  (EthernetLocal Loopback)
        RX packets 39350  bytes 2675910 (2670.50 KBB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 62870  bytes 330130140 (330.0 MBB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

노드 네트워크 확인

코드 블럭

root@w3-k8s:~# ifconfig -a
docker0cali9794446aa53: flags=4099<UP4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 15001480
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:6a:17:c5:80ether ee:ee:ee:ee:ee:ee  txqueuelen 01000  (Ethernet)
        RX packets 034654  bytes 05118573 (05.01 BMB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 041396  bytes 09084184 (09.0 BMB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens3calib4cfe5eb958: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 15001480
        inet 192.168.110.104  netmask 255.255.0.0  broadcast 192.168.255.255ether ee:ee:ee:ee:ee:ee  txqueuelen 1000  (Ethernet)
        RX inet6 fe80::f816:3eff:fe54:bc4packets 3885  prefixlenbytes 64218205  scopeid 0x20<link>
        ether fa:16:3e:54:0b:c4  txqueuelen 1000  (Ethernet)
        RX packets 1353299  bytes 1304887824 (1.3 GB(218.2 KB)
        RX errors 0  dropped 886030  overruns 0  frame 0
        TX packets 1912065122  bytes 2078935041619671 (2041.76 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

kube-ipvs0eth0: flags=130<BROADCAST,NOARP>4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.2330.02.115  netmask 255.255.255.2550  broadcast 010.0.02.0255
        ether 6608:2d00:b300:6c00:5000:9a00  txqueuelen 01000  (Ethernet)
        RX packets 0196682  bytes 0287889664 (0287.08 BMB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 013578  bytes 01005482 (01.0 BMB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

loeth1: flags=73<UP4163<UP,LOOPBACKBROADCAST,RUNNING,RUNNING>MULTICAST>  mtu 655361500
        inet 127192.0168.01.1103  netmask 255.0255.0255.0         inet6 ::1  prefixlen 128  scopeid 0x10<host>broadcast 192.168.1.255
        ether 08:00:27:3f:6d:d5     loop  txqueuelen 1000  (Local LoopbackEthernet)
        RX packets 162061189996  bytes 2229821154057708 (2254.20 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 162061171448  bytes 2229821122285772 (22.2 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

nodelocaldnslo: flags=130<BROADCAST73<UP,LOOPBACK,NOARP>RUNNING>  mtu 150065536
        inet 169127.2540.250.101  netmask 255.255.255.255  broadcast 0.0.0.0
        ether 16:84:53:46:fe:65 loop  txqueuelen 01000  (EthernetLocal Loopback)
        RX packets 0246745  bytes 067681709 (067.06 BMB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0246745  bytes 067681709 (067.06 BMB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

tunl0: flags=193<UP,RUNNING,NOARP>  mtu 1480
  #터널 인터페이스
        inet 10172.23316.78132.0  netmask 255.255.255.255
        tunnel   txqueuelen 1000  (IPIP Tunnel)
        RX packets 69129  bytes 938024080 (924.30 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 76137  bytes 512524502 (24.5.1 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

multi-container 관련 컨테이너 확인

코드 블럭

root@sungroot@w3-ubuntu05k8s:~# dockercrictl ps | grep multi-container
64c1938850a2 7e77cac0a9ec8       e784f4560448b  nginx     20 minutes ago      Running     "/docker-entrypoint.…"   26 minutes ago   Upnginx 25 minutes             k8s_nginx_multi-container_default_1d0e0776-18b1-4c7f-b05f-20b8c54fb230_0
b4c4045ac7770              ubuntu     2c8ad4a63a674       multi-container
2e4672f196a4e     "/bin/sleep 3650d"  2abc4dfd83182       2620 minutes ago    Up 26 minutesRunning             k8s_ubuntu_multi-container_default_1d0e0776-18b1-4c7f-b05f-20b8c54fb230_0
1eaedb9c9d55   k8s.gcr.io/pause:3.5   "/pause"ubuntu              0                   2c8ad4a63a674       multi-container

root@w3-k8s:~# crictl ps
CONTAINER           IMAGE               CREATED             STATE               NAME         27 minutes ago   Up 26 minutesATTEMPT             k8s_POD_multi-container_default_1d0e0776-18b1-4c7f-b05f-20b8c54fb230_0

...

Pause Container

...

파드 내부의 container들을 위한 일종의 '부모 container' 로서의 역할을 수행합니다.

...

파드가 실행될 때 Pause Container가 먼저 실행되고 Pause Container의 리눅스 네임스페이스를 파드 내부의 모든 컨테이너들이 상속받아서 사용합니다.

...

POD ID              POD
7e77cac0a9ec8       e784f4560448b       20 minutes ago      Running             nginx               0                   2c8ad4a63a674       multi-container
2e4672f196a4e       2abc4dfd83182       20 minutes ago      Running             ubuntu              0                   2c8ad4a63a674       multi-container
3041e3871fce9       825aff16c20cc       3 hours ago         Running             controller          0                   f13095e35bd52       ingress-nginx-controller-568fb54f96-mr8p4
ab79b29f18c4d       44f52c09decec       6 hours ago         Running             calico-node         1                   69f307f64ef92       calico-node-9dmp8
6a36a93a5e2fc       738c5d221d601       6 hours ago         Running             speaker             2                   1b868a54d4c49       speaker-6hc8s
e1450ce254e91       2019bbea5542a       6 hours ago         Running             kube-proxy          1                   424a7426434a9       kube-proxy-wwdpz

정보

리눅스 namespace

리눅스 커널에서 제공 기능으로 Container의 기반이 되는 기술. 하나의 시스템에서 프로세스를 격리시킬 수 있는 가상화 기술입니다.

3.4 파드 간 통신

Image Removed

Pod 간 route 경로 확인

...

Image Added

Pod 간 route 경로 확인

코드 블럭

root@cp-k8s:~# kubectl get pod -o wide
NAME                                      READY   STATUS    RESTARTS       AGE     IP               NODE     NOMINATED NODE   READINESS GATES
multi-container                           2/2    READY Running  STATUS 0   RESTARTS   AGE   IP     23m     172.16.132.6  NODE   w3-k8s   <none>      NOMINATED NODE   READINESS GATES
multi-container<none>
nfs-client-provisioner-5cf87f6995-vg6fq   21/21     Running    028 (18m ago)   7h13m   172.16.221.133   w1-k8s   <none>           <none>
25mubuntu-test   10.233.78.3      sung-ubuntu05   <none>           <none> ubuntu-test       1/1     Running   0              57m45m     10172.23316.99103.1134   sungw2-ubuntu04k8s   <none>           <none>
;

#ubuntu-test 
root@ubuntu-test:/# apt install traceroute

root@ubuntu-test:/# traceroute 10172.23316.78132.36
traceroute to 10172.23316.78132.36 (10172.23316.78132.36), 30 hops max, 60 byte packets
 1  19210.1680.1102.10315 (19210.1680.1102.10315)  01.202308 ms  0.032025 ms  0.028020 ms 
#sung-ubuntu04 ens3
 2  10172.23316.78132.0 (10172.23316.78132.0)  12.169099 ms  03.990995 ms  03.928182 ms   
      #sung-ubuntu05 tunl0
 3  10172.23316.78132.36 (10172.23316.78132.36)  12.096967 ms  16.111487 ms  15.087188 ms          #multi-container
IP

노드 route table 확인

코드 블럭

root@sungroot@w2-ubuntu04k8s:~# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         _gateway10.0.2.2        0.0.0.0         UG    100    0        0 ens3eth0
10.2330.782.0     sung-ubuntu05.c 255.255.255   0.0.0.0   UG    0  255.255.255.0   U 0    100    0 tunl0 10.233.91.0     sung-ubuntu02.c 255.255.255.0   UG    0      0eth0
10.0.2.2         0 tunl0
10.2330.950.0        sung-ubuntu01.c 255.255.255.0255 UH  UG    0100      0        0 tunl0eth0
10.2330.99.02.3        0.0.0.0         255.255.255.0255 UH  U     0  100    0        0 *eth0
10172.23316.99103.1128     0.0.0.0         255.255.255.255192 U UH    0      0        0 calie3df4d89b13*
10172.23316.99.2   103.134  0.0.0.0         255.255.255.255 UH    0      0        0 calia85a668c715calie3df4d89b13
10172.23316.112132.0    sung-ubuntu03.cw3-k8s          255.255.255.0192   UG    0      0        0 tunl0
169172.25416.169.254 192.168.51.110196.128  cp-k8s          255.255.255.255 UGH   100192 UG    0        0 ens3 172.17.0.0      0 tunl0
172.016.0.0221.128  w1-k8s          255.255.0255.0192 UG    U     0      0        0 docker0tunl0
192.168.01.0     0.0.0.0         255.255.0255.0     U     0      0        0 ens3eth1


root@sungroot@w2-ubuntu04k8s:~# cat /etc/hosts
127.0.0.1 # Ansible inventory hosts BEGIN
localhost
192.168.1101.10010 sung-ubuntu01.cluster.local sung-ubuntu01
cp-k8s
192.168.1101.101 sung-ubuntu02.cluster.local sung-ubuntu02w1-k8s
192.168.1101.102 sung-ubuntu03.cluster.local sung-ubuntu03w2-k8s
192.168.1101.103 sung-ubuntu04.cluster.local sung-ubuntu04
192.168.110.104 sung-ubuntu05.cluster.local sung-ubuntu05w3-k8s

참고

https://kubernetes.io/ko/docs/concepts/cluster-administration/networking/

...

버전	이전 버전 30	새 버전 31
변경한 사람	김호진	김호진
에 저장	2024-05-25	2024-05-25

Content Comparison

버전 비교

키

3.1 파드 네크워크

3.4 파드 간 통신