容器網絡其實并不難

使用容器總是感覺像使用魔法一樣。對于那些理解底層原理的人來說容器很好用，但是對于不理解的人來說就是個噩夢。很幸運的是，我們已經研究容器技術很久了，甚至成功揭秘容器只是隔離并受限的Linux進程，運行容器并不需要鏡像，以及另一個方面，構建鏡像需要運行一些容器。

現在是時候解決容器網絡問題了。或者更準確地說，單主機容器網絡問題。本文會回答這些問題：

如何虛擬化網絡資源，讓容器認為自己擁有獨占網絡？

如何虛擬化網絡資源，讓容器認為自己擁有獨占網絡？

如何讓容器們和平共處，之間不會互相干擾，并且能夠互相通信？

從容器內部如何訪問外部世界（比如，互聯網）？

從外部世界如何訪問某臺機器上的容器呢（比如，端口發布）？

網絡命名空間（namespace）

虛擬Ethernet設備（veth）

虛擬網絡交換機（網橋）

IP路由和網絡地址翻譯（NAT）

$?vagrant?init?centos/8

$?vagrant?up

$?vagrant?ssh

[vagrant@localhost?~]$?uname?-a

Linux?localhost.localdomain?4.18.0-147.3.1.el8_1.x86_64

#!/usr/bin/env?bash

echo??">?Network?devices"

ip?link

echo?-e?"\n>?Route?table"

ip?route

echo?-e?"\n>?Iptables?rules"

iptables?--list-rules

$?sudo?iptables?-N?ROOT_NS

$?sudo?./inspect-net-stack.sh

>?Network?devices

1:?lo:??mtu?65536?qdisc?noqueue?state?UNKNOWN?mode?DEFAULT?group?default?qlen?1000?link/loopback?00:00:00:00:00:00?brd?00:00:00:00:00:00

2:?eth0:??mtu?1500?qdisc?fq_codel?state?UP?mode?DEFAULT?group?default?qlen?1000?link/ether?52:54:00:e3:27:77?brd?ff:ff:ff:ff:ff:ff

>?Route?table

default?via?10.0.2.2?dev?eth0?proto?dhcp?metric?100

10.0.2.0/24?dev?eth0?proto?kernel?scope?link?src?10.0.2.15?metric?100

>?Iptables?rules

-P?INPUT?ACCEPT

-P?FORWARD?ACCEPT

-P?OUTPUT?ACCEPT

-N?ROOT_NS

$?sudo?ip?netns?add?netns0

$?ip?netns

netns0

$?sudo?nsenter?--net=/var/run/netns/netns0?bash

#?新建的bash進程在netns0里

$?sudo?./inspect-net-stack.sh

>?Network?devices?1:?lo:??mtu?65536?qdisc?noop?state?DOWN?mode?DEFAULT?group?default?qlen?1000

link/loopback?00:00:00:00:00:00?brd?00:00:00:00:00:00

>?Route?table

>?Iptables?rules

-P?INPUT?ACCEPT

-P?FORWARD?ACCEPT

-P?OUTPUT?ACCEPT

$?sudo?ip?link?add?veth0?type?veth?peer?name?ceth0

$?ip?link

1:?lo:??mtu?65536?qdisc?noqueue?state?UNKNOWN?mode?DEFAULT?group?default?qlen?1000

link/loopback?00:00:00:00:00:00?brd?00:00:00:00:00:00

2:?eth0:??mtu?1500?qdisc?fq_codel?state?UP?mode?DEFAULT?group?default?qlen?1000

link/ether?52:54:00:e3:27:77?brd?ff:ff:ff:ff:ff:ff

5:?ceth0@veth0:??mtu?1500?qdisc?noop?state?DOWN?mode?DEFAULT?group?default?qlen?1000

link/ether?66:2d:24:e3:49:3f?brd?ff:ff:ff:ff:ff:ff

6:?veth0@ceth0:??mtu?1500?qdisc?noop?state?DOWN?mode?DEFAULT?group?default?qlen?1000

link/ether?96:e8:de:1d:22:e0?brd?ff:ff:ff:ff:ff:ff

$?sudo?ip?link?set?ceth0?netns?netns0

#?列出所有設備，可以看到ceth0已經從root棧里消失了

$?ip?link?1:?lo:??mtu?65536?qdisc?noqueue?state?UNKNOWN?mode?DEFAULT?group?default?qlen?1000

link/loopback?00:00:00:00:00:00?brd?00:00:00:00:00:00

2:?eth0:??mtu?1500?qdisc?fq_codel?state?UP?mode?DEFAULT?group?default?qlen?1000

link/ether?52:54:00:e3:27:77?brd?ff:ff:ff:ff:ff:ff

6:?veth0@if5:??mtu?1500?qdisc?noop?state?DOWN?mode?DEFAULT?group?default?qlen?1000

link/ether?96:e8:de:1d:22:e0?brd?ff:ff:ff:ff:ff:ff?link-netns?netns0

$?sudo?ip?link?set?veth0?up

$?sudo?ip?addr?add?172.18.0.11/16?dev?veth0

$?sudo?nsenter?--net=/var/run/netns/netns0

$?ip?link?set?lo?up

$?ip?link?set?ceth0?up

$?ip?addr?add?172.18.0.10/16?dev?ceth0

$?ip?link

1:?lo:??mtu?65536?qdisc?noqueue?state?UNKNOWN?mode?DEFAULT?group?default?qlen?1000

link/loopback?00:00:00:00:00:00?brd?00:00:00:00:00:00

5:?ceth0@if6:??mtu?1500?qdisc?noqueue?state?UP?mode?DEFAULT?group?default?qlen?1000

link/ether 66:2d:24:e3:49:3f brd ff:ff:ff:ff:ff:ff link-netnsid 0

#?在netns0里ping?root的?veth0

$?ping?-c?2?172.18.0.11

PING?172.18.0.11?(172.18.0.11)?56(84)?bytes?of?data.

64?bytes?from?172.18.0.11:?icmp_seq=1?ttl=64?time=0.038?ms

64?bytes?from?172.18.0.11:?icmp_seq=2?ttl=64?time=0.040?ms

---?172.18.0.11?ping?statistics?---

2?packets?transmitted,?2?received,?0%?packet?loss,?time?58ms

rtt?min/avg/max/mdev?=?0.038/0.039/0.040/0.001?ms

#?離開?netns0

$?exit

#?在root命名空間里ping?ceth0

$?ping?-c?2?172.18.0.10

PING?172.18.0.10?(172.18.0.10)?56(84)?bytes?of?data.

64?bytes?from?172.18.0.10:?icmp_seq=1?ttl=64?time=0.073?ms

64?bytes?from?172.18.0.10:?icmp_seq=2?ttl=64?time=0.046?ms

---?172.18.0.10?ping?statistics?---

2?packets?transmitted,?2?received,?0%?packet?loss,?time?3ms

rtt?min/avg/max/mdev?=?0.046/0.059/0.073/0.015?ms

#?在?root?命名空間

$?ip?addr?show?dev?eth0

2:?eth0:??mtu?1500?qdisc?fq_codel?state?UP?group?default?qlen?1000

link/ether?52:54:00:e3:27:77?brd?ff:ff:ff:ff:ff:ff

inet?10.0.2.15/24?brd?10.0.2.255?scope?global?dynamic?noprefixroute?eth0

valid_lft?84057sec?preferred_lft?84057sec

inet6?fe80::5054:ff:fee3:2777/64?scope?link

valid_lft?forever?preferred_lft?forever

#?記住這里IP是10.0.2.15

$?sudo?nsenter?--net=/var/run/netns/netns0

#?嘗試ping主機的eth0

$?ping?10.0.2.15

connect:?Network?is?unreachable

#?嘗試連接外網

$?ping?8.8.8.8

connect:?Network?is?unreachable

#?在netns0命名空間:

$?ip?route

172.18.0.0/16?dev?ceth0?proto?kernel?scope?link?src?172.18.0.10

#?從?root?命名空間

$?sudo?ip?netns?add?netns1

$?sudo?ip?link?add?veth1?type?veth?peer?name?ceth1

$?sudo?ip?link?set?ceth1?netns?netns1

$?sudo?ip?link?set?veth1?up

$?sudo?ip?addr?add?172.18.0.21/16?dev?veth1

$?sudo?nsenter?--net=/var/run/netns/netns1

$?ip?link?set?lo?up

$?ip?link?set?ceth1?up

$?ip?addr?add?172.18.0.20/16?dev?ceth1

#?從netns1無法連通root?命名空間!

$?ping?-c?2?172.18.0.21

PING?172.18.0.21?(172.18.0.21)?56(84)?bytes?of?data.

From?172.18.0.20?icmp_seq=1?Destination?Host?Unreachable

From?172.18.0.20?icmp_seq=2?Destination?Host?Unreachable

---?172.18.0.21?ping?statistics?---

2?packets?transmitted,?0?received,?+2?errors,?100%?packet?loss,?time?55ms?pipe?2

#?但是路由是存在的!

$?ip?route

172.18.0.0/16?dev?ceth1?proto?kernel?scope?link?src?172.18.0.20

#?離開?`netns1`

$?exit

#?從?root?命名空間無法連通`netns1`

$?ping?-c?2?172.18.0.20

PING?172.18.0.20?(172.18.0.20)?56(84)?bytes?of?data.

From?172.18.0.11?icmp_seq=1?Destination?Host?Unreachable

From?172.18.0.11?icmp_seq=2?Destination?Host?Unreachable

---?172.18.0.20?ping?statistics?---

2?packets?transmitted,?0?received,?+2?errors,?100%?packet?loss,?time?23ms?pipe?2

#?從netns0可以連通`veth1`

$?sudo?nsenter?--net=/var/run/netns/netns0

$?ping?-c?2?172.18.0.21

PING?172.18.0.21?(172.18.0.21)?56(84)?bytes?of?data.

64?bytes?from?172.18.0.21:?icmp_seq=1?ttl=64?time=0.037?ms

64?bytes?from?172.18.0.21:?icmp_seq=2?ttl=64?time=0.046?ms

---?172.18.0.21?ping?statistics?---

2?packets?transmitted,?2?received,?0%?packet?loss,?time?33ms

rtt?min/avg/max/mdev?=?0.037/0.041/0.046/0.007?ms

#?但是仍然無法連通netns1

$?ping?-c?2?172.18.0.20

PING?172.18.0.20?(172.18.0.20)?56(84)?bytes?of?data.

From?172.18.0.10?icmp_seq=1?Destination?Host?Unreachable

From?172.18.0.10?icmp_seq=2?Destination?Host?Unreachable

---?172.18.0.20?ping?statistics?---

2?packets?transmitted,?0?received,?+2?errors,?100%?packet?loss,?time?63ms?pipe?2

$?ip?route

#?...?忽略無關行...?#

172.18.0.0/16?dev?veth0?proto?kernel?scope?link?src?172.18.0.11

172.18.0.0/16?dev?veth1?proto?kernel?scope?link?src?172.18.0.21

$?sudo?ip?netns?delete?netns0

$?sudo?ip?netns?delete?netns1

$?sudo?ip?link?delete?veth0

$?sudo?ip?link?delete?ceth0

$?sudo?ip?link?delete?veth1

$?sudo?ip?link?delete?ceth1

$?sudo?ip?netns?add?netns0

$?sudo?ip?link?add?veth0?type?veth?peer?name?ceth0

$?sudo?ip?link?set?veth0?up

$?sudo?ip?link?set?ceth0?netns?netns0

$?sudo?nsenter?--net=/var/run/netns/netns0

$?ip?link?set?lo?up

$?ip?link?set?ceth0?up

$?ip?addr?add?172.18.0.10/16?dev?ceth0

$?exit

$?sudo?ip?netns?add?netns1

$?sudo?ip?link?add?veth1?type?veth?peer?name?ceth1

$?sudo?ip?link?set?veth1?up

$?sudo?ip?link?set?ceth1?netns?netns1

$?sudo?nsenter?--net=/var/run/netns/netns1

$?ip?link?set?lo?up

$?ip?link?set?ceth1?up

$?ip?addr?add?172.18.0.20/16?dev?ceth1

$?exit

$?ip?route

default?via?10.0.2.2?dev?eth0?proto?dhcp?metric?100

10.0.2.0/24?dev?eth0?proto?kernel?scope?link?src?10.0.2.15?metric?100

$?sudo?ip?link?add?br0?type?bridge

$?sudo?ip?link?set?br0?up

$?sudo?ip?link?set?veth0?master?br0

$?sudo?ip?link?set?veth1?master?br0

$?sudo?nsenter?--net=/var/run/netns/netns0

$?ping?-c?2?172.18.0.20

PING?172.18.0.20?(172.18.0.20)?56(84)?bytes?of?data.

64?bytes?from?172.18.0.20:?icmp_seq=1?ttl=64?time=0.259?ms

64?bytes?from?172.18.0.20:?icmp_seq=2?ttl=64?time=0.051?ms

---?172.18.0.20?ping?statistics?---

2?packets?transmitted,?2?received,?0%?packet?loss,?time?2ms

rtt?min/avg/max/mdev?=?0.051/0.155/0.259/0.104?ms

$?sudo?nsenter?--net=/var/run/netns/netns1

$?ping?-c?2?172.18.0.10

PING?172.18.0.10?(172.18.0.10)?56(84)?bytes?of?data.

64?bytes?from?172.18.0.10:?icmp_seq=1?ttl=64?time=0.037?ms

64?bytes?from?172.18.0.10:?icmp_seq=2?ttl=64?time=0.089?ms

---?172.18.0.10?ping?statistics?---

2?packets?transmitted,?2?received,?0%?packet?loss,?time?36ms

rtt?min/avg/max/mdev?=?0.037/0.063/0.089/0.026?ms

$?sudo?nsenter?--net=/var/run/netns/netns0

$?ip?neigh

172.18.0.20?dev?ceth0?lladdr?6e:9c:ae:02:60:de?STALE

$?exit

$?sudo?nsenter?--net=/var/run/netns/netns1

$?ip?neigh

172.18.0.10?dev?ceth1?lladdr?66:f3:8c:75:09:29?STALE

$?exit

$?sudo?nsenter?--net=/var/run/netns/netns0

$?ping?10.0.2.15?#?eth0?address

connect:?Network?is?unreachable

$?ip?route

172.18.0.0/16?dev?ceth0?proto?kernel?scope?link?src?172.18.0.10

$?sudo?ip?addr?add?172.18.0.1/16?dev?br0

$?ip?route

#?...忽略無關行?...

172.18.0.0/16?dev?br0?proto?kernel?scope?link?src?172.18.0.1

$?ping?-c?2?172.18.0.10

PING?172.18.0.10?(172.18.0.10)?56(84)?bytes?of?data.

64?bytes?from?172.18.0.10:?icmp_seq=1?ttl=64?time=0.036?ms

64?bytes?from?172.18.0.10:?icmp_seq=2?ttl=64?time=0.049?ms

---?172.18.0.10?ping?statistics?---

2?packets?transmitted,?2?received,?0%?packet?loss,?time?11ms

rtt?min/avg/max/mdev?=?0.036/0.042/0.049/0.009?ms

$?ping?-c?2?172.18.0.20

PING?172.18.0.20?(172.18.0.20)?56(84)?bytes?of?data.

64?bytes?from?172.18.0.20:?icmp_seq=1?ttl=64?time=0.059?ms

64?bytes?from?172.18.0.20:?icmp_seq=2?ttl=64?time=0.056?ms

---?172.18.0.20?ping?statistics?---

2?packets?transmitted,?2?received,?0%?packet?loss,?time?4ms

rtt?min/avg/max/mdev?=?0.056/0.057/0.059/0.007?ms

$?sudo?nsenter?--net=/var/run/netns/netns0

$?ip?route?add?default?via?172.18.0.1

$?ping?-c?2?10.0.2.15

PING?10.0.2.15?(10.0.2.15)?56(84)?bytes?of?data.

64?bytes?from?10.0.2.15:?icmp_seq=1?ttl=64?time=0.036?ms

64?bytes?from?10.0.2.15:?icmp_seq=2?ttl=64?time=0.053?ms

---?10.0.2.15?ping?statistics?---

2?packets?transmitted,?2?received,?0%?packet?loss,?time?14ms

rtt?min/avg/max/mdev?=?0.036/0.044/0.053/0.010?ms

#?為`netns1`也做上述配置

#?在?root?命名空間

sudo?bash?-c?'echo?1?>?/proc/sys/net/ipv4/ip_forward'

$?sudo?nsenter?--net=/var/run/netns/netns0

$?ping?8.8.8.8

#?hung住了...

$?sudo?iptables?-t?nat?-A?POSTROUTING?-s?172.18.0.0/16?!?-o?br0?-j?MASQUERADE

sudo?iptables?-S

-P?INPUT?ACCEPT

-P?FORWARD?ACCEPT

-P?OUTPUT?ACCEPT

$?sudo?iptables?-t?filter?--list-rules

-P?INPUT?ACCEPT

-P?FORWARD?DROP

-P?OUTPUT?ACCEPT

-N?DOCKER

-N?DOCKER-ISOLATION-STAGE-1

-N?DOCKER-ISOLATION-STAGE-2

-N?DOCKER-USER

-A?FORWARD?-j?DOCKER-USER

-A?FORWARD?-j?DOCKER-ISOLATION-STAGE-1

-A?FORWARD?-o?docker0?-m?conntrack?--ctstate?RELATED,ESTABLISHED?-j?ACCEPT

-A?FORWARD?-o?docker0?-j?DOCKER

-A?FORWARD?-i?docker0?!?-o?docker0?-j?ACCEPT

-A?FORWARD?-i?docker0?-o?docker0?-j?ACCEPT

-A?DOCKER?-d?172.17.0.2/32?!?-i?docker0?-o?docker0?-p?tcp?-m?tcp?--dport?5000?-j?ACCEPT

-A?DOCKER-ISOLATION-STAGE-1?-i?docker0?!?-o?docker0?-j?DOCKER-ISOLATION-STAGE-2

-A?DOCKER-ISOLATION-STAGE-1?-j?RETURN

-A?DOCKER-ISOLATION-STAGE-2?-o?docker0?-j?DROP

-A?DOCKER-ISOLATION-STAGE-2?-j?RETURN

-A?DOCKER-USER?-j?RETURN

$?sudo?iptables?-t?nat?--list-rules

-P?PREROUTING?ACCEPT

-P?INPUT?ACCEPT

-P?POSTROUTING?ACCEPT

-P?OUTPUT?ACCEPT

-N?DOCKER

-A?PREROUTING?-m?addrtype?--dst-type?LOCAL?-j?DOCKER

-A?POSTROUTING?-s?172.17.0.0/16?!?-o?docker0?-j?MASQUERADE

-A?POSTROUTING?-s?172.17.0.2/32?-d?172.17.0.2/32?-p?tcp?-m?tcp?--dport?5000?-j?MASQUERADE

-A?OUTPUT?!?-d?127.0.0.0/8?-m?addrtype?--dst-type?LOCAL?-j?DOCKER

-A?DOCKER?-i?docker0?-j?RETURN

-A?DOCKER?!?-i?docker0?-p?tcp?-m?tcp?--dport?5005?-j?DNAT?--to-destination?172.17.0.2:5000

$?sudo?iptables?-t?mangle?--list-rules

-P?PREROUTING?ACCEPT

-P?INPUT?ACCEPT

-P?FORWARD?ACCEPT

-P?OUTPUT?ACCEPT

-P?POSTROUTING?ACCEPT

$?sudo?iptables?-t?raw?--list-rules

-P?PREROUTING?ACCEPT

-P?OUTPUT?ACCEPT

$?sudo?nsenter?--net=/var/run/netns/netns0

$?python3?-m?http.server?--bind?172.18.0.10?5000

#?從?root?命名空間

$?curl?172.18.0.10:5000

#?...?忽略無關行?...

$?curl?10.0.2.15:5000

curl:?(7)?Failed?to?connect?to?10.0.2.15?port?5000:?Connection?refused

#?外部流量

sudo?iptables?-t?nat?-A?PREROUTING?-d?10.0.2.15?-p?tcp?-m?tcp?--dport?5000?-j?DNAT?--to-destination?172.18.0.10:5000

#?本地流量?(因為它沒有通過?PREROUTING?chain)

sudo?iptables?-t?nat?-A?OUTPUT?-d?10.0.2.15?-p?tcp?-m?tcp?--dport?5000?-j?DNAT?--to-destination?172.18.0.10:5000

sudo?modprobe?br_netfilter

curl?10.0.2.15:5000

#?...?忽略無關行?...

slirp4netns可以用完全非特權的方式將網絡命名空間連接到Internet上，通過網絡命名空間里的一個TAP設備連接到用戶態的TCP/IP棧（slirp）。

https://docs.docker.com/network/#network-drivers

https://www.redhat.com/sysadmin/container-networking-podman

https://github.com/rootless-containers/slirp4netns

https://developers.redhat.com/blog/2018/10/22/introduction-to-linux-interfaces-for-virtual-networking/

容器 網絡 虛擬化

版權聲明：本文內容由網絡用戶投稿，版權歸原作者所有，本站不擁有其著作權，亦不承擔相應法律責任。如果您發現本站中有涉嫌抄襲或描述失實的內容，請聯系我們jiasou666@gmail.com 處理，核實后本網站將在24小時內刪除侵權內容。