coredns故障 现象：coredns链接apiserver失败： 1 2 3 4 [INFO] plugin/ready: Still waiting on: "kubernetes" [INFO] plugin/ready: Still waiting on: "kubernetes" [WARNING] plugin/kubernetes: Kubernetes API connection failure: Get "https://10.68.0.1:443/version": dial tcp 10.68.0.1:443: i/o timeout
排查过程 查看k8s的apiserver的状态：
1 2 3 NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.68.0.1 <none> 443/TCP 172m
在宿主机上去去访问apiserver的地址：
1 2 3 4 5 6 7 root@xxxxxx-sxyq:/opt#curl 10.68.0.1:443 404 page not found root@xxxxxx-sxyq:/opt#ping 10.68.0.1 PING 10.68.0.1 (10.68.0.1) 56(84) bytes of data. 64 bytes from 10.68.0.1: icmp_seq=1 ttl=64 time=0.107 ms 64 bytes from 10.68.0.1: icmp_seq=2 ttl=64 time=0.085 ms
怀疑是iptables转发的问题，查看iptables，但是机器上并没有iptables命令，这就很奇怪。
1 2 3 (base) root@xxxxxx-sxyq:/opt# ipt iptables-apply iptables-restore iptables-save iptables-xml (base) root@xxxxxx-sxyq:/opt# ipt
使用apt 安装iptables。限制iptables已安装。只能使用apt-get install --reinstall iptables 重新安装iptabels。
1 2 3 4 5 6 7 8 9 10 (base) root@xxxxxx-sxyq:/opt# apt install iptables Reading package lists... Done Building dependency tree Reading state information... Done iptables is already the newest version (1.6.1-2ubuntu2.1). The following packages were automatically installed and are no longer required: ieee-data python-certifi python-chardet python-jmespath python-kerberos python-libcloud python-lockfile python-netaddr python-openssl python-requests python-selinux python-simplejson python-urllib3 python-xmltodict Use 'apt autoremove' to remove them. 0 upgraded, 0 newly installed, 0 to remove and 10 not upgraded.
重新安装iptables之后，iptables命令重新出现了。看来问题就在这里。于是清除了iptables规则iptables -F
新的问题 本以为问题就这样解决了，发现其他的pod根本不能访问coredns进行dns解析。
通过tcp抓cni0网卡的包发现：
1 2 3 4 5 6 7 8 11:44:20.643974 IP 10.58.0.172.57526 > 10.58.0.192.domain: Flags [S], seq 2885522300, win 64860, options [mss 1410,sackOK,TS val 4275158535 ecr 0,nop,wscale 7], length 0 11:44:21.665944 IP 10.58.0.172.57526 > 10.58.0.192.domain: Flags [S], seq 2885522300, win 64860, options [mss 1410,sackOK,TS val 4275159557 ecr 0,nop,wscale 7], length 0 11:44:23.681946 IP 10.58.0.172.57526 > 10.58.0.192.domain: Flags [S], seq 2885522300, win 64860, options [mss 1410,sackOK,TS val 4275161573 ecr 0,nop,wscale 7], length 0 11:44:25.825941 ARP, Request who-has xxxxxx-sxyq tell 10.58.0.172, length 28 11:44:27.873945 IP 10.58.0.172.57526 > 10.58.0.192.domain: Flags [S], seq 2885522300, win 64860, options [mss 1410,sackOK,TS val 4275165765 ecr 0,nop,wscale 7], length 0 11:44:36.065944 IP 10.58.0.172.57526 > 10.58.0.192.domain: Flags [S], seq 2885522300, win 64860, options [mss 1410,sackOK,TS val 4275173957 ecr 0,nop,wscale 7], length 0 11:44:52.193949 IP 10.58.0.172.57526 > 10.58.0.192.domain: Flags [S], seq 2885522300, win 64860, options [mss 1410,sackOK,TS val 4275190085 ecr 0,nop,wscale 7], length 0 11:44:57.313946 ARP, Request who-has xxxxxx-sxyq tell 10.58.0.172, length 28
查看k3s得网络模式，是使用的ipvs，
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 ExecStart=/usr/local/bin/k3s \ server \ '--disable=traefik' \ '--disable=coredns' \ '--data-dir=/data/rancher/k3s' \ '--service-cidr=10.68.0.0/16' \ '--cluster-cidr=10.58.0.0/16' \ '--cluster-dns=10.68.0.2' \ '--cluster-domain=cluster.local.' \ '--flannel-backend=vxlan' \ '--kubelet-arg=topology-manager-policy=single-numa-node' \ '--kubelet-arg=cpu-manager-policy=static' \ '--kubelet-arg=kube-reserved=cpu=1' \ '--kube-apiserver-arg=service-node-port-range=20000-40000' \ '--kube-apiserver-arg=authorization-mode=Node,RBAC' \ '--kube-apiserver-arg=allow-privileged=true' \ '--kube-proxy-arg=proxy-mode=ipvs' \ '--kube-proxy-arg=masquerade-all=true' \ '--kube-proxy-arg=metrics-bind-address=0.0.0.0' \ '--kube-scheduler-arg=config=/etc/rancher/k3s/scheduler-policy-config.yaml'
在网上排查找了个帖子https://ask.kubesphere.io/forum/d/4699-k8s-pod-ping-kubesphere-devops/52，感觉和我的抓包结果很像，发出去了包但是没有回应。/etc/sysctl.conf1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 vm.swappiness=0 net.core.rmem_default=256960 net.core.rmem_max=16777216 net.core.wmem_default=256960 net.core.wmem_max=16777216 net.core.netdev_max_backlog=2000 net.core.somaxconn=65535 net.core.optmem_max=81920 net.ipv4.tcp_mem=8388608 12582912 16777216 net.ipv4.tcp_rmem=8192 87380 16777216 net.ipv4.tcp_wmem=8192 65536 16777216 net.ipv4.tcp_keepalive_time=180 net.ipv4.tcp_keepalive_intvl=30 net.ipv4.tcp_keepalive_probes=3 net.ipv4.tcp_sack=1 net.ipv4.tcp_fack=1 net.ipv4.tcp_window_scaling=1 net.ipv4.tcp_syncookies=1 net.ipv4.tcp_tw_reuse=1 net.ipv4.tcp_tw_recycle=0 net.ipv4.tcp_fin_timeout=10 net.ipv4.tcp_max_syn_backlog=100000 fs.file-max=1100000 fs.nr_open=1100000 fs.inotify.max_user_watches=524288 kernel.pid_max=655350
后来还发现一个奇怪的现象，其实pod里面不能访问的只有coredns的10.68.0.2这个ip、pod的ip以及外部的ip，其他k8s的service都可以访问。这就很奇怪了。
故障解决 于是，清理了已经部署的k3s，然后重启了机器，重新安装k3s 1.25版本，重启之后进行安装，ansible脚本对系统配置进行了以下修改：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 - name: Disable daily security update remote_user: root become: yes shell: "{{ item }}" with_items: - "systemctl kill --kill-who=all apt-daily.service" - "systemctl stop apt-daily.timer" - "systemctl stop apt-daily-upgrade.timer" - "systemctl stop apt-daily.service" - "systemctl stop apt-daily-upgrade.service" - "systemctl disable apt-daily.timer" - "systemctl disable apt-daily-upgrade.timer" - "systemctl disable apt-daily.service" - "systemctl disable apt-daily-upgrade.service" when: ansible_os_family == "Debian" ignore_errors: True tags: init_env #- name: create user # user: name={{ item }} shell=/bin/bash createhome=yes # with_items: # - "{{ username }}" # - "readonly" # tags: create_user - name: Set limits pam_limits: dest: "{{ item.dest }}" domain: '*' limit_type: "{{ item.limit_type }}" limit_item: "{{ item.limit_item }}" value: "{{ item.value }}" with_items: - { dest: '/etc/security/limits.conf',limit_type: 'soft',limit_item: 'nofile', value: '655350' } - { dest: '/etc/security/limits.conf',limit_type: 'hard',limit_item: 'nofile', value: '655350'} #- { dest: '/etc/security/limits.conf',limit_type: 'soft',limit_item: 'nproc', value: '102400' } #- { dest: '/etc/security/limits.conf',limit_type: 'hard',limit_item: 'nproc', value: '102400' } - { dest: '/etc/security/limits.conf',limit_type: 'soft',limit_item: 'sigpending', value: '255377' } - { dest: '/etc/security/limits.conf',limit_type: 'hard',limit_item: 'sigpending', value: '255377' } - { dest: '/etc/security/limits.d/90-nproc.conf', limit_type: 'soft',limit_item: 'nproc', value: '262144' } - { dest: '/etc/security/limits.d/90-nproc.conf', limit_type: 'hard',limit_item: 'nproc', value: '262144' } tags: init_env - sysctl: name: "{{ item.name }}" value: "{{ item.value}}" state: present reload: yes with_items: - { name: 'vm.swappiness', value: '0'} - { name: 'net.core.rmem_default ',value: '256960'} - { name: 'net.core.rmem_max',value: '16777216'} - { name: 'net.core.wmem_default',value: '256960'} - { name: 'net.core.wmem_max ',value: '16777216'} - { name: 'net.core.netdev_max_backlog ',value: '2000'} - { name: 'net.core.somaxconn',value: '65535'} - { name: 'net.core.optmem_max',value: '81920'} - { name: 'net.ipv4.tcp_mem',value: '8388608 12582912 16777216'} - { name: 'net.ipv4.tcp_rmem',value: '8192 87380 16777216'} - { name: 'net.ipv4.tcp_wmem',value: '8192 65536 16777216'} - { name: 'net.ipv4.tcp_keepalive_time',value: '180'} - { name: 'net.ipv4.tcp_keepalive_intvl',value: '30'} - { name: 'net.ipv4.tcp_keepalive_probes',value: '3'} - { name: 'net.ipv4.tcp_sack',value: '1'} - { name: 'net.ipv4.tcp_fack',value: '1'} - { name: 'net.ipv4.tcp_window_scaling',value: '1'} - { name: 'net.ipv4.tcp_syncookies',value: '1'} - { name: 'net.ipv4.tcp_tw_reuse',value: '1'} - { name: 'net.ipv4.tcp_tw_recycle',value: '0'} - { name: 'net.ipv4.tcp_fin_timeout',value: '10'} #- { name: 'net.ipv4.ip_local_port_range',value: '1024 65000'} - { name: 'net.ipv4.tcp_max_syn_backlog',value: '100000'} - { name: 'fs.file-max',value: '1100000'} - { name: 'fs.nr_open',value: '1100000'} - { name: 'fs.inotify.max_user_watches', value: '524288' } - { name: 'kernel.pid_max', value: '655350' } ignore_errors: True tags: init_env
安装成功后，发现coredns依旧无法启动，但是查看内核日志：
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: E1108 10:07:06.376295 1867 proxier.go:1562] "Failed to execute iptables-restore" err=< 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: exit status 1: iptables-restore: invalid option -- 'w' 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: iptables-restore: invalid option -- 'W' 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: Unknown arguments found on commandline 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: > rules=< 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: *nat 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: :KUBE-SERVICES - [0:0] 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: :KUBE-POSTROUTING - [0:0] 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: :KUBE-NODE-PORT - [0:0] 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: :KUBE-LOAD-BALANCER - [0:0] 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: :KUBE-MARK-MASQ - [0:0] 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: -A KUBE-POSTROUTING -m comment --comment "Kubernetes endpoints dst ip:port, source ip for solving hairpin purpose" -m set --match-se 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: -A KUBE-NODE-PORT -p tcp -m comment --comment "Kubernetes nodeport TCP port for masquerade purpose" -m set --match-set KUBE-NODE-POR 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: -A KUBE-SERVICES -m comment --comment "Kubernetes service cluster ip + port for masquerade purpose" -m set --match-set KUBE-CLUSTER- 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: -A KUBE-SERVICES -m addrtype --dst-type LOCAL -j KUBE-NODE-PORT 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: -A KUBE-LOAD-BALANCER -j KUBE-MARK-MASQ 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: -A KUBE-SERVICES -m set --match-set KUBE-CLUSTER-IP dst,dst -j ACCEPT 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: -A KUBE-POSTROUTING -m mark ! --mark 0x00004000/0x00004000 -j RETURN 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: -A KUBE-POSTROUTING -j MARK --xor-mark 0x00004000 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: -A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -j MASQUERADE --random-fully 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: -A KUBE-MARK-MASQ -j MARK --or-mark 0x00004000 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: COMMIT 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: *filter 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: :KUBE-FORWARD - [0:0] 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: :KUBE-NODE-PORT - [0:0] 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: :KUBE-PROXY-FIREWALL - [0:0] 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: :KUBE-SOURCE-RANGES-FIREWALL - [0:0] 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: -A KUBE-SOURCE-RANGES-FIREWALL -j DROP 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: -A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x00004000/0x00004000 -j ACCEPT 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: -A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: -A KUBE-NODE-PORT -m comment --comment "Kubernetes health check node port" -m set --match-set KUBE-HEALTH-CHECK-NODE-PORT dst -j ACC 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: COMMIT 11月 08 10:07:06 xxxxxx-sxyq k3s[1867]: >
iptables-restore没有-w这个选项，但是我用的模式是ipvs啊。同时iptables命令又没了。
1 kernel: [569918.603973] IPVS: rr: UDP 10.68.0.2:53 - no destination available
结合上一条没有，应该是iptables的问题。apt-get install --reinstall iptables 重新安装iptables，再重启k3s，再次查看，coredns正常启动，再进pod尝试dns解析，发现pod内解析正常，也能正常访问pod的ip。查看k3s日志和内核日志，均无报错。
问题 虽然问题就这么解决了，但是我有点想不通。自己还是太菜了。
1、为啥这个系统重启后iptables会消失？
2、是什么导致的无法访问coredns的services和pod？
3、为啥ipvs模式会使用iptables的命令？ipvs和iptables有什么区别？
