k8s服务访问不通排查思路

一、服务访问不通

1、排查思路

telnet 服务访问的地址=》telnet 服务svc端口 =》telnet pod的端口号

1)telnet 服务访问地址:

服务是使用svc的nodeport方式提供访问,先telnet 服务映射到宿主机的端口号,如果不通,使用netstat -nutlp 检查宿主机端口是否监听,如果不监听,查下svc端口对应服务端口是否写正确,暴露方式是否正确;如果监听, 在继续往下;

2)telnet 服务svc端口:

如果telnet服务svc端口不通,则kubectl describe svc 看下服务svc后端对应的pods ip是否正确,如果正确,在继续往下;

3)telnet pod的端口号:

telnet pod的端口号不通,则进入容器内部,看下服务启动的端口是否正确,有可能服务内部启动使用的80,被写成了8048;

4)如果上述情况检查来配置文件都是对的,pod内部也能访问,直接curl pod的某个接口也行, 但是telnet svc地址或者宿主机地址不通,可以去检查下k8s的kube-proxy的网络模式;

[root@master03 ~]# kubectl describe cm -n kube-system kube-proxy |grep -i mode
detectLocalMode: ""
mode: ipvs

再去看下对应端口号网络转发规则是否有问题;

[root@master03 ~]# ipvsadm -L -n |grep -A5 8181
TCP  169.254.25.10:8181 rr
  -> 10.233.74.9:80               Masq    1      0          0         
  -> 10.233.123.231:80            Masq    1      0          0         
  -> 10.233.123.232:80            Masq    1      0          0         
TCP  169.254.25.10:8196 rr
  -> 10.233.126.139:8096          Masq    1      0          0         
--
TCP  169.254.25.10:18181 rr
  -> 10.233.74.63:2181            Masq    1      0          0         
  -> 10.233.123.234:2181          Masq    1      0          0         
  -> 10.233.126.155:2181          Masq    1      0          0         
TCP  169.254.25.10:31267 rr
  -> 10.233.74.63:9141            Masq    1      0          0         
--
TCP  192.168.42.1:18181 rr
  -> 10.233.74.63:2181            Masq    1      0          0         
  -> 10.233.123.234:2181          Masq    1      0          0         
  -> 10.233.126.155:2181          Masq    1      0          0         
TCP  192.168.42.1:31267 rr
  -> 10.233.74.63:9141            Masq    1      0          0         
--
TCP  192.168.122.1:18181 rr
  -> 10.233.74.63:2181            Masq    1      0          0         
  -> 10.233.123.234:2181          Masq    1      0          0         
  -> 10.233.126.155:2181          Masq    1      0          0         
TCP  192.168.122.1:31267 rr
  -> 10.233.74.63:9141            Masq    1      0          0         
--
TCP  10.49.0.188:8181 rr
  -> 10.233.74.9:80               Masq    1      1          0         
  -> 10.233.123.231:80            Masq    1      0          0         
  -> 10.233.123.232:80            Masq    1      0          0         
TCP  10.49.0.188:8196 rr
  -> 10.233.126.139:8096          Masq    1      0          0         
--
TCP  10.49.0.188:18181 rr
  -> 10.233.74.63:2181            Masq    1      0          0         
  -> 10.233.123.234:2181          Masq    1      0          0         
  -> 10.233.126.155:2181          Masq    1      0          0         
TCP  10.49.0.188:31267 rr
  -> 10.233.74.63:9141            Masq    1      0          0         
--
TCP  10.233.74.0:8181 rr
  -> 10.233.74.9:80               Masq    1      0          0         
  -> 10.233.123.231:80            Masq    1      0          0         
  -> 10.233.123.232:80            Masq    1      0          0         
TCP  10.233.74.0:8196 rr
  -> 10.233.126.139:8096          Masq    1      0          0

如果没有ipvsadm工具,需要自己安装:

yum install -y ipvsadm

2、前端服务界面,可以正常访问,但是无法作修改或者新增数据等操作

原因 :有可能是后端数据库出现问题了

排查:查看数据库状态

1)mysql单节点版本
# 登录数据库
mysql -uroot -p
#查看数据的当前连接数,注意sleep的连接数
show processlist;
#查看数据设置的最大连接数
show variables like '%max_connection%';
设置“max_connections=1000”,这样就可设置mysql的最大连接数为1000。
#设置最大连接数
set global max_connections=1000;
# 连接总数、活跃数、最大并发数
mysql> show status like  'Threads%';

+-------------------+-------+

| Variable_name     | Value |

+-------------------+-------+

| Threads_cached    | 32    |

| Threads_connected | 10    |

| Threads_created   | 50    |

| Threads_rejected  | 0     |

| Threads_running   | 1     |

+-------------------+-------+

5 rows in set (0.00 sec)

Threads_connected :这个数值指的是打开的连接数.

Threads_running :这个数值指的是激活的连接数,这个数值一般远低于connected数值.

Threads_connected 跟show processlist结果相同,表示当前连接数。准确的来说,Threads_running是代表当前并发数查询数据库当前设置的最大连接数

2)mysql主从版本
#查看数据主从状态
MySQL> show slave status\G;

主要看Slave_IO_Running与Slave_SQL_Running这两个字段都显示yes就行了。

3)mysql集群版本
# k8s安装使用此命令查看数据库集群状态
kubectl exec -it xxx -n xxx -c xenon -- xenoncli cluster status
+------------------------------------------------+-------------------------------+--------+---------+--------------------------+---------------------+----------------+------------------------------------------------+
|                       ID                       |             Raft              | Mysqld | Monitor |          Backup          |        Mysql        | IO/SQL_RUNNING |                    MyLeader                    |
+------------------------------------------------+-------------------------------+--------+---------+--------------------------+---------------------+----------------+------------------------------------------------+
| det-tn-mysql-ha-0.det-tn-mysql-ha.default:8801 | [ViewID:3 EpochID:2]@FOLLOWER | UNKNOW | OFF     | state:[NONE]␤            | [ALIVE] [READONLY]  | [true/true]    | det-tn-mysql-ha-1.det-tn-mysql-ha.default:8801 |
|                                                |                               |        |         | LastError:               |                     |                |                                                |
+------------------------------------------------+-------------------------------+--------+---------+--------------------------+---------------------+----------------+------------------------------------------------+
| det-tn-mysql-ha-1.det-tn-mysql-ha.default:8801 | [ViewID:3 EpochID:2]@LEADER   | UNKNOW | OFF     | state:[NONE]␤            | [ALIVE] [READWRITE] | [true/true]    | det-tn-mysql-ha-1.det-tn-mysql-ha.default:8801 |
|                                                |                               |        |         | LastError:               |                     |                |                                                |
+------------------------------------------------+-------------------------------+--------+---------+--------------------------+---------------------+----------------+------------------------------------------------+
| det-tn-mysql-ha-2.det-tn-mysql-ha.default:8801 | [ViewID:3 EpochID:2]@FOLLOWER | UNKNOW | OFF     | state:[NONE]␤            | [ALIVE] [READONLY]  | [true/true]    | det-tn-mysql-ha-1.det-tn-mysql-ha.default:8801 |
|                                                |                               |        |         | LastError:               |                     |                |                                                |
+------------------------------------------------+-------------------------------+--------+---------+--------------------------+---------------------+----------------+------------------------------------------------+
(3 rows)

mysql集群状态

© 版权声明
THE END
喜欢就支持一下吧
点赞13 分享
评论 抢沙发
头像
欢迎您留下宝贵的见解!
提交
头像

昵称

取消
昵称表情代码图片