2022/1/6 西安疫情日记 疫情在接近2022的最后几天，疫情让西安封城了。前奏上班每天都要核酸检测结果。然而，一码通出现问题，导致做了核酸出不了码。一半的员工在楼下拿不到最新的结果上不了楼办公。平时能力超强的领导在这关键的时候哑了火，留下一地人提着早餐在楼下一脸愣逼。好在西安地区开始了居家办公，不然吐槽要准备两包瓜子才能成。疫情和一码通，买菜，孕妇等等事宜，迅速成了最近最受嘲讽的城市之一。各种热搜第一，似乎所有都有问题，轻轻一碰就能起大浪，大佬们为了乌纱帽各种骚话骚操作，我也是看的醉醉的对西安没有感情，如果不是离家近，再者囊中羞涩，我大抵是不会选择西安的可是，要适应下把。我们都需要自我安慰，给自己一点时间，让自己来安抚自己的情绪。居家隔离居家上班半个月，太忙。可以理解成居家上班就等于不下班，没有下班。上班和下班，是陆地和海岸的分界线。当你开始相信稻盛和夫的那一套的时候，你要明白，你不是他。请注意，当你相信的时候，你已经是老板了。但不得不说，稻盛老爷子的书还是能看一看的，没准你下辈子就成了老板，或许还拿出来和别人吹下水有用也不一定呢。20222022夸年是在隔离中度过。新房没有装修好，在外已经租房半年，从头到尾都没有收到免费的菜，也没有业主群，什么都没，倒也利利索索。吃了两天泡面，我那可怜的牛奶和水果汁都在京东的路上，只是暂时不能拿到了，可叹的未能早点下单。买菜热潮终于褪去，伴着寒风我闪亮登场，买了几包菜，2斤肉，三下五除二，2天干掉。吃着汤锅炖的菜，生活有了些须改善，有了些喝茶的欲望，这无疑是一件好事情。2021年12月23日零点开始封城，半个月。手机短信温馨提示，大意是如果你在和花花草草说话的话，不必担心这时候如果花花草草回答你了，需要注意一下了。好在这糟糕的事情，并没有发生。有个伙计用他的妩媚的水床拒绝了地暖。此时，外面迎来了第一场雪，外面天气仍然寒冷，打开落地窗，兴许是楼上楼下也有暖气的，倒也不显得冷，只是心里有些蹉跎。关于感动与眼泪最近没有去学习，一直在看西瓜视频。我个人拒绝抖音这类的东西，但是西瓜视频最近的确翻的比较多。今天想说一个感动和眼泪的事情，生活中有很多这个场景，我最刚好刷到这个。当然其他也有很多。“达人秀2021”里面有很多的达人，都很有特色。就像“新疆棉花哥“自制架子鼓翻唱小虎的”爱“。棉花哥年轻的时候就喜欢打架子鼓。棉花哥有个老婆，棉花哥在三十年前答应老婆收拾好棉花就买个钻戒，然而三十年后还是相同的台词在忽悠老婆，到现在还没给老婆买上，希望对所有人唱一首歌给他老婆表达这三十年来的爱意。接着棉花哥打着自己的架子鼓，唱着小虎队的爱把邓紫棋唱的抹眼泪，而谢霆锋说很久没看到这么纯粹的真实的音乐表达方式，孟非说音乐要有直击人心的力量。还有个女人似乎是打酱油的，她说了些好像什么都有说的，关键是也很面生，哈哈。那为什么同样的演出对不同的人感触不一样，而偏偏邓紫棋却在抹眼泪。泪点，我称作阈值，那么更多时候，它映射的是一种身受感同。请注意，我的措辞，身受感同绝非感同身受，这两个词的表达意思差之千里。什么是身受感同，这里就有很多的例子：人生中同样处理过此类糟糕事情的人在得知一方的悲剧，是身受感同的。一个人就一部手机，手机号码都绑定了七七八八一大堆的东西，什么都在里面，一个丢了手机的人和一个丢过手机的人是身受感同的。一个顺产的女人和另外一个顺产的女人所受到的疼痛是身受感同的。如果你没有身受，而去感同说的是两回事情。我们在电视剧里面经常会看到，当一方遇到糟糕的事情的时候， 成熟的方式是不去假设身受感同，而是让对方释放情绪，而又开始引导到其他地方，这很高明。而去假装感同身受的时候，你大概会发现，这人常常要么扮演的是一个幼稚的角色，要么就是一些反派，总之不怎么有智慧的样子。这些都是题外的话人生很长，也很短，很多事情过去了，忘记了，但并不代表事情没有来过，留下一些隐藏的开关，这个就是我说的阈值。这和佛教的空，一样。说到这里就很有趣的，也是我为什么多次翻阅佛经的书，因为很有意思。空说的就是事情过去了，我们忘记了，但是从来都没有否认事情没有来过。这个就是空性。在上面我说人生很长，但是另外有一个叫人生如梦。当你觉得人生的时候，就应该是梦醒的状态。不然的话，那虽然是睁开了眼，但还是在梦中的状态。有些人醒来了还在想那梦，此情可待成追忆，只是当时已惘然。恋梦的状态来了那么对于过去的事情，人生常常如此怀念。比如， 当时只是平常事，过后思量倍有情。你我人生经验都有这么一出，各自归各自的过去不可得。这就是人生。我们的人生很容易欺骗自己。觉时恋梦梦宜醒。这些出自金刚经南怀瑾第六品。其实这只有一个真相，眼泪这个东西非常奇妙，绝非偶然。有些快乐也是痛苦的，快乐后是痛苦，痛苦后是快乐的这么一个意思，因为人总是在向前走，只比鱼多不了多少记忆，只是学会了一些手段。这和周星驰的电影一样，你会发现一些高明的悲剧都是以喜剧的形式出现，往往让人乐在其中，却又透着难过和心酸，使人不自禁的对好的喜剧拍手叫好。次的就更别提了，凡所有相，皆是虚妄。

linuxea:清理kube-prometheus历史数据 通常在k8s中，pod是随时可以被替换的，在整个环境里往往我们不太关注某一条鱼，只关注整个鱼群的状态，因此监控数据不会存储太长，因为借鉴意义并不大。但是有时的确想要从 Prometheus 中删除一些指标，如果这些指标不需要，或者只需要释放一些磁盘空间。Prometheus 中的时间序列只能通过管理 HTTP API 删除（默认禁用）。--web.enable-admin-apiAs of Prometheus 2.0, the --web.enable-admin-api flag controls access to the administrative HTTP API which includes functionality such as deleting time series. This is disabled by default. If enabled, administrative and mutating functionality will be accessible under the /api/*/admin/ paths. The --web.enable-lifecycle flag controls HTTP reloads and shutdowns of Prometheus. This is also disabled by default. If enabled they will be accessible under the /-/reload and /-/quit paths.要启用它--web.enable-admin-api，根据安装方法通过启动脚本或 docker-compose 文件将标志传递给 Prometheus。删除时间序列指标使用以下语法删除与某个标签匹配的所有时间序列指标：curl -X POST \ -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={name="linuxea.com"}'要删除与 somejob或匹配的时间序列指标instance，请运行：curl −X POST −g 'http://localhost:9090/api/v1/admin/tsdb/deleteseries?match[]=job="nodeexporter"' curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={job="node_exporter"}' curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="192.168.0.1:9100"}'要从 Prometheus 中删除所有数据，请运行：curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={name=~".+"}'请注意，上述 API 调用不会立即删除数据实际数据仍然存在于磁盘上，将在未来的压缩中被清除。要确定何时删除旧数据，使用--storage.tsdb.retention选项 eg --storage.tsdb.retention='365d'（默认情况下，Prometheus 将数据保留 15 天）要完全删除通过delete_series发送clean_tombstonesAPI 调用删除的数据：curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/clean_tombstones'删除所有历史数据我们使用的kube-prometheus，添加--web.enable-admin-api方式有所不同打开prometheus-prometheus.yaml 添加enableAdminAPI: true字段 probeNamespaceSelector: {} probeSelector: {} replicas: 2 resources: requests: memory: 400Mi enableAdminAPI: true ruleSelector: matchLabels: prometheus: k8s而后kubectl apply -f prometheus-prometheus.yaml，pod起来后查看 containers: - args: - '--web.console.templates=/etc/prometheus/consoles' - '--web.console.libraries=/etc/prometheus/console_libraries' - '--config.file=/etc/prometheus/config_out/prometheus.env.yaml' - '--storage.tsdb.path=/prometheus' - '--storage.tsdb.retention.time=15d' - '--web.enable-lifecycle' - '--storage.tsdb.no-lockfile' - '--web.route-prefix=/' - '--web.enable-admin-api' image: 'quay.io/prometheus/prometheus:v2.26.0' 准备开始删除curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={__name__=~".+"}'查看删除前的大小[root@linuxea.com prometheus-db]# ls 01FPZGK1S0BZ9FZMSVPYWA1F52 01FQPP5PJTEMT1BARN8ARJTBKB 01FR222TD3ZWYAZTNRJP5SCJ6M 01FR4VB8N4J0D2DY979PTHZQY1 wal 01FQ59ZR5Z9VNNGY3JSXP6F9AQ 01FQWFJAC72YCBKEN19F39PYCM 01FR46R88Q9NN7GR99MMGX13VG 01FR526TSKK8JTN493PTBQH0XC 01FQB3CCFVNSP890V5KE07YD3M 01FQYDBKTHH8KB8K5GT581BKGE 01FR4MFC9JX6NK81G4Q1W0CJNY chunks_head 01FQGWS2M82Q23H0H2TD0JXK1Y 01FR0B54YSBJ0R54B5171P4Z7R 01FR4VB3HHRN0EE1GQCM6G0A77 queries.active [root@linuxea.com nfs-k8s]# du -sh monitoring-prometheus-k8s-db-prometheus-k8s-0-pvc-ac911baf-f1f1-4bf3-a1f8-af2cb13c5d90/ 7.1G monitoring-prometheus-k8s-db-prometheus-k8s-0-pvc-ac911baf-f1f1-4bf3-a1f8-af2cb13c5d90/找到svc[root@linuxea.com nfs-k8s]# kubectl -n monitoring get svc |grep prometheus-k8s prometheus-k8s NodePort 10.68.110.123 <none> 9090:30090/TCP 128d尝试curl一下[root@linuxea.com nfs-k8s]# curl 10.68.110.123:9090 <a href="/graph">Found</a>.执行[root@linuxea.com manifests]# curl -X POST -g 'http://10.68.110.123:9090/api/v1/admin/tsdb/delete_series?match[]={__name__=~".+"}'而后在查看大小[root@linuxea.com manifests]# kubectl -n monitoring exec -it prometheus-k8s-0 -- /bin/sh Defaulting container name to prometheus. Use 'kubectl describe pod/prometheus-k8s-0 -n monitoring' to see all of the containers in this pod. /prometheus $ du -sh ./ 1004.4M ./ /prometheus $ ls 01FR46R88Q9NN7GR99MMGX13VG 01FR4VB8N4J0D2DY979PTHZQY1 chunks_head 01FR4MFC9JX6NK81G4Q1W0CJNY 01FR526TSKK8JTN493PTBQH0XC queries.active 01FR4VB3HHRN0EE1GQCM6G0A77 01FR592J0H8ANAWMFP2VVQ3NX9 wal /prometheus $ du -sh ./ 1006.6M ./完全删除通过delete_series发送clean_tombstonesAPI 调用删除的数据[root@linuxea.com manifests]# curl -X POST -g 'http://10.68.110.123:9090/api/v1/admin/tsdb/clean_tombstones'在查看大小[root@linuxea.com manifests]# kubectl -n monitoring exec -it prometheus-k8s-0 -- /bin/sh Defaulting container name to prometheus. Use 'kubectl describe pod/prometheus-k8s-0 -n monitoring' to see all of the containers in this pod. /prometheus $ du -sh ./ 986.6M ./删除完成后，将--web.enable-admin-api关闭。修改enableAdminAPI: false字段即可延申参考tsdb-admin-apisAdd flag to enable prometheus web admin API and fixes #1215kube-prometheusspecprometheus-prometheus.yaml

linuxea:calico间歇失败探针Readiness /Liveness probe faile 我们有两个版本的内核节点，分别是两个批次的节点，安装了3.10.95和5.4.x的内核在出现Readiness probe failed和Liveness probe failed是一次pod驱逐而后就发生的查看事件# kubectl get events --sort-by='.lastTimestamp' -n kube-system LAST SEEN TYPE REASON OBJECT MESSAGE 50m Warning Unhealthy pod/calico-node-dbjr4 Readiness probe failed: 48m Warning Unhealthy pod/calico-node-8t7rc Liveness probe failed: 43m Warning Unhealthy pod/calico-node-msj2z Liveness probe failed: 34m Warning Unhealthy pod/calico-node-9gxqf Liveness probe failed: 34m Warning Unhealthy pod/calico-node-cj8qk Readiness probe failed: 32m Warning Unhealthy pod/calico-node-tph5b Liveness probe failed:回到Kuboard面板查看事件有一种可能tcp_tw_recycle，tcp_tw_recycle 在低版本的内核里面是可以被启用的，于是我们查看，发现过是打开状态关闭tcp_tw_recycle在 kernel 的4.12之后的版本中tcp_tw_recycle已经被移除了net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_tw_reuse = 1于是我们关闭sysctl -w net.ipv4.tcp_tw_recycle=0当打开tcp_tw_recycle的时候很有可能会发生calico的状态存在断续的或者一直起不来的情况，原因就是这种连接资源快速回收和重用的机制会在k8s中出现问题,特别是在有会话session的时候tcp_tw_recyle=1无法接收 NATed 数据包的服务器。默认情况下，当tcp_tw_reuse和tcp_tw_recycle都被禁用时，内核将确保处于TIME_WAIT状态的套接字将保持在该状态足够长的时间——足够长的时间以确保属于未来连接的数据包不会被误认为是旧连接的延迟数据包。当您启用时tcp_tw_reuse，处于TIME_WAIT状态的套接字可以在它们到期之前使用，并且内核将尝试确保没有关于 TCP 序列号的冲突。如果您启用tcp_timestamps（又名 PAWS，用于防止包装序列号），它将确保不会发生这些冲突。然而，你需要TCP时间戳上启用两个端（至少，这是我的理解）。有关详细信息，请参阅tcp_twsk_unique的定义。当您启用 时tcp_tw_recycle，内核会变得更加激进，并且会对远程主机使用的时间戳做出假设。它将跟踪每个具有连接TIME_WAIT状态的远程主机使用的最后一个时间戳），如果时间戳正确增加，则允许重新使用套接字。但是，如果主机使用的时间戳发生变化（即及时回溯），SYN数据包将被悄悄丢弃，并且连接将无法建立（您将看到类似于“连接超时”的错误）。如果您想深入了解内核代码，tcp_timewait_state_process的定义可能是一个很好的起点。现在，时间戳永远不应该回到过去；除非：主机重新启动（但是，当它恢复时，TIME_WAIT套接字可能已经过期，所以这将不是问题）；IP 地址很快被其他东西重用（TIME_WAIT连接会保留一点，但其他连接可能会被攻击TCP RST，这会释放一些空间）；网络地址转换（或 smarty-pants 防火墙）涉及连接中间。在后一种情况下，您可以在同一个 IP 地址后面有多个主机，因此，不同的时间戳序列（或者，所述时间戳在每个连接上由防火墙随机化）。在这种情况下，一些主机将随机无法连接，因为它们被映射到TIME_WAIT服务器存储桶具有更新时间戳的端口。这就是为什么文档会告诉您“NAT 设备或负载平衡器可能会因为设置而开始丢帧”的原因。有些人建议不要管tcp_tw_recycle，但启用tcp_tw_reuse并降低tcp_fin_timeout. 我同意 ：-）阅读-> 这篇文章有助于理解net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-arptables = 1 net.ipv4.tcp_tw_reuse = 0 net.core.somaxconn = 32768 net.netfilter.nf_conntrack_max=1000000 vm.swappiness = 0 vm.max_map_count=655360 fs.file-max=6553600 net.ipv4.tcp_keepalive_time = 600 net.ipv4.tcp_keepalive_intvl = 30 net.ipv4.tcp_keepalive_probes = 10问题仍然没有解决，此时我们并没有发现有用的日志信息，查看kublet日志如下，仍然提示探针失败如下12月 29 10:52:10 master1 kubelet[1458]: E1229 10:52:10.468294 1458 remote_runtime.go:392] ExecSync 312728508abf1a1bfea28a70a174e8a97e57685fc8bae741fa1702fb341e4b06 '/bin/calico-node -felix-ready -bird-ready' from runtime service failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded 12月 29 10:52:10 master1 kubelet[1458]: I1229 10:52:10.468344 1458 prober.go:117] Readiness probe for "calico-node-dbjr4_kube-system(8d8d3b92-b7d3-424d-a6af-48b8559cbe15):calico-node" failed (failure): 12月 29 10:52:10 master1 kubelet[1458]: E1229 10:52:10.543839 1458 remote_runtime.go:392] ExecSync 4805cd505d5b6494d26130cc0bf83a458c1b099c1eefebac8dff942ffe04b980 '/usr/bin/check-status -r' from runtime service failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded 12月 29 10:52:10 master1 kubelet[1458]: I1229 10:52:10.543899 1458 prober.go:117] Readiness probe for "calico-kube-controllers-5b75d4f8db-pn5ll_kube-system(b4c2985a-6476-448c-9e44-c2a1e7a7db07):calico-kube-controllers" failed (failure):查看系统，目前并没有其他特别的异常，并且业务正常，我们试图调整探针返回时间来测试调整Liveness/Readinesscalico探针的命令如下/bin/calico-node '-felix-live' - '-bird-live' /bin/calico-node '-felix-ready' '-bird-ready' 我们在yaml文件中默认配置的是1秒，在1.20版本也是如此 livenessProbe: exec: command: - /bin/calico-node - '-felix-live' - '-bird-live' failureThreshold: 6 initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 name: calico-node readinessProbe: exec: command: - /bin/calico-node - '-felix-ready' - '-bird-ready' failureThreshold: 3 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1但是这在一个负载较高的环境中可能需要增加timeoutSeconds关于探针阅读这篇linuxea:kubernetes 存活状态liveness probe(7)failureThreshold：探测几次，默认3次探测失败，则失败 periodSeconds：间隔时间，10秒 timeoutSeconds：探测超时时间，1秒 initialDelayseconds : 初始化探测延迟时间(在容器启动的时候，直接修改 timeoutSeconds: 5，而后观察一小时。问题解决# kubectl get events --sort-by='.lastTimestamp' -n kube-system No resources found in kube-system namespace.参考Dropping of connections with tcp_tw_recycleWhy does the TCP protocol have a TIME_WAIT state?calico-node unexplained readiness/liveness probe failsCalico pod Intermittently triggers probe failed eventslinuxea:kubernetes 存活状态liveness probe(7)

linuxea:nexus3排序清理docker镜像列表 nexus3的出现可以替代很多产品，nexus3整合了常用的功能，比如docker镜像仓库等。一般我们在配置docker仓库的时候，在blob stores中创建的新的stores来存储镜像。这样方便管理早期的nexus-cli已经没有维护，nexus-cli删除的镜像并没有按照时间的顺序进行保留 。需要在13rentgen的仓库中下载修复版的，所以你需要下载最新的进行使用登录nexus下载最新的修复版本https://github.com/13rentgen/nexus-cli/releases/tag/v1.1.0这里演示用的是旧的版本[root@nexus ]# ./nexus-cli --version Nexus CLI version 1.0.0-beta登录nexus[root@nexus ~]# nexus-cli configure Enter Nexus Host: http://127.0.0.1:38081 Enter Nexus Repository Name: docker-host Enter Nexus Username: admin Enter Nexus Password: yxt@..123!com列出镜像[root@nexus ~]# nexus-cli image ls删除命令nexus-cli image delete -name IMAGE_NAME -keep X，-keep X 表示保留几个tag如果是单个删除，那么命令如下nexus-cli image delete -name dnotask1 -keep 10 nexus-cli image delete -name outage2 -keep 10 nexus-cli image delete -name 1speed -keep 10 nexus-cli image delete -name mongo2 -keep 10 nexus-cli image delete -name mongo1 -keep 10清理脚本#!/bin/bash # Copyright 2022 marksugar, personal # 2022 linuxea.com # Author: mark <usertzc#163.com> USER=admin PASS="password" REPO=docker-host HOST="http://127.0.0.1:38081" CLEAN_DISK_NAME=/data/ touchnet(){ local recode=`curl -o /dev/null -s --connect-timeout 3 $1 -w %{http_code}` if [ "$recode" == "200" ]; then return 1 fi } Create_CRE(){ # protect file since it contains secrets touch ~/.credentials && chmod 600 ~/.credentials # update file using environment variables cat << _EOF > ~/.credentials # Nexus Credentials nexus_host = "${HOST}" nexus_username = "${USER}" nexus_password = "${PASS}" nexus_repository = "${REPO}" _EOF } touchnet ${HOST} if [ `echo $?` == "1" ]; then [ -f ~/.credentials ] || Create_CRE ; else echo "Cannot communicate with the ${HOST}";exit -1; fi PATHS=yxt-vehicle PATH_NAME="dnotask1 outage21speed mongo2 mongo1" TAKE_NUM=10 type nexus-cli > /dev/null 2>&1 && echo " nexus-cli is ok" || { echo "error nexus-cli is not install";exit -1 ;} if [ `df -hT ${CLEAN_DISK_NAME} | tail -1| awk '{print $6}'| awk -F% '{print $1}'` -gt 80 ];then for i in `echo ${PATH_NAME}`;do echo -e "\033[34m $i Start cleaning, keep the last 10 \033[0m" && nexus-cli image delete -name ${PATHS}/${i} -keep ${TAKE_NUM} ;done fi清理完成后进行删除Tasks --> create task --> delete unused manifests and images Task执行完就消失了Tasks --> create task ---> compact blob store Task而后指定store清理Compact blob store。

linuxea:处理k8s kubelet.go node "master" not found问题 一个项目的master节点down了之后kubelet起不来，由于是远古时期的项目，早期未作任何监控手段。因此也没有监控，没有文档，非常棘手经过查看，master节点的kubelet未启动，查看kubelet日志: failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directoryfailed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory随即在/usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf中去掉--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf后重启master的kubelet已经启动起来，但是还是有问题，status是NotReady查看master状态也是NotReady，而后就暂停了master的调度SchedulingDisabled[root@master ~]# kubectl get node NAME STATUS ROLES AGE VERSION master NotReady,SchedulingDisabled master 647d v1.16.3 worker1 Ready <none> 647d v1.16.3 worker2 Ready <none> 647d v1.16.3我们留意到worker是ready的，于是我们查看worker节点。发现worker节点调度正常，并且可以进行拉取镜像，并且此时业务也恢复了。我们使用kubectl describe master查看到master节点还是有问题[root@master ~]# kubectl describe node master Name: master Roles: master Labels: beta.kubernetes.io/arch=amd64 beta.kubernetes.io/os=linux kubernetes.io/arch=amd64 kubernetes.io/hostname=master kubernetes.io/os=linux node-role.kubernetes.io/master= Annotations: kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock node.alpha.kubernetes.io/ttl: 0 projectcalico.org/IPv4Address: 192.168.133.6/25 projectcalico.org/IPv4IPIPTunnelAddr: 10.100.219.64 volumes.kubernetes.io/controller-managed-attach-detach: true CreationTimestamp: Sat, 14 Mar 2020 11:37:43 +0800 Taints: node.kubernetes.io/unreachable:NoExecute node-role.kubernetes.io/master:NoSchedule node.kubernetes.io/unreachable:NoSchedule node.kubernetes.io/unschedulable:NoSchedule Unschedulable: true Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- NetworkUnavailable False Thu, 04 Mar 2021 10:26:49 +0800 Thu, 04 Mar 2021 10:26:49 +0800 CalicoIsUp Calico is running on this node MemoryPressure Unknown Tue, 21 Dec 2021 09:17:47 +0800 Tue, 21 Dec 2021 09:18:41 +0800 NodeStatusUnknown Kubelet stopped posting node status. DiskPressure Unknown Tue, 21 Dec 2021 09:17:47 +0800 Tue, 21 Dec 2021 09:18:41 +0800 NodeStatusUnknown Kubelet stopped posting node status. PIDPressure Unknown Tue, 21 Dec 2021 09:17:47 +0800 Tue, 21 Dec 2021 09:18:41 +0800 NodeStatusUnknown Kubelet stopped posting node status. Ready Unknown Tue, 21 Dec 2021 09:17:47 +0800 Tue, 21 Dec 2021 09:18:41 +0800 NodeStatusUnknown Kubelet stopped posting node status. Addresses: InternalIP: 192.168.133.6 Hostname: master Capacity: cpu: 4 ephemeral-storage: 36805060Ki hugepages-2Mi: 0 memory: 16259676Ki pods: 110 Allocatable: cpu: 4 ephemeral-storage: 33919543240 hugepages-2Mi: 0 memory: 16157276Ki pods: 110 System Info: Machine ID: abfbcf67c72dca40affb0cd0c8debd6d System UUID: 63B3A0C6-AEB7-3841-BA81-2724178E0890 Boot ID: 02424d77-4b14-45ea-a632-038dbc70a2c4 Kernel Version: 3.10.0-957.el7.x86_64 OS Image: CentOS Linux 7 (Core) Operating System: linux Architecture: amd64 Container Runtime Version: docker://18.9.7 Kubelet Version: v1.16.3 Kube-Proxy Version: v1.16.3 PodCIDR: 10.100.0.0/24 PodCIDRs: 10.100.0.0/24 Non-terminated Pods: (6 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE --------- ---- ------------ ---------- --------------- ------------- --- kube-system calico-node-v2rmv 250m (6%) 0 (0%) 0 (0%) 0 (0%) 4h15m kube-system etcd-master 0 (0%) 0 (0%) 0 (0%) 0 (0%) 362d kube-system kube-apiserver-master 250m (6%) 0 (0%) 0 (0%) 0 (0%) 362d kube-system kube-controller-manager-master 200m (5%) 0 (0%) 0 (0%) 0 (0%) 362d kube-system kube-proxy-k7shj 0 (0%) 0 (0%) 0 (0%) 0 (0%) 4h15m kube-system kube-scheduler-master 100m (2%) 0 (0%) 0 (0%) 0 (0%) 362d Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 800m (20%) 0 (0%) memory 0 (0%) 0 (0%) ephemeral-storage 0 (0%) 0 (0%) Events: <none>接着查看日志,提示node "master" not found[root@master ~]# journalctl -u kubelet -f -- Logs begin at Sun 2021-10-17 15:04:51 CST. -- Dec 21 14:56:18 master kubelet[34678]: E1221 14:56:18.159602 34678 kubelet.go:2267] node "master" not found Dec 21 14:56:18 master kubelet[34678]: E1221 14:56:18.259813 34678 kubelet.go:2267] node "master" not found Dec 21 14:56:18 master kubelet[34678]: E1221 14:56:18.329948 34678 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: services is forbidden: User "system:anonymous" cannot list resource "services" in API group "" at the cluster scope Dec 21 14:56:18 master kubelet[34678]: E1221 14:56:18.360614 34678 kubelet.go:2267] node "master" not found Dec 21 14:56:18 master kubelet[34678]: E1221 14:56:18.460822 34678 kubelet.go:2267] node "master" not found Dec 21 14:56:18 master kubelet[34678]: E1221 14:56:18.531167 34678 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:459: Failed to list *v1.Node: nodes "master" is forbidden: User "system:anonymous" cannot list resource "nodes" in API group "" at the cluster scope Dec 21 14:56:18 master kubelet[34678]: E1221 14:56:18.560990 34678 kubelet.go:2267] node "master" not found Dec 21 14:56:18 master kubelet[34678]: E1221 14:56:18.661174 34678 kubelet.go:2267] node "master" not found Dec 21 14:56:18 master kubelet[34678]: E1221 14:56:18.731398 34678 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" at the cluster scope Dec 21 14:56:18 master kubelet[34678]: E1221 14:56:18.761342 34678 kubelet.go:2267] node "master" not found Dec 21 14:56:18 master kubelet[34678]: E1221 14:56:18.861818 34678 kubelet.go:2267] node "master" not found Dec 21 14:56:18 master kubelet[34678]: E1221 14:56:18.930789 34678 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User "system:anonymous" cannot list resource "csidrivers" in API group "storage.k8s.io" at the cluster scope Dec 21 14:56:18 master kubelet[34678]: E1221 14:56:18.962003 34678 kubelet.go:2267] node "master" not found Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.062175 34678 kubelet.go:2267] node "master" not found Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.131629 34678 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User "system:anonymous" cannot list resource "runtimeclasses" in API group "node.k8s.io" at the cluster scope Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.162315 34678 kubelet.go:2267] node "master" not found Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.262472 34678 kubelet.go:2267] node "master" not found Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.331431 34678 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: services is forbidden: User "system:anonymous" cannot list resource "services" in API group "" at the cluster scope Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.362644 34678 kubelet.go:2267] node "master" not found Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.462865 34678 kubelet.go:2267] node "master" not found Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.532574 34678 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:459: Failed to list *v1.Node: nodes "master" is forbidden: User "system:anonymous" cannot list resource "nodes" in API group "" at the cluster scope Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.563028 34678 kubelet.go:2267] node "master" not found Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.663176 34678 kubelet.go:2267] node "master" not found Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.732705 34678 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/config/apiserver.go:46: Failed to list *v1.Pod: pods is forbidden: User "system:anonymous" cannot list resource "pods" in API group "" at the cluster scope Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.763328 34678 kubelet.go:2267] node "master" not found Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.863481 34678 kubelet.go:2267] node "master" not found Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.932104 34678 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User "system:anonymous" cannot list resource "csidrivers" in API group "storage.k8s.io" at the cluster scope Dec 21 14:56:19 master kubelet[34678]: E1221 14:56:19.963637 34678 kubelet.go:2267] node "master" not found Dec 21 14:56:20 master kubelet[34678]: E1221 14:56:20.063823 34678 kubelet.go:2267] node "master" not found Dec 21 14:56:20 master kubelet[34678]: E1221 14:56:20.133218 34678 reflector.go:123] k8s.io/client-go/informers/factory.go:134: Failed to list *v1beta1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User "system:anonymous" cannot list resource "runtimeclasses" in API group "node.k8s.io" at the cluster scope Dec 21 14:56:20 master kubelet[34678]: E1221 14:56:20.164019 34678 kubelet.go:2267] node "master" not found Dec 21 14:56:20 master kubelet[34678]: E1221 14:56:20.264632 34678 kubelet.go:2267] node "master" not found Dec 21 14:56:20 master kubelet[34678]: E1221 14:56:20.332795 34678 reflector.go:123] k8s.io/kubernetes/pkg/kubelet/kubelet.go:450: Failed to list *v1.Service: services is forbidden: User "system:anonymous" cannot list resource "services" in API group "" at the cluster scope Dec 21 14:56:20 master kubelet[34678]: E1221 14:56:20.364844 34678 kubelet.go:2267] node "master" not found然而此时master的kubelet 是runing的，但是没有状态还是没有ready,接着我重启了docker程序，仍然没有进展，反倒让proxy和coredns，calico Pending了。而etcd,controlle,api-server却是正常running的。由于项目太旧，没有任何文档，半年前做过证书更新，于是查看证书[root@master ]# kubeadm alpha certs check-expiration CERTIFICATE EXPIRES RESIDUAL TIME EXTERNALLY MANAGED admin.conf Dec 22, 2030 06:19 UTC 9y no apiserver Dec 22, 2030 06:19 UTC 9y no apiserver-etcd-client Dec 22, 2030 06:19 UTC 9y no apiserver-kubelet-client Dec 22, 2030 06:19 UTC 9y no controller-manager.conf Dec 22, 2030 06:19 UTC 9y no etcd-healthcheck-client Dec 22, 2030 06:19 UTC 9y no etcd-peer Dec 22, 2030 06:19 UTC 9y no etcd-server Dec 22, 2030 06:19 UTC 9y no front-proxy-client Dec 22, 2030 06:19 UTC 9y no scheduler.conf Dec 22, 2030 06:19 UTC 9y no发现证书也没有问题而后我们确认master是否配置正确[root@master kubernetes]# cat kubelet.conf apiVersion: v1 clusters: - cluster: certificate-authority-data: server: https://master:6443 name: kubernetes contexts: - context: cluster: kubernetes user: kubernetes-admin name: kubernetes-admin@kubernetes current-context: kubernetes-admin@kubernetes kind: Config preferences: {} users: - name: kubernetes-admin user: client-certificate-data: client-key-data: L在查看hosts[root@master kubernetes]# cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.133.6 master一切都没有问题，现在怀疑是kubelet.conf的证书权限有问题，于是将node的kubelet拿到master尝试。发现还是没有解决最后在尝试将master的admin.conf替换成kubelet.conf[root@master kubernetes]# systemctl restart kubelet查看日志[root@master kubernetes]# journalctl -u kubelet -f -- Logs begin at Sun 2021-10-31 03:30:27 CST. -- Dec 21 15:15:48 master kubelet[39135]: I1221 15:15:48.782529 39135 reconciler.go:301] Volume detached for volume "calico-node-token-sfzfw" (UniqueName: "kubernetes.io/secret/b7039ad3-f568-47b7-a323-737898628333-calico-node-token-sfzfw") on node "master" DevicePath "" Dec 21 15:15:48 master kubelet[39135]: I1221 15:15:48.782541 39135 reconciler.go:301] Volume detached for volume "coredns-token-6rmpw" (UniqueName: "kubernetes.io/secret/bf57f81b-911a-48b7-aa1e-624193cd7c51-coredns-token-6rmpw") on node "master" DevicePath "" Dec 21 15:15:48 master kubelet[39135]: I1221 15:15:48.782552 39135 reconciler.go:301] Volume detached for volume "config-volume" (UniqueName: "kubernetes.io/configmap/f4d237c7-2212-4c91-8535-4cad2bb88644-config-volume") on node "master" DevicePath "" Dec 21 15:15:48 master kubelet[39135]: I1221 15:15:48.782564 39135 reconciler.go:301] Volume detached for volume "coredns-token-6rmpw" (UniqueName: "kubernetes.io/secret/f4d237c7-2212-4c91-8535-4cad2bb88644-coredns-token-6rmpw") on node "master" DevicePath "" Dec 21 15:15:48 master kubelet[39135]: W1221 15:15:48.993096 39135 kubelet_getters.go:292] Path "/var/lib/kubelet/pods/3114871e-b583-4e02-a8f3-b63a25b2dffa/volumes" does not exist Dec 21 15:15:49 master kubelet[39135]: W1221 15:15:49.215461 39135 kubelet_getters.go:292] Path "/var/lib/kubelet/pods/02f1318a-3b06-48cf-a96e-83322f66c17b/volumes" does not exist Dec 21 15:15:49 master kubelet[39135]: W1221 15:15:49.215524 39135 kubelet_getters.go:292] Path "/var/lib/kubelet/pods/b7039ad3-f568-47b7-a323-737898628333/volumes" does not exist Dec 21 15:15:49 master kubelet[39135]: W1221 15:15:49.215548 39135 kubelet_getters.go:292] Path "/var/lib/kubelet/pods/f4d237c7-2212-4c91-8535-4cad2bb88644/volumes" does not exist Dec 21 15:15:49 master kubelet[39135]: W1221 15:15:49.215570 39135 kubelet_getters.go:292] Path "/var/lib/kubelet/pods/83515e8e-8ba5-459d-9bc6-3d897a4f18e9/volumes" does not exist Dec 21 15:15:49 master kubelet[39135]: W1221 15:15:49.215593 39135 kubelet_getters.go:292] Path "/var/lib/kubelet/pods/bf57f81b-911a-48b7-aa1e-624193cd7c51/volumes" does not exist Dec 21 15:15:49 master kubelet[39135]: I1221 15:15:49.384045 39135 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "lib-modules" (UniqueName: "kubernetes.io/host-path/809554bc-4e5c-4645-b8dd-9963182a8196-lib-modules") pod "calico-node-k4vx7" (UID: "809554bc-4e5c-4645-b8dd-9963182a8196") Dec 21 15:15:49 master kubelet[39135]: I1221 15:15:49.384101 39135 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "xtables-lock" (UniqueName: "kubernetes.io/host-path/809554bc-4e5c-4645-b8dd-9963182a8196-xtables-lock") pod "calico-node-k4vx7" (UID: "809554bc-4e5c-4645-b8dd-9963182a8196") Dec 21 15:15:49 master kubelet[39135]: I1221 15:15:49.384152 39135 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "flexvol-driver-host" (UniqueName: "kubernetes.io/host-path/809554bc-4e5c-4645-b8dd-9963182a8196-flexvol-driver-host") pod "calico-node-k4vx7" (UID: "809554bc-4e5c-4645-b8dd-9963182a8196") Dec 21 15:15:49 master kubelet[39135]: I1221 15:15:49.384194 39135 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "cni-net-dir" (UniqueName: "kubernetes.io/host-path/809554bc-4e5c-4645-b8dd-9963182a8196-cni-net-dir") pod "calico-node-k4vx7" (UID: "809554bc-4e5c-4645-b8dd-9963182a8196") Dec 21 15:15:49 master kubelet[39135]: I1221 15:15:49.384233 39135 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "host-local-net-dir" (UniqueName: "kubernetes.io/host-path/809554bc-4e5c-4645-b8dd-9963182a8196-host-local-net-dir") pod "calico-node-k4vx7" (UID: "809554bc-4e5c-4645-b8dd-9963182a8196") Dec 21 15:15:49 master kubelet[39135]: I1221 15:15:49.384271 39135 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "policysync" (UniqueName: "kubernetes.io/host-path/809554bc-4e5c-4645-b8dd-9963182a8196-policysync") pod "calico-node-k4vx7" (UID: "809554bc-4e5c-4645-b8dd-9963182a8196") Dec 21 15:15:49 master kubelet[39135]: I1221 15:15:49.384305 39135 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "calico-node-token-sfzfw" (UniqueName: "kubernetes.io/secret/809554bc-4e5c-4645-b8dd-9963182a8196-calico-node-token-sfzfw") pod "calico-node-k4vx7" (UID: "809554bc-4e5c-4645-b8dd-9963182a8196") Dec 21 15:15:49 master kubelet[39135]: I1221 15:15:49.384332 39135 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "cni-bin-dir" (UniqueName: "kubernetes.io/host-path/809554bc-4e5c-4645-b8dd-9963182a8196-cni-bin-dir") pod "calico-node-k4vx7" (UID: "809554bc-4e5c-4645-b8dd-9963182a8196") Dec 21 15:15:49 master kubelet[39135]: I1221 15:15:49.384369 39135 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "var-run-calico" (UniqueName: "kubernetes.io/host-path/809554bc-4e5c-4645-b8dd-9963182a8196-var-run-calico") pod "calico-node-k4vx7" (UID: "809554bc-4e5c-4645-b8dd-9963182a8196") Dec 21 15:15:49 master kubelet[39135]: I1221 15:15:49.384398 39135 reconciler.go:207] operationExecutor.VerifyControllerAttachedVolume started for volume "var-lib-calico" (UniqueName: "kubernetes.io/host-path/809554bc-4e5c-4645-b8dd-9963182a8196-var-lib-calico") pod "calico-node-k4vx7" (UID: "809554bc-4e5c-4645-b8dd-9963182a8196") Dec 21 15:15:49 master kubelet[39135]: W1221 15:15:49.986228 39135 kubelet_getters.go:292] Path "/var/lib/kubelet/pods/754ff5ae-8b4f-4f67-9027-68b5f59a8b90/volumes" does not exist Dec 21 15:15:50 master kubelet[39135]: I1221 15:15:50.387442 39135 reconciler.go:207]问题暂时得到解决[root@master kubernetes]# kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system calico-kube-controllers-dc4d48847-mmwdk 1/1 Running 0 4h30m kube-system calico-node-9z59b 1/1 Running 0 21m kube-system calico-node-k4vx7 1/1 Running 0 22m kube-system calico-node-xpktb 1/1 Running 0 21m kube-system coredns-dffb59cff-4tnrn 1/1 Running 0 4h30m kube-system coredns-dffb59cff-hjt7v 1/1 Running 0 4h30m kube-system eip-nfs-minio-6c44f59b76-nq8fm 1/1 Running 4 237d kube-system etcd-master 1/1 Running 5 362d kube-system kube-apiserver-master 1/1 Running 3 362d kube-system kube-controller-manager-master 1/1 Running 244 362d kube-system kube-proxy-7d5qb 1/1 Running 0 22m kube-system kube-proxy-k755k 1/1 Running 0 21m kube-system kube-proxy-pxhb6 1/1 Running 0 22m kube-system kube-scheduler-master 1/1 Running 241 362d kube-system kuboard-7d745b566-wwp8w 1/1 Running 0 42d kube-system metrics-server-7dcd7b44c-dvpp4 1/1 Running 0 87d[root@master kubernetes]# kubectl get node NAME STATUS ROLES AGE VERSION master Ready,SchedulingDisabled master 647d v1.16.3 worker1 Ready <none> 647d v1.16.3 worker2 Ready <none> 647d v1.16.3[root@master kubernetes]# kubectl describe node master Name: master Roles: master ... Unschedulable: true Conditions: Type Status LastHeartbeatTime LastTransitionTime Reason Message ---- ------ ----------------- ------------------ ------ ------- NetworkUnavailable False Tue, 21 Dec 2021 15:15:52 +0800 Tue, 21 Dec 2021 15:15:52 +0800 CalicoIsUp Calico is running on this node MemoryPressure False Tue, 21 Dec 2021 15:46:58 +0800 Tue, 21 Dec 2021 15:15:47 +0800 KubeletHasSufficientMemory kubelet has sufficient memory available DiskPressure False Tue, 21 Dec 2021 15:46:58 +0800 Tue, 21 Dec 2021 15:15:47 +0800 KubeletHasNoDiskPressure kubelet has no disk pressure PIDPressure False Tue, 21 Dec 2021 15:46:58 +0800 Tue, 21 Dec 2021 15:15:47 +0800 KubeletHasSufficientPID kubelet has sufficient PID available Ready True Tue, 21 Dec 2021 15:46:58 +0800 Tue, 21 Dec 2021 15:15:47 +0800 KubeletReady kubelet is posting ready status Addresses: InternalIP: 192.168.133.6 Hostname: master Capacity: cpu: 4 ephemeral-storage: 36805060Ki hugepages-2Mi: 0 memory: 16259676Ki pods: 110 Allocatable: cpu: 4 ephemeral-storage: 33919543240 hugepages-2Mi: 0 memory: 15211100Ki pods: 110 System Info: Machine ID: abfbcf67c72dca40affb0cd0c8debd6d System UUID: 63B3A0C6-AEB7-3841-BA81-2724178E0890 Boot ID: 02424d77-4b14-45ea-a632-038dbc70a2c4 Kernel Version: 3.10.0-957.el7.x86_64 OS Image: CentOS Linux 7 (Core) Operating System: linux Architecture: amd64 Container Runtime Version: docker://18.9.7 Kubelet Version: v1.16.3 Kube-Proxy Version: v1.16.3 PodCIDR: 10.100.0.0/24 PodCIDRs: 10.100.0.0/24 Non-terminated Pods: (6 in total) Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits AGE --------- ---- ------------ ---------- --------------- ------------- --- kube-system calico-node-k4vx7 250m (6%) 0 (0%) 0 (0%) 0 (0%) 31m kube-system etcd-master 0 (0%) 0 (0%) 0 (0%) 0 (0%) 362d kube-system kube-apiserver-master 250m (6%) 0 (0%) 0 (0%) 0 (0%) 362d kube-system kube-controller-manager-master 200m (5%) 0 (0%) 0 (0%) 0 (0%) 362d kube-system kube-proxy-pxhb6 0 (0%) 0 (0%) 0 (0%) 0 (0%) 31m kube-system kube-scheduler-master 100m (2%) 0 (0%) 0 (0%) 0 (0%) 362d Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 800m (20%) 0 (0%) memory 0 (0%) 0 (0%) ephemeral-storage 0 (0%) 0 (0%) Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Starting 31m kubelet, master Starting kubelet. Normal NodeHasSufficientMemory 31m (x2 over 31m) kubelet, master Node master status is now: NodeHasSufficientMemory Normal NodeHasNoDiskPressure 31m (x2 over 31m) kubelet, master Node master status is now: NodeHasNoDiskPressure Normal NodeHasSufficientPID 31m (x2 over 31m) kubelet, master Node master status is now: NodeHasSufficientPID Normal NodeNotReady 31m kubelet, master Node master status is now: NodeNotReady Normal NodeNotSchedulable 31m kubelet, master Node master status is now: NodeNotSchedulable Normal NodeAllocatableEnforced 31m kubelet, master Updated Node Allocatable limit across pods Normal NodeReady 31m kubelet, master Node master status is now: NodeReady Normal Starting 31m kube-proxy, master Starting kube-proxy.