K8s --The connection to the server 192.168.56.120:6443 was refused报错解决

news/2024/10/18 10:17:47/

K8s The connection to the server 192.168.56.120:6443 was refused报错解决

  • 背景说明
  • 环境说明
  • 部署信息
  • 场景复现
  • 问题排查
    • 基础环境排查
    • 组件服务排查
  • 根因定位
  • 问题解决
    • 安装go环境
    • 下载k8s源码
    • 修改CA有效期
    • 修改证书有效期
    • 编译 kubeadm
    • 更新证书
    • 检查结果

背景说明

自己本地虚拟机有一套使用kuebadm部署的3节点k8s集群，因为很长时间不使用了，今天登陆后在使用kubectl命令时一直报The connection to the server 192.168.56.120:6443 was refused - did you specify the right host or port?错误，接下来就将解决的整个过程分享给大家。

环境说明

操作系统信息以及k8s集群版本信息请看下面

[root@k8s-master][/usr/local]
$cat /etc/redhat-release 
CentOS Linux release 7.7.1908 (Core)[root@k8s-master][/usr/local]
$kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T18:16:20Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"linux/amd64"}

部署信息

序号ip角色
1192.168.56.120k8s-master
2192.168.56.121k8s-node1
3192.168.56.122k8s-node2

场景复现

[root@k8s-master][/usr/local]
$kubectl get node
The connection to the server 192.168.56.120:6443 was refused - did you specify the right host or port?

问题排查

基础环境排查

#1 基础环境排查
#1.0 端口排查（端口已经不存在）
[root@k8s-master][~]
$ss -antpl |grep 6443#1.1 查看内存（都够用）
[root@k8s-master][~]
$free -htotal        used        free      shared  buff/cache   available
Mem:           4.7G        1.2G        366M         58M        3.1G        3.1G
Swap:            0B          0B          0B#1.2 查看存储（也没问题）
[root@k8s-master][~]
$df -h
Filesystem               Size  Used Avail Use% Mounted on
devtmpfs                 2.4G     0  2.4G   0% /dev
tmpfs                    2.4G     0  2.4G   0% /dev/shm
tmpfs                    2.4G   58M  2.3G   3% /run
tmpfs                    2.4G     0  2.4G   0% /sys/fs/cgroup
/dev/mapper/centos-root   37G   22G   16G  59% /#1.3 查看防火墙（防火墙是关闭的）
[root@k8s-master][~]
$systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemonLoaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)Active: inactive (dead)Docs: man:firewalld(1)

组件服务排查

#2 排查组件
#2.1 查看docker信息（没有报错信息）
[root@k8s-master][~]
$systemctl status docker#2.2 查看kubelet（发现master节点丢失的信息，但是内容不详细）
[root@k8s-master][/usr/local]
$systemctl status kubeletJul 17 10:45:45 k8s-master kubelet[14485]: E0717 10:45:45.603977   14485 kubelet.go:2422] "Error getting node" err="node \"k8s-master\" not found"
Jul 17 10:45:45 k8s-master kubelet[14485]: E0717 10:45:45.634017   14485 controller.go:144] failed to ensure lease exists, will retry in 7s, error: Get "https://192.168....ion refused#3 查看etcd情况
[root@k8s-master][~]
$docker ps |grep k8s_etcd_etcd#从以下的信息可以看出证书已经过期了，在确认一遍
[root@k8s-master][~]
$docker logs dc2c0e42972e
{"level":"warn","ts":"2024-07-17T02:42:22.385Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:34074","server-name":"","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-07-17T02:43:33.346Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:34428","server-name":"","error":"tls: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2024-07-17T02:43:33Z is after 2024-03-14T07:34:55Z"}#4 查看证书过期情况（发现已经过期了，到此找到报错根因）
[root@k8s-master][~]
$kubeadm certs check-expiration

根因定位

#从以下的信息可以看出证书过期，导致api-server服务未启动
[root@k8s-master][~]
$docker logs dc2c0e42972e
{"level":"warn","ts":"2024-07-17T02:42:22.385Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:34074","server-name":"","error":"remote error: tls: bad certificate"}
{"level":"warn","ts":"2024-07-17T02:43:33.346Z","caller":"embed/config_logging.go:169","msg":"rejected connection","remote-addr":"127.0.0.1:34428","server-name":"","error":"tls: failed to verify client certificate: x509: certificate has expired or is not yet valid: current time 2024-07-17T02:43:33Z is after 2024-03-14T07:34:55Z"}

问题解决

既然已经定位到是证书过期的问题，那接下来就针对证书过期进行解决。

安装go环境

# 如果 k8s 版本较高，则需要 go 语言版本 1.19 以上(尽量使用和k8s版本一致的go版本，要不会报很多错误)
kubectl version
[root@k8s-master][~]
$kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T18:16:20Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"linux/amd64"}#1  下载 go 语言软件包
mkdir -p /root/pac
cd /root/pac
wget https://studygolang.com/dl/golang/go1.17.3.linux-amd64.tar.gz#2 解压并配置环境变量
tar xf go1.19.1.linux-amd64.tar.gz -C /usr/localvim /etc/profile
export GOROOT=/usr/local/go
export PATH=$PATH:/usr/local/go/bin
export GOPATH=/gosource /etc/profile#3 验证
[root@k8s-master][~]
$go version
go version go1.17.3 linux/amd64

下载k8s源码

#1 查看当前环境 k8s 版本
kubectl version#2 下载源码
cd /root/pac
wget https://github.com/kubernetes/kubernetes/archive/refs/tags/v1.23.0.tar.gz#2 解压
tar xf v1.23.0.tar.gz

修改CA有效期

#1 进入到指定目录
[root@k8s-master][~/xxx/go/kubernetes-1.23.0/staging/src/k8s.io/client-go/util/cert]
# cd /root/xxx/go/kubernetes-1.23.0/staging/src/k8s.io/client-go/util/cert#2 备份cert.go
[root@k8s-master][~/xxx/go/kubernetes-1.23.0/staging/src/k8s.io/client-go/util/cert]
# cp cert.go{,.bak}#3 修改cert.go
[root@k8s-master][~/xxx/go/kubernetes-1.23.0/staging/src/k8s.io/client-go/util/cert]
# vim cert.go// NewSelfSignedCACert creates a CA certificate
func NewSelfSignedCACert(cfg Config, key crypto.Signer) (*x509.Certificate, error) {now := time.Now()tmpl := x509.Certificate{SerialNumber: new(big.Int).SetInt64(0),Subject: pkix.Name{CommonName:   cfg.CommonName,Organization: cfg.Organization,},DNSNames:              []string{cfg.CommonName},NotBefore:             now.UTC(),//NotAfter:              now.Add(duration365d * 10).UTC(),// 修改为 100 年，默认为 10 年NotAfter:              now.Add(duration365d * 100).UTC(),KeyUsage:              x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign,BasicConstraintsValid: true,IsCA:                  true,}certDERBytes, err := x509.CreateCertificate(cryptorand.Reader, &tmpl, &tmpl, key.Public(), key)if err != nil {return nil, err}return x509.ParseCertificate(certDERBytes)
}

修改证书有效期

[root@k8s-master][~/xxx/go/kubernetes-1.23.0/staging/src/k8s.io/client-go/util/cert]
# cd /root/xxx/go/kubernetes-1.23.0/cmd/kubeadm/app/constants#2 备份constants.go
[root@k8s-master][~/xxx/go/kubernetes-1.23.0/cmd/kubeadm/app/constants]
# cp constants.go{,.bak}#3 修改文件
[root@k8s-master][~/xxx/go/kubernetes-1.23.0/cmd/kubeadm/app/constants]
# vim constants.goconst (// KubernetesDir is the directory Kubernetes owns for storing various configuration filesKubernetesDir = "/etc/kubernetes"// ManifestsSubDirName defines directory name to store manifestsManifestsSubDirName = "manifests"// TempDirForKubeadm defines temporary directory for kubeadm// should be joined with KubernetesDir.TempDirForKubeadm = "tmp"// CertificateValidity defines the validity for all the signed certificates generated by kubeadm//CertificateValidity = time.Hour * 24 * 365//修改为 100 年，默认为 1 年CertificateValidity = time.Hour * 24 * 365 * 100// DefaultCertificateDir defines default certificate directoryDefaultCertificateDir = "pki"// CACertAndKeyBaseName defines certificate authority base nameCACertAndKeyBaseName = "ca"// CACertName defines certificate nameCACertName = "ca.crt"// CAKeyName defines certificate name

编译 kubeadm

# 提前装rsync
yum install rsync -y#1 重新编译kubeadm
[root@k8s-master][~/xxx/go/kubernetes-1.23.0]
# pwd
/root/xxx/go/kubernetes-1.23.0[root@k8s-master][~/xxx/go/kubernetes-1.23.0]
# make all WHAT=cmd/kubeadm GOFLAGS=-v#2 替换原来的 kubeadm
[root@k8s-master][~]
# mv /usr/bin/kubeadm{,.bak}[root@k8s-master][~/xxx/go/kubernetes-1.23.0]
# pwd
/root/xxx/go/kubernetes-1.23.0[root@k8s-master][~/xxx/go/kubernetes-1.23.0]
# ll _output/local/bin/linux/amd64/kubeadm
-rwxr-xr-x. 1 root root 43667456 Mar  6 11:26 _output/local/bin/linux/amd64/kubeadm[root@k8s-master][~/xxx/go/kubernetes-1.23.0]
# cp _output/local/bin/linux/amd64/kubeadm /usr/bin/kubeadm[root@k8s-master][~/xxx/go/kubernetes-1.23.0]
# chmod +x /usr/bin/kubeadm

更新证书

#1 备份原文件
[root@k8s-master][~]
# cp -a /etc/kubernetes{,.bak_20240717}#2 更新证书
[root@k8s-master][/etc/kubernetes]
# kubeadm certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewedDone renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

检查结果

[root@k8s-master][~]
$kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jun 23, 2124 02:49 UTC   99y                                     no      
apiserver                  Jun 23, 2124 02:49 UTC   99y             ca                      no      
apiserver-etcd-client      Jun 23, 2124 02:49 UTC   99y             etcd-ca                 no      
apiserver-kubelet-client   Jun 23, 2124 02:49 UTC   99y             ca                      no      
controller-manager.conf    Jun 23, 2124 02:49 UTC   99y                                     no      
etcd-healthcheck-client    Jun 23, 2124 02:49 UTC   99y             etcd-ca                 no      
etcd-peer                  Jun 23, 2124 02:49 UTC   99y             etcd-ca                 no      
etcd-server                Jun 23, 2124 02:49 UTC   99y             etcd-ca                 no      
front-proxy-client         Jun 23, 2124 02:49 UTC   99y             front-proxy-ca          no      
scheduler.conf             Jun 23, 2124 02:49 UTC   99y                                     no      CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Mar 12, 2033 07:34 UTC   8y              no      
etcd-ca                 Mar 12, 2033 07:34 UTC   8y              no      
front-proxy-ca          Mar 12, 2033 07:34 UTC   8y              no

再次使用kubectl命令，看看问题是否解决

[root@k8s-master][~]
$kubectl get node
NAME         STATUS   ROLES                  AGE    VERSION
k8s-master   Ready    control-plane,master   489d   v1.23.0
k8s-node1    Ready    <none>                 489d   v1.23.0
k8s-node2    Ready    <none>                 489d   v1.23.0

至此，问题得到解决。


http://www.ppmy.cn/news/1476837.html

相关文章

Nginx的HA高可用的搭建

Nginx的HA高可用的搭建

1. 什么是高可用 高可用&#xff08;High Availability, HA&#xff09;是一种系统设计策略&#xff0c;旨在确保服务或应用在面对硬件故障、软件缺陷或任何其他异常情况时&#xff0c;仍能持续稳定地运行。它通过实现冗余性、故障转移、负载均衡、数据一致性、监控自动化、预防…
阅读更多...
阿里云服务器 篇三：提交搜索引擎收录

阿里云服务器 篇三：提交搜索引擎收录

文章目录 系列文章推荐：为网站注册域名判断网站是否已被搜索引擎收录主动提交搜索引擎收录未查询到收录结果时，根据提示进行提交网站提交网站时一般需要登录账号主动提交网站可缩短爬虫发现网站链接时间，但不保证一定能够收录所提交的网站百度提交地址360搜索提交地址搜狗提…
阅读更多...
【乐吾乐2D可视化组态编辑器】快捷键

【乐吾乐2D可视化组态编辑器】快捷键

快捷键 乐吾乐2D可视化组态编辑器demo&#xff1a;https://2d.le5le.com/ 快捷键描述空格 鼠标拖拽移动画布鼠标右键拖拽移动画布Ctrl 滚轮缩放画布Ctrl 点击 Pen多选Ctrl A全选Ctrl C复制Ctrl X剪切Ctrl V粘贴&#xff0c;alt视图中心粘贴&#xff0c;shift原位粘贴…
阅读更多...
C#qq邮件发送

C#qq邮件发送

先授权自己的qq邮箱 给咱一个授权码,记住授权码 需要在代码进行使用(不是qq密码) 设置SMTP客户端 using System.Net; using System.Net.Mail;public void SendQQMail(string host,int port,string fromEnail,string password,string toEmail,string subject,string body){Smtp…
阅读更多...
JavaScript之Web APIs-DOM

JavaScript之Web APIs-DOM

目录 DOM获取元素一、Web API 基本认知1.1 变量声明1.2 作用和分类1.3 DOM树1.4 DOM对象 二、获取DOM对象2.1 通过CSS选择器来获取DOM元素2.2 通过其他方式来获取DOM元素 三、操作元素内容3.1 元素.innerTest属性3.2 元素.innerHTML属性 四、操作元素属性4.1 操作元素常用属性4…
阅读更多...
Chromium CI/CD 之Jenkins实用指南2024-在Windows节点上创建任务（九）

Chromium CI/CD 之Jenkins实用指南2024-在Windows节点上创建任务（九）

1. 引言 在现代软件开发流程中&#xff0c;持续集成&#xff08;CI&#xff09;和持续交付&#xff08;CD&#xff09;已成为确保代码质量和加速发布周期的关键实践。Jenkins作为一款广泛应用的开源自动化服务器&#xff0c;通过其强大的插件生态系统和灵活的配置选项&#xf…
阅读更多...
Keka for Mac v1.4.3 中文下载 解压/压缩工具

Keka for Mac v1.4.3 中文下载 解压/压缩工具

Mac分享吧 文章目录 效果一、下载软件二、开始安装1、双击运行软件&#xff0c;将其从左侧拖入右侧文件夹中&#xff0c;等待安装完毕2、应用程序显示软件图标&#xff0c;表示安装成功 三、运行测试1、打开软件2、文件访问权限修改3、访达扩展 安装完成&#xff01;&#xff…
阅读更多...
单例模式~

单例模式~

概要 确保一个类只有一个实例&#xff0c;并提供一个全局访问点来访问该实例。 主要解决的问题 频繁创建和销毁全局使用的实例&#xff0c;减少开销。 解决思路 如果有该实例就返回&#xff0c;没有就创建。 应用场景 数据库连接池、线程池、spring中service等 几种实现…
阅读更多...
最新文章

Copyright @ 2022~2023