k8s问题记录与解决

一、问题：error: open /var/lib/kubelet/config.yaml: no such file or directory
解决：关键文件缺失，多发生于没有做 kubeadm init就运行了systemctl start kubelet。要先成功运行kubeadm init

二、kubelet.service has more than one ExecStart= setting, which is only allowed for Type=oneshot services. Refusing.
解决：打开/etc/systemd/system/kubelet.service.d/10-kubeadm.conf 中的配置：
   [root@k8s-master ~]# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
   # Note: This dropin only works with kubeadm and kubelet v1.11+
   [Service]
   Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
   Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
   # This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
   EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
   # This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
   # the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
   EnvironmentFile=-/etc/sysconfig/kubelet
   要打开注释此# ExecStart=
   ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

三、journalctl -f -u kubelet （-f是 --follow, -u是过滤出kubelet日志）
centos7 查看日志

四、kubeadm安装的k8s，重新安装k8s-mst
   检查 /etc/systemd/system/kubelet.service.d/10-kubeadm.conf 中的配置
   systemctl restart kubelet
   kubeadm reset
   kubeadm init --kubernetes-version=v1.13.4 --pod-network-cidr=10.244.0.0/16 --service-cidr=10.96.0.0/12 --ignore-preflight-errors=Swap
   其他节点执行
   kubeadm reset
   kubeadm join ...命令

五、kubeadm 生成的token过期后，集群增加节点
参考：https://www.jianshu.com/p/a5e379638577

六、[reset] unmounting mounted directories in "/var/lib/kubelet" 卡主
重启机器，重新执行命令

-------------------20190518 update-------------------

七、master所在虚机重启后，kube-master1 kubelet[34770]: E0419 13:52:09.511348 34770 kubelet.go:2266] node "kube-master1" not found，并发现获取到的ip地址为空，ifconfig命令去查看网卡配置情况，却发现根本没有配置eth0/ens33网卡

解决：依次systemctl stop kubelet、systemctl stop docker、systemctl restart network、systemctl restart docker、systemctl restart kubelet、ifconfig、kubectl get node

-------------------20190614 update-------------------

八、kubelet, k8s-node1 Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "1b7fb9d83e89dbe2815cc10fb1daf342162cb74da30568c0a59585e1dc9329a4" network for pod "wxapp-redis-1": NetworkPlugin cni failed to set up pod "wxapp-redis-1_pnup" network: failed to set bridge addr: "cni0" already has an IP address different from 10.244.1.1/24

解决：到有问题的机器，执行如下命令：

[root@k8s-node1 redis]# cd /var/lib/cni/flannel/
[root@k8s-node1 flannel]# ll
total 32
-rw-------. 1 root root 206 Apr 19 11:30 0727fe1a742f28b9a5d5d3188496bdc0aec220599caf6ce8c28f1b9c8ef1b8d4
-rw-------. 1 root root 206 Apr 19 14:30 13776cebe870d3f58982a123e2e32a4a89780c421db1bc425f13f13756822f81
-rw-------. 1 root root 206 Apr 19 11:30 1f249dd31dae8177a4fa5d3009eec7a36a4dccd8a836975f1d798adf43afda51
-rw-------. 1 root root 206 Apr 19 14:34 317e1ce2f78fddd1b36879be5dff169d27fefd4ca191dcd9d85781ab65cc14d8
-rw-------. 1 root root 187 Jun 14 10:05 5b104d16ea2042bd67f8958b8042fff01de3ba6a69f1a62f4b3ded81955c24bb
-rw-------. 1 root root 206 Apr 23 15:30 5bfe39c5fb73ecb09b7343260b8fc2526bb99b7d00a216ab0c76d87d247f3bc0
-rw-------. 1 root root 187 Jun 14 10:05 85de9489e5a4b5091bc40b5dc216a9f15fdb9aa077a28df32cfef97f8abd0c81
-rw-------. 1 root root 206 Apr 19 11:30 db015f887bbedf2ad7731aaa4a321183594a1d2c9b95398bb25f61ad5b052092
[root@k8s-node1 flannel]# systemctl stop docker
[root@k8s-node1 flannel]# systemctl stop kubelet
[root@k8s-node1 flannel]# systemctl stop kube-proxy
Failed to stop kube-proxy.service: Unit kube-proxy.service not loaded.
[root@k8s-node1 flannel]# rm -rf /var/lib/cni/flannel/ && rm -rf /var/lib/cni/networks/cbr0/ && ip link delete cni0
[root@k8s-node1 flannel]# rm -rf /var/lib/cni/networks/cni0/*
[root@k8s-node1 flannel]# systemctl start docker
[root@k8s-node1 flannel]# systemctl start kubelet

-------------------20190622 update-------------------

九、转（kubernetes --> kube-dns 安装 https://blog.csdn.net/zhuchuangang/article/details/76093887 https://www.cnblogs.com/chimeiwangliang/p/8809280.html）

十、转（kubernetes中网络报错问题排查 http://www.mamicode.com/info-detail-2315259.html）

-------------------20200523 update-------------------

十一、May 23 06:21:59 master dockerd-current[14690]: time="2020-05-23T06:21:59.555312805-04:00" level=error msg="Create container failed with error: oci runtime error: container_linux.go:235: starting container process caused \"process_linux.go:258: applying cgroup configuration for process caused \\\"Cannot set property TasksAccounting, or unknown property.\\\"\"\n"
May 23 06:21:59 master kubelet[21888]: E0523 06:21:59.654065 21888 kubelet.go:2266] node "master" not found
May 23 06:21:59 master kubelet[21888]: E0523 06:21:59.754397 21888 kubelet.go:2266] node "master" not found
May 23 06:21:59 master kubelet[21888]: E0523 06:21:59.855185 21888 kubelet.go:2266] node "master" not found
Docker创建容器报错：Cannot set property TasksAccounting, or unknown property.
最近又新配了一个服务器，想用docker简单的配置一下mysql，没想到创建容器时报错：
Error response from daemon: oci runtime error: container_linux.go:235: starting container process caused “process_linux.go:258: applying cgroup configuration for process caused “Cannot set property TasksAccounting, or unknown property.””
问题原因：主要原因还是centos系统版本兼容性问题，如果将系统做更新升级，即可解决。
执行：yum update后在执行以下操作：

[root@web03 .kube]# yum remove -y kubectl.x86_64 kubeadm.x86_64 kubelet.x86_64
Loaded plugins: fastestmirror, langpacks, product-id, search-disabled-repos, subscription-manager
...
Complete!
[root@web03 .kube]# yum list installed |grep kube
cri-tools.x86_64                      1.13.0-0                       @kubernetes
kubectl.x86_64                        1.13.4-0                       @kubernetes
[root@web03 .kube]# yum install -y kubelet-1.13.4-0.x86_64
Loaded plugins: fastestmirror, langpacks, product-id, search-disabled-repos, subscription-manager
...
Complete!
[root@web03 .kube]# yum install -y kubeadm-1.13.4-0.x86_64
Loaded plugins: fastestmirror, langpacks, product-id, search-disabled-repos, subscription-manager
...
Complete!
[root@web03 .kube]# yum list installed |grep kube
cri-tools.x86_64                      1.13.0-0                       @kubernetes
kubeadm.x86_64                        1.13.4-0                       @kubernetes
kubectl.x86_64                        1.13.4-0                       @kubernetes
kubelet.x86_64                        1.13.4-0                       @kubernetes
kubernetes-cni.x86_64                 0.6.0-0                        @kubernetes

十二、k8s使用kube-router网络插件并监控流量状态
https://www.jianshu.com/p/1a3caecc3b6b
附：kubeadm-kuberouter-all-features.yaml

apiVersion: v1
kind: ConfigMap
metadata:name: kube-router-cfgnamespace: kube-systemlabels:tier: nodek8s-app: kube-router
data:cni-conf.json: |{"cniVersion":"0.3.0","name":"mynet","plugins":[{"name":"kubernetes","type":"bridge","bridge":"kube-bridge","isDefaultGateway":true,"ipam":{"type":"host-local"}}]}
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:labels:k8s-app: kube-routertier: nodename: kube-routernamespace: kube-system
spec:template:metadata:labels:k8s-app: kube-routertier: nodeannotations:scheduler.alpha.kubernetes.io/critical-pod: ''spec:serviceAccountName: kube-routerserviceAccount: kube-routercontainers:- name: kube-routerimage: docker.io/cloudnativelabs/kube-routerimagePullPolicy: IfNotPresentargs:- --run-router=true- --run-firewall=true- --run-service-proxy=true- --kubeconfig=/var/lib/kube-router/kubeconfigenv:- name: NODE_NAMEvalueFrom:fieldRef:fieldPath: spec.nodeName- name: KUBE_ROUTER_CNI_CONF_FILEvalue: /etc/cni/net.d/10-kuberouter.conflistlivenessProbe:httpGet:path: /healthzport: 20244initialDelaySeconds: 10periodSeconds: 3resources:requests:cpu: 250mmemory: 250MisecurityContext:privileged: truevolumeMounts:- name: lib-modulesmountPath: /lib/modulesreadOnly: true- name: cni-conf-dirmountPath: /etc/cni/net.d- name: kubeconfigmountPath: /var/lib/kube-routerreadOnly: trueinitContainers:- name: install-cniimage: busyboximagePullPolicy: Alwayscommand:- /bin/sh- -c- set -e -x;if [ ! -f /etc/cni/net.d/10-kuberouter.conflist ]; thenif [ -f /etc/cni/net.d/*.conf ]; thenrm -f /etc/cni/net.d/*.conf;fi;TMP=/etc/cni/net.d/.tmp-kuberouter-cfg;cp /etc/kube-router/cni-conf.json ${TMP};mv ${TMP} /etc/cni/net.d/10-kuberouter.conflist;fivolumeMounts:- name: cni-conf-dirmountPath: /etc/cni/net.d- name: kube-router-cfgmountPath: /etc/kube-routerhostNetwork: truetolerations:- key: CriticalAddonsOnlyoperator: Exists- effect: NoSchedulekey: node-role.kubernetes.io/masteroperator: Exists- effect: NoSchedulekey: node.kubernetes.io/not-readyoperator: Existsvolumes:- name: lib-moduleshostPath:path: /lib/modules- name: cni-conf-dirhostPath:path: /etc/cni/net.d- name: kube-router-cfgconfigMap:name: kube-router-cfg- name: kubeconfigconfigMap:name: kube-proxyitems:- key: kubeconfig.confpath: kubeconfig
---
apiVersion: v1
kind: ServiceAccount
metadata:name: kube-routernamespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:name: kube-routernamespace: kube-system
rules:- apiGroups:- ""resources:- namespaces- pods- services- nodes- endpointsverbs:- list- get- watch- apiGroups:- "networking.k8s.io"resources:- networkpoliciesverbs:- list- get- watch- apiGroups:- extensionsresources:- networkpoliciesverbs:- get- list- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:name: kube-router
roleRef:apiGroup: rbac.authorization.k8s.iokind: ClusterRolename: kube-router
subjects:
- kind: ServiceAccountname: kube-routernamespace: kube-system

十三、k8s小知识点：如何安装指定版本的kubeadm
https://www.jianshu.com/p/4b22b5d2f69b

-------------------20200523 update-------------------

文献：

1.kubernetes---CentOS7安装kubernetes1.11.2图文完整版

2.安装k8s 1.9.0 实践：问题集锦