基于AnolisOS8.6+Kubespray2.23部署Kubernetes-v1.27

测试环境

Virtual Box，AnolisOS-8.6-x86_64-minimal.iso，4 vCPU, 8G RAM, 50 vDisk。最小化安装。需联网。

系统环境

关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
systemctl status firewalld

selinux关闭
cat /etc/selinux/config

定制Kubespray

下载kubespray-2.23.1.tar.gz
tar -zxvf kubespray-2.23.1.tar.gz
cd kubespray-2.23.1

自定义版本、开启Multus等，kubespray-v2.23.1, kubernetes 1.27.7 is the default version. [可选]
vi roles/kubespray-defaults/defaults/main.yml

kube_version: v1.27.7
kube_network_plugin_multus: true
enable_dual_stack_networks: true

修改镜像，二进制文件等下载地址. [可选]
vi roles/download/defaults/main/main.yml

注释掉OS检查。注：Kubespray不支持Anolis OS，仅支持openEuler。
vi roles/kubernetes/preinstall/tasks/0040-verify-settings.yml

#- name: Stop if the os does not support
#  assert:
#    that: (allow_unsupported_distribution_setup | default(false)) or ansible_distribution in supported_os_distributions
#    msg: "{{ ansible_distribution }} is not a known OS"
#  when: not ignore_assert_errors

增加containerd支持的OS:
vi roles/container-engine/containerd/defaults/main.yml

containerd_supported_distributions:- "CentOS"- "OracleLinux"- "RedHat"- "Ubuntu"- "Debian"- "Fedora"- "AlmaLinux"- "Rocky"- "Amazon"- "Flatcar"- "Flatcar Container Linux by Kinvolk"- "Suse"- "openSUSE Leap"- "openSUSE Tumbleweed"- "Kylin Linux Advanced Server"- "UnionTech"- "UniontechOS"- "openEuler"- "Anolis"

开启详细日志:
vi inventory/sample/group_vars/all/all.yml

unsafe_show_logs: true

增加Anolis时证书目的目录:
vi roles/etcd/tasks/upd_ca_trust.yml

- name: Gen_certs | target ca-certificate store fileset_fact:ca_cert_path: |-{% if ansible_os_family == "Debian" -%}/usr/local/share/ca-certificates/etcd-ca.crt{%- elif ansible_os_family == "RedHat" -%}/etc/pki/ca-trust/source/anchors/etcd-ca.crt{%- elif ansible_os_family == "Anolis" -%}/etc/pki/ca-trust/source/anchors/etcd-ca.crt{%- elif ansible_os_family in ["Flatcar", "Flatcar Container Linux by Kinvolk"] -%}/etc/ssl/certs/etcd-ca.pem{%- elif ansible_os_family == "Suse" -%}/etc/pki/trust/anchors/etcd-ca.pem{%- elif ansible_os_family == "ClearLinux" -%}/usr/share/ca-certs/etcd-ca.pem{%- endif %}tags:- facts

重新打包
tar czvf kubespray-2.23.1.tar.gz kubespray-2.23.1

定制kubespray-offline拉取离线镜像

https://kubespray.io/#/docs/operations/offline-environment
https://github.com/kubespray-offline/kubespray-offline
Kubespray-Offline是一个针对Kubernetes的离线部署工具包，旨在帮助用户在无网络连接或限制网络访问的环境中部署 Kubernetes 集群。
tar -zxvf kubespray-offline-2.23.1-0.tar.gz
cd kubespray-offline-2.23.1-0
mkdir -p outputs/files
kubespray-2.23.1.tar.gz放到outputs/files目录下

编辑config.sh，修改kubespray版本

vi config.sh
KUBESPRAY_VERSION=${KUBESPRAY_VERSION:-2.23.1}

安装Docker
编辑install-docker.sh文件，修改为阿里源

        $sudo yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

自定义Docker版本

        $sudo yum install -y docker-ce-20.10.24 docker-ce-cli-20.10.24

安装docker

./install-docker.sh

下载离线文件
注意: 拉取过程会访问github, registry.k8s.io, quay.io, docker.io, gcr.io, ghcr.io, dl.k8s.io等地址, 可能会受限.

./download-all.sh

下载后的文件会存放在outputs目录下。
download-all.sh会依次调用执行下列脚本

#install-docker.sh: 默认注释，建议单独安装docker
precheck.sh: 检查是否安装docker，是否禁用selinux
prepare-pkgs.sh: 安装python39, epel.
prepare-py.sh: Setup python venv, install required python packages.
get-kubespray.sh: 从github下载，Download and extract kubespray, if KUBESPRAY_DIR does not exist. 也可以手动创建目录，拷贝kubespray进去
pypi-mirror.sh: Download PyPI mirror files
download-kubespray-files.sh: Download kubespray offline files (containers, files, etc)
download-additional-containers.sh: Download additional containers. You can add any container image repoTag to imagelists/*.txt.
create-repo.sh: Download RPM or DEB repositories.
copy-target-scripts.sh: Copy scripts for target node.

下载完成后打包:
tar czvf kubespray-offline-2.23.1-v1.27.7-an.tar.gz kubespray-offline-2.23.1-0

部署节点初始化

拷贝制作的离线文件到部署节点

tar zxvf kubespray-offline-2.23.1-v1.27.7-an.tar.gz
cd kubespray-offline-2.23.1-0/outputs

set-all.sh调用以下脚本:

setup-container.sh: Install containerd from local files. Load nginx and registry images to containerd.
start-nginx.sh: Start nginx container.
setup-offline.sh: Setup yum/deb repo config and PyPI mirror config to use local nginx server.
setup-py.sh: Install python3 and venv from local repo.
start-registry.sh: Start docker private registry container.
load-push-all-images.sh: Load all container images to containerd. Tag and push them to the private registry.

修改load-push-all-images.sh中的registry地址同时push增加insecure-registry参数，默认是localhost。

LOCAL_REGISTRY=${LOCAL_REGISTRY:-"192.168.31.48:${REGISTRY_PORT}"}sudo $NERDCTL push ${newImage} --insecure-registry

执行初始化

./setup-all.sh

初始化完成后检查
nerdctl images
nerdctl ps

复制一个用于Anolis的离线repo文件:
cp playbook/roles/offline-repo/tasks/RedHat.yml playbook/roles/offline-repo/tasks/Anolis.yml

配置互信

ssh-keygen
ssh-copy-id root@192.168.31.65

./extract-kubespray.sh
cd kubespray-2.23.1
pip3 install -U pip
pip3 install -r requirements.txt

配置/etc/hosts

vi /etc/hosts
192.168.31.48 node
192.168.31.65 k8s-node

验证并拷贝hosts文件到集群node:
ansible -i inventory/sample/inventory.ini all -m ping
ansible -i inventory/sample/inventory.ini all -m copy -a “src=/etc/hosts dest=/etc/hosts”
ansible -i inventory/sample/inventory.ini all -m shell -a “cat /etc/hosts”

部署 offline repo
使用ansible将使用yum_rep的离线存储库配置部署到所有目标节点.
cp -r playbook kubespray-2.23.1/
ansible -i inventory/sample/inventory.ini all -m shell -a “mv /etc/yum.repos.d /tmp”
ansible -i inventory/sample/inventory.ini all -m shell -a “mkdir /etc/yum.repos.d”

修改repo地址，默认是localhost

vi playbook/roles/offline-repo/defaults/main.yml
yum_repo: http://192.168.31.48/rpms

集群node生成offline-repo文件:
ansible-playbook -i inventory/sample/inventory.ini playbook/offline-repo.yml

自定义离线镜像库:
mv inventory/sample/group_vars/all/offline.yml inventory/sample/group_vars/all/offline.yml.old
vi inventory/sample/group_vars/all/offline.yml

http_server: "http://192.168.31.48"
registry_host: "192.168.31.48:35000"containerd_insecure_registries: # Kubespray #8340"192.168.31.48:35000": "http://192.168.31.48:35000"files_repo: "{{ http_server }}/files"
yum_repo: "{{ http_server }}/rpms"
ubuntu_repo: "{{ http_server }}/debs"# Registry overrides
kube_image_repo: "{{ registry_host }}"
gcr_image_repo: "{{ registry_host }}"
docker_image_repo: "{{ registry_host }}"
quay_image_repo: "{{ registry_host }}"# Download URLs: See roles/download/defaults/main.yml of kubespray.
kubeadm_download_url: "{{ files_repo }}/kubernetes/{{ kube_version }}/kubeadm"
kubectl_download_url: "{{ files_repo }}/kubernetes/{{ kube_version }}/kubectl"
kubelet_download_url: "{{ files_repo }}/kubernetes/{{ kube_version }}/kubelet"
# etcd is optional if you **DON'T** use etcd_deployment=host
etcd_download_url: "{{ files_repo }}/kubernetes/etcd/etcd-{{ etcd_version }}-linux-amd64.tar.gz"
cni_download_url: "{{ files_repo }}/kubernetes/cni/cni-plugins-linux-{{ image_arch }}-{{ cni_version }}.tgz"
crictl_download_url: "{{ files_repo }}/kubernetes/cri-tools/crictl-{{ crictl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"
# If using Calico
calicoctl_download_url: "{{ files_repo }}/kubernetes/calico/{{ calico_ctl_version }}/calicoctl-linux-{{ image_arch }}"
# If using Calico with kdd
calico_crds_download_url: "{{ files_repo }}/kubernetes/calico/{{ calico_version }}.tar.gz"runc_download_url: "{{ files_repo }}/runc/{{ runc_version }}/runc.{{ image_arch }}"
nerdctl_download_url: "{{ files_repo }}/nerdctl-{{ nerdctl_version }}-{{ ansible_system | lower }}-{{ image_arch }}.tar.gz"
containerd_download_url: "{{ files_repo }}/containerd-{{ containerd_version }}-linux-{{ image_arch }}.tar.gz"#containerd_insecure_registries:
#    "{{ registry_addr }}"："{{ registry_host }}"# CentOS/Redhat/AlmaLinux/Rocky Linux
## Docker / Containerd
docker_rh_repo_base_url: "{{ yum_repo }}/docker-ce/$releasever/$basearch"
docker_rh_repo_gpgkey: "{{ yum_repo }}/docker-ce/gpg"# Fedora
## Docker
docker_fedora_repo_base_url: "{{ yum_repo }}/docker-ce/{{ ansible_distribution_major_version }}/{{ ansible_architecture }}"
docker_fedora_repo_gpgkey: "{{ yum_repo }}/docker-ce/gpg"
## Containerd
containerd_fedora_repo_base_url: "{{ yum_repo }}/containerd"
containerd_fedora_repo_gpgkey: "{{ yum_repo }}/docker-ce/gpg"

vi inventory/sample/group_vars/all/containerd.yml

containerd_registries_mirrors:- prefix: 192.168.31.48:35000mirrors:- host: http://192.168.31.48:35000capabilities: ["pull", "resolve"]skip_verify: false

Kubernetes集群部署

在kubespray-2.23.1目录下, 填写集群节点信息:
vi inventory/sample/inventory.ini
开始部署
ansible-playbook -i inventory/sample/inventory.ini --become --become-user=root cluster.yml
部署后检查
kubectl get nodes -o wide
kubectl get pod -A -o wide