K8S - Secret 的简介和使用

在这里插入图片描述

Secret 的定义

Kubernetes（k8s）中的 Secret 是一种用于存储敏感信息的 Kubernetes 资源对象，如密码、API 密钥、证书等。Secret 被设计为用于安全地存储和管理敏感数据，并且可以通过 Volume 或环境变量的方式将这些数据提供给 Pod 中的容器。

简单来讲， K8S 的secret 和 configmap 的作用都是存放configuration 配置数据
但是 configmap 不适合存放证书，密码等敏感数据。
官方推荐把敏感配置存放在 secret 中，会更加安全， ~~虽然个人认为也不怎么安全~~

Secret 创建

跟configmap 类似
screct 也可以用如下kubeclt命令来查看创建的例子

gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl create secret -h
Create a secret with specified type.A docker-registry type secret is for accessing a container registry.A generic type secret indicate an Opaque secret type.A tls type secret holds TLS certificate and its associated key.Available Commands:docker-registry   Create a secret for use with a Docker registrygeneric           Create a secret from a local file, directory, or literal valuetls               Create a TLS secretUsage:kubectl create secret (docker-registry | generic | tls) [options]Use "kubectl create secret <command> --help" for more information about a given command.
Use "kubectl options" for a list of global command-line options (applies to all commands).

Secret 的3个子类

由上面看出 secret 也分3个子类

docker-repository 这个类别是存放docker 私服的验证token, 会保存为json 格式，假如k8s deployment拉取的镜像是from docker 私服，则很可能需要provide 这个 docker-repository 类别的secret
tls 用于存放ssl 证书和 passwork 等信息
generic 正常的text ，密码等配置

本文主要详细介绍 generic 类型，不会cover 其他两种

generic 类型 secret的创建

1. by kubectl command line

还是先看一下官方例子

gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl create secret generic -h
Create a secret based on a file, directory, or specified literal value.A single secret may package one or more key/value pairs.When creating a secret based on a file, the key will default to the basename of the file, and the value will default to
the file content. If the basename is an invalid key or you wish to chose your own, you may specify an alternate key.When creating a secret based on a directory, each file whose basename is a valid key in the directory will be packaged
into the secret. Any directory entries except regular files are ignored (e.g. subdirectories, symlinks, devices, pipes,
etc).Examples:# Create a new secret named my-secret with keys for each file in folder barkubectl create secret generic my-secret --from-file=path/to/bar# Create a new secret named my-secret with specified keys instead of names on diskkubectl create secret generic my-secret --from-file=ssh-privatekey=path/to/id_rsa
--from-file=ssh-publickey=path/to/id_rsa.pub# Create a new secret named my-secret with key1=supersecret and key2=topsecretkubectl create secret generic my-secret --from-literal=key1=supersecret --from-literal=key2=topsecret# Create a new secret named my-secret using a combination of a file and a literalkubectl create secret generic my-secret --from-file=ssh-privatekey=path/to/id_rsa --from-literal=passphrase=topsecret# Create a new secret named my-secret from env fileskubectl create secret generic my-secret --from-env-file=path/to/foo.env --from-env-file=path/to/bar.env
...

这里就不一一作例子了，选最常用那个

gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl create secret generic test-sec1 --from-literal=username=u123 --from-literal=password=p123456
secret/test-sec1 created

这里的 test-sec1 是secret item的名字，在这个item 其实里面创建了2个 secret 对象
分别是
username:u123
password:p12345

注意，如果value 包含特殊字符，则需要用引号包住，否则base64 encode 会有gap

2. by yaml

格式

apiVersion: v1
kind: Secret
metadata:name: my-secret
type: Opaque
data:username: <base64-encoded-username>password: <base64-encoded-password>

注意的是， secret里的值都需要 base64 encoded之后才写入yaml文件，但是base64 encoded 的text 是十分容易decoded, 所以建议密码还是要先加密再encoded 写入

Secret 的查看

gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl get secret 
NAME                  TYPE                                  DATA   AGE
default-token-7khb9   kubernetes.io/service-account-token   3      175d
test-sec1             Opaque                                2      25m
gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$

注意这里的Opaque 就是 generic的意思， Data 列2 是有连个子项(username , password)

用 Kubectl describe secret 只能看到size 信息

gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl describe secret test-sec1
Name:         test-sec1
Namespace:    default
Labels:       <none>
Annotations:  <none>Type:  OpaqueData
====
password:  7 bytes
username:  4 bytes

用 kubectl get secret xxx -o yaml 可以查看encoded 后的值

gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl get secret test-sec1 -o yaml
apiVersion: v1
data:password: cDEyMzQ1Ng==username: dTEyMw==
kind: Secret
metadata:creationTimestamp: "2024-08-17T13:47:38Z"name: test-sec1namespace: defaultresourceVersion: "6132863"uid: 2a027099-090b-478b-b460-e42991a50c62
type: Opaque

很方便地 decoded 出来真正的值

gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ echo cDEyMzQ1Ng== | base64 --decode
p123456

所以说也不是很安全

Secret 的使用

例子介绍

这里我们用springboot 访问mysql的用户名和密码作例子

原本 cloud order service 的配置文件是这样的
application-prod.yml

...
spring:datasource:url: jdbc:mysql://192.168.0.42:3306/demo_cloud_order?useUnicode=true&characterEncoding=utf-8&useSSL=false&allowPublicKeyRetrieval=trueusername: cloud_orderpassword: ENC(xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx)driver-class-name: com.mysql.cj.jdbc.Driver
...

我们增加1个yaml 文件作为例子
application-k8sprod

...
spring:datasource:url: jdbc:mysql://192.168.0.42:3306/demo_cloud_order?useUnicode=true&characterEncoding=utf-8&useSSL=false&allowPublicKeyRetrieval=truedriver-class-name: com.mysql.cj.jdbc.Driver
...

吧user name 和密码去掉了
那么怎么启动这个springboot service呢，当然是在启动参数中传入

方法1, 吧secret map 到环境变量

创建 secret 包含db 的username 和 password (加密后的)

这里用yaml方式

cloud-order-db-secret.yaml

apiVersion: v1
kind: Secret
metadata:name: cloud-order-db-secret
type: Opaque
data:# echo cloud-order | base64username: Y2xvdWQtb3JkZXIKpassword: RU5DKE9KaDRYVGFXejFixxxxxxxxxxxxx

执行

gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl apply -f cloud-order-db-secret.yaml 
secret/cloud-order-db-secret created

修改deployment yaml

deployment-cloud-order-with-secret-command-notwork.yaml

apiVersion: apps/v1
kind: Deployment
metadata:labels: # label of this deploymentapp: cloud-order # custom definedauthor: nvd11name: deployment-cloud-order # name of this deploymentnamespace: default
spec:replicas: 3            # desired replica count, Please note that the replica Pods in a Deployment are typically distributed across multiple nodes.revisionHistoryLimit: 10 # The number of old ReplicaSets to retain to allow rollbackselector: # label of the Pod that the Deployment is managing,, it's mandatory, without it , we will get this error # error: error validating data: ValidationError(Deployment.spec.selector): missing required field "matchLabels" in io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelector ..matchLabels:app: cloud-orderstrategy: # Strategy of upodatetype: RollingUpdate # RollingUpdate or RecreaterollingUpdate:maxSurge: 25% # The maximum number of Pods that can be created over the desired number of Pods during the updatemaxUnavailable: 25% # The maximum number of Pods that can be unavailable during the updatetemplate: # Pod templatemetadata:labels:app: cloud-order # label of the Pod that the Deployment is managing. must match the selector, otherwise, will get the error Invalid value: map[string]string{"app":"bq-api-xxx"}: `selector` does not match template `labels`spec: # specification of the Podcontainers:- image: europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/cloud-order:1.0.2 # image of the containerimagePullPolicy: Alwaysname: container-cloud-ordercommand: ["bash"]# does not work, as the password format is ENV(xxxxx) , and the java -jar command could not handle (),  after I added double quotes "ENV(xxxx)", service failed to start due to db password error# unless I set the plain text password to secret , but I don't think it's secure enoughargs: - "-c"- |java -jar -Dserver.port=8080 app.jar --spring.profiles.active=$APP_ENVIRONMENT --spring.datasource.username=$DB_USER --spring.datasource.password=$DB_PASSWORD env: # set env varaibles- name: APP_ENVIRONMENT # name of the environment variablevalue: k8sprod # value of the environment variable- name: APP_TAG valueFrom: # value from config mapconfigMapKeyRef: name: cloud-order-app-tag-config # name of the configMap itemkey: APP_TAG # key from the config map item- name: DB_USERvalueFrom: # value from secretsecretKeyRef:name: cloud-order-db-secretkey: username- name: DB_PASSWORDvalueFrom: # value from secretsecretKeyRef:name: cloud-order-db-secretkey: passwordrestartPolicy: Always # Restart policy for all containers within the PodterminationGracePeriodSeconds: 10 # The period of time in seconds given to the Pod to terminate gracefully

但是实际上这个方法对这个case 不work

因为java -jar 无法处理 ENV(XXX)
例如

gateman@MoreFine-S500:~/tmp$ java -jar -Dserver.port=8085 app.jar --spring.profiles.active=k8sprod --spring.datasource.username=cloud-order --spring.datasource.password=ENC(OJh4XTaWz1beLRY/1cMxxxx)
bash: syntax error near unexpected token `('

不能正确地转义括号

当我加上"" 双引号包住ENV()
则启动失败，报错db 密码错误
我暂时无法解决这个问题，除非我把明文密码放入secret ，不使用ENV() ，但是上面说过了，把明文密码放入secret 我认为不够安全，很容易decoded 出来

方法2, 吧secret 保存到文件

前提是其实
java 启动spring boot 命令有1个特性，可以额外指定包含另1个配置文件

java -jar -Dserver.port=8085 -Dspring.config.additional-location=db-config.yaml app.jar --spring.profiles.active=k8sprod

注意这里的 spring.config.additional-location 必须用 -D 作为VM 参数传入，而不是 --spring.boot 参数传入

所以我们的方向是吧username 和 password 的配置写入到1个卷中

创建 secret 包含db 的username 和 password (加密后的)

cloud-order-db-secret-file.yaml

apiVersion: v1
kind: Secret
metadata:name: cloud-order-db-secret-file
type: Opaque
data:# atasource:#   username: cloud_order#   password: ENC(OJh4XTaWz1beLRY/1cM4Xaxxx)dbconfig: c3ByaW5nOgogIGRhdGFzb3VyY2U6Cxxxx

注意这里的encoded 码是原本的格式，必须符合spring 的配置

修改 deployment yaml

apiVersion: apps/v1
kind: Deployment
metadata:labels: # label of this deploymentapp: cloud-order # custom definedauthor: nvd11name: deployment-cloud-order # name of this deploymentnamespace: default
spec:replicas: 3            # desired replica count, Please note that the replica Pods in a Deployment are typically distributed across multiple nodes.revisionHistoryLimit: 10 # The number of old ReplicaSets to retain to allow rollbackselector: # label of the Pod that the Deployment is managing,, it's mandatory, without it , we will get this error # error: error validating data: ValidationError(Deployment.spec.selector): missing required field "matchLabels" in io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelector ..matchLabels:app: cloud-orderstrategy: # Strategy of upodatetype: RollingUpdate # RollingUpdate or RecreaterollingUpdate:maxSurge: 25% # The maximum number of Pods that can be created over the desired number of Pods during the updatemaxUnavailable: 25% # The maximum number of Pods that can be unavailable during the updatetemplate: # Pod templatemetadata:labels:app: cloud-order # label of the Pod that the Deployment is managing. must match the selector, otherwise, will get the error Invalid value: map[string]string{"app":"bq-api-xxx"}: `selector` does not match template `labels`spec: # specification of the Podcontainers:- image: europe-west2-docker.pkg.dev/jason-hsbc/my-docker-repo/cloud-order:1.0.2 # image of the containerimagePullPolicy: Alwaysname: container-cloud-ordercommand: ["bash"]args: - "-c"- |java -jar -Dserver.port=8080 -Dspring.config.additional-location=/app/config/db-config.yaml app.jar --spring.profiles.active=$APP_ENVIRONMENTenv: # set env varaibles- name: APP_ENVIRONMENT # name of the environment variablevalue: k8sprod # value of the environment variable- name: APP_TAG valueFrom: # value from config mapconfigMapKeyRef: name: cloud-order-app-tag-config # name of the configMap itemkey: APP_TAG # key from the config map item- name: DB_USERvalueFrom: # value from secretsecretKeyRef:name: cloud-order-db-secretkey: username- name: DB_PASSWORDvalueFrom: # value from secretsecretKeyRef:name: cloud-order-db-secretkey: passwordvolumeMounts: # volume mount- name: db-configmountPath: /app/configvolumes:- name: db-configsecret:secretName: cloud-order-db-secret-file  items:- key: dbconfigpath: db-config.yamlrestartPolicy: Always # Restart policy for all containers within the PodterminationGracePeriodSeconds: 10 # The period of time in seconds given to the Pod to terminate gracefully

注意上面有几点

command 容器启动命令被改成 java -jar -Dserver.port=8080 -Dspring.config.additional-location=/app/config/db-config.yaml app.jar --spring.profiles.active=$APP_ENVIRONMENT
把 secret的内容写入到/app/config/db-config

检查和测试

gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl apply -f deployment-cloud-order-with-secret.yaml 
deployment.apps/deployment-cloud-order created

首先部署

再进入容器，查看/app/config/db-config.yaml 是否被生成

gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ kubectl exec -it deployment-cloud-order-588c9444dc-7cphs /bin/bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
bash-4.4# cat /app/config/db-config.yaml 
spring:datasource:username: cloud_orderpassword: ENC(OJh4XTaWz1beLRY/1cM4Xa6IP/mrxxxxxx)

正确生成

测试接口：

gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ curl http://www.jp-gcp-vms.cloud:8085/cloud-order/actuator/info | jq .% Total    % Received % Xferd  Average Speed   Time    Time     Time  CurrentDload  Upload   Total   Spent    Left  Speed
100  1922    0  1922    0     0   3616      0 --:--:-- --:--:-- --:--:--  3612
{"app": "Cloud Order Service","appEnvProfile": "k8sprod","version": "1.0.2","hostname": "deployment-cloud-order-588c9444dc-cjtkv","dbUrl": "jdbc:mysql://192.168.0.42:3306/demo_cloud_order?useUnicode=true&characterEncoding=utf-8&useSSL=false&allowPublicKeyRetrieval=true","description": "This is a simple Spring Boot application to for cloud order...","SystemVariables": {"PATH": "/usr/java/openjdk-17/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","CLUSTERIP_CLOUD_ORDER_SERVICE_HOST": "10.98.117.97","CLUSTERIP_BQ_API_SERVICE_SERVICE_PORT": "8080","KUBERNETES_PORT": "tcp://10.96.0.1:443","DB_USER": "cloud-order\n","JAVA_HOME": "/usr/java/openjdk-17","KUBERNETES_SERVICE_HOST": "10.96.0.1","LANG": "C.UTF-8","CLUSTERIP_CLOUD_ORDER_PORT": "tcp://10.98.117.97:8080","CLUSTERIP_BQ_API_SERVICE_PORT_8080_TCP": "tcp://10.100.68.154:8080","CLUSTERIP_CLOUD_ORDER_PORT_8080_TCP": "tcp://10.98.117.97:8080","CLUSTERIP_BQ_API_SERVICE_PORT_8080_TCP_PROTO": "tcp","PWD": "/app","JAVA_VERSION": "17.0.2","_": "/usr/java/openjdk-17/bin/java","CLUSTERIP_CLOUD_ORDER_PORT_8080_TCP_ADDR": "10.98.117.97","KUBERNETES_PORT_443_TCP": "tcp://10.96.0.1:443","KUBERNETES_PORT_443_TCP_ADDR": "10.96.0.1","CLUSTERIP_CLOUD_ORDER_PORT_8080_TCP_PORT": "8080","CLUSTERIP_BQ_API_SERVICE_PORT": "tcp://10.100.68.154:8080","CLUSTERIP_BQ_API_SERVICE_PORT_8080_TCP_PORT": "8080","KUBERNETES_PORT_443_TCP_PROTO": "tcp","APP_ENVIRONMENT": "k8sprod","KUBERNETES_SERVICE_PORT": "443","CLUSTERIP_CLOUD_ORDER_SERVICE_PORT": "8080","CLUSTERIP_BQ_API_SERVICE_PORT_8080_TCP_ADDR": "10.100.68.154","APP_TAG": "cloud-order-app-tag","HOSTNAME": "deployment-cloud-order-588c9444dc-cjtkv","CLUSTERIP_CLOUD_ORDER_PORT_8080_TCP_PROTO": "tcp","CLUSTERIP_BQ_API_SERVICE_SERVICE_HOST": "10.100.68.154","KUBERNETES_PORT_443_TCP_PORT": "443","KUBERNETES_SERVICE_PORT_HTTPS": "443","SHLVL": "1","HOME": "/root","DB_PASSWORD": "ENC(OJh4XTaWz1beLRY/1cM4Xa6IP/mrKgx2CdVRysJl+BZ/+7eHswcEzQWFF1PR/Hhl)"}
}
gateman@MoreFine-S500:~/projects/coding/k8s-s/service-case/cloud-order$ curl http://www.jp-gcp-vms.cloud:8085/cloud-order/actuator/health | jq .% Total    % Received % Xferd  Average Speed   Time    Time     Time  CurrentDload  Upload   Total   Spent    Left  Speed
100   364    0   364    0     0    735      0 --:--:-- --:--:-- --:--:--   736
{"status": "UP","components": {"db": {"status": "UP","details": {"database": "MySQL","validationQuery": "isValid()"}},"diskSpace": {"status": "UP","details": {"total": 20891439104,"free": 12675047424,"threshold": 10485760,"path": "/app/.","exists": true}},"livenessState": {"status": "UP"},"ping": {"status": "UP"},"readinessState": {"status": "UP"}},"groups": ["liveness","readiness"]
}

测试通过！