开局一张图！文接上篇，上一篇文章只是完成了管理集群和工作集群的初始化创建。
TKG 作为企业发行版，除了VMware发行的 Kubernetes，同时也集成了CNCF生态的日志、监控、Ingress等重要VMware发行版本开源组件。这些组件帮助TKG实现初步生产就绪。TKG通过Tanzu Package Plugin for managing packages进行管理和发布。
Tanzu Package采用的是Carvel 项目的API，Tanzu的产品化实现。
Carvel 项目是由 VMware 开源的一套云原生开发工具集，提供了一套遵循 Unix 哲学的工具来帮助开发者将应用构建和部署到 Kubernetes 集群。每个工具只做一件事情，可自由组合，这些工具包括：
ytt：通过 YAML 结构而不是文本文档为 Kubernetes 配置生成模板和覆盖
kapp：将多个 Kubernetes 资源当做一个“应用”一样进行管理，比如安装、升级和删除等操作
kbld：以不可变的方式在 Kubernetes 配置中构建或引用容器镜像
imgpkg：通过 Docker 镜像仓库来打包和迁移应用程序的容器镜像及其配置文件
kapp-controller：符合 GitOps 理念的包管理器，实现 K8s 应用程序与包的持续交付
vendir：声明性地说明目录中应该包含哪些文件
Tanzu Package 分为 Core packages 和 User-Managed Packages，Core packages 是在部署和发布TKGm管理集群和工作集群的时候就同时安装部署了（CNI Core packages 可以选择不部署），User-Managed Packages 是在部署和发布TKGm管理集群和工作集群之后可以选择性部署。下面表格是 Core packages 与User-Managed Packages 当前包含的组件和功能，组件一直在持续迭代中。
Core Packages 清单：
Package 名字 | 类别 | 安装集群 | 功能描述 |
addons-manager | addons mgmt. | management | Tanzu addons manager |
ako-operator | networking | management | NSX Advanced Load Balancer using ako-operator |
antrea | cni | management/workload | networking and network security solution for containers |
calico | cni | management/workload | Networking and network security solution for containers |
kapp-controller | addons-management | workload | Kubernetes package manager for lifecycle management of packages and package repositories |
load-balancer-and-ingress-service | networking | management/workload | |
metrics-server | metrics | management/workload | Metrics Server is a scalable, efficient source of container resource metrics for Kubernetes built-in autoscaling pipelines |
pinniped | authentication | management/workload | Pinniped provides identity services to Kubernetes |
vsphere-cpi | cpi | management/workload | Cluster API Provider for vSphere is a concrete implementation of Cluster API for vSphere |
vsphere-csi | csi | management/workload | vSphere CSI provider |
User-Managed Packages清单
| Package 名字 | 依赖 | 目标集群 | 功能描述 |
|---|---|---|---|
| cert-manager | Required bycontour, external-dns, harbor, multus-cni, and prometheus and grafana. | Workload or shared services | Certificate management |
| multus-cni | n/a | Workload | Container networking |
| harbor | n/a | Shared services | Container registry |
| contour | Required byharbor, external-dns if you want to create DNS records for Contour HTTPProxy resources, and grafana. | Workload or shared services | Ingress control |
| fluent-bit | n/a | Workload | Log forwarding |
| grafana | n/a | Workload | Monitoring |
| prometheus | Required by grafana. | Workload | Monitoring |
| external-dns | Recommended for harbor on infrastructure platforms with load balancing such as Amazon EC2, Azure, and vSphere with NSX Advanced Load Balancer, especially in production or other environments in which Harbor availability is important. | Workload or shared services | Service discovery |
我们需要在根据需要工作集群安装User-Managed Packages清单的部分软件。本次主要测试以下几个软件功能。
测试内容
测试拓扑
备注：为了测试方便，便于理解，管理网、业务网、节点网络都使用同一个网络mgmt （生产部署需要分开）
开始配置之前介绍一个kubie 小工具，能够快速在多个集群context 和ns进行切换
https://github.com/sbstp/kubie
首先要重新发布一个workload02 集群 （1个控制节点，3个worker节点）
切换到工作集群workload02
[root@bootstrap ~]# tanzu cluster listNAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES PLANworkload02 default running 1/1 3/3 v1.21.2+vmware.1 <none> prod
[root@bootstrap ~]# kubie ctxtkgm-admin@tkgm
workload02-admin@workload02
[root@bootstrap ~]# kubie ctx workload02-admin@workload02
[workload02-admin@workload01|default] [root@bootstrap ~]#查看当前tanzu package repository ，
[workload02-admin@workload02|default] [root@bootstrap ~]# tanzu package repository list -A
- Retrieving repositories...NAME REPOSITORY STATUS DETAILS NAMESPACEtanzu-standard projects.registry.vmware.com/tkg/packages/standard/repo:v1.4.0 Reconcile succeeded tanzu-package-repo-globaltanzu-core projects.registry.vmware.com/tkg/packages/core/repo:v1.21.2_vmware.1-tkg.1 Reconcile succeeded tkg-system当前显示的是使用的VMware在线仓库的地址。如果使用的是私有仓库，使用如下命令进行添加
tanzu package repository add custom-image-repository-01 --url <REPOSITORY-URL> -n tkg-custom-image-repository --create-namespace查看当前已经安装的package，是初始化的时候自动安装的core package
[workload02-admin@workload02|default] [root@bootstrap ~]# tanzu package installed list -A
- Retrieving installed packages...
NAME PACKAGE-NAME PACKAGE-VERSION STATUS NAMESPACE
antrea antrea.tanzu.vmware.com Reconcile succeeded tkg-system
load-balancer-and-ingress-service load-balancer-and-ingress-service.tanzu.vmware.com Reconcile succeeded tkg-system
metrics-server metrics-server.tanzu.vmware.com Reconcile succeeded tkg-system
vsphere-cpi vsphere-cpi.tanzu.vmware.com Reconcile succeeded tkg-system
vsphere-csi vsphere-csi.tanzu.vmware.com Reconcile succeeded tkg-system查看当前可用的package，以及package功能描述
[workload02-admin@workload02|default] [root@bootstrap ~]# tanzu package available list -A
- Retrieving available packages...
NAME DISPLAY-NAME SHORT-DESCRIPTION NAMESPACE
cert-manager.tanzu.vmware.com cert-manager Certificate management tanzu-package-repo-global
contour.tanzu.vmware.com Contour An ingress controller tanzu-package-repo-global
external-dns.tanzu.vmware.com external-dns This package provides DNS synchronization functionality. tanzu-package-repo-global
fluent-bit.tanzu.vmware.com fluent-bit Fluent Bit is a fast Log Processor and Forwarder tanzu-package-repo-global
grafana.tanzu.vmware.com grafana Visualization and analytics software tanzu-package-repo-global
harbor.tanzu.vmware.com Harbor OCI Registry tanzu-package-repo-global
multus-cni.tanzu.vmware.com multus-cni This package provides the ability for enabling attaching multiple network interfaces to pods in Kubernetes tanzu-package-repo-global
prometheus.tanzu.vmware.com prometheus A time series database for your metrics tanzu-package-repo-global
addons-manager.tanzu.vmware.com tanzu-addons-manager This package provides TKG addons lifecycle management capabilities. tkg-system
ako-operator.tanzu.vmware.com ako-operator NSX Advanced Load Balancer using ako-operator tkg-system
antrea.tanzu.vmware.com antrea networking and network security solution for containers tkg-system
calico.tanzu.vmware.com calico Networking and network security solution for containers. tkg-system
kapp-controller.tanzu.vmware.com kapp-controller Kubernetes package manager tkg-system
load-balancer-and-ingress-service.tanzu.vmware.com load-balancer-and-ingress-service Provides L4+L7 load balancing for TKG clusters running on vSphere tkg-system
metrics-server.tanzu.vmware.com metrics-server Metrics Server is a scalable, efficient source of container resource metrics for Kubernetes built-in autoscaling pipelines. tkg-system
pinniped.tanzu.vmware.com pinniped Pinniped provides identity services to Kubernetes. tkg-system
vsphere-cpi.tanzu.vmware.com vsphere-cpi The Cluster API brings declarative, Kubernetes-style APIs to cluster creation, configuration and management. Cluster API Provider for vSphere is a concrete implementation of Cluster API for vSphere. tkg-system
vsphere-csi.tanzu.vmware.com vsphere-csi vSphere CSI provider tkg-system1.安装Cert-Manager Package
cert-manager 是一个云原生证书管理开源项目，用于在 Kubernetes 集群中自动管理和颁发来自各种颁发源的 TLS 证书，它可以从各种受支持的来源颁发证书，包括 Let’s Encrypt、HashiCorp Vault和Venafi以及私有 PKI，它将确保证书定期有效和更新，并在到期前的适当时间尝试更新证书。
Issuers/ClusterIssuers：定义使用 什么证书颁发机构 (CA) 来去颁发证书，Issuers和ClusterIssuers区别是issuers是一个名称空间级别的资源， 只能用来签发自己所在 namespace 下的证书，ClusterIssuer是个集群级别的资源 可以签发任意 namespace 下的证书
Certificate：定义所需的 X.509 证书，该证书将更新并保持最新。Certificate是一个命名空间资源，当Certificate被创建时，它会去创建相应的CertificateRequest资源来去申请证书。
查看可用的版本信息
[workload02-admin@workload02|default] [root@bootstrap ~]# tanzu package available list cert-manager.tanzu.vmware.com -A
- Retrieving package versions for cert-manager.tanzu.vmware.com...
NAME VERSION RELEASED-AT NAMESPACE
cert-manager.tanzu.vmware.com 1.1.0+vmware.1-tkg.2 2020-11-24T18:00:00Z tanzu-package-repo-global执行cert-manager安装
[workload02-admin@workload02|default] [root@bootstrap ~]# tanzu package install cert-manager --package-name cert-manager.tanzu.vmware.com --namespace cert-manager --version 1.1.0+vmware.1-tkg.2 --create-namespace
- Installing package 'cert-manager.tanzu.vmware.com'
| Creating namespace 'cert-manager'
| Getting package metadata for 'cert-manager.tanzu.vmware.com'
| Creating service account 'cert-manager-cert-manager-sa'
| Creating cluster admin role 'cert-manager-cert-manager-cluster-role'
| Creating cluster role binding 'cert-manager-cert-manager-cluster-rolebinding'
- Creating package resource
/ Package install status: ReconcilingAdded installed package 'cert-manager' in namespace 'cert-manager'查看是否安装成功
[workload02-admin@workload02|default] [root@bootstrap ~]# tanzu package installed list -A | grep cert-manager
cert-manager cert-manager.tanzu.vmware.com 1.1.0+vmware.1-tkg.2 Reconcile succeeded cert-manager2.安装配置Contour Package
Contour是VMware开源并贡献给CNCF的Ingress控制器产品，目前是CNCF孵化项目。
Contour是用于Kubernetes的高性能入口控制器，可为Envoy提供控制平面。Contour通过将Envoy部署为反向代理和负载均衡来工作。Contour支持动态配置更新和多团队Kubernetes集群，能够限制可配置虚拟主机和TLS凭证的命名空间，并提供高级负载均衡策略。
除了支持原生的ingres规则外，因为ingress-nginx注解很驳杂，不利于使用， Contour还抽象了HTTPProxy概念。
HTTPProxy的主要优势：
安全地支持多团队Kubernetes集群
并具有限制哪些命名空间可以配置虚拟主机和TLS凭据的能力。
允许包括来自另一个HTTPProxy（可能在另一个命名空间中）的路径或域的路由配置。
在一条路由中接受多种服务，并在它们之间负载均衡流量。
本机允许定义服务加权和负载平衡策略而无需注释。
在创建时验证HTTPProxy对象，并为创建后的有效性进行状态报告
查看AVI AKO 是否正在运行，AKO 是AVI负载均衡运行在tkg集群中pod，提供 Ingress-Controller 功能（需要企业级license后续会测试）和 Avi 配置功能，自动监控 k8s 对象，并将其转换成 Avi Controller APIs，例如创建Loadbalancer 类型的服务暴露（Contour with AVI ）。
[workload02-admin@workload02|default] [root@bootstrap ~]# kubectl get pods -n avi-system
NAME READY STATUS RESTARTS AGE
ako-0 1/1 Running 0 34m创建contour-data-values.yaml
envoy:
service:
type: LoadBalancer #指定envoy服务暴露方式
certificates:
useCertManager: true查看可用的contour Package 版本
[workload02-admin@workload02|default] [root@bootstrap ~]# tanzu package available list contour.tanzu.vmware.com -A
- Retrieving package versions for contour.tanzu.vmware.com...
NAME VERSION RELEASED-AT NAMESPACE
contour.tanzu.vmware.com 1.17.1+vmware.1-tkg.1 2021-07-23T18:00:00Z tanzu-package-repo-global安装Contour Package
[workload02-admin@workload02|tanzu-system-monitoring] [root@bootstrap ~]# tanzu package install contour -p contour.tanzu.vmware.com --version 1.17.1+vmware.1-tkg.1 --values-file contour-data-values.yaml
- Installing package 'contour.tanzu.vmware.com'
| Getting namespace 'default'
| Getting package metadata for 'contour.tanzu.vmware.com'
| Creating service account 'contour-default-sa'
| Creating cluster admin role 'contour-default-cluster-role'
| Creating cluster role binding 'contour-default-cluster-rolebinding'
| Creating secret 'contour-default-values'
- Creating package resource
| Package install status: ReconcilingAdded installed package 'contour' in namespace 'default'查看是否安装成功
备注：service/envoy 通过AVI/AKO LoadBalancer方式进行暴露，作为Ingress控制器的IP，与FQDN关联，实现Ingress数据引流。当前环境IP是
192.168.110.45：
[workload02-admin@workload02|default] [root@bootstrap ~]# tanzu package installed list -A | grep contour
contour contour.tanzu.vmware.com 1.17.1+vmware.1-tkg.1 Reconcile succeeded tanzu-system-contour
[workload02-admin@workload02|default] [root@bootstrap ~]# kubectl get pod,svc -n tanzu-system-ingress
NAME READY STATUS RESTARTS AGE
pod/contour-d5b6f4b68-nq74n 1/1 Running 0 60m
pod/contour-d5b6f4b68-w549q 1/1 Running 0 60m
pod/envoy-bnlnl 2/2 Running 0 60m
pod/envoy-krstk 2/2 Running 0 60m
pod/envoy-mmc82 2/2 Running 0 60mNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/contour ClusterIP 100.69.44.188 <none> 8001/TCP 60m
service/envoy LoadBalancer 100.67.252.117 192.168.110.45 80:31373/TCP,443:30691/TCP 60m3.安装配置ExternalDNS Package
ExternalDNS项目的是将Kubernetes的Service/Ingress暴露的服务的DNS记录同步给外部的DNS Provider。ExternalDNS的设计思想类似于KubeDNS，都是从多种K8S API资源中推断需要生成的DNS记录。不同之处是，ExternalDNS本身不提供DNS服务，它必须集成一个外部的DNS服务器，将DNS记录写进去。大量场景下，使用ExternalDNS你可以基于K8S资源（主要是Ingress和LoadBalancer类型的Service）来动态的控制DNS记录，而不需要知晓DNS服务器的技术细节。这是因为ExternalDNS项目已经集成了多种知名DNS服务提供商。
查看ExternalDNS Package 版本
[workload02-admin@workload02|default] [root@bootstrap ~]# tanzu package available list external-dns.tanzu.vmware.com
\ Retrieving package versions for external-dns.tanzu.vmware.com...
NAME VERSION RELEASED-AT
external-dns.tanzu.vmware.com 0.8.0+vmware.1-tkg.1 2021-06-11T18:00:00Z创建一个external-dns.yaml文件
namespace: tanzu-system-service-discovery
deployment:args:- --registry=txt- --txt-prefix=external-dns-- --txt-owner-id=workload02- --provider=rfc2136- --rfc2136-host=192.168.110.10 # DNS 地址- --rfc2136-port=53- --rfc2136-zone=corp.local - --rfc2136-insecure- --rfc2136-tsig-axfr- --source=service- --source=contour-httpproxy- --source=ingress- --domain-filter=corp.local更新Windows DNS 配置打开支持非安全的动态更新和Zone Transfers
安装 ExternalDNS Package ：
[workload02-admin@workload02|default] [root@bootstrap ~]# tanzu package install external-dns -p external-dns.tanzu.vmware.com -v 0.8.0+vmware.1-tkg.1 --values-file external-dns.yaml
\ Installing package 'external-dns.tanzu.vmware.com'
| Getting namespace 'default'
/ Getting package metadata for 'external-dns.tanzu.vmware.com'
| Creating service account 'external-dns-default-sa'
| Creating cluster admin role 'external-dns-default-cluster-role'
| Creating cluster role binding 'external-dns-default-cluster-rolebinding'
| Creating secret 'external-dns-default-values'
- Creating package resource
- Package install status: ReconcilingAdded installed package 'external-dns' in namespace 'default'查看是否安装成功，并查看日志验证部署：
[workload02-admin@workload02|default] [root@bootstrap ~]# tanzu package installed list -A |grep external
external-dns external-dns.tanzu.vmware.com 0.8.0+vmware.1-tkg.1 Reconcile succeeded default
[workload02-admin@workload02|default] [root@bootstrap ~]# kubectl get pod,svc -n tanzu-system-service-discovery
NAME READY STATUS RESTARTS AGE
pod/external-dns-65d7d94fbf-r676t 1/1 Running 0 4m8s[workload02-admin@workload02|default] [root@bootstrap ~]# kubectl logs pod/external-dns-5bb46b6dfb-6ldd9 -n tanzu-system-service-discovery
time="2021-12-14T05:42:40Z" level=info msg="config: {APIServerURL: KubeConfig: RequestTimeout:30s ContourLoadBalancerService:heptio-contour/contour GlooNamespace:gloo-system SkipperRouteGroupVersion:zalando.org/v1 Sources:[service contour-httpproxy ingress] Namespace: AnnotationFilter: LabelFilter: FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false IgnoreIngressTLSSpec:false Compatibility: PublishInternal:false PublishHostIP:false AlwaysPublishNotReadyAddresses:false ConnectorSourceServer:localhost:8080 Provider:rfc2136 GoogleProject: GoogleBatchChangeSize:1000 GoogleBatchChangeInterval:1s DomainFilter:[corp.local] ExcludeDomains:[] RegexDomainFilter: RegexDomainExclusion: ZoneNameFilter:[] ZoneIDFilter:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType: AWSZoneTagFilter:[] AWSAssumeRole: AWSBatchChangeSize:1000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AWSPreferCNAME:false AWSZoneCacheDuration:0s AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: AzureSubscriptionID: AzureUserAssignedIdentityClientID: BluecatConfigFile:/etc/kubernetes/bluecat.json CloudflareProxied:false CloudflareZonesPerPage:50 CoreDNSPrefix:/skydns/ RcodezeroTXTEncrypt:false AkamaiServiceConsumerDomain: AkamaiClientToken: AkamaiClientSecret: AkamaiAccessToken: AkamaiEdgercPath: AkamaiEdgercSection: InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InfobloxView: InfobloxMaxResults:0 DynCustomerName: DynUsername: DynPassword: DynMinTTLSeconds:0 OCIConfigFile:/etc/kubernetes/oci.yaml InMemoryZones:[] OVHEndpoint:ovh-eu OVHApiRateLimit:20 PDNSServer:http://localhost:8081 PDNSAPIKey: PDNSTLSEnabled:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:sync Registry:txt TXTOwnerID:workload02 TXTPrefix:external-dns- TXTSuffix: Interval:1m0s MinEventSyncInterval:5s Once:false DryRun:false UpdateEvents:false LogFormat:text MetricsAddress::7979 LogLevel:info TXTCacheInterval:0s TXTWildcardReplacement: ExoscaleEndpoint:https://api.exoscale.ch/dns ExoscaleAPIKey: ExoscaleAPISecret: CRDSourceAPIVersion:externaldns.k8s.io/v1alpha1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] CFAPIEndpoint: CFUsername: CFPassword: RFC2136Host:192.168.110.10 RFC2136Port:53 RFC2136Zone:corp.local RFC2136Insecure:true RFC2136GSSTSIG:false RFC2136KerberosRealm: RFC2136KerberosUsername: RFC2136KerberosPassword: RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:true RFC2136MinTTL:0s NS1Endpoint: NS1IgnoreSSL:false NS1MinTTLSeconds:0 TransIPAccountName: TransIPPrivateKeyFile: DigitalOceanAPIPageSize:50 ManagedDNSRecordTypes:[A CNAME] GoDaddyAPIKey: GoDaddySecretKey: GoDaddyTTL:0 GoDaddyOTE:false}"
time="2021-12-14T05:42:40Z" level=info msg="Instantiating new Kubernetes client"
time="2021-12-14T05:42:40Z" level=info msg="Using inCluster-config based on serviceaccount-token"
time="2021-12-14T05:42:40Z" level=info msg="Created Kubernetes client https://100.64.0.1:443"
time="2021-12-14T05:42:41Z" level=info msg="Created Dynamic Kubernetes client https://100.64.0.1:443"
time="2021-12-14T05:42:43Z" level=info msg="Configured RFC2136 with zone 'corp.local.' and nameserver '192.168.110.10:53'"4. 安装配置 Prometheus and Grafana Package
Prometheus 是一个开源的系统监控及报警工具，在2016年加入了 Cloud Native Computing Foundation，是继Kubernetes之后的第二个托管项目。
Grafana是一个图形化工具, 它可以从很多种数据源(例如Prometheus)中读取数据信息, 使用很漂亮的图表来展示数据, 并且有很多开源的dashborad可以使用,可以快速地搭建起一个非常精美的监控平台。它与Prometheus的关系就类似于Kibana与ElasticSearch。
安装Prometheus and Grafana Package前提是确保cert-manager and contour已经安装成功，Prometheus and Grafana Package服务暴露采用了contour的HTTPProxy特性实现Ingress功能；如需使用动态更新外部DNS服务器记录，需要 ExternalDNS。
[workload02-admin@workload02|test] [root@harbor new]# kubectl get app -A
NAMESPACE NAME DESCRIPTION SINCE-DEPLOY AGE
cert-manager cert-manager Reconcile succeeded 102s 30m
default contour Reconcile succeeded 52s 25m
default external-dns Reconcile succeeded 61s 24m
tanzu-system-dashboard grafana Reconcile succeeded 59s 17m
tanzu-system-monitoring prometheus Reconcile succeeded 83s 22m
tkg-system antrea Reconcile succeeded 3m5s 43m
tkg-system load-balancer-and-ingress-service Reconcile succeeded 68s 43m
tkg-system metrics-server Reconcile succeeded 92s 43m
tkg-system vsphere-cpi Reconcile succeeded 50s 43m
tkg-system vsphere-csi Reconcile succeeded 2m17s 43m
查看 可用的Prometheus Package版本
[workload01-admin@workload01|default] [root@bootstrap ~]# tanzu package available list prometheus.tanzu.vmware.com -A
\ Retrieving package versions for prometheus.tanzu.vmware.com...
NAME VERSION RELEASED-AT NAMESPACE
prometheus.tanzu.vmware.com 2.27.0+vmware.1-tkg.1 2021-05-12T18:00:00Z tanzu-package-repo-global创建 prometheus.yaml，设置fqdn
ingress:
enabled: true
virtual_host_fqdn: "prometheus.corp.local"
prometheus_prefix: "/"
alertmanager_prefix: "/alertmanager/"
prometheusServicePort: 80alertmanagerServicePort: 80
安装 prometheus Package
[workload02-admin@workload02|default] [root@bootstrap ~]# tanzu package install prometheus --package-name prometheus.tanzu.vmware.com --version 2.27.0+vmware.1-tkg.1 --values-file prometheus.yaml --namespace tanzu-system-monitoring --create-namespace
\ Installing package 'prometheus.tanzu.vmware.com'
| Creating namespace 'tanzu-system-monitoring'
| Getting package metadata for 'prometheus.tanzu.vmware.com'
| Creating service account 'prometheus-tanzu-system-monitoring-sa'
| Creating cluster admin role 'prometheus-tanzu-system-monitoring-cluster-role'
| Creating cluster role binding 'prometheus-tanzu-system-monitoring-cluster-rolebinding'
| Creating secret 'prometheus-tanzu-system-monitoring-values'
- Creating package resource
| Package install status: ReconcilingAdded installed package 'prometheus' in namespace 'tanzu-system-monitoring'查看是否安装成功
[workload01-admin@workload01|default] [root@bootstrap ~]# tanzu package installed list -A | grep prometheusprometheus prometheus.tanzu.vmware.com 2.27.0+vmware.1-tkg.1 Reconcile succeeded tanzu-system-monitoring
[workload01-admin@workload01|default] [root@bootstrap ~]# kubectl get pod,svc,pvc -n tanzu-system-monitoring
NAME READY STATUS RESTARTS AGE
pod/alertmanager-7db55c9bbb-5qjvc 1/1 Running 0 2m49s
pod/prometheus-cadvisor-2xzdm 1/1 Running 0 2m50s
pod/prometheus-cadvisor-ssg5s 1/1 Running 0 2m50s
pod/prometheus-cadvisor-wd4jm 1/1 Running 0 2m50s
pod/prometheus-kube-state-metrics-5d798787b4-p5txr 1/1 Running 0 2m48s
pod/prometheus-node-exporter-ggqbc 1/1 Running 0 2m48s
pod/prometheus-node-exporter-pltxk 1/1 Running 0 2m48s
pod/prometheus-node-exporter-prvdr 1/1 Running 0 2m48s
pod/prometheus-node-exporter-vwf68 1/1 Running 0 2m48s
pod/prometheus-pushgateway-57f78d864d-5lk4f 1/1 Running 0 2m48s
pod/prometheus-server-65f6bf94b9-hmbzq 2/2 Running 0 2m48sNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/alertmanager ClusterIP 100.68.150.195 <none> 80/TCP 2m49s
service/prometheus-kube-state-metrics ClusterIP None <none> 80/TCP,81/TCP 2m49s
service/prometheus-node-exporter ClusterIP 100.70.147.126 <none> 9100/TCP 2m48s
service/prometheus-pushgateway ClusterIP 100.66.142.78 <none> 9091/TCP 2m50s
service/prometheus-server ClusterIP 100.69.238.221 <none> 80/TCP 2m48sNAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/alertmanager Bound pvc-149c3a99-53d0-471d-b5b9-347b9bdd7373 2Gi RWO default 2m53s
persistentvolumeclaim/prometheus-server Bound pvc-874ec4c9-0924-49ac-81cb-fcf36e1507e1 150Gi RWO default 2m51s查看httpproxy状态，并解析prometheus.corp.local
[workload02-admin@workload02|default] [root@bootstrap ~]# kubectl get httpproxy -A
NAMESPACE NAME FQDN TLS SECRET STATUS STATUS DESCRIPTION
tanzu-system-monitoring prometheus-httpproxy prometheus.corp.local prometheus-tls valid Valid HTTPProxy
[workload02-admin@workload02|default] [root@bootstrap ~]# nslookup prometheus.corp.local
Server: 192.168.110.10
Address: 192.168.110.10#53Name: prometheus.corp.local
Address: 192.168.110.45[workload02-admin@workload02|default] [root@bootstrap ~]# kubectl logs pod/external-dns-5bb46b6dfb-6ldd9 -n tanzu-system-service-discovery
time="2021-12-14T05:42:40Z" level=info msg="config: {APIServerURL: KubeConfig: RequestTimeout:30s ContourLoadBalancerService:heptio-contour/contour GlooNamespace:gloo-system SkipperRouteGroupVersion:zalando.org/v1 Sources:[service contour-httpproxy ingress] Namespace: AnnotationFilter: LabelFilter: FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false IgnoreIngressTLSSpec:false Compatibility: PublishInternal:false PublishHostIP:false AlwaysPublishNotReadyAddresses:false ConnectorSourceServer:localhost:8080 Provider:rfc2136 GoogleProject: GoogleBatchChangeSize:1000 GoogleBatchChangeInterval:1s DomainFilter:[corp.local] ExcludeDomains:[] RegexDomainFilter: RegexDomainExclusion: ZoneNameFilter:[] ZoneIDFilter:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType: AWSZoneTagFilter:[] AWSAssumeRole: AWSBatchChangeSize:1000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AWSPreferCNAME:false AWSZoneCacheDuration:0s AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: AzureSubscriptionID: AzureUserAssignedIdentityClientID: BluecatConfigFile:/etc/kubernetes/bluecat.json CloudflareProxied:false CloudflareZonesPerPage:50 CoreDNSPrefix:/skydns/ RcodezeroTXTEncrypt:false AkamaiServiceConsumerDomain: AkamaiClientToken: AkamaiClientSecret: AkamaiAccessToken: AkamaiEdgercPath: AkamaiEdgercSection: InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InfobloxView: InfobloxMaxResults:0 DynCustomerName: DynUsername: DynPassword: DynMinTTLSeconds:0 OCIConfigFile:/etc/kubernetes/oci.yaml InMemoryZones:[] OVHEndpoint:ovh-eu OVHApiRateLimit:20 PDNSServer:http://localhost:8081 PDNSAPIKey: PDNSTLSEnabled:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:sync Registry:txt TXTOwnerID:workload02 TXTPrefix:external-dns- TXTSuffix: Interval:1m0s MinEventSyncInterval:5s Once:false DryRun:false UpdateEvents:false LogFormat:text MetricsAddress::7979 LogLevel:info TXTCacheInterval:0s TXTWildcardReplacement: ExoscaleEndpoint:https://api.exoscale.ch/dns ExoscaleAPIKey: ExoscaleAPISecret: CRDSourceAPIVersion:externaldns.k8s.io/v1alpha1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] CFAPIEndpoint: CFUsername: CFPassword: RFC2136Host:192.168.110.10 RFC2136Port:53 RFC2136Zone:corp.local RFC2136Insecure:true RFC2136GSSTSIG:false RFC2136KerberosRealm: RFC2136KerberosUsername: RFC2136KerberosPassword: RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:true RFC2136MinTTL:0s NS1Endpoint: NS1IgnoreSSL:false NS1MinTTLSeconds:0 TransIPAccountName: TransIPPrivateKeyFile: DigitalOceanAPIPageSize:50 ManagedDNSRecordTypes:[A CNAME] GoDaddyAPIKey: GoDaddySecretKey: GoDaddyTTL:0 GoDaddyOTE:false}"
time="2021-12-14T05:42:40Z" level=info msg="Instantiating new Kubernetes client"
time="2021-12-14T05:42:40Z" level=info msg="Using inCluster-config based on serviceaccount-token"
time="2021-12-14T05:42:40Z" level=info msg="Created Kubernetes client https://100.64.0.1:443"
time="2021-12-14T05:42:41Z" level=info msg="Created Dynamic Kubernetes client https://100.64.0.1:443"
time="2021-12-14T05:42:43Z" level=info msg="Configured RFC2136 with zone 'corp.local.' and nameserver '192.168.110.10:53'"
time="2021-12-14T05:50:51Z" level=info msg="Adding RR: prometheus.corp.local 0 A 192.168.110.45"
time="2021-12-14T05:50:51Z" level=info msg="Adding RR: external-dns-prometheus.corp.local 0 TXT \"heritage=external-dns,external-dns/owner=workload02,external-dns/resource=HTTPProxy/tanzu-system-monitoring/prometheus-httpproxy\""
[workload02-admin@workload02|default] [通过FQDN方式访问prometheus
查看可用的grafana Package版本
[workload02-admin@workload02|tanzu-system-monitoring] [root@bootstrap ~]# tanzu package available list grafana.tanzu.vmware.com -A
\ Retrieving package versions for grafana.tanzu.vmware.com...
NAME VERSION RELEASED-AT NAMESPACE
grafana.tanzu.vmware.com 7.5.7+vmware.1-tkg.1 2021-05-19T18:00:00Z tanzu-package-repo-global创建 TKG 1.4 有个bug，需要修改 ns 为 tanzu-system-dashboard，默认是 tanzu-system-monitoring 会安装失败
创建
grafana:
config:
datasource_yaml: |-
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
url: prometheus-server.tanzu-system-monitoring.svc.cluster.local
access: proxy
isDefault: true
namespace: tanzu-system-dashboard
ingress:virtual_host_fqdn: "grafana.corp.local"安装grafana Package
[workload02-admin@workload02|default] [root@bootstrap ~]# tanzu package install grafana -p grafana.tanzu.vmware.com -v 7.5.7+vmware.1-tkg.1 --values-file grafana-data-values.yaml --namespace tanzu-system-dashboard --create-namespace
\ Installing package 'grafana.tanzu.vmware.com'
| Creating namespace 'tanzu-system-dashboard'
/ Getting package metadata for 'grafana.tanzu.vmware.com'
| Creating service account 'grafana-tanzu-system-dashboard-sa'
| Creating cluster admin role 'grafana-tanzu-system-dashboard-cluster-role'
| Creating cluster role binding 'grafana-tanzu-system-dashboard-cluster-rolebinding'
| Creating secret 'grafana-tanzu-system-dashboard-values'
- Creating package resource
- Package install status: ReconcilingAdded installed package 'grafana' in namespace 'tanzu-system-dashboard'查看httpproxy状态，并解析grafana.corp.local
备注：granfa 服务也同时采用 LoadBalancer 暴露，也可以通过 LoadBalancer IP进行访问。
[workload02-admin@workload02|default]
[root@harbor ~]# tanzu package installed list -A |grep grafana
grafana grafana.tanzu.vmware.com 7.5.7+vmware.1-tkg.1 Reconcile succeeded tanzu-system-dashboard
[workload02-admin@workload02|default] [root@harbor ~]# kubectl get pod,svc,httpproxy -n tanzu-system-dashboard
NAME READY STATUS RESTARTS AGE
pod/grafana-6f76f5984b-jh4hr 2/2 Running 0 4m10sNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/grafana LoadBalancer 100.66.121.43 192.168.110.47 80:31690/TCP 4m11sNAME FQDN TLS SECRET STATUS STATUS DESCRIPTION
httpproxy.projectcontour.io/grafana-httpproxy grafana.corp.local grafana-tls valid Valid HTTPProxy
[workload02-admin@workload02|default] [root@harbor ~]#
[workload02-admin@workload02|default] [root@harbor ~]# kubectl get pod,svc,httpproxy -n tanzu-system-dashboard
NAME READY STATUS RESTARTS AGE
pod/grafana-6f76f5984b-jh4hr 2/2 Running 0 4m31sNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/grafana LoadBalancer 100.66.121.43 192.168.110.47 80:31690/TCP 4m32sNAME FQDN TLS SECRET STATUS STATUS DESCRIPTION
httpproxy.projectcontour.io/grafana-httpproxy grafana.corp.local grafana-tls valid Valid HTTPProxy
[workload02-admin@workload02|default] [root@harbor ~]#备注：
登陆AVI 负载均衡查看，自动创建的LB
contour
grafana
http方式登陆：http://grafana.corp.local
也可以采用Ip方式登陆：
http://192.168.110.47
默认用户名密码admin/admin ，提示修改
也可以导入其他的Dashboard模版
5.测试Contour ingress 方式发布应用
应用yaml文件示例内容如下：
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: cafe-ingress
spec:
rules:
- host: avitest.corp.local
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: coffee-svc
port:
number: 80
---
apiVersion: v1
kind: Service
metadata:
name: coffee-svc
labels:
app: coffee
spec:
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
selector:
app: coffee
type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
spec:
selector:
matchLabels:
app: coffee
replicas: 2
template:
metadata:
labels:
app: coffee
spec:
containers:
- name: avi-demoimage: xxx/tkg/avi-demo:latest （需要修改为真实的）
ports:
- containerPort: 80
livenessProbe:
tcpSocket:
port: 80
initialDelaySeconds: 5
periodSeconds: 5
readinessProbe:
tcpSocket:
port: 80
initialDelaySeconds: 5
periodSeconds: 5发布应用，查看pod，svc，ingress，以及验证解析已经添加
[workload02-admin@workload02|test] [root@harbor new]# k apply -f contouringress.yaml
ingress.networking.k8s.io/cafe-ingress created
service/coffee-svc created
deployment.apps/nginx-deployment created[workload02-admin@workload02|test] [root@harbor new]# k get po,svc,ingress
NAME READY STATUS RESTARTS AGE
pod/nginx-deployment-57b5b79798-7vjc5 1/1 Running 0 41s
pod/nginx-deployment-57b5b79798-cndzf 1/1 Running 0 41sNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/coffee-svc ClusterIP 100.70.177.45 <none> 80/TCP 41sNAME CLASS HOSTS ADDRESS PORTS AGE
ingress.networking.k8s.io/cafe-ingress <none> avitest.corp.local 192.168.110.46 80 41s[workload02-admin@workload02|test] [root@harbor new]# nslookup avitest.corp.local
Server: 192.168.110.10
Address: 192.168.110.10#53Name: avitest.corp.local
Address: 192.168.110.46通过域名方式进行访问应用：
未完待续！
要想了解云原生、机器学习和区块链等技术原理，请立即长按以下二维码，关注本公众号亨利笔记 （ henglibiji )，以免错过更新。
