前文我们聊到了k8s上crd资源的使用和相关说明，回顾请参考：https://www.cnblogs.com/qiuhom-1874/p/14267400.html；今天我们来了解下k8s的第二种扩展机制自定义apiserver，以及apiservice资源的相关话题；
在开始聊自定义apiserver前，我们先来了解下k8s原生的apiserver；其实apiserver就是一个https服务器，我们可以使用kubectl工具通过https协议请求apiserver创建资源，删除资源，查看资源等等操作；每个请求都对应着RESTful api中的请求方法，对应资源就是http协议中的url路径；比如我们要创建一个pod，其kubectl请求apiserver 使用post方法将资源定义提交给apiserver；pod资源就是对应群组中的某个版本下某个名称空间下的某个pod资源；
apiserver资源组织逻辑
提示：客户端访问apiserver，对应资源类似上图中的组织方式；比如访问default名称空间下某个pod，其路径就为/apis/core/v1/namespace/default/pod/mypod；对应resource包含名称空间以及对应资源类型；
k8s原生apiserver组成
k8s原生apiserver主要有两个组件组成，第一个组件aggregator，其功能类似web代理服务器，第二个组件就是真正的apiserver；其工作逻辑是，用户请求首先送达给aggregator，由aggregator根据用户请求的资源，将对应请求路由至apiserver；简单讲aggregator这个组件主要作用就是用来路由用户请求；默认情况aggregator会把所有请求都路由至原生的apiserver上进行响应；如果我们需要自定义apiserver，就需要在默认的aggregator上使用apiservice资源将自定义apiserver注册到原生的apiserver上，让其用户请求能够被路由至自定义apiserver进行响应；如下图
提示：apiserver是k8s的唯一访问入口，默认客户端的所有操作都是发送给apiserver进行响应，我们自定义的apiserver要想能够被客户端访问，就必须通过内建apiserver中的aggregator组件中的路由信息，把对应路径的访问路由至对应apiserver进行访问；对应aggregator中的路由信息，由k8s内建apiservice资源定义；简单讲apiservice资源就是用来定义原生apiserver中aggregator组件上的路由信息；该路由就是将某某端点的访问路由至对应apiserver；
查看原生apiserver中的群组/版本信息
[root@master01 ~]# kubectl api-versions admissionregistration.k8s.io/v1 admissionregistration.k8s.io/v1beta1 apiextensions.k8s.io/v1 apiextensions.k8s.io/v1beta1 apiregistration.k8s.io/v1 apiregistration.k8s.io/v1beta1 apps/v1 authentication.k8s.io/v1 authentication.k8s.io/v1beta1 authorization.k8s.io/v1 authorization.k8s.io/v1beta1 autoscaling/v1 autoscaling/v2beta1 autoscaling/v2beta2 batch/v1 batch/v1beta1 certificates.k8s.io/v1 certificates.k8s.io/v1beta1 coordination.k8s.io/v1 coordination.k8s.io/v1beta1 crd.projectcalico.org/v1 discovery.k8s.io/v1beta1 events.k8s.io/v1 events.k8s.io/v1beta1 extensions/v1beta1 flowcontrol.apiserver.k8s.io/v1beta1 mongodb.com/v1 networking.k8s.io/v1 networking.k8s.io/v1beta1 node.k8s.io/v1 node.k8s.io/v1beta1 policy/v1beta1 rbac.authorization.k8s.io/v1 rbac.authorization.k8s.io/v1beta1 scheduling.k8s.io/v1 scheduling.k8s.io/v1beta1 stable.example.com/v1 storage.k8s.io/v1 storage.k8s.io/v1beta1 v1 [root@master01 ~]#
提示：只有上面列出的群组版本才能够被客户端访问，即客户端只能访问上述列表中群组版本中的资源，没有出现群组版本是一定不会被客户端访问到；
apiservice资源的使用
示例：创建apiservice资源
[root@master01 ~]# cat apiservice-demo.yaml apiVersion: apiregistration.k8s.io/v1 kind: APIService metadata: name: v2beta1.auth.ilinux.io spec: insecureSkipTLSVerify: true group: auth.ilinux.io groupPriorityMinimum: 1000 versionPriority: 15 service: name: auth-api namespace: default version: v2beta1 [root@master01 ~]#
提示：apiservice资源属于apiregistration.k8s.io/v1群组，其类型为APIService，其中spec.insecureSkipTLSVerify字段用来描述是否忽略安全验证，即不验证https证书；true表示不验证，false表示要验证；group字段用来描述对应自定义apiserver的群组；groupPriorityMinimum字段用来描述对应群组的优先级；versionPriority字段用来描述对应群组版本的优先级；service字段是用来描述把对应群组版本的请求路由至某个service；该service就是对应自定义apiserver关联的service；version字段用来描述对应apiserver属于对应群组中的某个版本；上述资源清单表示在aggregator上注册auth.ilinux.io/v2beta1这个端点，该端点对应的后端apiserver的service是default名称空间下的auth-api service；即客户端访问auth.ilinux.io/v2beta1下的资源都会被路由至default名称空间下的auth-api service进行响应；
应用清单
[root@master01 ~]# kubectl apply -f apiservice-demo.yaml apiservice.apiregistration.k8s.io/v2beta1.auth.ilinux.io created [root@master01 ~]# kubectl get apiservice |grep auth.ilinux.io v2beta1.auth.ilinux.io default/auth-api False (ServiceNotFound) 16s [root@master01 ~]# kubectl api-versions |grep auth.ilinux.io auth.ilinux.io/v2beta1 [root@master01 ~]#
提示：可以看到应用清单以后，对应的端点信息就出现在api-versions中；
上述清单只是用来说明对应apiservice资源的使用；其实应用上述清单创建apiservice资源没有实质的作用，其原因是我们对应名称空间下并没有对应的服务，也没有对应自定义apiserver；所以通常自定义apiserver，我们会用到apiservice资源来把自定义apiserver整合进原生apiserver中；接下来我们部署一个真正意义上的自定义apiserver
部署metrics-server
metrics-server是用来扩展k8s的第三方apiserver，其主要作用是收集pod或node上的cpu，内存，磁盘等指标数据，并提供一个api接口供kubectl top命令访问；默认情况kubectl top 命令是没法正常使用，其原因是默认apiserver上没有对应的接口提供收集pod或node的cpu，内存，磁盘等核心指标数据；kubectl top命令主要用来显示pod/node资源的cpu，内存，磁盘的占用比例；该命令能够正常使用必须依赖Metrics API；
默认没有部署metrics server使用kubectl top pod/node查看pod或节点的cpu，内存占用比例
[root@master01 ~]# kubectl top Display Resource (CPU/Memory/Storage) usage. The top command allows you to see the resource consumption for nodes or pods. This command requires Metrics Server to be correctly configured and working on the server. Available Commands: node Display Resource (CPU/Memory/Storage) usage of nodes pod Display Resource (CPU/Memory/Storage) usage of pods Usage: kubectl top [flags] [options] Use "kubectl <command> --help" for more information about a given command. Use "kubectl options" for a list of global command-line options (applies to all commands). [root@master01 ~]# kubectl top pod error: Metrics API not available [root@master01 ~]# kubectl top node error: Metrics API not available [root@master01 ~]#
提示：默认没有部署metrics server，使用kubectl top pod/node命令，它会告诉我们没有可用的metrics api；
部署metrics server
下载部署清单
[root@master01 ~]# mkdir metrics-server [root@master01 ~]# cd metrics-server [root@master01 metrics-server]# wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.4.0/components.yaml --2021-01-14 23:54:30-- https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.4.0/components.yaml Resolving github.com (github.com)... 52.74.223.119 Connecting to github.com (github.com)|52.74.223.119|:443... connected. HTTP request sent, awaiting response... 302 Found Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/92132038/c700f080-1f7e-11eb-9e30-864a63f442f4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210114%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210114T155432Z&X-Amz-Expires=300&X-Amz-Signature=fc5a6f41ca50ec22e87074a778d2cb35e716ae6c3231afad17dfaf8a02203e35&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=92132038&response-content-disposition=attachment%3B%20filename%3Dcomponents.yaml&response-content-type=application%2Foctet-stream [following] --2021-01-14 23:54:32-- https://github-production-release-asset-2e65be.s3.amazonaws.com/92132038/c700f080-1f7e-11eb-9e30-864a63f442f4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210114%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210114T155432Z&X-Amz-Expires=300&X-Amz-Signature=fc5a6f41ca50ec22e87074a778d2cb35e716ae6c3231afad17dfaf8a02203e35&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=92132038&response-content-disposition=attachment%3B%20filename%3Dcomponents.yaml&response-content-type=application%2Foctet-stream Resolving github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)... 52.217.39.44 Connecting to github-production-release-asset-2e65be.s3.amazonaws.com (github-production-release-asset-2e65be.s3.amazonaws.com)|52.217.39.44|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 3962 (3.9K) [application/octet-stream] Saving to: ‘components.yaml’ 100%[===========================================================================================>] 3,962 11.0KB/s in 0.4s 2021-01-14 23:54:35 (11.0 KB/s) - ‘components.yaml’ saved [3962/3962] [root@master01 metrics-server]# ls components.yaml [root@master01 metrics-server]#
修改部署清单内容
[root@master01 metrics-server]# cat components.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
k8s-app: metrics-server
rbac.authorization.k8s.io/aggregate-to-admin: "true"
rbac.authorization.k8s.io/aggregate-to-edit: "true"
rbac.authorization.k8s.io/aggregate-to-view: "true"
name: system:aggregated-metrics-reader
rules:
- apiGroups:
- metrics.k8s.io
resources:
- pods
- nodes
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
k8s-app: metrics-server
name: system:metrics-server
rules:
- apiGroups:
- ""
resources:
- pods
- nodes
- nodes/stats
- namespaces
- configmaps
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
k8s-app: metrics-server
name: metrics-server-auth-reader
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
k8s-app: metrics-server
name: metrics-server:system:auth-delegator
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
k8s-app: metrics-server
name: system:metrics-server
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:metrics-server
subjects:
- kind: ServiceAccount
name: metrics-server
namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
spec:
ports:
- name: https
port: 443
protocol: TCP
targetPort: https
selector:
k8s-app: metrics-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
k8s-app: metrics-server
name: metrics-server
namespace: kube-system
spec:
selector:
matchLabels:
k8s-app: metrics-server
strategy:
rollingUpdate:
maxUnavailable: 0
template:
metadata:
labels:
k8s-app: metrics-server
spec:
containers:
- args:
- --cert-dir=/tmp
- --secure-port=4443
- --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
- --kubelet-use-node-status-port
- --kubelet-insecure-tls
image: k8s.gcr.io/metrics-server/metrics-server:v0.4.0
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /livez
port: https
scheme: HTTPS
periodSeconds: 10
name: metrics-server
ports:
- containerPort: 4443
name: https
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /readyz
port: https
scheme: HTTPS
periodSeconds: 10
securityContext:
readOnlyRootFilesystem: true
runAsNonRoot: true
runAsUser: 1000
volumeMounts:
- mountPath: /tmp
name: tmp-dir
nodeSelector:
kubernetes.io/os: linux
priorityClassName: system-cluster-critical
serviceAccountName: metrics-server
volumes:
- emptyDir: {}
name: tmp-dir
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
labels:
k8s-app: metrics-server
name: v1beta1.metrics.k8s.io
spec:
group: metrics.k8s.io
groupPriorityMinimum: 100
insecureSkipTLSVerify: true
service:
name: metrics-server
namespace: kube-system
version: v1beta1
versionPriority: 100
[root@master01 metrics-server]#
提示：在deploy中，spec.template.containers.args字段中加上–kubelet-insecure-tls选项，表示不验证客户端证书；上述清单主要用deploy控制器将metrics server运行为一个pod，然后授权metrics-server用户能够对pod/node资源进行只读权限；然后把metrics.k8s.io/v1beta1注册到原生apiserver上，让其客户端访问metrics.k8s.io下的资源能够被路由至metrics-server这个服务上进行响应；
应用资源清单
[root@master01 metrics-server]# kubectl apply -f components.yaml serviceaccount/metrics-server created clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created clusterrole.rbac.authorization.k8s.io/system:metrics-server created rolebinding.rbac.authorization.k8s.io/metrics-server-auth-reader created clusterrolebinding.rbac.authorization.k8s.io/metrics-server:system:auth-delegator created clusterrolebinding.rbac.authorization.k8s.io/system:metrics-server created service/metrics-server created deployment.apps/metrics-server created apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created [root@master01 metrics-server]#
验证：查看原生apiserver是否有metrics.k8s.io/v1beta1?
[root@master01 metrics-server]# kubectl api-versions|grep metrics metrics.k8s.io/v1beta1 [root@master01 metrics-server]#
提示：可以看到metrics.k8s.io/v1beta1群组已经注册到原生apiserver上；
查看metrics server pod是否运行正常？
[root@master01 metrics-server]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE calico-kube-controllers-744cfdf676-kh6rm 1/1 Running 4 5d7h canal-5bt88 2/2 Running 20 11d canal-9ldhl 2/2 Running 22 11d canal-fvts7 2/2 Running 20 11d canal-mwtg4 2/2 Running 23 11d canal-rt8nn 2/2 Running 21 11d coredns-7f89b7bc75-k9gdt 1/1 Running 32 37d coredns-7f89b7bc75-kp855 1/1 Running 31 37d etcd-master01.k8s.org 1/1 Running 36 37d kube-apiserver-master01.k8s.org 1/1 Running 14 13d kube-controller-manager-master01.k8s.org 1/1 Running 43 37d kube-flannel-ds-fnd2w 1/1 Running 5 5d5h kube-flannel-ds-k9l4k 1/1 Running 7 5d5h kube-flannel-ds-s7w2j 1/1 Running 4 5d5h kube-flannel-ds-vm4mr 1/1 Running 6 5d5h kube-flannel-ds-zgq92 1/1 Running 37 37d kube-proxy-74fxn 1/1 Running 10 10d kube-proxy-fbl6c 1/1 Running 8 10d kube-proxy-n82sf 1/1 Running 10 10d kube-proxy-ndww5 1/1 Running 11 10d kube-proxy-v8dhk 1/1 Running 11 10d kube-scheduler-master01.k8s.org 1/1 Running 39 37d metrics-server-58fcfcc9d-drbw2 1/1 Running 0 32s [root@master01 metrics-server]#
提示：可以看到对应pod已经正常运行；
查看pod里的日志是否正常？
[root@master01 metrics-server]# kubectl logs metrics-server-58fcfcc9d-drbw2 -n kube-system
I0114 17:52:03.601493 1 serving.go:325] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
E0114 17:52:04.140587 1 pathrecorder.go:107] registered "/metrics" from goroutine 1 [running]:
runtime/debug.Stack(0x1942e80, 0xc00069aed0, 0x1bb58b5)
/usr/local/go/src/runtime/debug/stack.go:24 +0x9d
k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).trackCallers(0xc00028afc0, 0x1bb58b5, 0x8)
/go/pkg/mod/k8s.io/apiserver@v0.19.2/pkg/server/mux/pathrecorder.go:109 +0x86
k8s.io/apiserver/pkg/server/mux.(*PathRecorderMux).Handle(0xc00028afc0, 0x1bb58b5, 0x8, 0x1e96f00, 0xc0006d88d0)
/go/pkg/mod/k8s.io/apiserver@v0.19.2/pkg/server/mux/pathrecorder.go:173 +0x84
k8s.io/apiserver/pkg/server/routes.MetricsWithReset.Install(0xc00028afc0)
/go/pkg/mod/k8s.io/apiserver@v0.19.2/pkg/server/routes/metrics.go:43 +0x5d
k8s.io/apiserver/pkg/server.installAPI(0xc00000a1e0, 0xc000589b00)
/go/pkg/mod/k8s.io/apiserver@v0.19.2/pkg/server/config.go:711 +0x6c
k8s.io/apiserver/pkg/server.completedConfig.New(0xc000589b00, 0x1f099c0, 0xc0001449b0, 0x1bbdb5a, 0xe, 0x1ef29e0, 0x2cef248, 0x0, 0x0, 0x0)
/go/pkg/mod/k8s.io/apiserver@v0.19.2/pkg/server/config.go:657 +0xb45
sigs.k8s.io/metrics-server/pkg/server.Config.Complete(0xc000589b00, 0xc000599440, 0xc000599b00, 0xdf8475800, 0xc92a69c00, 0x0, 0x0, 0xdf8475800)
/go/src/sigs.k8s.io/metrics-server/pkg/server/config.go:52 +0x312
sigs.k8s.io/metrics-server/cmd/metrics-server/app.runCommand(0xc00001c6e0, 0xc000114600, 0x0, 0x0)
/go/src/sigs.k8s.io/metrics-server/cmd/metrics-server/app/start.go:66 +0x157
sigs.k8s.io/metrics-server/cmd/metrics-server/app.NewMetricsServerCommand.func1(0xc0000d9340, 0xc0005a4cd0, 0x0, 0x5, 0x0, 0x0)
/go/src/sigs.k8s.io/metrics-server/cmd/metrics-server/app/start.go:37 +0x33
github.com/spf13/cobra.(*Command).execute(0xc0000d9340, 0xc00013a130, 0x5, 0x5, 0xc0000d9340, 0xc00013a130)
/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:842 +0x453
github.com/spf13/cobra.(*Command).ExecuteC(0xc0000d9340, 0xc00013a180, 0x0, 0x0)
/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950 +0x349
github.com/spf13/cobra.(*Command).Execute(...)
/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887
main.main()
/go/src/sigs.k8s.io/metrics-server/cmd/metrics-server/metrics-server.go:38 +0xae
I0114 17:52:04.266492 1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0114 17:52:04.267021 1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0114 17:52:04.266641 1 secure_serving.go:197] Serving securely on [::]:4443
I0114 17:52:04.266670 1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0114 17:52:04.266682 1 dynamic_serving_content.go:130] Starting serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key
I0114 17:52:04.266688 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0114 17:52:04.267120 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0114 17:52:04.266692 1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0114 17:52:04.267301 1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0114 17:52:04.367448 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0114 17:52:04.367472 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0114 17:52:04.367462 1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
[root@master01 metrics-server]#
提示：只要metrics server pod没有出现错误日志，或者无法注册等信息，就表示pod里的容器运行正常；
验证：使用kubectl top 命令查看pod的cpu ，内存占比，看看对应命令是否可以正常执行？
[root@master01 metrics-server]# kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% master01.k8s.org 235m 11% 1216Mi 70% node01.k8s.org 140m 3% 747Mi 20% node02.k8s.org 120m 3% 625Mi 17% node03.k8s.org 133m 3% 594Mi 16% node04.k8s.org 125m 3% 700Mi 19% [root@master01 metrics-server]# kubectl top pods -n kube-system NAME CPU(cores) MEMORY(bytes) calico-kube-controllers-744cfdf676-kh6rm 2m 23Mi canal-5bt88 50m 118Mi canal-9ldhl 22m 86Mi canal-fvts7 49m 106Mi canal-mwtg4 57m 113Mi canal-rt8nn 56m 113Mi coredns-7f89b7bc75-k9gdt 3m 12Mi coredns-7f89b7bc75-kp855 3m 15Mi etcd-master01.k8s.org 25m 72Mi kube-apiserver-master01.k8s.org 99m 410Mi kube-controller-manager-master01.k8s.org 14m 88Mi kube-flannel-ds-fnd2w 3m 45Mi kube-flannel-ds-k9l4k 3m 27Mi kube-flannel-ds-s7w2j 4m 46Mi kube-flannel-ds-vm4mr 3m 45Mi kube-flannel-ds-zgq92 2m 19Mi kube-proxy-74fxn 1m 27Mi kube-proxy-fbl6c 1m 23Mi kube-proxy-n82sf 1m 25Mi kube-proxy-ndww5 1m 25Mi kube-proxy-v8dhk 2m 23Mi kube-scheduler-master01.k8s.org 3m 33Mi metrics-server-58fcfcc9d-drbw2 6m 23Mi [root@master01 metrics-server]#
提示：可以看到kubectl top命令可以正常执行，说明metrics server 部署成功没有问题；
以上就是使用apiservice资源结合自定义apiserver扩展k8s功能的示例，简单总结apiservice资源的主要作用就是在aggregator上创建对应的路由信息，该路由信息的主要作用是将对应端点访问路由至自定义apiserver所对应的service进行响应；
开心洋葱 , 版权所有丨如未注明 , 均为原创丨未经授权请勿修改 , 转载请注明容器编排系统K8s之APIService资源！
