k8s的service

一、Service简介1.服务发现与负载均衡资源Service Ingress这类资源决定了如何访问运行中的应用。Service服务为一组 Pod 提供固定的访问入口和负载均衡。它主要有四种类型ClusterIP集群内部访问、NodePort节点端口映射通过节点 IP端口访问、LoadBalancer对接云厂商负载均衡器和Headless用于 StatefulSet 的稳定 DNS。Ingress入口管理集群外部访问集群内服务的七层HTTP/HTTPS路由。例如根据域名api.example.com路由到api-service根据路径/web路由到web-service。2.理解标签与service中的endpoint建立一个service[rootk8s-master ~]# mkdir service/ [rootk8s-master ~]# cd service/ [rootk8s-master service]# kubectl create service clusterip --tcp 80:80 testservice --dry-runclient -o yaml testservice.yaml [rootk8s-master service]# vim testservice.yaml apiVersion: v1 kind: Service metadata: labels: app: testservice name: testservice spec: ports: - name: webport port: 80 protocol: TCP targetPort: 80 selector: app: webserver type: ClusterIP [rootk8s-master service]# kubectl apply -f testservice.yml service/testservice created [rootk8s-master service]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 none 443/TCP 13d testservice ClusterIP 10.100.99.225 none 80/TCP 4s [rootk8s-master service]# kubectl describe svc testservice建立pod并测试pod是否可以被testservice暴漏[rootk8s-master service]# kubectl run webserver --image myapp:v1 --dry-runclient -o yaml webserver.yml [rootk8s-master service]# vim webserver.yml apiVersion: v1 kind: Pod metadata: labels: run: webserver name: webserver spec: containers: - image: myapp:v1 name: webserver [rootk8s-master service]# kubectl get pods -o wide;kubectl describe service testservice由于标签没有匹配上所以此时并没有被暴露更改标签然后观察[rootk8s-master service]# kubectl label pods webserver appwebserver pod/webserver labeled [rootk8s-master service]# kubectl describe service testservice [rootk8s-master service]# curl 10.100.99.225 Hello MyApp | Version: v1 | a hrefhostname.htmlPod Name/a3.利用service暴漏控制器监控[rootk8s-master service]# watch -n 1 kubectl describe svc webservice ; kubectl get deploy --show-labels;kubectl get pods -o wide --show-labels 生成控制器和service的组合yam[rootk8s-master service]# kubectl create service clusterip --tcp 80:80 webservice --dry-runclient -o yaml webservice.yaml [rootk8s-master service]# kubectl create deployment webcluster --image myapp:v1 --dry-runclient -o yaml webservice.yaml [rootk8s-master service]# vim webservice.yaml apiVersion: v1 kind: Service metadata: labels: app: webservice name: webservice spec: port: 80 protocol: TCP targetPort: 80 selector: app: webcluster type: ClusterIP --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: webcluster name: webcluster spec: replicas: 2 selector: matchLabels: app: webcluster template: metadata: labels: app: webcluster spec: containers: - image: myapp:v1 name: myapp生成控制器和service[rootk8s-master service]# kubectl apply -f webservice.yaml #测试 [rootk8s-master service]# curl 10.100.140.222 [rootk8s-master service]# curl 10.100.140.222/hostname.html [rootk8s-master service]# curl 10.100.140.222/hostname.html二、service的ipvs模式Service 是由 kube-proxy 组件加上 iptables 来共同实现的kube-proxy 通过 iptables 处理 Service 的过程需要在宿主机上设置相当多的 iptables 规则如果宿主机有大量的Pod不断刷新iptables规则会消耗大量的CPU资源IPVS模式的service可以使K8s集群支持更多量级的Pod注意切换ipvs模式后kube-proxy会在宿主机上添加一个虚拟网卡kube-ipvs0并分配所有service IP1.在所有节点中安装ipvsadm工具[rootk8s-master service]# for name in 100 10 20 200 do ssh -l root 172.25.254.$name dnf install ipvsadm -y done2.配置集群中的kube-proxy重启pod在pod运行时配置文件中采用默认配置当改变配置文件后已经运行的pod状态不会变化所以要重启pod[rootk8s-master service]# kubectl -n kube-system edit cm kube-proxy 59 mode: ipvs [rootk8s-master service]# kubectl -n kube-system get pods kube-proxy-n7zrb 1/1 Running 1 (28h ago) 8d kube-proxy-pq7qn 1/1 Running 0 8d kube-proxy-r6zc7 1/1 Running 1 (28h ago) 8d [rootk8s-master service]# kubectl -n kube-system get pods |awk /proxy/{system(kubectl -n kube-system delete pods $1)} pod kube-proxy-n7zrb deleted from kube-system namespace pod kube-proxy-pq7qn deleted from kube-system namespace pod kube-proxy-r6zc7 deleted from kube-system namespace3.效果[rootk8s-master service]# ipvsadm -Ln三、Service的类型微服务类型作用描述ClusterIP默认值k8s系统给service自动分配的虚拟IP只能在集群内部访问NodePort将Service通过指定的Node上的端口暴露给外部访问任意一个NodeIP:nodePort都将路由到ClusterIPLoadBalancer在NodePort的基础上,借助cloud provider创建一个外部的负载均衡器并将请求转发到 NodeIP:NodePort此模式只能在云服务器上使用ExternalName将服务通过 DNS CNAME 记录方式转发到指定的域名通过 spec.externlName 设定1.ClusterIPyaml文件内容[rootk8s-master service]# vim webservice.yaml apiVersion: v1 kind: Service metadata: labels: app: webservice name: webservice spec: ports: - name: webport port: 80 protocol: TCP targetPort: 80 selector: app: webcluster type: ClusterIP --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: webcluster name: webcluster spec: replicas: 2 selector: matchLabels: app: webcluster template: metadata: labels: app: webcluster spec: containers: - image: myapp:v1 name: myapp [rootk8s-master service]# kubectl apply -f webservice.yaml集群内部解析注意集群内部通常使用service的解析名称进行通信地址的指定因为资源的ip会变但是名字不会[rootk8s-master service]# kubectl -n kube-system get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE calico-typha ClusterIP 10.97.97.73 none 5473/TCP 28h kube-dns ClusterIP 10.96.0.10 none 53/UDP,53/TCP,9153/TCP 13d [rootk8s-master service]# dig webservice.default.svc.cluster.local 10.96.0.10clusterIP的headless模式当设定此模式那么service不会被分配一个ip而是直接把endpoints上的ip直接通过dns裸漏[rootk8s-master service]# vim webservice.yaml apiVersion: v1 kind: Service metadata: labels: app: webservice name: webservice spec: ports: - name: webport port: 80 protocol: TCP targetPort: 80 selector: app: webcluster type: ClusterIP clusterIP: None ##加上这条 --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: webcluster name: webcluster spec: replicas: 2 selector: matchLabels: app: webcluster template: metadata: labels: app: webcluster spec: containers: - image: myapp:v1 name: myapp [rootk8s-master service]# kubectl apply -f webservice.yaml#查看解析 [rootk8s-master service]# dig webservice.default.svc.cluster.local 10.96.0.102.nodePort模式nodePort的yaml文件内容[rootk8s-master service]# vim webservice.yaml apiVersion: v1 kind: Service metadata: labels: app: webservice name: webservice spec: ports: - name: webport port: 80 protocol: TCP targetPort: 80 nodePort: 30001 #指定端口如不写可以从30000~32767之间随机分 selector: app: webcluster type: NodePort #NodePort模式 --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: webcluster name: webcluster spec: replicas: 2 selector: matchLabels: app: webcluster template: metadata: labels: app: webcluster spec: containers: - image: myapp:v1 name: myapp [rootk8s-master service]# kubectl apply -f webservice.yaml查看策略与访问测试[rootk8s-master service]# ipvsadm -Ln TCP 172.25.254.100:30001 rr - 10.224.36.78:80 Masq 1 0 0 - 10.224.169.157:80 Masq 1 0 0 TCP 10.244.0.0:30001 rr - 10.224.36.78:80 Masq 1 0 0 - 10.224.169.157:80 Masq 1 0 0 [rootk8s-master service]# curl 172.25.254.100:30001 Hello MyApp | Version: v1 | a hrefhostname.htmlPod Name/a [rootk8s-master service]# curl 172.25.254.100:30001/hostname.html webcluster-77c87d9946-tg8qr [rootk8s-master service]# curl 172.25.254.100:30001/hostname.html webcluster-77c87d9946-kjc9n端口破限默认端口范围30000~32767[rootk8s-master service]# vim /etc/kubernetes/manifests/kube-apiserver.yaml 42 - --service-node-port-range30000-50000 [rootk8s-master service]# kubectl -n kube-system delete pods all [rootk8s-master service]# vim webservice.yaml nodePort: 44444 [rootk8s-master service]# kubectl apply -f webservice.yaml3.loadbalancerloadbalancer的yaml文件内容[rootk8s-master service]# cp webservice.yaml loadbalancer.yml [rootk8s-master service]# vim loadbalancer.yml apiVersion: v1 kind: Service metadata: labels: app: webservice name: webservice spec: ports: - name: webport port: 80 protocol: TCP targetPort: 80 selector: app: webcluster type: LoadBalancer #指定LoadBalancer模式 --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: webcluster name: webcluster spec: replicas: 2 selector: matchLabels: app: webcluster template: metadata: labels: app: webcluster spec: containers: - image: myapp:v1 name: myapp [rootk8s-master service]# kubectl apply -f loadbalancer.yml [rootk8s-master metaLLB]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 none 443/TCP 13d webservice LoadBalancer 10.102.221.3 none 80:43957/TCP 3m59s构建metalLB[rootk8s-master service]# tar zxf metallb-v0.16.1.tar.gz [rootk8s-master service]# cd metaLLB/ [rootk8s-master metaLLB]# ls 1-metallb-native.yaml 2-metallb-confmap.yml metallb-v0.16.1.tar #镜像推入harbor仓库 [rootk8s-master metaLLB]# docker load -i metallb-v0.16.1.tar [rootk8s-master metaLLB]# docker tag quay.io/metallb/controller:v0.16.1 reg.timinglee.org/metallb/controller:v0.16.1 [rootk8s-master metaLLB]# docker tag quay.io/metallb/speaker:v0.16.1 reg.timinglee.org/metallb/speaker:v0.16.1 [rootk8s-master metaLLB]# docker push reg.timinglee.org/metallb/controller:v0.16.1 [rootk8s-master metaLLB]# docker push reg.timinglee.org/metallb/speaker:v0.16.1 #更新proxy [rootk8s-master mateLLB]# kubectl edit configmap -n kube-system kube-proxy 59 mode: ipvs 60 ipvs: 61 strictARP: true [rootk8s-master metaLLB]# kubectl -n kube-system get pods |awk /proxy/{system(kubectl -n kube-system delete pods $1)} #修改配置文件的镜像位置 [rootk8s-master metaLLB]# vim 1-metallb-native.yaml [rootk8s-master metaLLB]# grep image: 1-metallb-native.yaml 1-metallb-native.yaml: image: metallb/controller:v0.16.1 1-metallb-native.yaml: image: metallb/speaker:v0.16.1 #设定获取ip范围 [rootk8s-master metaLLB]# vim 2-metallb-confmap.yml apiVersion: metallb.io/v1beta1 kind: IPAddressPool metadata: name: first-pool namespace: metallb-system spec: addresses: - 172.25.254.50-172.25.254.99 --- apiVersion: metallb.io/v1beta1 kind: L2Advertisement metadata: name: example namespace: metallb-system spec: ipAddressPools: - first-pool验证[rootk8s-master metaLLB]# kubectl apply -f 1-metallb-native.yaml #等待1的配置加载完成 [rootk8s-master metaLLB]# kubectl apply -f 2-metallb-confmap.yml [rootk8s-master metaLLB]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 none 443/TCP 13d webservice LoadBalancer 10.102.221.3 172.25.254.50 80:43957/TCP 26m4.extrnalname[rootk8s-master service]# kubectl create svc externalname externalname --external-namewww.timinglee.org --dry-runclient -o yaml externalname.yaml [rootk8s-master service]# vim externalname.yaml apiVersion: v1 kind: Service metadata: labels: app: externalname name: externalname spec: externalName: www.timinglee.org selector: app: externalname type: ExternalName [rootk8s-master service]# kubectl apply -f externalname.yaml [rootk8s-master service]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE externalname ExternalName none www.timinglee.org none 39s kubernetes ClusterIP 10.96.0.1 none 443/TCP 13d四、ingress七层代理1.ingress的部署下载deploy文件[rootk8s-master service]# tar zxf ingress.1.15.1.tar.gz [rootk8s-master service]# cd ingress-v1.15.1 [rootk8s-master ingress-v1.15.1]# wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.15.1/deploy/static/provider/baremetal/deploy.yaml准备ingress所需镜像[rootk8s-master ingress-v1.5.1]# docker load -i ingress-v1.15.1.tar [rootk8s-master ingress-v1.5.1]# docker tag registry.k8s.io/ingress-nginx/controller:v1.15.1 reg.timinglee.org/ingress-nginx/controller:v1.15.1 [rootk8s-master ingress-v1.5.1]# docker push reg.timinglee.org/ingress-nginx/controller:v1.15.1 [rootk8s-master ~]# docker tag registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.6.9 reg.timinglee.org/ingress-nginx/kube-webhook-certgen:v1.6.9 [rootk8s-master ~]# docker push reg.timinglee.org/ingress-nginx/kube-webhook-certgen:v1.6.9更改deploy.yaml[rootk8s-master ingress-v1.5.1]# vim deploy.yaml 444: image: ingress-nginx/controller:v1.15.1 547: image: ingress-nginx/kube-webhook-certgen:v1.6.9 603: image: ingress-nginx/kube-webhook-certgen:v1.6.9 365 type: LoadBalancer [rootk8s-master ingress-v1.5.1]# kubectl apply -f deploy.yaml [rootk8s-master ingress-v1.5.1]# kubectl get -n ingress-nginx svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx-controller LoadBalancer 10.111.155.83 172.25.254.51 80:30805/TCP,443:32333/TCP 7d1h ingress-nginx-controller-admission ClusterIP 10.101.96.196 none 443/TCP 7d1h2.ingress功能的基本实现pathtypeImplementationSpecific通配符匹配| Exact完全匹配| Prefix前缀匹配注意ingress必须和输出的service资源处于同一namespace建立ingress代理的微服务[rootk8s-master ingress-v1.5.1]# kubectl create deployment myappv1 --image myapp:v1 --replicas 2 --dry-runclient -o yaml myappv1.yaml [rootk8s-master ingress-v1.5.1]# kubectl create service clusterip servicev1 --tcp 80:80 --dry-runclient -o yaml myappv1.yaml [rootk8s-master ingress-v1.5.1]# vim myappv1.yaml apiVersion: apps/v1 kind: Deployment metadata: labels: app: myappv1 name: myappv1 spec: replicas: 2 selector: matchLabels: app: myappv1 template: metadata: labels: app: myappv1 spec: containers: - image: myapp:v1 name: myapp --- apiVersion: v1 kind: Service metadata: labels: app: servicev1 name: servicev1 spec: ports: - name: serviceportv1 port: 80 protocol: TCP targetPort: 80 selector: app: myappv1 type: ClusterIP #同样方式建立myappv2 [rootk8s-master ingress-v1.5.1]# kubectl apply -f myappv1.yaml [rootk8s-master ingress-v1.5.1]# kubectl apply -f myappv2.yaml建立ingress[rootk8s-master ingress-v1.5.1]# kubectl create ingress ingress1 --class nginx --rule */servicev1:80 --dry-runclient -o yaml 1-ingress.yml [rootk8s-master ingress-v1.5.1]# vim 1-ingress.yml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress1 annotations: nginx.ingress.kubernetes.io/rewrite-target: / spec: ingressClassName: nginx rules: - host: www.timinglee.org http: paths: - backend: service: name: servicev1 port: number: 80 path: /v1 pathType: Prefix - backend: service: name: servicev2 port: number: 80 [rootk8s-master ingress-v1.5.1]# kubectl apply -f 1-ingress.yml [rootk8s-master ingress-v1.5.1]# kubectl describe ingress ingress1测试[rootk8s-master ingress-v1.5.1]# vim /etc/hosts 172.25.254.51 www.timinglee.org [rootk8s-master ingress-v1.5.1]# curl www.timinglee.org/v1 Hello MyApp | Version: v1 | a hrefhostname.htmlPod Name/a [rootk8s-master ingress-v1.5.1]# curl www.timinglee.org/v2 Hello MyApp | Version: v2 | a hrefhostname.htmlPod Name/a3.基于域名的访问[rootk8s-master ingress-v1.5.1]# vim 2-ingress.yml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress1 annotations: nginx.ingress.kubernetes.io/rewrite-target: / spec: ingressClassName: nginx rules: - host: myapp1.timinglee.org http: paths: - backend: service: name: servicev1 port: number: 80 path: / pathType: Prefix - host: myapp2.timinglee.org http: paths: - backend: service: name: servicev2 port: number: 80 path: / pathType: Prefix [rootk8s-master ingress-v1.5.1]# kubectl apply -f 2-ingress.yml测试[rootk8s-master ingress-v1.5.1]# vim /etc/hosts 172.25.254.51 www.timinglee.org myapp1.timinglee.org myapp2.timinglee.org [rootk8s-master ingress-v1.5.1]# curl myapp1.timinglee.org Hello MyApp | Version: v1 | a hrefhostname.htmlPod Name/a [rootk8s-master ingress-v1.5.1]# curl myapp2.timinglee.org Hello MyApp | Version: v2 | a hrefhostname.htmlPod Name/a4.基于动静分离的方式[rootk8s-master ingress-v1.5.1]# vim 3-ingress.yml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress1 annotations: nginx.ingress.kubernetes.io/use-regex: true nginx.ingress.kubernetes.io/rewrite-target: /index.html spec: ingressClassName: nginx rules: - host: www.timinglee.org http: paths: - backend: service: name: servicev1 port: number: 80 path: /php pathType: ImplementationSpecific - backend: service: name: servicev2 port: number: 80 path: /static pathType: ImplementationSpecific5.tls加密访问[rootk8s-master ingress-v1.5.1]# openssl req -newkey rsa:2048 -nodes -keyout tls.key -x509 -days 365 -subj /CNnginxsvc/Onginxsvc -out tls.crt [rootk8s-master ingress-v1.5.1]# kubectl create secret tls web-tls-secret --key tls.key --cert tls.crt [rootk8s-master ingress-v1.5.1]# vim 4-ingress.yml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress1 annotations: nginx.ingress.kubernetes.io/ssl-redirect: true nginx.ingress.kubernetes.io/rewrite-target: / spec: ingressClassName: nginx tls: - hosts: - www.timinglee.org secretName: web-tls-secret rules: - host: www.timinglee.org http: paths: - backend: service: name: servicev1 port: number: 80 path: / pathType: Prefix6.auth认证生成认证文件并注入集群中[rootk8s-master ingress-v1.5.1]# dnf install httpd-tools -y [rootk8s-master ingress-v1.5.1]# htpasswd -cm auth admin #注入集群 [rootk8s-master ingress-v1.5.1]# kubectl create secret generic web-auth --from-file auth在ingress中配置auth认证[rootk8s-master ingress-v1.5.1]# vim 5-ingress.yml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress1 annotations: nginx.ingress.kubernetes.io/auth-type: basic nginx.ingress.kubernetes.io/auth-secret: web-auth nginx.ingress.kubernetes.io/auth-realm: Please input username and password nginx.ingress.kubernetes.io/rewrite-target: / spec: ingressClassName: nginx rules: - host: www.timinglee.org http: paths: - backend: service: name: servicev1 port: number: 80 path: /v1 pathType: Prefix7.ingress中的网页重写利用网页从写定义默认发布文[rootk8s-master ingress-v1.5.1]# vim 6-ingress.yml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress1 annotations: nginx.ingress.kubernetes.io/rewrite-target: /$2 nginx.ingress.kubernetes.io/use-regex: true spec: ingressClassName: nginx rules: - host: www.timinglee.org http: paths: - backend: service: name: servicev1 port: number: 80 path: / pathType: Prefix - backend: service: name: servicev2 port: number: 80 path: /lee/(.*) pathType: ImplementationSpecific [rootk8s-master ingress-v1.5.1]# kubectl apply -f 6-ingress.yml利用正侧表达式从定向网页[rootk8s-master ingress-v1.5.1]# vim 7-ingress.yml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress1 annotations: nginx.ingress.kubernetes.io/rewrite-target: /$2 nginx.ingress.kubernetes.io/use-regex: true spec: ingressClassName: nginx rules: - host: www.timinglee.org http: paths: - backend: service: name: servicev1 port: number: 80 path: / pathType: Prefix - backend: service: name: servicev2 port: number: 80 path: /lee/(.*) pathType: ImplementationSpecific五、金丝雀发布cannary金丝雀发布Canary Release也称为灰度发布是一种软件发布策略。主要目的是在将新版本的软件全面推广到生产环境之前先在一小部分用户或服务器上进行测试和验证以降低因新版本引入重大问题而对整个系统造成的影响。是一种Pod的发布方式。金丝雀发布采取先添加、再删除的方式保证Pod的总量不低于期望值。并且在更新部分Pod后暂停更新当确认新Pod版本运行正常后再进行其他版本的Pod的更新。1.基于headr的灰度发布[rootk8s-master ingress-v1.5.1]# vim cannary-header.yml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress1 annotations: spec: ingressClassName: nginx rules: - host: www.timinglee.org http: paths: - backend: service: name: servicev1 port: number: 80 path: / pathType: Prefix --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress2 annotations: nginx.ingress.kubernetes.io/canary: true nginx.ingress.kubernetes.io/canary-by-header: version nginx.ingress.kubernetes.io/canary-by-header-value: 2 spec: ingressClassName: nginx rules: - host: www.timinglee.org http: paths: - backend: service: name: servicev2 port: number: 80 path: / pathType: Prefix2.基于权重的灰度发布[rootk8s-master ingress-v1.5.1]# vim cannary-weight.yml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress1 annotations: spec: ingressClassName: nginx rules: - host: www.timinglee.org http: paths: - backend: service: name: servicev1 port: number: 80 path: / pathType: Prefix --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: ingress2 annotations: nginx.ingress.kubernetes.io/canary: true nginx.ingress.kubernetes.io/canary-weight: 50 nginx.ingress.kubernetes.io/canary-weight-total: 100 spec: ingressClassName: nginx rules: - host: www.timinglee.org http: paths: - backend: service: name: servicev2 port: number: 80 path: / pathType: Prefix [rootk8s-master ingress-v1.5.1]# kubectl apply -f cannary-weight.yml编写测试脚本[rootk8s-master ingress-v1.5.1]# vim check_cannary.sh #!/bin/bash v10 v20 for (( i0; i100; i)) do responsecurl -s www.timinglee.org |grep -c v1 v1expr $v1 $response v2expr $v2 1 - $response done echo v1:$v1, v2:$v2 [rootk8s-master ingress-v1.5.1]# sh check_cannary.sh v1:52, v2:48