Cilium BGP 控制面服务路由通告实战：基于 containerlab 的 ClusterIP / ExternalIP / LoadBalancerIP 通告与验证

Cilium BGP 控制面服务路由通告实战基于 containerlab 的 ClusterIP / ExternalIP / LoadBalancerIP 通告与验证【免费下载链接】ciliumeBPF-based Networking, Security, and Observability项目地址: https://gitcode.com/GitHub_Trending/ci/cilium本指南基于 Cilium 仓库中的 containerlab 实验环境contrib/containerlab/auto-discovery/default-gateway/service/README.md完整讲解 Cilium BGP 控制面如何向外部路由器通告 Kubernetes Service 的虚拟 IPVIP包括 ClusterIP、ExternalIP 与 LoadBalancer IP 三类地址以及internalTrafficPolicy/externalTrafficPolicy对通告行为的影响。读完本文你将掌握如何编写CiliumBGPAdvertisement的 Service 通告配置、如何用容器化拓扑复现实验并借助 FRR 路由器验证 BGP 路由与 Community 属性。实验背景为什么需要通告 Service VIP在 Cilium BGP 控制面中CiliumBGPAdvertisement资源可以通告两类核心前缀Pod CIDR通告每个 Cilium 节点上的 Pod 网段使外部网络知道如何到达 PodService VIP以精确匹配路由/32 或 /128通告 Service 的 ClusterIP、ExternalIP 或 LoadBalancer ingress IP让外部路由器可以把流量引导到集群内。本实验将 Service 通告拆分为三个阶段对应 Makefile 中的三个目标阶段Makefile 目标说明BGP 建连与通告资源下发make apply-bgp创建CiliumBGPClusterConfig、CiliumBGPPeerConfig、CiliumBGPAdvertisementClusterIP 与 ExternalIP 通告make apply-service创建 NodePort 类型的 Service 及其后端 DeploymentLoadBalancer IP 通告make apply-lb创建CiliumLoadBalancerIPPool与 LoadBalancer 类型 Service实验拓扑的核心是一个运行 FRR 的外部路由器router0两个部署了 Cilium 的 kind 节点server0/server1以 eBGP 方式与其建立对等另有一个不含 Cilium 的server2用于连通性测试。完整拓扑定义见 topo.yaml。部署实验环境Makefilecontrib/containerlab/auto-discovery/default-gateway/service/Makefile封装了从建集群到安装 Cilium 的完整流程make deploydeploy目标依次完成用 kind 创建双栈集群cluster.yaml、启动 containerlab 拓扑、移除 control-plane 节点 taint、为 BGP 认证创建 Secret、构建并加载本地 Cilium 镜像最后使用本地 Helm chart 安装 Ciliumkind create cluster --config cluster.yaml sudo containerlab -t topo.yaml deploy kubectl taint nodes bgp-cplane-dev-service-control-plane node-role.kubernetes.io/control-plane:NoSchedule- kubectl -n kube-system create secret generic --typestring bgp-auth-secret --from-literalpasswordcilium123 cd $(git rev-parse --show-toplevel) KIND_CLUSTER_NAMEbgp-cplane-dev-service make kind-image cilium install --chart-directoryinstall/kubernetes/cilium \ -f values.yaml \ --set image.overridelocalhost:5000/cilium/cilium-dev:local \ --set image.pullPolicyNever \ --set operator.image.overridelocalhost:5000/cilium/operator-generic:local \ --set operator.image.pullPolicyNever cilium status --wait --namespace kube-systemCilium 的关键配置在 values.yaml 中routingMode: native、bgpControlPlane.enabled: true、双栈 IPv4/IPv6、ipv4NativeRoutingCIDR: 10.0.0.0/8与ipv6NativeRoutingCIDR: fd00::/16并以hostNetwork: true方式部署。清理环境使用make destroy重来一遍使用make reload等价于先 destroy 再 deploy。BGP 对等与自动发现本实验省略了与其他 lab 相似的 instance 与 peering 配置但 BGP 对等的关键点值得展开。在 bgp.yaml 中CiliumBGPClusterConfig通过nodeSelector选中带有bgp65001标签的节点该标签由 cluster.yaml 中的node-labels: bgp65001打上并声明本地 ASN 为 65001apiVersion: cilium.io/v2 kind: CiliumBGPClusterConfig metadata: name: cilium-bgp spec: nodeSelector: matchLabels: bgp: 65001 bgpInstances: - name: 65001 localASN: 65001 peers: - name: 65000 peerASN: 65000 autoDiscovery: mode: DefaultGateway defaultGateway: addressFamily: ipv6 peerConfigRef: name: cilium-peer这里对端发现使用的是DefaultGateway模式通过节点的默认路由网关自动发现对端这正是本实验目录位于auto-discovery/default-gateway之下的原因对端 ASN 为 65000与实际运行的 FRR 路由器对应。CiliumBGPPeerConfig则声明了 BGP 认证 Secretbgp-auth-secret密码cilium123、优雅重启gracefulRestart.enabled: truerestartTimeSeconds: 15并为 IPv4/IPv6 unicast 两个地址族绑定advertise: bgp标签用于关联通告资源。注拓扑中的 FRR 路由器topo.yaml实际以 AS 65010 启动并通过neighbor CILIUM local-as 65000 no-prepend replace-as在会话上伪装为 65000且启用了bgp bestpath as-path multipath-relax与bgp listen range动态监听以实现与两个 Cilium 节点的 eBGP 多路径对等。Service 通告配置详解本实验定义了三种通告bgp.yaml 中的CiliumBGPAdvertisementPodCIDR通告节点 Pod 网段附带 Community 属性65000:99Service标签bgpblue通告匹配该标签的 Service 的 ClusterIP/ExternalIP/LoadBalancerIP附带 Community65000:100Service标签bgpred同上附带 Community65000:200。apiVersion: cilium.io/v2 kind: CiliumBGPAdvertisement metadata: name: bgp-advertisements labels: advertise: bgp spec: advertisements: - advertisementType: PodCIDR attributes: communities: standard: [ 65000:99 ] - advertisementType: Service service: addresses: - ClusterIP - ExternalIP - LoadBalancerIP selector: matchExpressions: - { key: bgp, operator: In, values: [ blue ] } attributes: communities: standard: [ 65000:100 ] - advertisementType: Service service: addresses: - ClusterIP - ExternalIP - LoadBalancerIP selector: matchExpressions: - { key: bgp, operator: In, values: [ red ] } attributes: communities: standard: [ 65000:200 ]Service 通告的地址类型service.addresses列表是必填项可取值来自 API 类型定义BGPServiceAddressType见 pkg/k8s/apis/cilium.io/v2/bgp_advert_types.goClusterIP通告 Service 的 ClusterIP 前缀ExternalIP通告 Service 配置的 externalIPsLoadBalancerIP通告 LoadBalancer 类型 Service 的 ingress IP要求 Service 的loadBalancerClass为空或为 Cilium 支持的类如io.cilium/bgp-control-plane。API 层面的约束还包括Addresses至少一项MinItems1通过 CEL 校验强制Service通告类型必须携带service字段、非Service类型禁止携带service字段、PodCIDR类型禁止携带selector见 bgp_advert_types.go。此外BGPServiceOptions还支持aggregationLengthIPv40–31与aggregationLengthIPv60–127用于对 Service 前缀做聚合。标签选择与属性selector使用标准的 Kubernetes 标签选择器本实验中 Service 通过labels: bgp: blue/bgp: red被区分并为各自前缀打上不同的 standard Community。attributes.communities.standard可以携带任意合法的 ASN:Value 组合用于下游路由器做策略匹配。ClusterIP 与 ExternalIP 通告实验创建 NodePort 类型的 Service 与后端 Deploymentmake apply-service该步骤应用 service.yamltenant-blue命名空间中的service-blue带bgp: blue标签配置了externalIPs: 192.168.100.10、externalTrafficPolicy: Local与internalTrafficPolicy: Localtenant-red中的service-red带bgp: red标签配置了externalIPs: 192.168.200.10但未设置流量策略默认为 Cluster。两个 Service 都是ipFamilyPolicy: PreferDualStack的 NodePort并各有一个curlimages/curl后端 Deployment。查看 Service 列表NAMESPACE↑ NAME TYPE CLUSTER-IP EXTERNAL-IP PORTS AGE tenant-blue service-blue NodePort 10.2.51.216 192.168.100.10 1234►31394 66s tenant-red service-red NodePort 10.2.13.151 192.168.200.10 1236►31012 65s在 FRR 路由器上查看完整 BGP 路由表docker exec -it clab-bgp-cplane-dev-service-router0 vtysh -c sh bgp ipv4BGP table version is 8, local router ID is 10.0.0.1, vrf id 0 Default local pref 100, local AS 65000 Status codes: s suppressed, d damped, h history, * valid, best, multipath, i internal, r RIB-failure, S Stale, R Removed Nexthop codes: NNN nexthops vrf id, announce-nh-self Origin codes: i - IGP, e - EGP, ? - incomplete RPKI validation codes: V valid, I invalid, N Not found Network Next Hop Metric LocPrf Weight Path * 10.1.0.0/24 fd00:10:0:1::2 0 65001 i * 10.1.1.0/24 fd00:10:0:2::2 0 65001 i * 10.2.13.151/32 fd00:10:0:2::2 0 65001 i * fd00:10:0:1::2 0 65001 i * 10.2.51.216/32 fd00:10:0:2::2 0 65001 i * 192.168.100.10/32 fd00:10:0:2::2 0 65001 i * 192.168.200.10/32 fd00:10:0:2::2 0 65001 i * fd00:10:0:1::2 0 65001 i Displayed 6 routes and 8 total paths逐条解读这些路由10.1.0.0/24与10.1.1.0/24是来自 control-plane 与 worker 两个节点的Pod CIDR路由10.2.51.216/32是 blue 服务的ClusterIP。由于该 Service 设置了internalTrafficPolicy: Local只在存在本地后端backend的节点上才会收到这条路由10.2.13.151/32是 red 服务的ClusterIP。未设置 internal traffic policy默认 iTPCluster因此从两个 Cilium 节点都能收到该路由形成多路径192.168.100.10/32与192.168.200.10/32是ExternalIP行为遵循相同的规律依赖externalTrafficPolicy配置。各通告类型的 BGP 属性验证PodCIDRCommunity 为65000:99docker exec -it clab-bgp-cplane-dev-service-router0 vtysh -c sh bgp ipv4 10.1.1.0BGP routing table entry for 10.1.1.0/24, version 4 Paths: (1 available, best #1, table default) Advertised to non peer-group peers: fd00:10:0:1::2 fd00:10:0:2::2 65001 fd00:10:0:2::2 from fd00:10:0:2::2 (10.0.2.2) Origin IGP, valid, external, best (First path received) Community: 65000:99 Last update: Thu May 16 18:21:57 2024Blue ServiceinternalTrafficPolicyLocal仅单一路径Community 为65000:100docker exec -it clab-bgp-cplane-dev-service-router0 vtysh -c sh bgp ipv4 10.2.51.216/32BGP routing table entry for 10.2.51.216/32, version 7 Paths: (1 available, best #1, table default) Advertised to non peer-group peers: fd00:10:0:1::2 fd00:10:0:2::2 65001 fd00:10:0:2::2 from fd00:10:0:2::2 (10.0.2.2) Origin IGP, valid, external, best (First path received) Community: 65000:100 Last update: Thu May 16 18:21:57 2024Red ServiceinternalTrafficPolicyCluster来自两个对端的多路径Community 为65000:200docker exec -it clab-bgp-cplane-dev-service-router0 vtysh -c sh bgp ipv4 10.2.13.151/32BGP routing table entry for 10.2.13.151/32, version 5 Paths: (2 available, best #2, table default) Advertised to non peer-group peers: fd00:10:0:1::2 fd00:10:0:2::2 65001 fd00:10:0:2::2 from fd00:10:0:2::2 (10.0.2.2) Origin IGP, valid, external, multipath Community: 65000:200 Last update: Thu May 16 18:21:57 2024 65001 fd00:10:0:1::2 from fd00:10:0:1::2 (10.0.1.2) Origin IGP, valid, external, multipath, best (Older Path) Community: 65000:200 Last update: Thu May 16 18:21:55 2024LoadBalancer IP 通告实验LoadBalancer 服务的配置方式类似需要先为服务提供可分配的负载均衡 IP 池make apply-lbl b-ip.yaml 定义了名为lb-ip.yaml的两个CiliumLoadBalancerIPPoolip-pool-blue20.0.10.0/24与2004::0/64通过serviceSelector匹配bgpblue和ip-pool-red20.1.10.0/24与2004:1::0/64匹配bgpred。对应的lb-service-blueexternalTrafficPolicy: Local与lb-service-red默认策略分别落在各自池中NAMESPACE↑ NAME TYPE CLUSTER-IP EXTERNAL-IP PORTS AGE tenant-blue lb-service-blue LoadBalancer 10.2.165.65 20.0.10.1 1234►32398 2m17s tenant-red lb-service-red LoadBalancer 10.2.97.3 20.1.10.1 1236►30547 2m17s关键验证点在于流量策略与端点状态的组合tenant-blueexternalTrafficPolicyLocal且没有任何关联端点由于本节点没有该 Service 的本地端点20.0.10.1/32不会被通告docker exec -it clab-bgp-cplane-dev-service-router0 vtysh -c sh bgp ipv4 20.0.10.1/32% Network not in tabletenant-redexternalTrafficPolicyCluster无论端点状态如何20.1.10.1/32前缀都会被通告且来自两个对端、携带 Community65000:200docker exec -it clab-bgp-cplane-dev-service-router0 vtysh -c sh bgp ipv4 20.1.10.1/32BGP routing table entry for 20.1.10.1/32, version 11 Paths: (2 available, best #1, table default) Advertised to non peer-group peers: fd00:10:0:1::2 fd00:10:0:2::2 65001 fd00:10:0:1::2 from fd00:10:0:1::2 (10.0.1.2) Origin IGP, valid, external, multipath, best (Router ID) Community: 65000:200 Last update: Thu May 16 18:37:28 2024 65001 fd00:10:0:2::2 from fd00:10:0:2::2 (10.0.2.2) Origin IGP, valid, external, multipath Community: 65000:200 Last update: Thu May 16 18:37:28 2024源码视角Service 通告的实现原理从源码层面可以印证上述行为完全由 Cilium BGP Service Reconciler 控制核心实现位于 pkg/bgp/manager/reconciler/service.go。标签匹配matchService将通告中的selector转为标准 LabelSelector 后与 Service 的标签集匹配不匹配则直接跳过service.go地址类型分发对service.addresses中的每项分别调用getLoadBalancerIPPaths/getClusterIPPaths/getExternalIPPaths计算期望前缀service.go流量策略与端点判断ClusterIP 路径中当IntTrafficPolicy Local且无本地端点时跳过当无任何后端且未开启EnableNoServiceEndpointsRoutable时同样跳过service.goExternalIP 路径对ExtTrafficPolicy Local做同样的本地端点检查service.goLoadBalancerIP 路径额外检查loadBalancerClass非 Cilium 支持的类直接忽略并对externalTrafficPolicy: Local且无本地端点的情况跳过通告如果 Service 由本地 Envoy 代理ProxyRedirects例如 Gateway API / Ingress处理流量则不受此限制service.go。这解释了本实验中的全部现象blue 服务因internalTrafficPolicy: Local/externalTrafficPolicy: Local只从有本地后端的节点通告单一路由red 服务默认Cluster策略则从所有节点通告多路径而lb-service-blue因Local策略且无本地端点、也无本地代理前缀完全不出现在 BGP 表中。扩展与排障建议修改通告标签或属性后重放使用make delete-bgp make apply-bgp重新应用 bgp.yaml即可验证新的 selector 与 Community 组合查看 Cilium 侧的路由与配置cilium-dbg bgp routes对应 cilium-dbg_bgp_routes.md与cilium-dbg bgp peerscilium-dbg_bgp_peers.md可检查本机通告状态与对等关系关联测试用例仓库中的 BGP 集成测试如 pkg/bgp/test/testdata/svc-traffic-policy.txtar、pkg/bgp/test/testdata/svc-adverts.txtar覆盖了流量策略、端点增减、通告修改等场景是理解边界行为的补充材料双栈注意实验全程同时启用 IPv4/IPv6kind 集群、Service 均为PreferDualStack路由器通过 IPv6 链路本地地址建立 BGP 会话并接收 IPv4 前缀排查时注意区分地址族sh bgp ipv4/sh bgp ipv6。至此你已经可以基于本实验模板在自己的集群中按需通告 ClusterIP、ExternalIP 与 LoadBalancer IP并通过internalTrafficPolicy/externalTrafficPolicy精确控制每条 Service 路由的发布范围与多路径行为。【免费下载链接】ciliumeBPF-based Networking, Security, and Observability项目地址: https://gitcode.com/GitHub_Trending/ci/cilium创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考