kube-proxy 的 ipvs 模式已经有了删除日期

Kubernetes v1.37 于 2026 年 8 月 26 日发布，带来 67 项增强，其中恰好只有一项是废弃——而它就是 kube-proxy 的 ipvs 模式。比起 gang scheduling 或者 Pod 证书，这条标题小得多，但运维上的分量大得多：整个版本里，只有这一件事的结局是从一个正在生产上跑着的数据面里删掉代码。时间表已经写在纸上了：1.37 加一个特性门控，1.40 默认关闭，1.43 删除 pkg/proxy/ipvs。

这件事有两种版本：一种是改一行 ConfigMap 就完事，另一种是过一个很难受的周二，而区别完全在于你先查了什么。下面写的是全套：KEP 的五个阶段以及其中哪些是日期而不是意向，怎么弄清楚集群实际在跑什么（包括那个比 ipvs 更糟的答案），为什么这个模式从来不是大家记忆里那条“逃离 iptables”的路，为什么 IPVS 调度算法并没有在干你留着它的那份活，内核底线以及哪些节点镜像过得去，四处有文档记载的行为变化，按节点铺开的做法，必须手工清理的残留状态，一份出错就返回非零退出码的验证脚本，以及一次真的只有两条命令的回滚。

1.37 只废弃了一样东西，就是它

下面这些现象没有一个会自报家门说“我是废弃”，所以这件事往往发现得很晚——通常是被那个做 1.40 升级的人撞上，而此时距离第一次警告已经过了三个版本。这个模式不会逐渐劣化。它会一直好好地工作，直到它不存在的那个版本为止，而中间那些警告都落进了一份没人 grep 的日志里。^[rel137]

那张表的第二行值得看两遍，因为它跟 IPVS 一点关系都没有，而且专门收拾那些从来没用过 IPVS 的集群。Kubernetes 1.37 里的默认模式仍然是 iptables，而文档明明白白写着：未来某个版本会把这个默认值改成 nftables。如果你的 kube-proxy 配置里没有写模式，那你并不是“选择了 iptables”，而是“跟着上游当前的推荐值走”——也就是说，你有一次数据面替换，排在别人的日程表上。^[vips]

上游自己的建议，也是本文里最便宜的一件事：为了避免集群的代理后端在升级过程中被意外改掉，你应该确保所有集群的 kube-proxy 配置都明确写出要用哪个模式。这是五分钟的改动，不管你最后落在哪个模式上，今天就值得做。

五个阶段，其中两个是日期

KEP-5495 把这件事拆成五个阶段，而有用的做法是把“公告”和“日期”分开看。第 1、2 阶段已经发生，对集群怎么跑没有任何改变。第 3 阶段改的是一个默认值——也就是说，它改变的是“没人做任何事”的集群会遇到什么。第 4 阶段删代码，这是唯一不可逆的一步。^[kep5495]

其中两行值得加个注脚。1.46 的清理阶段写在 KEP 里，而几乎所有关于这次废弃的二手报道都没提，这一点只有在你写会去读特性门控的工具时才要紧——在那之前，这个门控是一个真实存在、被正式列出的 GA 门控。另外，SIG Network 给出的理由值得知道，因为它不是性能：这个小组里没有熟悉 ipvs 后端代码的维护者，而且相当长一段时间以来，凡是有人报 ipvs 的 bug，得到的答复都是迁到 nftables 去。一个没人在修的后端，不管它当年的跑分多好看，都是负债。^{[depol][k8srel]}

这个集群到底跑在哪个模式上

先看实际配置了什么，而不是运维手册上写了什么，因为在任何一个超过两年的集群上，这是两份不同的文档。答案有三种，而第三种才是有意思的那个：ipvs、iptables，或者什么都没写。^[kpcfg]

# The one-liner from the v1.37 release announcement. On a kubeadm-built cluster
# the whole of kube-proxy's configuration lives in one ConfigMap key.
kubectl -n kube-system get configmap kube-proxy \
  -o jsonpath='{.data.config\.conf}' | grep 'mode:'
# mode: ipvs

# Careful with the empty answer, because it is the most common one and it does
# NOT mean "iptables forever". An unset mode means "whatever kube-proxy decides
# is the recommended default", and upstream says in as many words that a future
# release will change that default from iptables to nftables. If this comes back
# blank, you have a data-plane change scheduled that nobody in your team chose.
#   mode: ""     <- pin it, whichever mode you intend to be on

# Managed clusters and non-kubeadm installers do not necessarily use that
# ConfigMap. Ask the process instead - this is true wherever the config came from.
kubectl -n kube-system get ds kube-proxy \
  -o jsonpath='{.spec.template.spec.containers[0].args}' | tr ',' '\n'

# And from the node, which is the only answer that cannot be out of date:
ps -o args= -C kube-proxy
ipvsadm -L -n --daemon 2>/dev/null; ipvsadm -L -n | head -20
# IP Virtual Server version 1.2.1 (size=4096)
# Prot LocalAddress:Port Scheduler Flags
# TCP  10.96.0.1:443 rr
#   -> 192.168.4.11:6443            Masq    1      3          0

# Not every cluster runs kube-proxy at all. If this returns nothing, your CNI
# is implementing Services itself and none of this deprecation reaches you.
kubectl -n kube-system get ds -l k8s-app=kube-proxy

然后去找那条警告，它从 Kubernetes 1.35 起就一直在你的日志里，是“这件事和你有关”最省事的证据。顺手把 1.37 新增的那个特性门控也找出来——不是因为你需要去设它（它默认开启，这个版本什么都不变），而是因为它是 1.40 会递到你手里的那根杠杆，最好提前知道：它只把期限往后推三个小版本，不会把期限拿掉。^[gates]

# Stage 1 of the deprecation landed in Kubernetes 1.35: kube-proxy logs a
# warning on every start in ipvs mode. It has been in your logs for two releases.
kubectl -n kube-system logs ds/kube-proxy --tail=-1 --prefix \
  | grep -i -m5 'ipvs.*deprecat'
# The ipvs proxier is now deprecated and may be removed in a future release.
# Please use 'nftables' instead.

# Stage 2 is what 1.37 added: a feature gate. Today it is on by default, so
# nothing changes yet. In 1.40 the default flips and kube-proxy in ipvs mode
# exits with an error unless the gate is set back by hand; in 1.43 the gate
# cannot save you because the code is gone.
kubectl -n kube-system get configmap kube-proxy \
  -o jsonpath='{.data.config\.conf}' | grep -A3 featureGates

# What the 1.40 override would look like. Write this down as the thing you are
# choosing NOT to depend on, rather than as a plan:
#
#   featureGates:
#     KubeProxyIPVS: true
#
# It buys three minor releases, or roughly nine months at the current cadence,
# and it stops working entirely in 1.43.

# Count how much of the fleet this is really about, because in most clusters
# the answer is "some node pools, historically".
kubectl get nodes -o custom-columns=\
'NODE:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion,'\
'KERNEL:.status.nodeInfo.kernelVersion,OS:.status.nodeInfo.osImage'

ipvs 模式是什么，以及它被记成了什么

有必要把“到底在退役什么”说准，因为流行的那个版本是错的，而且错在会改变决策的地方。ipvs 模式是在 Kubernetes 1.8 加进来的，目的是摆脱“规则条数与 Service 数量成正比”的代价。这一点它做到了。它从来没做到的是取代 iptables：内核的 IPVS API 本身无法表达完整的 Kubernetes Service 语义——伪装（masquerade）判断、NodePort 过滤、LoadBalancer 源地址段、没有端点的 Service 要返回 reject——所以这个模式在 IPVS 表底下还驱动着 iptables 和 ipset。v1.37 的发布公告就是这么写的。不愿意听信任何人的话，就在自己的节点上把规则数出来。^[kep3866]

# Three separate pieces of kernel state, which is the first surprise for anyone
# who believed ipvs mode meant "no iptables".

# (a) The IPVS virtual servers. This part is what people think of as ipvs mode.
ipvsadm -L -n | wc -l

# (b) A dummy interface holding every ClusterIP - and every LoadBalancer IP -
#     as a /32 on the node. This is why the node answers ARP for addresses it
#     does not own, and therefore why MetalLB in layer-2 mode needs strictARP.
ip -brief addr show kube-ipvs0 | head
# kube-ipvs0  DOWN  10.96.0.1/32 10.96.0.10/32 10.107.44.9/32 ...

# (c) The iptables rules and ipsets that ipvs mode drives underneath, because
#     the kernel IPVS API on its own cannot express masquerade decisions,
#     LoadBalancer source ranges, NodePort filtering or the reject rules for a
#     Service with no endpoints. Count them before you claim to be iptables-free.
ipset list -name | grep -c '^KUBE-'
iptables-save -t nat | grep -c '^-A KUBE-'

# The scheduler in use, which is the setting this whole mode is usually kept for:
kubectl -n kube-system get configmap kube-proxy \
  -o jsonpath='{.data.config\.conf}' | grep -A4 '^ipvs:'
# ipvs:
#   scheduler: "lc"
#   strictARP: true

# One field, for the whole cluster. There is no per-Service scheduler in
# Kubernetes: it is not in the Service API, and kube-proxy does not read one.
# Whatever is on that line is what every Service on every node gets.

流行版本的后半段是调度算法，而这也正是绝大多数还留在 ipvs 上的集群留在这里的原因。有人在很多年前设了一行 scheduler: "lc"，从那以后整个集群就被理解为“在做最小连接数负载均衡”。它不是，而原因是架构性的，不是 bug。kube-proxy 在每个节点上都跑一份，每一份都维护自己的 IPVS 表，而这张表只统计本节点发起的连接。于是“最小连接数”是按客户端节点分别算的：二十个繁忙节点意味着二十个各自独立的本地判断，而不是一个全局判断，而从集群外面进来的流量根本不在任何一份统计里。SIG Network 在写这次废弃时，专门点了这个误解。^[vips]

# The claim to test: "we run lc so connections go to the least loaded pod".
#
# Every node runs its own kube-proxy with its own IPVS table, and that table
# only counts connections this node opened. Ask two nodes about the same
# Service and the connection counts will not agree - because they are answers
# to different questions.

SVC_IP=$(kubectl get svc -n prod api -o jsonpath='{.spec.clusterIP}')
for node in $(kubectl get nodes -o name | head -3); do
  echo "== ${node#node/}"
  kubectl debug "$node" -q -it --image=busybox --profile=general -- \
    chroot /host ipvsadm -L -n -t "$SVC_IP:8080" 2>/dev/null | tail -n +4
done
# == node-01
#   -> 10.244.1.7:8080   Masq  1  118  4
#   -> 10.244.2.4:8080   Masq  1    0  0      <- zero, from THIS node
# == node-02
#   -> 10.244.1.7:8080   Masq  1    2  1
#   -> 10.244.2.4:8080   Masq  1   96  3      <- the other pod, same Service

# Neither node is wrong. "Least connections" is being computed per client node,
# so with N busy client nodes you get N independent local decisions, not one
# global one. Add a client outside the cluster, or a client behind a
# LoadBalancer that lands on a different node, and IPVS never sees it at all.
#
# Clean up the debug pods. `kubectl debug node/...` names them
# node-debugger-<node>-<suffix> and sets no label, so match on the name.
kubectl get pods -o name | grep '^pod/node-debugger-' | xargs -r kubectl delete

# The two things people actually want from a scheduler have supported answers
# that survive this migration, and neither of them is an IPVS scheduler:
#   sticky clients   -> Service .spec.sessionAffinity: ClientIP
#   keep it local    -> Service .spec.internalTrafficPolicy: Local
kubectl get svc -A -o json | jq -r '
  .items[] | select(.spec.sessionAffinity == "ClientIP")
  | "\(.metadata.namespace)/\(.metadata.name) sessionAffinity=ClientIP"'

另外，从来就没有过按 Service 设置的调度算法。ipvs.scheduler 是 kube-proxy 配置里的一个字段，作用于每个节点上的每一个 Service——它不属于 Service API，也没有任何注解能改它。这一点值得停下来想一想，因为它意味着：大家真正想从调度算法那里拿到的两样东西——客户端粘性和节点本地性——在 API 里都有受支持的做法，而且这次迁移完全碰不到它们：sessionAffinity: ClientIP 和 internalTrafficPolicy: Local。^[svc][stp]

这些节点跑得动 nftables 模式吗

接下来是唯一一条硬约束。nftables 模式要求 Linux 5.13 或更新的内核，没有部分支持，也没有回退——低于这个版本，kube-proxy 在该模式下根本起不来。这项检查决定了这场迁移是“改一个 ConfigMap”还是“做一轮节点镜像”，所以在写任何计划之前先做它。^[vips]

#!/usr/bin/env bash
# Precheck. nftables mode needs Linux 5.13 or newer; there is no partial
# support and no fallback - kube-proxy will not start in nftables mode below it.
set -euo pipefail

need_major=5 need_minor=13
fail=0

while read -r node kernel os; do
  ver=${kernel%%-*}                       # 6.8.0-51-generic -> 6.8.0
  IFS=. read -r maj min _ <<<"$ver"
  if (( maj > need_major )) || { (( maj == need_major )) && (( min >= need_minor )); }; then
    printf '  ok    %-22s %s\n' "$node" "$kernel"
  else
    printf '  TOO OLD %-20s %-24s %s\n' "$node" "$kernel" "$os"
    fail=1
  fi
done < <(kubectl get nodes -o custom-columns=\
'NAME:.metadata.name,KERNEL:.status.nodeInfo.kernelVersion,OS:.status.nodeInfo.osImage' \
  --no-headers)

(( fail == 0 )) && echo "all nodes can run nftables mode" \
                || echo "some nodes cannot: rebuild the image or stay on iptables mode"

# On a node that reports too old, the answer is almost never "patch the kernel".
# It is "this node image is end of life". Check what the distro offers before
# planning anything: an Ubuntu 20.04 node on the HWE kernel is fine, the same
# release on the GA kernel is not.
uname -r
nft --version

如果有节点过不了这条线，实话通常是这个节点镜像已经到了生命周期终点，而不是内核需要打补丁，正确的修法是重建镜像。如果在 1.40 之前来不及重建，退路是 iptables 模式，而不是那个特性门控：iptables 后端没有被废弃，它的性能在 ipvs 模式引入之后有过大幅改进，而且上游针对这种情况明确推荐用它而不是 ipvs。对正在权衡时间点的人还可以补一句：这次废弃里提到，所有旧到跑不了 nftables 模式的内核，都会在 2026 年底之前退出长期支持——也就是说，“内核太老”这个反对意见本身也有保质期。^{[kernrel][kep5495]}

切过去之后有四处行为不一样

有四处行为会变，其中三处写在一个标题底下——从 iptables 模式迁移到 nftables——而 ipvs 用户根本没有理由读过那一节。它们在这里同样成立，因为它们是目的地的属性，不是来源的属性。^[vips]

• NodePort 不再监听每一个本地地址。在 iptables 和 ipvs 模式下，一个 type: NodePort 的 Service 在所有本地 IP 上都能访问，除非你自己收窄过。nftables 模式默认是 --nodeport-addresses primary，意思是只有 Node 对象里那个主 IPv4 和／或主 IPv6 地址，别的一个都不算。任何从第二张网卡、管理地址或者浮动 VIP 上访问 NodePort 的东西都会失效。如果确实需要原来的覆盖面，就把这个选项显式写出来——0.0.0.0/0 可以恢复——但先做审计，因为上游认为这个新默认值才是大家本来想要的。
• localhost 上的 NodePort 是另一回事，而 1.37 改了它。连 127.0.0.1:<nodePort> 在 iptables 模式下是通的，在 nftables 模式下则完全不通。从 Kubernetes 1.37 起它可以通，前提是打开 alpha 的 KubeProxyNFTablesLocalhostNodePorts 门控，并且把 nodePortAddresses 设成 primary,localhost。在你判断自己在不在乎之前，先去读那个计数器：kube-proxy 一直在统计从回环口进来、被 NodePort 接受的报文数。
• kube-proxy 不再替你开防火墙。iptables 模式会为每个 NodePort 加一条 accept 规则，理由是本地防火墙如果配得太狠，否则会把它们挡掉。这套做法对基于 nftables 的防火墙根本不成立，所以 nftables 模式在这件事上什么都不做。如果你有主机防火墙，现在必须由它自己放行 NodePort 端口段——可以说本来就该是这样，但它仍然是一个变化。
• 那个 conntrack 绕行方案默认不再安装。6.1 之前的内核有一个 bug，会重置发往 Service IP 的长连接。iptables 模式装了一个绕行方案；后来发现这个方案自己也会带来问题，所以 nftables 模式不装它。你到底需不需要它是可以测出来的，不是一个判断题——计数器就在指标里——如果确实需要，--conntrack-tcp-be-liberal 就是把这个行为拿回来的受支持做法。
• 凡是你自己挂在 kube-proxy 链上的东西，现在都坏了。如果某个防火墙脚本、某个 CNI 钩子或者某条监控规则按名字引用了 KUBE-SERVICES 或 KUBE-SEP-*，这些链已经没有了：kube-proxy 的状态现在住在 table ip kube-proxy 和 table ip6 kube-proxy 里。这从来就不是受支持的用法——上游有一篇标题就是这个意思的长期告示——但值得在铺开之前而不是之后 grep 一遍，因为它是静默失败的。

# The behaviour difference most likely to page you. In ipvs and iptables mode,
# NodePort Services are reachable on every local address unless you said
# otherwise. nftables mode defaults to --nodeport-addresses primary: the node's
# primary IPv4 and/or IPv6 address from the Node object, and nothing else.
#
# So the question to answer before the switch is: does anything reach a NodePort
# on a secondary address, a VIP, a management NIC, or on loopback?

# What is currently configured, if anything:
kubectl -n kube-system get configmap kube-proxy \
  -o jsonpath='{.data.config\.conf}' | grep -i nodePortAddresses
# (empty means "all local addresses")

# Which NodePorts exist at all, and who might be pointed at them:
kubectl get svc -A -o json | jq -r '
  .items[] | select(.spec.type == "NodePort" or .spec.type == "LoadBalancer")
  | .spec.ports[]? | select(.nodePort)
  | "\(.nodePort)\t\(.protocol)"' | sort -u

# Localhost NodePorts are the sharp edge: health checks, sidecars and a
# surprising number of monitoring agents connect to 127.0.0.1:<nodePort>.
# kube-proxy counts them for you, and a non-zero value means something out
# there depends on it.
kubectl -n kube-system exec ds/kube-proxy -- \
  wget -qO- http://127.0.0.1:10249/metrics \
  | grep kubeproxy_iptables_localhost_nodeports_accepted_packets_total
# kubeproxy_iptables_localhost_nodeports_accepted_packets_total 41822

# Kubernetes 1.37 gives that case a way out, as an alpha feature gate. If you
# need it, you need it on 1.37+ and you need both halves:
#   featureGates:
#     KubeProxyNFTablesLocalhostNodePorts: true
#   nodePortAddresses: ["primary", "localhost"]

# And the conntrack workaround: iptables mode installs one for a pre-6.1 kernel
# bug that resets long-lived TCP connections. nftables mode does not, by default.
# Non-zero here means you are relying on it - carry --conntrack-tcp-be-liberal.
kubectl -n kube-system exec ds/kube-proxy -- \
  wget -qO- http://127.0.0.1:10249/metrics \
  | grep kubeproxy_iptables_ct_state_invalid_dropped_packets_total

这五条的形态是一样的：没有一条会让进程起不来。kube-proxy 正常启动、上报健康、绝大部分流量照常转发，然后某一条具体的路径不通了。这就是为什么下面那份切换前的检查比切换本身更值钱，也是为什么第一个节点要单独放满一个工作日。^{[reset][ctsysctl]}

切换本身，一次一个节点

改动只有三行。真正需要纪律的是删掉 ipvs 配置块，而不是把它留在新模式旁边，因为无用的残留配置正是下一个人得出“这个集群还在跑 ipvs”这个结论的原因。用 kubeadm 搭的集群还要多查一件事：这个 ConfigMap 会被 kubeadm upgrade 重新生成，所以只用 kubectl 做的改动可能在下次升级时被悄悄改回去。同样的改动也要落到集群配置里。^[kubeadm]

# The change is three lines, and the discipline is in what you delete.
kubectl -n kube-system get configmap kube-proxy \
  -o jsonpath='{.data.config\.conf}' > kube-proxy.conf.bak
cp kube-proxy.conf.bak kube-proxy.conf

# --- before ----------------------------------------------------------------
#   mode: ipvs
#   ipvs:
#     scheduler: "lc"
#     strictARP: true
#     syncPeriod: 30s
#
# --- after -----------------------------------------------------------------
#   mode: nftables
#   nftables:
#     minSyncPeriod: 1s
#     syncPeriod: 30s
#
# Delete the whole ipvs block rather than leaving it. In nftables mode it is
# dead configuration: scheduler has no equivalent and no effect, and strictARP
# was only ever there to stop the node answering ARP for the ClusterIPs that
# ipvs mode bound onto kube-ipvs0 - an interface nftables mode never creates.
# Leaving it behind is how the next person concludes the cluster is still ipvs.

# Apply. Note that on a kubeadm cluster this ConfigMap is regenerated by
# `kubeadm upgrade`, so make the same change in the cluster configuration or
# the next upgrade will quietly put ipvs back.
kubectl -n kube-system create configmap kube-proxy \
  --from-file=config.conf=kube-proxy.conf \
  --dry-run=client -o yaml | kubectl apply -f -

然后按“任何一个掌管每个节点数据面的东西”该有的方式铺开，也就是先一个节点、再一个节点池、最后全量。这一次灰度的成本异常低，因为状态完全是按节点的，而且每次启动都从 API Server 重建：没有任何共享状态可以被弄坏，而跑不了新模式的节点会在启动时直接失败，错误信息里就带着内核版本。^[drain]

# Do not restart the DaemonSet across the fleet. kube-proxy owns the data plane
# on every node it runs on; a bad rollout is a cluster-wide outage, and this one
# is cheap to canary because the state is per node.

# Pause the DaemonSet so the ConfigMap change does not roll on its own.
kubectl -n kube-system patch ds kube-proxy \
  -p '{"spec":{"updateStrategy":{"rollingUpdate":{"maxUnavailable":1}}}}'

# One node. Cordon it, move the workloads off, restart only that pod.
NODE=node-07
kubectl cordon "$NODE"
kubectl drain "$NODE" --ignore-daemonsets --delete-emptydir-data --timeout=5m
kubectl -n kube-system delete pod \
  --field-selector "spec.nodeName=$NODE" -l k8s-app=kube-proxy

# Watch it come up in the new mode. A node that cannot run nftables mode fails
# here, loudly, with the kernel version in the message - which is the correct
# place to find that out.
kubectl -n kube-system logs -f --tail=40 \
  "$(kubectl -n kube-system get pod -l k8s-app=kube-proxy \
     --field-selector "spec.nodeName=$NODE" -o name)"

kubectl uncordon "$NODE"

# Give it real traffic and a working day before the second node. The failures
# this migration produces are not startup failures; they are "one client on a
# secondary interface stopped reaching a NodePort", and that takes a shift to
# surface. Then a pool, then the fleet.

ipvs 会留下什么

清理是这场迁移与以往任何一次 kube-proxy 模式切换都不同的地方，而且是往好的方向不同。kube-proxy 过去会尝试替别的模式收尾，后来不干了——KEP-2448 把那段逻辑删掉了——因为 iptables、ipvs 和 userspace 这三个后端都会往同一批链里写东西，于是清理其中一个模式的规则会连带删掉正在运行的那个模式的规则。nftables 模式不与任何人共享状态：它的全部状态都在自己那两张表里。这就是为什么朝这个方向切换在设计上会在启动时移除旧规则，也是为什么在需要的时候手工清理是安全的。^{[kep2448][ipvsadm]}

# kube-proxy in nftables mode is designed to remove the iptables and ipvs rules
# it finds on startup - the modes do not share state, which is exactly why this
# direction is safe when switching between the iptables-family modes was not.
# Verify rather than assume, on the first node, before the second one.

ipvsadm -L -n | tail -n +4 | wc -l         # want 0
ip link show kube-ipvs0 2>/dev/null        # want "does not exist"
ipset list -name | grep -c '^KUBE-'        # want 0
iptables-save -t nat | grep -c '^-A KUBE-' # want 0 (or only your own rules)

# If something survived - an older kube-proxy, a node that was rebooted mid-way,
# a third party that wrote into those chains - clear it explicitly. Every one of
# these is safe once kube-proxy is confirmed running in nftables mode on the node.
ipvsadm --clear
ip link delete kube-ipvs0                  # recreated only by ipvs mode
for s in $(ipset list -name | grep '^KUBE-'); do ipset destroy "$s"; done

# The stale kube-ipvs0 addresses are the ones that actually hurt. Left in place
# with no IPVS rules behind them, the node still claims those ClusterIPs and
# still answers ARP for them, and traffic that lands there is dropped rather
# than redirected. That is a black hole that looks like an application problem.

# The strictARP sysctls are set at runtime and do not revert on their own. They
# are harmless, but if you want the node back to stock:
sysctl -w net.ipv4.conf.all.arp_ignore=0
sysctl -w net.ipv4.conf.all.arp_announce=0

# There is also a supported flush. It is documented as cleaning up iptables and
# ipvs rules, so it is the right tool for this direction and the wrong one for
# the other. Run it with kube-proxy stopped on that node.
kube-proxy --cleanup

有一样残留比其余的都值得盯。kube-ipvs0 是 ipvs 模式用来把每一个 ClusterIP——以及每一个 LoadBalancer IP——以 /32 绑到节点上的哑接口，这正是 MetalLB 在二层模式下把 strictARP 列为 ipvs 集群必需项的全部原因。nftables 模式不会创建这个接口，所以那条要求随之失效，这个设置变成无用配置。但如果这个接口在切换之后活了下来、地址还挂在上面、背后却已经没有 IPVS 规则，那么这个节点会继续宣称拥有那些 ClusterIP、继续为它们应答 ARP，然后把流量丢掉。这是一个只在某一个节点上出现的黑洞，表现出来是“应用偶发失败”，也是“清理要去验证而不是假设”最硬的一条理由。^{[metallb][chains]}

# Everything kube-proxy now owns lives in two tables of its own, one per IP
# family, which is the property that makes the whole thing inspectable.
nft list tables
# table ip kube-proxy
# table ip6 kube-proxy

# The Service map, which is the equivalent of what `ipvsadm -L -n` used to show.
# Note that it is a map lookup rather than a rule chain: this is the performance
# argument for the new backend, and it is visible in the output.
nft list table ip kube-proxy | head -40

# One Service end to end:
SVC_IP=$(kubectl get svc -n prod api -o jsonpath='{.spec.clusterIP}')
nft list table ip kube-proxy | grep -A3 "$SVC_IP"

# Live rule changes, which is the closest thing to watching kube-proxy think:
nft monitor rules

# What kube-proxy does NOT own any more - and must not, if the cleanup worked:
nft list ruleset | grep -c 'KUBE-SVC\|KUBE-SEP'   # want 0
ipvsadm -L -n | tail -n +4 | wc -l               # want 0

# A standing caution that predates all of this: kube-proxy's chains and tables
# are not API. If something of yours matched on KUBE-SERVICES by name, it is
# broken now, and it was unsupported before. Hook your own table into the same
# netfilter priorities instead of writing into kube-proxy's.

去验证，而不是去指望

在这场迁移里，验证有一个由故障形态决定的特定形状：几乎所有出错的情况都会让 kube-proxy 继续跑着、继续健康。所以检查 Pod 是不是 Running 什么都证明不了。下面这份脚本检查的是 kube-proxy 实际选中的模式（而不是 ConfigMap 里写的那个）、有没有 ipvs 状态活下来、哑接口是不是没了、规则同步有没有成功——然后做一件任何检查都做不到的事，就是从这个节点上的一个 Pod 里，向一个真实的 Service 发起一次真实的连接。^[dbgsvc]

#!/usr/bin/env bash
# Run on a migrated node, from a machine with kubectl and cluster access.
# Exits non-zero on anything that would be silently wrong. The failures this
# migration produces do not stop kube-proxy, so "the pod is Running" proves
# nothing at all.
set -uo pipefail
NODE=${1:?usage: verify.sh <node>}
NS=${NS:-default}
rc=0
say() { printf '%-46s %s\n' "$1" "$2"; }
chk() { if [[ $2 == "$3" ]]; then say "$1" "ok"; else say "$1" "FAIL ($2 != $3)"; rc=1; fi; }

POD=$(kubectl -n kube-system get pod -l k8s-app=kube-proxy \
      --field-selector "spec.nodeName=$NODE" -o jsonpath='{.items[0].metadata.name}')

# 1. the mode kube-proxy actually chose, not the one in the ConfigMap
mode=$(kubectl -n kube-system logs "$POD" | grep -om1 'Using .* Proxier' | awk '{print $2}')
chk "proxy mode" "$mode" "nftables"

# 2. no ipvs state left on the node
left=$(kubectl debug "node/$NODE" -q --image=busybox --profile=general -- \
       chroot /host sh -c 'ipvsadm -L -n 2>/dev/null | tail -n +4 | wc -l' 2>/dev/null | tr -d ' ')
chk "ipvs virtual servers remaining" "${left:-0}" "0"

# 3. the dummy interface is gone, so no stale ClusterIP black holes
iface=$(kubectl debug "node/$NODE" -q --image=busybox --profile=general -- \
        chroot /host sh -c 'ip link show kube-ipvs0 >/dev/null 2>&1 && echo present || echo absent' \
        2>/dev/null | tr -d ' ')
chk "kube-ipvs0" "${iface:-absent}" "absent"

# 4. sync is succeeding, which is the counter that replaces "is it up"
m=$(kubectl -n kube-system exec "$POD" -- wget -qO- http://127.0.0.1:10249/metrics)
fails=$(awk '/^kubeproxy_sync_proxy_rules_nftables_sync_failures_total/ {s+=$2} END{print s+0}' <<<"$m")
chk "nftables sync failures" "$fails" "0"
awk '/^kubeproxy_sync_proxy_rules_last_timestamp_seconds/ {print "  last successful sync:", $2}' <<<"$m"

# 5. and the part no inspection can establish: a real connection to a real
#    Service, from a pod on this node, plus a NodePort from off-box.
kubectl -n "$NS" run "nftcheck-$$" --rm -i --restart=Never \
  --overrides="{\"spec\":{\"nodeName\":\"$NODE\"}}" \
  --image=curlimages/curl -- \
  curl -sS -o /dev/null -w '%{http_code}\n' --max-time 5 \
  http://kubernetes.default.svc.cluster.local:443 >/dev/null 2>&1 \
  && say "in-cluster Service connect" "ok" \
  || { say "in-cluster Service connect" "FAIL"; rc=1; }

exit $rc

之后有三个计数器值得配两周告警。注意这份清单里缺了什么：Kubernetes 的指标参考里有 iptables 专属计数器，有 nftables 专属计数器，唯独没有任何一个 ipvs 的。你今天对 ipvs 模式的所有监控，都是你自己拿 ipvsadm 拼出来的——这正是整个废弃背后那条“维护”论据一个很小也很具体的例证。^[metrics]

# kube-proxy serves these on 10249 on every node. Three of them are worth an
# alert for the fortnight after the migration; the rest are for the postmortem.

# Sync is failing on this node - the data plane is drifting from the API server:
sum by (node) (rate(kubeproxy_sync_proxy_rules_nftables_sync_failures_total[5m])) > 0

# Cleanup is failing - usually leftovers from the previous mode, or something
# else writing into the same tables:
sum by (node) (rate(kubeproxy_sync_proxy_rules_nftables_cleanup_failures_total[5m])) > 0

# Rules are stale. This is the one that catches the silent failure, because
# kube-proxy stays Running while it happens:
time() - max by (node) (kubeproxy_sync_proxy_rules_last_timestamp_seconds) > 300

# And the pre-migration baseline worth keeping: sync duration before and after.
# In clusters with a few thousand Services the improvement is the point; in a
# cluster with forty Services there is nothing to see and that is fine too.
histogram_quantile(0.99, sum by (le) (rate(kubeproxy_sync_proxy_rules_duration_seconds_bucket[5m])))

# Note what is not in this list. The Kubernetes metrics reference has
# iptables-specific counters and nftables-specific counters and no ipvs-specific
# counter at all. Whatever observability you have for ipvs mode today, you built
# it yourself out of ipvsadm - which is its own argument about maintenance.

怎么回滚，以及回滚能换来什么

这里的回滚是真便宜，而今年落在这批节点上的其他迁移都做不到这一点。没有数据要转换，没有磁盘格式要降级，也没有能活过重启的状态：整个改动就是一个 ConfigMap 键，加上每次启动都由 kube-proxy 从 API Server 重建的、按节点存在的内核状态。把旧的键放回去，删掉这个节点上的 Pod，一分钟之内节点就回到原处。^[kep3866]

# Rollback here is genuinely cheap, which is not true of most migrations on
# these nodes. There is no data to convert and no format to downgrade: the whole
# of the change is one ConfigMap key and per-node kernel state that is rebuilt
# from the API server on every start.

kubectl -n kube-system create configmap kube-proxy \
  --from-file=config.conf=kube-proxy.conf.bak \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl -n kube-system delete pod \
  --field-selector "spec.nodeName=$NODE" -l k8s-app=kube-proxy

# kube-proxy in ipvs mode removes the nftables rules it finds on startup. If it
# does not - and rollback matters most exactly when the new backend is
# misbehaving - the manual version is two commands, because every rule
# kube-proxy owns is inside its own tables:
nft delete table ip kube-proxy
nft delete table ip6 kube-proxy

# Then confirm the old data plane is actually back, rather than assuming:
ipvsadm -L -n | tail -n +4 | wc -l     # want non-zero again
ip -brief addr show kube-ipvs0         # the dummy interface returns

# Be honest about what the rollback bought. ipvs mode is off by default in
# Kubernetes 1.40 and the code is deleted in 1.43. A rollback ends a bad
# maintenance window; it does not move the date.

有两点提醒，都是关于范围而不是机制。ipvs 模式下的 kube-proxy 在设计上会删掉启动时发现的 nftables 规则，但回滚最要紧的场合恰恰是新后端行为不正常的时候，所以手动版本值得知道：kube-proxy 拥有的每一条规则都在它自己的表里，两条 nft delete table 就能全部清掉。另外要把回滚到底换来了什么说清楚。它结束了一个搞砸的维护窗口。它挪不动 1.40，也挪不动 1.43。^[skew]

另一个答案：干脆不跑 kube-proxy

这次废弃还有第二个答案，值得点名，因为对某些集群来说它是更好的那个：干脆不跑 kube-proxy。Cilium 和 Calico 都直接用 eBPF 实现 Kubernetes Service，用哈希表查找代替规则列表，而且两者的文档都是“把 kube-proxy 撤掉”而不是“和它并存”。如果你本来就在跑其中之一，这次 ipvs 废弃可能正好是一个契机，去做一件你迟早要做的改动。^[cilium]

但它比本文描述的那件事大得多，也应该按“大得多”来掂量。替掉 kube-proxy 等于把 Service 的实现挪进 CNI，从此你的数据面被绑在那个项目的发版节奏、它的内核要求（比 5.13 更高，不是更低）和它的调试工具上，这不是在一个有时间压力的废弃窗口里该做的决定。对大多数集群来说，诚实的顺序是：现在先切到 nftables，因为那只是改一个 ConfigMap；至于 eBPF 那个问题，按它自身的价值和自身的排期单独评估。^{[calico][netpol]}

这件事该按什么顺序做

压缩之后，这个决定比这篇文章小得多。如果你的内核都在 5.13 或更新，这就是一次配置变更加一次谨慎的铺开。如果不是，这就是一轮节点镜像的活，而过渡答案是 iptables 模式，不是那个特性门控。而如果你的 kube-proxy 配置里压根没写模式，那不管上面这些跟你有没有关系，这一周该修的就是它。^[kep5495]

1. 把模式钉死，不管它是哪个。如果 kube-proxy ConfigMap 里的 mode: 是空的，今天就把它显式写出来。这是清单上唯一一条对“完全没有 ipvs”的集群也成立的事项，也是唯一一条期限不是你自己定的。
2. 先查内核，再写计划。全机群跑一条命令，就能决定这件事是一个下午还是一个季度。低于 5.13 的节点需要的是新镜像，不是补丁。
3. 把决定尖锐边角的那两个计数器量出来。localhost 上的 NodePort 和 conntrack 的 invalid 丢包，这两件事在你动任何东西之前，都能从 kube-proxy 自己的指标里问出答案。顺手把 NodePort 的监听地址一起审计掉。
4. 一个节点，一个完整工作日，然后才是一个节点池。这里的回归不是启动失败，而是“第二张网卡上的某一个客户端”，这种东西要一个班次才浮得出来。第一个节点上的 kube-ipvs0 和那些 ipset，请手工确认清理干净。
5. 删掉 ipvs 配置块和 strictARP 设置。它们在 nftables 模式下都不起作用，留着它们正是下一个人得出“这次迁移根本没做过”的原因。然后到 ConfigMap 真正生成的地方做同样的改动，否则下一次 kubeadm upgrade 会把它撤回去。

这是同一年落在同一批节点上的几项变更之一，合起来读更省事：从 Ingress NGINX 迁移到 Gateway API，网络这边的另一半工作，铺开的纪律也一模一样；containerd 1.7 升级到 2.x，底下那层运行时，有它自己的退役时间表；以及 从 cgroup v1 迁移到 cgroup v2，Kubernetes 已经强制要求的那项节点级变更。如果你在权衡这些东西到底需要多少，什么时候不该用 Kubernetes 是这个论证的另一面。

常见问题

Kubernetes 1.37 里 kube-proxy 的 ipvs 模式被移除了吗？

没有。Kubernetes 1.37 加的是 KubeProxyIPVS 特性门控，默认值为 true，所以 ipvs 模式的行为和以前完全一样。真正要记的日期是 1.40 和 1.43：1.40 时门控默认翻成 false，在 ipvs 模式下的 kube-proxy 如果不去覆盖它就会退出报错；1.43 时 pkg/proxy/ipvs 被删除，门控也再帮不上忙。门控本身在 1.46 被移除。

怎么知道我的集群在用哪个代理模式？

在 kubeadm 搭的集群上：kubectl -n kube-system get configmap kube-proxy -o jsonpath='{.data.config\.conf}' | grep 'mode:'。如果结果是空的，说明这个集群没有选过模式，跑的是 kube-proxy 当前推荐的那个——在 1.37 里是 iptables，未来某个版本会变成 nftables。如果集群不使用那个 ConfigMap，就去读 DaemonSet 的启动参数，或者直接在节点上跑 ipvsadm -L -n。

一定要迁到 nftables 吗，还是可以用 iptables 模式？

iptables 模式是一个正当的终点。它没有被废弃，没有内核底线，而且在 ipvs 模式引入之后的这些年里性能有过大幅改进——对于老到跑不了 nftables 模式的系统，上游明确推荐用它而不是 ipvs。在内核允许的地方，nftables 是更好的目标，默认值也在往那边走；但“从 ipvs 切到 iptables”对老节点镜像来说是一个真实的答案，不是敷衍。

kube-proxy 的 nftables 模式需要什么内核版本？

Linux 5.13 或更新，而且只支持 Linux 节点。没有部分支持：低于这个版本，kube-proxy 不会以 nftables 模式启动。实际上这排除了 RHEL 8、CentOS 7、Debian 11 以及使用 GA 内核的 Ubuntu 20.04，而 RHEL 9 和 10、Debian 12、Ubuntu 22.04 与 24.04、Amazon Linux 2023 都过得去。请用 kubectl get nodes -o custom-columns=NAME:.metadata.name,KERNEL:.status.nodeInfo.kernelVersion 去查，而不是按发行版名字推断。

我配的 IPVS 调度算法会丢吗？

设置会丢，而你以为它带来的那个行为，基本上从来就不存在。ipvs.scheduler 是整个集群共用的一个字段，不是按 Service 设置的，而且每个节点的 IPVS 表只统计本节点发起的连接——所以 lc 是“按客户端节点算的最小连接数”，不是按集群算的。如果你要的是客户端粘性，那是 Service 上的 .spec.sessionAffinity: ClientIP；如果你要的是节点本地性，那是 .spec.internalTrafficPolicy: Local。这两个都不受这次迁移影响。

切到 nftables 模式之后还需要 strictARP 吗？

不需要。strictARP 之所以存在，是因为 ipvs 模式会把每一个 ClusterIP 和 LoadBalancer IP 绑到 kube-ipvs0 这个哑接口上，导致节点为 MetalLB 想要控制的地址应答 ARP。nftables 模式从不创建这个接口，所以底层问题消失了，这个设置变成无用配置——把它和整个 ipvs 配置块一起删掉。但请务必在每个迁移完的节点上确认 kube-ipvs0 真的没了，因为一个还挂着过期地址的残留接口会把流量吞进黑洞。

在跑着的集群上切模式安全吗？

在 iptables 系模式和 nftables 之间切换，设计上是安全的：nftables 模式把全部状态放在自己那两张表里，并会在启动时移除它发现的 iptables 和 ipvs 规则，回滚方向同理。这和在 iptables 与 ipvs 之间切换是两回事——那两者共享链，也正因如此 kube-proxy 的自动清理才在 KEP-2448 里被删掉。但它仍然是一次落在每个被触及节点上的数据面变更，所以要 cordon、drain、只重启一个 kube-proxy Pod、验证，然后让这个节点单独待满一个工作日。

这场迁移里最常出问题的是什么？

NodePort 的可达性。nftables 模式默认走 --nodeport-addresses primary，所以原本通过第二张网卡、管理地址、浮动 VIP 或者 127.0.0.1 访问的 NodePort 会不再应答——与此同时 kube-proxy 一直健康，别的一切照常。这两种情况都能提前量出来：从 kube-proxy 自己的指标，以及当前的 nodePortAddresses 设置。

要不要干脆用 Cilium 或 Calico 的 eBPF 替掉 kube-proxy？

这是一个真实的选项，对某些集群还是更好的那个，但它比换一个模式大得多：它把 Service 的实现挪进 CNI，把你的数据面绑在那个项目的发版、内核要求和调试工具上。在废弃的压力下做这个决定，顺序是反的。切到 nftables 只是改一个 ConfigMap，而且能把期限拿掉；eBPF 那个问题放到之后，按它自己的排期评估。

如果 nftables 模式表现不正常，怎么回滚？

把之前那份 ConfigMap 的键恢复回去，然后删掉该节点上的 kube-proxy Pod；旧模式会在启动时从 API Server 重建自己的状态，所以节点一分钟之内就回去了。如果 kube-proxy 自己的清理没有跑——而这恰恰就是回滚要紧的那种场合——它拥有的每一条规则都在自己的表里，所以 nft delete table ip kube-proxy 和 nft delete table ip6 kube-proxy 就能全部清掉。回滚结束的是一个搞砸的维护窗口；它挪不动 1.40，也挪不动 1.43。

补一条版本层面的说明，因为 1.37 的账本和大多数报道说的并不一样：真正可能让 Pod 卡在 ContainerCreating 的是 SELinuxMount 转为 GA；cgroup v1 的失败在 1.35 落地，静态 Pod 的限制在 1.34，而 containerd 的悬崖还在前面的 1.38。升级到 Kubernetes 1.37 时真正会坏的东西把这三栏拆开，并给出应当在升级之前、而不是之后执行的排查。

参考资料

只放一手资料，核对时间为 2026 年 8 月 27 日。在这次废弃的版本号上，如果 Kubernetes 官方文档与某篇二手文章打架，该信的是文档和 KEP——阶段划分已经被修订过一次了。

1. Kubernetes - Virtual IPs and Service Proxies: the reference page for every kube-proxy mode. It carries the deprecation notice for ipvs mode with the 1.40 and 1.43 dates, the kernel 5.13 requirement for nftables mode, the full list of IPVS schedulers and the ipvs.scheduler field they are set through, the four documented behaviour differences when migrating to nftables, and the sentence that matters most to clusters that are not on ipvs at all: the default mode is iptables in 1.37 and a future release will change it to nftables
2. Kubernetes v1.37: Garhwal - the release announcement of 26 August 2026. One of the 67 enhancements is a deprecation and it is this one. The deprecation section states the timeline, gives the jsonpath one-liner for finding out which mode a cluster is running, and says outright that ipvs mode continues to use iptables underneath because the kernel IPVS API alone cannot implement Kubernetes Services
3. KEP-5495: Deprecate ipvs mode in kube-proxy - the enhancement proposal itself, and the only document that carries all five stages. Stage 2 in 1.37 adds the KubeProxyIPVS feature gate; stage 3 in 1.40 flips it off by default; stage 4 in 1.43 removes pkg/proxy/ipvs; the cleanup stage in 1.46 removes the gate. It also records why: SIG Network has no maintainers familiar with the ipvs backend, and the kernels too old for nftables mode will be out of LTS by the end of 2026
4. KEP-3866: Add an nftables-based kube-proxy backend - the design document for the mode you are migrating to, including the section titled "The ipvs mode of kube-proxy will not save us", the reasoning behind switching modes being safe in this direction when it was not between the iptables-family modes, and the two nft commands that remove every rule kube-proxy owns
5. KEP-265: IPVS load balancing mode in Kubernetes - the original 2017 proposal, worth reading now mainly to see which of its promises were kept and which were quietly not
6. KEP-2448: Remove kube-proxy automatic clean-up logic - why kube-proxy stopped trying to tidy up after the other modes, and therefore why the cleanup step in this migration is something you do rather than something that happens
7. kube-proxy command line reference: --proxy-mode, --cleanup, --nodeport-addresses, --conntrack-tcp-be-liberal, --ipvs-scheduler, --ipvs-strict-arp and the rest. Note that --cleanup is documented as cleaning up iptables and ipvs rules, which is the direction that matters here
8. KubeProxyConfiguration API reference: the schema of the config.conf that lives in the kube-proxy ConfigMap, including the mode field, the ipvs and nftables sections, and which options are read in which mode
9. Kubernetes feature gates: where KubeProxyIPVS and KubeProxyNFTablesLocalhostNodePorts are listed with their stage and default, and the reference for how a gate is passed to a component that is not the API server
10. Kubernetes metrics reference: the exact names of the kube-proxy counters used in this article, including kubeproxy_sync_proxy_rules_nftables_sync_failures_total, kubeproxy_iptables_ct_state_invalid_dropped_packets_total and kubeproxy_iptables_localhost_nodeports_accepted_packets_total. Also, by omission, the fact that there is no ipvs-specific counter anywhere in the list
11. Kubernetes deprecation policy: what a deprecation of a component flag or behaviour actually commits the project to, which is the frame for reading the KEP-5495 stages as dates rather than intentions
12. Kubernetes releases: the supported branches and their end-of-life dates, which is how you turn "1.40" and "1.43" into calendar quarters for your own cluster
13. Kubernetes version skew policy: how far kube-proxy is allowed to lag the API server and the kubelet, which bounds how long a partially migrated fleet can stay partially migrated
14. Kubernetes v1.35: Timbernetes - the release where stage 1 of this deprecation landed and kube-proxy started logging a warning on startup in ipvs mode. If nobody in your organisation noticed, that is the point
15. Kubernetes v1.36: Haru - the release in between, useful for placing the deprecation on the same timeline as the other node-level changes of 2026
16. Kubernetes v1.37 sneak peek: the pre-announcement of the same deprecation, published four weeks before the release
17. Kubernetes - Service: the API that all of this implements, and the reference for sessionAffinity, which is the feature people mistakenly believe the IPVS sh scheduler is providing
18. Kubernetes - EndpointSlices: the objects kube-proxy actually watches, and the reason rule-sync cost scales with endpoint churn rather than with Service count alone
19. Kubernetes - Service internal traffic policy: the supported way to keep traffic on the local node, which is the thing IPVS locality-based schedulers are sometimes reached for instead
20. Kubernetes - Cluster networking: where kube-proxy sits relative to the CNI plugin, which decides whether any of this applies to your cluster at all
21. Kubernetes - Debug Services: the official checklist for a Service that does not answer, and the first thing to run when a node comes back on a new proxy mode
22. Kubernetes - Safely drain a node: the cordon, drain and uncordon sequence this migration slots into, one node at a time
23. Kubernetes - Upgrading kubeadm clusters: for kubeadm-built clusters, the place the kube-proxy DaemonSet and its ConfigMap come from, and the reason a config change can be reverted by the next upgrade if it is not also made in the cluster configuration
24. Kubernetes blog - Kubernetes's iptables chains are not API: the standing warning that anything of yours which hooks into kube-proxy's own chains is unsupported. It is the single best predictor of what will break when the chains are replaced by nftables tables
25. Kubernetes blog - IPVS-based in-cluster load balancing deep dive: the 2018 introduction to the mode being retired, including the kube-ipvs0 dummy interface and the ipset usage that this article tells you to go and clean up
26. Kubernetes blog - kube-proxy subtleties, debugging an intermittent connection reset: the original write-up of the conntrack invalid-state problem whose workaround nftables mode does not install by default
27. kubernetes/kubernetes, pkg/proxy/ipvs: the directory KEP-5495 stage 4 deletes. Worth a look if you want to see for yourself how much iptables the ipvs mode is driving
28. kubernetes/kubernetes, pkg/proxy/nftables: the implementation you are moving to, and the authority on which table and chain names to expect on a migrated node
29. nftables wiki: the syntax reference for reading what kube-proxy now writes, in particular sets, maps and verdict maps, which are the features the iptables API cannot express and the reason the new backend is faster
30. nft(8) manual page: list, delete, monitor and the ruleset commands used in the verification section
31. The netfilter project's nftables page: the upstream statement of what nftables replaces and why development moved there
32. Linux kernel documentation - nf_conntrack sysctls: nf_conntrack_tcp_be_liberal, which is what --conntrack-tcp-be-liberal sets, and the surrounding timeouts kube-proxy also manages
33. kernel.org - active kernel releases: the longterm branches and their projected end-of-life dates, which is how to check the KEP's claim that every kernel too old for nftables mode leaves LTS by the end of 2026
34. ipvsadm(8): the tool for reading and clearing the IPVS table that kube-proxy leaves behind, including -L -n for inspection and -C for the flush used in the cleanup step
35. MetalLB installation: the source of the strict ARP requirement for kube-proxy in ipvs mode. It is an ipvs-only requirement because it works around an ipvs-only behaviour, which is why it stops applying after this migration
36. Cilium - Kubernetes without kube-proxy: the other answer to this deprecation, which is to stop running kube-proxy at all and let an eBPF data plane implement Services
37. Calico - enabling the eBPF data plane: the same answer from the other major CNI, including the requirement to disable kube-proxy rather than run both