这些并非都是 1.37 的新变化。

Kubernetes v1.37 已于 2026 年 8 月 26 日发布，而围绕它的那些解读，有一个值得注意的形状。有好几项变更被算到了这个版本头上，可它们并不属于这里：kubelet 在 cgroup v1 上拒绝启动（那个默认值是在 v1.35 翻面的）、静态 Pod 失去对 Secret 和 ConfigMap 的引用（自 v1.34 起就默认开启），以及 kube-proxy 的 ipvs 模式被移除（它自 v1.35 起就带着弃用告示，移除目标是 v1.43）。还有一项被朝相反方向说错了：containerd 1.x 至今仍能跑在 v1.37 的 kubelet 下，悬崖在 v1.38。与此同时，这个版本里真正可能在升级当天让 Pod 卡在 ContainerCreating 的那一项——SELinuxMount 转正为 GA 并默认开启——只分到一个段落，因为对于集群不跑 SELinux 的人来说，它是隐形的。

这个区分不是抠字眼，因为每一栏对应的活儿都不一样。如果 cgroup 那件事是在 1.37 上让你吃了一惊，那你其实已经晚了两个版本，而且修法是每个节点重启一次，不是回滚。如果有人告诉你 containerd 1.x 已经不能用了、你因此慌慌张张地重建容器运行时，那你会把一个维护窗口花在一个还差一个版本才到期的期限上——它是真的，但不是今天。而如果 SELinux 那件事是在 1.37 上让你吃了一惊，那你手上就有起不来的 Pod，而本来能提前找出它们的那次审计，在升级完成之后会明显更难做。所以这篇文章做两件事：先老老实实把账分开，然后深入讲需要动手的部分——完整的 SELinux 审计和退出开关、ipvs 与 containerd 各自真实的时间线，以及一份把不可逆步骤排在正确位置上的操作手册。

各种故障现象，以及它们各自属于哪个版本

先从你实际会看到的现象说起，因为这些故障没有一个会自报家门。一个 Pod 在某个节点上永远停在 ContainerCreating，而同样的 Pod 在另一个节点上跑得好好的——这是 SELinux 标签冲突，决定成败的是谁先拿到了那个卷。一个节点升级完回来，kubelet 根本起不来——这几乎肯定是 cgroup v1，而且从 v1.35 起就是这样了。一个从 2021 年就以静态 Pod 形式运行的监控代理，偏偏只在你刚升级的那一个节点上挂了——那是它本来就不该能用的 Secret 引用，而拿走这项能力的那个默认值，早在 v1.34 就已经落地。^[sneak]

值得内化的规律是：一次版本升级会把你上次心里有底的那个版本之后的所有变更一次性摊到你面前，而不只是你要升到的那个版本里的变更。现实中集群并不是一个小版本一个小版本地升；它们在 1.34 上待一年，然后一步迈过来。Service 的 ExternalIPs 就是眼下的例子：它在 v1.36 被弃用，从那时起 API 服务器在每次使用时都会发出警告，而从更早版本迈向 v1.37 的人，是第一次撞见这条警告。目前它什么都还没被移除——kube-proxy 这一侧的支持预计不早于 v1.40 才默认关闭，移除也不早于 v1.43——而这恰恰就是一项还在三个版本之外的变更如何变成一次故障的：警告出现在一个没人读过说明的版本里。这也正是本文第二节是一本账、而不是一份发布说明摘要的原因。^[extip]

把账算清楚：1.37 的新变化，与早就落地的东西

下面是老实的算法。左栏是发布团队自己的预告里为 v1.37 列出的内容；右边几栏是这项变更真正的出处，或者它还要往哪里去。凡是落在“更早”那几行里的东西，都是你本该已经处理掉的；如果还没处理，升级窗口就是你发现它的时刻。而落在“更晚”那几行里的，是该写进日历的日期，不是这周的活儿。^[sneak]

本文最有用的一句话：如果你的节点没有以 enforcing 模式运行 SELinux，这里最长的那一节对你完全不适用。当 SELinux 不可用或在内核中被禁用时，kubelet 会跳过整条 SELinux 代码路径。先查这一项——一条命令就够——因为它决定了这是半天的审计工作，还是十分钟的阅读。

其中有两项值得说说怎么自己去核实，而不是听别人怎么说。特性门控参考文档会公布 SELinuxMount、SELinuxChangePolicy 和 KubeProxyIPVS 在每个版本上的状态，这是核对“哪个版本翻了哪个默认值”这类说法最快的办法。另外，每个版本的发布团队博客都带着自己的弃用清单；读三篇要二十分钟，而这比升级窗口里塞进去的大多数事情都更值。^[gates]

你实际站在什么位置

在这一切变得可执行之前，你需要四个数字，而且它们彼此独立：每个节点的 kubelet 版本、该节点上 SELinux 是否处于 enforcing、它跑的是哪个 cgroup 版本，以及 kube-proxy 处于哪个模式。在任何有规模的机器群里，答案都不会整齐划一——节点池各自按自己的节奏推进，而版本偏差策略明确允许 kubelet 落后于 API 服务器，所以版本参差的机器群是受支持的配置，而不是疏于管理的证据。^[skew]

# Four numbers decide how much of this article applies to you, and they are
# independent of each other. Ask for all four rather than assuming.

# 1. Control plane and kubelet versions. Node pools drift; this is normal.
kubectl get nodes -o custom-columns=\
'NODE:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion,'\
'RUNTIME:.status.nodeInfo.containerRuntimeVersion,'\
'KERNEL:.status.nodeInfo.kernelVersion,OS:.status.nodeInfo.osImage'
# NODE     KUBELET   RUNTIME              KERNEL          OS
# node-01  v1.36.4   containerd://2.3.2   6.8.0-51        Ubuntu 24.04.3 LTS
# node-02  v1.35.9   containerd://1.7.28  5.15.0-118      Ubuntu 22.04.5 LTS   <- two problems

# 2. Is SELinux actually in play? If the answer is "no" on every node, the
#    largest section of this article does not apply to you at all.
for n in $(kubectl get nodes -o name); do
  printf '%-22s ' "${n#node/}"
  kubectl debug "$n" -q -it --image=busybox --profile=general -- \
    chroot /host sh -c 'getenforce 2>/dev/null || echo "not installed"' 2>/dev/null
done
# node-01  Enforcing        <- section "SELinux" applies
# node-02  not installed    <- it does not

# 3. Which cgroup version. v2 shows cgroup2fs; v1 shows tmpfs.
kubectl debug node/node-01 -q -it --image=busybox --profile=general -- \
  chroot /host stat -fc %T /sys/fs/cgroup
# cgroup2fs

# 4. Which kube-proxy mode. This is the one people are most often wrong about,
#    because the answer usually predates everyone currently on the team.
kubectl -n kube-system get configmap kube-proxy \
  -o jsonpath='{.data.config\.conf}' | grep -E '^\s*mode:'
#   mode: "ipvs"

# Clean up the debug pods. `kubectl debug node/...` names them
# node-debugger-<node>-<suffix> and applies no label, so there is nothing to
# select on - match the name, or they accumulate silently.
kubectl get pods -o name | grep '^pod/node-debugger-' | xargs -r kubectl delete

另一个要紧的数字，是你当前所在的版本实际还剩多少余量。Kubernetes 支持最近三个小版本，每个大约十四个月，其中最后两个月进入维护模式，只有关键安全修复才会落地。这个时间表正是升级并非可选项的原因，当有人提议把这次升级往后拖时，把它摆在桌面上很有用。^[k8srel]

SELinuxMount 转正为 GA：会让 Pod 起不来的那一项

这一节才是这篇文章存在的理由。SELinuxMount 在 v1.37 转正为 GA，并且默认启用。这项变更本身是性能上的收益，机制也很漂亮：容器运行时不再遍历整个卷、给每个 inode 重新打标签——在大卷或远端文件系统上这确实很慢——而是由 kubelet 用 -o context=<label> 挂载该卷，内核以常数时间把标签应用到这个挂载点上的每个 inode。问题正是这个机制的直接后果。一个挂载点只能持有一个 SELinux 上下文。在递归重新标记的做法下，两个标签不同的 Pod 可以共享一个卷；在上下文挂载的做法下则不行，其中一个会一直停在 ContainerCreating，直到另一个消失。^{[selblog][kep1710]}

影响范围比听上去要窄，而且值得精确算一遍，而不是往最坏处想。要让一个卷的行为发生变化，五个条件必须同时成立，其中最常被漏掉的是 CSI 驱动这一条：只有当驱动在自己的 CSIDriver 对象上设置了 seLinuxMount: true、声明自己能接受上下文挂载时，kubelet 才会使用上下文挂载。没有设置这个字段的驱动——而未设置不等于 true——保持原来的递归行为，完全不受这次翻面影响。树内卷类型中支持该挂载选项的是 fc、iscsi 和 rbd；其余树内插件一律仍然递归重新标记。^[csidriver]

# The blast radius of the SELinuxMount change is not "clusters with SELinux".
# It is the intersection of three things, and all three have to be true before
# a single pod is at risk.

# (a) SELinux enforcing on the node. Checked above. If not, stop here.

# (b) A CSI driver that has opted in. The kubelet only uses the mount option
#     when the driver declares it can take one. Drivers that do not set this
#     keep the old recursive relabel and are unaffected by the flip.
kubectl get csidrivers \
  -o custom-columns='DRIVER:.metadata.name,SELINUXMOUNT:.spec.seLinuxMount'
# DRIVER                    SELINUXMOUNT
# ebs.csi.aws.com           true      <- volumes on this driver change behaviour
# efs.csi.aws.com           false     <- unchanged
# csi.trident.netapp.io     <none>    <- unset is not true; unchanged

# The in-tree volume types that support the mount option are fc, iscsi and rbd.
# Everything else in tree relabels recursively regardless.

# (c) Two pods with different SELinux labels sharing one volume. This is the
#     part you cannot infer from a manifest, because the label is often
#     assigned by the runtime rather than written down. The cheapest proxy is
#     to find the volumes that more than one workload mounts at all:
kubectl get pods -A \
  -o jsonpath='{range .items[*]}{range .spec.volumes[?(@.persistentVolumeClaim)]}'\
'{.persistentVolumeClaim.claimName}{"\t"}{end}{.metadata.namespace}{"\n"}{end}' \
  | awk -F'\t' 'NF>1' | sort | uniq -c | sort -rn | awk '$1>1'
#   3 shared-media   default
#   2 build-cache    ci

# That list is a starting point, not an answer. The answer comes from the
# controller below, which knows the labels.

# One more thing worth knowing before you panic: a pod that mounts a volume
# through different subPaths used to be able to share it across labels too.
# That case also stops working, and it is rare enough that upstream says it has
# never been seen in practice.

会出问题的共享模式有两种，而现实中只会遇到其中一种。第一种是两个 Pod 通过不同的 subPath、带着不同标签共享同一个卷，上游称之为极其小众，并且说从未在实践中见过。第二种是一个特权 Pod 和一个非特权 Pod 共享一个卷——同样不常见，但在真实应用中确实观察到过，也正是你该去找的那一种。如果你是那个要为这次升级签字的人，KEP 自己写的升级说明值得一读，那是目前最接近官方操作手册的东西。^[kepstory3]

升级之前必须做完的审计

Kubernetes v1.36 就为这个问题发布了一个控制器，而几乎没有人把它打开，因为它需要显式启用，也因为它要警告的那件事当时还没发生。selinux-warning-controller 运行在 kube-controller-manager 内部，通过 --controllers=*,selinux-warning-controller 启用，它监视集群里的每一个 Pod，并报告每一对以 SELinuxMount 不允许的方式共享卷的 Pod。即使这两个 Pod 当前位于不同节点，它也照样报告，理由很正当：调度器明天可能就把它们放到一起。^[k8s136]

# Kubernetes v1.36 shipped a controller whose entire job is to find these
# conflicts before the upgrade turns them into stuck pods. It is off by
# default and it is the single most useful thing in this article.
#
# It runs inside kube-controller-manager. `*` keeps every default controller
# and adds this one; listing it alone would disable all the others.

# kubeadm clusters: edit the static pod manifest on each control plane node.
sudo vi /etc/kubernetes/manifests/kube-controller-manager.yaml
#   spec:
#     containers:
#     - command:
#       - kube-controller-manager
#       - --controllers=*,bootstrapsigner,tokencleaner,selinux-warning-controller
#                                                      ^^^^^^^^^^^^^^^^^^^^^^^^^
# The kubelet restarts the pod when the file changes. Give it a minute.

# Note the second requirement, which is easy to miss: you must NOT have
# explicitly disabled the SELinuxChangePolicy feature gate. It is GA and on by
# default, so this only bites clusters carrying an old --feature-gates line.
kubectl -n kube-system get pod -l component=kube-controller-manager \
  -o jsonpath='{.items[*].spec.containers[*].command}' | tr ',' '\n' \
  | grep -E 'feature-gates|controllers='

# Confirm it is running before you trust its silence:
kubectl -n kube-system logs -l component=kube-controller-manager --tail=200 \
  | grep -i 'selinux'
# ... "Starting controller" controller="selinux-warning-controller"

# Enabling it has one privacy consequence worth stating: the metric it emits
# carries namespace names as labels, so it can leak namespace names to anyone
# who can read kube-controller-manager metrics. Upstream's assumption is that
# only cluster administrators can.

然后要把两个指标都读一遍，因为它们回答的是不同的问题，单看任何一个都不够。控制器的 selinux_warning_controller_selinux_volume_conflict 把冲突双方的 Pod 名称和命名空间作为标签带出来——那就是你的待办清单。kubelet 的 volume_manager_selinux_volume_context_mismatch_warnings_total 完全没有 Pod 名称标签，但它是“真正会失败的 Pod 有多少个”的诚实计数，而且是在 SELinuxMount 仍处于关闭状态时发出的。最后这半句正是为什么要在 v1.36 上做这件事：警告计数不为零，而一切仍然正常工作。升级之后，同一次测量会以 ..._errors_total 的形式出现，而那时候 Pod 已经卡住了，不再只是有风险。^[selblog]

# There are two metrics and they answer two different questions. You need both:
# one tells you WHICH pods, the other tells you HOW MANY will actually fail.

# --- (1) From kube-controller-manager: which pods conflict, by name ---------
# Reported even when the pods are on different nodes, because the scheduler
# may put them together tomorrow.
kubectl get --raw /metrics 2>/dev/null | true   # via your metrics stack, or:
kubectl -n kube-system exec -it \
  "$(kubectl -n kube-system get pod -l component=kube-controller-manager \
     -o name | head -1)" -- \
  curl -sk https://127.0.0.1:10257/metrics \
  | grep '^selinux_warning_controller_selinux_volume_conflict'
# selinux_warning_controller_selinux_volume_conflict{
#   pod1_name="my-other-pod",pod1_namespace="default",
#   pod1_value="system_u:object_r:container_file_t:s0:c0,c1",
#   pod2_name="my-pod",pod2_namespace="default",
#   pod2_value="system_u:object_r:container_file_t:s0:c0,c2",
#   property="SELinuxLabel"} 1

# --- (2) From the kubelet: how many pods would actually fail ---------------
# Emitted while SELinuxMount is still DISABLED, which is exactly the window
# you are in before the upgrade. It has no pod-name label - that is what the
# controller above is for - but it is the honest count.
kubectl get --raw "/api/v1/nodes/node-01/proxy/metrics" \
  | grep '^volume_manager_selinux_volume_context_mismatch_warnings_total'
# volume_manager_selinux_volume_context_mismatch_warnings_total{...} 2

# After the upgrade the same measurement lives under a different name, and by
# then the pods are already stuck rather than merely at risk:
#   volume_manager_selinux_volume_context_mismatch_errors_total

# The whole point of doing this on v1.36 is that the warnings metric is
# non-zero while everything still works. Read it while it is still cheap.

对指标点到名的每一个负载，你有两个选择：把共享方式改掉，或者让这个 Pod 退出这项机制。退出开关是一个 Pod 字段 spec.securityContext.seLinuxChangePolicy，它自 v1.36 起就是稳定 API——也就是说你现在就可以在当前版本上把它应用上去，行为不会有任何变化，而升级落地的那一刻它会做正确的事。这是整个版本里最便宜的一份保险。把它设为 Recursive，该 Pod 保持旧行为，代价是放弃那份性能收益；而对于一个本来就在跨标签共享卷的负载来说，这笔交易你原本就一直在做。^{[seccontext][mutadm]}

# For every workload the metrics named, you have two options: fix the sharing,
# or opt that pod out of the mount-option path. The opt-out is a Pod field and
# it is stable API as of Kubernetes v1.36.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: legacy-shared-cache
spec:
  template:
    spec:
      securityContext:
        # Three values:
        #   unset       - follow the cluster default. On v1.37 with SELinuxMount
        #                 enabled, that means mount options.
        #   MountOption - use mount options explicitly. Only valid while the
        #                 SELinuxMount feature gate is on.
        #   Recursive   - the pre-1.37 behaviour: the runtime relabels every
        #                 file. Slower on large volumes, and it is what lets two
        #                 differently labelled pods share one volume.
        seLinuxChangePolicy: Recursive
      containers:
        - name: app
          image: registry.internal.example.com/app:1.4.2
---
# Applying this to every affected workload by hand does not scale, and both
# the SELinux blog and the KEP say so. Prefer a policy. In-tree:

apiVersion: admissionregistration.k8s.io/v1beta1
kind: MutatingAdmissionPolicy
metadata:
  name: selinux-recursive-optout
spec:
  matchConstraints:
    resourceRules:
      - apiGroups:   [""]
        apiVersions: ["v1"]
        operations:  ["CREATE"]
        resources:   ["pods"]
  # Scope this to the namespaces the metrics actually named. A cluster-wide
  # opt-out works, but it also throws away the performance win for every
  # workload that was never at risk.
  matchConditions:
    - name: only-flagged-namespaces
      expression: "request.namespace in ['default', 'ci']"
  failurePolicy: Fail
  reinvocationPolicy: IfNeeded
  mutations:
    - patchType: ApplyConfiguration
      applyConfiguration:
        expression: >
          Object{ spec: Object.spec{
            securityContext: Object.spec.securityContext{
              seLinuxChangePolicy: "Recursive" } } }

靠人手给每一个受影响的 Deployment 和 StatefulSet 加上这个字段，超出小集群规模就撑不住了，SELinux 那篇博客也直说了这一点，并点名 MutatingAdmissionPolicy、变更准入 Webhook、Kyverno 和 Gatekeeper 作为批量落地的手段。关于范围有一条建议：不要图省事、把 Recursive 作为一刀切的预防措施铺到整个集群。这样做确实有效，同时也把性能改进从每一个本来就没有风险的负载身上扔掉了——而在大多数集群里，那几乎是全部负载。把范围收在指标真正点到名的那几个命名空间上。^[kyverno]

静态 Pod 和它们的 Secret 引用：这件事发生在 1.34

这是最容易被从 1.33 一步升上来的人算到 1.37 头上的一种故障。静态 Pod 由 kubelet 从磁盘上的一个目录管理，而不是通过 API 服务器创建，所以它们本来就完全不该能读取 API 对象；一个缺陷让它们可以通过 configMapRef、secretRef 这类字段引用 Secret 和 ConfigMap。堵上这个缺陷的是 PreventStaticPodAPIReferences 特性门控，而它自 v1.34 起就默认开启——所以在任何真正逐个走过 1.34、1.35、1.36 的集群上，这件事早就发生过了，受影响的 Pod 也早就坏过一次。v1.37 的预告宣布这个门控本身被移除，退出开关随之消失；但随 v1.37 一起发布的特性门控参考文档，仍然把 PreventStaticPodAPIReferences 列为默认为 true 的 beta 门控，所以这条逃生口在技术上也许还在。规划时请当它不在。它本来就是缺陷而不是特性，不会再回来，而把静态 Pod 目录 grep 一遍，成本远低于一个节点一个节点地去发现。^{[staticpod][iss140226]}

# Static pods are managed by the kubelet from a directory on disk, not by the
# API server. They were never supposed to be able to read API objects; a bug
# let them, through envFrom.configMapRef, envFrom.secretRef, valueFrom and
# volume references. The gate that closes it - PreventStaticPodAPIReferences -
# has defaulted to true since v1.34, so this is only "new in 1.37" for a
# cluster that skipped releases. The v1.37 sneak peek says the gate was
# removed in this release; the shipped v1.37 feature-gates reference still
# lists it as Beta/true. Plan as if the opt-out is gone: it closed a defect,
# and it is not coming back. Find the references before the upgrade.

# Where the manifests live. Do not assume /etc/kubernetes/manifests: read it
# from the kubelet's own configuration.
sudo grep -E '^staticPodPath:' /var/lib/kubelet/config.yaml
# staticPodPath: /etc/kubernetes/manifests

# The audit, per node. Any hit is a pod that will fail to start on v1.37.
sudo grep -rnE 'configMapRef|secretRef|configMapKeyRef|secretKeyRef|(configMap|secret):' \
  /etc/kubernetes/manifests/
# /etc/kubernetes/manifests/node-exporter.yaml:24:            secretRef:
# /etc/kubernetes/manifests/node-exporter.yaml:25:              name: scrape-creds

# Fleet-wide, without logging into every box. Note that the control plane's
# own static pods (kube-apiserver, etcd, kube-scheduler,
# kube-controller-manager) are generated by kubeadm and do not use these
# references, so a clean result there is expected rather than reassuring.
for n in $(kubectl get nodes -o name); do
  printf '%-22s ' "${n#node/}"
  kubectl debug "$n" -q -it --image=busybox --profile=general -- \
    chroot /host sh -c \
    'grep -rlE "configMapRef|secretRef|configMapKeyRef|secretKeyRef" \
       /etc/kubernetes/manifests/ 2>/dev/null | tr "\n" " " || true' 2>/dev/null
  echo
done

# The fix is to stop being a static pod, or stop needing the reference:
#   * A DaemonSet is the right answer for almost everything that is a static
#     pod today for historical reasons. It can read Secrets normally.
#   * If it has to stay static, put the value in the manifest, or bind-mount a
#     file from the host and read it from there. Both are worse than a
#     DaemonSet, and both work.

修法几乎总是：别再当静态 Pod 了。大量静态 Pod 的存在都出于历史原因——在 DaemonSet 还没有今天这么能干的年代，它是运行节点级代理的办法——而 DaemonSet 可以正常读取 Secret、有滚动更新，也会出现在人们习惯去看的地方。如果某个东西确实必须保持静态，那么选项是把值直接内联写进清单，或者从宿主机 bind-mount 一个文件再从那里读。两种都比 DaemonSet 差，两种都能用。另外单独提一句：kubectl run --filename/-f 在这个版本里也被弃用了，理由是它生成的 Pod 一直都是纯粹由命令行参数拼出来的——这一项影响脚本，不影响集群。^[iss138671]

kube-proxy ipvs：自 1.35 起就是弃用，不是移除

接下来是被过度报道得最厉害的那项变更。kube-proxy 的 ipvs 模式自 v1.35 起就带着弃用告示，不是从 v1.37 才开始。v1.37 新增的是 KubeProxyIPVS 特性门控，而这个门控本身也被标注为弃用；除此之外就是 kube-proxy 启动时记的那条警告。没有任何东西停止工作，没有特性门控翻面，也没有任何流量受影响。弃用背后的理由值得理解，因为它解释了为什么这件事从来就没被修好：内核的 ipvs API 无法表达 Kubernetes Service 需要的全部语义，所以 ipvs 模式一直在底层为其中一部分工作回落到 iptables。KEP-3866 在一个小节标题里说得很直白——kube-proxy 的 ipvs 模式救不了我们。^{[kep5495][kep3866]}

去处是 nftables 模式，它自 v1.33 起就是 GA，也是 Linux 节点上的推荐模式。集群不会自己迁移；你得显式设置 mode: "nftables"。内核要求是真实存在的，但并不苛刻——所有老到不支持 nftables 模式的内核都会在 2026 年底之前退出 LTS——而且 nftables 的缺陷修复被专门回合到了 1.33 和 1.34 分支，就是为了让旧版本上的 ipvs 用户可以先迁移、而不必先升级 Kubernetes。^[nftblog]

# What actually happens on v1.37 if you run ipvs mode: kube-proxy logs the
# same deprecation warning it has logged since v1.35. That is all. Nothing
# stops working, no feature gate flips, and no traffic is affected. What v1.37
# adds is the KubeProxyIPVS gate, itself marked deprecated - the switch that
# will later be used to turn the mode off by default.
kubectl -n kube-system logs -l k8s-app=kube-proxy --tail=50 | grep -i deprecat
# W0826 ... "ipvs mode of kube-proxy is deprecated and will be removed in a
#            future release; see KEP-5495"

# The reason is worth knowing, because it explains why there is no fixing it:
# the kernel's ipvs API cannot express everything a Kubernetes Service needs,
# so ipvs mode has always fallen back to iptables underneath for parts of the
# job. It was never the clean escape from iptables it was sold as.

# The destination is nftables mode, GA since v1.33. Clusters never migrate on
# their own - you have to set it.

# --- migrating, on a kubeadm cluster ---------------------------------------
# 1. Check the kernel. nftables mode wants a reasonably modern kernel; every
#    kernel too old for it leaves LTS by the end of 2026.
kubectl get nodes -o jsonpath='{range .items[*]}{.status.nodeInfo.kernelVersion}{"\n"}{end}' \
  | sort -u

# 2. Change the mode in the ConfigMap.
kubectl -n kube-system get configmap kube-proxy -o yaml > /tmp/kube-proxy.bak.yaml
kubectl -n kube-system patch configmap kube-proxy --type merge -p \
  "$(printf '{"data":{"config.conf":%s}}' \
     "$(kubectl -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' \
        | sed 's/^\(\s*mode:\).*/\1 "nftables"/' | jq -Rs .)")"

# 3. Roll the DaemonSet one node at a time and watch, rather than all at once.
kubectl -n kube-system rollout restart daemonset/kube-proxy
kubectl -n kube-system rollout status daemonset/kube-proxy --timeout=10m

# 4. Verify from the data plane, not the control plane. A Service that
#    resolves but does not connect is the failure mode here.
kubectl run nft-check --rm -it --restart=Never --image=busybox -- \
  sh -c 'wget -qO- --timeout=5 http://kubernetes.default.svc/healthz || echo FAILED'

# Rolling back is the same edit in reverse; keep /tmp/kube-proxy.bak.yaml.
# Do this as its own change, on its own day. Bundling a proxy-mode migration
# into a version upgrade means that when connectivity breaks you will not know
# which one did it.

关于顺序有一条建议，说得带点情绪：把代理模式的迁移放进它自己的维护窗口、自己的那一天，不要和版本升级捆在一起。这两项变更都动到数据面，当连通性在一个同时做了两件事的窗口里出问题时，你会把整个故障时间花在判断到底是哪一个干的，而不是花在修复上。这里也没有任何时间压力值得你把它们合并——弃用策略保证一个 beta 及以上的特性有很长的余量，而 KEP 自己的毕业标准把特性门控默认为 false 排在 v1.40、把移除排在 v1.43。^{[kubeproxycfg][deprecpolicy]}

cgroup v1：这件事发生在 1.35

cgroup v1 这件事是最常被张冠李戴的，而把它归对位置会改变你该采取的动作。kubelet 的 failCgroupV1 设置自 Kubernetes v1.35 起就默认为 true。因此，一个仍在 cgroup v1 上的节点，从那时候起 kubelet 就一直起不来，除非有人加上了 failCgroupV1: false——而在 v1.35 升级期间，不少人手速很快地加了，然后再也没回头看过。所以走向 v1.37 时，有用的问题不是“这会不会坏”，而是“谁还背着这个覆盖设置，还能背多久”。^{[kep5573][kubeletcfg]}

# This is the change most often misattributed to v1.37. The kubelet setting
# `failCgroupV1` has defaulted to TRUE since Kubernetes v1.35. A node still on
# cgroup v1 has therefore been refusing to start its kubelet since v1.35,
# unless somebody added the override - which many people did, in a hurry, and
# then forgot.

# So the useful question on the way to v1.37 is not "will this break" but
# "who is still carrying the override?"
for n in $(kubectl get nodes -o name); do
  printf '%-22s ' "${n#node/}"
  kubectl debug "$n" -q -it --image=busybox --profile=general -- \
    chroot /host sh -c \
    'printf "cgroup=%s override=%s\n" \
       "$(stat -fc %T /sys/fs/cgroup)" \
       "$(grep -c failCgroupV1 /var/lib/kubelet/config.yaml 2>/dev/null)"' 2>/dev/null
done
# node-01  cgroup=cgroup2fs override=0     <- fine
# node-02  cgroup=tmpfs     override=1     <- v1, running on borrowed time

# The override itself, for reference. It is a stopgap and upstream says so:
#   apiVersion: kubelet.config.k8s.io/v1beta1
#   kind: KubeletConfiguration
#   failCgroupV1: false

# v1.37 still honours it. KEP-5573 removes cgroup v1 support outright in a
# later release, and no date has been committed to, so the honest planning
# assumption is "the next one that suits SIG Node" rather than a fixed month.

# What you lose in the meantime is not theoretical. In-place pod resizing and
# tiered memory protection - both features people are actively asking for -
# depend on cgroup v2 and simply do not work on a v1 node.

# Switching the node is a kernel command line change and a reboot:
#   systemd.unified_cgroup_hierarchy=1
# ...and then the container runtime's cgroup driver has to agree with the
# kubelet's. That is a longer job than it looks, which is why it has its own
# article.

v1.37 仍然认这个覆盖设置，而 KEP-5573 会在之后某个版本里彻底移除 cgroup v1 支持，且没有承诺日期——所以老实的规划假设是“SIG Node 觉得合适的那个版本”，而不是一个你能写进计划表的月份。在此期间你放弃的东西并不抽象：原地调整 Pod 资源和分级内存保护都依赖 cgroup v2，在 v1 节点上根本不工作，而这两项都是团队正在主动要的功能。切换本身是改一行内核命令行加一次重启，再让容器运行时的 cgroup driver 和 kubelet 的保持一致——这件事比听上去费劲，也有它自己的一篇文章。^[cgroups]

containerd 1.x：悬崖在 1.38，不是现在

这一项是朝另一个方向被说错的，而搞错它的代价，是白白搭进去一个本不必花的维护窗口。containerd 1.x 的支持并没有在 v1.36 被移除，在 v1.37 里也没有。v1.37 的 kubelet 仍然能跑在它上面：启用 KubeletCgroupDriverFromCRI 之后，kubelet 会通过 RuntimeConfig 这个 CRI RPC 向运行时询问 cgroup driver，而 containerd 1.y 并不实现这个 RPC，于是 kubelet 悄悄回落到自己的 --cgroup-driver 取值，同时把 cri_losing_support 指标加一。这条回落路径原本安排在 v1.37 消失，后来被推迟了一个版本，明确是为了和 containerd v1.7 自己的支持窗口对齐。Kubernetes 的运行时文档现在把话说得很白：在 v1.38，这条回落被去掉，更旧的 containerd 版本在更新的 kubelet 下就会失败。^{[k8s134][ctrrel]}

# The change most often stated backwards. containerd 1.x was NOT removed in
# v1.36, and it still runs against a v1.37 kubelet. What it runs on is a
# fallback: the kubelet asks the runtime for its cgroup driver over the
# RuntimeConfig CRI RPC, containerd 1.y does not implement that RPC, and the
# kubelet falls back to its own --cgroup-driver value. That fallback was
# scheduled to go in v1.37 and was deferred one release to align with
# containerd v1.7's support window, so it disappears in v1.38.

# The audit is a metric, not a spreadsheet. Every node relying on the fallback
# increments this, so scrape it rather than walking nodes by hand.
kubectl get --raw /metrics | grep '^cri_losing_support'
# cri_losing_support{version="1.38.0"} 1

# Then confirm which nodes, and with what:
kubectl get nodes -o custom-columns=\
'NODE:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion,'\
'RUNTIME:.status.nodeInfo.containerRuntimeVersion' | sort -k3
# NODE     KUBELET   RUNTIME
# node-02  v1.37.0   containerd://1.7.28   <- works today, fails on 1.38
# node-01  v1.37.0   containerd://2.3.2

crictl version
# RuntimeName:        containerd
# RuntimeVersion:     v2.3.2
# RuntimeApiVersion:  v1

# Two things make this less comfortable than "one release of runway" sounds:
# containerd 1.7's own extended support ends in September 2026, and
# containerd's published Kubernetes support matrix has no row for 1.37 yet -
# the last row is 1.36 (2.3.0+, 2.2.0+). Book the migration before 1.38, in
# its own window. Landing two runtime-level changes together means that when a
# node comes back wrong you will be bisecting instead of fixing.

所以这个期限是真的，而且刚好还有一个版本——比那种恐慌所暗示的处境要好，也比什么都不做所暗示的要糟。有两件事让它更紧。运行时这一侧的时钟先走完：这件事系于 containerd 1.7 这条 LTS 分支，而它的扩展支持在 2026 年 9 月结束，也就是现在。另外，containerd 自己的 Kubernetes 支持矩阵目前根本没有 Kubernetes 1.37 这一行——最后公布的一行是 1.36，列的是 2.3.0+ 和 2.2.0+——所以任何人给你报一个“containerd 官方认可的 1.37 搭配版本”，那都是外推，而不是引用。落到实处：去抓 cri_losing_support，凡是需要这条回落的节点都已经在上报它了，这比人工遍历节点是更好的审计手段；然后再用 containerRuntimeVersion 确认。在 1.38 之前、并且在自己的窗口里，把运行时迁移做完。把两项运行时层面的变更放在一起，意味着当某个节点回来时状态不对，你要做的是二分定位，而不是修复。^[runtimes]

metrics.k8s.io 终于走到 v1

这个版本里的好消息，而且是真的好。metrics.k8s.io 在 beta 里待了将近九年之后毕业到 v1。这是 kubectl top 背后的 API，也是 HorizontalPodAutoscaler 的 CPU 和内存指标背后的 API，这让它成为 Kubernetes 中使用最广泛的接口之一，也让它在 beta 里待这么久显得很奇怪。这次毕业是对既有稳定性的承认，而不是引入变化：预期没有功能差异，过渡期内 v1 和 v1beta1 都会继续提供服务。^{[kep5207][metricspipe]}

# The good news in this release. metrics.k8s.io graduates to v1 after nearly
# nine years in beta. Both versions stay served during the transition, so
# there is nothing to do on upgrade day - this is a thing you can adopt on
# your own schedule rather than a thing that happens to you.

kubectl get --raw /apis/metrics.k8s.io | jq -r '.versions[].groupVersion'
# metrics.k8s.io/v1
# metrics.k8s.io/v1beta1

# What consumes it: `kubectl top`, and the HorizontalPodAutoscaler's cpu and
# memory metrics. Both keep working without changes.
kubectl top nodes
kubectl top pods -A --sort-by=memory | head

# Where it matters is code you own. Anything that talks to the API directly -
# a custom autoscaler, a capacity report, a dashboard backend - should move
# off v1beta1 while both are available rather than after one is not.
kubectl get --raw /apis/metrics.k8s.io/v1/nodes | jq '.items[0]'

# Find the clients still on the beta path, from the API server's own counters:
kubectl get --raw /metrics \
  | grep 'apiserver_requested_deprecated_apis\|metrics.k8s.io.*v1beta1' | head

# There is no removal date for v1beta1 yet. Kubernetes' deprecation policy
# guarantees a beta API at least nine months or three releases after
# deprecation, so this is a housekeeping item, not a deadline.

所以升级当天没有事要做，而这正是重点——这是一件你按自己的节奏去采纳的事，而不是一件发生在你身上的事。它真正要紧的地方在你自己的代码：自研的弹性伸缩器、容量报表、看板后端，任何直接和这个 API 打交道的东西，都该趁两个版本都还在提供服务时从 beta 路径挪走，而不是等到其中一个不在了再挪。v1beta1 目前还没有移除日期，而弃用策略保证一个 beta API 至少还有九个月或三个版本，所以把它当成日常整理，而不是一个截止期限。^[hpa]

这个版本的其余部分，简短地说

这个版本里还有三件事值得知道，尽管它们都不会影响升级。三项都是毕业而不是移除，其中第一项最值得留意。^[kep4960]

• kubelet 运行在用户命名空间中——即 rootless 模式——进入 beta。节点组件历来以宿主机上的 root 身份运行。这项能力让它们以宿主机上的非特权用户身份运行，同时在 Linux 用户命名空间内部仍然表现为 root，从而收窄了节点组件漏洞的影响范围。这里的 beta 意味着门控默认打开，但仅仅启用它本身并不会让 kubelet 跑进用户命名空间——背后还有一整套宿主机配置——而且这项变更并没有进入发布公告。请把它当成一件值得在测试节点上试试的事，而不是一件已经发生在你身上的事。它和面向 Pod 的用户命名空间也不是同一个特性，后者在 v1.35 进入 beta，在 v1.36 转正为 GA。
• 面向 ReadWriteOncePod 卷的 SELinux 重新标记在 v1.36 就已经是 GA。那是 SELinuxMountReadWriteOncePod，一个比本文讨论的 SELinuxMount 更窄的门控，这也是为什么有些集群已经用上下文选项挂载了一段时间却什么都没坏——RWOP 卷按定义就不能被共享，所以本文描述的这种冲突在那里不可能出现。
• 卷健康监测从 alpha 重新开始。最初的实现落在 v1.21，此后一直没有毕业；KEP-1432 在 CSIVolumeHealth 门控之后把它重置，并引入四个 CSI RPC——ControllerListVolumeHealth、ControllerGetVolumeHealth、NodeGetVolumeHealth 和 NodeGetStorageHealth——分别上报到 PersistentVolumeClaim.status.healthStatus、Pod.status.volumeHealth 和 CSINode.status.storageHealth。取值词汇被刻意做得很小且便于机器读取：Inaccessible、DataLoss 和 Degraded，再配上 condition 上的 reason 和 message 承载驱动特有的细节。

alpha 意味着默认关闭、不用于生产，但如果你曾经为了搞清楚哪里出了问题，而不得不把一个挂死的挂载点和存储厂商的看板来回对照，那这项健康监测工作值得跟进。这是 Kubernetes 第一次为那个问题给出机器可读的答案。^{[kep1432][userns]}

升级，按顺序来

顺序比命令更重要。这次升级里真正难的部分，都难在升级之前；而唯一在升级之后会明显变贵的步骤，就是 SELinux 审计——所以它排在最前面，提前的量级是几周，不是几分钟。^[kubeadmup]

# The order matters more than the commands, and the SELinux audit has to come
# first because it is the only step that is materially harder after the
# upgrade than before it.

# --- WEEKS BEFORE, on v1.36 ------------------------------------------------
# 1. Turn on selinux-warning-controller, read both metrics, apply the opt-outs.
# 2. Audit static pods for Secret and ConfigMap references. Move them to
#    DaemonSets where you can.
# 3. Get every node onto containerd 2.x and cgroup v2, if any are not.
#    Both of these are already overdue rather than upcoming.
# 4. Decide about ipvs - and then do it in a DIFFERENT window.

# --- THE DAY ---------------------------------------------------------------
# Control plane first, one node at a time. Nothing here is 1.37-specific;
# it is the standard kubeadm sequence and it is standard because it works.
sudo apt-mark unhold kubeadm && sudo apt-get update \
  && sudo apt-get install -y kubeadm='1.37.0-*' && sudo apt-mark hold kubeadm

sudo kubeadm upgrade plan
sudo kubeadm upgrade apply v1.37.0        # first control plane node
# sudo kubeadm upgrade node               # every other control plane node

# Then the kubelet and kubectl on that same node:
sudo apt-mark unhold kubelet kubectl && sudo apt-get update \
  && sudo apt-get install -y kubelet='1.37.0-*' kubectl='1.37.0-*' \
  && sudo apt-mark hold kubelet kubectl
sudo systemctl daemon-reload && sudo systemctl restart kubelet

# --- WORKER NODES, ONE AT A TIME -------------------------------------------
NODE=node-02
kubectl drain "$NODE" --ignore-daemonsets --delete-emptydir-data --timeout=15m
# ... upgrade kubeadm, run `kubeadm upgrade node`, upgrade kubelet, restart ...
kubectl uncordon "$NODE"

# Then STOP and look, before the next node. The failure this release can
# produce is a pod stuck in ContainerCreating, and it is per-node:
kubectl get pods -A --field-selector spec.nodeName="$NODE" \
  -o wide | grep -v Running | grep -v Completed

# The version skew policy is what makes the staged rollout legal: a kubelet
# may be up to three minor versions behind the API server, so a fleet halfway
# through this is a supported configuration rather than a risk in itself.

关于分批推进有一句可以让人安心的话：版本偏差策略允许 kubelet 落后 API 服务器最多三个小版本，所以一个推到一半的机器群是受支持的配置，本身并不构成风险。慢慢来。升一个节点，认真看一看，然后再继续——因为这个版本可能造成的故障是按节点发生的，表现为一个永远起不来的 Pod，而不是某个会有人呼你的报错。^[skew]

怎么回滚，以及什么回不去

回滚这件事值得一个直白的答案，而不是一个安慰性的答案，而在这次升级上，这个答案有三个部分。可逆的那些变更是真的可逆：kube-proxy 模式就是一个 ConfigMap 加一次 DaemonSet 重启，而 seLinuxChangePolicy 是一个在 v1.36 和 v1.37 上行为完全相同的 Pod 字段——这也正是为什么提前把它加上去不花任何代价，却把整套回滚故事都买了下来。^[kubeadmup]

# Rollback deserves a straight answer. Most of this release rolls back; one
# part of it does not roll back in the way people assume.

# --- What rolls back cleanly ----------------------------------------------
# The kube-proxy mode change: it is a ConfigMap and a DaemonSet restart.
kubectl -n kube-system apply -f /tmp/kube-proxy.bak.yaml
kubectl -n kube-system rollout restart daemonset/kube-proxy

# The seLinuxChangePolicy opt-out: it is a Pod field. Setting it to Recursive
# is safe on v1.36 and v1.37 alike, which is why applying it BEFORE the
# upgrade costs nothing and buys the whole rollback story.

# --- What does not ---------------------------------------------------------
# The control plane. `kubeadm upgrade` has no downgrade path: going back means
# restoring the etcd snapshot you took before you started, which means losing
# everything written to the cluster since. If you did not take one, you do not
# have a rollback - you have a forward fix.
sudo ETCDCTL_API=3 etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save "/var/backups/etcd-pre-1.37-$(date +%F).db"

# The static pod references. If a static pod was relying on a secretRef, the
# reference is gone on 1.37 and comes back on a downgrade - but the downgrade
# is the control plane operation above, so in practice the fix is forward:
# move it to a DaemonSet.

# --- The one that surprises people ----------------------------------------
# Downgrading the kubelet does NOT un-stick a pod that failed to mount because
# of an SELinux label conflict, because the pod that WON the volume is still
# holding it with its own context. Terminate one of the two, or set
# seLinuxChangePolicy: Recursive on both and let them share again. Version
# numbers are not the lever here; the Pod field is.

控制面才是那个回不去的部分。kubeadm upgrade 没有降级路径；往回走意味着恢复你开始之前拍的那份 etcd 快照，并丢掉此后写进集群的一切。如果你没拍，那你就没有回滚——你有的是向前修复，而那是凌晨两点该有的另一种对话。还有一种失败会让人措手不及：把 kubelet 降级并不会让一个因为 SELinux 标签冲突而挂载失败的 Pod 脱困，因为抢到那个卷的 Pod 仍然用它自己的上下文占着它。这里的杠杆是那个 Pod 字段，不是版本号。^[selblog]

去验证，而不是去指望

这次升级的验证有一个特定的形状，因为它的典型故障会让所有指示灯都是绿的。节点是 Ready。kubelet 健康。控制面没问题。只不过有一个 Pod 永远创建不完，在一个节点上，属于某一个团队。所以“检查集群是不是起来了”什么也证明不了——你必须去查那些可能悄无声息就是错的东西。^[chlog]

#!/usr/bin/env bash
# Post-upgrade verification. Exits non-zero when something is wrong, so it can
# run between nodes in a pipeline rather than being read by a person at 3am.
set -uo pipefail
rc=0
fail() { printf '  FAIL  %s\n' "$*"; rc=1; }
pass() { printf '  ok    %s\n' "$*"; }

echo '== versions =='
kubectl version -o json | jq -r '.serverVersion.gitVersion'

echo '== every node Ready and on the expected version =='
bad=$(kubectl get nodes --no-headers | awk '$2!="Ready"{print $1}')
[ -z "$bad" ] && pass 'all nodes Ready' || fail "not Ready: $bad"

echo '== no pod stuck creating (the SELinux failure mode) =='
stuck=$(kubectl get pods -A --no-headers \
  | awk '$4=="ContainerCreating"{print $1"/"$2}')
[ -z "$stuck" ] && pass 'nothing in ContainerCreating' || fail "stuck: $stuck"

echo '== SELinux context mismatches that became real failures =='
for n in $(kubectl get nodes -o name); do
  v=$(kubectl get --raw "/api/v1/nodes/${n#node/}/proxy/metrics" 2>/dev/null \
      | awk '/^volume_manager_selinux_volume_context_mismatch_errors_total/{s+=$2} END{print s+0}')
  [ "$v" = "0" ] && pass "${n#node/}: 0 mismatch errors" \
                 || fail "${n#node/}: $v SELinux mismatch errors"
done

echo '== static pods all running =='
sp=$(kubectl get pods -A -o json \
  | jq -r '.items[] | select(.metadata.annotations["kubernetes.io/config.source"]=="file")
           | select(.status.phase!="Running") | .metadata.namespace+"/"+.metadata.name')
[ -z "$sp" ] && pass 'static pods Running' || fail "static pods not Running: $sp"

echo '== the resource metrics API answers on both versions =='
kubectl get --raw /apis/metrics.k8s.io | jq -e \
  '[.versions[].groupVersion] | index("metrics.k8s.io/v1")' >/dev/null \
  && pass 'metrics.k8s.io/v1 served' || fail 'metrics.k8s.io/v1 missing'
kubectl top nodes >/dev/null 2>&1 && pass 'kubectl top works' || fail 'kubectl top broken'

echo '== service traffic actually flows =='
kubectl run verify-net --rm -i --restart=Never --image=busybox --timeout=60s -- \
  sh -c 'wget -qO- --timeout=5 http://kubernetes.default.svc/healthz' >/dev/null 2>&1 \
  && pass 'in-cluster Service reachable' || fail 'in-cluster Service unreachable'

exit $rc

在节点之间跑它，而不是等到最后再跑。这个脚本出错时返回非零退出码，也就是说它可以待在流水线的一个步骤里，而不是等着某个人在凌晨三点去读它；而在升级已经过去之后仍然值得保留的两项检查，是 ContainerCreating 扫描和 SELinux 不匹配计数器。两项都很便宜，而且都能抓住那一类问题——否则它会一直等到有人发现自己的负载再也没回来。^[k8spatch]

这件事该按什么顺序做

压缩之后，这次升级比本文的篇幅显得要小——前提是属于前面两个版本的那些活你都干完了。下面这张决策表，其实问的是你当初跳过了其中哪一项。^[sneak]

1. 先用一条命令回答 SELinux 那个问题。如果没有任何节点以 enforcing 模式运行 SELinux，就整节跳过这里最长的一段，把 1.37 当成一次常规升级。只要有任何一个节点在跑，下面全部适用。
2. 在 v1.36 上打开 selinux-warning-controller，并把两个指标都读一遍。控制器点出冲突的 Pod 名字；kubelet 数出真正会失败的有多少个。要提前几周做，因为升级之后这件事明显更难做，而那时候 Pod 已经卡住了，不再只是有风险。
3. 给指标点到名的负载加上 seLinuxChangePolicy: Recursive，用策略来做，不要靠人手。它在 v1.36 上就是稳定 API，今天不改变任何行为，而升级落地的那一刻会做正确的事。把范围收在涉及的命名空间上，而不是整个集群。
4. 在每个节点的静态 Pod 目录里 grep secretRef 和 configMapRef。这项默认行为自 v1.34 起就生效了，所以在逐个版本走过来的集群上它早已发生；而在跳版本的集群上，它还在前面等着。把找到的东西挪成 DaemonSet，它几乎总是本来就该是的样子。
5. 把 1.35 欠下的 cgroup v1 那笔债还掉，并把 containerd 迁移排进 1.38 之前的日历。还在 cgroup v1 上的节点，已经背着覆盖设置过了两个版本。还在 containerd 1.x 上的节点今天照常能用，但再过一个版本就不能用了。两件事各自占一个窗口——并且把 ipvs 到 nftables 的迁移彻底留到另外一天。

上面提到的两笔债各有自己的操作手册，因为它们都不是五分钟能干完的活：containerd 1.7 迁移到 2.x，那是从运行时这一侧看 containerd 1.x 的移除；以及 从 cgroup v1 迁移到 cgroup v2，那是 failCgroupV1 覆盖设置背后的审计与转换。如果同一个维护窗口还要顺手重建你的入口层，从 ingress-nginx 迁移到 Gateway API 讲的就是那次迁移。而如果到了这一步，屋子里有人开口问这一整套到底值不值，什么时候不该用 Kubernetes 是这个论证的另一面，也说得同样老实。

常见问题

Kubernetes 1.37 什么时候发布，到底什么会坏？

v1.37 已于 2026 年 8 月 26 日发布。其中只有一项变更能让负载停摆：SELinuxMount 转正为 GA 并默认启用，在 SELinux 处于 enforcing 的节点上，如果两个标签不同的 Pod 共享同一个卷、且 CSI 驱动已声明启用，就可能让 Pod 停在 ContainerCreating。其余真正属于这个版本的新变化都很安静——新增 KubeProxyIPVS 特性门控并随即标注为弃用；metrics.k8s.io 毕业到 v1，两个版本都继续提供服务；以及 kubectl run --filename/-f 被弃用，这一项影响脚本而不是集群。其余被算到这个版本头上的东西，要么早就落地了，要么还没落地：cgroup v1 上的 kubelet 启动失败属于 v1.35，静态 Pod 的 Secret 限制属于 v1.34，ipvs 模式自 v1.35 起弃用、到 v1.43 才移除，而 containerd 1.x 目前仍然能用——它的 CRI 回落要到 v1.38 才被去掉。

Kubernetes 1.37 移除了 kube-proxy 的 ipvs 模式吗？

没有，而且它连弃用都不是在这个版本做的——弃用发生在 v1.35，从那时起 kube-proxy 每次启动都在记那条警告。v1.37 新增的是 KubeProxyIPVS 特性门控，它本身也被标注为弃用，而它就是之后用来把这个模式关掉的那个开关。没有任何东西停止工作，也没有流量受影响。KEP-5495 给出了时间线：该门控预计在 v1.40 默认为 false，支持在 v1.43 彻底移除。弃用的理由是内核的 ipvs API 无法表达 Kubernetes Service 需要的全部语义，所以 ipvs 模式一直在底层为其中一部分工作回落到 iptables。迁移目标是 nftables 模式，自 v1.33 起为 GA——而集群永远不会自己切换，你得自己设置 mode: "nftables"。请把它放在和版本升级不同的维护窗口里做。

升级到 1.37 之后，我的 Pod 为什么卡在 ContainerCreating？

如果该节点以 enforcing 模式运行 SELinux，那么最可能的原因是 SELinuxMount 转正为 GA 引入的卷共享冲突。卷现在用 -o context=<label> 挂载，而不再被递归重新标记，而一个挂载点只能持有一个 SELinux 上下文——所以同一个节点上、两个标签不同的 Pod 共享一个卷这件事已经不可能了。其中一个会一直停在 ContainerCreating，直到另一个终止。典型场景是一个特权 Pod 和一个非特权 Pod 共享一个卷。到该节点上读 kubelet 指标 volume_manager_selinux_volume_context_mismatch_errors_total 来确认。修法是给受影响的 Pod 设置 spec.securityContext.seLinuxChangePolicy: Recursive——注意把 kubelet 降级不会释放那个卷，因为抢到它的那个 Pod 仍然占着上下文。

升级之前怎么找出 SELinux 卷冲突？

启用 v1.36 就已经发布的 selinux-warning-controller：给 kube-controller-manager 传 --controllers=*,selinux-warning-controller，并确认你没有显式禁用 SELinuxChangePolicy 特性门控。然后读两个指标。selinux_warning_controller_selinux_volume_conflict 把冲突双方的 Pod 名称和命名空间作为标签带出来——那就是你的待办清单，而且即使两个 Pod 当前不在同一节点它也照样报告，因为调度器之后可能把它们放到一起。volume_manager_selinux_volume_context_mismatch_warnings_total 由 kubelet 在 SELinuxMount 仍然关闭时发出，没有 Pod 名称标签，但给出的是“真正会失败的 Pod 有多少个”的诚实计数。两个你都需要。要在 v1.36 上做这件事，那时候警告计数不为零，而一切仍然正常工作。

seLinuxChangePolicy: Recursive 到底做什么，现在设置它安全吗？

它让该 Pod 退出上下文挂载这条路径，保持 1.37 之前的行为：由容器运行时遍历这个卷、给每个文件重新打标签。代价是那份性能收益——在大卷或远端卷上，递归重新标记确实很慢——收益是两个标签不同的 Pod 可以重新共享这个卷。它自 v1.36 起就是稳定的 Pod API，所以今天设置它不会改变你集群当下的任何行为，而升级落地的那一刻它会做正确的事。这让它成为这个版本里最便宜的一份保险。请通过 MutatingAdmissionPolicy 或策略引擎来落地，而不是去逐个改 Deployment；并且把范围收在冲突指标真正点到名的命名空间上，而不是整个集群——一刀切的 Recursive 会把改进从每一个本来就没有风险的负载身上扔掉。

我的集群不用 SELinux，这些还和我有关吗？

基本没关系。当 SELinux 不可用或在内核中被禁用时，kubelet 会跳过整条 SELinux 代码路径，所以 SELinuxMount 这项变更对你是空操作，本文最长的那一节可以直接忽略。仍然适用的是静态 Pod 那条限制——把每个节点的静态 Pod 目录 grep 一遍 secretRef 和 configMapRef，它自 v1.34 起就是默认行为，因此只会咬到跳过版本的集群——另外还有 v1.35 欠下的 cgroup v2 那笔债（如果你还没还），以及 containerd 2.x 的迁移，它在 1.37 上并不紧急，但必须在 1.38 之前做完。不过请在每个节点上实际执行 getenforce，而不是想当然：某个节点池用的镜像启用了 SELinux、整个机器群因此参差不齐，这比人们以为的要常见。

1.37 会让 kubelet 在 cgroup v1 上起不来吗？

这项变更不是 1.37 的。kubelet 的 failCgroupV1 设置自 v1.35 起就默认为 true，所以一个在 cgroup v1 上的节点，从那时候起 kubelet 就一直起不来，除非有人加了 failCgroupV1: false。v1.37 仍然认这个覆盖设置。KEP-5573 会在之后某个版本里彻底移除 cgroup v1 支持，但没有承诺日期，所以规划假设应该是“SIG Node 觉得合适的那个版本”，而不是某个月份。不该继续背着这个覆盖设置的理由是：原地调整 Pod 资源和分级内存保护都需要 cgroup v2，没有它就根本不工作。切换一个节点意味着改一行内核命令行（systemd.unified_cgroup_hierarchy=1）、重启一次，并让运行时的 cgroup driver 和 kubelet 的保持一致。

Kubernetes 1.37 需要 containerd 2.0 吗？

不需要——而这正是最常被说反的一条。containerd 1.x 至今仍能跑在 v1.37 的 kubelet 下。它靠的是一条回落路径：kubelet 通过 RuntimeConfig 这个 CRI RPC 向运行时询问 cgroup driver，而 containerd 1.y 并不实现这个 RPC，于是 kubelet 回落到自己的 --cgroup-driver 取值，同时把 cri_losing_support 指标加一。这条回落原本安排在 v1.37 移除，为了和 containerd v1.7 的支持窗口对齐而推迟了一个版本，所以它会在 v1.38 消失——到那时，更旧的 containerd 版本在更新的 kubelet 下确实会失败。有两点让这件事没有听上去那么舒服：containerd 1.7 自己的扩展支持在 2026 年 9 月结束，而 containerd 公布的 Kubernetes 支持矩阵目前还没有 1.37 这一行，所以眼下没有人能给出一个官方认可的搭配。用 cri_losing_support 和 containerRuntimeVersion 做审计，并在拿 1.38 之前把迁移做完——放在它自己的窗口里，不要和版本升级捆在一起。

一个要读 Secret 的静态 Pod 该换成什么？

几乎所有情况下都是 DaemonSet。静态 Pod 由 kubelet 从磁盘上的一个目录管理，而不是通过 API 服务器创建，所以读取 API 对象这件事本来就不该成立——一个缺陷让它可以通过 configMapRef、secretRef 这类字段实现。堵上这个缺陷的 PreventStaticPodAPIReferences 门控自 v1.34 起就默认开启，所以这并不是 1.37 的新变化——它只是对跳了好几个版本的集群来说像是新的。v1.37 的预告说这个门控被整个移除了；而随 v1.37 发布的特性门控参考文档，仍然把它列为默认为 true 的 beta 门控。不管是哪种情况，都不要围着那个退出开关做规划。大多数静态 Pod 的存在都出于历史原因，来自 DaemonSet 还没有今天这么能干的年代；而 DaemonSet 可以正常读取 Secret、有滚动更新，也会出现在人们习惯去找它的地方。如果某个东西确实必须保持静态，就把值内联写进清单，或者从宿主机 bind-mount 一个文件再从那里读。两种都比 DaemonSet 差，两种都能用。请在升级前把它们找出来，而不是一个节点一个节点地撞见。

可以从 1.34 或 1.35 直接升到 1.37 吗？

可以——版本偏差策略允许 kubelet 落后 API 服务器最多三个小版本，而 kubeadm 对控制面是一个版本一个版本地处理——但风险不在机制上，在阅读量上。跳版本意味着你跳过的那些版本里的每一条弃用都会一次性到齐，而它们是按版本分别记录的，不是累积的。从 1.35 过来，你还会一并继承 v1.36 的变更：Service ExternalIPs 的弃用警告、SELinuxMountReadWriteOncePod 和 SELinuxChangePolicy 转正为 GA，以及面向 Pod 的用户命名空间进入稳定。从 1.34 过来，你还额外继承 v1.35 里 failCgroupV1 默认值的翻面——它会在 cgroup v1 节点上直接让 kubelet 起不来。请把你跨过的每一个版本的发布团队博客都读一遍。三篇大概二十分钟，而这比升级窗口里塞进去的大多数事情都更值。

1.37 的升级能回滚吗？

部分能，而且分清楚哪部分很重要。kube-proxy 模式的变更可以干净地回滚——它就是一个 ConfigMap 加一次 DaemonSet 重启。seLinuxChangePolicy 是一个在 v1.36 和 v1.37 上行为相同的 Pod 字段，这也正是提前加上它不花任何代价的原因。控制面回不去：kubeadm upgrade 没有降级路径，所以往回走意味着恢复你开始之前拍的那份 etcd 快照，并丢掉此后写进去的一切。请务必拍那份快照。还有一种失败会让人措手不及：把 kubelet 降级并不会让一个因为 SELinux 冲突而挂载失败的 Pod 脱困，因为抢到那个卷的 Pod 仍然占着这个挂载点的上下文。杠杆是两个 Pod 上的 seLinuxChangePolicy，不是版本号。

metrics.k8s.io 的 v1beta1 要没了吗？

还没有，而且不会毫无预告地没。metrics.k8s.io 在 beta 里待了将近九年之后，于 v1.37 毕业到 v1，而过渡期内 v1 和 v1beta1 都会继续提供服务，正是为了让采纳可以按你自己的节奏进行。预期没有功能变化——这次毕业是承认既有的稳定性，而不是引入稳定性。kubectl top 和 HorizontalPodAutoscaler 什么都不用做就照常工作。值得做的是把你自己的代码——自研的弹性伸缩器、容量报表、看板后端——趁两个版本都还在服务时从 beta 路径挪走。Kubernetes 的弃用策略保证一个 beta API 在弃用之后至少还有九个月或三个版本，而 v1beta1 根本还没有被给出移除日期，所以这属于日常整理，不是截止期限。

参考资料

先放一手资料。发布团队的预告和 v1.37 变更日志，是关于“这个版本里有什么”唯一有权威性的说法；KEP 则是那些跨版本时间线唯一被写下来的地方，这也正是它们能解掉“1.37 移除了某个其实要到 1.43 才移除的东西”这类摘要的原因。凡是本文做了纠正的地方——cgroup 那次失败属于 v1.35、静态 Pod 那条限制属于 v1.34、ipvs 是自 v1.35 起弃用而不是在 v1.37 被移除，以及 containerd 那道悬崖在你前面的 v1.38 而不是在你身后的 v1.36——分歧针对的是那些二手报道，而不是项目本身。有一处分歧出在项目内部，值得点明：预告说 PreventStaticPodAPIReferences 门控在这个版本被移除，而随 v1.37 发布的特性门控参考文档里它仍然在列。就“到底发布了什么”而言，该信的是参考文档那一页。

1. Kubernetes v1.37 Sneak Peek - the release team's own list of what is deprecated, removed and breaking in this release, published 31 July 2026. This is the document that separates "new in 1.37" from "still in progress", and it carries its own caveat that the information reflects the state of the release before the release date
2. Kubernetes CHANGELOG-1.37.md - the authoritative record once the release is cut on 26 August 2026. Where this article and the changelog disagree after that date, the changelog is right and this page is a snapshot of the plan
3. SELinux Volume Label Changes goes GA (and likely implications in v1.37) - the pre-announcement by the feature's own authors. It contains the five conditions for a mount-option relabel, the two conflict scenarios, the seLinuxChangePolicy opt-out, the selinux-warning-controller and the recommended upgrade path. This is the single most important source for this article
4. KEP-1710: Speed up SELinux volume relabeling using mounts. The enhancement proposal behind SELinuxMount, including why a mount can hold only one context and therefore why volume sharing across differently labelled pods stops working
5. KEP-1710, "Story 3: cluster upgrade" - the upgrade scenario written by the authors, which is the closest thing to an official runbook for this change
6. Kubernetes - Configure a Security Context for a Pod or Container: seLinuxOptions, the seLinuxChangePolicy field, efficient SELinux volume relabeling and the selinux-warning-controller
7. Kubernetes API reference - CSIDriver: the seLinuxMount field a driver has to set to true before the kubelet will mount its volumes with a context option. Drivers that do not set it keep the old recursive behaviour, which is why the blast radius of this change is driver-specific
8. Kubernetes - Feature Gates: the per-release state of SELinuxMount, SELinuxChangePolicy, SELinuxMountReadWriteOncePod and KubeProxyIPVS. The table is the fastest way to check a claim about which release flipped which default
9. KEP-5495: Deprecate ipvs mode in kube-proxy. The deprecation timeline this article quotes - warning now, feature gate defaulting to false by v1.40, removal by v1.43 - comes from its graduation criteria and nowhere else
10. KEP-5495 README on GitHub: the same document at its source, including the graduation criteria table with the release numbers
11. KEP-3866: nftables kube-proxy backend, including the section titled "The ipvs mode of kube-proxy will not save us" - the technical argument that ipvs mode never stopped depending on iptables underneath, which is the reason for the deprecation
12. NFTables mode for kube-proxy - the introduction to the mode that replaces both iptables and ipvs, and the migration notes for moving to it
13. Kubernetes - kube-proxy configuration (v1alpha1) reference: the `mode` field this article tells you to read and change, and the rest of the KubeProxyConfiguration surface
14. KEP-5573: Remove CGroup v1 support. The staged removal plan behind the kubelet's failCgroupV1 setting, and the statement that the override is temporary
15. Kubernetes - About cgroup v2: how to check which version a node is on, the requirements for cgroup v2, and the features that depend on it
16. Kubernetes - kubelet configuration (v1beta1) reference: failCgroupV1 and the rest of the KubeletConfiguration fields this article edits
17. Kubernetes - Static Pods: what they are, why the kubelet manages them directly rather than through the API server, and therefore why referencing API objects from one was never supposed to work
18. kubernetes/kubernetes issue 140226 - the discussion behind prohibiting Secret and ConfigMap references from static pods, and the fate of the PreventStaticPodAPIReferences feature gate that allowed an opt-out. The gate has defaulted to on since v1.34; the v1.37 sneak peek says it was removed in this release while the shipped v1.37 feature-gates reference still lists it, so the reference page is the one to trust
19. kubernetes/kubernetes issue 138671 - the deprecation of `kubectl run --filename/-f`, on the grounds that the pod it produces is always built purely from the command-line arguments
20. KEP-5207: metrics.k8s.io API definition. The enhancement that graduates the resource metrics API to stable after nearly nine years in beta, and the statement that v1 and v1beta1 both remain available during the transition
21. Kubernetes - Resource metrics pipeline: what metrics.k8s.io actually serves, who serves it, and how `kubectl top` and the HorizontalPodAutoscaler consume it
22. Kubernetes - Horizontal Pod Autoscaling: the largest consumer of the resource metrics API, and the reason its graduation matters beyond `kubectl top`
23. KEP-2033 / KEP-4960: Kubelet in UserNS, also known as rootless mode. Graduating to beta in v1.37, which lets node components run as an unprivileged user on the host while still appearing as root inside the namespace
24. Kubernetes - User namespaces for pods: the workload-level feature that reached GA in v1.36, distinct from the rootless kubelet but built on the same kernel mechanism
25. KEP-1432: Volume health monitoring. Reset to alpha in v1.37 with four new CSI RPCs and three new status fields, after an initial implementation in v1.21 that never graduated
26. Kubernetes v1.34: Of Wind & Will - the release that opened the containerd 1.x end-of-support discussion and the CRI cgroup-driver work behind it. Read alongside the container runtimes page, which records where that timeline actually ended up: the fallback is dropped in v1.38, not v1.36
27. Kubernetes v1.36 release announcement - the release that made user namespaces GA, graduated SELinuxMountReadWriteOncePod, and shipped the selinux-warning-controller that this article tells you to switch on
28. Kubernetes v1.36: Deprecation and removal of Service ExternalIPs - note that despite the title, v1.36 only deprecates the field and starts emitting warnings; kube-proxy support goes off by default no earlier than v1.40 and removal is no earlier than v1.43. A good illustration both of why reading one release's notes is not enough, and of how a headline turns into a rumour
29. Kubernetes - Releases: the supported branches and their end-of-life dates. The source for the support window this article uses to argue about how much runway a cluster actually has
30. Kubernetes - Patch releases: the cadence, the support period and the maintenance-mode window at the end of each minor release's life
31. Kubernetes - Version skew policy: how far the kubelet may lag the API server, which is what makes a staged node upgrade legal in the first place
32. Kubernetes - Upgrading kubeadm clusters: the control plane first, then one node at a time, with drain and uncordon around each. The sequence the runbook in this article slots into
33. Kubernetes - Deprecation policy: the rules that govern how long a deprecated feature must survive before removal, which is why an ipvs deprecation in v1.37 cannot become a removal before v1.43
34. Kubernetes - Container runtimes: installing and configuring containerd or CRI-O, including the cgroup driver requirement that ties this article to the cgroup v2 migration. It is also the page that settles the containerd question, stating that older containerd versions still work today through the kubelet's cgroup-driver fallback and that in Kubernetes 1.38 that fallback is dropped and they will fail with newer kubelets
35. containerd - Versioning and release: the release-status table and the Kubernetes/containerd support matrix. As of September 2026 the matrix stops at Kubernetes 1.36 (2.3.0+, 2.2.0+) and has no row for 1.37, and containerd 1.7's extended LTS support ends in September 2026
36. Kubernetes - Releases: the release and support dates quoted in this article, including v1.35 on 17 December 2025, v1.36 on 22 April 2026 and v1.37 on 26 August 2026
37. Kubernetes - MutatingAdmissionPolicy: the in-tree way to apply the seLinuxChangePolicy opt-out across a namespace or a cluster without editing every workload by hand
38. Kyverno: a policy engine the SELinux blog names explicitly as a way to apply the opt-out fleet-wide. Listed because the alternative - patching every Deployment and StatefulSet individually - does not scale past a small cluster
39. Gateway API v1.6: TCPRoute and UDPRoute graduate to Standard. Out of scope for this article but on the same upgrade window for most clusters, and the reason the ingress migration keeps appearing in the same maintenance plan