containerd 1.x 已经没有退路

这场迁移有两种读法：一种像待办事项，一种像最后期限，而你落在哪一种，取决于一条脚注。containerd 的版本表把 1.7 分支标为 LTS，支持到 2026 年 9 月，看上去还有余地。下面那条脚注说明，自 2026 年 3 月起这段支持由两位具名维护者提供，并且只面向 Kubernetes 1.32、1.31、1.30 在 Google Kubernetes Engine 上的使用场景，与该场景无关的改动可以被拒绝。Kubernetes 1.32 已于 2026 年 2 月停止维护。如果你不是在 GKE 上跑一套已经停止维护的 Kubernetes，这根救命绳从来就没朝你扔过。

所以这是一场有真实期限的迁移，不该只是在运维手册里把版本号加一就算完事。下面写的是全套：怎么读 containerd 的版本策略而不被 LTS 这个词误导，为什么 2.1 是最糟糕的落脚点，配置从 version 2 到 version 3 的重写与挪了位置的插件 ID，带着自己独有 bug 和独有故障形态的镜像仓库改造，2.1 里悄悄换掉的镜像拉取路径，2.0 删掉的全部内容，一份按节点执行的操作手册，一份关于回滚能救回什么、救不回什么的实话，以及一份出错时返回非零退出码的验证脚本。

这些故障现象，没有一个会提到 containerd

这一类故障没有一个会自报家门说是运行时版本的问题，所以往往查得很晚。一个节点升级后回来，CRI 插件根本没起来：守护进程在跑，systemctl status 是绿的，而节点上每个 Pod 都卡在 ContainerCreating。一个四年来一直从内部镜像源拉的镜像，忽然开始从 Docker Hub 拉，最先被察觉的是出网流量账单。某个团队两年前手工加的 RuntimeClass 解析不到了，只有用它的那些负载会挂。每一种情况里，运行时都在跑，而运行时是错的。^[ctrrel]

共同的线索是：containerd 2.x 对 version 2 的配置文件是刻意宽容的——读进来，在内存里转换，然后照常启动。升级当下这是善意，之后就是负债，因为这意味着迁移可以无限期地停在半途，而没有任何东西逼你收尾。守护进程把抱怨写进日志然后照样启动；没加载成功的插件报了一个错误状态，而没有任何东西会把它端到你面前。从某种意义上说，整篇文章都是在论证一件事：把迁移做完，别让兼容层替你扛着。^[ctr20]

把支持周期表读对

先从版本表开始，因为它是唯一能拍板的文档，也是被系统性读错的那一份。containerd 维护两类分支。普通版本支持八个月。每年有一个版本被指定为 LTS，至少支持两年。在这之上，某个分支还可以在常规窗口关闭之后，由具名维护者提供延长支持——同一列里挂着同一个标签，实质上却是另一回事。^[ctrrel]

接着把 Kubernetes 的支持矩阵叠上去，这是两个项目交汇的地方。containerd 会为每个 Kubernetes 小版本给出一份推荐版本清单。对 Kubernetes 1.36，这份清单写的是 2.3.0+、2.2.0+——完全没有 1.x 的条目。kubelet 里没有任何机制会强制这一点：不受支持的组合照样启动、照样运行、看起来一切正常，直到它不正常，那时候就只剩你自己在排查了。这张矩阵表达的是“测过什么”，而这些测试是你和一个别人没遇到过的运行时 bug 之间唯一的东西。^[k8srel]

坑里还有一个坑，叫 containerd 2.1。对 2025 年下半年做升级的人来说，它是最顺手的落脚点，很多内部文档到今天还这么写，而它已经在 2026 年 7 月 3 日停止维护——比支持到 2026 年 11 月的 2.2 还早，更远早于当前 LTS、支持到 2028 年 4 月的 2.3。“升到 2.x”不是计划。从零开始只有一个值得瞄准的分支，就是 2.3。

这些节点上到底装了什么

动手之前先把“实际装的是什么”搞清楚，因为在任何规模的机器群里，答案都不会只有一个版本。有三件事要看，而且互相独立：守护进程版本、配置文件版本、以及实际提供给 kubelet 的 CRI API 版本。配置版本是最容易被忽略的，也是唯一可能“缺席”的那个：没有 version 这一行的文件，会被当成 version 1 文件处理。而在这一点上，项目自己的文档互相打架，值得知道分歧在哪：CRI 配置指南说 version 1 已在 containerd 2.0 中被删除，而 RELEASES.md 说缺少版本号时按 version 1 解析，并且所有旧版本都能通过迁移得到支持，源码里也确实还留着一个 v1 的迁移函数。所以对 version 1 的文件，正确的态度是看见就修，而不是指望自己能推断出它到底会怎么被对待。^[cfgtoml]

# The daemon, the client and the shim are three separate versions and they are
# allowed to disagree. Ask all three rather than assuming.
containerd --version
# containerd github.com/containerd/containerd/v2 v2.3.2 <revision>
ctr version                        # client and server, side by side
runc --version                     # the OCI runtime is a separate install now

# The configuration version is the single most useful number here. There is no
# `version` line in very old files: absent means version 1, which containerd
# 2.0 removed outright rather than migrating.
head -1 /etc/containerd/config.toml
# version = 2

# What the plugins are doing. A plugin in state "error" is the daemon telling
# you a migration went wrong; it does not stop the daemon from starting.
ctr plugins ls | awk '$4!="ok"'
# TYPE  ID  PLATFORMS  STATUS

# From the Kubernetes side, which is what actually matters:
kubectl get nodes -o custom-columns=\
'NODE:.metadata.name,RUNTIME:.status.nodeInfo.containerRuntimeVersion,'\
'KUBELET:.status.nodeInfo.kubeletVersion,OS:.status.nodeInfo.osImage'
# NODE     RUNTIME                KUBELET   OS
# node-01  containerd://1.7.28    v1.34.9   Ubuntu 24.04.3 LTS

# And the CRI API version the kubelet is really getting. containerd 2.0 removed
# v1alpha2; if anything on this node still speaks it, it stops working here.
crictl version
# RuntimeName:  containerd
# RuntimeApiVersion:  v1

然后去问守护进程，它其实早就想告诉你的事。从 1.6.27 和 1.7.12 起，containerd 就通过 introspection API 暴露废弃告警，目的正是让这场迁移可以被计划，而不是被撞见。子命令是 ctr deprecations list——复数，值得特意说一句，因为至少有一份官方文档把它写成了单数，而那个写法并不存在。用 --format json 在全部节点上跑一遍。它不是健康证明，因为告警是在使用时才产生的：一个自上次重启以来没拉过 schema 1 镜像的节点，不会报出这一条。它是“你已经知道有问题”的那份初始清单。^[depsrc]

# containerd has been telling you what will break since 1.6.27 / 1.7.12, through
# the introspection API. Almost nobody reads it, because the warnings go into
# the daemon log rather than anywhere you look. Ask directly.
#
# Note the subcommand is `deprecations`, plural. Some documentation writes it
# in the singular; that form does not exist and returns a usage error.
ctr deprecations list
# ID                                          LAST OCCURRENCE   MESSAGE
# io.containerd.deprecation/pull-schema-1-image  2026-08-24...   Schema 1 image...
# io.containerd.deprecation/cri-registry-mirrors 2026-08-24...   `mirrors` is deprecated...

# Machine-readable, which is the form you want across a fleet:
ctr deprecations list --format json | jq -r '.[].id' | sort -u

# Run it on every node and count, rather than sampling. The warnings are
# emitted on use, so a node that has not pulled a schema 1 image since the last
# daemon restart will not report one - which is why this is a starting point
# and not a clean bill of health.
for n in $(kubectl get nodes -o name); do
  printf '%-22s ' "${n#node/}"
  kubectl debug "$n" -it --image=busybox --profile=general -- \
    chroot /host ctr deprecations list --format json 2>/dev/null \
    | jq -r '[.[].id] | join(",")' || echo '(unavailable)'
done

# Clean up afterwards. `kubectl debug node/...` names its pods
# node-debugger-<node>-<suffix> and applies no label of its own, so there is
# nothing to select on - match the name instead, or they accumulate silently.
kubectl get pods -n default -o name | grep '^pod/node-debugger-' | xargs -r kubectl delete

配置文件，从 version 2 到 version 3——现在还有 version 4

配置文件是这场迁移的主体，而第一件要弄清楚的事是“最新版本”到底指哪一个，因为它在今年又往前挪了一格。version 3 随 containerd 2.0 到来，把 CRI 插件一分为二的正是它。version 4 随 2.3 到来，做的是另一件事，下一节专门讲。version 2 至今仍然会被读取，并在每次启动时于内存中转换——守护进程会为此在日志里留下一行，这是判断一个节点究竟是真的迁移过、还是只是被兼容层收留着的最省事的办法。守护进程自带一个转换器 containerd config migrate，它读你当前的文件，把最新版本的内容打到标准输出。man 手册里没有它——containerd-config(8) 只写了 default——这在很大程度上解释了为什么这么少人知道它存在。^{[cricfg][cfgsrc]}

# containerd 2.x reads a version 2 file and converts it in memory on every start.
# That is a compatibility shim, not a plan: it costs startup time, it is where
# the registry bug below lives, and the daemon says so on every boot.
journalctl -u containerd | grep -m1 'Configuration migrated from version'
# Configuration migrated from version 2, use `containerd config migrate` to
# avoid migration

# `containerd config migrate` reads your current file and prints the LATEST
# version on stdout. It is not in the man page - only `default` is - but it has
# been in the binary since 2.0.
#
# Note which version "latest" means, because it moved. Version 3 arrived in
# containerd 2.0 and is the one that split the CRI plugin in two. Version 4
# arrived in 2.3 and moves the server sockets into plugins (see below).
containerd config migrate > /tmp/config.new.toml
head -1 /tmp/config.new.toml
# version = 4        <- on containerd 2.3. On 2.0-2.2 this says 3.

# Two things to know before you trust the output. First, `migrate` and `dump`
# share one implementation, so the result is the FULLY POPULATED configuration,
# defaults and all - not a minimal file. Every default you did not choose is now
# pinned in your file and stops following the daemon when upstream changes it.
wc -l /etc/containerd/config.toml /tmp/config.new.toml
#   41 /etc/containerd/config.toml
#  318 /tmp/config.new.toml

# Second, and this is upstream's own warning: migrating the file to the latest
# version limits which containerd versions can read it. A version 4 file needs
# 2.3.0 or newer. If you might want to roll the binary back tonight, write a
# version 3 file instead - 2.0 and later read it, and it still gets you the
# plugin split, which is the part that matters.

# So: use the output to learn the new names, then hand-write the short version.
# What did it actually change? Compare the keys, not the files.
grep -oE '^\s*\[[^]]+\]' /tmp/config.new.toml | tr -d ' []' | sort > /tmp/new.keys
grep -oE '^\s*\[[^]]+\]' /etc/containerd/config.toml | tr -d ' []' | sort > /tmp/old.keys
diff -u /tmp/old.keys /tmp/new.keys

# Validate before you restart anything. `config dump` loads the file the daemon
# would load, including everything pulled in by `imports`, and fails loudly on
# a file it cannot parse. Note that --config is a global flag: it goes BEFORE
# the subcommand. Putting it after `config dump` is a usage error, not a check -
# urfave/cli rejects it with "flag provided but not defined: -config".
containerd --config /tmp/config.new.toml config dump >/dev/null && echo 'parses'

# Keep the old one. It is the fastest rollback you have.
cp -a /etc/containerd/config.toml /etc/containerd/config.toml.v2.bak

在把它的输出直接盖到线上配置之前，有两件事要知道。第一，migrate 和 dump 走的是同一条代码路径，所以返回的是完整展开的配置：守护进程的每一个默认值都被显式写出来。四十行的文件会变成三百行，而每一个你从没选过的默认值，现在都被钉死在你的文件里，上游改动时它不会跟着走。用它的输出去学新的键名，然后自己手写那份短的。第二，在重启任何东西之前先校验候选文件——但要注意 --config 是全局参数而不是子命令参数，所以它得写在 config dump 之前，也就是 containerd --config <文件> config dump；写在后面不是校验，而是一个用法错误。写对了，它就会按守护进程的方式加载文件、跟进 imports，遇到解析不了的东西明确报错——这比在一个起不来的节点上发现拼写错误要好得多。^[cfgman]

# Configuration version 4 (containerd 2.3 and later). It changes nothing about
# CRI: the whole of the plugin split above is version 3 work. What it moves is
# the daemon's own sockets, out of top-level tables and into server plugins.

# --- version 3 and earlier -------------------------------------------------
#   [grpc]
#     address = "/run/containerd/containerd.sock"
#     uid = 0
#     gid = 0
#   [ttrpc]
#     address = "/run/containerd/containerd.sock.ttrpc"
#   [metrics]
#     address = "127.0.0.1:1338"
#   [debug]
#     address = "/run/containerd/debug.sock"
#     level = "info"

# --- version 4 -------------------------------------------------------------
version = 4

[plugins.'io.containerd.server.v1.grpc']
  address = '/run/containerd/containerd.sock'
  uid = 0
  gid = 0

[plugins.'io.containerd.server.v1.ttrpc']
  address = '/run/containerd/containerd.sock.ttrpc'

[plugins.'io.containerd.server.v1.metrics']
  address = '127.0.0.1:1338'

[plugins.'io.containerd.server.v1.debug']
  address = '/run/containerd/debug.sock'

# `[debug]` does not disappear: level, format and log_trace_id stay at the top
# level. Only the socket fields move.
#
# One behaviour change hides in here. Before version 4, an unset ttrpc address
# was derived from the grpc address as "<grpc address>.ttrpc" and inherited its
# uid and gid. In version 4 the ttrpc plugin is independent and falls back to
# its own default. If anything of yours connects to that socket by path -
# a shim debugger, a monitoring agent - set it explicitly rather than assuming.

version 4 值得单独看一眼，因为几乎所有谈这场迁移的文章都停在 version 3，也因为它带着一条回滚上的约束。它对 CRI 没有任何改动：它把守护进程自己的 socket 从顶层的 [grpc]、[ttrpc]、[metrics]、[debug] 表里挪了出来，放进 io.containerd.server.v1.* 这组插件。由此带来两个后果。行为上的那个：在 version 4 之前，没有显式设置的 ttrpc 地址是从 gRPC 地址推导出来的，形如 <grpc address>.ttrpc，并继承它的 uid 和 gid；而在 version 4 里，ttrpc 插件是独立的，会回落到它自己的默认值——所以你手上任何按路径连这个 socket 的东西，现在都应该把地址显式写出来。操作上的那个是上游自己的提醒：把文件迁到最新版本，会缩小能读懂它的 containerd 版本范围。version 4 的文件需要 2.3.0 或更新的版本，而 version 3 的文件从 2.0 起都能读。如果你的方案里包含“当晚就能把二进制换回去”，那就写 version 3——插件拆分这个真正要紧的部分，你照样拿到了。^{[cfgver][srvmig]}

一个插件拆成两个，配置项也跟着挪了

结构上的变化是：原本唯一的 CRI 插件被拆成了两个。io.containerd.grpc.v1.cri 过去装着所有东西；在 version 3 里它只剩流式服务器的选项，实质内容搬到了两个新的 ID 下：io.containerd.cri.v1.runtime 管一切与“跑容器”有关的东西——runtimes、CNI、sandbox、SELinux、OOM 处理；io.containerd.cri.v1.images 管一切与镜像有关的东西——snapshotter、镜像仓库、被固定的 sandbox 镜像、拉取并发。这个划分比原来的好，同时也意味着：对插件 ID 做机械式查找替换，会把你大约一半的配置项放进错误的表里。^[ctrarch]

# /etc/containerd/config.toml - containerd 1.7, the file most clusters have.
# Everything lives under one plugin ID: io.containerd.grpc.v1.cri
version = 2

[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "registry.k8s.io/pause:3.10"

  [plugins."io.containerd.grpc.v1.cri".containerd]
    snapshotter = "overlayfs"
    default_runtime_name = "runc"

    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
      runtime_type = "io.containerd.runc.v2"
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
        SystemdCgroup = true

    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.gvisor]
      runtime_type = "io.containerd.runsc.v1"

  [plugins."io.containerd.grpc.v1.cri".cni]
    bin_dir = "/opt/cni/bin"
    conf_dir = "/etc/cni/net.d"

  # The block that causes the most trouble in this migration.
  [plugins."io.containerd.grpc.v1.cri".registry]
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
      endpoint = ["https://mirror.internal.example.com"]

# /etc/containerd/config.toml - containerd 2.x. The single CRI plugin has been
# split in two, and the settings moved with the split: anything about running
# containers is now under io.containerd.cri.v1.runtime, anything about images
# under io.containerd.cri.v1.images. io.containerd.grpc.v1.cri still exists,
# but only for the streaming server options.
version = 3

[plugins.'io.containerd.cri.v1.images']
  snapshotter = 'overlayfs'                 # moved: was under ...cri.containerd

  [plugins.'io.containerd.cri.v1.images'.pinned_images]
    sandbox = 'registry.k8s.io/pause:3.10.2'  # replaces sandbox_image

  [plugins.'io.containerd.cri.v1.images'.registry]
    config_path = '/etc/containerd/certs.d'   # replaces the mirrors block

[plugins.'io.containerd.cri.v1.runtime']
  [plugins.'io.containerd.cri.v1.runtime'.containerd]
    default_runtime_name = 'runc'

    [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
      runtime_type = 'io.containerd.runc.v2'
      [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
        SystemdCgroup = true

    [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.gvisor]
      runtime_type = 'io.containerd.runsc.v1'

  [plugins.'io.containerd.cri.v1.runtime'.cni]
    bin_dirs = ['/opt/cni/bin']   # bin_dir is deprecated since 2.1: plural now
    conf_dir = '/etc/cni/net.d'

# Note what is NOT here: no registry.mirrors alongside config_path. Setting
# both is an error - "`mirrors` cannot be set when `config_path` is provided" -
# and the CRI plugin refuses to load rather than picking one.
#
# And note the clock on the old keys. registry.mirrors and registry.configs
# were deprecated in containerd 1.5, registry.auths in 1.3, and cni.bin_dir in
# 2.1. All four carry the same removal target: containerd 2.4. Converting them
# is not housekeeping you can defer past the next release.

这次拆分里有两处改名造成了大部分损失。sandbox_image 变成了 pinned_images.sandbox，于是原本把 pause 镜像指向内部镜像源的集群，会悄无声息地退回 registry.k8s.io——在节点还能出网的日子里毫无问题。另一处是 snapshotter，它从运行时那一侧挪到了镜像那一侧，反直觉到值得去核对而不是想当然。下表里其余的条目都只是路径变化，不是行为变化。^[ctrplug]

镜像仓库：真正会让集群趴下的那一段

镜像仓库配置是这场迁移从“繁琐”变成“有风险”的地方，而且它带着日期：mirrors 和 configs 在 containerd 1.5 中被废弃，auths 在 1.3 中被废弃，三者的移除目标都是 containerd 2.4——也就是本文推荐的那个版本的下一个版本。前两个的替代品是一棵目录树：在单个 config_path 之下，每个仓库主机命名空间一个子目录，各自放一个 hosts.toml。文件更多，魔法更少，而且确实更好用：多个 host 按顺序尝试，capabilities 是显式写出来的，每个仓库自己的 CA 只是文件里的一行而不是一个特例。第三个属性 auths 故意没有对应的文件写法：凭据应该放进 Kubernetes 的 imagePullSecret，而不是放在一份节点级配置里，让这台机器上的每个负载都继承它。^{[crireg][hosts]}

# The mirrors / configs / auths properties are deprecated. The replacement is a
# directory of hosts.toml files, one per registry host namespace, pointed at by
# a single config_path. It is more files and considerably less magic.

# Directory naming, which is where this goes wrong silently. containerd looks
# for the host namespace in three forms, in order:
#     <host>_<port>_        e.g. registry.internal.example.com_5000_
#     <host>:<port>         e.g. registry.internal.example.com:5000
#     _default
# The first form is the portable one - a colon is not a legal filename on
# Windows - so prefer it. A directory named anything else looks perfectly
# correct and simply never matches.

mkdir -p /etc/containerd/certs.d/docker.io
cat > /etc/containerd/certs.d/docker.io/hosts.toml <<'TOML'
server = "https://docker.io"

[host."https://mirror.internal.example.com"]
  capabilities = ["pull", "resolve"]

# Fall through to the real registry if the mirror does not have the layer.
# Order matters: hosts are tried top to bottom.
[host."https://registry-1.docker.io"]
  capabilities = ["pull", "resolve"]
TOML

# A private registry on a port, with its own CA:
mkdir -p /etc/containerd/certs.d/registry.internal.example.com_5000_
cat > /etc/containerd/certs.d/registry.internal.example.com_5000_/hosts.toml <<'TOML'
server = "https://registry.internal.example.com:5000"

[host."https://registry.internal.example.com:5000"]
  capabilities = ["pull", "resolve", "push"]
  ca = "/etc/containerd/certs.d/internal-ca.crt"
TOML

# registry.auths has no file equivalent, on purpose. Credentials belong in a
# Kubernetes imagePullSecret, not in the node's runtime configuration where
# every workload on the node inherits them.
kubectl create secret docker-registry regcred \
  --docker-server=registry.internal.example.com:5000 \
  --docker-username=ci --docker-password="$REG_PASSWORD"

# Verify resolution without restarting anything. --hosts-dir makes ctr read the
# same tree the CRI plugin will read.
ctr images pull --hosts-dir /etc/containerd/certs.d docker.io/library/alpine:3.22

# And confirm the daemon agrees once it has restarted:
containerd config dump | grep -A3 "cri.v1.images'.registry"
# config_path = '/etc/containerd/certs.d'

这个 bug 值得把话说准，因为不准的那个版本会把你带去错的地方翻。它并不要求你写过什么奇怪的东西。拿一份普通的 version 2 文件，里面只有一个 registry.mirrors 块，别的什么都没有，用它启动 containerd 2.2.0——内存里的迁移会在旁边补上默认的 config_path，而 CRI 插件对这种组合直接拒绝：`mirrors` cannot be set when `config_path` is provided。插件加载不了。守护进程照样启动。所有调度到这个节点的 Pod 随即创建失败，而这台机器上每个服务的顶层状态都还是绿的。你拿这两个键去 grep 自己的文件，什么也找不到，因为其中一个根本不是你写的。该问题是针对 2.2.0 报告的，修复在 pull request 12617 中，它在 2.3.0 之前合入，并被回合到 2.2 分支——所以在当前的 2.3 或较新的 2.2 补丁版上你不受影响，在 2.0 或 2.1 上则会中招。无论哪种情况，都成立的那条指令是把改造做完：建好 certs.d 目录树，把 config_path 指过去，然后删掉 mirrors 块，别给迁移留下需要猜的余地。^{[iss12612][pr12617]}

镜像拉取的路径在你不知道的时候变了

从 containerd 2.1 起，CRI 插件通过 Transfer Service 拉取镜像，而不再在自己的进程里拉。这是默认行为，不是你打开的开关，单看这一点也没什么特别。值得单开一节的是它的回退机制：如果 CRI 的镜像配置里有 Transfer Service 无法照办的设置，containerd 会为整个节点设上 use_local_image_pull 并记一条警告。它不报错，不在使用现场告诉你，也不告诉集群。^[ctrxfer]

# Since 2.1 the CRI plugin pulls images through the Transfer Service instead of
# pulling in-process. This is not a flag you set; it is the default. What makes
# it worth knowing is the fallback: if the CRI image configuration contains
# anything the Transfer Service cannot honour, containerd silently switches the
# whole node back to local pull and logs a warning.
#
# The triggers, from the CRI config guide:
#   Registry.Mirrors set          Registry.Configs set      Registry.Auths set
#   MaxConcurrentDownloads != 3   DiscardUnpackedLayers = true
#   ImagePullWithSyncFs = true    DisableSnapshotAnnotations = false
#
# Which means the perfectly reasonable act of raising the download concurrency
# quietly changes the code path your images are pulled through.

journalctl -u containerd --since '10 min ago' \
  | grep -iE 'transfer|use_local_image_pull|falling back'

# If you want local pull, ask for it rather than triggering it by accident:
#   [plugins.'io.containerd.cri.v1.images']
#     use_local_image_pull = true
#
# If you want the Transfer Service, move the settings to where it reads them:
#   [plugins.'io.containerd.transfer.v1.local']
#     max_concurrent_downloads = 6

# Check which path a real pull took, end to end:
crictl pull registry.k8s.io/pause:3.10.2
crictl images | head
ctr -n k8s.io images ls | wc -l

把触发列表读一遍，含义就很清楚了：把 max_concurrent_downloads 从 3 调到 6——在带宽好的节点上是再正常不过的做法——就会把这个节点上所有的镜像拉取切到另一条代码路径。保留那个已废弃的 mirrors 块也是同样效果，这是“把镜像仓库改造做完而不是搁一半”的第二个理由。如果你要的是本地拉取，就把 use_local_image_pull = true 写明白。如果你要的是 Transfer Service，就把并发设置挪到真正会读它的 [plugins.'io.containerd.transfer.v1.local'] 下面。^[cricfg]

真正被删掉的东西

接下来是移除项，也就是真正消失、而不只是改名的那部分。清单很短，上面每一项也都有文档写明的替代品，但要去读版本号，别读摘要——containerd 2.0 的迁移文档和 RELEASES.md 在两处对不上，而两次被人引用的都是迁移文档。Docker schema 1 拉取是其中要紧的一例：它在 2.0 中被禁用，当时还有一个环境变量能把它打开；在 2.1 中被移除，此后什么都打不开它。既然这里的目标是 2.3，就当它已经没了。这一点之所以要紧，是因为还停留在 schema 1 的镜像，按定义就是大约 2017 年以后没人重新构建过的镜像，也就意味着没人手上还有那份 Dockerfile。要在升级前找出来，不要升级后再找：从 1.7.8 和 1.6.25 起，转换过的镜像带有一个标签，可以据此检索。^{[ctr20][ctrrelmd]}

# Docker schema 1 manifests. Get the timeline right, because it decides whether
# you have a workaround or a deadline:
#   containerd 2.0  pulling is DISABLED by default, and the environment variable
#                   CONTAINERD_ENABLE_DEPRECATED_PULL_SCHEMA_1_IMAGE=1 re-enables it
#   containerd 2.1  support REMOVED. The variable does nothing. So does anything else.
# Since 2.3 is the target, treat this as removed and find the images NOW.
#
# Since 1.7.8 / 1.6.25 converted images carry a label, so they are findable:
ctr namespaces list --quiet | xargs -I{} -- \
  ctr --namespace={} image list \
    'labels."io.containerd.image/converted-docker-schema1"'

# On a node still running 1.7, the same list from the CRI side:
crictl images -o json | jq -r '.images[].repoTags[]' | sort -u > /tmp/node-images.txt

# For each one, ask the registry what media type it actually serves. A schema 1
# manifest answers with application/vnd.docker.distribution.manifest.v1+prettyjws.
# Anything that does needs rebuilding in schema 2 or OCI before the node moves.

# The runtime v1 shims were removed in 2.0. Anything still asking for them
# fails to start the container, with an error about an unknown runtime:
grep -rn 'io.containerd.runtime.v1.linux\|io.containerd.runc.v1' \
  /etc/containerd/ /etc/crio/ 2>/dev/null
kubectl get runtimeclass -o custom-columns=NAME:.metadata.name,HANDLER:.handler

# The AUFS snapshotter was removed. Almost nobody sets this, and the ones who
# do have a kernel from 2016 underneath it:
containerd config dump | grep -E "snapshotter\s*=" | sort -u

# LimitNOFILE is no longer set in the reference unit. On systemd 240 and newer
# the default is fine; below that the kernel default of 4096 applies, and
# containers inherit it.
systemctl show containerd -p LimitNOFILE -p LimitNOFILESoft
systemctl --version | head -1

有一处移除比其余的都安静，值得点名。参考用的 containerd.service 单元不再显式设置 LimitNOFILE。containerd 的 rlimit 会被它启动的容器继承，所以这并不是只影响守护进程的设置：在 systemd 240 及以后，默认值是合理的，这件事没有影响；在更早的版本上，生效的是内核默认值 4096，而这台机器上的每个容器都会继承它。上游对这类主机的建议是把 LimitNOFILE=1024:524288 手工加回去。^{[pr8924][sdexec]}

没打招呼就翻面的默认值

除了移除项之外，还有几个默认值换了取值。正是它们让一个“你一个配置都没改”的升级之后表现不同，所以对每一条都该明确做个决定，而不是默认照单全收。^[ctr20]

• 非特权端口和 ICMP 默认打开。CRI 插件现在会为既不使用宿主网络命名空间、也不使用用户命名空间的容器设置 net.ipv4.ip_unprivileged_port_start=0 和 net.ipv4.ping_group_range=0 2147483647。绑定 1024 以下的端口不再需要 CAP_NET_BIND_SERVICE，ping 也不再需要 CAP_NET_RAW。方便，同时也是容器安全姿态的变化——把 enable_unprivileged_ports 和 enable_unprivileged_icmp 设为 false 就能恢复旧行为。
• NRI 默认启用。Node Resource Interface 允许插件在容器创建时修改容器。访问控制靠的是对系统级 NRI socket 的访问权限，也就是说无论你是否真的跑了哪怕一个 NRI 插件，这个 socket 已经成为节点攻击面的一部分。
• CDI 默认启用，而这个开关本身也快没了。Container Device Interface 处于开启状态，cdi_spec_dirs 默认指向 /etc/cdi 和 /var/run/cdi，所以任何能往这两个目录写 spec 文件的东西，都能为容器描述设备访问。要注意 enable_cdi 这个开关在 containerd 2.2 中已被废弃，移除目标是 2.4，之后 CDI 就是一直开着——如果你的打算是把它关掉，那个打算是有保质期的。
• io_uring 不再在默认 seccomp 白名单里。io_uring_enter、io_uring_register 和 io_uring_setup 在一连串内核漏洞之后被移出白名单，项目认为默认放行不再安全。围绕 io_uring 构建的负载需要一份显式的 profile——以及一次“它是否应该拥有这份 profile”的讨论。
• 带 sandbox 的 CRI 实现成为默认。CRI 插件使用稳定的 sandbox 控制器，而不是旧的 CRI server。正常运行时这完全看不出来；跑 Kata 或 gVisor 这类带 sandbox 的运行时时非常明显——而这恰恰是全量铺开之前该测的场景。

以上没有一条是不升级的理由。它们是“先升一个节点、看一看、然后才把改动写进镜像构建”的理由——这就是一场迁移和一次全机群意外之间的区别。^{[ctrnri][cdi]}

单节点升级，按顺序来

机械操作的部分很短，变化更多在做法上而不在内容上。合并包 cri-containerd-cni-VERSION-OS-ARCH.tar.gz 在 2.0 中被移除，因此 containerd、runc 和 CNI 插件现在是三次独立安装、三个独立的版本决定。这更显式，也稍微多点活，同时消除了一个存在多年的混淆来源：过去有人升级 containerd 的同时不知不觉把 runc 也升了。关于这一跳本身有一点可以放心：containerd 支持相邻小版本之间的升级，也单独支持相邻 LTS 版本之间的直接升级——而 1.7（LTS）到 2.3（LTS）正是它自己的发布文档给出的例子。你并没有跳过任何本该落脚的地方。^[ctrstart]

#!/usr/bin/env bash
# One node, from containerd 1.7 to 2.3 LTS. Run it on a drained node.
#
# 1.7 -> 2.3 is a supported jump. containerd supports sequential minor upgrades
# and, separately, direct upgrades between sequential LTS releases - and it
# names 1.7 (LTS) to 2.3 (LTS) as an example. That is exactly this path.
set -euo pipefail
VER=2.3.4                 # check https://containerd.io/releases/ before pinning
RUNC_VER=1.5.0
CNI_VER=1.9.1
ARCH=amd64

# 0. Get the workloads off, and keep the node out of rotation until verified.
#    (From the control plane, not from the node.)
#    kubectl drain node-01 --ignore-daemonsets --delete-emptydir-data

# 1. Back up everything the daemon owns. /var/lib/containerd is the image and
#    snapshot store; losing it means re-pulling every image on the node.
systemctl stop kubelet containerd
cp -a /etc/containerd/config.toml /etc/containerd/config.toml.bak
tar -C /var/lib -czf "/var/backups/containerd-lib-$(date +%F).tgz" containerd

# 2. Install the three components separately. The combined
#    cri-containerd-cni-*.tar.gz bundles were removed in 2.0; this is now the
#    supported route rather than a workaround.
curl -fsSLO "https://github.com/containerd/containerd/releases/download/v${VER}/containerd-${VER}-linux-${ARCH}.tar.gz"
curl -fsSLO "https://github.com/containerd/containerd/releases/download/v${VER}/containerd-${VER}-linux-${ARCH}.tar.gz.sha256sum"
sha256sum -c "containerd-${VER}-linux-${ARCH}.tar.gz.sha256sum"
tar -C /usr/local -xzf "containerd-${VER}-linux-${ARCH}.tar.gz"

#    runc and the CNI plugins, pinned deliberately rather than left behind.
#    These used to ride along inside the removed bundles; now they are yours
#    to choose, which also means yours to forget.
curl -fsSL -o /usr/local/sbin/runc \
  "https://github.com/opencontainers/runc/releases/download/v${RUNC_VER}/runc.${ARCH}"
chmod 755 /usr/local/sbin/runc

mkdir -p /opt/cni/bin
curl -fsSLO "https://github.com/containernetworking/plugins/releases/download/v${CNI_VER}/cni-plugins-linux-${ARCH}-v${CNI_VER}.tgz"
tar -C /opt/cni/bin -xzf "cni-plugins-linux-${ARCH}-v${CNI_VER}.tgz"

# 3. Refresh the systemd unit from the release, then reapply any drop-in of
#    your own. Note the reference unit no longer sets LimitNOFILE, and that the
#    directory does not exist on a host that came from a distribution package.
mkdir -p /usr/local/lib/systemd/system
curl -fsSL -o /usr/local/lib/systemd/system/containerd.service \
  "https://raw.githubusercontent.com/containerd/containerd/v${VER}/containerd.service"
systemctl daemon-reload

# 4. Put the new configuration in place - the hand-written one, not the
#    318-line dump - and prove it parses before anything restarts.
install -m 0644 /tmp/config.new.toml /etc/containerd/config.toml
containerd --config /etc/containerd/config.toml config dump >/dev/null

# 5. Bring it back, runtime first, kubelet second.
systemctl start containerd
sleep 3
ctr plugins ls | awk '$4!="ok"'          # must print only the header
systemctl start kubelet

# 6. Then verify from the cluster's point of view before uncordoning:
#    kubectl get node node-01 -o jsonpath='{.status.nodeInfo.containerRuntimeVersion}'
#    kubectl uncordon node-01

关于顺序有两点，赶时间的时候特别容易搞错。开始之前先备份 /var/lib/containerd，不是因为升级大概率会弄坏它，而是因为它就是镜像和快照的存储：丢了它意味着这个节点上所有镜像都要重新拉，在大节点上这是几十分钟加上一大笔出网流量。另外，先把运行时拉起来再拉 kubelet，并且在启动 kubelet 之前先用 ctr plugins ls 检查有没有插件不处于 ok 状态。一个没加载成功的 CRI 插件，等于一个会接收 Pod 却创建不了的节点。^[runc][cni]

同一件事从 Kubernetes 这一侧看

从集群这一侧看，没有事要做，但有一件事要查。支持矩阵是项目文档，不是准入策略：没有哪个 kubelet 会因为 containerd 版本不受支持而拒绝启动，也不会在任何地方冒出一条事件。所以审计必须是显式的——遍历节点，读 containerRuntimeVersion，再和同一节点上 kubelet 版本对应的矩阵行去比。节点池版本参差是常态而不是例外，尤其是在节点镜像按自己的节奏推进的环境里。^[k8sruntime]

# The support matrix is a project document, not a runtime check: nothing stops
# a kubelet from talking to an unsupported containerd. That is precisely the
# problem - you find out from a bug, not from a startup error. So audit it.

kubectl get nodes -o json | jq -r '
  .items[] | [.metadata.name,
              .status.nodeInfo.kubeletVersion,
              .status.nodeInfo.containerRuntimeVersion] | @tsv' \
| while IFS=$'\t' read -r node kubelet runtime; do
    ctd=${runtime#containerd://}
    case "${kubelet%.*}/${ctd%%.*}" in
      v1.36/1|v1.37/1) verdict='NOT LISTED - upgrade the runtime' ;;
      *)               verdict='check against containerd.io/releases' ;;
    esac
    printf '%-22s kubelet=%-9s containerd=%-9s %s\n' \
           "$node" "$kubelet" "$ctd" "$verdict"
  done

# Carry the cgroup driver across explicitly. Kubernetes 1.28 added the ability
# for the kubelet to read it from the CRI runtime instead of its own config file
# - but that arrived as an alpha feature behind the KubeletCgroupDriverFromCRI
# gate, and on the containerd side it needs 2.0 or later. Either way the
# containerd setting has to be right, so do not treat it as automatic:
grep -rn 'SystemdCgroup' /etc/containerd/config.toml
grep -E '^cgroupDriver:' /var/lib/kubelet/config.yaml

# Runtime handlers are the part most often lost, because they are the part
# somebody added by hand. Every handler referenced by a RuntimeClass must still
# exist in the rewritten configuration:
kubectl get runtimeclass -o jsonpath='{range .items[*]}{.handler}{"\n"}{end}' \
  | sort -u | while read -r h; do
      grep -q "runtimes\.${h}\b" /etc/containerd/config.toml \
        && echo "ok      $h" || echo "MISSING $h"
    done

# And the sandbox image. It moved from sandbox_image to pinned_images.sandbox,
# and if you had it pointed at an internal mirror, that is a setting to carry
# across rather than a default to accept.
crictl info | jq -r '.config.sandboxImage // .config.containerd.sandboxImage'

这场迁移中最常丢掉的配置是 runtime handler，因为它恰好是最常被手工加上去的那一项。集群里每一个 RuntimeClass 都点名了一个 handler，它必须在重写后的配置里存在；如果不存在，只有请求它的负载会失败——也就是说故障范围被限制在那个用 gVisor 或 Kata 的团队里，其他人一周都不会察觉。按名字逐个对照新文件检查。cgroup driver 同理，而且这里那个听着让人安心的说法并不完全成立：Kubernetes 1.28 确实让 kubelet 可以向 CRI 运行时询问它用的是哪个 driver，但那是一个由 KubeletCgroupDriverFromCRI 特性开关控制的 alpha 功能，containerd 这一侧还要求 2.0 或更高版本。所以它并不是自动的，SystemdCgroup = true 仍然必须活过这次重写，而且它现在在另一个路径下。^{[k8srtc][k8skubeadm]}

怎么回滚，以及什么回不去

回滚这件事值得给一个直白的答案，而不是一个安慰性的答案。二进制文件和配置文件可以干净地回滚——两者都是磁盘上的文件，只要你留了旧的压缩包和旧的 config.toml，十分钟就能回到起点。这已经比大多数迁移能给的多了。^[ctrgh]

# Rolling back is realistic here, which is not true of every migration on this
# site - but only if you kept the two things that matter and only within
# limits. Know which of these applies before you start the window.

# --- what rolls back cleanly ---------------------------------------------
# The binaries and the configuration file. Both are files on disk.
systemctl stop kubelet containerd
tar -C /usr/local -xzf /var/backups/containerd-1.7.28-linux-amd64.tar.gz
cp -a /etc/containerd/config.toml.v2.bak /etc/containerd/config.toml
systemctl daemon-reload && systemctl start containerd kubelet
containerd --version

# --- what does not ---------------------------------------------------------
# 1. The image and snapshot store, in the sense that nobody promises it will.
#    containerd's stability document puts file system layout, storage formats
#    and snapshot formats explicitly OUTSIDE its guarantees and says the project
#    may migrate these formats between minor versions. A downgrade against a
#    store that 2.x has already written to is therefore undefined rather than
#    documented-as-broken. Restore the tarball instead of finding out:
#      systemctl stop containerd
#      mv /var/lib/containerd /var/lib/containerd.v2
#      tar -C /var/lib -xzf /var/backups/containerd-lib-2026-08-24.tgz
#    (Container root filesystems are maintained on upgrade; it is the metadata
#     around them that has no promise attached.)
#
# 2. A configuration file you already migrated. A version 4 file needs
#    containerd 2.3.0 or newer, and a version 3 file needs 2.0 or newer. This
#    is why the config.toml backup matters as much as the binary one, and why
#    writing version 3 rather than 4 keeps your options open for a while.
#
# 3. Nothing about the Kubernetes control plane. This is a node-level change:
#    do NOT roll the cluster back because one node's runtime misbehaved.

# The honest limit on all of this: rollback buys you a night, not a quarter.
# containerd 1.7 leaves extended support in September 2026, and that extension
# only ever covered Kubernetes 1.30, 1.31 and 1.32 on GKE - all three of which
# are already out of support upstream.

有两样东西没那么容易回去。第一样是状态目录，而实话是：没有人承诺过它能回去。containerd 的稳定性文档把文件系统布局、存储格式和快照格式明确排除在保证范围之外，并且写明项目可能在小版本之间迁移这些格式。所以对一个已经被 2.x 写过的 /var/lib/containerd 做降级，属于“未定义”，而不是“文档写明会坏”——这个区别不值得在凌晨三点拿生产去验证：要么恢复那份压缩包，要么接受把所有镜像重新拉一遍。第二样是配置文件本身，前提是你已经迁移过它：version 4 的文件需要 containerd 2.3.0 或更新的版本才读得动。更根本的一点值得说得直接些：回滚给你的是一个晚上，不是一个季度。1.7 分支在 2026 年 9 月离开延长支持，而在那之前它也不会为某家厂商托管服务之外的任何东西打补丁。回滚是把一个搞砸的维护窗口安全收尾的办法，不是把决定往后推的办法。^[ctrsec]

去验证，而不是去指望

验证不是风格问题，而且在这场迁移里它有一个特定的形状：几乎所有出错的情况都会让守护进程继续跑着。所以“检查 containerd 是否在运行”什么也证明不了。下面的脚本检查那些可能悄悄出错的东西——配置版本、有没有插件处于错误状态、CRI 是不是在 v1 上应答、cgroup driver 和 runtime handler 有没有活下来——然后做两件光靠检查完全确认不了的事。^[ctrcrictl]

#!/usr/bin/env bash
# Post-upgrade verification. Every check prints OK or explains itself; the exit
# code is the number of failures, so this can run straight from your config
# management after the node comes back.
fail=0
chk() { if eval "$2" >/dev/null 2>&1; then printf 'OK   %s\n' "$1";
        else printf 'FAIL %s\n' "$1"; fail=$((fail+1)); fi; }

chk 'containerd is 2.x'      'containerd --version | grep -qE " v2\."'
chk 'config is version 3+'   'grep -qE "^version = [34]$" /etc/containerd/config.toml'
chk 'no in-memory migration' '! journalctl -u containerd -b | grep -q "Configuration migrated from version"'
chk 'config parses'          'containerd --config /etc/containerd/config.toml config dump'
chk 'no plugin in error'     '[ "$(ctr plugins ls | awk "NR>1 && \$4!=\"ok\"" | wc -l)" -eq 0 ]'
chk 'CRI answers on v1'      'crictl version | grep -q "RuntimeApiVersion:  v1"'
chk 'runtime handler runc'   'containerd config dump | grep -q "runtimes.runc"'
chk 'systemd cgroup driver'  'containerd config dump | grep -q "SystemdCgroup = true"'
chk 'no legacy shims'        "! containerd config dump | grep -qE 'io\.containerd\.runtime\.v1\.linux|io\.containerd\.runc\.v1'"
chk 'mirrors not set'        "! containerd config dump | grep -q 'registry.mirrors'"
chk 'sandbox image pinned'   'containerd config dump | grep -q "pinned_images"'
chk 'kubelet is running'     'systemctl is-active --quiet kubelet'

# The two that are worth reading rather than counting. First: registry
# resolution has to be exercised, not inspected - a hosts.toml with the wrong
# directory name looks perfectly fine and simply never matches.
crictl pull registry.k8s.io/pause:3.10.2 >/dev/null 2>&1 \
  && echo 'OK   pull through the configured hosts' \
  || { echo 'FAIL pull through the configured hosts'; fail=$((fail+1)); }

# Second: the deprecation list should be shorter than it was before, not
# longer. A new entry here is something the migration introduced.
ctr deprecations list --format json 2>/dev/null | jq -r '.[].id' | sed 's/^/     still deprecated: /'

# And a real workload, because none of the above proves a container starts.
ctr run --rm docker.io/library/alpine:3.22 verify-"$$" /bin/true \
  && echo 'OK   container runs' \
  || { echo 'FAIL container runs'; fail=$((fail+1)); }

printf '\n%d failure(s)\n' "$fail"; exit "$fail" 

这两件事是一次真实的拉取和一个真实的容器。一个放在名字与仓库主机命名空间不完全一致的目录里的 hosts.toml，看起来完全正确，却永远匹配不上，你把文件读一百遍也看不出来；只有拉一次才看得出来。另外，一份在各方面都合法的配置，仍然可能起不了容器，因为运行时二进制版本不对，或者不在 shim 期望的路径上。这两件事都在第一个节点上跑完，再动第二个节点。^[critools]

这件事该按什么顺序做

压缩之后，这个决定比这篇文章小得多。目标版本只有一个，就是 2.3——当前的 LTS，支持到 2028 年 4 月。表上其余的要么已经没有支持，要么几个月内就没有支持，要么是一个为别人的托管服务量身定制的延长支持分支。真正的工作量不是换二进制，那只要十五分钟；真正的工作量是配置重写，而其中的核心是镜像仓库改造。^[ctrrel]

1. 先盘点，再计划。逐节点记录守护进程版本、配置文件版本、CRI API 版本和 kubelet 版本。然后在全部节点上跑 ctr deprecations list --format json 并保存输出——那就是“你已经知道有问题”的清单。
2. 手写配置，把转换器当字典用。跑 containerd config migrate 学新的键名，然后自己写一份简短的 version 3 文件。不要直接安装那份三百行的 dump：它会把你从没选过的默认值全部钉死。
3. 镜像仓库改造要先做、单独做。建好 certs.d 目录树，用 ctr images pull --hosts-dir 验证，并确认 mirrors 和 config_path 绝不会出现在同一个文件里。这一步就是会造成故障的那一步。
4. 对变动的默认值做决定，而不是被动继承。非特权端口、NRI、CDI 以及 io_uring 的 seccomp 变更都与安全相关。选定、写下来、放进镜像构建。
5. 先一个节点，再一个节点池，最后全量。备份 /var/lib/containerd、升级、跑验证脚本、解除 cordon。之后才把改动写进节点镜像——并且把 RuntimeClass 依赖的 runtime handler 检查一遍，因为不会有别的东西替你检查。

这是同一年落在同一批节点上的四项变更之一，合起来做比分开做便宜：从 cgroup v1 迁移到 cgroup v2，因为 cgroup driver 的设置必须活过两次重写，而 kubelet 现在直接向运行时询问它；Docker Engine 29 的破坏性变更，同一套容器底座从 Docker 那一侧看过去；以及 从 ingress-nginx 迁移到 Gateway API，如果这批节点本来就要为入口层的变更重建。如果你在权衡这些东西到底需要多少，什么时候不该用 Kubernetes 是这个论证的另一面。

常见问题

containerd 1.7 还有支持吗？

只在很窄的意义上还有。containerd 的版本表把 1.7 列为 LTS 至 2026 年 9 月，但关键在脚注：committer 提供的常规支持已于 2026 年 3 月结束，延长部分由两位具名维护者提供，并且面向 Kubernetes 1.32、1.31、1.30 在 Google Kubernetes Engine 上的使用场景，与该场景无关的改动可能被拒绝。这三个 Kubernetes 版本在上游都已停止维护。如果你不是在 GKE 上跑一套已停止维护的 Kubernetes，那就把 1.7 当作今天就已经没有支持，而不是等到九月。

应该升级到 containerd 2.x 的哪个版本？

2.3。它是当前的 LTS 分支，2026 年 4 月 30 日开始，支持到 2028 年 4 月 30 日。其他选项各有各的问题：2.1 已于 2026 年 7 月 3 日停止维护，2.2 只支持到 2026 年 11 月 6 日，2.0 则和 1.7 一样处在面向特定厂商的延长支持之下。如果你现在在 2.1 或 2.2 上，到 2.3 的路很短，因为配置已经是 version 3 了。

必须重写 config.toml 吗，还是 containerd 2.x 会读旧文件？

它会读 version 2 的文件，并在每次启动时在内存里转换，所以没有东西强迫你——而且它每转换一次就记一条 Configuration migrated from version 2, use `containerd config migrate` to avoid migration，这是全机群审计最快的办法。仍然建议重写的理由是，兼容路径恰恰是已知问题所在，尤其是镜像仓库迁移会在你的 mirrors 块旁边塞进一个 config_path，导致 CRI 插件加载不了。用 containerd config migrate 学新键名，然后手写一份简短的文件。版本要有意识地选：version 3 从 containerd 2.0 起都能读，version 4 需要 2.3.0 或更新的版本，而在 2.3 上 migrate 输出的就是 version 4。

什么是配置 version 4，我需要它吗？

version 4 随 containerd 2.3 到来。它对 CRI 没有任何改动——这场迁移里绝大部分内容围绕的那次插件拆分，是 version 3 的事，随 2.0 引入。version 4 做的是把守护进程自己的 socket 从顶层的 [grpc]、[ttrpc]、[metrics]、[debug] 表里挪出来，放进 io.containerd.server.v1.grpc、…v1.ttrpc、…v1.metrics 和 …v1.debug 这几个插件块；[debug] 的 level、format 和 log_trace_id 仍然留在顶层。有一处行为变化很容易漏掉：在 version 4 之前，没有显式设置的 ttrpc 地址是从 gRPC 地址推导出来的，形如 <grpc address>.ttrpc，并继承它的 uid 和 gid；而在 version 4 里，ttrpc 插件用的是它自己的默认值。你并不需要 version 4，而且有理由暂时继续用 version 3：version 4 的文件在 2.0、2.1、2.2 上都读不了，写了它就等于收窄了自己的回滚余地。

“`mirrors` cannot be set when `config_path` is provided”是什么意思？

意思是 CRI 镜像服务插件拒绝加载，因为镜像仓库配置里最终同时出现了已废弃的 mirrors 表和更新的 config_path。containerd 仍然会启动——守护进程健康，插件不健康——而所有调度到该节点的 Pod 都会创建失败。坑在于这两个键未必都是你写的：该问题是针对 containerd 2.2.0 报告的，用的是一份只含 registry.mirrors 块的普通 version 2 文件，因为内存里的迁移会在旁边补上默认的 config_path。所以你去 grep 自己的文件什么也找不到。修复在 pull request 12617 中，它在 2.3.0 之前合入，并被回合到 2.2，因此当前的 2.3 和较新的 2.2 补丁版不受影响，2.0 和 2.1 则会中招。无论哪种情况，长期有效的修法都是让两者只留其一：建一棵 certs.d 目录树，把 config_path 指过去，然后删掉 mirrors 块。

镜像仓库的 mirrors 块被什么取代了？

一棵目录树。在 [plugins.'io.containerd.cri.v1.images'.registry] 下设置 config_path——习惯上是 /etc/containerd/certs.d——然后为每个仓库主机命名空间建一个子目录，各放一个 hosts.toml。每个文件写明一个 server 和一个或多个带显式 capabilities 的 [host."…"] 条目，按顺序尝试，因此当某个镜像源没有对应层时可以回落到上游仓库。每个仓库自己的 CA 和客户端证书就是同一个文件里的键。目录名必须与主机命名空间完全一致，包括端口——这是“配置看起来没错却永远匹配不上”最常见的原因。

sandbox_image 去哪了？

它变成了 [plugins.'io.containerd.cri.v1.images'.pinned_images] 下的 sandbox。任何一次重写之后都值得手工核对这一项，因为故障是延迟出现且与环境相关的：能上外网的节点会安安静静地从 registry.k8s.io 拉 pause 镜像，一切看起来正常；而离线或者出网受限的节点则一个 Pod 都创建不了。如果你原本把它指向内部镜像源，请显式把这个值搬过去。

我只是调高了下载并发，为什么拉取行为变了？

因为从 containerd 2.1 起，CRI 插件默认通过 Transfer Service 拉取，而 Transfer Service 不会从 CRI 镜像配置里读 max_concurrent_downloads。当 containerd 发现某个设置 Transfer Service 无法照办时，它会为该节点设上 use_local_image_pull = true，记一条警告，然后继续运行。完整的触发列表是 Registry.Mirrors、Registry.Configs、Registry.Auths、不等于 3 的 MaxConcurrentDownloads、DiscardUnpackedLayers、ImagePullWithSyncFs 以及 DisableSnapshotAnnotations = false。想调高并发又不换代码路径，就把它写到 [plugins.'io.containerd.transfer.v1.local'] 下面。

我的老镜像会用不了吗？

只有 Docker schema 1 的镜像，而且只影响拉取——但要去核对版本号，因为这件事上很多文章都落后了一个版本。该支持在 containerd 2.0 中默认禁用，当时环境变量 CONTAINERD_ENABLE_DEPRECATED_PULL_SCHEMA_1_IMAGE=1 还能把它打开；在 2.1 中被移除，此后什么都打不开它。既然目标是 2.3，就没有后门可走：这些镜像必须重新构建。升级前先找出来：从 containerd 1.7.8 和 1.6.25 起，从 schema 1 转换来的镜像会带上 io.containerd.image/converted-docker-schema1 标签，用 ctr image list 加这个标签可以在所有 namespace 里找到它们。修法是用 schema 2 或 OCI 重新构建，而这些镜像通常老到“找到 Dockerfile”才是最难的一步。

能从 containerd 2.x 回滚到 1.7 吗？

二进制可以干净地回滚——留好旧的压缩包，十分钟就能完成。有两件事会让它复杂起来。首先是状态目录：containerd 的稳定性文档把文件系统布局、存储格式和快照格式明确排除在保证范围之外，并且写明项目可能在小版本之间迁移这些格式，所以对一个已经被 2.x 写过的 /var/lib/containerd 做降级，属于“未定义”，而不只是“有风险”。要么恢复升级前做的那份压缩包，要么接受把节点上所有镜像重新拉一遍。其次是配置文件：如果你已经迁移过它，version 4 的文件需要 2.3.0 或更新的版本，version 3 的文件需要 2.0 或更新的版本，所以把 version 2 的原件留着。把回滚当作把一个搞砸的维护窗口收尾的手段，而不是推迟迁移的手段：1.7 在 2026 年 9 月离开延长支持。

如果 containerd 是不受支持的版本，kubelet 会停止工作吗？

不会，而这正是需要显式审计的原因。Kubernetes 与 containerd 的支持矩阵表达的是“两个项目测过哪些组合”，不是准入策略：kubelet 里没有任何东西会检查运行时版本，不会产生事件，不受支持的组合照样启动、看起来照样能用。你失去的是那些测试：对 Kubernetes 1.36，矩阵只列了 containerd 2.3.0+ 和 2.2.0+，完全没有 1.x 条目，所以在 1.36 集群上跑 1.7 节点，是一个没有任何人替你验证过的组合。自己遍历节点、读 containerRuntimeVersion、逐一对照。

同一批节点上的 Service 数据面走的是另一张时间表：Kubernetes 1.37 把 kube-proxy 的 ipvs 模式放进 feature gate 标记为弃用，1.40 默认关闭，1.43 直接删代码。把 kube-proxy 从 IPVS 切到 nftables讲的是 5.13 的内核门槛、会在你不知情时改变的 NodePort 行为，以及没人清理就会吞掉流量的残留 kube-ipvs0。

补一条版本层面的说明，因为 1.37 的账本和大多数报道说的并不一样：真正可能让 Pod 卡在 ContainerCreating 的是 SELinuxMount 转为 GA；cgroup v1 的失败在 1.35 落地，静态 Pod 的限制在 1.34，而 containerd 的悬崖还在前面的 1.38。升级到 Kubernetes 1.37 时真正会坏的东西把这三栏拆开，并给出应当在升级之前、而不是之后执行的排查。

参考资料

先放一手资料。containerd 的版本策略和它的 2.0 迁移文档，是关于“删了什么”“某个分支打补丁到什么时候”唯一有权威性的说法；CRI 配置指南是改名键位唯一完整的清单。凡是本文说了而二手资料没说的地方——比如 2.1 已经停止维护、1.7 的延长支持只针对本身也已停止维护的 Kubernetes 版本——分歧针对的是那些摘要，而不是项目本身。

1. containerd - Versioning and release (RELEASES.md): the release-status table quoted throughout this article, including the end-of-life dates for 1.7, 2.0, 2.1, 2.2 and 2.3, the footnotes explaining that 1.7's and 2.0's extended support is scoped to specific Kubernetes versions on GKE, the Kubernetes/containerd support matrix, the daemon-configuration version table (version 3 needs 2.0, version 4 needs 2.3), the deprecation tables with their removal targets, the upgrade-path rules, and the "Not Covered" section that places storage and snapshot formats outside the stability guarantee
2. containerd RELEASES.md on GitHub - the same document at its source, which is worth reading directly because the rendered docs site and the repository occasionally disagree with the older containerd-2.0 transition page (config version 1, and the deprecation release of the cri-containerd bundles, are two places where they do)
3. containerd source, version/version.go on release/2.3: `const ConfigVersion = 4`. This is what `containerd config migrate` targets, and the reason a migrated file on 2.3 comes back as version 4 rather than version 3
4. containerd source, the config migration table and serviceMigrate: the function that moves the top-level [grpc], [ttrpc], [metrics] and debug socket settings into io.containerd.server.v1.* plugins for version 4, including the note that an unset ttrpc address is no longer derived from the grpc address
5. containerd 2.0 - what's new, what's breaking, what's changing: the single authoritative list of removals (CRI v1alpha2, the AUFS snapshotter, the runtime v1 shims, LimitNOFILE, the cri-containerd release bundles), the default flips, and the deprecation of the CRI registry properties
6. CRI Plugin Config Guide - config versions 1, 2 and 3 side by side, the renamed plugin IDs, the full annotated default configuration, and the table of which image-pull options the Transfer Service does and does not support
7. containerd - CRI registry configuration: how the deprecated mirrors, configs and auths properties map onto a certs.d directory tree, which is the conversion this migration actually turns on
8. containerd - Registry Configuration (hosts.toml): the host-namespace directory layout under config_path, the capabilities key, and the per-host CA and client-certificate settings
9. containerd-config(8): the manual page. Note that it documents only the `default` subcommand - `dump` and `migrate` exist in the binary but not in this page, which is why so few people know about them
10. containerd-config.toml(5): the daemon configuration file itself - the version header, the plugins table, imports, and the state and root directories
11. containerd source, cmd/containerd/command/config.go: the definition of `containerd config default`, `dump` and `migrate`. `migrate` and `dump` share one implementation, which is why the migrated file comes back fully populated with defaults instead of as a minimal diff
12. containerd source, ctr deprecations: the subcommand is `deprecations` (plural), it takes --format json, and it sets CONTAINERD_SUPPRESS_DEPRECATION_WARNINGS while it runs. Some documentation writes it in the singular, which does not exist
13. containerd issue 12612 - the CRI plugin fails to load when a version 2 config with registry.mirrors is migrated, because the result carries both config_path and mirrors: "`mirrors` cannot be set when `config_path` is provided". Reported against 2.2.0
14. containerd pull request 12617 - the fix for the migration that emitted both config_path and mirrors. Worth checking against the exact patch release you are installing rather than assuming
15. containerd - Plugins: the plugin model behind the renamed IDs, and the distinction between built-in, proxy and binary external plugins that matters if you still load Go plugin libraries from plugin_dir
16. containerd - Ops: running the daemon, the systemd unit, the socket and state directories, and the configuration import mechanism
17. containerd - Getting started: the officially supported installation route now that the cri-containerd bundles are gone, which is containerd, runc and the CNI plugins installed as three separate components
18. containerd - Transfer service: the stable API that the CRI plugin uses for image pull by default from 2.1 onwards, and the reason a handful of registry settings now behave differently
19. containerd - Snapshotters: overlayfs as the default and the replacement for the removed AUFS snapshotter, plus the blockfile, devmapper and erofs alternatives
20. containerd - NRI, the Node Resource Interface: enabled by default from 2.0, which means access to the NRI socket is now part of your node's security surface whether or not you use it
21. containerd - user namespaces in CRI: supported from 2.0 and requiring runc v1.2.0 or later, which is one of the reasons the runtime binary needs upgrading alongside the daemon
22. containerd - CRI plugin architecture: how the kubelet, the CRI plugin, the snapshotters and the shims fit together, which is the mental model the renamed plugin IDs now reflect
23. containerd - crictl: the CRI-level debugging tool, and the right way to confirm that the kubelet's view of the runtime matches yours
24. containerd releases on GitHub: the binary tarballs, the checksums and the per-release notes. Also the place to confirm that the cri-containerd-(cni-)VERSION-OS-ARCH.tar.gz bundles really are gone rather than moved
25. containerd - Security and audits: the project's security policy and advisory history, which is the argument for not staying on a branch that only accepts patches for someone else's managed service
26. containerd pull request 8924 - the discussion behind removing the explicit LimitNOFILE from the reference systemd unit, including why hosts on systemd older than 240 must set it back by hand
27. Kubernetes - Releases: the supported branches and their end-of-life dates. This is what turns containerd's extended-support footnotes into a dead end, because the Kubernetes versions they name are already out of support
28. Kubernetes - Container runtimes: installing and configuring containerd for a cluster, including the cgroup driver requirement and the sandbox image setting
29. Kubernetes - Container Runtime Interface: the API the kubelet speaks, and the reason the removal of CRI v1alpha2 in containerd 2.0 is a compatibility statement rather than an implementation detail
30. Kubernetes - Configuring a cgroup driver: the kubelet side of the SystemdCgroup setting that has to be carried across when the containerd configuration is rewritten
31. Kubernetes - Runtime Class: the resource that maps a pod onto one of the runtime handlers defined in the containerd configuration, which is the part of the config most likely to be hand-written and therefore most likely to be lost in a migration
32. Kubernetes - User namespaces for pods: one of the capabilities that only exists once the node is on containerd 2.x with a recent enough runc
33. Kubernetes - Pull an image from a private registry: the ImagePullSecrets mechanism that replaces the deprecated registry.auths block in the containerd configuration
34. Kubernetes - Upgrading kubeadm clusters: the drain, upgrade, uncordon sequence this migration slots into, and the reminder that node components are upgraded one node at a time
35. runc releases: the OCI runtime that has to be installed separately now that the combined containerd bundles are gone, and whose version gates CRI user namespaces
36. CNI plugins releases: the third component of the install, previously bundled in cri-containerd-cni-*.tar.gz and now shipped on its own
37. cri-tools: crictl and critest, the CRI-level client used throughout this article to verify that the runtime is answering on v1 and that images and pods survived the upgrade
38. Container Device Interface: the specification behind enable_cdi and cdi_spec_dirs, both enabled by default from containerd 2.0
39. Google Security Blog - learnings from the kCTF VRP: the exploit history that led to io_uring_enter, io_uring_register and io_uring_setup being dropped from containerd's default seccomp allowlist in 2.0
40. OCI/Docker image manifest version 2, schema 2: the format that replaced the Docker schema 1 manifests whose pull support is disabled by default from containerd 2.0
41. systemd.exec(5) - LimitNOFILE and the rest of the resource limits a unit inherits, which containers then inherit from containerd. Relevant because the reference unit stopped setting it explicitly