Skip to content
← Blog

containerd 1.x has run out of road.

containerd 1.7 leaves extended support in September 2026, and that extension only ever covered Kubernetes versions that are already end of life. Here is the version-3 config rewrite, the registry conversion that breaks clusters, and the one branch worth upgrading to.

·26 min read
  • containerd
  • Kubernetes
  • Containers
  • Linux

There is a version of this migration that reads like a chore and a version that reads like a deadline, and which one you are in depends on a footnote. containerd's release table shows the 1.7 branch as LTS until September 2026, which sounds like room to breathe. The footnote underneath says that since March 2026 that support has been provided by two named maintainers, and that it is focused on usage with Kubernetes 1.32, 1.31 and 1.30 via Google Kubernetes Engine — with changes that are not needed for that usage liable to be refused. Kubernetes 1.32 went end of life in February 2026. If you are not running an out-of-support Kubernetes on GKE, the lifeline you thought you had was never pointed at you.

Three-panel cover image. The left panel, headed "containerd 1.x", shows a single configuration block labelled io.containerd.grpc.v1.cri holding runtimes, snapshotter, sandbox_image and a registry mirrors table, with a red end-of-life stamp reading "1.7 — extended support ends September 2026". The centre column, headed "what moves", lists five conversions: version 2 to version 3, io.containerd.grpc.v1.cri split into cri.v1.runtime and cri.v1.images, sandbox_image becoming pinned_images.sandbox, registry.mirrors becoming a certs.d directory with config_path, and cri-containerd bundles becoming three separate installs. The right panel, headed "containerd 2.3 LTS", shows two configuration blocks side by side, one for runtime and one for images, above a support bar running to April 2028.
What actually moves: one configuration version, one plugin split into two, one setting renamed, one registry block replaced by a directory tree, and one bundled download replaced by three separate ones.

So this is a migration with a real date on it, and it deserves more than a version bump in a playbook. What follows is the whole of it: how to read containerd's release policy without being misled by the word LTS, why 2.1 is the worst possible landing spot, the configuration rewrite from version 2 to version 3 with the plugin IDs that moved, the registry conversion that has its own bug and its own class of outage, the image-pull path that quietly changed in 2.1, everything 2.0 removed, a per-node runbook, an honest account of what rollback can and cannot recover, and a verification script that exits non-zero when something is wrong.

The failure modes, and why none of them says containerd

Nothing about this failure set announces itself as a runtime version problem, which is why it tends to be diagnosed late. A node comes back after an upgrade and the CRI plugin is simply not there — the daemon is running, systemctl status is green, and every pod on the node is stuck in ContainerCreating. An image that has pulled from the internal mirror for four years starts pulling from Docker Hub instead, and the first sign is the egress bill. A RuntimeClass that a team added by hand two years ago stops resolving, and only the workloads that use it fail. In each case the runtime is running and the runtime is wrong.[ctrrel]

What you seeWhat it usually meansWhere it is dealt with
Every pod on a node stuck in ContainerCreating, daemon healthyThe CRI plugin failed to load. containerd starts anyway and reports the failure only in its own logRegistries
Images suddenly pulling from the upstream registry, not the internal mirrorThe registry.mirrors block did not survive the rewrite into config_pathRegistries
The pause container pulls from registry.k8s.io on an air-gapped nodesandbox_image was not carried across to pinned_images.sandboxPlugin split
Only the gVisor or Kata workloads fail; everything else is fineA runtime handler named by a RuntimeClass is missing from the new configurationKubernetes
An old image that pulled last week now fails with a manifest errorDocker schema 1 pulling: disabled by default in containerd 2.0, removed outright in 2.1Removals
Image pulls behave differently after a change nobody connected to pullingA setting the Transfer Service cannot honour flipped the node back to local pullImage pull
The daemon logs "Configuration migrated from version 2" on every bootThe file was never rewritten. The compatibility shim is carrying it, and that is where the registry bug livesConfiguration

The common thread is that containerd 2.x is deliberately tolerant of a version 2 configuration file: it reads it, converts it in memory, and starts. That is a kindness at upgrade time and a liability afterwards, because it means the migration can be half-done indefinitely and nothing forces the issue. The daemon logs its complaints and starts anyway; the plugin that failed to load reports an error status that nothing surfaces to you. The whole of this article is, in one sense, an argument for finishing the migration rather than leaving the shim to carry it.[ctr20]

Read the support table properly

Start with the release table, because it is the only document that decides anything and it is routinely read wrong. containerd runs two kinds of branch. A regular release is supported for eight months. One release a year is designated LTS and supported for at least two years. On top of that, individual branches can be given extended support by named maintainers after the general support window closes — and that is a different thing wearing the same label in the same column.[ctrrel]

BranchStatusEnd of lifeWhat it means for you
1.6End of life23 August 2025Unsupported for a year. Nothing is coming, including security fixes
1.7LTS, extendedSeptember 2026Extended support only, from two named maintainers, scoped to Kubernetes 1.30–1.32 on GKE
2.0LTS, extendedMarch 2027Same shape: extended support scoped to Kubernetes 1.33 on GKE, which is itself end of life
2.1End of life3 July 2026Already gone. The version many people upgraded to first, and the worst place to stop
2.2Active6 November 2026Patched, but ten weeks of runway. Fine as a staging post, wrong as a destination
2.3LTS30 April 2028The target. Current long-term branch, nearly two years of support ahead of it
2.4Futuretentatively April 2027A regular eight-month release. Not a substitute for the LTS

Now overlay the Kubernetes support matrix, which is where the two projects meet. containerd publishes a recommended-version list per Kubernetes minor release. For Kubernetes 1.36 that list reads 2.3.0+, 2.2.0+ — and there is no 1.x entry at all. Nothing in the kubelet enforces this: an unsupported pairing will start, run, and appear to work, right up until it does not, and then you are debugging alone. The matrix is a statement about what has been tested, and testing is the only thing standing between you and a runtime bug nobody else has seen.[k8srel]

Kubernetescontainerd versions listed as recommendedKubernetes end of lifeReading
1.332.1.0+, 2.0.4+, 1.7.24+, 1.6.36+28 June 2026Already out of support. This is the pairing containerd 2.0's extension names
1.342.1.3+, 2.0.6+, 1.7.28+, 1.6.39+27 October 2026Two months left, and two of the four containerd options — 2.1 and 1.6 — are themselves EOL
1.352.2.0+, 2.1.5+, 1.7.28+28 February 2027The last row where a 1.x runtime appears at all
1.362.3.0+, 2.2.0+28 June 2027No 1.x entry. This is the line where the migration stops being optional

The trap inside the trap is containerd 2.1. It was the obvious place to land for anyone who upgraded in the second half of 2025, it is still what a lot of internal documentation says, and it reached end of life on 3 July 2026 — before 2.2, which runs to November 2026, and long before 2.3, which is the current LTS and is supported to April 2028. "Move to 2.x" is not a plan. There is exactly one branch worth targeting from a standing start, and it is 2.3.

What is actually installed on these nodes

Before touching anything, establish what is actually installed, because in a fleet of any size the answer is not one version. Three things matter and they are independent: the daemon version, the configuration file version, and the CRI API version the kubelet is actually being served. The configuration version is the one people forget, and it is the one that can be absent — a file with no version line is treated as a version 1 file. Here the project's own documents disagree with each other, and it is worth knowing which way: the CRI configuration guide says version 1 was removed in containerd 2.0, while RELEASES.md says an absent version is parsed as version 1 and that all prior versions are supported by migration, and the source still ships a v1 migration function. Treat a version 1 file as something to fix on sight rather than something you can reason about with confidence.[cfgtoml]

# The daemon, the client and the shim are three separate versions and they are
# allowed to disagree. Ask all three rather than assuming.
containerd --version
# containerd github.com/containerd/containerd/v2 v2.3.2 <revision>
ctr version                        # client and server, side by side
runc --version                     # the OCI runtime is a separate install now

# The configuration version is the single most useful number here. There is no
# `version` line in very old files: absent means version 1, which containerd
# 2.0 removed outright rather than migrating.
head -1 /etc/containerd/config.toml
# version = 2

# What the plugins are doing. A plugin in state "error" is the daemon telling
# you a migration went wrong; it does not stop the daemon from starting.
ctr plugins ls | awk '$4!="ok"'
# TYPE  ID  PLATFORMS  STATUS

# From the Kubernetes side, which is what actually matters:
kubectl get nodes -o custom-columns=\
'NODE:.metadata.name,RUNTIME:.status.nodeInfo.containerRuntimeVersion,'\
'KUBELET:.status.nodeInfo.kubeletVersion,OS:.status.nodeInfo.osImage'
# NODE     RUNTIME                KUBELET   OS
# node-01  containerd://1.7.28    v1.34.9   Ubuntu 24.04.3 LTS

# And the CRI API version the kubelet is really getting. containerd 2.0 removed
# v1alpha2; if anything on this node still speaks it, it stops working here.
crictl version
# RuntimeName:  containerd
# RuntimeApiVersion:  v1

Then ask the daemon what it has been trying to tell you. Since 1.6.27 and 1.7.12 containerd has exposed deprecation warnings through the introspection API, specifically so that this migration could be planned rather than discovered. The subcommand is ctr deprecations list — plural, which is worth saying because at least one official document writes it in the singular and that form does not exist. Run it with --format json across the fleet. It is not a clean bill of health, because warnings are emitted on use: a node that has not pulled a schema 1 image since its last restart will not report one. It is a starting list of things you know are wrong.[depsrc]

# containerd has been telling you what will break since 1.6.27 / 1.7.12, through
# the introspection API. Almost nobody reads it, because the warnings go into
# the daemon log rather than anywhere you look. Ask directly.
#
# Note the subcommand is `deprecations`, plural. Some documentation writes it
# in the singular; that form does not exist and returns a usage error.
ctr deprecations list
# ID                                          LAST OCCURRENCE   MESSAGE
# io.containerd.deprecation/pull-schema-1-image  2026-08-24...   Schema 1 image...
# io.containerd.deprecation/cri-registry-mirrors 2026-08-24...   `mirrors` is deprecated...

# Machine-readable, which is the form you want across a fleet:
ctr deprecations list --format json | jq -r '.[].id' | sort -u

# Run it on every node and count, rather than sampling. The warnings are
# emitted on use, so a node that has not pulled a schema 1 image since the last
# daemon restart will not report one - which is why this is a starting point
# and not a clean bill of health.
for n in $(kubectl get nodes -o name); do
  printf '%-22s ' "${n#node/}"
  kubectl debug "$n" -it --image=busybox --profile=general -- \
    chroot /host ctr deprecations list --format json 2>/dev/null \
    | jq -r '[.[].id] | join(",")' || echo '(unavailable)'
done

# Clean up afterwards. `kubectl debug node/...` names its pods
# node-debugger-<node>-<suffix> and applies no label of its own, so there is
# nothing to select on - match the name instead, or they accumulate silently.
kubectl get pods -n default -o name | grep '^pod/node-debugger-' | xargs -r kubectl delete

The configuration file, version 2 to version 3 — and now 4

The configuration file is the substance of the migration, and the first thing to establish is what "the latest version" means, because it moved this year. Version 3 arrived with containerd 2.0 and is the one that split the CRI plugin in two. Version 4 arrived with 2.3 and does something different, which the next section covers. Version 2 is still read and converted in memory on every start — the daemon logs a line saying so, which is the cheapest way to find out whether a node has actually been migrated or is merely tolerated. The daemon ships a converter, containerd config migrate, which reads your current file and prints the latest version to standard output. It is not in the manual page — containerd-config(8) documents only default — which is a large part of why so few people know it exists.[cricfg][cfgsrc]

# containerd 2.x reads a version 2 file and converts it in memory on every start.
# That is a compatibility shim, not a plan: it costs startup time, it is where
# the registry bug below lives, and the daemon says so on every boot.
journalctl -u containerd | grep -m1 'Configuration migrated from version'
# Configuration migrated from version 2, use `containerd config migrate` to
# avoid migration

# `containerd config migrate` reads your current file and prints the LATEST
# version on stdout. It is not in the man page - only `default` is - but it has
# been in the binary since 2.0.
#
# Note which version "latest" means, because it moved. Version 3 arrived in
# containerd 2.0 and is the one that split the CRI plugin in two. Version 4
# arrived in 2.3 and moves the server sockets into plugins (see below).
containerd config migrate > /tmp/config.new.toml
head -1 /tmp/config.new.toml
# version = 4        <- on containerd 2.3. On 2.0-2.2 this says 3.

# Two things to know before you trust the output. First, `migrate` and `dump`
# share one implementation, so the result is the FULLY POPULATED configuration,
# defaults and all - not a minimal file. Every default you did not choose is now
# pinned in your file and stops following the daemon when upstream changes it.
wc -l /etc/containerd/config.toml /tmp/config.new.toml
#   41 /etc/containerd/config.toml
#  318 /tmp/config.new.toml

# Second, and this is upstream's own warning: migrating the file to the latest
# version limits which containerd versions can read it. A version 4 file needs
# 2.3.0 or newer. If you might want to roll the binary back tonight, write a
# version 3 file instead - 2.0 and later read it, and it still gets you the
# plugin split, which is the part that matters.

# So: use the output to learn the new names, then hand-write the short version.
# What did it actually change? Compare the keys, not the files.
grep -oE '^\s*\[[^]]+\]' /tmp/config.new.toml | tr -d ' []' | sort > /tmp/new.keys
grep -oE '^\s*\[[^]]+\]' /etc/containerd/config.toml | tr -d ' []' | sort > /tmp/old.keys
diff -u /tmp/old.keys /tmp/new.keys

# Validate before you restart anything. `config dump` loads the file the daemon
# would load, including everything pulled in by `imports`, and fails loudly on
# a file it cannot parse. Note that --config is a global flag: it goes BEFORE
# the subcommand. Putting it after `config dump` is a usage error, not a check -
# urfave/cli rejects it with "flag provided but not defined: -config".
containerd --config /tmp/config.new.toml config dump >/dev/null && echo 'parses'

# Keep the old one. It is the fastest rollback you have.
cp -a /etc/containerd/config.toml /etc/containerd/config.toml.v2.bak

Two things about that converter are worth knowing before you pipe its output over your live configuration. First, migrate and dump are the same code path, so what comes back is the fully populated configuration: every default the daemon has, written out explicitly. A forty-line file becomes three hundred lines, and every default you did not choose is now pinned in your file and will stop tracking upstream when it changes. Use the output to learn the new key names, then hand-write the short version. Second, validate the candidate file before you restart anything — but note that --config is a global flag rather than a subcommand flag, so it goes before config dump, and putting it after is a usage error rather than a check. Run correctly, it loads the file the daemon would load, follows imports, and fails loudly on something it cannot parse, which is a much better place to discover a typo than a node that will not come back.[cfgman]

# Configuration version 4 (containerd 2.3 and later). It changes nothing about
# CRI: the whole of the plugin split above is version 3 work. What it moves is
# the daemon's own sockets, out of top-level tables and into server plugins.

# --- version 3 and earlier -------------------------------------------------
#   [grpc]
#     address = "/run/containerd/containerd.sock"
#     uid = 0
#     gid = 0
#   [ttrpc]
#     address = "/run/containerd/containerd.sock.ttrpc"
#   [metrics]
#     address = "127.0.0.1:1338"
#   [debug]
#     address = "/run/containerd/debug.sock"
#     level = "info"

# --- version 4 -------------------------------------------------------------
version = 4

[plugins.'io.containerd.server.v1.grpc']
  address = '/run/containerd/containerd.sock'
  uid = 0
  gid = 0

[plugins.'io.containerd.server.v1.ttrpc']
  address = '/run/containerd/containerd.sock.ttrpc'

[plugins.'io.containerd.server.v1.metrics']
  address = '127.0.0.1:1338'

[plugins.'io.containerd.server.v1.debug']
  address = '/run/containerd/debug.sock'

# `[debug]` does not disappear: level, format and log_trace_id stay at the top
# level. Only the socket fields move.
#
# One behaviour change hides in here. Before version 4, an unset ttrpc address
# was derived from the grpc address as "<grpc address>.ttrpc" and inherited its
# uid and gid. In version 4 the ttrpc plugin is independent and falls back to
# its own default. If anything of yours connects to that socket by path -
# a shim debugger, a monitoring agent - set it explicitly rather than assuming.

Version 4 is worth its own look, because almost every write-up on this migration stops at version 3 and because it carries a rollback constraint. It changes nothing about CRI: it moves the daemon's own sockets out of the top-level [grpc], [ttrpc], [metrics] and [debug] tables and into io.containerd.server.v1.* plugins. Two consequences follow. The behavioural one: before version 4, an unset ttrpc address was derived from the gRPC address as <grpc address>.ttrpc and inherited its uid and gid, while in version 4 the ttrpc plugin is independent and falls back to its own default — so anything of yours that connects to that socket by path should now set it explicitly. The operational one is upstream's own warning: migrating a file to the latest version limits which containerd versions can read it. A version 4 file needs 2.3.0 or newer; a version 3 file is read by 2.0 and later. If a same-night binary rollback is part of your plan, write version 3 — you still get the plugin split, which is the part that matters.[cfgver][srvmig]

One plugin became two, and the settings moved with it

The structural change is that the single CRI plugin was split in two. io.containerd.grpc.v1.cri used to hold everything; in version 3 it holds only the streaming server options, and the substance lives under two new IDs: io.containerd.cri.v1.runtime for everything about running containers — runtimes, CNI, sandboxes, SELinux, OOM handling — and io.containerd.cri.v1.images for everything about images: the snapshotter, the registry, the pinned sandbox image, pull concurrency. It is a better division than the one it replaces, and it means a mechanical find-and-replace of the plugin ID will put roughly half your settings in the wrong table.[ctrarch]

# /etc/containerd/config.toml - containerd 1.7, the file most clusters have.
# Everything lives under one plugin ID: io.containerd.grpc.v1.cri
version = 2

[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "registry.k8s.io/pause:3.10"

  [plugins."io.containerd.grpc.v1.cri".containerd]
    snapshotter = "overlayfs"
    default_runtime_name = "runc"

    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
      runtime_type = "io.containerd.runc.v2"
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
        SystemdCgroup = true

    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.gvisor]
      runtime_type = "io.containerd.runsc.v1"

  [plugins."io.containerd.grpc.v1.cri".cni]
    bin_dir = "/opt/cni/bin"
    conf_dir = "/etc/cni/net.d"

  # The block that causes the most trouble in this migration.
  [plugins."io.containerd.grpc.v1.cri".registry]
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
      endpoint = ["https://mirror.internal.example.com"]
# /etc/containerd/config.toml - containerd 2.x. The single CRI plugin has been
# split in two, and the settings moved with the split: anything about running
# containers is now under io.containerd.cri.v1.runtime, anything about images
# under io.containerd.cri.v1.images. io.containerd.grpc.v1.cri still exists,
# but only for the streaming server options.
version = 3

[plugins.'io.containerd.cri.v1.images']
  snapshotter = 'overlayfs'                 # moved: was under ...cri.containerd

  [plugins.'io.containerd.cri.v1.images'.pinned_images]
    sandbox = 'registry.k8s.io/pause:3.10.2'  # replaces sandbox_image

  [plugins.'io.containerd.cri.v1.images'.registry]
    config_path = '/etc/containerd/certs.d'   # replaces the mirrors block

[plugins.'io.containerd.cri.v1.runtime']
  [plugins.'io.containerd.cri.v1.runtime'.containerd]
    default_runtime_name = 'runc'

    [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
      runtime_type = 'io.containerd.runc.v2'
      [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
        SystemdCgroup = true

    [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.gvisor]
      runtime_type = 'io.containerd.runsc.v1'

  [plugins.'io.containerd.cri.v1.runtime'.cni]
    bin_dirs = ['/opt/cni/bin']   # bin_dir is deprecated since 2.1: plural now
    conf_dir = '/etc/cni/net.d'

# Note what is NOT here: no registry.mirrors alongside config_path. Setting
# both is an error - "`mirrors` cannot be set when `config_path` is provided" -
# and the CRI plugin refuses to load rather than picking one.
#
# And note the clock on the old keys. registry.mirrors and registry.configs
# were deprecated in containerd 1.5, registry.auths in 1.3, and cni.bin_dir in
# 2.1. All four carry the same removal target: containerd 2.4. Converting them
# is not housekeeping you can defer past the next release.
containerd 1.x (version 2)containerd 2.x (version 3)Note
version = 2version = 3 (2.0) / version = 4 (2.3)Version 2 is still read and converted in memory. A version 4 file needs 2.3.0 or newer
plugins."io.containerd.grpc.v1.cri"plugins.'io.containerd.cri.v1.runtime'Everything about running containers: runtimes, CNI, SELinux, OOM, sandboxes
plugins."io.containerd.grpc.v1.cri"plugins.'io.containerd.cri.v1.images'Everything about images: snapshotter, registry, pinned images, pull settings
plugins."io.containerd.grpc.v1.cri"plugins.'io.containerd.grpc.v1.cri'Still exists, but only for the streaming server options
sandbox_image = "…"pinned_images.sandbox = '…'Renamed and moved. Lose it and an air-gapped node reaches for registry.k8s.io
…cri".containerd.snapshotter…cri.v1.images'.snapshotterMoved across the split, from the runtime side to the images side
…cri".containerd.runtimes.*…cri.v1.runtime'.containerd.runtimes.*Path change only. runtime_type = io.containerd.runc.v2 is unchanged
…cri".registry.mirrors…cri.v1.images'.registry.config_pathDifferent mechanism. A directory of hosts.toml files, not a table. Removal target 2.4
…cri".registry.auths— (imagePullSecrets)No replacement, by design. Credentials move to the cluster. Removal target 2.4
…cri".cni.bin_dir…cri.v1.runtime'.cni.bin_dirsPlural, and a list. Deprecated in 2.1, removal target 2.4
plugin_dir (Go plugin .so)— (proxy or binary plugins)Already removed in 2.1, not merely deprecated

Two renames inside that split cause most of the damage. sandbox_image became pinned_images.sandbox, so a cluster that pointed its pause image at an internal mirror will silently revert to registry.k8s.io — which is fine until the day the node has no egress. And snapshotter moved from the runtime side to the images side, which is unintuitive enough that it is worth checking rather than assuming. Everything else in the table below is a path change rather than a behaviour change.[ctrplug]

Registries: the part that breaks clusters

Registry configuration is where this migration turns from tedious into risky, and it has a date on it: mirrors and configs were deprecated in containerd 1.5, auths in 1.3, and all three carry a removal target of containerd 2.4 — the release after the one this article recommends. The replacement for the first two is a directory tree: one subdirectory per registry host namespace under a single config_path, each containing a hosts.toml. It is more files and much less magic, and it is genuinely better — hosts are tried in order, capabilities are explicit, and a per-registry CA is a line in a file rather than a special case. The third property, auths, has no file replacement on purpose: credentials belong in a Kubernetes image pull secret, not in a node-level configuration that every workload on the node inherits.[crireg][hosts]

# The mirrors / configs / auths properties are deprecated. The replacement is a
# directory of hosts.toml files, one per registry host namespace, pointed at by
# a single config_path. It is more files and considerably less magic.

# Directory naming, which is where this goes wrong silently. containerd looks
# for the host namespace in three forms, in order:
#     <host>_<port>_        e.g. registry.internal.example.com_5000_
#     <host>:<port>         e.g. registry.internal.example.com:5000
#     _default
# The first form is the portable one - a colon is not a legal filename on
# Windows - so prefer it. A directory named anything else looks perfectly
# correct and simply never matches.

mkdir -p /etc/containerd/certs.d/docker.io
cat > /etc/containerd/certs.d/docker.io/hosts.toml <<'TOML'
server = "https://docker.io"

[host."https://mirror.internal.example.com"]
  capabilities = ["pull", "resolve"]

# Fall through to the real registry if the mirror does not have the layer.
# Order matters: hosts are tried top to bottom.
[host."https://registry-1.docker.io"]
  capabilities = ["pull", "resolve"]
TOML

# A private registry on a port, with its own CA:
mkdir -p /etc/containerd/certs.d/registry.internal.example.com_5000_
cat > /etc/containerd/certs.d/registry.internal.example.com_5000_/hosts.toml <<'TOML'
server = "https://registry.internal.example.com:5000"

[host."https://registry.internal.example.com:5000"]
  capabilities = ["pull", "resolve", "push"]
  ca = "/etc/containerd/certs.d/internal-ca.crt"
TOML

# registry.auths has no file equivalent, on purpose. Credentials belong in a
# Kubernetes imagePullSecret, not in the node's runtime configuration where
# every workload on the node inherits them.
kubectl create secret docker-registry regcred \
  --docker-server=registry.internal.example.com:5000 \
  --docker-username=ci --docker-password="$REG_PASSWORD"

# Verify resolution without restarting anything. --hosts-dir makes ctr read the
# same tree the CRI plugin will read.
ctr images pull --hosts-dir /etc/containerd/certs.d docker.io/library/alpine:3.22

# And confirm the daemon agrees once it has restarted:
containerd config dump | grep -A3 "cri.v1.images'.registry"
# config_path = '/etc/containerd/certs.d'

The bug is worth stating precisely, because the imprecise version of it sends you looking in the wrong place. It does not require you to have written anything odd. Start containerd 2.2.0 with an ordinary version 2 file that contains a registry.mirrors block and nothing else, and the in-memory migration adds the default config_path alongside it — and the CRI plugin rejects that combination outright: `mirrors` cannot be set when `config_path` is provided. The plugin does not load. The daemon starts anyway. Every pod scheduled to that node then fails to create while the top-level status of every service on the box stays green. Grepping your own file for both keys finds nothing, because you never wrote one of them. It was reported against 2.2.0 and fixed in pull request 12617, which landed before 2.3.0 and was backported to the 2.2 branch — so on a current 2.3 or a recent 2.2 patch you are not exposed, and on 2.0 or 2.1 you are. The instruction that survives either way is to finish the conversion: build the certs.d tree, point config_path at it, and delete the mirrors block, so that no migration has to guess.[iss12612][pr12617]

The image pull path changed underneath you

From containerd 2.1 the CRI plugin pulls images through the Transfer Service rather than pulling in-process. This is a default, not an option you enabled, and on its own it is unremarkable. What makes it worth a section is the fallback. If the CRI image configuration contains anything the Transfer Service cannot honour, containerd sets use_local_image_pull for the whole node and logs a warning. It does not fail, it does not tell you at the point of use, and it does not tell the cluster.[ctrxfer]

# Since 2.1 the CRI plugin pulls images through the Transfer Service instead of
# pulling in-process. This is not a flag you set; it is the default. What makes
# it worth knowing is the fallback: if the CRI image configuration contains
# anything the Transfer Service cannot honour, containerd silently switches the
# whole node back to local pull and logs a warning.
#
# The triggers, from the CRI config guide:
#   Registry.Mirrors set          Registry.Configs set      Registry.Auths set
#   MaxConcurrentDownloads != 3   DiscardUnpackedLayers = true
#   ImagePullWithSyncFs = true    DisableSnapshotAnnotations = false
#
# Which means the perfectly reasonable act of raising the download concurrency
# quietly changes the code path your images are pulled through.

journalctl -u containerd --since '10 min ago' \
  | grep -iE 'transfer|use_local_image_pull|falling back'

# If you want local pull, ask for it rather than triggering it by accident:
#   [plugins.'io.containerd.cri.v1.images']
#     use_local_image_pull = true
#
# If you want the Transfer Service, move the settings to where it reads them:
#   [plugins.'io.containerd.transfer.v1.local']
#     max_concurrent_downloads = 6

# Check which path a real pull took, end to end:
crictl pull registry.k8s.io/pause:3.10.2
crictl images | head
ctr -n k8s.io images ls | wc -l
SettingLocal pullTransfer Service (default from 2.1)
snapshotterSupportedSupported
ImagePullProgressTimeoutSupportedSupported
PinnedImagesSupportedSupported
Registry.Mirrors / Configs / AuthsSupported (all deprecated)Not supported — triggers fallback to local pull
MaxConcurrentDownloadsRead from the CRI image configMust move to plugins.'io.containerd.transfer.v1.local'; anything other than 3 triggers fallback
DiscardUnpackedLayersSupportedNot supported — triggers fallback
ImagePullWithSyncFsSupportedNot supported — triggers fallback
DisableSnapshotAnnotationsSupportedConfigure in the snapshotter plugin instead; false triggers fallback

Read the trigger list once and the implication becomes obvious: raising max_concurrent_downloads from 3 to 6 — an ordinary, sensible thing to do on a node with a fat pipe — moves every image pull on that node onto a different code path. So does keeping the deprecated mirrors block, which is a second reason to finish the registry conversion rather than leave it. If you want local pull, set use_local_image_pull = true and mean it. If you want the Transfer Service, move the concurrency setting to [plugins.'io.containerd.transfer.v1.local'], where it is actually read.[cricfg]

What was removed outright

Now the removals, which are the part that is genuinely gone rather than merely renamed. The list is short and everything on it has a documented replacement, but read the version numbers rather than the summaries — the containerd 2.0 transition document and RELEASES.md disagree in two places, and both times the transition document is the one people quote. Docker schema 1 pulling is the important case: it was disabled in 2.0, where an environment variable brought it back, and removed in 2.1, where nothing does. Since the target here is 2.3, treat it as gone. That matters because the images still in schema 1 are, by definition, images nobody has rebuilt since roughly 2017, which means nobody has the Dockerfile either. Find them before the upgrade, not after: since 1.7.8 and 1.6.25 converted images carry a label that makes them searchable.[ctr20][ctrrelmd]

# Docker schema 1 manifests. Get the timeline right, because it decides whether
# you have a workaround or a deadline:
#   containerd 2.0  pulling is DISABLED by default, and the environment variable
#                   CONTAINERD_ENABLE_DEPRECATED_PULL_SCHEMA_1_IMAGE=1 re-enables it
#   containerd 2.1  support REMOVED. The variable does nothing. So does anything else.
# Since 2.3 is the target, treat this as removed and find the images NOW.
#
# Since 1.7.8 / 1.6.25 converted images carry a label, so they are findable:
ctr namespaces list --quiet | xargs -I{} -- \
  ctr --namespace={} image list \
    'labels."io.containerd.image/converted-docker-schema1"'

# On a node still running 1.7, the same list from the CRI side:
crictl images -o json | jq -r '.images[].repoTags[]' | sort -u > /tmp/node-images.txt

# For each one, ask the registry what media type it actually serves. A schema 1
# manifest answers with application/vnd.docker.distribution.manifest.v1+prettyjws.
# Anything that does needs rebuilding in schema 2 or OCI before the node moves.

# The runtime v1 shims were removed in 2.0. Anything still asking for them
# fails to start the container, with an error about an unknown runtime:
grep -rn 'io.containerd.runtime.v1.linux\|io.containerd.runc.v1' \
  /etc/containerd/ /etc/crio/ 2>/dev/null
kubectl get runtimeclass -o custom-columns=NAME:.metadata.name,HANDLER:.handler

# The AUFS snapshotter was removed. Almost nobody sets this, and the ones who
# do have a kernel from 2016 underneath it:
containerd config dump | grep -E "snapshotter\s*=" | sort -u

# LimitNOFILE is no longer set in the reference unit. On systemd 240 and newer
# the default is fine; below that the kernel default of 4096 applies, and
# containers inherit it.
systemctl show containerd -p LimitNOFILE -p LimitNOFILESoft
systemctl --version | head -1
FeatureDeprecated inRemoved inWhat to do instead
Runtime V1, io.containerd.runtime.v1.linux1.42.0io.containerd.runc.v2
Runc V1, io.containerd.runc.v11.42.0io.containerd.runc.v2
Built-in AUFS snapshotter1.52.0overlayfs
containerd.io/restart.logpath label1.52.0containerd.io/restart.loguri
cri-containerd-*.tar.gz bundles1.62.0Install containerd, runc and the CNI plugins separately
CRI v1alpha2 API1.72.0CRI v1 only. Check crictl version reports RuntimeApiVersion: v1
Legacy CRI podsandbox implementation2.02.0The sandbox controller, which is the default
Pulling Docker schema 1 images1.72.1 (disabled in 2.0)Rebuild in schema 2 / OCI. The env-var escape hatch stopped working in 2.1
Go-plugin library (*.so) runtime plugins2.02.1External plugins: proxy or binary
Explicit LimitNOFILE in the reference unit2.0Use the systemd default; below systemd 240 set 1024:524288 by hand
io_uring_* in the default seccomp profile2.0An explicit seccomp profile, and a discussion about whether you want one

One removal is quieter than the rest and worth calling out. The reference containerd.service unit no longer sets LimitNOFILE explicitly. containerd's rlimits are inherited by the containers it starts, so this is not a daemon-only setting: on systemd 240 and newer the default is sensible and this is a non-event, but on anything older the kernel default of 4096 applies and every container on the host inherits it. Upstream's own guidance for those hosts is to set LimitNOFILE=1024:524288 back by hand.[pr8924][sdexec]

Defaults that flipped without asking

Separately from the removals, several defaults changed value. These are the ones that make a node behave differently after an upgrade in which you changed no setting at all, and they are worth a deliberate decision rather than acceptance by default.[ctr20]

Defaultcontainerd 1.xcontainerd 2.xWhy it matters
enable_unprivileged_portsfalsetrueContainers bind below 1024 without CAP_NET_BIND_SERVICE
enable_unprivileged_icmpfalsetrueping works without CAP_NET_RAW
enable_cdiofftrueSpec files under /etc/cdi and /var/run/cdi describe device access. The switch itself is deprecated in 2.2, gone in 2.4
NRIdisabledenabledThe NRI socket becomes part of the node's attack surface
Sandboxed CRIlegacy CRI serversandbox controllerInvisible for runc, worth testing for Kata and gVisor
Image pull pathin-processTransfer Service (from 2.1)Falls back to local pull silently on several settings
io_uring_* syscallsallowedblockedDropped from the default seccomp allowlist after repeated kernel exploits
Sandbox imagesandbox_imagepinned_images.sandboxSame value, different key. Easy to lose in the rewrite
  • Unprivileged ports and ICMP are on. The CRI plugin now sets net.ipv4.ip_unprivileged_port_start=0 and net.ipv4.ping_group_range=0 2147483647 for containers that use neither the host network namespace nor user namespaces. Binding below port 1024 no longer needs CAP_NET_BIND_SERVICE, and ping no longer needs CAP_NET_RAW. Convenient, and a change in your container security posture — setting enable_unprivileged_ports and enable_unprivileged_icmp to false restores the old behaviour.
  • NRI is enabled. The Node Resource Interface lets plugins modify containers as they are created. Access is controlled by access to the systemwide NRI socket, which means that socket is now part of your node's attack surface whether or not you run a single NRI plugin.
  • CDI is enabled, and the switch is on its way out. The Container Device Interface is on with cdi_spec_dirs defaulting to /etc/cdi and /var/run/cdi, so anything that can write a spec file into those directories can describe device access for containers. Note that enable_cdi itself was deprecated in containerd 2.2 with a removal target of 2.4, after which CDI is simply always on — if your plan was to turn it off, that plan has an expiry date.
  • io_uring is no longer in the default seccomp allowlist. io_uring_enter, io_uring_register and io_uring_setup were dropped after a long enough run of kernel exploits that the project judged them unsafe to allow by default. A workload built around io_uring will need an explicit profile — and a conversation about whether it should have one.
  • The sandboxed CRI implementation is the default. The CRI plugin uses the stable sandbox controller rather than the legacy CRI server. This is invisible in normal operation and highly visible if you run a sandboxed runtime such as Kata or gVisor, which is exactly the case worth testing before the fleet rollout.

None of these is a reason not to upgrade. They are a reason to upgrade one node, look at it, and only then write the change into the image build — which is the difference between a migration and a fleet-wide surprise.[ctrnri][cdi]

The node upgrade, in order

The mechanical part is short, and there is one change to how it is done rather than what it does. The combined cri-containerd-cni-VERSION-OS-ARCH.tar.gz bundles were removed in 2.0, so containerd, runc and the CNI plugins are now three separate installs with three separate version decisions. That is more explicit and slightly more work, and it removes a long-standing source of confusion in which people upgraded containerd and silently upgraded runc at the same time. One reassurance about the jump itself: containerd supports upgrades between sequential minor releases and, separately, direct upgrades between sequential LTS releases — and 1.7 (LTS) to 2.3 (LTS) is the example its own release document gives. You are not skipping anything you were supposed to land on.[ctrstart]

#!/usr/bin/env bash
# One node, from containerd 1.7 to 2.3 LTS. Run it on a drained node.
#
# 1.7 -> 2.3 is a supported jump. containerd supports sequential minor upgrades
# and, separately, direct upgrades between sequential LTS releases - and it
# names 1.7 (LTS) to 2.3 (LTS) as an example. That is exactly this path.
set -euo pipefail
VER=2.3.4                 # check https://containerd.io/releases/ before pinning
RUNC_VER=1.5.0
CNI_VER=1.9.1
ARCH=amd64

# 0. Get the workloads off, and keep the node out of rotation until verified.
#    (From the control plane, not from the node.)
#    kubectl drain node-01 --ignore-daemonsets --delete-emptydir-data

# 1. Back up everything the daemon owns. /var/lib/containerd is the image and
#    snapshot store; losing it means re-pulling every image on the node.
systemctl stop kubelet containerd
cp -a /etc/containerd/config.toml /etc/containerd/config.toml.bak
tar -C /var/lib -czf "/var/backups/containerd-lib-$(date +%F).tgz" containerd

# 2. Install the three components separately. The combined
#    cri-containerd-cni-*.tar.gz bundles were removed in 2.0; this is now the
#    supported route rather than a workaround.
curl -fsSLO "https://github.com/containerd/containerd/releases/download/v${VER}/containerd-${VER}-linux-${ARCH}.tar.gz"
curl -fsSLO "https://github.com/containerd/containerd/releases/download/v${VER}/containerd-${VER}-linux-${ARCH}.tar.gz.sha256sum"
sha256sum -c "containerd-${VER}-linux-${ARCH}.tar.gz.sha256sum"
tar -C /usr/local -xzf "containerd-${VER}-linux-${ARCH}.tar.gz"

#    runc and the CNI plugins, pinned deliberately rather than left behind.
#    These used to ride along inside the removed bundles; now they are yours
#    to choose, which also means yours to forget.
curl -fsSL -o /usr/local/sbin/runc \
  "https://github.com/opencontainers/runc/releases/download/v${RUNC_VER}/runc.${ARCH}"
chmod 755 /usr/local/sbin/runc

mkdir -p /opt/cni/bin
curl -fsSLO "https://github.com/containernetworking/plugins/releases/download/v${CNI_VER}/cni-plugins-linux-${ARCH}-v${CNI_VER}.tgz"
tar -C /opt/cni/bin -xzf "cni-plugins-linux-${ARCH}-v${CNI_VER}.tgz"

# 3. Refresh the systemd unit from the release, then reapply any drop-in of
#    your own. Note the reference unit no longer sets LimitNOFILE, and that the
#    directory does not exist on a host that came from a distribution package.
mkdir -p /usr/local/lib/systemd/system
curl -fsSL -o /usr/local/lib/systemd/system/containerd.service \
  "https://raw.githubusercontent.com/containerd/containerd/v${VER}/containerd.service"
systemctl daemon-reload

# 4. Put the new configuration in place - the hand-written one, not the
#    318-line dump - and prove it parses before anything restarts.
install -m 0644 /tmp/config.new.toml /etc/containerd/config.toml
containerd --config /etc/containerd/config.toml config dump >/dev/null

# 5. Bring it back, runtime first, kubelet second.
systemctl start containerd
sleep 3
ctr plugins ls | awk '$4!="ok"'          # must print only the header
systemctl start kubelet

# 6. Then verify from the cluster's point of view before uncordoning:
#    kubectl get node node-01 -o jsonpath='{.status.nodeInfo.containerRuntimeVersion}'
#    kubectl uncordon node-01

Two notes on sequencing that are easy to get wrong under time pressure. Back up /var/lib/containerd before you start, not because the upgrade is likely to corrupt it but because it is the image and snapshot store: losing it means re-pulling every image on the node, which on a large node is measured in tens of minutes and a lot of egress. And bring the runtime up before the kubelet, then check ctr plugins ls for anything not in state ok before starting the kubelet at all. A CRI plugin that failed to load is a node that accepts pods and cannot create them.[runc][cni]

The Kubernetes side of the same change

From the cluster's side there is nothing to do and one thing to check. The support matrix is a project document rather than an admission control policy: no kubelet refuses to start against an unsupported containerd, and no event appears anywhere. So the audit has to be explicit — walk the nodes, read containerRuntimeVersion, and compare it against the matrix for the kubelet version on the same node. Mixed node pools are the normal case rather than the exception, particularly where node images roll forward on their own schedule.[k8sruntime]

# The support matrix is a project document, not a runtime check: nothing stops
# a kubelet from talking to an unsupported containerd. That is precisely the
# problem - you find out from a bug, not from a startup error. So audit it.

kubectl get nodes -o json | jq -r '
  .items[] | [.metadata.name,
              .status.nodeInfo.kubeletVersion,
              .status.nodeInfo.containerRuntimeVersion] | @tsv' \
| while IFS=$'\t' read -r node kubelet runtime; do
    ctd=${runtime#containerd://}
    case "${kubelet%.*}/${ctd%%.*}" in
      v1.36/1|v1.37/1) verdict='NOT LISTED - upgrade the runtime' ;;
      *)               verdict='check against containerd.io/releases' ;;
    esac
    printf '%-22s kubelet=%-9s containerd=%-9s %s\n' \
           "$node" "$kubelet" "$ctd" "$verdict"
  done

# Carry the cgroup driver across explicitly. Kubernetes 1.28 added the ability
# for the kubelet to read it from the CRI runtime instead of its own config file
# - but that arrived as an alpha feature behind the KubeletCgroupDriverFromCRI
# gate, and on the containerd side it needs 2.0 or later. Either way the
# containerd setting has to be right, so do not treat it as automatic:
grep -rn 'SystemdCgroup' /etc/containerd/config.toml
grep -E '^cgroupDriver:' /var/lib/kubelet/config.yaml

# Runtime handlers are the part most often lost, because they are the part
# somebody added by hand. Every handler referenced by a RuntimeClass must still
# exist in the rewritten configuration:
kubectl get runtimeclass -o jsonpath='{range .items[*]}{.handler}{"\n"}{end}' \
  | sort -u | while read -r h; do
      grep -q "runtimes\.${h}\b" /etc/containerd/config.toml \
        && echo "ok      $h" || echo "MISSING $h"
    done

# And the sandbox image. It moved from sandbox_image to pinned_images.sandbox,
# and if you had it pointed at an internal mirror, that is a setting to carry
# across rather than a default to accept.
crictl info | jq -r '.config.sandboxImage // .config.containerd.sandboxImage'

The setting most often lost in this migration is the runtime handler, because it is the setting most often added by hand. Every RuntimeClass in the cluster names a handler that must exist in the rewritten configuration; if it does not, only the workloads that ask for it fail, which means the failure is scoped to whichever team was using gVisor or Kata and nobody else notices for a week. Check them by name against the new file. The same applies to the cgroup driver, and here the reassuring version of the story is not quite true: Kubernetes 1.28 added the ability for the kubelet to ask the CRI runtime which driver it uses, but as an alpha feature behind the KubeletCgroupDriverFromCRI gate, and the containerd side of it needs 2.0 or later. So it is not automatic, SystemdCgroup = true still has to survive the rewrite, and it now lives at a different path.[k8srtc][k8skubeadm]

Rolling back, and what cannot be rolled back

Rollback deserves a straight answer rather than a reassuring one. The binaries and the configuration file roll back cleanly — both are files on disk, and if you kept the old tarball and the old config.toml you are ten minutes from where you started. That is genuinely more than most migrations offer.[ctrgh]

# Rolling back is realistic here, which is not true of every migration on this
# site - but only if you kept the two things that matter and only within
# limits. Know which of these applies before you start the window.

# --- what rolls back cleanly ---------------------------------------------
# The binaries and the configuration file. Both are files on disk.
systemctl stop kubelet containerd
tar -C /usr/local -xzf /var/backups/containerd-1.7.28-linux-amd64.tar.gz
cp -a /etc/containerd/config.toml.v2.bak /etc/containerd/config.toml
systemctl daemon-reload && systemctl start containerd kubelet
containerd --version

# --- what does not ---------------------------------------------------------
# 1. The image and snapshot store, in the sense that nobody promises it will.
#    containerd's stability document puts file system layout, storage formats
#    and snapshot formats explicitly OUTSIDE its guarantees and says the project
#    may migrate these formats between minor versions. A downgrade against a
#    store that 2.x has already written to is therefore undefined rather than
#    documented-as-broken. Restore the tarball instead of finding out:
#      systemctl stop containerd
#      mv /var/lib/containerd /var/lib/containerd.v2
#      tar -C /var/lib -xzf /var/backups/containerd-lib-2026-08-24.tgz
#    (Container root filesystems are maintained on upgrade; it is the metadata
#     around them that has no promise attached.)
#
# 2. A configuration file you already migrated. A version 4 file needs
#    containerd 2.3.0 or newer, and a version 3 file needs 2.0 or newer. This
#    is why the config.toml backup matters as much as the binary one, and why
#    writing version 3 rather than 4 keeps your options open for a while.
#
# 3. Nothing about the Kubernetes control plane. This is a node-level change:
#    do NOT roll the cluster back because one node's runtime misbehaved.

# The honest limit on all of this: rollback buys you a night, not a quarter.
# containerd 1.7 leaves extended support in September 2026, and that extension
# only ever covered Kubernetes 1.30, 1.31 and 1.32 on GKE - all three of which
# are already out of support upstream.

Two things do not roll back so easily. The state directory is the first, and the honest framing is that nobody promises it will: containerd's stability document places file system layout, storage formats and snapshot formats explicitly outside its guarantees and says the project may migrate these formats between minor versions. So a downgrade against a /var/lib/containerd that 2.x has already written to is undefined rather than documented-as-broken, which is not a distinction worth testing at three in the morning — restore the tarball or accept re-pulling every image. The second is the configuration file itself, if you already migrated it: a version 4 file needs containerd 2.3.0 or newer to be read at all. And the wider point deserves bluntness: rollback buys you a night, not a quarter. The 1.7 branch leaves extended support in September 2026, and it is not receiving patches for anything outside one vendor's managed service in the meantime. A rollback is a way to end a bad maintenance window safely, not a way to defer the decision.[ctrsec]

Verifying, rather than hoping

Verification is not a matter of taste, and on this migration it has a specific shape: almost everything that goes wrong leaves the daemon running. So checking that containerd is up proves nothing. The script below checks the things that can be silently wrong — the configuration version, whether any plugin is in an error state, whether CRI is answering on v1, whether the cgroup driver and the runtime handlers survived — and then does the two things that cannot be established by inspection at all.[ctrcrictl]

#!/usr/bin/env bash
# Post-upgrade verification. Every check prints OK or explains itself; the exit
# code is the number of failures, so this can run straight from your config
# management after the node comes back.
fail=0
chk() { if eval "$2" >/dev/null 2>&1; then printf 'OK   %s\n' "$1";
        else printf 'FAIL %s\n' "$1"; fail=$((fail+1)); fi; }

chk 'containerd is 2.x'      'containerd --version | grep -qE " v2\."'
chk 'config is version 3+'   'grep -qE "^version = [34]$" /etc/containerd/config.toml'
chk 'no in-memory migration' '! journalctl -u containerd -b | grep -q "Configuration migrated from version"'
chk 'config parses'          'containerd --config /etc/containerd/config.toml config dump'
chk 'no plugin in error'     '[ "$(ctr plugins ls | awk "NR>1 && \$4!=\"ok\"" | wc -l)" -eq 0 ]'
chk 'CRI answers on v1'      'crictl version | grep -q "RuntimeApiVersion:  v1"'
chk 'runtime handler runc'   'containerd config dump | grep -q "runtimes.runc"'
chk 'systemd cgroup driver'  'containerd config dump | grep -q "SystemdCgroup = true"'
chk 'no legacy shims'        "! containerd config dump | grep -qE 'io\.containerd\.runtime\.v1\.linux|io\.containerd\.runc\.v1'"
chk 'mirrors not set'        "! containerd config dump | grep -q 'registry.mirrors'"
chk 'sandbox image pinned'   'containerd config dump | grep -q "pinned_images"'
chk 'kubelet is running'     'systemctl is-active --quiet kubelet'

# The two that are worth reading rather than counting. First: registry
# resolution has to be exercised, not inspected - a hosts.toml with the wrong
# directory name looks perfectly fine and simply never matches.
crictl pull registry.k8s.io/pause:3.10.2 >/dev/null 2>&1 \
  && echo 'OK   pull through the configured hosts' \
  || { echo 'FAIL pull through the configured hosts'; fail=$((fail+1)); }

# Second: the deprecation list should be shorter than it was before, not
# longer. A new entry here is something the migration introduced.
ctr deprecations list --format json 2>/dev/null | jq -r '.[].id' | sed 's/^/     still deprecated: /'

# And a real workload, because none of the above proves a container starts.
ctr run --rm docker.io/library/alpine:3.22 verify-"$$" /bin/true \
  && echo 'OK   container runs' \
  || { echo 'FAIL container runs'; fail=$((fail+1)); }

printf '\n%d failure(s)\n' "$fail"; exit "$fail" 

Those two are a real pull and a real container. A hosts.toml in a directory whose name does not exactly match the registry host namespace looks completely correct and simply never matches, and no amount of reading the file will show you that; only a pull will. And a configuration can be valid in every respect and still not start a container, because the runtime binary is the wrong version or is not on the path the shim expects. Run both, on the first node, before the second one.[critools]

The order to do this in

Compressed, the decision is smaller than the article. There is one target version, and it is 2.3 — the current LTS, supported to April 2028. Everything else on the table is either already out of support, out of support within months, or an extended-support branch scoped to someone else's managed service. The work is not the binary swap, which is fifteen minutes; the work is the configuration rewrite, and inside that, the registry conversion.[ctrrel]

If your situation is…Then the target is…And the work is…
containerd 1.7 on Kubernetes 1.34 or 1.352.3 LTS, in one hopThe full configuration rewrite. LTS to LTS is an explicitly supported jump, and there is no reason to stop at 2.2
containerd 1.6, anywhere2.3 LTS, urgentlyUnsupported since August 2025, and 1.6 to 2.3 is neither sequential nor LTS-to-LTS — go via 1.7. Treat it as a security item, not a maintenance one
containerd 2.1, upgraded last year2.3 LTSThe config is already version 3, so it is mostly a binary swap — but 2.1 to 2.3 skips 2.2 and is outside the supported upgrade path, so test it rather than assuming
containerd 2.2, current2.3 LTS before November 2026Minimal, but do not let it drift — 2.2 ends before the next LTS arrives
Kubernetes 1.36 already, containerd 1.x2.3 LTS, this windowYou are running an untested pairing. The matrix has no 1.x row for 1.36
A managed service (GKE, EKS, AKS)Whatever the provider shipsRead their node image release notes; the runtime is theirs, the RuntimeClasses are yours
  1. Inventory before you plan. Daemon version, config file version, CRI API version and the kubelet version, per node. Then ctr deprecations list --format json across the fleet and keep the output — it is the list of things you already know are wrong.
  2. Rewrite the configuration by hand, using the converter as a dictionary. Run containerd config migrate to learn the new key names, then write a short version 3 file yourself. Do not install the three-hundred-line dump: it pins every default you never chose.
  3. Do the registry conversion first and separately. Build the certs.d tree, prove it with ctr images pull --hosts-dir, and make sure mirrors and config_path never appear in the same file. This is the step that causes outages.
  4. Decide about the changed defaults rather than inheriting them. Unprivileged ports, NRI, CDI and the io_uring seccomp change are security-relevant. Pick, write it down, and put it in the image build.
  5. One node, then a pool, then the fleet. Back up /var/lib/containerd, upgrade, run the verification script, uncordon. Only then write the change into the node image — and check the runtime handlers your RuntimeClasses depend on, because nothing else will.

This is one of four changes landing on the same nodes in the same year, and they are easier together than separately: migrating from cgroup v1 to cgroup v2, because the cgroup driver setting has to survive both rewrites and the kubelet now reads it from the runtime; the breaking changes in Docker Engine 29, which is the same container stack seen from the Docker side; and moving from ingress-nginx to the Gateway API, if the node is also being rebuilt for the ingress change. If you are weighing up how much of this you need at all, when not to use Kubernetes is the other side of the argument.

Frequently asked questions

Is containerd 1.7 still supported?

Only in a narrow sense. containerd's release table lists 1.7 as LTS until September 2026, but the footnote is the important part: general support from the committers ended in March 2026, and the extension is provided by two named maintainers and is focused on usage with Kubernetes 1.32, 1.31 and 1.30 via Google Kubernetes Engine, with changes liable to be refused if they are not needed for that usage. All three of those Kubernetes versions are already end of life upstream. If you are not on GKE running an out-of-support Kubernetes, treat 1.7 as unsupported today rather than in September.

Which containerd 2.x version should I upgrade to?

2.3. It is the current LTS branch, it started on 30 April 2026 and it is supported until 30 April 2028. The alternatives are worse in specific ways: 2.1 reached end of life on 3 July 2026, 2.2 is supported only until 6 November 2026, and 2.0 is on the same kind of vendor-scoped extended support as 1.7. If you are on 2.1 or 2.2 today the move to 2.3 is small, because your configuration is already version 3.

Do I have to rewrite config.toml, or will containerd 2.x read my old file?

It will read a version 2 file and convert it in memory on every start, so nothing forces you — and it logs Configuration migrated from version 2, use `containerd config migrate` to avoid migration every time it does, which is the quickest way to audit a fleet. The argument for rewriting anyway is that the compatibility path is where the known problems live, in particular the registry migration that injects config_path next to your mirrors block and stops the CRI plugin from loading. Use containerd config migrate to learn the new key names and hand-write a short file. Choose the version deliberately: version 3 is read by containerd 2.0 and later, version 4 needs 2.3.0 or newer, and version 4 is what migrate emits on 2.3.

What is configuration version 4, and do I need it?

Version 4 arrived in containerd 2.3. It changes nothing about CRI — the plugin split that most of this migration is about is version 3 work, introduced in 2.0. What version 4 does is move the daemon's own sockets out of the top-level [grpc], [ttrpc], [metrics] and [debug] tables and into io.containerd.server.v1.grpc, …v1.ttrpc, …v1.metrics and …v1.debug plugin blocks; [debug] keeps level, format and log_trace_id at the top level. One behaviour change is easy to miss: before version 4 an unset ttrpc address was derived from the gRPC address as <grpc address>.ttrpc and inherited its uid and gid, and in version 4 the ttrpc plugin uses its own default instead. You do not need version 4, and there is a reason to prefer version 3 for a while: a version 4 file cannot be read by 2.0, 2.1 or 2.2, so writing one narrows your rollback options.

What does "`mirrors` cannot be set when `config_path` is provided" mean?

It means the CRI image service plugin refused to load because the registry configuration ended up specifying both the deprecated mirrors table and the newer config_path. containerd starts regardless — the daemon is healthy, the plugin is not — and every pod scheduled to that node fails to create. The trap is that you may not have written both: it was reported against containerd 2.2.0 with an ordinary version 2 file containing only a registry.mirrors block, because the in-memory migration adds the default config_path alongside it. So grepping your own file finds nothing. It was fixed in pull request 12617, which landed before 2.3.0 and was backported to 2.2, so current 2.3 and recent 2.2 patches are not exposed while 2.0 and 2.1 are. Either way the durable fix is to keep exactly one of the two: build a certs.d tree, point config_path at it, and delete the mirrors block.

What replaced the registry mirrors block?

A directory tree. Set config_path under [plugins.'io.containerd.cri.v1.images'.registry] — conventionally /etc/containerd/certs.d — and create one subdirectory per registry host namespace, each containing a hosts.toml. Each file names a server and one or more [host."…"] entries with explicit capabilities, tried in order, so a mirror that does not have a layer can fall through to the upstream registry. Per-registry CA and client certificates are keys in the same file. The directory name must match the host namespace exactly, including the port — that is the most common reason a correct-looking configuration never matches.

Where did sandbox_image go?

It became sandbox under [plugins.'io.containerd.cri.v1.images'.pinned_images]. This one is worth checking by hand after any rewrite, because the failure is delayed and environment-specific: a node with internet access will happily pull the pause image from registry.k8s.io and nothing looks wrong, while an air-gapped or egress-restricted node fails to create any pod at all. If you had it pointed at an internal mirror, carry the value across explicitly.

Why did image pulls change behaviour when I only raised the download concurrency?

Because from containerd 2.1 the CRI plugin pulls through the Transfer Service by default, and the Transfer Service does not read max_concurrent_downloads from the CRI image configuration. When containerd finds a setting the Transfer Service cannot honour it sets use_local_image_pull = true for the node and logs a warning, then carries on. The full trigger list is Registry.Mirrors, Registry.Configs, Registry.Auths, MaxConcurrentDownloads other than 3, DiscardUnpackedLayers, ImagePullWithSyncFs and DisableSnapshotAnnotations = false. To raise concurrency without changing the code path, set it under [plugins.'io.containerd.transfer.v1.local'] instead.

Will my old images stop working?

Only Docker schema 1 images, and only for pulling — but check the version numbers, because a lot of writing on this is a release behind. Support was disabled by default in containerd 2.0, where the environment variable CONTAINERD_ENABLE_DEPRECATED_PULL_SCHEMA_1_IMAGE=1 brought it back, and removed in 2.1, where nothing does. Since 2.3 is the target, there is no escape hatch: the images have to be rebuilt. Find them before you upgrade: since containerd 1.7.8 and 1.6.25, images converted from schema 1 are labelled io.containerd.image/converted-docker-schema1, so ctr image list with that label finds them across every namespace. Rebuilding in schema 2 or OCI is the fix, and the images concerned are usually old enough that finding the Dockerfile is the hard part.

Can I roll back from containerd 2.x to 1.7?

The binaries roll back cleanly — keep the old tarball and it is a ten-minute operation. Two things complicate it. The state directory first: containerd's stability document puts file system layout, storage formats and snapshot formats explicitly outside its guarantees and says the project may migrate these formats between minor versions, so a downgrade against a /var/lib/containerd that 2.x has already written to is undefined rather than merely risky. Restore the tarball you took before the upgrade, or accept re-pulling every image on the node. The configuration file second: if you migrated it, a version 4 file needs 2.3.0 or newer and a version 3 file needs 2.0 or newer, so keep the version 2 original. Treat rollback as a way to end a bad maintenance window, not as a way to defer the migration: 1.7 leaves extended support in September 2026.

Does the kubelet stop working if containerd is an unsupported version?

No, and that is precisely why this needs an explicit audit. The Kubernetes/containerd support matrix is a statement about which pairings the projects test, not an admission control policy — nothing in the kubelet checks the runtime version, no event is emitted, and an unsupported pairing will start and appear to work. What you lose is the testing: for Kubernetes 1.36 the matrix lists only containerd 2.3.0+ and 2.2.0+, with no 1.x entry at all, so a 1.7 node on a 1.36 cluster is a combination nobody has exercised on your behalf. Walk the nodes, read containerRuntimeVersion, and compare it yourself.

The Service data plane on the same nodes is on its own clock: Kubernetes 1.37 deprecated kube-proxy's ipvs mode behind a feature gate, 1.40 turns it off by default and 1.43 deletes the code. migrating kube-proxy from IPVS to nftables covers the kernel 5.13 floor, the NodePort behaviour that changes underneath you, and the leftover kube-ipvs0 that black-holes traffic if nobody cleans it up.

One release-level note, because the ledger for 1.37 is not what most coverage says it is: the change that can actually leave pods in ContainerCreating is SELinuxMount reaching GA, while the cgroup v1 failure landed in 1.35, the static pod restriction in 1.34, and the containerd cliff is still ahead in 1.38. what actually breaks when you upgrade to Kubernetes 1.37 separates the three columns and gives the audit to run before the upgrade rather than after it.

Sources

Primary sources first. containerd's release policy and its 2.0 transition document are the only authoritative statements about what was removed and until when a branch is patched; the CRI configuration guide is the only complete list of the renamed keys. Where this article says something a secondary source does not — that 2.1 is already end of life, that 1.7's extended support is scoped to Kubernetes versions that are themselves out of support — the disagreement is with the summary, not with the project.

  1. containerd - Versioning and release (RELEASES.md): the release-status table quoted throughout this article, including the end-of-life dates for 1.7, 2.0, 2.1, 2.2 and 2.3, the footnotes explaining that 1.7's and 2.0's extended support is scoped to specific Kubernetes versions on GKE, the Kubernetes/containerd support matrix, the daemon-configuration version table (version 3 needs 2.0, version 4 needs 2.3), the deprecation tables with their removal targets, the upgrade-path rules, and the "Not Covered" section that places storage and snapshot formats outside the stability guarantee
  2. containerd RELEASES.md on GitHub - the same document at its source, which is worth reading directly because the rendered docs site and the repository occasionally disagree with the older containerd-2.0 transition page (config version 1, and the deprecation release of the cri-containerd bundles, are two places where they do)
  3. containerd source, version/version.go on release/2.3: `const ConfigVersion = 4`. This is what `containerd config migrate` targets, and the reason a migrated file on 2.3 comes back as version 4 rather than version 3
  4. containerd source, the config migration table and serviceMigrate: the function that moves the top-level [grpc], [ttrpc], [metrics] and debug socket settings into io.containerd.server.v1.* plugins for version 4, including the note that an unset ttrpc address is no longer derived from the grpc address
  5. containerd 2.0 - what's new, what's breaking, what's changing: the single authoritative list of removals (CRI v1alpha2, the AUFS snapshotter, the runtime v1 shims, LimitNOFILE, the cri-containerd release bundles), the default flips, and the deprecation of the CRI registry properties
  6. CRI Plugin Config Guide - config versions 1, 2 and 3 side by side, the renamed plugin IDs, the full annotated default configuration, and the table of which image-pull options the Transfer Service does and does not support
  7. containerd - CRI registry configuration: how the deprecated mirrors, configs and auths properties map onto a certs.d directory tree, which is the conversion this migration actually turns on
  8. containerd - Registry Configuration (hosts.toml): the host-namespace directory layout under config_path, the capabilities key, and the per-host CA and client-certificate settings
  9. containerd-config(8): the manual page. Note that it documents only the `default` subcommand - `dump` and `migrate` exist in the binary but not in this page, which is why so few people know about them
  10. containerd-config.toml(5): the daemon configuration file itself - the version header, the plugins table, imports, and the state and root directories
  11. containerd source, cmd/containerd/command/config.go: the definition of `containerd config default`, `dump` and `migrate`. `migrate` and `dump` share one implementation, which is why the migrated file comes back fully populated with defaults instead of as a minimal diff
  12. containerd source, ctr deprecations: the subcommand is `deprecations` (plural), it takes --format json, and it sets CONTAINERD_SUPPRESS_DEPRECATION_WARNINGS while it runs. Some documentation writes it in the singular, which does not exist
  13. containerd issue 12612 - the CRI plugin fails to load when a version 2 config with registry.mirrors is migrated, because the result carries both config_path and mirrors: "`mirrors` cannot be set when `config_path` is provided". Reported against 2.2.0
  14. containerd pull request 12617 - the fix for the migration that emitted both config_path and mirrors. Worth checking against the exact patch release you are installing rather than assuming
  15. containerd - Plugins: the plugin model behind the renamed IDs, and the distinction between built-in, proxy and binary external plugins that matters if you still load Go plugin libraries from plugin_dir
  16. containerd - Ops: running the daemon, the systemd unit, the socket and state directories, and the configuration import mechanism
  17. containerd - Getting started: the officially supported installation route now that the cri-containerd bundles are gone, which is containerd, runc and the CNI plugins installed as three separate components
  18. containerd - Transfer service: the stable API that the CRI plugin uses for image pull by default from 2.1 onwards, and the reason a handful of registry settings now behave differently
  19. containerd - Snapshotters: overlayfs as the default and the replacement for the removed AUFS snapshotter, plus the blockfile, devmapper and erofs alternatives
  20. containerd - NRI, the Node Resource Interface: enabled by default from 2.0, which means access to the NRI socket is now part of your node's security surface whether or not you use it
  21. containerd - user namespaces in CRI: supported from 2.0 and requiring runc v1.2.0 or later, which is one of the reasons the runtime binary needs upgrading alongside the daemon
  22. containerd - CRI plugin architecture: how the kubelet, the CRI plugin, the snapshotters and the shims fit together, which is the mental model the renamed plugin IDs now reflect
  23. containerd - crictl: the CRI-level debugging tool, and the right way to confirm that the kubelet's view of the runtime matches yours
  24. containerd releases on GitHub: the binary tarballs, the checksums and the per-release notes. Also the place to confirm that the cri-containerd-(cni-)VERSION-OS-ARCH.tar.gz bundles really are gone rather than moved
  25. containerd - Security and audits: the project's security policy and advisory history, which is the argument for not staying on a branch that only accepts patches for someone else's managed service
  26. containerd pull request 8924 - the discussion behind removing the explicit LimitNOFILE from the reference systemd unit, including why hosts on systemd older than 240 must set it back by hand
  27. Kubernetes - Releases: the supported branches and their end-of-life dates. This is what turns containerd's extended-support footnotes into a dead end, because the Kubernetes versions they name are already out of support
  28. Kubernetes - Container runtimes: installing and configuring containerd for a cluster, including the cgroup driver requirement and the sandbox image setting
  29. Kubernetes - Container Runtime Interface: the API the kubelet speaks, and the reason the removal of CRI v1alpha2 in containerd 2.0 is a compatibility statement rather than an implementation detail
  30. Kubernetes - Configuring a cgroup driver: the kubelet side of the SystemdCgroup setting that has to be carried across when the containerd configuration is rewritten
  31. Kubernetes - Runtime Class: the resource that maps a pod onto one of the runtime handlers defined in the containerd configuration, which is the part of the config most likely to be hand-written and therefore most likely to be lost in a migration
  32. Kubernetes - User namespaces for pods: one of the capabilities that only exists once the node is on containerd 2.x with a recent enough runc
  33. Kubernetes - Pull an image from a private registry: the ImagePullSecrets mechanism that replaces the deprecated registry.auths block in the containerd configuration
  34. Kubernetes - Upgrading kubeadm clusters: the drain, upgrade, uncordon sequence this migration slots into, and the reminder that node components are upgraded one node at a time
  35. runc releases: the OCI runtime that has to be installed separately now that the combined containerd bundles are gone, and whose version gates CRI user namespaces
  36. CNI plugins releases: the third component of the install, previously bundled in cri-containerd-cni-*.tar.gz and now shipped on its own
  37. cri-tools: crictl and critest, the CRI-level client used throughout this article to verify that the runtime is answering on v1 and that images and pods survived the upgrade
  38. Container Device Interface: the specification behind enable_cdi and cdi_spec_dirs, both enabled by default from containerd 2.0
  39. Google Security Blog - learnings from the kCTF VRP: the exploit history that led to io_uring_enter, io_uring_register and io_uring_setup being dropped from containerd's default seccomp allowlist in 2.0
  40. OCI/Docker image manifest version 2, schema 2: the format that replaced the Docker schema 1 manifests whose pull support is disabled by default from containerd 2.0
  41. systemd.exec(5) - LimitNOFILE and the rest of the resource limits a unit inherits, which containers then inherit from containerd. Relevant because the reference unit stopped setting it explicitly

Was this useful?