Your containers are disposable. The host under them is a pet

Containers come with a tidy story about immutability. You build an image once, tag it, scan it, and when something needs changing you replace the whole thing instead of patching it in place. Now look at the machine that runs it. On most clusters that’s a general purpose Linux distribution with a package manager, a shell, an SSH daemon, a Python interpreter, and whatever a configuration management run left behind three months ago. Half the packages are there because something else wanted them. All of them are yours to patch.

That gap costs you twice. Every package you didn’t choose is attack surface you still have to track, and the host is where a container escape lands. It also makes updates a judgement call: in-place package upgrades depend on the state already on the box, so the node built in January and the node built in March drift apart, and the rollback plan is a snapshot and some hope.

AWS went at that problem by subtraction. Bottlerocket is a Linux distribution with only the pieces needed to run containers, announced as a public preview on March 10, 2020 and generally available since August 31, 2020 for Amazon EKS, with the Amazon ECS variant in preview. Version 1.0.7 landed on March 17, 2021, roughly half a year of point releases after 1.0.0. Our read of that was that most of the childhood problems had been sorted out and it was safe enough to try for real, so we put it into production on EKS, and the experience has been positive. This is the report from the field: what it removes, what that breaks, and what it takes to put it under an EKS cluster you already have.

What’s actually left in the image

Start with the parts list, because the whole argument follows from how short it is.

What does a container host OS contain if it only runs containers?

A kernel, glibc, systemd as init, wicked for networking, GRUB with patches for partition flip updates, a container runtime, and the agent your orchestrator needs.

That’s the list in the README. The runtime and the agent are what the variant decides. On the Kubernetes variants it’s containerd, with kubelet and aws-iam-authenticator on top. The aws-ecs-1 variant is the one to read carefully before you assume containerd everywhere: AWS’s walkthrough of the build system (January 25, 2021) shows its package list carrying docker-engine, docker-cli, docker-init and docker-proxy next to ecs-agent. So everything below about containerd and CRI is a Kubernetes-variant statement. The packages are built with RPM definitions, and RPM itself never makes it into the image: there’s no package manager on the running system, because nothing is meant to be installed there. Almost all of the first-party code is written in Rust, which is what you notice first if you open the repository and look at the language bar.

The subtraction goes further than packages. In non-developer builds there’s no shell installed and no interpreters, Python included. The security docs give two reasons for that. A shell lets an attacker pivot from a bug that grants local code execution, because the programs worth chaining together are already on the box. The second reason points inward: the missing shell is a forcing function on the people building the OS, so new OS code goes into a language like Rust or Go, which has built-in protection against memory safety problems such as buffer overflows. A third reason follows from the first two rather than from the document: drift usually starts with somebody’s helper script, and a host nobody can script on gives drift fewer places to start.

What you get on top of that baseline is already hardened rather than hardenable: an immutable root filesystem backed by dm-verity, /etc on a stateless tmpfs rendered from templates at every boot, binaries built with PIE and stack protector by default, and SELinux loaded in enforcing mode with no setting to turn it off. If your current hardening backlog says “enable SELinux” somewhere near the bottom, that’s a line item this deletes.

Settings, not shell scripts

A host with no shell needs a different way in for configuration, and this is where Bottlerocket stops looking like a normal distribution.

How do you configure a machine you can’t log into?

You send settings to an API, as TOML in EC2 user data at launch or as JSON through apiclient afterwards, and the system renders the config files from templates.

In user data it looks like a small .toml file: settings.kubernetes.cluster-name, settings.kubernetes.api-server, settings.kubernetes.cluster-certificate, and the node labels and taints you want the kubelet to register with. At runtime the same model is a transaction: apiclient -m PATCH -u /settings -d '{...}' stages a change, apiclient -m POST -u /tx/commit_and_apply applies it. Settings persist across reboots and are migrated through OS updates, which is the part that makes /etc on tmpfs survivable.

Two settings are worth knowing before you launch anything. settings.kernel.sysctl takes the kernel parameters you’d otherwise drop into a bootstrap script, quoted on both sides because the keys contain dots. And settings.kernel.lockdown turns the Linux lockdown module on in one of three modes. The security guidance recommends integrity, which blocks most ways of overwriting the running kernel and refuses unsigned kernel modules. confidentiality goes further and stops userspace reading kernel memory, which also breaks BPF, perf and anything else that relies on it. The default is none, for compatibility with deployments that still need those things.

The gotcha is directional. Lockdown can be raised at runtime but not lowered: going back toward none needs a reboot. On a node you treat as replaceable that’s academic, which is rather the point.

Updates are a partition flip

Configuration is only half of why a package manager isn’t there. The other half is what happens on update day.

What does an OS update look like with no packages to upgrade?

The image has two identical partition sets, A and B. An update writes a whole filesystem image into the inactive set, flips the partition table, and reboots into it.

Failed boots are tracked, and the system reverts to the previous working set by itself. If the new image boots fine but your workload hates it, you roll back on purpose with signpost rollback-to-inactive and a reboot. The images are signed and distributed through The Update Framework. Swapping an image also drops the dependency that makes yum update unpredictable across a fleet, because there’s no package database or half-applied upgrade on the node for the next one to build on.

Going backwards is the part to check per release. Settings and agent state outside the image carry forward, so a schema change there can close the door behind you. The 1.0.5 release notes (January 15, 2021) say the aws-ecs-1 variant cannot be downgraded once it has updated, because the ECS agent’s data store schema changed. Trying anyway can leave ECS and the container instance inconsistent, so check the notes of any version you haven’t run before you count on a rollback.

“Automated security updates” needs a footnote too: the design makes updates automatable, it doesn’t apply them on its own. Something has to drive the update API, and that something is yours to configure. On Kubernetes the recommended driver is the Bottlerocket update operator, a controller that checks for updates, cordons and drains a node, reboots it, then moves to the next. For direct control, or on the ECS preview variant where the operator doesn’t apply, the same API is two commands from a host container: apiclient update check, then apiclient update apply --check --reboot. Whichever route you pick, enabling automatic updates is the first item on the security guidance list.

The other thing to plan for is variants. Backwards-incompatible changes get siloed into new variants instead of being pushed into existing ones, and each Kubernetes variant is supported for about a year after release, following the Amazon EKS support policy. So the kernel under you doesn’t jump a major version on a Tuesday, and in exchange you’re on an annual variant migration. If you’re already doing EKS version upgrades on that cadence, it rides along with work you’ve committed to anyway.

When there’s no SSH, you get in through two containers

Everyone asks the same question at this point, usually within a minute of hearing there’s no shell: what do you do at 3am when a node is misbehaving?

What’s the escape hatch, and what does it cost?

Two host containers that run outside your orchestrator, in their own containerd instance. The control container is enabled by default and reaches you through AWS Systems Manager Session Manager; the admin container is disabled by default and runs an SSH server.

The control container runs the SSM agent, so you get a session from the console or with aws ssm start-session --target INSTANCE_ID, and from there you can make API calls to change settings. The instance needs the SSM role and read access to ECR to pull the containers at all. The admin container is the bigger hammer: SSH as ec2-user with your EC2-registered key, and sheltie for a full root shell on the host. You can enable it in user data or with one command from the control container.

Both take a superpowered flag, and that flag is where the hardening story gets honest. A superpowered host container runs with extra privileges, extra mounts, and an unrestricted SELinux label: the permissive super_t domain, where policy violations are logged and not blocked. Enforcing mode with no off switch is true of the OS, and the escape hatch is the exception. The same logic makes the host API socket at /run/api.sock a critical thing to keep away from orchestrated pods: write access to it means defining an arbitrary host container and running it superpowered, which is full control of the node.

We disabled both containers in production and weren’t certain we could turn them back on from outside. The guidance answers that plainly, and it’s a warning rather than a tip: disabling the control container “could leave you with no way to access the API and change settings on an existing node”. That’s survivable when a node is cattle and alarming when it isn’t, which is why “design for host replacement” is in the same document. Containers share the host kernel, the kernel interface can be narrowed but not removed, so replace hosts periodically even with updates automated.

The agents you used to install on the host have nowhere to go

So far this reads like a list of things you no longer have to do. Here’s the invoice.

What actually breaks when you move a cluster onto Bottlerocket?

Anything of yours that expected to be installed on the host: intrusion detection agents, scanners that build a kernel module, log shippers with Docker-shaped assumptions, and every runbook that starts with ssh.

The hard case is an out-of-tree kernel module. We’ve been blocked by this before: an image scanner that wanted a module built against the running kernel, which meant the kernel upgrade and the scanner upgrade became one coupled release. On Bottlerocket that pattern needs a superpowered host container and rules out lockdown = "integrity", so you’d be giving back two of the protections you came for. The honest answer is usually to stop trying. Run it as a privileged DaemonSet through the orchestrator if it can work that way, or replace it with something that does the job elsewhere, like scanning images at build time instead of on the node.

The soft cases are cheaper than they look, but they’ll find you on day one. Ours was logging. Filebeat was pinned to the Docker JSON log format, and the Kubernetes variant runs containerd, which writes CRI format. Log streaming stopped until the input format went back to its default: auto, docker or cri, where auto detects the format and anything else turns detection off. One setting, and it’s the sort of thing a canary node finds for free and a cluster-wide migration finds at the worst moment.

The rest of the security guidance reads as a list most teams have already half-adopted: avoid containers with elevated privileges, keep the container runtime socket away from workloads, limit host namespaces, system mounts and block devices, don’t run as UID 0. Each item comes with examples, including pod security policies you can lift. Those examples have aged into a caveat: PodSecurityPolicy was deprecated in Kubernetes 1.21, announced five days ago, so take the intent from those examples and expect the mechanism to change.

Putting it under an EKS cluster you already have

The quickstart tells you to run eksctl. If your cluster was made with Terraform, that advice is not for you, and you don’t need a fork of your module either.

What changes in your Terraform to get Bottlerocket nodes?

Two things: the AMI id of a Bottlerocket image, and your own user data template. Everything else about the worker group stays as it is.

We expected to patch the terraform-aws-eks module to make this work. Other people had expected that too, and opened pull requests adding Bottlerocket-specific handles, which the maintainers declined on the grounds that the module was complicated enough already. They were right. A worker group takes ami_id, so a data "aws_ami" filtered on bottlerocket-aws-k8s-1.19-x86_64-* picks the image for a 1.19 cluster. The variant is pinned to the Kubernetes version, so that filter is one of the things that moves when you upgrade the cluster. userdata_template_file swaps in a TOML template instead of the default shell bootstrap, and the module renders the cluster endpoint, name and certificate authority into it, because it already knows them. userdata_template_extra_args passes your own variables into the same render, so switches like enabling the admin container stay variables rather than copies of the file.

Our template sets lockdown = "integrity", leaves settings.host-containers.admin disabled and makes the control container a variable, and uses additional_userdata for node labels. The one extra resource is an IAM policy attachment for AmazonSSMManagedInstanceCore, without which the control container can’t reach Session Manager and your way in isn’t there. We’ve put the whole thing up as a Bottlerocket example on the module, so the next person doesn’t have to rediscover which two knobs matter.

Who else has tried this, and what happened to them

Bottlerocket isn’t a new idea, and the history is worth a minute before you commit a fleet to it.

The genre starts with CoreOS Container Linux and Red Hat’s Project Atomic: the same bet that a host running only containers should be small, updated as an image, and configured declaratively. Red Hat acquired CoreOS in early 2018, and Container Linux reached end of life on May 26, 2020, with Fedora CoreOS named as its successor. A few months later the CentOS Project moved its focus to CentOS Stream and cut CentOS Linux 8’s life to the end of 2021. Our read is that the IBM acquisition is showing up in these decisions, which is a guess rather than something Red Hat said. The pattern isn’t a guess: host OS lineages get retired, and a migration you didn’t schedule is the cost.

That cuts both ways for Bottlerocket. It’s an AWS product with AWS-built variants, which is lock-in of the ordinary kind. It’s also Apache 2.0 and MIT licensed, built from source with a documented variant mechanism, so a build for another environment is work rather than a petition. We’d like a Consul and Nomad variant, and nothing in the design says no.

What this means for teams

If you run EKS and your node AMI is a general purpose distribution you patch, Bottlerocket is a smaller surface and a better update story. What the switch costs you is decided by what you’ve bolted onto your hosts: agents that need the host, and runbooks that need a shell. Neither is a reason to stay, and both are a reason to try it on one node group before all of them.

This week:

• Launch one Bottlerocket worker group into a non-production cluster with the AMI id and TOML user data above, and move a real workload onto it.
• On that node group, validate the three things most likely to break before you plan a wider move: your log shipper against CRI format, every agent you install outside the orchestrator, and a full update cycle including a rollback.
• List what you install on your current nodes outside the orchestrator. Anything that can’t run as a DaemonSet is your migration list.
• Set lockdown = "integrity" from the start to find out immediately whether anything you run needs unsigned kernel modules, deploy the update operator, and leave the admin container disabled until a specific incident needs it.

Common questions, answered

How do I get a shell on a Bottlerocket node?

Start an SSM session into the control container, which is enabled by default: aws ssm start-session --target INSTANCE_ID, or Session Manager in the console. That gives you a shell inside a container with apiclient, so you can read and change host settings. For a root shell on the host itself, enable the admin container with enable-admin-container from that session, SSH in as ec2-user with your EC2 key, and run sheltie. The instance needs the SSM role and ECR read access for any of this to work, so attach those when you create the node group, not when you need them. Disable the admin container again afterwards.

How do I fix missing container logs after moving to containerd?

Set your log shipper back to format auto-detection: in Filebeat’s container input, format takes auto, docker or cri, and auto is the default. A config pinned to docker reads nothing on a Bottlerocket Kubernetes node, because containerd writes CRI format rather than Docker’s JSON. The ECS variant still packages Docker, so this one bites on the Kubernetes variants. The logs are on disk the whole time, which is what makes this confusing to debug. Anything else parsing those files, Fluent Bit or Vector included, needs the same check.

Can I run Bottlerocket outside AWS?

Not off the shelf today, but the door is open: it’s open source under Apache 2.0 or MIT and builds from source, and the variant mechanism exists so that other environments and orchestrators can be supported. The builds AWS publishes target EKS, which is generally available, and ECS, which is in preview, and those carry the orchestrator agents. A variant for somewhere else is a build you maintain, so treat it as a project rather than a download.

Related episodes

• Episode 23, How Do We Run Kubernetes In The Cloud? covers what a managed control plane doesn’t manage. Nodes are the biggest item on that list, and this is one answer to it.
• Episode 4, Docker Runtime Security is the argument about containers sharing a host kernel. That’s the premise behind designing for host replacement rather than trusting the host.
• Episode 17, Best Practices for Building Docker Images is the same subtraction one layer up: ship fewer packages in the image, patch fewer things later.
• Episode 1, Infrastructure as Code is where replacing a machine instead of fixing it stops being a slogan and starts being a build.

Resources

• Bottlerocket README, v1.0.7 settles what’s in the image, every setting you can send in TOML or through apiclient, how the two partition sets flip on update, and how the admin and control containers are enabled.
• Bottlerocket security features settles the hardening that ships on by default: dm-verity, tmpfs /etc, no shell, hardening flags, SELinux enforcing. It’s also candid about the limits, including that a superpowered host container runs in a permissive domain.
• Bottlerocket security guidance settles what’s left for you, ranked by priority, with examples. The kernel lockdown and host container sections are the ones to read before your first launch.
• How the Bottlerocket build system works, AWS Open Source Blog (January 25, 2021) settles which packages each variant carries, including Docker on aws-ecs-1 and the Kubernetes packages on aws-k8s-*.
• General availability of Bottlerocket, AWS Open Source Blog (August 31, 2020) settles what’s GA and what isn’t: EKS generally available, ECS in preview, x86_64 and arm64, Apache 2.0 or MIT.
• Bottlerocket update operator settles how automated updates happen on Kubernetes: a controller you deploy that drives the update API and takes nodes through cordon, drain and reboot in order.
• Bottlerocket example for terraform-aws-eks is our working example: AMI lookup, TOML user data template, kernel lockdown, host container switches and the SSM policy. It’s an open pull request, so read it as a worked example rather than module documentation.
• Filebeat container input settles the log format question: auto, docker or cri, with auto detecting and anything else disabling detection.

Since we recorded

EKS managed node groups added native Bottlerocket support in October 2021, so picking it is now an AMI type on a node group and most of the launch template plumbing above is optional. The pod security policy examples in the hardening guidance aged out: PodSecurityPolicy was removed in Kubernetes 1.25 in August 2022, with Pod Security Admission as the in-tree replacement. Bottlerocket moved past EC2 unevenly: VMware variants ship and it is the default operating system for EKS Anywhere on vSphere, while the bare metal line stopped at Kubernetes 1.28, after a February 2024 decision not to publish metal-k8s-1.29 or any later bare metal variant. We went one step further down this road in May 2021, in episode 29, Unikernels, where the question is what’s left if you remove the general purpose OS entirely.