Podcast about DevSecOps

#29 - Unikernels

One application per VM changes how teams deploy, collect metrics and answer audit questions.

17 min read

This time we are talking unikernles! Ian Eyberg from NanoVMs joins us to discuss how far this technology is from prime time. And it turns out that you don’t have to be a kernel developer to take advantage of unikernes. Today, there are tools available to package, distribute, and run them locally as well as in the public cloud. While talking to Ian, it felt that the state of the technology is very similar to Linux containers at the beginning of 2010x, just before Docker made Linux containers available for everyone.

Discuss the episode or ask us anything on LinkedIn


You ship one application and deploy a hundred other things

Boot a stock Ubuntu or Debian image on EC2, install nothing at all, and your code is already in a crowd. “There’s like a hundred processes already running on it”, and on a one thread instance the ones with work to do take turns on that single CPU. The ones asleep still hold memory. None of it is your application. It’s the machinery that makes a general purpose machine usable: an init system, a package manager, remote login, log rotation, and the timers that tidy up after them. Every installed package there is something to update and something to answer for when an auditor asks what runs on the box. Then we add a container runtime, and an orchestrator above that, so the one process we actually wanted ends up three layers down in a stack we also have to secure.

Unikernels ask the obvious question. If a virtual machine runs one application, why is there a general purpose operating system underneath it? Give the application only the operating system functions it needs, boot that as the VM, and the layers in between stop existing. The research case is old: Unikernels: Library Operating Systems for the Cloud, from ASPLOS 2013, reported an order of magnitude less code with no significant performance penalty. Those library operating systems compile your source and the operating system functions into one image together, and that is where the practical case kept failing. To use any of it, you had to be a kernel engineer.

That part has changed, which is why we went looking for someone building the tooling. Ian Eyberg is the founder and CEO of NanoVMs, based in Oakland, California. He got into the field about five years ago by reading the papers, had the first Go unikernel built because nobody else had done it, and now works on Nanos, a kernel that runs exactly one application, and ops, the command line tool that packages and deploys it. Nanos doesn’t ask you to compile against it. It takes the ordinary Linux ELF binary your toolchain already produces, puts it on a disk image and boots that, so go build stays your build step.

What a unikernel is, and why you haven’t run one

Start with the word, because the definitions in circulation are loose and one of them is wrong.

What is a unikernel?

One application, and only one, with just the operating system functions it needs underneath it, and no second program on the machine.

That single application is all the implementations share, and there are well over ten of them now. They disagree on the rest: some run as a virtual machine, some don’t, some keep a syscall boundary, some don’t, and whether your code is compiled into the kernel image or loaded as an ordinary binary is another of those splits. Nanos takes the ordinary binary and boots it as a VM, which is the model everything below assumes. What they don’t share is the line you’ll hear repeated, that a unikernel has no operating system. It has one. Paging, access management and a network stack are all in there, and somebody had to write them. It just isn’t general purpose.

The interest in them is wider than the security angle we came in with. Telcos look at unikernels for serverless and for latency in NFV projects. ARM and Baidu look at them for edge. People who came from the security side look at what disappears from the attack surface when there are no users, no shell and no package manager.

Why haven’t you run one? Using the older projects meant cross-compiling code bases you didn’t write, in languages you didn’t know, against kernel concepts you’d never needed. NanoVMs treated that, rather than performance, as the thing blocking adoption, and set the bar at a DBA who doesn’t write code being able to spin up an application without writing any.

So ops is deliberately thin. Install it with curl https://ops.city/get.sh -sSfL | sh. Locally it builds a disk image out of your application and drives QEMU for you, and that’s close to the whole of it. Build for Linux first, because the image boots an ELF binary: GOOS=linux go build main.go on a Mac, plain go build main.go on Linux, as the NanoVMs Go tutorial has it. For Ruby, Python or Node you load a package with a known good interpreter, because nobody should have to discover on their own that DNS resolution wants libresolv and libnss on the image.

We’ve seen this shape before. Linux containers existed for years before most teams touched them, and what changed was packaging and distribution, not a kernel feature.

On AWS it’s an AMI, not another layer

The next thing that stops people is the hypervisor, because the clouds don’t hand theirs over.

How do you run a unikernel on a cloud that won’t give you its hypervisor?

You don’t need the hypervisor: ops image create turns your application into an AMI, and that AMI boots as an ordinary EC2 instance.

The common misconception runs the other way, that you stand up Kubernetes on EC2 and run unikernels on top of it. NanoVMs goes in the opposite direction. Your account already sells you storage, networking and security primitives, so use those and skip the layer. In practice the image is staged into an S3 bucket you own (CloudConfig.BucketName), ops reads ~/.aws/credentials, and it creates a security group per instance unless you point it at an existing CloudConfig.VPC and CloudConfig.SecurityGroup (the AWS pages of the ops docs carry the full set).

The orchestration you were about to add is already in the account. An auto scaling group is the thing that keeps five copies of your web server alive, and the load balancer in front of it is your ingress. If you want somewhere to script against, put Nginx or HAProxy in front as a reverse proxy and treat that as your junction point. None of it is new infrastructure, which is the appeal: it’s the architecture your account has supported for a decade.

If you do want the hypervisor itself, it’s available and uncommon. AWS has had bare metal instances since 2017, and Google Compute Engine supports nested virtualization with KVM. For this workflow you need neither.

Where the performance difference comes from

The single application rule isn’t only a packaging constraint. It’s also why the same binary can serve more traffic, though not quite for the reason you’ll hear.

Why would the same binary run faster as a unikernel?

Because nothing else on the instance is competing with it for memory and CPU, and the kernel underneath it has only your program to serve.

Threads are fine. Go, Java or Rust can have as many as they want, and more of them is how you use a bigger instance. Processes aren’t supported at all.

The explanation usually offered is those hundred processes on the fresh Debian instance, stealing the CPU from your program. On its own it doesn’t hold up. Linux schedules the tasks that are runnable, not every task that exists, so a daemon blocked on a socket costs you memory rather than CPU time, and a unikernel with ten busy threads is giving the scheduler work too. What you do remove is the rest of the machine: the services and their timers, the memory and page cache they hold, and a kernel built to arbitrate between many programs and many users when you brought one.

NanoVMs’ own benchmarks put numbers on it. ops.city and nanos.org run as Go unikernels on Google Cloud, where the company measures roughly twice the requests per second of the same application and the same Go version on a stock Debian image, and roughly three times on AWS. MySQL and Redis measured faster as well. Those are the vendor’s figures from the vendor’s benchmarks. The interview gives no methodology for these figures, so benchmark your own workload. The reasoning behind them is the honest part: “It’s merely architecture. We didn’t write a magic algorithm or anything like that”.

Hypervisor and networking differences could help explain the gap between the Google figure and the AWS figure: Google Cloud is built on KVM, AWS ran on Xen for years and uses a modified KVM on its newer instances, and the Nitro system offloads AWS networking onto its own silicon. Nothing in the comparison isolates which of those does the work, or how much.

What runs unported, and what you’d be sizing

That same rule is the constraint, and it decides which half of your stack can move at all.

What can you actually run this way today?

Server side applications that live in one process: a Node.js server, a GraphQL server and MongoDB run with no porting, and MySQL, Redis and Nginx come as packages.

What’s out is anything multi-process by design. PostgreSQL forks a process per connection and coordinates through shared memory and IPC, so it isn’t a candidate without work. NanoVMs has turned processes into threads for customers who insisted on a specific piece of software, but that’s a port, not a deployment. Desktop software is out of scope entirely. The target is your web tier.

Sizing changes too, because you can’t log in and watch top. Start from the concurrency your application actually uses rather than the reputation the language has. Node runs your JavaScript on one thread unless you reach for worker threads, and CPython has real threads but the global interpreter lock lets only one of them execute Python bytecode at a time, so CPU-bound Python won’t fill a large instance however many cores you rent. Several interpreters on small instances behind a load balancer will. The lock isn’t the whole story either: extension modules written for it release the GIL during compression, hashing and I/O, so if your hot path sits in C, size that workload from a measurement instead of from the rule. Go and Rust spread work across native threads, so scaling up works there, and Google will rent you instances with hundreds of threads if you have the workload for it.

The size you pick has to come from what the application reports about itself, the request rate it sustains and the memory it holds, and that telemetry has to be in place before you deploy, because there’s no interactive session to open when the question comes up later. Predefined instance types don’t help much, since they come in fixed steps: one thread and four gigabytes, two threads and sixteen. Google is one exception, with custom machine types announced in November 2015 that let you set the vCPU count and the memory separately. Oracle is another, with flexible VM shapes added in 2020, where you pick the core count and the memory per core when you create or resize the instance. On the providers still shipping fixed shapes, there’s room for them to get more fluid.

One second order effect is worth watching. If micro VMs take off, demand shifts to the small sizes, t3a.nano and t3a.micro, because the stack you were paying for isn’t there.

Logs and metrics when there’s no shell

Sizing is the easy half of operating a machine you can’t log into. The other half is the objection this technology has carried for years.

How do you debug something with no SSH and no shell?

With the signals from inside your application, because that’s where the useful ones were already coming from.

The sharp version of the objection is Bryan Cantrill’s Unikernels are unfit for production from January 2016, arguing that you can’t understand them when they misbehave. Logging answers cleanly. For something small you don’t care much about, write to standard out, let it go out over the serial interface and pull it back with ops instance logs. For anything with traffic, ship syslog to what you already run, Elasticsearch, Splunk, Papertrail. The configuration moves into the application: your logging library, told which host and port to send to, instead of /var/log/syslog and a systemd unit on the host.

Metrics are the same move, with one deployment detail that an SDK alone doesn’t settle. Prometheus is the clean case, because the client library is already inside your process, it exposes an endpoint, and whatever you run today scrapes it. Datadog works the other way round. Its client sends StatsD datagrams to a DogStatsD server hosted by the Datadog Agent, so that Agent still has to run somewhere, and the somewhere can’t be the unikernel. Keep a host for it, point your instances at its address the way you point them at syslog, and ask the same question of any other APM you run. What you can’t do is drop an agent binary next to your application, because there is no next to.

That answers logs and metrics. It doesn’t answer the debugger: no shell at three in the morning, no strace on a misbehaving box. NanoVMs sells crash dumping and metrics collection as a product, which is a fair indication of where the gap still is.

Packages, images, and the hub that isn’t there yet

Distribution was the other half of what made containers ordinary, and it’s the part that isn’t finished here.

What’s the difference between an ops package and an ops image?

A package is a tarball of binaries and libraries, closer to apt-get than to Docker Hub, and an image is the bootable disk image built from that package and your application, on demand.

Images get rebuilt constantly, and that’s deliberate. Config files and environment variables are baked in, so every deploy produces a new image instead of a modification of the old one. The format follows the target, VHDX for Hyper-V, raw for KVM, and with a base around ten megabytes, converting between them is effectively instant.

If you’ve been selling immutable infrastructure to your team, that’s the argument with fewer exceptions than usual, and it’s worth being exact about where it stops. Deployments replace images and nobody administers the host. Your application still writes its data and still answers on the interfaces you gave it, so the attack surface it exposes is its own.

The hub is the missing piece. Packages today are the ones NanoVMs builds, you can build your own, and there’s no open central place to publish them. That matters, because the Docker Hub model is a large part of why containers spread. You don’t compile Postgres, you pull an image somebody who understands Postgres built, and you throw it away afterwards. Our attempt to map the terms onto Docker, with the package as the image and the image as the container, didn’t survive contact with the person who named them. “Maybe the terms could use some work”, Ian said. Build the model fresh instead.

The audit questions that stop applying

Immutability starts mattering to the people who sign off on your systems before it matters to the people running them.

What does a unikernel do to a compliance checklist?

It removes the controls that assume a multi-user machine: no user accounts, no passwords to rotate, no interactive commands to record.

Two examples came from NanoVMs’ customers. A credit card processor, Paywax, got through PCI DSS level 1 on this architecture. In US government work, compliance teams walk through STIGs, long documents of controls covering password rotation, which person ran which command, and who has access to what. NanoVMs’ claim, from its own engagements, is that around 70% of the controls in one of those documents don’t apply here, because the architecture rules them out rather than satisfying them. That’s their count, and your auditor still has to agree with it.

What doesn’t go away is the application. Your code still handles the data, still needs its own authentication and logging, and still has dependencies with CVEs. Patching one means rebuilding and redeploying the whole image, which is cheap here but not automatic. The checklist gets shorter at the operating system layer and stays exactly as long at the application layer.

Most of us have sat in an audit explaining where the firewall is when the honest answer is a service mesh. You’re going to be there a long time. The appeal of this model is that several of those conversations don’t start at all.

What this means for teams

Where we landed: this looks like Linux containers just before they became ordinary. The technology runs, the tooling is young, the distribution story is unfinished, and a small company in Oakland maintains the kernel. Nanos and ops are Apache 2.0, and the company makes its money beside them, on support plans, a graphical administration tool and a product that collects crash dumps and metrics. That’s a reasonable combination to pilot with and a hard one to bet a platform migration on. The question to ask is the one you’d ask of any young tool: what happens to it if the company has a bad year.

Four things you can do this week:

  • Install ops and run one stateless Go or Node service locally with ops run. A clean local boot is the pilot’s first checkpoint, and if the service won’t come up there, nothing in a cloud account fixes it.
  • Push that same service into a sandbox account with ops image create -t aws, put it behind an auto scaling group, and compare the path with the ECS or EKS path you have now.
  • Check your log and metric paths. Anything depending on a host agent needs the syslog or SDK route instead, plus a receiver of its own somewhere. Cheaper to find out now than mid-pilot.
  • List the packages on your current AMI and mark the ones that exist for the host rather than for your application. That list is what you’d stop patching, and it’s the number worth taking to your security team.

Common questions, answered

How do I try a unikernel without touching my cloud account?

Run one locally: install ops, then ops run your binary, which builds a disk image and starts QEMU for you. No cloud credentials are involved and nothing gets uploaded. Build the binary for Linux first, since the image boots an ELF file: GOOS=linux go build main.go on a Mac, go build main.go on Linux. For Node, Python or Ruby, load the packaged interpreter first, because that package carries the shared libraries the interpreter expects to find.

Can I run PostgreSQL as a unikernel?

No, not without porting it. PostgreSQL forks a separate process for each connection and coordinates through shared memory, and a unikernel runs one process with threads. MySQL and Redis ship as packages and have been benchmarked, so a cache or a MySQL instance is a fairer first move. Keep PostgreSQL on your managed service.

Do I need Kubernetes to run unikernels on AWS?

No, and avoiding it is most of the point. ops image create produces an AMI, you boot it as an EC2 instance, an auto scaling group keeps the number of copies you asked for, and a load balancer handles ingress. If you want one place to script against, run Nginx or HAProxy as a reverse proxy in front of them.

Resources

  • Nanos settles what the kernel does and doesn’t do: one process only, no users, no remote administration over ssh, Apache 2.0.
  • ops settles the local path: one install line, a disk image built from your application, QEMU driven for you.
  • ops on AWS settles the cloud path: ops image create into an AMI, the S3 bucket it stages through, and the VPC, security group and elastic IP settings.
  • NanoVMs settles who maintains this and how they make money, which is support plans and commercial tooling beside the open source.
  • Unikernels: Library Operating Systems for the Cloud, ASPLOS 2013, settles where the claims originate: compile time specialisation, sealed images, an order of magnitude less code.
  • Unikernels are unfit for production, January 2016, settles the strongest objection, that production systems have to be debuggable. Read it before you pilot.
  • Running Go unikernels settles the build step: an ordinary ELF binary, cross-compiled with GOOS=linux if you work on a Mac, with no kernel source in sight.
  • DISA STIGs settles what the compliance claim is measured against.

Since we recorded

We came back to this on the hundredth episode, and the verdict was short: the hype collapsed, unikernels stayed niche, and they might find a niche again in AI workloads where boot speed matters (100 Episodes Later: What Still Matters in DevSecOps). Nanos and ops are still worked on and still Apache 2.0, with the project tagging release 0.1.56 in September 2026. The research line kept moving too: Unikraft, which started inside the Xen Project, shipped v0.19.0 in May 2025 with POSIX compatibility as its headline, which is the usability problem from this conversation approached from the other end. For where mainstream packaging went instead, episode 109, Docker Got Quiet. Did You Miss Anything? picks up Docker’s own trajectory.