The parts you glue together yourself are the parts nobody ships

Two layers of a team’s estate tend to accumulate rather than get designed. The first is how a person reaches a private host: a VPN, plus whatever you bolted onto it after the VPN turned out to grant more than you meant. The second is how source code becomes a running process: a makefile, a CI script, and three steps that live in one engineer’s shell history. Both work. Neither has a maintenance owner beyond whoever touched it last, and the upkeep rarely reaches a roadmap until the day it breaks.

At HashiConf Digital, HashiCorp announced two open source projects aimed straight at them. Boundary gives access to hosts and services without giving access to the network they sit on. Waypoint puts build, deploy and release in one file behind one command. Around those came a release slate: Terraform 0.14 in beta, Consul 1.9 in beta, Consul and Vault offered as managed services. The question that decides whether any of it matters is the one to ask of any tool at 0.1: how much do you have to build around it before it does anything?

Jacob Lärfors, CTO and co-founder of Verifa, has been working through that question with Nomad. Verifa is a small independent consultancy out of Finland working on continuous integration, delivery and cloud infrastructure, and it deliberately keeps no vendor partnerships, which is why we wanted his read rather than a partner’s. He runs Nomad in production for customers and published Nomad: the workload orchestrator you may have missed in September.

How much platform you build before anything ships

Start with the scheduler, because that choice sets the size of everything stacked on top of it.

Should you start with Nomad or Kubernetes?

Nomad puts less in front of your first workload. The delivery platform is then yours to add piece by piece. Kubernetes wants that platform first. How long it takes depends on what your team has built before, and on whether a managed control plane carries the parts you would otherwise run yourself.

Nobody actually wants an orchestrator. What teams want is an application delivery platform: the thing that takes source from a developer and gets it running, with CI and CD attached to it. That platform is what you build on Kubernetes, and building it is most of the distance between an empty cluster and a developer shipping through it.

The case where it isn’t close is a hybrid footprint. On one client we had several private data centres, GPUs on premises, general purpose workloads in AWS, and a requirement that all of it talk to the rest securely. Consul Connect tied the services together and Nomad scheduled across the lot. Kubernetes could have done it, with overlay networking as the first project instead of the application. Workloads that aren’t containers land in the same column: the exec driver runs a binary, a Java application or a shell script, and Windows is a first-class target rather than a port.

Look at who runs Nomad at scale and the pattern holds. Cloudflare schedules services across its edge data centres with it, and CircleCI maps each CI job to a Nomad job. Cloudflare’s workloads are its own. CircleCI’s are its customers’ code, but every job reaches the scheduler through one execution model the operator defined, not as an application with its own networking, storage and release needs. Mesos is on the way out, so for anything new the shortlist is these two, with ECS if you’re in AWS with a handful of services.

One binary, until you need a network

The simplicity that gets you started is also the claim that gets oversold.

Is Nomad really just one binary?

For scheduling, yes. For the rest of what you expect from an orchestrator, no.

Service discovery, DNS and health checking mean a Consul cluster next to it. Consul isn’t Nomad’s datastore, the way etcd is Kubernetes’: Nomad servers replicate their own state with Raft. Consul supplies the service catalogue your jobs register into, the DNS to find each other, health checks, and Connect if you want a service mesh. Past that the list is set by what you asked for. Container networking beyond what the task driver gives you puts CNI plugins on every client node. Storage is either a directory already on the machine, declared as a host_volume, or a CSI plugin when you want the scheduler to attach one from a storage provider. Connect usually means an Envoy sidecar beside each service, unless the application speaks the mesh itself and runs as a Connect-native task. Every plugin and sidecar there is another binary, another service to supervise, another version to match.

Matching those versions is where the difference from Kubernetes becomes work. A Kubernetes release ships with the component versions its maintainers tested together, and you inherit that pairing. With Nomad and Consul the matrix is yours: we’ve had problems picking versions of the two that got along, and the answer came from trying rather than from a support table. What you get back is that nothing is hidden: you can read what you’ve assembled, because you assembled it.

The feature gaps are real too. We felt Nomad’s development had moved slowly. Horizontal application autoscaling arrived with the Nomad Autoscaler in 0.11 in March, and cluster autoscaling followed in 0.12 in July, still in beta, alongside CNI support. If you’re coming from Kubernetes, assume a primitive you rely on may not be there, and check first.

Knowing what a job did is the same trade. Nomad ships a UI with jobs and allocations in it, the dashboard you go hunting for on your first day with Kubernetes, and it exports to Prometheus. What it won’t do is come and tell you. We had a job being OOM killed over and over, so we wrote another Nomad job to collect that and post it to Slack. We think we open sourced it. An events API is coming, close in shape to the Kubernetes events API, and that’s what we’d build the next one on.

One file describes the job, until HCL won’t say it

What you get in exchange for those gaps is a job you can read in one sitting.

What do you do when the HCL parser won’t accept your job?

Submit the same job as JSON, which reaches the API without going through that parser.

Nomad’s HTTP API is JSON all the way down, and the /job/parse endpoint converts HCL into it. JSON skips the parser, not Nomad’s job model: the server validates the job either way, so what you get past is a gap in the parser. A colleague who needed Consul ingress gateways defined as Nomad jobs, which HCL wouldn’t express, submitted them as JSON and filed a pull request to fix the HCL side. Going through the back door is always a slippery slope.

What the schema buys shows up in the shapes. A Nomad job file holds the networking, the resource limits, the health checks and the connect block that puts the service on the mesh over mTLS, with the allow rules sitting on the Consul side. The Kubernetes equivalent is a Helm chart of twenty objects. A newcomer has to know that a HorizontalPodAutoscaler scaling on CPU utilization measures that percentage against the container’s CPU requests value, and does nothing without one. They also have to know that NetworkPolicy, PodSecurityPolicy and PodDisruptionBudget exist, and that the list keeps growing. One file is also a ceiling: what the schema says is what you can do.

The first ceiling we hit was mounting a whole configuration file, the job a ConfigMap does in Kubernetes and a config does in Docker Swarm. Templating a snippet of a file was no trouble. A whole file was, so the workflow became Levant, templating the job file itself before it reaches Nomad so the configuration travels inside the job: Kustomize by another name, one layer lower.

Nomad does render whole files, to be fair to the template stanza: data holds the content inline in the job spec, and source points at a template already on the client node, usually fetched there by an artifact block. Picture the file you want sitting beside the job spec in your repository. source won’t find it, because it reads a path on the node, and artifact means first putting it somewhere the node can fetch from. Templating the job file is a way round that too.

Boundary authorizes a person to a resource

Scheduling is one half of the platform you’d rather not build. Access is the other, and it’s where Boundary lands.

What does Boundary give you that a VPN doesn’t?

Authorization per session and per resource, instead of one decision at the door that puts you on a network.

The shape is familiar if you’ve run Vault. A control plane holds the policy, worker nodes sit at the edge and proxy sessions, and the CLI authenticates you and hands back a token. We ran 0.1 during the conference, control plane in our cloud, client on a laptop, and reached resources there: you ask the CLI for the connection you want, SSH, HTTP or Postgres, the worker asks the control plane whether you may, and proxies the session if you do.

A VPN can be narrowed, to be fair to it: rules on the VPN side will hold a given user to a given set of hosts. That is complicated networking, it tends to go undone, and the VPN most teams run grants the network and lets the network decide the rest.

The second-order effect is on the network, and it’s yours to do. Installing Boundary takes no route away: your security groups allow tomorrow exactly what they allowed yesterday. What it gives you is one source address worth allowing, so you can narrow the rules on your boxes to the worker nodes and delete the rest. You’re still doing SSH. You’ve stopped accepting it from everywhere.

At 0.1 the stopper is authentication: a local database of usernames and passwords, which few companies will accept. OIDC is on HashiCorp’s published roadmap along with a Vault integration and dynamic host catalogs from Consul, AWS, Azure and GCP, and it is already visible in the 0.1 source, unreleased. That’s the release we’d start testing properly. The Vault integration is the one we’d wait for, promised rather than shipped: a database connection whose credentials come short-lived from Vault instead of a password somebody stored. How tightly those credentials track the session isn’t something the announcement settles.

One use case needs none of that. Take a tool with a decade of advisories behind it, a Jenkins you must expose, and put Boundary in front. Your engineers need the controller to authenticate against and a worker to carry the session, so both are reachable from wherever they work; Jenkins is reachable from the worker and, once you have closed the rules that say otherwise, from nothing else.

Everyone is selling zero trust this month

Boundary landed in the middle of a race, which is worth knowing before you read any launch post as a solution to your problem.

Does Cloudflare One compete with Boundary?

Not head on. They take different halves of the same problem.

Cloudflare announced Cloudflare One on 12 October, days before HashiConf. A WARP client on desktop and mobile puts employee traffic onto Cloudflare’s network, Gateway filters what leaves the device, and Access applies identity rules per application. The post is blunt about what it replaces, calling traditional VPN clients “clunky battery sucks”.

The part we’d try is the routing. In an all-remote company there are people who tell us every week how slow the VPN is, and a better authorization model won’t make their connection faster. A different route might: hop onto the nearest Cloudflare edge instead of riding the ISP’s path to the data centre. Whether it helps depends on where those people sit and what the ISP in between is doing, so it’s a test to run from their locations before anyone calls it a fix. Add a government doing deep packet inspection on everything and the case for trying gets stronger. Cloudflare doesn’t much care what runs in your data centre, while Boundary goes deeper into it, where Vault and Consul already are. The zero trust story Cloudflare tells about SaaS applications is the piece we don’t follow yet, and we’d read it properly before repeating it.

Tailscale builds the same idea on WireGuard. We went through those alternatives, SSH certificates and hardware keys in episode 15, before any of this month’s products existed.

Waypoint is the makefile HashiCorp wants to maintain for you

The third thing you build yourself is the glue between a commit and a running process, and this is the controversial one.

What problem does Waypoint solve?

One waypoint.hcl describes build, deploy and release, and waypoint up runs the three of them.

The launch post lists over a dozen plugins, Kubernetes, Nomad, ECS, Google Cloud Run, Azure Container Instances, Docker and Buildpacks among them. You also get waypoint logs, waypoint exec for a shell in the deployed application, and an optional waypoint.run URL with a Let’s Encrypt certificate, hosted by HashiCorp and switchable off. If you remember Otto, this is Otto with less ambition and a tighter scope. HashiCorp markets it as a replacement for your makefile.

That framing splits us. Jacob still writes makefiles for most of the projects he works on: forty lines written once beat retyping flags and special commands, even with a fuzzy finder and shell history behind you. Andrey built a makefile-based build system for an enterprise, calls it a childhood trauma, and says the thing is unmaintainable after a while. His answer to those forty lines is what becomes of them: three years on it’s a hundred lines, it isn’t declarative any more because there’s logic and conditionals in it, and nobody knows what is going on in there.

That’s the risk here: you commit to Waypoint, the deployment tool you want next isn’t supported, and you wait on a plugin. Two things get you round the wait, and both leave glue in your hands. Waypoint runs on Terraform’s plugin system and invites you to write your own, which is the glue maintenance you adopted Waypoint to be rid of. The exec plugin, in the same post, calls the deployment logic you already have: your script keeps working, and keeps being yours to maintain.

The angle that makes it worth trying anyway is CI. Jenkins went from a plugin per build tool to “just run a bash script”, and the bash scripts grew; a CI job that calls waypoint up, with Waypoint holding the environment-specific mess, is a reasonable middle.

Mattias won’t take the developer primitives. Nobody execs into pods in his clusters, Falco is watching for exactly that, and logs belong in the logging system rather than a CLI tail. Where he’d use it is before the merge: waypoint up into an integration environment, confirm it runs on Kubernetes, then merge. One more read, speculation rather than fact: Pulumi gives you a stack and a URL from its own SaaS, and now waypoint up gives you the same from HashiCorp’s.

The quieter releases are the ones you’ll upgrade to

The new projects took the attention. The releases are what actually changes your week.

Which of these is worth your time this quarter?

Terraform 0.14, for two things you’ll see on every plan.

Input variables can be marked sensitive, so the value stays out of console output the way sensitive outputs already do, which matters because that output gets shipped into logging and monitoring systems with weaker access control than Terraform’s. Remember that the redaction stops at the console: Terraform stores state in plaintext, including secret values from your configuration. The other change is the concise diff, on by default, printing the resources, attributes and blocks that change instead of the whole object. HashiCorp made more of it on stage than it sounds, and in a pipeline, where a plan is a wall of text nobody reads, they’re probably right. The beta also brings a provider dependency lock file and ARM64 Linux builds, and it points you at an upgrade guide rather than at an upgrade command, which is a change in itself after 0.12 and 0.13. No date was committed for 1.0; the number we heard on stage was eight to ten months, as far as we could tell.

Consul 1.9 beta is the kind of release that improves what already worked. Intentions match HTTP paths, headers and methods, where before they matched service identity alone. The UI gets a topology tab that draws error rates and latency on the connections between services. Streaming cuts server CPU and network traffic in large clusters. Kubernetes gets custom resource definitions, and OpenShift installs from a single setting.

Managed services are the rest of it. HCP Consul moved to public beta on AWS, and HCP Vault entered private beta, also on AWS, in us-west-2 only, with a single cluster size and a sign-up list. No pricing is published, and our guess is that it won’t be cheap. It’s still the announcement with the widest reach: running Vault well takes a specific skill set (episode 13), and plenty of teams have the need without the skill.

Nomad itself got almost nothing on stage, because 1.0 has its own event on 27 October.

What this means for teams

None of these announcements change what you run tomorrow. What they change is which of your homemade layers has a credible replacement coming, and the honest answer is: the access layer within a release or two, the delivery glue maybe, the platform not at all.

If you’re choosing a scheduler, choose on the shape of your estate rather than on the comparison posts. On premises, hybrid, Windows, or workloads that aren’t containers, and Nomad asks less of you before the first job runs. Cloud-native with a managed control plane available, and the Kubernetes platform you build is the one everyone else is building, which is worth something when you hire.

This week:

• Run Boundary 0.1 on a test box, put one Postgres database behind it, and write down what your security review will ask that the local password store can’t answer.
• Take the terraform plan output from your noisiest workspace, run it again on the 0.14 beta with one variable marked sensitive, and see whether the concise diff is reviewable.
• If you run Nomad, pick a job that got OOM killed last month and find out how you’d know. If the answer is that you wouldn’t, write the collector.
• If makefiles hold your deploys together, try waypoint up against one scratch service and count how many of your own lines it replaces.

Common questions, answered

How do I replace my VPN with Boundary right now?

You don’t replace it yet, you run it alongside for one resource. Boundary 0.1 authenticates against its own local database of usernames and passwords, which means a second identity store to provision and deprovision, and that’s a reason your security review will say no.

Put one target behind it, a database or a single host, keep the VPN for everything else, and treat it as a trial of the operating model rather than a migration. When the OIDC authentication method lands, the trial becomes a plan.

How do I mount a whole configuration file into a Nomad task?

Use a template stanza: source renders a template file that is already on the machine, and an artifact block is how it usually gets there. data does the same job with the content held inline in the job spec.

Neither one carries a file that lives beside the job spec in your repository: it has to be on the client node, or somewhere an artifact block can fetch it from. The way round that is to template the job file itself before it reaches Nomad, which is what Levant does.

How do I find out when a Nomad job gets OOM killed?

Write a small Nomad job that collects it and posts to Slack. That is what we did, and we think we open sourced it.

Nothing in Nomad pushes the event at you. The UI and the Prometheus integration will show you the allocation if you go and look, so what you are filling is the notification gap rather than a missing metric.

Related episodes

• Remote Work Security: VPN alternatives, SSH certificates, hardware keys and the BeyondCorp model, the argument Boundary and Cloudflare One now sell products into.
• What Are We Working On: the hybrid project where Consul and Nomad beat Kubernetes on bare metal, and what the Consul cluster cost us.
• All You Need to Know About Setting Up HashiCorp Vault: what running Vault yourself involves, which is what HCP Vault proposes to take off your hands.
• Terraform in CI: where a plan gets read, and why a diff that prints only what changes is more than cosmetics.

Resources

• Announcing HashiCorp Boundary: what 0.1 does, that workers proxy the sessions, and the roadmap items that aren’t in it yet, OIDC and Vault among them.
• Announcing HashiCorp Waypoint: the build, deploy and release model, the plugin list at launch, the exec plugin for existing deployment logic, and what the waypoint.run URL service is.
• Nomad: the workload orchestrator you may have missed: Jacob Lärfors on running Nomad for customers, why Consul comes with it, and where Levant fills a gap.
• Nomad template stanza: what source, data and a fetched artifact will and won’t do for configuration files.
• How we use HashiCorp Nomad: Cloudflare on scheduling services across edge data centres, which is what Nomad at scale looks like when the workloads are your own.
• Announcing HashiCorp Terraform 0.14 Beta: sensitive input variables, the concise diff, the provider lock file, and the absence of a date for 1.0.
• Announcing HashiCorp Consul 1.9: layer 7 intentions, the topology view, streaming, and which features are beta rather than generally available.
• Introducing Cloudflare One: the competing zero trust bundle, and which pieces are the client, the filter and the per-application policy.

Since we recorded

Nomad 1.0 went generally available on 8 December 2020, bringing the event stream we were waiting for, namespaces out of Enterprise and into open source, and a topology view. Boundary added OIDC authentication in 0.2, against Okta, Azure AD and anything else that speaks the protocol. Waypoint went another way: HashiCorp archived the Community Edition repository in January 2024, and the hosted service it kept is a different product, rebuilt around application templates and add-ons rather than the build, deploy and release command demoed in 2020. We came back to HashiConf the following year in episode 34.