The roadmaps this autumn were written for the racks you still own

Plenty of computing still runs on hardware somebody bought outright. That’s easy to forget if your week looks like ours. We work mostly with startups, workloads running in a public cloud, nothing to walk up to. The customer these hybrid products are built for looks nothing like that: a data center paid for years ago, a colo bill that’s a rounding error next to a cloud bill, and no appetite for managing any of it.

That customer is who October’s keynotes were written for. Google used Next to announce Google Distributed Cloud, managed hardware and software that extends its services into your data center and your edge sites, powered by Anthos. HashiCorp spent most of its HashiConf Global keynote on Boundary and Waypoint, two products that were already a year old, both of which make the most sense when the thing you deploy to isn’t one cloud’s API. Neither conference announced anything groundbreaking. Both made the direction clearer.

The problem underneath both keynotes is the one you inherit the moment part of your estate sits outside a single cloud. Deployments nobody can see from one place. Access granted a different way in every environment. Upgrades, patching and the rest of day two arriving months after the install, with no service owner but you. A cloud console puts the first two in one view, and managed services take specific duties off you: AWS says it handles guest OS and database patching, firewall configuration and disaster recovery on the services it manages. Responsibility for your own workloads stays with you either way. On EC2 that means the guest OS and its security patches, the application software you installed, and the security group you configured. Outside one provider you keep all of that and lose the single view as well.

Supply chain security got a ladder and a tool that climbs it

Start with the announcement that applies whether the servers are yours or rented.

What do SLSA and cosign give you that you don’t have?

A shared vocabulary for how much a build can be trusted, and a way to attach signed evidence to the artifact that build produced.

SLSA, Supply chain Levels for Software Artifacts, came out of Google in June 2021 as four levels of increasing rigor, from a scripted build up to provenance you can verify from dependency to production. Google says it’s derived from Binary Authorization for Borg, which it has run on its own production workloads for over eight years. The useful property is that it’s a ladder rather than a certification: you can sit at the first level and read exactly what the next one asks for.

cosign 1.0 shipped on 28 July and is the part you can run this week. Sigstore calls it ready for production use: the feature set for signing and verifying artifacts in OCI registries is stable, and so is the spec for how signatures are stored and discovered. The 1.0 feature list covers signing against KMS on GCP, AWS, Azure and Vault, against YubiKeys and against local keys, plus signing of arbitrary blobs. Keyless signing tied to an OIDC identity, through Fulcio and Rekor, was still marked experimental. The project moves quickly, features land close to weekly, so check your version against any tutorial you follow.

If you work somewhere with its own build platform, you may already have the equivalent in house and this changes little for you. The gain is for the open source you depend on, where maintainers now have something concrete to aim at and you have something to check.

Let somebody else store the metrics

The second theme both clouds pushed is handing back the parts of the stack nobody wanted to run.

If the service stores your metrics, what still runs in your cluster?

The scraper. Collection stays next to your targets, and what you hand over is storage, query and alerting.

That’s the first question everyone asks, because Prometheus pulls from endpoints inside the cluster and a hosted service can’t reach them. Amazon Managed Service for Prometheus went generally available on 29 September 2021 with an alert manager and recording and alerting rules: you keep a Prometheus that scrapes and sends on through remote_write, so what you really outsourced is the time-series database. Google’s version was announced at Next as a preview we couldn’t test, and as far as we could tell there was no pricing on the page yet.

Hand it over if storage is the part that hurts. Storage and scalability for metrics isn’t hard work, it’s slow work, and it’s work nobody put on a roadmap. Run more than one Prometheus and querying across them needs aggregation, which is doable and never free. Thanos answers that with a sidecar: Prometheus keeps writing its blocks locally, the sidecar uploads them to an object storage bucket, and a querier reads across the lot. Uber went further and built M3 as a remote store for Prometheus metrics, because a single node caps how much you can keep.

The same pattern turned up on the database side. Spanner got a PostgreSQL interface in preview at Next, which puts a familiar dialect in front of a store that scales past one machine and keeps distributed transactions. Read the small print: Google says full PostgreSQL compatibility isn’t the goal, so it’s a core subset, not a drop-in. What makes it feel like black magic is TrueTime, which leans on GPS receivers and atomic clocks in Google’s data centers. Hence the best line of the week: when you need cesium in your data center, that’s when you know you’ve gone too far. CockroachDB, the open source database people compare to Spanner, is deliberately built the other way and explains why it works without atomic clocks.

Your model is a product, and its API is the download link

One corner of the Next security slate is worth knowing about even though few teams need it yet.

Can somebody copy your model through its API?

Yes, given enough queries. Feed it inputs, keep the outputs, and train a second model on those pairs until it behaves the same way.

The copy doesn’t understand what is inside yours. It just lands on coefficients that balance themselves the same way, which is enough if what you sell is the prediction. The attack isn’t new: Stealing Machine Learning Models via Prediction APIs, from USENIX Security 2016, extracted models with near-perfect fidelity from BigML and Amazon Machine Learning, and found that removing confidence values from responses still leaves room for harmful extraction.

It’s also a problem few companies have earned yet. Most are still trying to get one model into production, and security doesn’t come first on that journey. If you’re past it, the hard part isn’t rate limiting, it’s telling a bad actor apart from a customer with heavy traffic. A model’s answers aren’t always deterministic either, which makes that separation harder than it sounds.

Waypoint isn’t a makefile, it’s the console you don’t have

HashiConf gave a year-old product most of its keynote, so it’s worth decoding what the product is for.

What does Waypoint give you that your deploy script doesn’t?

A server that holds every build, deployment and release across your apps, with logs and a shell into the running thing, on platforms where nothing else offers that.

The first impression is the one everybody has, including us when it launched. A waypoint.hcl of 20 or 30 lines says build this, push it to ECR, deploy it to ECS, and waypoint up runs build, deploy and release in order. It reads as a glorified makefile written in HCL. The launch post had more in it than that: waypoint logs, waypoint exec for a shell in the deployed application, and a web UI that was read-only at the time.

The UI is what the pitch undersells. The Waypoint server keeps all your builds and deployments, their logs and that exec access in one place, and if you’ve ever used the ECS console it’s roughly that, laid out better. Version 0.6, released on 14 October, pointed it at Kubernetes: install the server from an official Helm chart, build images inside unprivileged pods with Kaniko, deploy from Helm charts, and see the created resources and their health in the UI, along with horizontal pod autoscaling and ingress.

On a public cloud most of that is redundant. kubectl will do anything you want to a cluster. ECS is the case where it’s not redundant, because you juggle copilot, ecs-cli and aws ecs with a wall of arguments, and none of it is pleasant daily.

Now picture the same tooling in a private data center: Nomad and Kubernetes on your own hardware, a Lambda or two on the side, and one portal where teams watch deployments, read logs and generate preview environments. That’s a public-cloud experience for people who don’t have a public cloud, and a year of engineering stops looking odd.

Most IT spending still isn’t cloud spending

That product only makes sense next to a number.

How much of the world’s IT spending goes on the cloud?

A single-digit share of it, going by the figure Amazon’s chief executive gave at the end of 2020.

Analyst numbers land in the same place from a different direction. Gartner put public cloud end-user spending for 2021 at $304.9 billion against total IT spending of about $4.1 trillion, roughly 7%. The two forecasts don’t draw their categories the same way, and Gartner’s total counts devices, communications services and IT services alongside data center systems and software, so the ratio is about money and not about where workloads run. The split we actually wanted, companies on their own hardware against companies in the cloud, is one nobody could source.

Traffic is moving both ways across that line. Private data centers are warming to elasticity, and we’re setting one up now: an on-prem cluster that expands into the cloud when it needs more power, because sizing fixed hardware for your peak is the expensive way to buy a peak. Meanwhile the clouds are walking into the data center. Amazon ECS Anywhere went generally available in May 2021 and EKS Anywhere followed on 8 September, running on vSphere at launch with bare metal promised for 2022. Google answered with Distributed Cloud and a preview of Anthos for VMs, which attaches existing vSphere virtual machines or moves them onto bare metal with KubeVirt, so the workloads nobody could containerize end up managed as Kubernetes objects.

Two different things get called local here, and only one of them is disconnected. An Anthos cluster on bare metal keeps a thin connection back to Google Cloud called Connect, which is how the console sees it at all. Google Distributed Cloud Hosted, announced on 12 October, is the one that needs no connectivity to Google Cloud at any time, running a local Anthos control plane for customers whose rules won’t allow the link. You can’t try it yet: the preview is set for the first half of 2022.

The pitch to a migrating customer is blunt and it works: you bought those servers years ago, the colo costs very little, you don’t want to manage them, so connect them to our control plane. Data residency keeps a floor under that market, because sovereignty rules keep local hosters selling space in-country. That space is getting scarce: as far as we could tell, colocation in Stockholm is effectively full.

Day two is when you find out what you bought

Getting any of this to work is the cheap half.

When is a hybrid rollout worth celebrating?

After day two operations, not after the install.

Installing something, and still having it running, upgraded and secured several months later, are two different projects, and the second one is invisible. Nobody notices the features that keep it alive, they just expect them to be there. The share of that work still done by hand is higher than anyone admits in a talk, which is the honest case for the public cloud: you move faster there because you outsourced most of it. That makes it a strategy decision rather than a technical one: the question is when the business needs to be lean, not which tool reads nicer.

That’s also where the multi-cloud argument belongs. Corey Quinn’s case against multi-cloud is that running one workload across several providers in a provider-agnostic way confines you to the basic primitives and strips out the managed services that made a cloud worth paying for. We agree, for workloads. It’s not quite what HashiCorp is selling, though. Their bet is that the common layer is access, not runtime: Vault issuing short-lived credentials, Consul doing service discovery and connectivity, Boundary granting a session to one database rather than to the network it sits on, Nomad and Waypoint handling deployment. One IAM across clouds, racks and laptops, with one audit trail.

We can’t tell you how much of the industry that describes, because we live in the bubble of startups running on AWS. If it’s where the industry is going, HashiCorp is well positioned for it.

Zero trust is an identity problem in network costume

What makes that bet credible is that access control is where hybrid actually breaks.

Why is zero trust so hard to implement?

Because before a control plane can decide whether one thing may talk to another, both of them need an identity strong enough to verify.

People are the easier half. Workloads are where it gets thin. Kubernetes helps, since a pod has a service account, but most of an estate isn’t pods: virtual machines, batch utilities, third-party services calling your API. Somebody calling your API isn’t an identity. You need to know which workload is calling, and that means a credential issued to that workload rather than one secret shared by everything that talks to the endpoint. A signed request is one way to carry it. A short-lived certificate is another: SPIFFE puts the workload’s identity in an X.509 certificate, or in a JWT when a proxy sits in the path. Then the same question comes round again for anything that opens a network connection, and only once both ends can be named does the control plane get to make the allow decision.

The push isn’t only technical. Executive Order 14028, signed in May 2021 after a year of ransomware, started a government-wide move of US federal agencies toward a zero trust architecture, and that pulls the market along. The products are stacking up accordingly. HashiCorp’s keynote ran a segment on the pillars of zero trust, an update on Boundary, and a collaboration with Microsoft: Boundary is in the Azure app gallery, with deeper Azure Active Directory integration promised. Cloudflare for Teams with WARP and Gateway is in the same space, so is Tailscale. Which of them takes the market is an open question.

What actually shipped

Both keynotes were thinner than the build-up, so here is the slate they actually read out.

HashiConf Global was HashiConf North America with a replay in an Asia-Pacific time zone. The keynote opened on customers, downloads and metrics trending up, then gave its segments to the pillars of zero trust, a Boundary update, the Microsoft collaboration, Waypoint 0.6 and the HCP Packer public beta, free while in beta. HCP Packer is a registry for image metadata: which build produced which image, and which channel it sits on. Useful, incremental, not groundwork. The Terraform roadmap session amounted to “it’s stable”: Terraform Cloud for the web view of a plan, CDK for Terraform if you want infrastructure described in a programming language, which together read as an answer to Pulumi. Anyone who came for a big Terraform announcement went home empty-handed.

What this means for teams

Both vendors are building for teams whose estates span private infrastructure and public clouds, and for the build pipelines everybody shares. Neither theme needs a migration to act on. This week:

• Sign the artifacts of one service with cosign, then read the SLSA levels and write down which one that service is at and what the next one asks for.
• If you run Prometheus and the storage is what hurts, price the managed option against the cost of running Thanos yourself, and compare the query and alerting path as well as where the blocks land.
• If you have on-prem compute, run the Waypoint tutorials, about five minutes each, and decide whether a portal over Nomad and Kubernetes is worth the dependency. If everything you deploy already has a console, skip it.
• List the workloads that have a verifiable identity and the ones that hold a shared secret. That list is your zero trust backlog, and it exists before any product does.

Common questions, answered

How do I start with SLSA on a service I already ship?

Make your build fully scripted and generate provenance describing how it produced the artifact. That’s what Level 1 asks for, and the levels above want that same provenance authenticated and hard to falsify.

Signing is a separate task, and it doesn’t stand in for that one. cosign signs a container image against a key in your cloud KMS, in Vault, on a YubiKey or on disk, so you can sign the artifact you already publish without touching the build. A signature still isn’t a SLSA level: provenance is the part that takes real pipeline work.

How do I keep Prometheus costs down when storage is the expensive part?

Keep the scraping where it is and choose a long-term store on its own merits; the three differ. A Thanos sidecar leaves each Prometheus writing blocks to its own disk and uploads them to an object storage bucket. M3 takes a remote_write stream into a purpose-built store. Amazon’s managed service takes remote_write too, signed with your AWS credentials, which Prometheus has done natively since 2.26.0.

None of them is automatically cheaper. Price ingestion and retention at your real sample rate, and check what each does to your queries and alerting rules first.

How do I give someone access to one database without putting them on the VPN?

Put a broker in front of it that authorizes a person to a resource rather than a network. That’s what Boundary does: the session reaches one target, and Vault can issue credentials for that session rather than sharing them in advance.

Cloudflare for Teams and Tailscale solve it differently, and they cover machines too: Cloudflare Access has issued service tokens to automated clients since 2019. The mechanisms exist. What stays yours is deciding which identity each workload gets and writing the policy for what it may reach.

Related episodes

• Episode 18, HashiConf Special is the other end of this story: Boundary and Waypoint at version 0.1, and the argument about whether a makefile replacement is a thing you want maintained for you.
• Episode 15, Remote Work Security worked through identity as the gate before Boundary launched, including the VPN alternatives and hardware keys.
• Episode 28, Scaling Security covers the supply chain problem from the attack side, and why cloud IAM is the lock-in nobody budgets for.
• Episode 30, three pillars of observability has the longer version of the metrics argument, including what to aggregate and which store your team will actually query.

Resources

• Introducing SLSA, an End-to-End Framework for Supply Chain Integrity, June 2021, settles what the four levels are, including the scripted build and generated provenance that Level 1 asks for.
• Cosign 1.0!, 28 July 2021, settles that 1.0 shipped as the project’s first general availability release, that signing and verifying artifacts in OCI registries is the stable part, and that the remaining experimental features, keyless signing through Fulcio and Rekor, were still to be stabilized.
• Amazon Managed Service for Prometheus is now generally available, September 2021, settles what the managed service covers, including the alert manager and rules.
• The AWS Shared Responsibility Model and GDPR, 2018, settles which duties a managed service takes over and which stay with you on EC2: guest OS updates and security patches, application software, and security group configuration.
• HashiConf Global 2021 opening keynote settles what was announced and in what order: zero trust, Boundary, the Microsoft collaboration, Waypoint 0.6 and the HCP Packer public beta.
• Announcing HashiCorp Waypoint, October 2020, settles the build, deploy and release model, and that waypoint logs, waypoint exec and the read-only UI were there from the start.
• Announcing HashiCorp Waypoint 0.6, October 2021, settles the Kubernetes slate: Helm chart install, Kaniko builds, resource listings, autoscaling and ingress.
• Announcing Google Distributed Cloud Edge and Hosted, October 2021, settles which variant runs without any connectivity to Google Cloud, and that its preview was set for the first half of 2022.

Since we recorded

The supply chain ladder held up and got formal: SLSA reached a stable version 1.0 in April 2023, restructured into tracks after nearly two years of preview. HashiCorp ended active development of the original Waypoint in October 2023 and rebuilt HCP Waypoint around templates and golden patterns for platform teams, leaving the old source archived on GitHub since January 2024. HashiCorp’s own position changed more than its products did, through the Terraform license move and the fork that answered it, which we covered in episode 54. The hybrid question came back as a cost question rather than a capability one in episode 59.