Putting your manifests in Git isn’t the new part

Your Kubernetes manifests are already in Git. They went in the day you wrote them, next to the Terraform, the Dockerfiles and the pipeline config. So when someone says the team should move to GitOps, the fair question is what you’re supposed to do differently on Monday. We’ve been committing deployment specs for about ten years, and until recently we called that version control.

The new part isn’t the repository. It’s how a change gets from the repository into the cluster, and that one detail decides who holds credentials to your Kubernetes API, who can change production, and where your pipeline stops. Take the definition at face value and you’ll buy a tool that moves your deployment story somewhere you didn’t plan for it to go.

The term comes from Weaveworks, the company behind the Weave Net CNI plugin and the Flux agent. Argo CD came out of Intuit with the same shape and a web UI on top, and in November 2019 the Flux maintainers announced they were joining forces with the Argo CD project, starting with a shared GitOps Engine built by engineers from both sides.

The direction of travel is the idea

Start with the mechanics, because every other argument hangs off them.

What separates GitOps from a CI server that runs kubectl apply?

Where the thing that applies your manifests runs, and which way the change travels.

In the setup most of us already have, a CI server sits outside the cluster, holds a kubeconfig, calls the Kubernetes API across the network and watches the rollout until the pods come up. In the other one, an agent sits inside the cluster. It watches a repository, pulls the commit in and applies it locally. It still calls the Kubernetes API. It just calls it from a pod that was already there.

That rearranges the credential story, which is the point Flux made for itself in its README: “you don’t need to grant your CI access to the cluster”. If your CI is a hosted service, granting that access means something on somebody else’s network reaching your API server, so you expose the endpoint or punch a hole for it. Which you can leave as a shrug, unless you want someone mining cryptocurrency on your cluster.

The API server is the front door to everything the control plane can do, and Kubernetes puts several layers in front of it: TLS on the transport, then authentication, authorization and admission control on anything that writes, with auditing alongside. Which network it answers on is a separate decision, set with --bind-address and --secure-port. Every extra network that can reach it is a choice someone should make on purpose.

You can take the outside caller away without Git entering into it: run your CI workers inside your own network. An in-cluster agent gets you to the same place from the other direction. Neither one closes the endpoint. It keeps answering the internet until you change the cluster’s own configuration, and that switch is yours to flip. On EKS, restricting the public endpoint to IPv4 ranges in CIDR notation only arrived in December 2019, and before that the one option was turning it off. So do both: move the caller inside, then disable public access or narrow it to your ranges. Private workers also leave the credentials where they were, since they still need a kubeconfig that can deploy.

Why this is a Kubernetes story

The pull model asks something specific of whatever sits on the receiving end.

Why is nearly every GitOps discussion about Kubernetes?

Because pulling a desired state into a system only works if something inside that system knows how to converge on it, and Kubernetes ships that as its core design.

Kubernetes controllers are control loops, and each one “tries to move the current cluster state closer to the desired state”. That leaves the agent a small job: get the YAML from the repository into the API server. Convergence is somebody else’s work already.

Now try the same thing with Terraform. You can package the binary in a container and run it as a Job in the cluster, but no controller sits inside AWS reconciling your VPC against a repo. The tool that solved this for Terraform went the other way: Atlantis runs plan and apply from pull request comments, self hosted so the credentials stay with you. Push with a review gate, and it works fine.

Builds are where we get stuck, and we’re not claiming to have settled it. You can declare that you want four replicas of an image. Whether your build succeeds isn’t something you can declare, because nobody knows how it will go until it runs. That isn’t the same as saying builds can’t be declared: Jenkins X got named as a GitOps tool because it runs pipelines as Kubernetes workloads, and it writes a Jenkinsfile “for defining the CI/CD pipelines through declarative pipeline-as-code”. Its promotion model is the part that clearly fits the loop: each environment gets its own repository, and promoting a version opens a pull request there, automatic into Staging and manual into Production.

What the agent does between your commits

Granted the agent, the next question is how quickly it notices you.

How does the cluster find out that a commit happened?

It asks on a timer, unless you wire up a webhook to tell it sooner.

Flux’s daemon ships with --git-poll-interval and --sync-interval both set to 5m: the first is how often it fetches new commits, the second how often it applies the git config to the cluster, “at least this often”, with new commits provoking more frequent syncs. Argo CD polls every three minutes and takes a webhook to skip the wait. Read those as detection delay and nothing more. A commit landing just before a poll is noticed almost immediately, one landing just after waits a cycle, and either way the rollout then takes as long as pulling images and passing readiness checks takes.

The sync timer is also a reapply, which is where the two tools stop agreeing. Flux applies what’s in Git on that interval whether or not the commit changed, so a hand edit to a live object gets quietly put back. Argo CD doesn’t do that out of the box: automated sync runs only while an application is OutOfSync, and against a revision it has already synced “a second sync will not be attempted, unless selfHeal flag is set to true”, after a self heal timeout of five seconds. On Argo CD that flag is the switch for reverting hand edits, and it’s off until you set it.

Resources you didn’t put there are a different question, and the tools split on it. Flux’s garbage collection is narrow on purpose: it labels every resource it syncs from a given repo, branch and path, then deletes the ones carrying that label which the current sync didn’t apply. Anything it never created is invisible. Argo CD’s orphaned resources monitoring is the one that looks outward, flagging “a top-level namespaced resource which does not belong to any Argo CD Application”. Three things never count: the default ServiceAccount and its token, the kubernetes Service in default, and the kinds your project blacklists. The “if anything else is running here, alert me or kill it” instinct needs that second kind. Flux’s cleanup only tidies up after itself.

The hard part is the second year. Setting a cluster up and running one for two years aren’t the same job, and drift comes from incidents, not from carelessness. Somebody edits a live object at 2am because a pod is crash looping and the fix is one field. Now either the repository is wrong, or the sync timer put the crash loop back while they were still typing.

What counts as the spec you promised to run

Before a repository can be the source of truth, you have to say which file in it is the truth.

Which file is the thing the cluster will actually run?

Whatever your agent renders at sync time, which for a chart means the template plus the values, not the template alone.

helm template renders a chart locally against the values you pass it, so one chart and two value files describe two different clusters. Argo CD does that rendering itself, running helm template <CHART> each time it compares Git with the cluster, so a committed chart plus committed values is a legitimate source of truth. What it rests on is pinned inputs and deterministic output. The same page gives the counterexample: a template calling randAlphaNum gets a fresh value on every comparison, and the application sits permanently OutOfSync.

Committing the rendered output as well is the other approach, and it buys something the chart can’t: your diffs show what changes in the cluster instead of what changes in the inputs, which is what a reviewer needs before approving anything. Either answer works. What doesn’t is committing a chart whose output you can’t predict and calling the repository the desired state.

There’s a limit worth knowing before you chase total coverage. Argo CD’s best practices tell you not to track replicas in Git when the Horizontal Pod Autoscaler owns that field. That’s a separate problem from a resource nobody committed: the object is yours, one field in it isn’t, and two writers on one field show up forever as drift. Pick an owner per field, and the diff starts meaning something.

Who gets to change the cluster now

Once the cluster pulls on its own, the question of who can change production moves with it.

If a developer bumps a replica count and the cluster pulls it in, where do operations people fit?

In the repository, because that’s where the gate is now.

This is the uncomfortable part of the model. A developer commits, the cluster takes it in, the change is live, and there’s no point in that sequence where an operations person can step in. The gate moved from who holds a kubeconfig to who can merge. That isn’t worse, and it isn’t automatically better. It’s a different place to put the control, and it only exists if you configure it: branch protection, required reviewers, and commit rights that differ from the ones people hold on the application source. Argo CD’s best practices make the same point from the other side, listing separation of access as a reason to keep config in its own repository, so the people who write the code aren’t automatically the people who can push to production.

What makes this manageable is the split between the cluster and the things inside it. Infrastructure as code builds the cluster, the network and the node groups. GitOps covers the objects in it, your applications plus the operational set that’s easy to forget: log shippers, monitoring, ingress. Different repositories, different reviewers, often different people, and only the second kind is being pulled.

Pull requests, or a pipeline that stops you

Which leaves what has to happen before a merge, and here we don’t agree.

Does a cluster change need a pull request?

Mattias runs every cluster change through one. Andrey would rather the pipeline be the thing that stops you.

Mattias’s case is that it’s the same workflow as every other repository, so there’s nothing new to teach. You open the pull request, someone reads the YAML, it gets approved, it lands, the cluster takes it. Whether the author is a developer or an operator doesn’t matter, because the person who knows the cluster best can review it either way. The Weaveworks framing assumed exactly this, in Flux’s own words: “You push code not containers. Everything is controlled through pull requests.”

Andrey’s case is time. Life is too short to do pull requests: you open one, you ask for a review, and then you hunt the reviewer for two days while nothing ships. Push to master instead and spend that effort on a pipeline that won’t let you break the system. There’s a second reason that’s better than the first. When you come back to a project after two months you don’t remember how to verify it, so the first thing to do on any project is to write every verification step into the pipeline. Then returning means pushing, and no reviewer is guaranteed to understand what you did better than the checks you wrote when you did.

We stop at the same place, though. In an industry where regulation requires every production deployment to be approved or peer reviewed, review is a control rather than a preference. The argument runs that when a change exposes something it shouldn’t have, a peer review on record might reduce the legal damage quite a lot. That can sound silly from inside the team, and in that situation it isn’t.

Where your pipeline ends now

Settle the merge rules and you still have the pipeline you already built, which just got shorter.

Where does your pipeline end once the cluster pulls?

At the push. Build, test, lint, commit the manifest, done, and the deployment happens somewhere your pipeline can’t watch.

As the maintainer of a project you want one end to end view of a change: built, tested, deployed, healthy. Hand the last step to an in-cluster agent and the pipeline goes quiet at the commit, with a poll interval and a cluster between it and the answer. You find out in the cluster, in the agent’s logs, or from whoever notices first.

Two answers are honest. The first is to put the watching back into the pipeline, which Argo CD’s CI automation guide is written for: run argocd app sync, then argocd app wait. Both authenticate with ARGOCD_AUTH_TOKEN, a JWT generated from the project, against Argo’s own API rather than the cluster’s. Your pipeline blocks until the application reports healthy, and the Kubernetes API stays off the list of things your CI has to reach. The second is to accept the split and move your deployment view into the cluster’s tooling, where the state actually lives.

There’s a trap in the plumbing too. If your pipeline commits the new image tag back to the repository that triggered it, you trigger yourself, and Argo CD’s best practices list that loop as one reason to keep manifests in their own repository. One repository can still work, but then the pipeline needs logic that filters the trigger by path, so a commit touching only the manifests directory doesn’t start a build. It isn’t hypothetical: Flux’s image update automation writes a new tag back to your repository every time it finds one, and appends [ci skip] to those commits when you set --git-ci-skip.

Order, and the test you can’t run

The last gap shows up on the day you try to bring everything up at once.

What happens when the proxy starts before the backend it fronts?

It fails, restarts, and keeps failing until the backend answers, which is cheap when the retry is cheap and painful when it isn’t.

In theory nothing in a cluster should care about start order. In practice some of it does, and Kubernetes converges by retrying, which is often enough. Where it isn’t, make the order explicit: an init container blocks the app container until its check passes, and Argo CD orders a sync with phases and waves, lowest argocd.argoproj.io/sync-wave first, with pre-sync hooks ahead of everything.

The deeper gap is testing. With application code, master gets rebuilt from scratch and you learn whether the branch as a whole works. Cluster manifests work the other way: you apply the file that changed, and nobody rebuilds the cluster from master to check that the rest still stands up. The closest thing is creating a throwaway cluster from the same repository, which is exactly what the repository-as-inventory claim promises. If you’ve never tried it, you don’t know whether the claim holds for your repo.

What this means for teams

GitOps is a decision with consequences attached, not a product you install. The decision we’ve been circling is the pull direction: an agent that takes commits in and reconciles against them, with the repository as the gate in front of it. Where that agent runs is a second decision, and it’s the one that moves credentials. Argo CD v1.4.2 could already register other clusters with argocd cluster add, so pulling from Git doesn’t by itself mean the agent lives in the cluster it changes. The credential benefit arrives when the API server it calls is the one it’s already inside, and the exposure benefit arrives later still, when you close the public endpoint.

Be honest about which problem you’re solving, because the answers differ. If the problem is CI reaching an exposed API server, private runners and an in-cluster agent both let you shut off public access, and neither one does it for you. Closing the endpoint is a change to the cluster’s own configuration, and until somebody makes it the endpoint is still answering the internet. Credentials don’t disappear either: private runners still need a kubeconfig that can deploy, and an in-cluster agent still needs rights in the cluster, it just holds them where it runs. If the problem is not knowing what’s running, that’s orphan detection rather than garbage collection, and it’s the one we’d spend a week on.

This week:

• Write down everything that holds credentials to your cluster API today, which network each one reaches it from, and whether the endpoint still answers the public internet.
• Point an agent at a repository for one namespace, then edit a live object by hand and time how long your change survives. On Argo CD, run it twice, with selfHeal off and then on.
• Turn on Argo CD’s orphaned resources monitoring, or run Flux with --sync-garbage-collection-dry, and read what each one lists. They answer different questions.
• Add a wait-for-healthy step to one pipeline, so the pipeline ends where the deployment ends.

Common questions, answered

How do I deploy to Kubernetes from CI without exposing the API server?

Put the thing that applies manifests inside the cluster, let it pull from Git, then disable public access to the API endpoint, because moving the caller inside doesn’t close the endpoint by itself. Your pipeline’s last step becomes a commit, which the agent picks up on its poll interval or a webhook.

If you’d rather keep CI-driven applies, run those workers inside your own network and make the same configuration change. On EKS you can instead narrow the public endpoint to your own IPv4 ranges. Either way the credentials stay put: those workers still hold a kubeconfig that can deploy.

How do I stop a deployment commit from retriggering my build pipeline?

Keep your manifests in a repository of their own, separate from the application source. Argo CD’s best practices list exactly this: pushing config to the repository that triggers your build can produce an endless cycle of builds and commits.

If one repository is a hard requirement, filter the trigger by path so a change under the manifests directory doesn’t start a build, and have the writer mark its own commits. Flux appends [ci skip] to the commits it makes when --git-ci-skip is set.

How do I find out what’s running in my cluster that nobody committed?

Turn on Argo CD’s orphaned resources monitoring, which reports top-level namespaced resources belonging to no Argo CD Application and warns the applications in that namespace. Flux’s garbage collection won’t answer this: it only tracks what it labelled at sync time, so resources it never created stay invisible.

What comes back is shorter than you’d fear, because the default ServiceAccount and its token, the kubernetes Service in default, and your project’s blacklisted kinds are left out. Read what’s left before deleting from it.

Related episodes

• Infrastructure as Code: where the drift argument starts, and the difference between creating a cluster and running the things inside it.

Resources

• Flux README, v1.17.0: the GitOps principles as the project stated them, including why your CI doesn’t need cluster access.
• Flux daemon reference, v1.17.0: every flag the agent takes, with the five minute poll and sync defaults.
• Flux garbage collection, v1.17.0: how Flux labels what it syncs, and why it only deletes resources it created.
• Argo CD automated sync policy, v1.4.2: when automated sync runs, and what selfHeal adds on top of it.
• Argo CD orphaned resources, v1.4.2: what counts as orphaned, and the resources it never counts.
• Argo CD best practices, v1.4.2: why config belongs in its own repository, and which fields not to track in Git.
• Kubernetes: controlling access to the API: the layers a request passes through before anything is written, and where the API server listens.
• Amazon EKS public endpoint access restrictions: the December 2019 announcement, and the fact that closing a managed cluster’s public endpoint is its own setting.

Since we recorded

The two tools never became one: the projects’ own collaboration FAQ still files a single project under “the ultimate goal”, and records that Flux tried the shared GitOps Engine, extracted from Argo in the first place, then took a different approach. Flux was rewritten as Flux v2, and Flux Legacy reached end of life, so every v1.17.0 flag above is history rather than advice. The definition we tried to pin down here got written out properly as the four OpenGitOps principles: declarative, versioned and immutable, pulled automatically, continuously reconciled. We came back to where this sits next to everything else you provision in episode 101, “Infrastructure as Code in 2026”.