---
title: "#89 - Agents, Reviews, and Secrets: Real Talk on AI in Dev"
date: 2026-01-05
episode: 89
---


Are devs ignoring AI, misusing it, or getting real value? What happens when agents touch your env vars, repos, and pipelines? How do you share prompts, set team defaults, and keep trust? Could an AI engineer role lead culture as well as tools?

[Discuss the episode or ask us anything on LinkedIn](https://www.linkedin.com/company/devsecops-talks/)

<!--more-->

{{< whats-in-this-post >}}

<!-- Player -->

 {{<  podbean usm9e-19e4dd8-pb "DEVSECOPS Talks #89 - Agents, Reviews, and Secrets: Real Talk on AI in Dev"  >}}

---

<!-- Video -->

## Feeling fast and shipping fast are different things {#feeling-fast}

You can put a working app on the internet in an afternoon. Lovable and the tools around it earned that reputation honestly, and for a prototype or an internal tool they are hard to argue with. The argument starts on day 30. Reliability stops fitting the shape of the company, something breaks that nobody can explain, and customers start calling about the SLA. That is the stretch worth talking about, not the afternoon.

So the question isn't whether developers can start faster with AI. They can, visibly. It's whether anything reaches production sooner, and whether what reaches production can be changed next month without introducing more bugs than features. Those are different questions, and most of the market is still answering the first one.

Mark Shine, co-founder and CTO of [Ankra](https://www.ankra.io/), the Stockholm platform that provisions Kubernetes clusters and handles the day-two work afterwards, describes two ends of a spectrum he keeps running into. At one end, a team can predictably deliver on its goals. At the other, a team leaning hard on AI will get there, but nobody can say when. Both are shipping. Only one can be planned around.

## What the perception gap actually measures {#perception-gap}

Before arguing about which tool to buy, it helps to know that the feeling of speed has been measured against a clock, and the two don't agree.

#### Are developers actually shipping faster with AI?

In METR's trial, these experienced maintainers took longer to complete their tasks, and could not feel it.

We remembered the finding as a survey of about a thousand developers with CI/CD analytics behind it. The study that matches the shape is smaller and tighter than that. [METR ran a randomized controlled trial](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) with 16 experienced open-source developers across 246 real issues, on repositories they had contributed to for years. METR randomly assigned each issue to allow or forbid AI tools. With AI allowed, the work took 19% longer. Beforehand, the same developers forecast a 24% speedup. Afterwards, having just lived through it, they still estimated AI had sped them up by 20%.

{{< key-point >}}The number worth distrusting isn't the benchmark. It's your own estimate of how much faster you got.{{< /key-point >}}

METR timed task completion, not delivery to production, and other randomized work points the other way. [Three field experiments pooling 4,867 developers](https://www.microsoft.com/en-us/research/publication/the-effects-of-generative-ai-on-high-skilled-work-evidence-from-three-field-experiments-with-software-developers/) at Microsoft, Accenture and a Fortune 100 manufacturer estimate a 26% increase in completed tasks for developers given GitHub Copilot. The error band is wide, and the results vary by experiment. METR's own caveats are long too. Sixteen developers is a small group, and they were maintainers working in code they knew intimately. The tooling was mostly Cursor Pro with Claude 3.5 and 3.7 Sonnet, and most of them had only a few dozen hours of Cursor experience. METR says outright that the result doesn't show AI is unhelpful in general, for less experienced developers, or in unfamiliar code. What survives is the gap itself: people who had just been slowed down could not feel it.

## Two camps of tooling, and a management layer on top {#speed-or-specs}

That gap explains why the tools are splitting into camps, because they optimise for different halves of it.

#### What's the real difference between Cursor and Kiro?

Which workflow each one puts in front of you by default: get to code quickly, or get to a specification first.

[Cursor 2.0](https://cursor.com/blog/2-0), out on 29 October 2025, is the clearest example of the speed camp. It shipped Composer, Cursor's own coding model, which the company says is four times faster than similarly intelligent models and finishes most turns in under 30 seconds. The same release made it easy to run many agents in parallel on separate git worktrees, and to put one request through several models and keep the best answer. Run Composer, Sonnet 4.5 and Codex on the same task and Composer finishes while Codex is still deciding what to write. The speed is real, and so is the pile of code you then have to understand. Cursor has answers for that part too: [Plan Mode](https://cursor.com/blog/plan-mode), added in October 2025, has the agent research the codebase and produce an editable Markdown plan before it writes code.

[Kiro](https://kiro.dev/blog/introducing-kiro/), AWS's IDE, sits at the other end, and it runs both modes: [spec requests for the structured workflow, vibe requests for ordinary chat](https://kiro.dev/blog/understanding-kiro-pricing-specs-vibes-usage-tracking/). We use the spec workflow. From one prompt it drafts user stories with acceptance criteria, then a design, then ordered tasks with the tests for each one. It runs on the same Claude models, so the difference is the prompting and the workflow rather than the intelligence. It's slow, often hours for one feature, and it writes a lot of tests and uses them to check itself. The output worth having isn't only the code. <mark>A written spec keeps requirements and design intent explicit</mark>, in a form the next session, the next model and the next person can read.

Then there's the layer above both. [Google Antigravity](https://antigravity.google/blog/introducing-google-antigravity), announced on 18 November 2025, puts an Agent Manager in front of several agents at once and has them emit task lists, plans and browser recordings so you can check the work without reading every tool call. Underneath, it's largely orchestration of editors running agents. That is useful, and it is not a new way to write software.

## Where the cognitive load goes {#cognitive-load}

The spec camp's real claim isn't about code quality, it's about what the day costs you, and this is where we genuinely disagree.

#### Does handing work to an agent reduce cognitive load or just move it?

It moves it, and we don't agree on whether the move is a win.

Write code for three hours and you're drained. Focus goes, and you've developed an attachment to what you produced, which makes it hard to kill the baby even when the baby came out ugly. While the agent implements the feature, we can think through the approach, then review the result without the same attachment to the code. You can look at it, decide it isn't good, delete all of it, go back to the spec, refine it and run again, without mourning anything.

Mark Shine's objection is that the saved energy may not be saved. You can be good at many sports or great at one, and attention split across several agents feels like freedom without necessarily being it. He also points at the part nobody counts: you finish the day with a small bug unresolved, your subconscious works on it while you sleep, and the fix arrives in the morning with the energy to apply it. Hand the implementation over and that pass has less to chew on.

{{< key-point >}}Mark Shine: "you can be good at many sports or great at one".{{< /key-point >}}

Andrey disagrees. The forced pause while Kiro grinds through an implementation is what makes him ask whether the approach is right, so he counts it as thinking rather than waiting. The honest version is that both describe something real, and the difference is what you do with the wait. Thinking about the problem counts. So does the gym. Checking YouTube doesn't, and that's a personal choice rather than a property of the tool.

## The quality control you handed to the thing you're checking {#testing}

There's one place where moving the work has a cost that isn't personal preference, and that's testing.

#### What breaks when the AI writes your tests?

The independence of your last check on the AI's work.

Some developers want to delegate tests because they dislike writing them. Go looking on Reddit and you'll find the trade offered cheerfully: I'll write the code, the AI can write the tests, because I don't want to deal with that. That's an impression from the threads rather than a measurement, and plenty would argue with it. The trade is fine right up to the moment tests are what you trust. If the same thing writes the implementation and the assertions, the two share their blind spots. The tests still execute and they still catch regressions. What they stop being is an outside opinion on whether the implementation understood the requirement in the first place.

The survey data says that's happening inside a wider squeeze. [Stack Overflow's 2025 Developer Survey](https://stackoverflow.co/company/press/archive/stack-overflow-2025-developer-survey/) found 84% of developers using or planning to use AI tools, up from 76% the year before, while 46% said they don't trust the accuracy of the output, up from 31%. Favourability fell to 60% from 72%. And 45% reported that debugging AI-generated code is time-consuming, which is where some of the saved time goes.

{{< key-point >}}Generated assertions can encode the same mistaken assumption as the generated code, so someone who wrote neither has to read them.{{< /key-point >}}

Kiro's answer is to write a lot of tests and validate against them, and in practice that works well enough most of the time if you stuff it with context and documentation. It only counts as quality control if someone reads the tests.

## The trust wall {#trust-wall}

Testing is the check you give away by choice. The next one is built into the tools.

#### Why won't your security team approve agent mode?

Because the context that makes these tools good is the context you aren't allowed to send out.

{{< key-point >}}Mark Shine: "with Cursor, it can go through perspectives of echoing out your environment variables that can contain secrets and API tokens", and from there to a model provider's data centre.{{< /key-point >}}

That's the divide, and it's structural. These tools work well on context, and the only way they get it is if you hand it over. Whether you want to is the wall. The objection isn't paranoid: a business asking whether Microsoft or another provider ends up holding its intellectual property is asking a contract question.

Part of the answer is older than agents. Developer hygiene still applies: keep production tokens out of the shell you develop in, and don't leave a long-lived key in `.env` next to the code an agent reads. Issue short-lived credentials scoped to the task instead, because a secret that was never in the environment needs no control. It happens anyway, which is the point. Before agent mode, a leaked variable had to be exfiltrated by something. Now there's a process in that shell whose job is to read everything nearby, and `echo $AWS_SECRET_ACCESS_KEY` is a reasonable thing for it to try while debugging why your credentials don't work.

The controls under that are real, if you know which layer each sits at. `.cursorignore` keeps a file out of the index, so it isn't sent to the model as context. That isn't an access control: a command running outside the sandbox can still `cat` the file. [Cursor's agent sandboxing in 2.0](https://forum.cursor.com/t/agent-sandboxing-available-in-cursor-2-0/139449), announced in October 2025, narrows that gap where it applies. On macOS, with the sandbox setting on and a model that supports it, a sandboxed command writes only to your workspace and `/tmp`, reaches no network unless you enable auto-run network access, and cannot open `.cursorignore`d files. It can still read the rest of the filesystem, an unsupported model falls back to the older allowlist, and stepping outside needs your approval. Filesystem permissions sit under all of it: if the agent's account cannot read `~/.aws/credentials`, no prompt talks it into doing so.

The story widens as the tools spread. We expect the next wave of leaks to come from volume: people who were never going to build software now can, in healthcare, in schools, in places that lack apps. Those apps will collect data and run for ten years. We cleared a similar hurdle with PHP and MySQL web apps, slowly and with plenty of breaches.

## The outage everyone blamed on AI {#aws-outage}

That volume argument gets used in the other direction too, with the big providers as the example, and that's where it's worth checking the story against the record.

#### Did AI cause the October 2025 AWS outage?

The documented cause is a race condition in DynamoDB's DNS automation, and the report offers no evidence for the AI-replacement story.

One version going round at the time held that Amazon had replaced a large share of its DevOps staff with AI days before us-east-1 fell over. [AWS's own post-event summary](https://aws.amazon.com/message/101925/) describes something else. DNS failures for the DynamoDB endpoint started at 11:48 PM PDT on 19 October, engineers identified DynamoDB's DNS state as the source by 12:38 AM, and a second window of Network Load Balancer errors driven by health check failures ran until 2:09 PM on 20 October. The trigger was a latent race condition in the DynamoDB DNS management automation, where one process applied a stale plan over a newer one. The summary does not mention staffing or AI at all. On the timeline, [Amazon announced 14,000 corporate role reductions](https://www.aboutamazon.com/news/company-news/amazon-workforce-reduction) on 28 October, after the event, and gave its reasons as reducing bureaucracy, removing layers and shifting resources, with AI as strategic context rather than as the stated cause.

The intuition underneath the story is still worth keeping. A tremendously skilled pool of engineers was not enough to keep a latent race condition out of the automation that manages production DNS. Automation with the power to change production carries that risk whether or not a model is involved, and an agent holding credentials is the same shape.

## The work nobody had time for {#overdue-work}

Correcting the blame doesn't settle where the value is, and in practice it shows up in the work that was never going to get done otherwise.

#### What is AI reliably good at right now?

The overdue work, and the first pass on a review.

The clearest wins look boring. A rate limiting feature that a codebase had needed for five years, with no developer ever free to write it, got built in a day and took another couple of hours of iterating and feeding the problems back. Open source modules behave the same way: someone opens an issue, a background agent goes and does what it says, and twenty minutes later there's a pull request. Security fixes, package updates and the long tail of maintenance fit this shape exactly, which is why GitHub and others aimed their first agents there.

Review is the other one, because it's the last thing anyone wants to do. [Copilot code review went generally available on 4 April 2025](https://github.blog/changelog/2025-04-04-copilot-code-review-now-generally-available/) and can be requested as a reviewer on any pull request. [Cursor's Bugbot left beta in July 2025](https://cursor.com/blog/bugbot-out-of-beta) after reviewing over a million PRs, with more than half the issues it flagged fixed before merge. Run the machine first, then put a human on the questions worth human attention. [GitHub says plainly](https://docs.github.com/en/copilot/concepts/agents/code-review) that the review can miss problems and that you stay responsible for the change.

Here's a concrete catch. A Terraform change was spinning up GKE, wiring it to Google's Secret Manager, and creating the service account that bridges the two. Our implementation downloaded a kubeconfig and used `local-exec` to apply the Kubernetes side, which runs the command on whatever machine happens to be running Terraform. It didn't have to: the [Kubernetes provider accepts connection credentials directly](https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs), and [HashiCorp's own documentation](https://developer.hashicorp.com/terraform/language/resources/provisioners/syntax) says to exhaust the alternatives before reaching for a provisioner, because Terraform can't model what one does. Copilot's review caught it and said so. Without that, it would most likely have been merged, and the bug would have arrived later as works on my machine.

{{< key-point >}}An AI reviewer is worth having because it can catch mistakes before a human reads the change. The human still owns the whole change.{{< /key-point >}}

## Nobody shares what works {#culture}

So the tools pay off in places, and plenty of teams still aren't getting that payoff, which is a people problem rather than a tooling one.

#### Why isn't your team using the tools you bought?

Because three different kinds of non-adoption are sitting on the same four-person team, and nobody is talking about any of them.

There's the developer who knows their way, is genuinely good at it, and wants to be left alone. There's the camp that holds the whole thing is unreliable and only produces bugs. And there are the people who can't be bothered with another shiny tool while they have delivery to do. All three can be in one standup.

Talking about it is harder than it sounds. Deliberately talking AI down to avoid overhyping it has a side effect: once the lead sounds negative, the people getting value out of it stop mentioning it, and the conversation polarises into two camps that no longer exchange anything. Meanwhile plenty of people have quietly worked out pre-prompts and configuration that make their own results better, and nobody in the company has heard about any of it.

{{< key-point >}}Mark Shine: "who's got good prompts? Let's talk about how you have learned stuff that's better for coding for our company."{{< /key-point >}}

The precedent for fixing this isn't new. Ericsson ran a security master role inside teams. People cringed at the title, and the reasoning still held. Security knowledge is thin and unevenly spread, so you plant someone in each team and make it a discussion point. The AI engineer role being advertised now is undefined enough that it could be this, if the job is spreading what works rather than owning the prompt cache. [The 2025 DORA report](https://dora.dev/research/2025/dora-report/) lands in the same place from the data side: AI amplifies an organisation's existing strengths and weaknesses, and the returns come from the surrounding system rather than the tool.

What that looks like in practice is a golden path. The people with time to experiment work something out, and it then trickles down to the ones who are busy delivering, as a default that works on day one. The failure mode on the other side is standardisation: hand everyone one approved assistant, forbid the rest, and you kill the experimentation that produced the path in the first place. The version worth copying is the usual one. This path works, and it's fine if you go around it, as long as you come back and say what you found.

## What this means for teams {#what-this-means}

The polarised argument about whether AI replaces developers is not the one that will cost you anything this quarter. What will cost you is an agent with more reach than its brief, in a codebase where the tests meant to check the agent were written by it, on a team where nobody compares notes.

Four things worth doing this week:

- **Measure the delivery side, not the authoring side.** You already have the data in your pipelines: lead time, rework, change failure rate. Compare before and after, because in METR's trial self-reported speed and measured time differed by about 39 percentage points.
- **Pull production credentials out of any shell an agent runs in.** List what's in the environment, in `.env` files and in your kube contexts, scope what's left to the task, and back it with filesystem permissions rather than with an index exclusion alone.
- **Make the AI the first reviewer and keep the human accountable for the change.** Add it as a reviewer on the repositories where the cost of a missed `local-exec` is highest, and give the human the design questions it can't ask.
- **Run one meeting where people show their prompts and configuration.** Ask who got something working, write down the two or three defaults worth copying, and publish them as a path rather than a policy.

## Common questions, answered {#faq}

### How do I stop a coding agent from sending my secrets to a model provider? {#faq-agent-secrets}

Take the secrets out of the environment the agent runs in, then enforce what's left with controls the agent can't talk its way past. Use short-lived credentials, keep production tokens out of development shells, and set filesystem permissions so the agent's account cannot read credential files at all. The layers differ: an ignore file keeps a file out of the index, and a terminal command running outside the sandbox can still read it. Sandboxing the agent's shell is the separate setting that blocks that read, where it applies. Cursor's sandbox covers macOS, needs the setting on and a supported model, and falls back to the older allowlist otherwise. Then check what the vendor retains and where it processes, as a contract question rather than a product one.

### Can an AI reviewer replace a human reviewer on pull requests? {#faq-ai-review}

No, but it should go first, because it can catch mistakes before a human reads the change. It picks up mechanical problems well: a provisioner that shouldn't be there, a missing error path, a credential in a diff. GitHub says its own review can miss problems, and the person merging stays responsible for the change. Request it automatically, then have the human review what it flagged plus the questions it can't ask: whether the design fits how your team works, and whether the feature was worth building.

### Should we standardise on one AI coding tool for the whole team? {#faq-standardise}

Publish a default instead of mandating one. Collect the prompts and configuration from the people already getting results and ship them as a golden path, so someone with no time to experiment starts from a setup that works. Handing everyone one approved assistant and forbidding the rest kills the experimentation that produced the path in the first place.

## Related episodes {#related-episodes}

- [Episode 90, K8s vs Managed Services: Cost, Lock-In, and Reality](/episodes/090-k8s-vs-managed-services-cost-lock-in-and-reality/) is Mark Shine's previous appearance, on when a managed service beats a cluster and what the choice costs in team skills.
- [Episode 84, AI for DevSecOps: Current Wins and Ongoing Gaps](/episodes/084-ai-for-devsecops-current-wins-and-ongoing-gaps/) covers the security side of the same question: what AI genuinely helps with today and where the tools still fall short.
- [Episode 82, Tools, MCPs, and Attack Scenarios](/episodes/082-tools-mcps-and-attack-scenarios/) is the longer version of the trust wall argument, on how LLMs use tools and how an attacker gets at them.
- [Episode 81, Keeping Secrets Safe](/episodes/081-keeping-secrets-safe/) is the hygiene checklist underneath the environment variable problem: which secrets are at risk and which tool fits which consumer.

## Resources {#resources}

- [METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) is the trial itself, including the limitations that keep it from generalising.
- [The Effects of Generative AI on High-Skilled Work](https://www.microsoft.com/en-us/research/publication/the-effects-of-generative-ai-on-high-skilled-work-evidence-from-three-field-experiments-with-software-developers/) is the larger randomized evidence on the other side, measuring completed tasks rather than time per task.
- [Cursor, Cursor 2.0](https://cursor.com/blog/2-0) settles what Composer claims on speed, and the [agent sandboxing announcement](https://forum.cursor.com/t/agent-sandboxing-available-in-cursor-2-0/139449) draws the boundary a sandboxed command works inside on macOS with a supported model: it reads your filesystem, but writes only to your workspace and `/tmp`, reaches no network unless auto-run network access is on, and cannot open `.cursorignore`d files. An unsupported model gets the earlier allowlist instead.
- [AWS, Introducing Kiro](https://kiro.dev/blog/introducing-kiro/) is the spec-driven workflow in the vendor's own words: requirements, design, tasks and the tests attached to each.
- [AWS, summary of the DynamoDB service disruption in us-east-1](https://aws.amazon.com/message/101925/) is the timeline and the cause, which is what to check the outage stories against.
- [HashiCorp, Terraform provisioners](https://developer.hashicorp.com/terraform/language/resources/provisioners/syntax) explains why `local-exec` is the option you reach for after exhausting the alternatives.
- [Stack Overflow, 2025 Developer Survey](https://stackoverflow.co/company/press/archive/stack-overflow-2025-developer-survey/) has the adoption and trust numbers moving in opposite directions.
- [Ankra](https://www.ankra.io/) and [Mark Shine on LinkedIn](https://www.linkedin.com/in/markshine/), for the Kubernetes platform work behind his side of this.

## Since we recorded {#since}

METR's [follow-up of 24 February 2026](https://metr.org/blog/2026-02-24-uplift-update/) calls the result inconclusive: developers increasingly declined to join a study that made them work without AI, so selection effects run through the new numbers. GitGuardian's [State of Secrets Sprawl 2026](https://www.gitguardian.com/state-of-secrets-sprawl-report-2026), out in March 2026, puts a number on a different mechanism. Public GitHub commits co-authored by Claude Code leak secrets at roughly twice the baseline rate, which measures what gets committed rather than what an agent sends to a provider. [Episode 111, AI Agents: When Helpful Becomes Harmful](/episodes/111/) examines another risk: agents taking actions beyond the user's request.