Nobody has a number when they say “at our scale”

Someone tells you the approach won’t work at your scale. Ask them how many requests per second they serve. It’s the question we put to people who say they process too much to manage, and it’s a fair one: “at scale” is either a measurement or a feeling, and the number is what tells you which.

Guessing costs money. It’s where a service that would run on one box gets broken into a distributed system, and where a cloud bill grows for reasons nobody can point at. Social media doesn’t help: there’s always a company posting how many nodes it runs, and most people reading it don’t need that fluff.

The word is doing two jobs, which is why the argument never resolves. Scale is a number, a threshold your traffic has crossed or hasn’t. Scaling is a mechanism, capacity that moves up and down with load while you’re asleep. You can want the second without being anywhere near the first, and teams routinely buy the first because what they wanted was the second.

Scale is a number, scaling is a reaction

Start with the two definitions, because which one you’re arguing about decides everything that follows.

What’s the difference between scale and scaling?

Scale is the load you’ve measured; scaling is what your infrastructure does on its own when that load changes.

Julien treats scale as a threshold: above 10,000 requests per second you’ve got a different class of problem, and below it an ordinary website. Andrey starts from the mechanism: scaling is the elasticity of your capacity reacting to a signal, adding servers when load arrives and removing them when it leaves, so nobody logs in at 2am to do it by hand.

Julien’s post calls 10,000 requests per second “my personal definition of ‘at scale’”, and personal is the operative word. The number is borrowed from the C10K problem, which counts something else: ten thousand connections held open at the same time, not ten thousand requests arriving every second. One connection can carry many requests or sit idle between them, so the two don’t convert, and no architecture decision follows from crossing a line drawn by analogy.

What C10K settles is narrower than it looks. Dan Kegel asked the question in 1999, nginx was written to answer it, and his arithmetic on a $1,200 machine of the day ends “So hardware is no longer the bottleneck.” The rest of his page is the limits that were left, and they’re software: how the server handles I/O, the cap on open file descriptors, thread counts and per-thread stacks. Whether your Linux box carries ten thousand concurrent connections depends on the box, how that server is configured and what each connection is doing.

The database is where this gets decided first. We’d want convincing that a single instance serves 10,000 concurrent connections, which is why read replicas and a cache turn up early, long before the application tier needs anything clever.

Autoscaling has prerequisites, and that’s the expensive part

Say you want the mechanism rather than the threshold. It doesn’t arrive on its own.

What has to be true before an autoscaler can do anything?

Traffic has to be distributable across identical instances, and something has to decide when to add or remove one.

One EC2 instance with an A record pointing at its IPv4 address can’t autoscale. The primitives come first: instances in an Auto Scaling group, a load balancer in front, a metric the group watches, and something that acts on it. On AWS that’s the group plus a CloudWatch alarm. On Kubernetes it’s the HorizontalPodAutoscaler, a controller that reads pod metrics and changes the replica count on a deployment. Where a platform hands you neither, you write that logic yourself, and it’s harder than “more load, more servers” sounds.

The difficult half is down, and it’s timing. Scale in too eagerly and the next spike finds you empty. React too twitchily and you pay for instances that boot, serve nothing and terminate. EC2 Auto Scaling has two answers, and they belong to different policy types. A simple scaling policy pauses the group after it acts, 300 seconds by default, so the last change reaches the metric before the next decision; a scale-in policy can take a shorter one, because terminating is quicker than launching. Target tracking and step scaling can scale out immediately, without waiting for a cooldown to end, though a scale-in can still be delayed while the group is in one. What they add is an instance warmup: until it expires the instance stays out of the group’s aggregated metrics, and scale-in is blocked while a scale-out is still warming up. Keeping a starting instance out of the average is the point: what it reports while it boots describes initialization, not how it serves traffic. Kubernetes has the same idea as a stabilization window on scale down.

Leave room between the thresholds too: AWS advises an adequate margin between scale-out and scale-in, or the group flaps.

Sticky sessions change what the new capacity can do for you. ELB has routed repeated requests to the same instance “whenever possible” since 2010, so a new instance still picks up new sessions. What it can’t pick up is the load already pinned elsewhere, so an imbalance drains at the speed sessions end rather than at the speed you add capacity. Scale-in is the sharper edge: terminate an instance and the sessions it was holding locally go with it. Getting state out of the instance answers both, which is why it sits under all the other prerequisites.

Vertical first, distributed when you have to

The cheapest way to meet those prerequisites is often to not need them yet.

When should you add RAM instead of adding instances?

First, almost always, because a bigger box is a far smaller change than a distributed system.

Scaling a monolith vertically is an order of magnitude easier than building a distributed system. In a monolith a call is a function call, so you’re fairly sure it happens. Over a network you have a packet and a hope. Going distributed means pulling state out of the application, putting the data where it belongs, pooling connections, and owning the low level networking that comes with it.

Then comes the code you never had to write before. Retries need exponential backoff with jitter, or every client hammers a struggling service in lockstep, which is the thundering herd. A service already answering 429 Too Many Requests (RFC 6585) doesn’t need you asking again a millisecond later, and you want a circuit breaker that stops sending traffic to something returning errors.

Basecamp is the long-running counterexample to the rewrite instinct. DHH wrote in 2009 that faster hardware let them punt on sharding, repeated it three years later, and said in February 2020 that they’d “dodged the database sharding bullet for almost twenty years”. They fix performance problems when the problems occur, and that works because they measure.

The ceiling is real, which is the argument for thinking in years. Vertical scaling ends: eventually no vendor sells a box big enough. We know of a company that asked every vendor for more RAM and was told to come back in a year. AWS sells 24 TiB in a single instance, so the ceiling is high, and it’s still a ceiling. Hit it and you start re-engineering with your growth already capped, which is the worst moment to begin.

Sometimes the thing to scale is the organisation

Hardware isn’t the only thing with a ceiling in it.

What do you do when the software is fine and delivery still crawls?

Count the people and the lines between them, because that’s the system that’s saturated.

We don’t know of an organisation with 150 developers on a single monolith that is still quick at shipping features. Past some headcount the constraint sits in the coordination, not the code.

The cost shows up on the cloud bill. Corey Quinn put it as a myth and a fact in February 2020: the myth is that your AWS bill is a function of how many customers you have, and the fact is that it’s a function of how many engineers you have. Every team brings its own pipeline, its own language and its own ecosystem to interact with. Teams on the same open plan floor still don’t talk to each other, because seating isn’t collaboration.

Adding people to a problem is the move that looks like scaling and isn’t. Some call it the meat cloud: more nodes, except the nodes are human, and human communication bandwidth is low enough that synchronising ten of them can cost more than the work. A small, tight team evolving the software in the direction the business needs beats a large one evolving it in the direction of somebody’s CV. Being big is hard and painful, and if you’ve got a choice, don’t be big.

CPU is the metric you get, not the one you want

Whatever you’re scaling, the signal you scale on decides how well it works.

Why is CPU a weak signal?

It tells you the box is busy and nothing about what it’s busy doing.

We still reach for it first in an incident, and we’d defend that: time to recovery is the number that counts while customers are watching, so you throw hardware at 100% CPU everywhere, bring the service back, then find out why. RAM doesn’t come free either: on EC2 you install an agent before you can graph it.

What the profiler finds is rarely what the ticket said. Time and again the CPU is going on string manipulation. Strings are immutable in most languages, so changing one character allocates a new object, and concatenation in a loop and regular expressions do it at volume. One well written Java service we profiled spent 60% of its CPU on JSON serialization and deserialization. It was doing that by breathing. The allocations come back as garbage collection pressure, so the RAM graph moves too, and often the worst of it sits in a library you didn’t write and won’t think to profile.

Which leaves an engineering decision that’s really a budget decision. You can pay an expensive developer for a week to find the allocations, or you can add an instance. Hardware is almost always cheaper than developer time, and a cheap fix that holds is a fix: an early Node.js service we heard of, back on the 0.x releases, was restarted every ten requests because that cost less than hunting the leak.

A rewrite earns its budget when you can show the number it moves, and published numbers come with conditions. Google’s write-up of Deloitte’s mobile speed study reports 8.4% higher conversion on retail sites from a 0.1 second improvement. The measurement ran over 30 days at the end of 2019, on mobile journeys. It also took all four measured metrics, First Meaningful Paint, Estimated Input Latency, Observed Load and Time to First Byte, improving by that 0.1 seconds on every page of the funnel. That is what the study observed, not a return your site can book in advance. Measure your own funnel and you have a business case; without one you’re speculating, which means you don’t have the information to build a strategy on.

Turning requests per second into pods

CPU is a proxy for what you care about. Measure the thing itself.

How do you scale on requests per second?

Count requests at the proxy, load test one instance to find where it saturates, then target below that.

Envoy is the counting half: a proxy in front of your application that reconfigures without a restart, with its own security problems, and exports Prometheus metrics. Pick the counter you mean. Its HTTP connection manager counts several things under http.<stat_prefix>.: downstream_cx_total is connections, downstream_rq_total requests, downstream_rq_active the requests in flight, downstream_rq_time the duration histogram.

Those are Envoy’s internal stat names; the scrape renames them. http.<stat_prefix>.downstream_rq_total arrives in Prometheus as envoy_http_downstream_rq_total, the prefix lifted out into an envoy_http_conn_manager_prefix label. The value is yours: Envoy’s own example config sets stat_prefix: ingress_http. With stat_prefix: ingress on the connection manager in front of your application, the arrival rate is rate(envoy_http_downstream_rq_total{envoy_http_conn_manager_prefix="ingress"}[1m]). Match your own prefix or the query returns no series; drop the matcher and you scale on the admin interface too.

Counting is the easy half. The HorizontalPodAutoscaler doesn’t read Prometheus: it reads the Kubernetes metrics APIs, so something has to serve custom.metrics.k8s.io, and the Prometheus Adapter is the usual bridge. Its arithmetic is a ratio, desiredReplicas = ceil[currentReplicas * (currentMetricValue / desiredMetricValue)], where the current value is your rate averaged across pods.

The target is per pod, and it isn’t the number the load test gave you. The test finds capacity, the rate where latency and errors start to climb; the target goes under it. That gap pays for a pod’s slow start, an average minute against your worst, and a pod dying at a bad moment. Past the tipping point, clients time out and retry, piling more load on a service that already has too much.

Here we don’t agree. Julien would put Envoy in and scale on the real metric: above his own threshold a service doesn’t get maintenance windows. Andrey’s objection is setup cost: CPU is already there, 80% up and below 80% down, no experiment and no proxy to operate. What settles it is whether the signal tracks what hurts. CPU tracks CPU-bound work. Arrival rate tracks demand, and misses a saturated dependency: the same requests arrive and take longer, so the rate holds flat while they queue. downstream_rq_active and downstream_rq_time move then.

Run the experiment where the data is real: a temporary cluster won’t behave like production, chaos engineering will, and working monitoring is its entry requirement. SRE’s error budget pays for it: a quiet month buys you a node going away on purpose.

Predicted load, and the day the prediction is wrong

Reacting to load is one approach. Forecasting it is the other, and it’s where the AIOps pitch lands.

Where does predictive scaling fall over?

On the days that don’t look like your history: a launch, a viral post, a Black Friday unlike last year’s.

The idea is sound and the data is already there: a cluster running Prometheus collects health and load metrics; feed the history to a model and it forecasts the curve, so you buy capacity ahead of the traffic instead of chasing it. For a company running thousands of servers, cutting a cluster in half for half a day is real money, which is why an industry sells this.

AWS shipped it in November 2018. Predictive scaling needs at least one day of history, re-evaluates every 24 hours and forecasts the next 48, and the announcement is honest about the limit: it’s “not designed to help in situations where spikes in load are not cyclic or predictable”. That matches the couple of commercial tools we tried, which is a small sample rather than a survey. A repeating cycle gets predicted well: an online shop in one country, open from 8am to 8pm, is the easy case. The forecast is drawn from the history, so it handles the shape that recurs, and a new offering or an event with no precedent in the data is where it comes up short.

So keep reactive scaling under the forecast, with headroom. For an event you know is coming, AWS Support will plan it with you through Infrastructure Event Management, which the 2019 Prime Day write-up points at: colleagues “ready (and eager) to help you to plan for your large-scale product or application launch”. Prediction and anomaly detection are two different algorithms over the same metrics, and the second flags traffic you’ve never seen before, which is a security question as much as a capacity one.

Logging is not monitoring

All of this assumes you’re collecting something you can scale on, and most teams are collecting the wrong thing in the wrong place.

What belongs in a log and what belongs in a metric?

Anything you’d draw a graph of belongs in a time series database; logs keep the records you read one at a time.

A log is a text file on a disk somewhere. On-prem that meant the disk filled, almost predictably, and somebody got paged in the night because a developer added a debug line and forgot to remove it. The cloud doesn’t repeal that. Docker’s default json-file driver performs no rotation unless you configure it, which the logging documentation says “can lead to disk space exhaustion”. What a managed logging service changes is where the growth lands: on an invoice, which is how a logging bill reaches the order of millions without anyone deciding to spend it.

Prometheus stores metrics as time series with labels and gives you a query language over them. Its own documentation is clear about the edge: “If you need 100% accuracy, such as for per-request billing, Prometheus is not a good choice.” Bill from something else, and keep Prometheus for the monitoring.

Two levels, errors in production and debug on your laptop, is the short version, and too blunt to follow literally. A successful login, a privilege change, an administrative action: none is an error and all are records. NIST SP 800-92 treats them as security log data to store and review, and your auditor will have a view too.

You can alert on patterns in logs, on an error rate per minute, and that’s as close to monitoring as logs get. It’s also where most teams stop, which is why the metrics you already collect are the most undervalued data in the cluster. A time series holds a month of request durations in a fraction of the space the logs would, with a query language over it, so that is where “this call got slower” gets asked in practice. Where the time goes belongs to tracing, an id carried from the first service to the second to the third, and Google’s 2010 Dapper paper is where today’s tracing tools get their shape. The metric says the chain got slower, the trace says which link, and optimising either side of a bottleneck moves nothing.

What this means for teams

Both halves of the word come down to the same habit: measure before you architect.

This week:

• Find your actual peak requests per second and write it down. It’s what the next conversation that opens with “at our scale” needs.
• Load test one instance or pod of your busiest service until latency and errors climb. That rate is your capacity, not your target: set the policy under it, with room for a slow start, a bad minute and a lost instance.
• Check which policy type your group is on, then set the cooldown or the instance warmup deliberately instead of taking the defaults, and leave a margin between the scale-out and scale-in thresholds.
• Cut the DEBUG output and the metric-shaped lines from production logging, keep the records you need for audit, and put the countable ones in a time series database. Check what that does to next month’s bill.

Common questions, answered

Do I need a distributed system to handle 10,000 requests per second?

Load test one instance before you design anything, because 10,000 is a personal rule of thumb, not a property of your application. The figure is borrowed from C10K, which counts ten thousand connections open at once, not ten thousand requests a second. What one box carries depends on the hardware, the server implementation, how it is configured and what each connection is doing. Check the database early: one instance serving that many concurrent connections is what we’d want convincing of, and read replicas and a cache are the usual answer.

How do I stop my autoscaling group from scaling up and down all day?

Leave an adequate margin between your scale-out and scale-in thresholds, then set the waiting period that matches your policy type. A simple scaling policy pauses for the cooldown, 300 seconds by default, and a scale-in policy can take a shorter one. Target tracking and step scaling can scale out without waiting for a cooldown, though a scale-in can still be delayed by one. Each new instance also gets an instance warmup, so it stays out of the group’s aggregate and blocks scale-in until it finishes. If it still flaps, read the group’s scaling activity next to the metric history, because a scaling action moves the metric that triggered it.

How do I bring down a cloud logging bill that keeps growing?

Cut the DEBUG output and the lines that are really metrics, and keep the records you need for audit and security review. Most of the volume is debug output nobody reads, and you pay to store and index all of it. Metrics with a query language over them answer the questions people were grepping logs for, at a fraction of the size, and rotation on the container log driver stops it filling a node’s disk.

Related episodes

• What Are We Working On: picking an orchestrator and what Kubernetes costs after the first deploy, including the autoscalers it ships without configuring.
• DevOps What: where the error budget comes from, and why spending part of one on an experiment is a legitimate use of it.
• Infrastructure as Code: the same instinct pointed at tooling, automate the work that needs automating and leave the rest alone.

Resources

• What scale means: Julien Bisconti’s June 2020 post, calling 10,000 requests per second his personal definition of scale.
• The C10K problem: Dan Kegel’s 1999 framing of ten thousand simultaneous clients, and the limits he lands on: server design and operating system configuration.
• Scaling cooldowns for Amazon EC2 Auto Scaling: which policies wait for a cooldown, the 300 second default, and the scale-in a cooldown can still delay.
• Using load shedding to avoid overload: why a service needs headroom under its measured tipping point.
• Envoy HTTP connection manager statistics: what downstream_cx_total, downstream_rq_total, downstream_rq_active and downstream_rq_time each count, under the internal names the Prometheus scrape renames.
• Prometheus metrics collected by the CloudWatch agent: the exported Envoy names and the envoy_http_conn_manager_prefix label you filter on.
• Predictive scaling for EC2: how much history the forecast needs, and the spikes AWS says it won’t help with.
• Milliseconds make millions: the four metrics, every page of the funnel, and the lift Deloitte measured in late 2019.

Since we recorded

KEDA was already scaling Kubernetes workloads on queue depth and event rates when this episode aired, and it graduated from the CNCF in August 2023. AWS folded predictive scaling into EC2 Auto Scaling as a native policy in May 2021. A running pod’s CPU and memory can now be changed in place, since in-place pod resize reached beta in Kubernetes 1.33. The monitoring argument got episodes of its own, Monitoring Done Wrong and the three pillars of observability.