Kubernetes Cost Optimization: What I Actually Do on Production Clusters

Kubernetes makes infrastructure costs opaque in a way that even experienced cloud engineers underestimate. EC2 or Compute Engine instances are a simple billing model: you run a machine, you pay for it. Kubernetes adds a layer of abstraction — the scheduler decides which workloads run on which nodes, resource requests and limits determine how tightly packed the cluster is, and the gap between what you provisioned and what your applications actually use is where money disappears.

This is what I actually do when a client's Kubernetes spend is higher than it should be, based on real production clusters on GKE and EKS. Not theoretical recommendations — the specific interventions that have reduced bills meaningfully.

---

Understanding Where Kubernetes Money Goes

Before optimizing anything, I get a clear picture of the cost breakdown. The billing model for managed Kubernetes differs between providers:

GKE: Standard mode charges per node (e2-standard-2 at ~$0.067/hour, for example). The GKE cluster management fee was eliminated for the first cluster in a project as of 2023; additional clusters in the same project still incur a management fee. GKE Autopilot charges per pod resource request (CPU and memory), which is higher per unit than Standard but eliminates idle node capacity cost.

EKS: Charges $0.10/hour per cluster plus the underlying EC2 instance costs for nodes. For small clusters, the $0.10/hour cluster fee (~$72/month) is a meaningful fixed cost relative to node cost.

In both cases, the actual cost drivers are:

1. Node count and size — the biggest lever
2. Storage costs — Persistent Volume Claims in non-optimal storage classes
3. Networking costs — cross-AZ data transfer between pods on different nodes
4. Idle cluster overhead — dev/test clusters running 24/7 without need

I look at these in that order, largest to smallest.

---

Step 1: Node Pool Rightsizing

The most common waste pattern I find is a cluster sized for peak load running continuously at average load. A cluster with three e2-standard-8 nodes (8 vCPU, 32 GB RAM each) serving a workload that peaks at 12 vCPU for two hours per day and averages 4 vCPU the rest of the time is paying for peak capacity around the clock.

How I assess this:

# Get actual CPU/memory usage across all nodes (GKE)
kubectl top nodes

# Compare to what's been requested
kubectl describe nodes | grep -A5 "Allocated resources"

The difference between kubectl top nodes (actual usage) and allocated resources shows the buffer. On most production clusters I see 40–60% of allocated CPU sitting idle. This is not necessarily wrong — you need headroom for spikes — but the right amount of headroom is a function of your traffic variability, not a static factor of 2×.

Vertical Pod Autoscaler (VPA) is the tool I use to get data-driven rightsizing recommendations:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-service-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-service
  updatePolicy:
    updateMode: "Off"  # Recommendation only, no automatic updates

In Off mode, VPA collects historical metrics and surfaces recommendations without applying them. After 7–14 days of data collection:

kubectl describe vpa my-service-vpa

The output shows current resource requests vs. recommended requests based on actual usage. Typical result: a deployment with resources.requests.cpu: 500m that actually uses 80m–120m at peak. Rightsizing to 150m request releases node capacity for scheduling other pods.

I apply VPA recommendations manually after review, not automatically. Automatic VPA application involves pod restarts that need to be coordinated with deployment windows.

---

Step 2: Spot / Preemptible Nodes for Non-Critical Workloads

Spot instances (AWS) and Spot VMs / Preemptible VMs (GCP) offer 60–90% discount on compute costs at the cost of potential interruption. For workloads that are stateless and can tolerate interruption, this is the highest-leverage cost reduction available.

What runs on spot/preemptible in my clusters:

• CI/CD runners (restartable, ephemeral)
• Batch processing jobs
• Development and staging workloads
• Background workers with retry logic

What does not run on spot:

• Stateful workloads (databases, caches) where restart is disruptive
• Primary replicas of critical services
• Anything without a retry/restart mechanism

GKE configuration for a spot node pool:

# Spot node pool alongside a regular node pool
nodeConfig:
  spot: true
  machineType: e2-standard-4
  taint:
  - key: cloud.google.com/gke-spot
    value: "true"
    effect: NO_SCHEDULE

The taint prevents non-spot-tolerant workloads from landing on spot nodes. Deployments that should use spot nodes are given a matching toleration and an optional node affinity preference:

tolerations:
- key: cloud.google.com/gke-spot
  operator: Equal
  value: "true"
  effect: NoSchedule
affinity:
  nodeAffinity:
    preferredDuringSchedulingIgnoredDuringExecution:
    - weight: 100
      preference:
        matchExpressions:
        - key: cloud.google.com/gke-spot
          operator: In
          values:
          - "true"

Handling preemption gracefully:

The cluster needs a PodDisruptionBudget on critical deployments so the scheduler doesn't drain all replicas simultaneously during a preemption event:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: my-service-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: my-service

Without PDBs, a spot node being reclaimed can cause all pods of a deployment to be rescheduled simultaneously, causing an outage even if there are replicas on regular nodes.

---

Step 3: Namespace-Level Cost Attribution

On multi-team clusters where multiple services share nodes, cost attribution is the prerequisite for meaningful optimization conversations. "The Kubernetes bill is high" is not actionable. "The data pipeline namespace is consuming 38% of cluster resources and contributing roughly $1,200/month to the infrastructure bill" is actionable.

OpenCost or Kubecost provide this attribution. For GKE clusters I use the hosted Kubecost integration or self-hosted OpenCost:

helm install opencost opencost/opencost \
  --namespace opencost \
  --create-namespace

After installation and 24–48 hours of data collection, namespace-level cost reports show which teams or services are driving which costs. I deliver these reports monthly alongside the standard cost review.

The conversation that follows is typically: "The analytics namespace costs more than the API namespace, but it's running jobs that ran fine for three years. What changed?" The answer is often a new data source or a changed query pattern that caused job parallelism to spike. Namespace-level attribution makes this visible; aggregate cluster billing doesn't.

---

Step 4: Cluster Autoscaler Configuration

Cluster Autoscaler adds and removes nodes based on pod scheduling demand. Misconfigured autoscaler settings are a common source of both over-provisioning and unexpected costs.

Common misconfigurations I find:

• --scale-down-utilization-threshold set too high (default 0.5 = 50% utilization required before scale-down). On clusters with many small pods, utilization often never drops below 50% at node level even when significant spare capacity exists. Lowering to 0.3–0.4 allows more aggressive scale-down for bursty workloads.

• --scale-down-delay-after-add set to default 10 minutes. For batch workloads that add nodes for a job and then want to scale down, 10 minutes is fine. For interactive workloads with bursty patterns, nodes might be added and then not needed 15 minutes later — the 10 minute delay keeps them running unnecessarily.

• No node pool minimum set. Without a minimum, autoscaler will scale to zero for non-critical node pools — which is sometimes desirable (overnight scale-down) and sometimes not (production workloads need headroom to start up before the first node is scheduled).

I review autoscaler events with:

kubectl describe configmap cluster-autoscaler-status -n kube-system

This shows why the autoscaler is or isn't scaling down, including which pods are blocking scale-down (often pods without PDBs, pods with local storage, or DaemonSets).

---

Step 5: PVC Storage Class Audit

Persistent Volume Claims created in older clusters often use default storage classes that aren't cost-optimal. On GKE, the default storage class has historically been standard (backed by HDD Persistent Disk), but standard-rwo (backed by balanced SSD PD) became default in newer clusters. Meanwhile, premium-rwo (SSD PD) costs more than either.

A common pattern I find: a PVC created for a Postgres database three years ago on premium-rwo for performance requirements that no longer apply (the database was migrated to Cloud SQL and this PVC is now used for less critical data), now paying SSD pricing for HDD-appropriate workloads.

Audit:

kubectl get pvc --all-namespaces -o custom-columns=\
NAMESPACE:.metadata.namespace,\
NAME:.metadata.name,\
STORAGE:.spec.resources.requests.storage,\
CLASS:.spec.storageClassName

Migration to a cheaper storage class requires creating a new PVC, copying the data, and updating the deployment to reference the new PVC. Not a five-minute change, but on large clusters with many old PVCs, worth auditing.

---

The Client Conversation Around Kubernetes Costs

The hardest part of Kubernetes cost optimization is not technical — it's explaining why a cluster that "just runs our services" costs what it does.

I use this framework when presenting optimization opportunities:

1. Show the cost attribution first. Before recommendations, show who/what is generating cost. Clients accept optimization recommendations more readily when they can see where their money is going rather than receiving a list of "things to change."

2. Separate required costs from waste. A cluster sized for peak traffic with appropriate headroom is not wasted money. A cluster sized for a peak that happened once during a load test three years ago and hasn't recurred is. Be specific about which category each recommendation falls into.

3. Quantify the change effort. "Enable spot nodes for the CI pool" has a different implementation effort than "migrate the primary database to a smaller node pool." Clients make better decisions when they can see the savings per hour of implementation work.

4. Sequence recommendations by risk, not savings. The biggest savings opportunities often involve changing workload configurations or storage classes — changes with real rollback risk. I deliver low-risk wins (VPA rightsizing, autoscaler tuning) before higher-risk changes (spot migration, storage class changes). This builds trust and demonstrates care before asking clients to approve riskier modifications.

---

Resources

• OpenCost — open-source Kubernetes cost monitoring
• Kubernetes documentation — Vertical Pod Autoscaler
• GKE — spot VMs documentation
• EKS — managed spot node groups
• Cluster Autoscaler — FAQ and configuration reference
• AWS — Karpenter node provisioner (alternative to Cluster Autoscaler)

---

Monitoring Kubernetes Costs Proactively

Reactive cost reviews — looking at the bill after it arrives — miss cost spikes that develop and resolve within a single month. A three-day GPU experiment that someone forgot to terminate might spend $300, get noticed when the resource is finally cleaned up, and never appear in a monthly anomaly analysis because total month spend looked normal.

Proactive cost monitoring for Kubernetes requires:

Daily cost tracking with alerting. OpenCost or Kubecost can be configured to alert when namespace-level spend exceeds a daily threshold. For a cluster where the analytics namespace averages $40/day, an alert at $80/day catches runaway jobs the same day they start — not 30 days later.

Resource quota enforcement per namespace. Kubernetes ResourceQuotas prevent namespaces from consuming more than their allocated share of cluster resources:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: analytics-quota
  namespace: analytics
spec:
  hard:
    requests.cpu: "40"
    requests.memory: "80Gi"
    limits.cpu: "80"
    limits.memory: "160Gi"
    persistentvolumeclaims: "20"

A quota doesn't prevent cost surprises from within the allowed range, but it prevents one namespace from consuming the entire cluster's capacity — which is the failure mode that generates the largest unexpected bills.

Tracking cost per deployment, not just per namespace. Namespace attribution is team-level granularity. For clusters with large namespaces shared by multiple services, deployment-level attribution shows which specific service is the cost driver. Kubecost provides this at no additional setup cost once the base installation is running.

---

When to Choose GKE Autopilot vs. Standard Mode

The choice between GKE Autopilot and Standard mode has cost implications that aren't obvious from the feature comparison.

Standard mode charges per node. You provision nodes, you pay for them regardless of pod utilization. With good autoscaling configuration, you can approach high utilization — but idle node time is always some cost.

Autopilot charges per pod resource request (CPU and memory), not per node. Google manages the node infrastructure. You pay for what you request, not for what you provision. For clusters with highly variable workloads — batch jobs that run overnight, event-driven pipelines that are idle during business hours — Autopilot can be significantly cheaper because you're not paying for node idle time.

The crossover point depends on utilization. A cluster that runs at 80%+ utilization on Standard mode may cost less in Standard than Autopilot, because Autopilot pricing per resource unit is higher than Standard node pricing per resource unit. A cluster that runs at 30–40% utilization — common for dev and staging clusters — often costs less in Autopilot.

I recommend running the cost comparison before migrating an existing cluster. GCP provides a cost estimator, and Autopilot has some constraints (no privileged containers, specific CPU/memory ratio requirements) that may affect workloads.

---

Resources

• OpenCost — open-source Kubernetes cost monitoring
• Kubernetes documentation — Vertical Pod Autoscaler
• GKE — spot VMs documentation
• EKS — managed spot node groups
• Cluster Autoscaler — FAQ and configuration reference
• GKE Autopilot overview
• Karpenter — next-generation node provisioner for EKS