Per-Token Inference Costs on Kubernetes: OpenCost 1.121.0

If you only read this section: Install OpenCost, label your inference workloads with opencost.io/sku and opencost.io/customer, count tokens in your inference server (vLLM, TGI, Triton), and join the two in your data warehouse on the SKU label. That's the whole article.

If you've ever had a finance team ask "what did last month's LLM traffic actually cost us?" and watched your platform engineering team go quiet, OpenCost 1.121.0 is the release you've been waiting for.

OpenCost has been the de facto standard for Kubernetes cost allocation since Kubecost donated it to the CNCF in 2023. It's the open-source core of what Kubecost sells, and the cadence since donation has been steady. It tracks CPU, memory, storage, network, and GPU allocation. What's been missing is the unit-economics layer above it: cost per inference request, cost per token, cost per customer. OpenCost 1.121.0, shipped July 20, 2026 (with 1.121.1 as the latest patch), now exposes per-token cost attribution as a first-class concept. This is the first time you can answer "this chatbot cost $0.0034 per 1k output tokens on Tuesday" with data, not a back-of-envelope calculation.

This post walks through what changed, what you need to wire it in, and where the sharp edges still are.

What was actually missing

The pre-1.121.0 story was workable for traditional workloads. A namespace running a web service has a known pod footprint, OpenCost allocates CPU and memory costs back to it, you divide by request volume or transaction count, and finance gets a number.

Inference workloads don't fit that model for three reasons:

1. GPU memory is the cost driver, not pod count. A single inference pod might consume 35 GB of an A100's 80 GB of HBM. The remaining 45 GB is idle, but the pod is paying for all 80 GB worth of hardware cost allocation if you don't carve it up.
2. Cost scales with traffic, not uptime. An inference pod idle at 3 AM costs almost nothing. An inference pod saturating at noon with 200 RPS might cost 50x more than its idle baseline. Cost-per-hour allocation is the wrong denominator.
3. Finance asks about tokens, customers, and SKUs, not namespaces. "How much did GPT-4-class traffic for ACME Corp cost us in July?" is a real question. The answer requires a join between GPU cost data, inference server metrics, and customer/SKU metadata.

OpenCost 1.121.0 doesn't solve the third problem for you. It gives you the cost data with the right granularity to solve it.

What's new in 1.121.0

The release shipped four changes that matter for AI infrastructure teams, plus two operational fixes worth knowing about:

1. AI Inference Costs v1 [PR #3845]. The headline feature, built by Sima Nadler (IBM) as Sprint 1 of a multi-sprint effort. Adds per-token and per-request inference cost attribution as a first-class concept in OpenCost. Aggregation by pod, controller, and Service; metrics and REST APIs exposed. This is the change that makes the "per-token cost attribution on Kubernetes" story actually true.

2. AI Inference usage-cost basis bug fix [PR #3901]. Companion fix to #3845 - corrects the cost basis calculation that the new feature introduced. Without this, the new feature would have produced wrong numbers in some allocation paths.

3. Allocation JSON now includes NAT Gateway costs [PR #3937]. Adds networkNatGatewayEgressCost and networkNatGatewayIngressCost to the Allocation JSON response. Important for RAG workloads and large-context inference, where egress to a vector DB or model API drives NAT gateway costs that were previously invisible.

4. NAT gateway metrics in the scrape allowlist [PR #3973]. Companion to #3937 - adds the kubecost_network_nat_gateway_* metrics to the collector-source /metrics scrape target.

Plus two operational fixes worth noting:

• GPU allocation merge bug fix [PR #3913]. Prevents GPU allocation metadata from being silently dropped during aggregation when the receiver's GPUAllocation is nil. Without this fix, GPU costs could disappear in some aggregation paths. This bug was specifically triggered by multi-tenant GPU sharing paths (HAMi, MIG), which is why it's worth flagging here even though it's a bug fix rather than a feature.
• AWS Athena query result reuse [PR #3953]. Opt-in ResultReuseMaxAgeMinutes in AthenaConfiguration reduces S3 re-scans for multi-cluster deployments sharing a CUR workgroup. Defaults to 0 (disabled), preserving existing behavior.

How to wire in per-token cost attribution

Per-token cost attribution is not a single feature flag. It's a pipeline of three pieces.

Step 1: Make sure OpenCost sees your GPU allocation

If you're running on AWS, GCP, Azure, or bare metal with NVIDIA devices, OpenCost's GPU plugin should already be detecting them. Confirm with:

If you don't see GPU detection logs, you need the NVIDIA device plugin or HAMi device plugin running on GPU nodes, and OpenCost configured with the GPU pricing source enabled.

Step 2: Tag your inference workloads by SKU

OpenCost allocation keys on labels. For per-token attribution to be useful, you need labels that map to billable units:

These labels propagate into the Allocation records. Once they're there, you can query cost by customer, by SKU, or by model. The observability stack patterns we ship for AI infrastructure are how most teams get token counts and OpenCost allocations aligned on the same labels.

Step 3: Join with inference server metrics

This is where the per-token math happens, and OpenCost doesn't do it for you. You need to:

1. Capture token counts (input + output) per pod, per request, from your inference server (vLLM, TGI, Triton, or your own service).
2. Push those token counts into a metrics system with the same labels as the OpenCost allocation (the opencost.io/sku label is a natural join key).
3. Compute cost per 1k tokens by dividing total SKU cost by total SKU tokens.

A typical SQL query for this might look like:

This example uses time_bucket, a TimescaleDB / Timescale-flavored idiom. On BigQuery, Snowflake, or ClickHouse the pattern is the same (truncate a timestamp to an hour or day bucket, then join), but the function call is DATE_TRUNC (BigQuery / Snowflake) or toStartOfHour / toStartOfDay (ClickHouse). The shape of the query is the same: OpenCost owns the cost side, your inference server owns the token side, you join them.

Where this works well

The end-to-end pipeline works cleanly when:

• You run a small number of model SKUs (under 20).
• Your inference traffic is steady enough that 5-minute or 1-hour buckets are meaningful.
• Your billing boundaries map cleanly to Kubernetes namespaces or label selectors.
• Finance's questions map to dimensions you already have labels for (customer, SKU, model, region).

This is the realistic setup for most production AI features shipping today. A handful of models, a handful of customers, batched inference at a steady cadence. OpenCost 1.121.0 nails this.

Where the sharp edges still are

Spot-priced GPU nodes and pre-emption (Karpenter, Cluster Autoscaler with spot) break OpenCost's pricing model and create phantom allocations. OpenCost's pricing assumes on-demand rates and tracks what was scheduled, not what ran or at what realized price. On a spot-heavy GPU fleet your cost numbers will be overstated by 2-4x. The fix is to wire in custom pricing that reflects your realized spot rates, and filter your token-metric join to skip pods that didn't run for at least N minutes. This is why many spot-heavy GPU fleets hand OpenCost a custom pricing config that reflects realized spot rates, not the public on-demand API.

Bursty traffic breaks the per-token math at the bucket level. If an inference pod idles at 0 RPS for 23 hours then bursts at 1000 RPS for 1 hour, your cost-per-token for that burst is artificially low (because the bucket's numerator is dragged down by the idle time). The fix is to bucket by active time windows, not calendar time, but OpenCost doesn't ship that abstraction. You'll build it on top.

Multi-tenant GPU sharing (HAMi, MIG, DRA) is still rough. OpenCost's GPU allocation assumes one pod owns the full GPU. When a pod is sharing via HAMi or MIG, the allocation gets fuzzy: is the cost the full GPU, a fraction, or per-claim? 1.121.0 fixes the data-loss bug (PR #3913) that was specifically triggered by multi-tenant GPU sharing paths - if you're on a pre-1.121.0 OpenCost, validate that your GPU allocation totals match what the device plugin reported; they may not have. But even with the fix, the data model still doesn't have native ResourceClaim awareness. Expect a manual override or a custom pricing config that splits the GPU cost across known sharing tenants until OpenCost adds native ResourceClaim awareness.

Pre-1.121.0 GPU allocation validation. Beyond the manual override path, the immediate upgrade benefit for HAMi and MIG users is real: the merge bug in #3913 could cause a 30% gap between HAMi-reported GPU usage and OpenCost-reported allocation. If you're on a pre-1.121.0 OpenCost and your GPU allocation totals look low, that's probably why.

Custom hardware (your own DC, colocation, bare metal with hand-rolled procurement) requires the custom pricing path. The public pricing endpoint helps, but the workflow is still "keep a spreadsheet of hardware costs and refresh the pricing endpoint quarterly." We do this for several clients; it's not free.

A practical adoption path

If you're starting from scratch, count on 6-8 weeks for a clean greenfield deployment. The OpenCost install itself is a day. The label discipline is a month. The data warehouse plumbing is the long pole. If you already have vLLM or TGI metrics in Prometheus and a working data warehouse, you can compress to 2-3 weeks - most of that is creating the SKU/customer label taxonomy and validating the join. 2-3 months is realistic for multi-model, multi-region, multi-cloud deployments.

A representative greenfield path:

1. Week 1. Install OpenCost 1.121.0 on a non-production cluster. Confirm GPU detection works for your hardware. Confirm pricing is pulling from your cloud provider's public pricing API.
2. Week 2. Add opencost.io/sku and opencost.io/customer labels to one production inference deployment. Don't migrate everything yet. Add a Kyverno admission controller that requires the labels - this is how you keep label discipline sustainable as the team grows.
3. Week 3-4. Stand up the token-counting side. If you don't already have vLLM or TGI exposing token counts in Prometheus, this is the work that takes longer than the OpenCost side. Budget two weeks for it. Plan for pre-aggregation: OpenCost allocates in 1-hour or 1-day windows, vLLM exports in 1-minute windows, and the join needs a 5-minute pre-aggregation step.
4. Week 5-6. Build the join. Either in your data warehouse (BigQuery, Snowflake, ClickHouse) or as a Grafana panel that joins Prometheus + OpenCost metrics.
5. Week 7-8. Show it to finance. They will ask three questions. Answer them. Adjust labels and bucketing to match how they actually want to slice the data. Validate that your OpenCost GPU allocation totals match what your device plugin reported - if they don't, you're probably on a pre-1.121.0 OpenCost with the merge bug.

You will not be done in 6-8 weeks if your model surface is large or your GPU footprint is heterogeneous. Plan for 2-3 months if you're a mid-sized AI feature team running multiple model SKUs across regions.

What this means for FinOps and platform teams

The interesting shift here is that GPU cost attribution is moving from "platform engineering makes a quarterly estimate" to "finance gets a live dashboard." This is the same trajectory that CPU cost allocation took in 2018-2020, and the Kubernetes FinOps playbook is similar:

• The data has to be clean enough to trust. Finance will spot-check your numbers against invoice data, and if they don't match, they'll stop trusting the dashboard.
• The labels have to map to billing dimensions that already exist. If your CRM doesn't have a customer_id column matching your opencost.io/customer label, the join doesn't work.
• The data warehouse matters more than the dashboard. Once finance wants to slice by quarter or customer cohort, you need SQL-accessible data, not a Grafana panel.
• For a deeper walk through the cost optimization patterns that work on Kubernetes, our practical guide covers the join design, label discipline, and warehouse plumbing this section assumes.

If you're a platform team supporting AI features, the unsexy work here is label discipline and data warehouse plumbing, not the OpenCost install itself. That's consistent with how teams reduce Kubernetes costs more broadly - the install is cheap, the label discipline is expensive.

When this isn't enough

If you're running training workloads (which have a different cost shape: GPU-hours, not tokens), OpenCost 1.121.0 helps with allocation but won't give you per-experiment or per-checkpoint cost. For that, you need something like Weights & Biases, MLflow, or a homegrown training-orchestrator cost model on top of OpenCost.

If you're running a multi-cloud setup where inference happens across regions and providers, the per-token math still works but the data engineering multiplies. We help teams stand this up - it's not free, but the input is a spreadsheet, not a data engineering project. Our multi-cloud cost optimization work covers this exact pattern.

If you're at a stage where finance is asking questions about inference cost at all, you're past the stage where the answer is "we'll get back to you next quarter." That's the threshold OpenCost 1.121.0 was built for.

Sources and further reading

• OpenCost GitHub repository, release notes for v1.121.0 (July 20, 2026) and v1.121.1
• OpenCost documentation, custom pricing and GPU allocation
• Our production experience shipping per-token cost attribution on Kubernetes (gated, available to clients under NDA - the inputs we use for cost optimization work are the same inputs that make this work)
• PR #3845 (AI Inference Costs v1), PR #3901 (inference usage-cost basis fix), PR #3913 (GPUAllocation merge bug fix), PR #3937 (Allocation JSON NAT gateway costs), PR #3953 (Athena query result reuse), PR #3973 (NAT gateway scrape allowlist)

If you're standing up per-token inference cost attribution on Kubernetes and want a second opinion on the data model, book an Assessment Workshop. We'll walk through your GPU footprint, your inference traffic shape, and your finance team's actual questions - and tell you whether OpenCost 1.121.0 is the right starting point, or whether you need something custom on top.