Inference economics

LLM Inference Cost per Token: How to Calculate GPU Costs

LLM inference cost per token turns a measured GPU-hour bill and sustained output throughput into a workload-normalized unit cost. This guide shows how to choose the denominator, run a fair measurement, account for utilization and serving overhead, and compare the result without treating an example as a benchmark.

Published November 16, 2026 · Calculation method and example values are illustrative.

A GPU server processes a stream of language-model output tokens while its hourly compute cost accrues.
Token economics depend on measured useful output over the same period as the GPU bill.

Short answer

Use the same window for cost and completed output

Divide the cost incurred during a serving window by the completed output tokens produced during that exact window. Multiply the result by 1,000,000 to express cost per million output tokens. With a GPU-hour rate, the compact estimate is hourly GPU cost ÷ (sustained output tokens per second × 3,600) × 1,000,000.

The denominator must match the job: output tokens for generated text, input tokens for prompt processing, or both only when the comparison explicitly uses a combined total. This page isolates that unit-economics question; a complete GPU server cost and ownership estimate includes purchase, power, cooling, and operating costs.

Start with the unit

LLM inference cost per token: the direct calculation

Choose a fixed window, sum the serving costs charged in it, and divide by completed output tokens from those same requests. This avoids combining a monthly bill with throughput measured under a different load.

Cost per 1 million output tokens (serving cost during the window ÷ completed output tokens during the window) × 1,000,000

For one GPU billed at a stable hourly rate, sustained output throughput converts the time unit:

GPU-only estimate (GPU cost per hour ÷ (sustained output tokens per second × 3,600)) × 1,000,000

For a multi-GPU instance, use its full instance-hour rate and aggregate measured throughput. Check whether the listed price already covers every GPU; the units should resolve to dollars per output token.

This GPU-only estimate is a first comparison, not total cost of ownership. The example below is arithmetic, not a provider quote or performance claim.

Measure before extrapolating

Measure throughput before estimating GPU cost per million tokens

Throughput changes with the model, precision, prompt and output lengths, serving engine, batching, and concurrency. Test a repeatable profile that resembles the traffic you expect; a peak number from a different benchmark is not a reliable denominator.

  1. Freeze the workload. Record the model, precision, context, typical prompt and completion lengths, and token types counted as output.
  2. Match the deployment. Keep GPU count, serving software, memory limits, batching, and concurrency aligned with the planned service.
  3. Separate warmup. Stabilize model loading and kernels before steady-state measurement. Measure cold starts separately if production users incur them.
  4. Count completed output. Sum successful output tokens and record failures, retries, cancellations, and queue time for reconciliation.
  5. Repeat at realistic load. Test expected concurrency and a lower-load case; report throughput with latency percentiles.

Log the measurement window, instance shape, GPU count, input and output totals, request count, errors, latency, and throughput calculation. Repeat after a material model, serving, or instance change.

Concurrency changes how much parallel work reaches the GPU. Higher concurrency can improve aggregate output until memory, latency, or scheduling becomes the limit, so include the tested level with every result.

Keep token totals comparable

Separate prompt, completion, and billed token totals

Always name the token type. Prompt processing and generation stress a model differently, while managed services may bill them at different rates. Combining totals can hide a change in workload mix.

MeasureDenominatorUse it for
Output cost per millionCompleted generated output tokensComparing generation workloads with similar prompts and output quality
Input cost per millionProcessed prompt tokensSeparating prompt-heavy traffic or prefill economics
Combined cost per millionInput plus output tokensOnly when the compared service bills and reports the same combined definition

Use the model or service tokenizer and state whether cached, hidden, retried, or rejected tokens count. For API comparisons, match the provider’s input/output definitions; never divide a combined bill by output tokens alone.

Illustrative arithmetic

Work through a cost-per-million example

Suppose one GPU costs $2 per hour and sustains 100 completed output tokens per second. At that rate, an hour produces 360,000 output tokens.

Formula illustration relating GPU-hour price, sustained output tokens per second, and cost per million output tokens.
The worked values are illustrative; replace them with measured throughput and the rate for the chosen deployment.
Hourly GPU cost$2.00
Output tokens per hour100 × 3,600 = 360,000
Cost per output token$2 ÷ 360,000 = $0.00000556
Cost per 1 million output tokens$5.56

This unit conversion assumes a stable rate, one GPU, completed output only, and no other charges. Host costs, idle periods, retries, or lower throughput raise the actual result.

For several GPUs, use the node’s total hourly cost and aggregate measured output. Do not assume throughput scales linearly; sharding, interconnect, memory, and scheduling affect efficiency.

Capacity changes the denominator

Account for utilization, batching, and concurrency

Utilization describes how much paid capacity produces useful work. Batching can raise aggregate throughput while increasing latency; short requests or queueing may leave capacity underused.

Measured output tokens per second at expected production load already reflects utilization, batching, and concurrency. Use it directly. If only a peak rate is available, multiply it by an explicitly labeled effective-utilization assumption and treat the result as a scenario:

Scenario throughputpeak output tokens per second × effective utilization

Use the share of time spent producing successful output, not a generic GPU-busy percentage that may include warmup or non-output work. At low traffic, idle hours remain part of the paid cost; for intermittent demand, compare an autoscaled option using the same request volume and latency requirements.

Keep benchmark throughput, production utilization, and full-service cost as separate results. Each describes a different operating boundary.

Expand from GPU-only to all-in

Include costs missing from the GPU-hour rate

A GPU-hour rate may exclude CPU and RAM, storage, network transfer, orchestration, licensing, platform fees, or minimum commitments. Use the full instance or endpoint charge when capacity is sold as a bundle.

  • Compute: include every GPU and idle hours; check whether the node price already includes all accelerators.
  • Host: add CPU, RAM, local or persistent storage, and snapshots used by the service.
  • Data: include applicable network transfer between regions, storage, and application services.
  • Platform: add endpoint fees, warm-worker minimums, retries, monitoring, and support charged on the bill.
  • Ownership: for owned systems, decide how power, cooling, facilities, support, operations, and refresh risk are amortized.

Label the result as GPU-only or all-in, and keep every included cost on the same period and denominator. For purchase assumptions see the GPU server cost guide; for provider rates see the cloud GPU rental guide.

Make the comparison fair

Compare cost per token without confusing it with API rates

Managed API pricing may include the model, serving stack, reliability, and support; a self-hosted figure may include only GPU time. Match model quality, token definitions, traffic mix, latency, retries, and service availability, then include relevant hosting costs on both sides.

Break-even check: divide total monthly self-hosted cost by successful output tokens and compare it with the managed bill for the same workload. State any excluded warm capacity, operations, or failures.

There is no universal request volume where self-hosting wins. Utilization, staffing, model updates, reliability, and always-on capacity move the break-even point; measure the workload and keep its assumptions.

For integrated, quote-based enterprise capacity, see the NVIDIA DGX Cloud pricing guide; its contract scope should not be compared with a GPU-only hourly estimate.

Common questions

LLM inference cost per token FAQ

How do I calculate LLM inference cost per token?

Divide the total serving cost for a measurement window by the completed output tokens from that same window. Multiply by one million for cost per million output tokens. For a one-GPU estimate, divide the GPU-hour price by sustained output tokens per second times 3,600, then multiply by one million.

How much does one million LLM output tokens cost on a GPU?

There is no universal rate. It depends on the GPU or instance price, model, precision, prompt and output lengths, batching, concurrency, runtime overhead, and sustained throughput. Use a measured workload and a verified rate instead of treating an illustrative example as a quote.

Should input tokens be included in cost per token?

Keep input and output tokens separate when their performance or billing differs. Report cost per million output tokens for generated text, and report prompt processing separately when it matters. If a provider bills a combined total, show that denominator explicitly so the comparison stays like for like.

Does GPU utilization reduce cost per token?

Utilization changes how much useful output the paid capacity produces. If you use measured output tokens per second at the actual load, utilization is already reflected and should not be multiplied in a second time. If you start from a peak rate, apply a clearly labeled utilization assumption and treat the result as an estimate.

Is self-hosted inference cheaper than an LLM API?

It depends on sustained demand, model quality, latency, staffing, availability, and the full infrastructure bill. Compare equivalent output quality and token definitions, then include GPU or instance charges, CPU and memory, storage, network, idle capacity, operations, and reliability costs on the self-hosted side.

Where can I find cloud LLM inference cost per million tokens?

Start with the provider rate for the exact GPU, region, instance shape, and billing model, then measure output throughput with your model and serving configuration. A provider-rate comparison does not replace a workload benchmark.

Method, limitations, and sources

The $2 GPU-hour and 100 output tokens per second are illustrative, not a quote or benchmark. The method checks denominator consistency, measured throughput, utilization, cost coverage, and reproducibility.

GPU Cost publishes GPU and cloud-pricing comparisons and links to its own rate and ownership guides. Those pages provide context for the hourly input, not an endorsement or a source for the example result.

Prices, software behavior, and throughput depend on the deployment and can change. Re-run the measurement when any material workload or serving assumption changes.