Measure before extrapolating
Measure throughput before estimating GPU cost per million tokens
Throughput changes with the model, precision, prompt and output lengths, serving engine, batching, and concurrency. Test a repeatable profile that resembles the traffic you expect; a peak number from a different benchmark is not a reliable denominator.
- Freeze the workload. Record the model, precision, context, typical prompt and completion lengths, and token types counted as output.
- Match the deployment. Keep GPU count, serving software, memory limits, batching, and concurrency aligned with the planned service.
- Separate warmup. Stabilize model loading and kernels before steady-state measurement. Measure cold starts separately if production users incur them.
- Count completed output. Sum successful output tokens and record failures, retries, cancellations, and queue time for reconciliation.
- Repeat at realistic load. Test expected concurrency and a lower-load case; report throughput with latency percentiles.
Log the measurement window, instance shape, GPU count, input and output totals, request count, errors, latency, and throughput calculation. Repeat after a material model, serving, or instance change.
Concurrency changes how much parallel work reaches the GPU. Higher concurrency can improve aggregate output until memory, latency, or scheduling becomes the limit, so include the tested level with every result.