Model-serving capacity calculator
AI Inference Tokens per Second Calculator: TPS, Users and Latency
Tokens per second can mean per-user generation speed or the aggregate output of an entire inference service. Mixing those definitions causes bad capacity plans. This calculator starts with user demand and response targets, then estimates required aggregate output TPS, per-request generation rate and capacity headroom without assuming a particular accelerator.
Interactive calculator
AI Inference Tokens per Second Calculator
Enter your own workload or capacity assumptions. Public LPX benchmarks are workload-specific, and this calculator does not certify application performance, electrical design, cooling or OEM compatibility.
Live Amazon supporting hardware
Current supporting components and Price Options
Compare current Amazon listings relevant to this guide. Product availability and prices can change.
Quick answer
TPS is only useful when you say whose tokens and over what interval
Per-request TPS describes how fast one response is generated. Aggregate TPS describes how many output tokens the service emits across all requests. A fleet can have high aggregate throughput while individual users see slow generation, so both metrics belong in an inference capacity plan.
Calculate demand first, then compare candidate platforms
Use observed requests per minute, average output length and concurrency to describe the workload. Once required TPS is known, compare it with sustained measurements from LPX, GPU or cloud endpoints at the same context and quality settings.
Tokens-per-second metrics you should keep separate
Each metric answers a different serving question and can produce a different architecture decision.
| Metric | Simple definition | Useful for | Common mistake |
|---|---|---|---|
| Per-request output TPS | Output tokens ÷ generation seconds | User-perceived interactivity | Calling it total server throughput |
| Aggregate output TPS | All output tokens ÷ wall-clock second | Fleet capacity | Ignoring how slow each stream is |
| Request rate | Requests arriving per second/minute | Queueing and admission control | Assuming every request has equal token length |
| Time to first token | Delay before generation starts | Prompt/context responsiveness | Hiding it inside output TPS |
| End-to-end latency | TTFT + generation + app/tool overhead | Completed user experience | Optimizing only accelerator time |
| Headroom-adjusted TPS | Expected TPS × safety factor | Production capacity planning | Selling 100% of benchmark peak |
Before you use the result for procurement
Benchmark the exact workload
Keep the model, context length, output length, concurrency, precision and serving-software version beside every TPS result.
Separate context from generation
LPX is positioned around low-latency token generation; Rubin and Rubin CPX handle different context-heavy work. Measure both phases.
Treat rack specs as architecture
LPUs, SRAM and internal bandwidth explain the system design but do not by themselves predict application throughput.
Verify the current OEM design
Use qualified system documentation for rack power, cooling, networking, serviceability and supported configuration before procurement.
Per-request TPS describes interactivity
If a model emits 800 tokens over four seconds of generation, the user sees about 200 output tokens per second during that phase. This is the metric most people notice when watching text stream on screen. High per-request TPS can make coding agents and interactive assistants feel dramatically faster.
It does not tell you how many customers the platform can serve at once. A system may deliver 200 TPS to one user but lose that speed as concurrency rises, or it may preserve per-user interactivity by using more hardware.
Aggregate TPS describes the total output factory
Aggregate throughput sums output across every active request. A service producing 10,000 output tokens each second might deliver 200 TPS to 50 users, 100 TPS to 100 users or some uneven distribution shaped by scheduling. The same aggregate number can therefore create very different user experiences.
Capacity planning needs aggregate TPS because it determines how much total demand the fleet can carry. Product design needs per-request TPS because it determines how responsive each session feels.
Requests per minute converts product traffic into token demand
Start with the traffic metric product teams already track. Multiply requests per minute by average output tokens and divide by 60 to estimate average aggregate output TPS. For peak planning, use the peak request rate and a realistic high-percentile output length rather than a daily average that smooths bursts.
Request distributions are often heavy-tailed. A few coding or research tasks can generate far more tokens than simple chat questions. Segment workloads when one average stops representing the fleet.
Generation time creates an interactivity requirement
If a user should receive 1,000 output tokens in five seconds, the generation phase needs roughly 200 output TPS for that request. That target can be more demanding than the aggregate traffic requirement, especially for low-volume premium services where each user expects near-instant progress.
Set the target from product experience, not hardware marketing. Then benchmark candidate systems at the concurrency required to meet the business demand.
Concurrency links per-user speed and aggregate capacity
Peak concurrent generations tell you how many streams may be advancing at the same time. Multiplying concurrency by the target per-request TPS gives a rough instantaneous aggregate requirement. The request-rate method gives another view based on arrivals. The larger of the two is often a useful starting capacity envelope.
Real schedulers do not divide capacity perfectly evenly, so this is still a planning approximation. Load testing should reveal how the platform behaves when request lengths and arrival times are uneven.
Time to first token is outside pure output TPS
A service can generate at 500 TPS after it starts and still make the user wait ten seconds for the first token because a huge prompt is being processed. TTFT includes context processing, queueing and routing effects that output TPS alone does not capture.
For agentic systems with large context, track TTFT separately and plot it against context length. This is where context-specialized GPU capacity such as Rubin or Rubin CPX can matter even when LPX improves the generation phase.
Headroom turns a benchmark into a production target
A benchmark that reaches 10,000 aggregate TPS should not automatically be sold as 10,000 TPS of customer capacity. Bursts, model variance, maintenance and tail-latency objectives require reserve. The calculator increases demand by the headroom percentage before comparing it with measured platform throughput.
Choose headroom from operational data. A mature steady service may need less than a rapidly growing agent platform with unpredictable request sizes.
Model quality can change the economic value of TPS
Two endpoints can emit the same number of tokens per second but produce different task quality. A faster model that needs more retries or generates verbose unnecessary output can consume more tokens and take longer to complete the actual job.
Track task success, answer quality and total tokens per completed task alongside throughput. This converts infrastructure speed into a business metric rather than rewarding token production for its own sake.
Context and output tokens place different pressure on hardware
Input tokens are processed before and during inference through attention and cache mechanisms, while output tokens are generated sequentially. The cost per token is therefore not identical across input and output. A workload with a million-token prompt and a 100-token answer has a different hardware profile from one with a short prompt and a 10,000-token answer.
This is why the calculator focuses specifically on output TPS for generation demand. Context sizing needs its own measurements and may drive a different resource pool in a disaggregated architecture.
Compare platforms with the same workload trace
When evaluating LPX, GPUs or cloud APIs, send the same prompt distribution, context, output caps and quality settings. Measure the same concurrency and include warm-up behavior. Otherwise a faster number may come from an easier workload rather than better hardware.
Save the raw request trace and benchmark software version so the test can be repeated after upgrades. Reproducibility is more valuable than a single impressive screenshot.
Networking and storage can limit effective inference throughput
High accelerator TPS creates downstream data. Models and caches have to be loaded, requests traverse networks and distributed serving can exchange substantial state. If network or storage paths saturate, the accelerator may wait and observed service TPS will fall below its compute capability.
The supporting Amazon products on this page are useful for broad component price monitoring, not for validating a production fabric. High-end clusters should follow qualified network and storage designs.
Use TPS as one layer of an SLO, not the whole SLO
A strong inference service-level objective normally combines availability, TTFT, output-token latency, total completion time and perhaps cost or quality. TPS is one measurable layer inside that contract. Optimizing it in isolation can move a bottleneck somewhere else.
The best capacity model therefore starts with the user experience, translates it into traffic and token demand, then verifies the complete path under load. The calculator is a transparent first step in that process.
Methodology and sources
The calculator uses standard arithmetic on request rate, output length, generation time, concurrency and headroom. It is hardware-neutral. Platform capacity should come from sustained measurements using the same model and workload profile.
- NVIDIA technical blog: LPX long-context interactivity
- NVIDIA Groq 3 LPX product page
- NVIDIA Groq 3 LPX full-production announcement
- NVIDIA technical blog: inside Groq 3 LPX
- NVIDIA Vera Rubin NVL72
- NVIDIA Rubin CPX announcement
As an Amazon Associate, Cloudzat may earn from qualifying purchases. Marketplace listings on these pages cover adjacent networking, storage, memory or power hardware only. They are not represented as LP30 chips, LPX trays or qualified Vera Rubin rack components. Verify exact models, warranty, interfaces and current OEM documentation before purchase.
Frequently asked questions
How do I calculate output tokens per second?
Divide the number of output tokens by the seconds spent generating them. For fleet throughput, sum output across all active requests over the same wall-clock interval.
What is a good tokens-per-second number?
There is no universal number. The requirement comes from user experience, model size, context, output length and product tier.
Is TPS the same as time to first token?
No. TTFT is the delay before generation starts; output TPS is the rate after output begins.
How do I estimate aggregate TPS from requests per minute?
Multiply requests per minute by average output tokens, divide by 60, then add headroom for peak variance.
Why does concurrency matter?
Many simultaneous streams can compete for the same compute and change both aggregate and per-user throughput.
Should I include input tokens in output TPS?
No. Keep input/context processing and output generation as separate measurements because they stress the system differently.
Can I compare two providers by TPS alone?
Only if model, context, quality, concurrency and measurement method are aligned. Also compare TTFT, tail latency and task success.
How much headroom should I use?
Choose a percentage supported by traffic variability, growth and availability requirements. Twenty to thirty percent is a scenario, not a universal rule.
What is the measured platform TPS field for?
Enter a sustained aggregate result from the candidate system. The calculator compares your headroom-adjusted requirement with that measured capacity.
Does this calculator predict Groq 3 LPX performance?
No. It is hardware-neutral. Use actual LPX measurements when evaluating an LPX deployment.