Tokens per second is an AI inference metric that measures how many text tokens a language model generates each second. It indicates how quickly streamed AI output appears after generation begins. TPS is useful for judging output speed, but it must be considered alongside time to first token, model quality, prompt length, and concurrent-user load.
Tokens Per Second
What Are Tokens Per Second in AI?
Tokens per second, or TPS and tok/s, measures how many text tokens an AI language model generates each second during inference. It shows how quickly streamed output appears after a model starts responding. A token can be a word fragment, punctuation mark, number, or code symbol. AI models process and generate tokens during inference, the live response stage rather than training.
How Tokens Per Second Works
TPS usually measures the final output-generation stage of an AI response.
- A user submits a prompt, such as a request to explain an error or write a function.
- The model processes the prompt and conversation context.
- The first output token arrives. This delay is time to first token, or TTFT.
- The model predicts each next token from previous tokens.
- TPS measures this token-by-token decoding rate.
Tokens Per Second vs Time to First Token vs Throughput
These metrics answer different performance questions.
| Metric | What it measures | When it matters most |
|---|---|---|
| Tokens per second | Generation speed for one response | Streaming chat and coding assistance |
| Time to first token | Wait before output begins | Long prompts, document analysis, and interactive apps |
| Throughput | Total tokens served across requests | Multi-user AI products and background workloads |
Why Tokens Per Second Matters for AI Applications
Higher TPS can make an app feel more responsive, but there is no universal good number. The right target depends on the task and output length.
- Streaming replies appear smoother.
- Code suggestions, edits, and explanations finish sooner.
- Prompt-and-revise cycles are less disruptive.
- AI agents can move faster between generation steps when tool calls are not the bottleneck.
Limits of TPS as an AI Speed Metric
A high TPS figure alone does not guarantee a fast or useful AI experience.
- Long prompts and large context windows can increase time to first token.
- Model size, hardware, quantization, batching, and network delay affect results.
- Reasoning models may plan before generating visible text.
- Output length changes total wait time, even at the same TPS.
- Tokens per dollar measures cost efficiency, not response speed.
How to Compare TPS Benchmarks Fairly
Compare like with like to avoid misleading conclusions.
- Use the same model, version, and serving software.
- Record hardware, quantization, context length, and batch size.
- Measure prompt processing and decoding separately.
- Use realistic prompts and output lengths.
- Report median results from repeated runs.
- Test one-user responsiveness and throughput under realistic load.
Tokens Per Second in Emergent
In Emergent, generation speed can affect coding, debugging, testing, and iteration, but dependable delivery also relies on model capability, tool calls, test execution, and deployment workflows. Learn more in Emergent’s Mercury 3 speed update and this guide to AI website builders versus developers.
Start Building
on Emergent today
on Emergent today
Start Building
