Speed is the product
On a phone call, the caller hears every millisecond the model spends thinking. My AI receptionist ran llama-3.1-8b-instant on Groq for exactly that reason. Each turn sent about 1,500 tokens in and capped the reply at 80 tokens out. The caller heard a short filler phrase after about half a second and the full answer one to four seconds after they stopped talking. The model was never the slow part. Speech recognition waiting for the caller to finish was.
Groq's own table lists what it serves today and how fast:
| Model | Tokens per second | Input per 1M | Output per 1M |
|---|---|---|---|
| openai/gpt-oss-20b | 1,000 | $0.075 | $0.30 |
| openai/gpt-oss-120b | 500 | $0.15 | $0.60 |
| qwen/qwen3.8-27b | 450 | $0.80 | $4.00 |
| llama-3.1-8b-instant | 560 | Contact sales | Contact sales |
| llama-3.3-70b-versatile | 280 | Contact sales | Contact sales |
Speech-to-text runs there too. whisper-large-v3-turbo costs $0.04 per hour of audio, and my receptionist app used it to transcribe recorded audio. Batch jobs get 50 percent off.
Look at the last two rows. Those were the two models I built on.
August 2026: the model left
My chat assistant ran llama-3.3-70b-versatile first for a salon tenant and kept Claude as the fallback. Around August 21, 2026 that model stopped answering for my account. Nothing crashed. The fallback caught every message, which is what a fallback is for, and that silent success drained my Anthropic budget in a day. The full story is on the Claude page.
I swapped the setting to openai/gpt-oss-120b and found a second wall. On the free tier, Groq capped a request at 8,000 tokens a minute. My grounded prompts run 8,000 to 12,000 tokens. Every one came back as a 413 error, on every model I tried. Groq could not carry that tenant at any quality level until the prompt shrank or the account moved to a paid tier.
Humanize, my rewriting tool, hit the same retirement. It had called Llama models on Groq since April 20, 2026. It was down from September 5 to October 23 for two reasons, and the retired Llama models were one of them. I rebuilt the chain: Qwen first, then gpt-oss-120b, then Claude Haiku.
Limits, as Groq documents them
The two Groq pages I read on November 14, 2026 do not agree, so here is each one.
- The models page lists gpt-oss-120b at 250,000 tokens a minute and 1,000 requests a minute on the Developer plan.
- The rate limits page lists the same model at 30 requests a minute, 1,000 requests a day, 8,000 tokens a minute and 200,000 tokens a day, and calls those base limits.
The second set matches what I hit. Check the limits screen inside your own account before you size anything. When you go over, the API returns a 429 with a retry-after header.
How I use it now
Groq never sits alone in my code. Every call has a second provider behind it and a log line that says which one answered. I count fallbacks per day, because a rising count is the first sign a model was pulled or a limit changed.
I keep prompts for Groq small on purpose. A voice turn fits. A classification job fits. A 12,000-token customer-service prompt with a knowledge base does not fit on the free tier, and by the time you pay for headroom the price gap to Claude Haiku 5.5 at $0.10 per million input tokens is gone.