Token pricing looks simple on paper. A model provider publishes a per-million-token rate, you estimate your usage, and you put a number in the budget. Then the first month's bill arrives and it's two or three times that number. This pattern is playing out across Australian enterprises right now, and it isn't because the teams are careless. It's because token costs compound in ways that a flat rate doesn't capture.
Why token estimates go wrong from the start
Most teams estimate tokens by thinking about output: how many words will the model produce? That's the wrong place to start. Input tokens cost money too, and in many enterprise deployments they dwarf the output. Every system prompt, every retrieved document chunk, every conversation history segment injected into a request adds to the input count before the model writes a single character of response.
A retrieval-augmented generation setup, for instance, might inject 3,000 tokens of source document context for every 200-token user query. The ratio matters. If your retrieval-augmented generation pipeline pulls five document chunks per query and each chunk runs 600 tokens, you're spending 3,000 input tokens before the model has processed the question. At GPT-4o pricing, that adds up to real money across a few thousand daily queries.
System prompts are another underestimated cost. A well-crafted system prompt for an enterprise AI assistant might run 800 to 1,500 tokens. That cost is paid on every single API call. Teams frequently write and refine system prompts without considering that each added instruction is a recurring line item multiplied by request volume.
The multimodal multiplier
Text tokens are cheap relative to what image-capable models charge for visual input. A single high-resolution image passed to a vision model can consume anywhere from 1,000 to 6,000 tokens depending on the provider's tiling logic. Teams building document processing pipelines that handle scanned PDFs or product photos often discover this the hard way after a weekend batch job runs against expectations.
Video frames are worse. If a workflow passes video to a multimodal model, each frame is treated as an image. A 30-second clip at 1 frame per second is 30 separate image costs. This isn't exotic: document intelligence, inspection automation, and compliance review workflows regularly involve visual content, and the teams that scoped them as "text workloads" are the ones seeing the largest variances.
Context window size is not free
Provider marketing around context windows focuses on capability: you can now fit an entire codebase, a full legal contract, a year of customer transcripts. What gets less airtime is that every token inside that context window is billed. A 200,000-token context window doesn't mean a 200,000-token conversation costs the same as a 2,000-token one. It costs roughly 100 times more on the input side.
Long-context models are genuinely useful for certain tasks. But teams that enable large context windows as a default, because it's easier than designing a selective retrieval strategy, pay for capacity they don't need. The practical implications of context window size go well beyond what fits in a single prompt; the billing model is where the cost surfaces.
Conversation history management is the specific area where this bites hardest. A chatbot that appends the full conversation history to every request sees exponentially growing input costs as a conversation lengthens. By message 20, you might be paying for 15,000 tokens of history to generate a 100-token reply. Implementing a sliding window or a summarisation step can cut this dramatically without hurting response quality in most cases.
Hidden costs in agentic workflows
Single-turn interactions are expensive. Agentic workflows, where a model reasons across multiple steps, calls tools, checks outputs, and retries, multiply that cost by the number of turns in each reasoning loop. An agent that takes 8 steps to complete a task costs roughly 8 times what a single-call implementation would cost for the same outcome, assuming similar prompt lengths per step.
Tool call responses add another layer. When an agent calls an external API and receives a JSON payload back, that response gets injected into the next prompt as tokens. A verbose API response bloated with metadata can add hundreds of tokens per tool call. Teams building agentic AI systems for the enterprise need to consider response trimming as part of their pipeline design, not as an afterthought.
Retry logic is a quiet multiplier. An agent that retries a failed step three times before escalating has just tripled the cost of that step. Without circuit breakers and retry budgets, a failure mode in one part of a workflow can generate thousands of dollars of API spend in minutes.
What metering and observability actually require
Most teams instrument their applications for latency and error rates. Fewer instrument for token consumption at the request level. Without per-request token logging, it's impossible to identify which queries, which users, or which workflow paths are driving the cost. You end up with an aggregate number that tells you spend is high but not where to cut.
Useful token observability means logging input tokens, output tokens, and model name for every API call, tagged with enough metadata to trace the call back to a feature, user cohort, or workflow step. This isn't complex to implement, but it has to be built in from the start. Retrofitting observability onto an existing AI application is genuinely painful.
Spend alerts with hard ceilings matter too. A misconfigured prompt, a loop bug, or an unexpected traffic spike can generate thousands of dollars of API spend in hours. Provider-level rate limits are a blunt instrument. Application-level spend limits, triggered before the bill arrives, are sharper. Set them by feature and by day, not just by month.
Model selection is a cost decision, not just a capability decision
The most capable model is rarely the most cost-effective for the task. Sentiment classification, entity extraction, simple summarisation, and classification tasks perform well on smaller, cheaper models. Running them on a frontier model because it's already integrated is spending three to ten times more than necessary.
Routing logic, where a classifier determines which tier of model handles a given query, is one of the highest-leverage cost controls available. A query that can be answered by a smaller model costs a fraction of what a frontier model would charge. At scale, the savings are significant. The engineering effort to build a router is typically paid back within weeks on any meaningful query volume.
Caching is the other lever. Identical or near-identical prompts returning cached responses cost nothing in inference. Prompt-level caching is now a native feature on some providers; semantic caching, where embeddings match a query to a prior response above a similarity threshold, extends coverage further. Both approaches are worth implementing before reaching for a bigger model budget.
What to do before the next invoice
Three things make the most immediate difference. First, add per-request token logging if you don't have it. You can't optimise what you can't see. Second, audit your system prompts: remove redundant instructions, compress verbose sections, and test whether shorter prompts degrade output quality on your specific tasks. Third, map your workloads to model tiers and move anything that doesn't need frontier capability to a cheaper option.
Token costs are an engineering problem, not a pricing negotiation. The teams that treat them as a design constraint from the start spend a fraction of what teams spend when they treat AI compute as a utility that sorts itself out.