AI inference latency is the time between submitting a prompt and receiving a usable response. For a developer testing a model in a notebook, a two-second delay is a minor irritation. For a customer-facing application serving thousands of concurrent requests, that same delay is a support ticket, a bounce rate, and a growing cloud bill. Australian enterprises scaling AI workloads beyond pilot stage consistently find that latency is the problem they did not plan for.
What actually drives inference latency
Most discussions of AI performance focus on model accuracy. Latency gets less attention, even though it determines whether a model is usable in production. Four factors drive the bulk of it.
First, model size. Larger models process more parameters per token, and that arithmetic takes time. A 70-billion-parameter model does not respond in the same timeframe as a 7-billion-parameter one, even on identical hardware. The gap is not linear. It compounds at scale.
Second, hardware. Running inference on CPUs is dramatically slower than on GPUs or purpose-built accelerators like AWS Inferentia or Google Cloud TPUs. Many Australian teams prototype on GPU-backed instances and then attempt to cut costs by moving to CPU infrastructure. The latency jump surprises them every time.
Third, batching configuration. When requests arrive faster than the model can serve them, they queue. Poorly tuned batching adds wait time on top of processing time. Good batching configuration groups requests intelligently to maximise GPU utilisation without stacking user-visible delay.
Fourth, network topology. If your application sits in Sydney and your inference endpoint runs in us-east-1 because that is where the model was first deployed, every request carries 150-plus milliseconds of round-trip overhead before the model sees a single token. That is fixable, but it requires deliberate placement decisions.
Time to first token vs total generation time
Latency in generative AI has two distinct measures, and conflating them causes bad optimisation decisions.
Time to first token (TTFT) measures how long the model takes to start responding. For streaming applications, this is the latency the user perceives. A fast TTFT makes a response feel immediate, even if the full output takes several seconds to complete.
Total generation time (or end-to-end latency) measures how long the complete response takes. For batch processing, document summarisation, or back-end pipelines where results are consumed programmatically, this is the number that matters.
Optimising for TTFT without considering throughput can actually worsen cost efficiency. Optimising for throughput without considering TTFT produces applications that feel sluggish. The right target depends on the use case, and that means writing it down before tuning begins.
Quantisation and its trade-offs
Quantisation reduces the numerical precision of model weights, for example moving from 32-bit floating point to 8-bit integers. Smaller weights load faster, require less memory bandwidth, and fit on cheaper hardware. The latency improvement is real: INT8 quantisation can cut inference time by 30 to 50 percent on supported hardware, with modest accuracy loss on most tasks.
The trade-offs matter. Quantised models underperform their full-precision counterparts on tasks that require fine numerical distinctions, complex reasoning chains, or low-frequency vocabulary. For a general-purpose summarisation tool, quantisation is usually a sensible default. For a medical coding assistant where precision errors carry regulatory risk, it warrants careful evaluation against your specific outputs.
Australian teams evaluating quantisation should run it through the same structured process they would use for any model change. The silent model update problem applies here too: a quantised model is a different model, and production systems need to treat it as one.
Caching, batching, and routing as latency levers
Not every latency problem requires touching the model. Three operational levers often deliver faster results with less risk.
Prompt caching stores the computed key-value state of a long system prompt so that repeated requests do not recompute it from scratch. For applications with a stable, lengthy context window, caching the prefix can cut TTFT by more than half. Most major inference providers, including AWS Bedrock and Google Vertex AI, support some form of KV caching natively. AI output caching at the response layer is a complementary strategy for deterministic or near-deterministic queries.
Dynamic batching groups concurrent requests so the GPU processes multiple inputs simultaneously. The gain comes from hardware utilisation: a GPU waiting for a single request is wasted capacity. Most production inference servers, including NVIDIA Triton and vLLM, support dynamic batching out of the box. Tuning batch size and timeout thresholds requires load testing against realistic traffic patterns, not just benchmarks.
Request routing directs queries to the right model for the task. A short, factual question does not need a 70-billion-parameter model. A routing layer that classifies incoming queries and dispatches them to appropriately sized models reduces both latency and per-request cost. This is increasingly practical now that small language models have reached a capability level where they handle a substantial proportion of enterprise queries with acceptable accuracy.
Infrastructure placement for Australian workloads
Australian teams face a geographic constraint that their US and European counterparts do not. The nearest AWS, Azure, and GCP regions are in Sydney and Melbourne, but many third-party model APIs and inference endpoints default to US-based infrastructure. Benchmarking latency from within Australia against a US endpoint and then reporting those numbers as representative of production performance is a common error.
Where local inference endpoints are available, use them. AWS Bedrock, Azure AI Foundry, and Google Vertex AI all operate in Australian regions for supported models. Where local endpoints are not available for a specific model, factor the round-trip latency into your SLA calculations before committing to the deployment architecture. 150 milliseconds of network overhead is not recoverable through model-level optimisation.
Measuring latency properly
Teams that do not measure latency end up optimising based on feel. That produces inconsistent results and makes it impossible to demonstrate improvement to stakeholders.
Measure at the application layer, not just at the model API boundary. The round trip from a user action to a rendered response includes application server processing, network hops, and any post-processing steps. Focusing only on model response time misses the full picture.
Use percentiles, not averages. A p50 latency of 800 milliseconds looks acceptable. A p99 of 12 seconds in the same dataset means one in every 100 users is waiting more than 12 seconds. For customer-facing applications, that tail is the experience that generates complaints.
Set up continuous latency monitoring in production, not just during load testing. Model provider infrastructure changes, traffic patterns shift, and new model versions roll out. A latency baseline established at deployment can drift significantly within weeks without anyone noticing until users start complaining.
When latency is not the real problem
Sometimes what looks like a latency problem is actually a throughput problem or a cost problem in disguise. A system serving 10 requests per second with 800-millisecond latency behaves identically for individual users as one serving 100 requests per second with the same per-request latency. But the infrastructure costs are very different, and teams optimising for latency without tracking throughput can end up spending more to serve the same load.
Getting the measurement right first, then deciding what to optimise, saves a significant amount of wasted engineering time. In most Australian enterprise deployments, the biggest latency wins come from infrastructure placement, prompt caching, and model routing rather than from model-level tuning. Start there.