Chain-of-thought prompting is one of those techniques that sounds obvious once you hear it: ask the model to show its reasoning before it commits to an answer. The practice has grown from a research curiosity into a standard tool for enterprise AI teams, but it's widely misapplied. Teams bolt it onto every prompt by default, then wonder why their token costs are climbing without any visible quality gain.
The technique is worth understanding precisely, because the cases where it genuinely helps are narrow and specific. Get the targeting right, and chain-of-thought can rescue a model from errors that no amount of rephrasing would fix. Get it wrong, and it's an expensive way to generate plausible-sounding padding.
What chain-of-thought prompting actually does
A standard prompt asks a model to produce an answer. A chain-of-thought prompt asks it to produce a sequence of reasoning steps that leads to an answer. The simplest version adds a short phrase: "Think through this step by step before answering." More structured versions provide a worked example in the prompt itself, showing the model what a reasoning trace should look like before it encounters the real question.
The mechanism is not that the model becomes smarter. It doesn't. What changes is the computation path. Large language models generate tokens sequentially, and each token is conditioned on everything before it. When a model writes out intermediate reasoning, that reasoning becomes part of the context the final answer is generated from. A model that writes "the monthly rate is $250, there are 12 months, so the annual cost is $3,000" is far more likely to arrive at $3,000 than a model asked to skip directly to the total. The working is doing real work.
This is why chain-of-thought doesn't help uniformly. On simple factual retrieval, there's nothing to reason through. The model either knows the capital of France or it doesn't. Inserting a reasoning trace doesn't change that. On tasks that require multi-step logic, it can be the difference between a correct answer and a confident mistake.
Where it genuinely earns its place
Chain-of-thought prompting delivers real gains on tasks with a few shared characteristics. The problem must require more than one inference step. The intermediate steps must be the kind of thing the model can represent in language. And errors must be detectable from the trace itself, not just from the final output.
Arithmetic and symbolic reasoning are the textbook cases. Ask a model to calculate tax on a multi-line invoice directly, and it will hallucinate totals with quiet confidence. Ask it to work through each line, tally subtotals, then apply the rate, and errors become visible in the trace before they compound into the final number. Legal reasoning over clauses follows the same pattern: the model can trace which conditions are satisfied, which aren't, and why, before it renders a conclusion.
Classification tasks with overlapping categories also benefit. If you're asking a model to triage support tickets into one of five technical domains that frequently overlap, asking it to reason about which criteria apply narrows the space before a final label is assigned. Without the trace, the model is essentially pattern-matching on surface features. With the trace, it's checking criteria explicitly.
It's worth noting that models with built-in chain-of-thought reasoning, sometimes marketed under terms like "extended thinking" or "reasoning mode," handle some of this automatically. But those capabilities aren't available in every deployment context, and understanding the prompting technique is still valuable even when the model handles the scaffolding internally. Knowing how generation parameters like temperature interact with reasoning outputs matters just as much as knowing which tasks benefit from extended traces.
Where it doesn't help and can actively hurt
The failure mode isn't that chain-of-thought produces wrong answers. The failure mode is that it produces wrong answers with very convincing reasoning attached to them. A model that hallucinates a case citation will sometimes construct a detailed step-by-step argument around that citation. The trace looks rigorous. The foundation is fiction.
This matters most for knowledge retrieval tasks. If the answer depends on a fact the model doesn't actually know, asking it to reason doesn't fix the knowledge gap. It just wraps the gap in scaffolding. Teams that apply chain-of-thought to retrieval tasks without combining it with grounded sources, such as a retrieval-augmented generation pipeline, are often getting more confident-sounding errors, not fewer errors.
Simple classification and extraction tasks are another context where the overhead isn't justified. Asking a model to extract an ABN from a contract doesn't require a reasoning chain. Neither does identifying whether a sentence contains a company name. Adding chain-of-thought to these prompts increases latency and cost without changing accuracy in any measurable way.
Token budgets are a real constraint. Enterprise teams running chain-of-thought across high-volume pipelines can easily triple their token consumption per call. For teams already working through token cost pressures in production deployments, applying chain-of-thought indiscriminately is one of the fastest ways to blow a budget.
Zero-shot vs few-shot chain-of-thought
There are two main implementation approaches. Zero-shot chain-of-thought adds a phrase like "think step by step" to the prompt without providing any examples. It requires no prompt engineering beyond that one addition and works well on models that have been trained on reasoning traces. Few-shot chain-of-thought includes one or more worked examples in the prompt itself, showing the model the format and depth of reasoning expected before it encounters the actual task.
Few-shot is more reliable on tasks where the reasoning structure is domain-specific and unlikely to be intuited from a generic instruction. Tax calculation in Australian jurisdiction-specific scenarios, for instance, may need an example that shows the model where GST applies and where it doesn't. A zero-shot instruction to "think step by step" won't anchor the model to the right framework.
The cost is prompt length. Few-shot chain-of-thought prompts can be long. Two or three worked examples, each with a multi-step trace, can consume a significant portion of the available context window before the actual question even appears. For teams working with long documents or complex input, that tradeoff needs to be weighed carefully. Context window size matters here, and it's worth understanding what context window size actually means for enterprise use before committing to a prompting approach that eats heavily into it.
Practical guidance for Australian enterprise teams
The most useful framing is to treat chain-of-thought as a targeted intervention, not a default setting. Start without it. Measure accuracy on your specific task. If the model is making errors that look like incomplete or skipped reasoning steps, add chain-of-thought and retest. If accuracy improves and the cost increase is acceptable, keep it. If accuracy was already fine, remove it.
A few concrete signals that chain-of-thought is worth trying: the task requires more than two inference steps, errors in the output are hard to predict in advance, the same input is producing inconsistent outputs across runs, or domain experts reviewing outputs are finding errors that look like the model "jumped" to a conclusion.
Signals that it isn't worth adding: the task is purely extractive, the model is already hitting acceptable accuracy, you're operating under tight latency or token budget constraints, or the domain requires factual knowledge the model doesn't have regardless of how well it reasons.
One underused approach is to run chain-of-thought traces in evaluation rather than production. Generate reasoning traces during testing to diagnose where a model's logic breaks down, then use those findings to refine the base prompt or the task structure. Once the prompt is tuned, deploy without the chain-of-thought overhead. The trace served its purpose in testing. Production doesn't always need to pay for it.