What is chain-of-thought prompting and when should you use it?

TL;DR

• Chain-of-thought (CoT) prompting asks the model to produce intermediate reasoning steps, not just the final answer.
• CoT helps with multi-step math, logic, and planning tasks by exposing how the model arrives at an answer.
• It often raises accuracy on complex problems but increases token cost, latency, and exposure of internal reasoning. Key tradeoffs: higher accuracy on reasoning tasks versus cost, slower response, and possible hallucinated or sensitive intermediate steps.

In this question, we will learn what chain-of-thought prompting is, why it helps models reason, and when you should use it in an interview or product setting.

We will cover the following:

• The intuition
• How it actually works
• When to use chain-of-thought
• How to prompt it effectively
• Tradeoffs and failure modes
• Questions the interviewer might ask
• What the interviewer is really testing

Chain-of-thought prompting is a prompting technique that asks the model to show its intermediate steps or reasoning before giving a final answer. It increases reliability on multi-step problems like arithmetic, logic, and multi-hop inference, but comes at a cost in tokens, time, and potential exposure of incorrect or sensitive intermediate content. Use it for problems that need stepwise decomposition and avoid it for short factual queries or when you must not reveal internal reasoning.

The intuition (an analogy that makes it click)

Think of the model as a student who sometimes gets the right answer by pattern matching and sometimes by thinking aloud. If we only ask for the final answer we cannot tell if the student guessed or actually computed. When we ask the student to "show your work" we see the steps and can often find and fix mistakes. Chain-of-thought prompting is asking the model to "show its work."

Showing the chain reduces brittle jumps. You get more signal about how the answer was formed, which improves correct solutions on tasks that require sequential reasoning.

How it actually works (the real mechanics, with one concrete worked example)

At a technical level, autoregressive language models generate a sequence of tokens with probability

$$p(x_1,\dots,x_n)=\prod_{t=1}^n p(x_t\mid x_{<t}).$$

When we include intermediate steps in the ground truth or prompt, we change the conditional distributions the model learns and conditions on. The model is then more likely to produce stepwise decompositions that mirror correct reasoning.

Worked example. Compare a direct prompt and a chain-of-thought prompt for a simple arithmetic problem.

Prompt A (direct): "Calculate 23 times 17."

Model A likely returns: "391."

Prompt B (CoT): "Calculate 23 times 17. Show your steps."

Model B returns a chain and final answer:

1. $23 \times 17 = 23 \times (10 + 7)$
2. $23 \times 10 = 230$
3. $23 \times 7 = 161$
4. Sum: $230 + 161 = 391$ Final answer: $391$

Table: example outcomes on a toy benchmark (illustrative)

You can see CoT raises the correct rate on multi-step items because the model explicitly represents intermediate arithmetic. The cost is more tokens and higher latency.

When to use chain-of-thought

Use CoT when:

• The task requires multi-step reasoning, such as symbolic math, multi-hop reading comprehension, or program tracing.
• You need interpretability of the model's reasoning to validate or debug answers.
• Small improvements in accuracy matter and you can afford extra tokens and latency.

Avoid CoT when:

• The question is a short factual lookup where the final answer is enough.
• You must minimize cost or speed is critical.
• Revealing intermediate steps could leak sensitive information or mislead users.

A rough decision rule: if solving the problem needs more than one thought or operation, try CoT. If it is a single factual fact, skip CoT.

How to prompt it effectively

Good prompts guide the model into useful, concise chains. Try these patterns:

• Provide one or two exemplars that include reasoning steps before the test prompt. This is called few-shot CoT.
• Ask for numbered steps or phrases like "Step 1", "Step 2", to encourage structure.
• Limit verbosity by asking for "brief" steps, for example "Show brief steps and the final answer."

Example few-shot CoT excerpt: "Q: If a car travels 60 miles in 1.5 hours, what is its speed? Show steps. A: Step 1: Distance = 60 miles, Time = 1.5 hours. Step 2: Speed = distance / time = $60 / 1.5 = 40$ mph. Answer: $40$ mph."

When you use few-shot CoT, the model imitates the pattern of reasoning and is more likely to produce correct intermediate steps.

Tradeoffs and failure modes

CoT improves reasoning but has costs and specific failure modes.

• Cost and latency: more tokens per response increases monetary cost and response time.
• Overconfidence in incorrect chains: the model can produce plausible but wrong reasoning and a confident final answer.
• Prompt sensitivity: CoT performance depends on the wording and examples in the prompt.
• Safety and privacy: intermediate steps could reveal sensitive or unwanted content, especially when the model fabricates facts.

Questions the interviewer might ask

Some follow-up questions you might get:

How is few-shot CoT different from zero-shot CoT? Few-shot CoT provides exemplar question/answer pairs including reasoning steps in the prompt. Zero-shot CoT asks the model to reason without exemplars, often by adding an instruction like "Let's think step by step." Few-shot usually gives stronger guidance.

Does CoT always improve accuracy? No. CoT helps on tasks that require explicit decomposition. On short factual or pattern-matching tasks CoT can add noise or reduce performance. Empirical testing on your task is essential.

How do you detect when the chain is wrong? Add verification heuristics. For arithmetic, re-compute the result via a different method. For textual inference, check premises against the source. You can also run multiple chains with different random seeds and compare answers.

Can CoT be combined with model ensembles or verification? Yes. A common pattern is to generate multiple chains, vote or rerank answers using a verification model, or apply a symbolic checker to the final answer.

Is CoT applicable to code generation? Partially. Asking for stepwise design or comments can help plan code. But for precise code correctness, test execution or static checks are required because plausible logic can be incorrect.

How does temperature affect CoT outputs? Higher temperature increases diversity in reasoning and can reveal alternative solutions, but may also increase hallucinations. Lower temperature makes chains more deterministic. Choose temperature based on whether you want exploration or consistent outputs.

Some things to note:

• Test CoT on a validation set before deploying to measure real accuracy gains.
• Prefer concise steps and verification checks to control hallucinations.
• Measure cost and latency impact for your production constraints.

What the interviewer is really testing

The interviewer wants to see that you understand both the strengths and limits of prompting techniques. They expect you to explain when CoT meaningfully improves accuracy, how to craft prompts, and how to mitigate incorrect or expensive outputs. They want practical judgement on tradeoffs between correctness, cost, and safety.