Inside ChatGPT: What Happens After You Hit Enter?

In this question, we will learn the end-to-end steps that run after you press Enter and how each stage affects latency, correctness, and safety.

We will cover the following:

• The intuition
• How it actually works
• Decoding strategies and comparison
• Tradeoffs and failure modes
• Questions the interviewer might ask
• What the interviewer is really testing

Direct answer: After you hit Enter the client tokenizes your text and sends tokens to the model, the transformer stack computes logits for the next tokens using attention across the context, a decoding rule converts logits into discrete tokens, and post-processing applies safety filters and formatting before the final text is returned. This pipeline balances compute, memory, and probability choices like temperature or top-p to trade off speed, determinism, and creativity.

The intuition (an analogy that makes it click)

Imagine a conversation as a relay race. The runner hands a baton that encodes the conversation so far to the next runner. Tokenization is the handoff. The transformer layers are runners that look at the baton and the race history to decide the best next step. Decoding is the decision to sprint or take a measured step. Post-processing is the referee checking rules and polishing the result.

This keeps the sequence coherent because each layer attends to prior context and refines predictions.

How it actually works (the real mechanics, with one concrete worked example)

Step overview: tokenization, embedding, transformer inference, logits, softmax, decoding, detokenization, filtering.

Concrete example: you type "Hello, how are you?"

Tokens and ids example:

The model converts each token id to an embedding vector of dimension $d$. For a context of length $T$ the model works with a tensor of shape $T\times d$. Positional encodings are added so the model knows order.

The core compute is a stack of transformer layers. Each layer computes self-attention then a feed-forward network. Self-attention computes attention scores between all pairs of positions, which costs $O(T^2)$ time or memory with respect to the context length $T$. The attention weights are used to mix value vectors and produce contextualized representations.

The model produces logits vector $z$ over the vocabulary for the next token. We turn logits into probabilities with softmax:

Then a decoding strategy selects the next token, the token is appended to the context, and inference continues until a stop condition.

Latency and cost table (approximate contributions):

A small worked numeric step: with temperature $T=0.7$ and logits for three candidates $z=[2.0, 1.0, 0.0]$, probabilities become:

Compute these to see which token is likely and how temperature spreads probability mass.

Decoding strategies

We choose how to map probabilities to discrete tokens. Common options and tradeoffs:

Temperature is a scalar applied to logits before softmax. Higher temperature makes the distribution flatter and increases diversity. Lower temperature concentrates mass on the highest logit.

When we run autoregressive generation we often cache key and value tensors from previous steps so we do not recompute everything for prior tokens. Caching reduces compute per token from $O(T)$ to $O(1)$ per new token for the attention with respect to earlier tokens, but total memory grows with $T$.

Tradeoffs and failure modes

• Latency versus context: larger $T$ gives better context but costs $O(T^2)$ memory and higher latency.
• Determinism versus creativity: greedy or low temperature gives repeatable answers but may be bland. High temperature or sampling can hallucinate.
• Safety filtering versus fidelity: aggressive filters may remove correct but sensitive content or change tone.

Questions the interviewer might ask

Some follow-up questions you might get:

How does the context window limit affect generation? Context length $L$ bounds how many tokens the model can attend to. Older tokens fall out of context and the model may lose earlier details unless you summarize or use retrieval to reintroduce them.

Why is attention $O(T^2)$ and can it be improved? Self-attention computes pairwise scores between positions, so naive cost is $O(T^2)$. Sparse or linearized attention approximations reduce cost but trade off expressiveness and sometimes quality.

What is the role of logits versus probabilities? Logits are raw scores $z$ that the model outputs. Softmax converts logits into probabilities. Temperature and top-p operate on logits or probabilities to shape sampling behavior.

How do caching and batching affect throughput? Caching previous key/value tensors speeds incremental generation. Batching multiple requests increases GPU utilization but adds scheduling complexity and potential latency for real-time apps.

How do safety filters integrate into the pipeline? Filters run after detokenization or on token streams. They detect policy violations and can block, redact, or request reformulation. Filters add latency and can create false positives.

Some things to note:

• Retrieval augmentation and tool use can reduce hallucinations by supplying grounded context.
• Quantization or mixed precision reduces memory and latency but can slightly affect output fidelity.

What the interviewer is really testing

They want to know you can explain the full inference pipeline clearly and link design choices to practical tradeoffs. They also want to hear how you would mitigate common problems like hallucination, latency, and context limits by using caching, decoding knobs, retrieval, or safety controls. Show you understand both the math and operational implications.