What is a Large Language Model (LLM), and how does it work?

In this question, we will learn what a Large Language Model (LLM) is and how it works, from the high-level idea down to the main math and a concrete inference example.

We will cover the following:

• The intuition
• How it actually works
• Training and inference
• Scaling and optimizations
• Tradeoffs and failure modes
• Questions the interviewer might ask
• What the interviewer is really testing

A Large Language Model (LLM) is a deep neural network, typically a transformer, trained to model sequences of tokens and predict next tokens. It maps tokens to embeddings, applies layers of self-attention and feed-forward transforms to build contextual representations, and decodes probabilities with a final softmax. During training the model minimizes a token-level loss across many tokens, and during inference you sample or search from the predicted distribution.

The intuition (an analogy that makes it click)

Think of an LLM as a very fast and well-practiced autocomplete system. You give it a partial sentence and it uses patterns learned from reading a huge number of texts to guess what comes next. Self-attention is like highlighting nearby and distant words that matter for the guess, then combining the highlights to form a single confident suggestion.

How it actually works (the real mechanics, with one concrete worked example)

Steps at a glance:

1. Tokenization: split text into discrete token ids.
2. Embedding: map each token id to a vector and add positional encodings.
3. Transformer blocks: repeat layers of self-attention and feed-forward networks to get contextual vectors.
4. Linear projection and softmax: map the final vector to logits over the vocabulary and form probabilities.

The core operation in a transformer block is scaled dot-product attention. In compact math form:

Here $Q,K,V$ are learned linear projections of the input, and $d_k$ is the key dimension.

Concrete toy example: you have the prompt "The cat sat on the" and three candidate next tokens "mat", "dog", "table". After passing through the model we obtain logits. We convert logits to probabilities with softmax, where $softmax(z_i)=\frac{e^{z_i}}{\sum_j e^{z_j}}$. The table shows a plausible outcome.

The model will most likely pick "mat" under greedy decoding. If you use temperature or top-k, the sampling behavior changes.

Training and inference

Training objective. Most LLMs are trained with a next-token prediction objective and cross-entropy loss. A compact view:

Optimization adjusts billions of parameters to minimize this loss over many tokens. Data curation, tokenization design, and batching strategy matter a lot for final quality.

Inference modes. Common strategies include greedy decoding, beam search, top-k sampling, and nucleus (top-p) sampling. Temperature $T$ rescales logits before softmax to control randomness: higher $T$ means more random sampling, lower $T$ makes the model more deterministic.

Scaling and optimizations

Model compute and memory depend on parameters and sequence length. Self-attention cost is quadratic in sequence length, $O(n^2)$, where $n$ is the token count in context, which drives many engineering choices.

A quick comparison of typical sizes and uses:

Common optimizations include quantization to reduce memory, distillation to compress behavior into smaller models, caching past keys and values during decoding to avoid recomputing attention, and sparse attention variants to reduce $O(n^2)$ costs for long contexts.

Tradeoffs and failure modes

Choosing model size and training mix is a set of tradeoffs: larger models often generalize better but cost more to serve and may be harder to control. Data quality drives factuality more than sheer quantity.

Questions the interviewer might ask

Some follow-up questions you might get:

How does self-attention differ from recurrent models? Self-attention computes interactions between all pairs of tokens in parallel, which enables better long-range context and parallel training. Recurrent models process tokens sequentially and can struggle with very long dependencies.

Why do transformers use positional encodings? Because self-attention is permutation invariant. Positional encodings inject token order information so the model can distinguish sequences with the same tokens in different orders.

What causes hallucinations and how would you reduce them? Hallucinations arise when the model overgeneralizes statistical patterns beyond factual grounding. Reduce them with retrieval-augmented generation, grounding on verified data sources, calibration, and conservative decoding strategies.

What is the role of the softmax temperature? Temperature rescales logits before softmax to control randomness. A low temperature concentrates probability mass on high-logit tokens; a high temperature spreads mass and increases diversity.

When would you fine-tune vs use prompting? Fine-tune if you need consistent behavior on a specific domain and have labeled data. Use prompting or few-shot if you need quick adaptation without retraining or when labeled data is scarce.

How do you measure LLM quality? Use a mix of automatic metrics like perplexity and task-specific measures, plus human evaluation for fluency, factuality, and safety.

Some things to note:

• Perplexity measures average predictive uncertainty but does not capture truthfulness.
• Context length limits and attention costs directly affect deployment choices.

What the interviewer is really testing

They want to know you understand the building blocks: tokenization, embeddings, self-attention, transformer layers, and softmax decoding, and that you can explain tradeoffs between scale, latency, and trustworthiness. They also check that you can move from intuition to concrete mechanics and mention practical mitigations for common failure modes.