Are LLMs Deterministic? A Hacker News Thread, a Compiler Analogy and the Batch-Size Problem

On 27 April 2026 a programmer who writes as Lelanthran published a short essay called "LLMs Are Not a Higher Level of Abstraction". It answers a popular claim: that prompting a large language model is the next rung on the ladder from binary to assembly to C to Python. When the essay reached Hacker News on 3 May, it drew 166 points and 156 comments (Lelanthran, 2026a, 2026b).

One of those comments, by the user conorbergin, went straight at the essay's premise. LLMs are deterministic, it said: "the same model under the same conditions will produce the same output", unless randomness is added on purpose (conorbergin, 2026).

The replies to that comment, all on the page linked below, are a compact tour of how LLM inference really works. This post walks through them, checks each claim against primary sources, and then asks the question underneath: if LLMs were perfectly deterministic, would that make prompts an abstraction layer?

The essay's argument

Lelanthran's case fits in three steps (Lelanthran, 2026a):

1. Every earlier step up the stack is a function, f(x) → y. The same assembly, C or Python source always produces the same artifact.

2. An LLM is f(x) → P(y). Your prompt does not produce the program you wanted; it produces a probability of getting it.

3. Worse, the output can contain y plus things you never asked for, which the essay calls z1 to zN. A test that checks only for y will pass anyway.

The essay's example: ask for a to-do web app and you may get one, along with exposed credentials or a world-writable FTP share. Your tests looked for the to-do app, so nothing flagged the rest.

The comment's point: the randomness is added on purpose

conorbergin's objection is about where P(y) comes from, and on that narrow point it is correct.

A transformer's forward pass is fixed arithmetic. Given the tokens so far and the trained weights, it computes a score, or logit, for every token in the vocabulary. A softmax turns those scores into a probability distribution. The network's job ends there. A separate piece of code, the sampler, then picks the next token. As anon373839 put it in the thread, the sampler is software that is "NOT the LLM" (anon373839, 2026). Our explainer on attention shows the arithmetic inside the forward pass.

• At temperature 0, the sampler takes the highest-scoring token every time. This is greedy decoding.

• At higher temperatures, it draws from the distribution using a pseudo-random number generator.

• A seeded generator is repeatable. As 0-_-0 replied, you can have deterministic random number generators. Same seed, same draws.

So the P(y) in the essay is not a property of the network. It is a decision made at the sampling step, and with a fixed seed even sampling can be replayed. nowittyusername went further: with the same sampling parameters and the same server configuration, results repeat at any temperature (nowittyusername, 2026).

conorbergin also called neural networks "universal function approximators". That refers to theorems from 1989 showing that a network with one hidden layer can approximate any continuous function on a bounded domain as closely as you like, given enough units (Cybenko, 1989; Hornik et al., 1989). The theorems say what networks can represent, not what a trained one has learned, but they do underline the point: a network is a function.

Why the same prompt still gives different answers

Other replies pointed out that what is true on paper is not what users see.

• 2ndorderthought: temperature 0 and no conversation history would make a model deterministic, but cloud providers do not serve it that way.

• nnevatie: that is "not nearly enough". Inference spread across GPUs and nodes, and especially the batching of requests, makes hosted LLMs non-deterministic.

• slashdave: it is hard to get decent GPU performance and a deterministic answer at the same time. Temperature 0 also tends to send models into repetitive loops.

The usual explanation for this is floating-point arithmetic. Computers round every addition, so (a + b) + c can differ slightly from a + (b + c). GPUs add thousands of numbers in parallel, and if the order of those additions varies between runs, so do the rounding errors.

The batch-size effect

In September 2025, Horace He and colleagues at Thinking Machines Lab showed that this explanation is "not entirely wrong" but misses the main cause (He & Thinking Machines Lab, 2025). Running the same matrix multiplication on a GPU again and again gives bitwise-identical results. A typical forward pass does not need the atomic additions that would make order vary.

The real problem is that the kernels are not batch-invariant:

1. A server groups your request with other users' requests into a batch.

2. The batch size depends on how busy the server is.

3. Kernels pick different reduction strategies for different batch sizes, so the order of additions, and therefore the rounding, changes.

4. A tiny change in a logit can flip which token wins. Once one token differs, everything after it differs too.

Each forward pass is deterministic. What your request gets depends on who else is using the server at that moment.

The team's experiment makes this concrete. They asked Qwen3-235B-A22B-Instruct-2507 at temperature 0 to "Tell me about Richard Feynman" 1,000 times, 1,000 tokens each. They got 80 different completions. All 1,000 agreed for the first 102 tokens; at token 103, 992 said Feynman was born in "Queens, New York" and 8 said "New York City". With batch-invariant versions of RMSNorm, matrix multiplication and attention, all 1,000 completions were identical.

Determinism had a price. On Qwen3-8B, 1,000 sequences took 26 seconds with vLLM's defaults, 55 seconds with unoptimised deterministic kernels and 42 seconds once attention was improved. The authors call that "not disastrous".

Where the tools are now

• vLLM states that it does not guarantee reproducible results by default, "for the sake of performance". Results are reproducible only on the same hardware and vLLM version. A batch-invariant mode is available in beta and may cost performance (vLLM, n.d.-a, n.d.-b).

• SGLang shipped deterministic inference on Thinking Machines' kernels in September 2025. It reported 34.35% overhead, against 61.5% for the original kernels, and noted that output otherwise varies because requests share batches with other users' (SGLang Team, 2025).

What the API providers promise

The second row matters for the thread's advice. On Anthropic's newest models, "set temperature to zero" is no longer an option at all.

Chaotic is not the same as random

The most useful reply came from mrob. When people call LLMs non-deterministic, assume they mean chaotic: small changes to the input can cause large changes to the output, and the only way to find out is to run the whole calculation. For many uses, mrob added, that is just as troublesome.

conorbergin disagreed: models cope well with typos and reworded questions. mylifeandtimes offered a counterexample. Change "helpful" to "helpless" in a long system prompt, three or four characters out of a thousand, and the output is completely different. orbital-decay's answer is the key to the whole debate: that edit reversed the prompt's meaning, and the input is not formally specified, which is "inherent for the domain, not the model" (orbital-decay, 2026).

What actually separates a compiler from an LLM

Put those threads together and the essay's conclusion survives, for a different reason than the one it gives.

Compilers had to be made deterministic too. The Reproducible Builds project defines a build as reproducible when anyone can recreate "bit-by-bit identical copies" from the same source, environment and instructions (Reproducible Builds, n.d.-a). Getting there means hunting down build tools that embed the current time or the source directory path in their output (Reproducible Builds, n.d.-b, n.d.-c). Those were problems with the build environment, not with the compiler's maths, but bit-identical binaries were never automatic.

A compiler's outputs are equivalent; an LLM's are not. Two compilers, or two optimisation levels, can turn the same C program into very different machine code. That is fine, because the language standard defines what the program means, and every correct output must behave that way. You can reason about your program at the level of C without reading the assembly. Where the standard leaves behaviour undefined, it says so; the essay's own author wrote the previous chapter on C and undefined behaviour.

A prompt has no such contract. Make an LLM perfectly batch-invariant, pin the seed, and the same prompt will give the same program every time. But nothing defines which program a prompt means. A different, equally valid wording can produce a different program, and nothing guarantees the two behave the same. That is orbital-decay's point and mrob's: the problem is specification and sensitivity, not dice. Determinism makes an LLM reproducible. It does not make it an abstraction.

That is also why the essay's z1 to zN warning holds even for a deterministic model. A deterministic model that adds an open FTP share will add it every time. To trust code an agent wrote, you have to check it, as in our post on verifying what coding agents produce.

Practical takeaways

• Treat generated code as source. Commit it, review it and test it. Do not count on regenerating the same code later from the same prompt.

• Pin what you can. Record the exact model snapshot and any system fingerprint the API returns. Use a fixed seed where the provider supports one.

• Do not rely on temperature 0. Hosted APIs batch your request with others, and some newer models no longer accept the setting.

• Self-host batch-invariant inference when bit-exact replays matter. That covers evaluations, audits and reinforcement-learning training. vLLM and SGLang both offer it, at a speed cost.

• Test for what you did not ask for. Checks for the feature you wanted will not catch the extras. Add tests and scans for open ports, secrets and permissions.

References

anon373839. (2026, May 4). The model outputs a probability distribution for the next token [Comment on the online forum post LLMs are not a higher level of abstraction]. Hacker News. https://news.ycombinator.com/item?id=48003099

Anthropic. (n.d.). Create a message. Claude API documentation. Retrieved October 10, 2026, from https://platform.claude.com/docs/en/api/messages/create

conorbergin. (2026, May 3). LLMs are deterministic, the same model under the same conditions will produce the same output [Comment on the online forum post LLMs are not a higher level of abstraction]. Hacker News. https://news.ycombinator.com/item?id=48002418

Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals, and Systems, 2(4), 303–314. https://doi.org/10.1007/BF02551274

He, H., & Thinking Machines Lab. (2025, September 10). Defeating nondeterminism in LLM inference. Thinking Machines Lab: Connectionism. https://doi.org/10.64434/tml.20250910

Hornik, K., Stinchcombe, M., & White, H. (1989). Multilayer feedforward networks are universal approximators. Neural Networks, 2(5), 359–366. https://doi.org/10.1016/0893-6080(89)90020-8

Lelanthran. (2026a, April 27). LLMs are not a higher level of abstraction. https://www.lelanthran.com/chap15/content.html

Lelanthran. (2026b, May 3). LLMs are not a higher level of abstraction [Online forum post]. Hacker News. https://news.ycombinator.com/item?id=47999520

nowittyusername. (2026, May 4). There are a lot of misconceptions about LLMs [Comment on the online forum post LLMs are not a higher level of abstraction]. Hacker News. https://news.ycombinator.com/item?id=48004421

OpenAI. (n.d.). Advanced usage. OpenAI API documentation. Retrieved October 10, 2026, from https://developers.openai.com/api/docs/guides/advanced-usage

orbital-decay. (2026, May 4). Sure. What did you expect? You changed the semantic of your prompt to the complete opposite [Comment on the online forum post LLMs are not a higher level of abstraction]. Hacker News. https://news.ycombinator.com/item?id=48004339

Reproducible Builds. (n.d.-a). Definitions. Retrieved October 10, 2026, from https://reproducible-builds.org/docs/definition/

Reproducible Builds. (n.d.-b). Timestamps. Retrieved October 10, 2026, from https://reproducible-builds.org/docs/timestamps/

Reproducible Builds. (n.d.-c). Build path. Retrieved October 10, 2026, from https://reproducible-builds.org/docs/build-path/

SGLang Team. (2025, September 22). Towards deterministic inference in SGLang and reproducible RL training. LMSYS Org. https://lmsys.org/blog/2025-09-22-sglang-deterministic/

vLLM. (n.d.-a). Reproducibility. vLLM documentation. Retrieved October 10, 2026, from https://docs.vllm.ai/en/latest/usage/reproducibility/

vLLM. (n.d.-b). Batch invariance. vLLM documentation. Retrieved October 10, 2026, from https://docs.vllm.ai/en/latest/features/batch_invariance/