What You Are Actually Talking To

A language model predicts likely next text. That is not a criticism; it is the mechanism, and knowing it explains almost every surprise you will have.

It means the model is extraordinary at form — structure, tone, translation, rephrasing, drafting, summarizing what is in front of it — and unreliable at any fact it was not given. Nothing in the mechanism distinguishes a remembered truth from a plausible fabrication.

It also means the confidence of the wording carries no information. Standard training and evaluation reward a confident guess over an admission of uncertainty. So the sentence it is sure about and the sentence it made up arrive in the same voice. Every instinct you have for reading human confidence is misleading here, and it is the single most expensive habit to unlearn.

Hold two things at once. These systems are genuinely useful, and they will be more useful the more precisely you drive them. And they will produce a fabricated case citation, a wrong dosage or a dead link with exactly the same fluency as the real thing. Skepticism is not a mood to adopt; it is a procedure to run on the specifics.

The rule that follows from all of it: use the model for the shape, verify the substance. Anything specific and checkable that you did not supply — a number, a name, a citation, a date, a price — is unverified until you have opened the source yourself.