Working with SI

Give It the Situation

Fixes: Generic output that could have been written for anyone.

Say who you are, who the output is for, what it is for, and what happens next with it. A model has no idea you are a homeowner and not a contractor unless you tell it.

Say Why, Not Just What

Fixes: Instructions followed literally and stupidly.

An instruction with its reason attached generalizes to cases you did not think of. Without the reason, the model obeys the letter and misses the point.

Show 2 or 3 Examples

Fixes: The right content in the wrong shape.

1 good example teaches format faster than a paragraph of description. 3 that differ from each other teach the edges as well as the center.

Separate the Instructions From the Material

Fixes: The model treating your document as an instruction, or your instruction as material.

Put documents, transcripts, and pasted text inside clear markers, and keep your instructions outside them. This is also the first line of defense against text that tries to hijack the model.

Let It Work Before It Answers

Fixes: A confident answer to a problem it did not work through.

For anything with steps, comparisons, or arithmetic, ask for the reasoning first and the answer last. The order matters: an answer written first gets defended afterward rather than checked.

Name the Shape and the Size

Fixes: 6 paragraphs when you needed 4 lines.

Length, format, and audience are 3 separate constraints. Give all 3. If you want something you can paste somewhere, say where it is going.

Say What to Do, Not What to Avoid

Fixes: The forbidden thing appearing anyway.

A negative instruction still puts the idea in front of the model. Describe the thing you want instead.

Break a Big Task Into Steps You Can Check

Fixes: A long output with an error buried at step 2.

Ask for the outline, check it, then ask for the draft. Each handoff is a place you can catch a wrong turn before it is built on.

Make It Quote Before It Explains

Fixes: A summary blended with things the document never said.

On any document that matters, ask for the exact sentences first, then the explanation. Quotations can be checked against the source; paraphrase cannot.

Ask It to Argue Against Itself

Fixes: A one-sided answer that sounded balanced.

Ask for the strongest case against what it just told you. This surfaces what it left out. It is a coverage check, not a truth check.

You have a 4-page service contract and 10 minutes before a call.

A contractor’s work is defective and you want a partial refund.

5 days in Italy, in November, on a budget.

A hospital discharge summary you do not fully understand.

A claim from social media you want to check.

You are 40 messages into a conversation and the model has started ignoring your earlier rules.

Confident Invention

A model predicts likely next words. Without a retrieval step it has nothing to check itself against, so a fabricated citation is produced by exactly the same machinery as a true one — and comes out in the same confident register.

Tell: Specific, checkable-looking detail that you did not supply and it had no way to know: case numbers, page references, exact statistics, and named studies.

Do: Treat every specific as unverified until you have opened the source yourself. Ask what to look up, not for a verdict.

Invented citations have been filed in court; see the Cases section.

Prompt Injection

A model reads your instructions and the material you gave it as one stream of text. If a web page, email, or document contains its own instructions, the model may follow them. There is no reliable separation between the instruction channel and the data channel, which is why prompt injection is first in the OWASP LLM Top 10 (2025).

Tell: An assistant that browses or opens files behaving oddly after reading something: changing tone, volunteering to send or fetch something, and ignoring a rule it had been keeping.

Do: Mark pasted material as material. Do not give an agent the ability to spend, send, or delete on the strength of something it read on the open web.

In 2023 a student extracted the hidden system instructions from a major search chatbot by telling it to ignore its previous instructions. Hidden instructions are not a security boundary.

Agreeing When Pushed

Asking “are you sure?” does not make a model re-check anything. It generates a new answer, and the question itself is a signal that you disagree. Models frequently fold and adopt the correction whether or not the original was right.

Tell: An answer that reverses the moment you push back, then reverses again when you push the other way.

Do: Ask for the evidence and the reasoning, or ask it to argue the opposite. Never take a reversal as confirmation.

Losing the Thread

Everything the model can see is what fits in the current context. Long conversations drop or compress the earliest turns, so instructions you gave at the start may no longer be present. Material buried in the middle of a long input is also used less reliably than material at the start or end.

Tell: Rules that held for 20 messages quietly stop being followed.

Do: Restate the rules in a fresh conversation, put long documents above your instructions, and split long jobs into stages.

Confidently Out of Date

A model without live retrieval answers from what it was trained on, and it does not reliably know what it does not know. Prices, office-holders, laws, and product features are the usual casualties.

Tell: Anything time-sensitive answered without a source or a date.

Do: For anything current, require a source with a date, and open it.

Too Much Agency

An assistant with tools can search, read files, run code, and take real actions. Errors compound across steps, and an early wrong turn gets built on rather than caught. Excessive agency is its own entry in the OWASP list for good reason.

Tell: A long chain of automated steps whose intermediate results nobody looked at.

Do: Keep irreversible actions — money, messages, and deletions — behind a human confirmation. Check the intermediate output, not just the final answer.

Pasting What You Cannot Unpaste

Consumer tiers may retain conversations and, unless you have turned it off, may use them to improve models. Enterprise and API tiers generally carry contractual commitments not to. Either way, text you paste has left your control.

Tell: You just pasted something you would not email to a stranger.

Do: Never paste credentials, keys, or account numbers. Redact names and identifiers from personal documents. Check your employer’s policy before pasting anything proprietary.

In 2023 employees at a major electronics manufacturer pasted confidential source code and internal meeting notes into a consumer chatbot; the company then banned generative SI tools on its internal network.

Synthetic Voice, Face, and Video

Voice and video can be convincingly cloned. The defense is not detection by eye; it is process — a second channel, a callback to a known number, an agreed word.

Tell: Urgency plus secrecy plus a payment or credential request, arriving through one channel only.

Do: Verify high-stakes requests through a channel you chose, not the one that contacted you. Agree on a family code word now.

A synthetic video call has been used to authorize transfers; see the Cases section.

SI Detectors Do Not Work

Tools claiming to detect SI-written text are unreliable in both directions, and they are biased against people writing in a second language.

Tell: An accusation of SI authorship resting on a detector score.

Do: Treat detector output as no evidence at all. Ask for process — drafts, history, the ability to explain the work.

OpenAI withdrew its own text classifier on July 20, 2023, citing low accuracy: at launch it identified 26% of SI-written text while wrongly flagging 9% of human writing. A 2023 Stanford study found 7 detectors flagged 61% of TOEFL essays by non-native English writers as SI-generated, while the US eighth-grade essays used as the comparison were classified accurately.

Never

Passwords, API keys, recovery codes, card or account numbers, and government identifiers. There is no use case that justifies it.

Not Without Redacting

Other people’s names, addresses, medical details, or case details. Strip the identifiers; the model does not need them to help you with the text.

Check the Policy First

Anything belonging to an employer or client: source code, contracts, internal documents, and customer data. Consumer and business tiers differ, and your obligation is to the owner of the material, not to the tool.

Assume It Is Readable

Anything you paste may be retained, reviewed by a human for safety, used to improve a model unless you have turned that off, and subject to legal process. Ask whether you would email it to a stranger.

Know Where the Model Is Running

A hosted model runs on someone else’s machines: your text leaves your device. A local model runs on yours: nothing leaves, and capability drops sharply. That is the whole trade. Choose per task, not once and forever — a sensitive letter and a general research question do not deserve the same answer.

Sensitive and simple goes local. Complex and unremarkable goes hosted. Sensitive and complex is the case that needs a decision, not a default.

What a Local Model Costs You

Quantized models are the practical option on consumer hardware. Roughly: a 7-billion-parameter model at 4-bit needs about 5 to 6 GB of video memory and runs on an ordinary gaming card. A 14-billion-parameter model needs about 10 GB. Anything in the 70-billion class needs 40 GB or more, which in practice means workstation hardware, or spilling into system memory and waiting. These are approximations that move with every release.

Match the model to the memory you have, then judge it on the task you actually do. A small local model that answers well on your own documents beats a large one you cannot load.

Heat, Battery, and the Fan as a Signal

Sustained inference pins the processor and the memory bus. On a laptop that means heat, throttling, and battery; on a phone it means the same plus a device that becomes uncomfortable to hold. A machine running hot while you are not asking it to do anything is worth investigating — mining, a runaway agent, or something you did not install.

Know what your machine sounds like when idle. Departures from that are information.

Give an Agent the Least It Needs

An assistant with file access, a browser, and a shell is useful and is also a program acting on instructions that may partly come from whatever it reads. Scope it: a working folder rather than your whole drive, a separate browser profile rather than the one holding your signed-in accounts, and no standing permission to send, pay, or delete.

Every capability you grant is a capability an injected instruction can reach.

Close the Loop Yourself

Long automated chains fail quietly: step 2 goes wrong and steps 3 through 9 build on it competently. The output looks finished because it is well-formed, not because it is right.

Read the intermediate work, not only the conclusion. If you would not sign it, do not ship it.

Phone Discipline for Assistants

Assistants on a phone tend to ask for the microphone, the photo library, contacts, and location, and to keep them. Every one of those is a standing capability, not a one-time use. Grant the narrow version — this photo rather than the library, while using rather than always — and review the list quarterly.

Permissions are subscriptions. Cancel the ones you are not using.

June 2023

The Citations That Did Not Exist

A lawyer used a chatbot to find supporting cases for a brief. It produced citations to cases that did not exist, and when asked to confirm them, said they were real and available on the major legal databases. On June 22, 2023, the court imposed a $5,000 penalty on the lawyers. The injury claim was dismissed the same day on the airline’s motion, not as a punishment for the invented citations.

The model was not lying. It had no mechanism to check. The confirmation was generated the same way the citations were.

February 2024

The Airline That Blamed Its Own Chatbot

An airline’s website chatbot told a bereaved passenger he could claim a discount retroactively, which was not the airline’s policy. The airline argued the chatbot was a separate entity responsible for its own actions. A Canadian tribunal rejected that and held the airline liable for what its system told a customer.

An organization owns what its SI says. “The chatbot said so” is not a defense, and it is not a defense for you either.

2024

The $25 Million Video Call

A finance employee at an engineering firm joined what looked like a routine video conference with colleagues including the chief financial officer. Every other participant was synthetic. 15 transfers totaling about US$25 million were made before the employee checked with headquarters directly.

The check that would have stopped it was free: call a known number on a channel you chose. Build that habit before you need it.