What the tool is doing
A language model produces text that fits the pattern of the text it was trained on, continuing from what you gave it. That description sounds dismissive and is not meant to be — the capability is real, and a system that can restructure a rambling paragraph into something readable is genuinely useful. But the description explains the failures, which is why it is worth holding onto.
Producing plausible text and producing true text are different objectives, and the tool is built for the first. When they coincide, which is often, the output is correct. When they diverge — an obscure fact, a specific citation, a number, a rule that changed recently, a question about a thing that does not exist — the tool produces the text that best fits the shape of an answer. That text is fluent, confident, and formatted like everything else it produces.
The second half of the problem is that the system has no reliable way to know what it does not know. These systems do not have a dependable internal signal that separates a well-supported answer from an invented one, so asking whether it is sure produces another fluent answer rather than a diagnostic. Confidence in the output tells you nothing about accuracy, and this is the single most important thing to internalize.
Where fabrication shows up
| What you asked for | The typical failure | What to do instead |
|---|---|---|
| A source, citation or reference | A plausible author, a plausible title, a plausible year, and no such work. Formatted flawlessly. | Treat every reference as unverified until you have found it yourself. If you cannot find it, it very likely does not exist. |
| A quotation | Something the person plausibly could have said, attributed with confidence | Find the original. Paraphrase if you cannot. |
| A number, statistic or measurement | Numbers in the right order of magnitude and wrong in the specifics, presented without a range | Get it from the source that publishes it. Use the tool to find out who publishes it, not what the number is. |
| A current rule, price, deadline or specification | An answer reflecting an earlier state of the world, with no indication that it is out of date | Check the issuing body directly. This is where "it was right last year" does the most damage. |
| Anything about a specific named person or small organization | Details blended from similar names, or invented outright | Do not use it for this at all |
| Code, formulas and calculations | Syntactically clean and logically wrong, often in the edge cases. Arithmetic in prose is unreliable. | Run it, test it, or check the arithmetic separately |
| A summary of a document you supplied | Much more reliable, because the material is in front of it — but it can still smooth over a qualification that mattered | This is the strongest use. Spot-check the parts you will rely on. |
Note the pattern in that table. Reliability rises sharply when the material is supplied rather than recalled. Give it the document and ask it to reorganize, and you are on solid ground. Ask it to remember the document, and you are not.
What happens to what you type in
Anything you paste into a service leaves your device and reaches a company's systems. What happens next depends entirely on that service's terms, and those terms vary widely and change: some retain conversations for a period, some use submitted content to improve their systems by default, some offer a setting to opt out of that, some contractual and workplace tiers commit to not using it at all. There is no universal answer, and any article that gives you one is out of date the week after it is written. Read the current terms of the service you actually use, and look specifically for retention period, whether content is used for training, and whether a setting changes it.
Given that, a simple rule handles most situations: do not paste anything you would not be comfortable having sit on a company's servers indefinitely. That covers more than it first appears.
| Do not paste | Why |
|---|---|
| Other people's personal information | Names, addresses, health details, financial details. It is not yours to hand over, and at work it may breach an obligation you are personally subject to. |
| Anything covered by a confidentiality agreement | Employment contracts and client agreements generally do not carve out an exception for convenience |
| Credentials, keys, tokens | Obvious, and it still happens constantly inside pasted configuration files and code |
| Unredacted documents | Contracts, medical letters, statements. Strip identifying details before asking about the structure or the wording. |
| Anything you would not put in an email to a stranger | The working test, and it is a good one |
Check whether your employer has a policy before work material goes near a personal account — one usually exists.
Where confident wrong answers do the most damage
Medical, legal and financial questions are the worst case, and it is worth being precise about why rather than issuing a general warning. Three things stack up. The answers sound authoritative, because the training material for these domains is full of authoritative prose. The person asking usually cannot evaluate the answer, which is exactly why they asked. And the cost of acting on a wrong answer is high and sometimes not reversible.
Add a fourth factor specific to these fields: the correct answer depends on particulars the tool does not have. A medication interaction depends on the rest of your prescriptions and your kidney function. A contract clause depends on the jurisdiction, the rest of the agreement, and what was negotiated. A tax question depends on your full situation and on rules that change annually. The tool will answer confidently without any of that, because it produces the text that fits the question it was given.
This is not an argument for never touching these subjects. It is an argument for a specific role. Use it to build vocabulary before an appointment, to draft the questions you want to ask, to get a plain-language explanation of a term so the professional conversation starts further along, or to structure a summary of your own history to bring with you. What you do not do is substitute it for the appointment, or arrive having already decided based on what it said. The value is in preparing for expert advice, not replacing it.
Four shapes of task where it works
The distinction that predicts success is not the topic. It is whether the tool is generating knowledge, which it cannot reliably do, or transforming material you already have, which it does well.
| Shape | Example | Why it works |
|---|---|---|
| Drafting you will edit | A first version of a difficult email, a cover letter, a complaint, a description you have been avoiding writing | The hard part of writing is often starting. A mediocre draft you rewrite is faster than a blank page, and you supply the judgment. |
| Restructuring text you already have | Tightening a rambling paragraph, turning notes into an outline, summarizing a document you paste in, adjusting the tone of something you wrote | The facts come from you. The tool is doing arrangement, which is what it is best at. |
| Explaining something you can then verify | What a term in a document means, how a process generally works, what a piece of code appears to do | An explanation gives you the vocabulary to check the claim against a real source, which a bare answer does not |
| Generating options rather than answers | Twenty possible titles, five approaches to a problem, counterarguments to a position you hold, questions to ask a contractor | Judgment stays with you and volume is genuinely useful. A wrong option costs nothing because you are selecting, not accepting. |
The reverse framing is just as useful. When you catch yourself about to accept output as fact — a number, a rule, a citation, a diagnosis — that is the signal you have moved from transformation to generation, and the verification cost has just gone up.
Verifying without losing the time you saved
Verification that costs more than writing it yourself defeats the point, so scale it to consequence. Ask what happens if this specific line is wrong. A tone adjustment on an internal email needs no checking. A figure going into a document someone else will act on needs a source. A dosage, a legal deadline or a structural specification needs the authority that publishes it, not a second opinion from the same tool.
Two techniques are worth the effort and one is not. Asking for the source and then finding that source yourself is worth it, because the failure is visible — the reference either exists or it does not. Giving the tool the material and asking it to work only from that material is worth it, because it changes the task from recall to reading. What is not worth it is asking the same tool whether it is sure: you will get another fluent answer, generated the same way, and no additional information.
One habit prevents the most common professional embarrassment. If you did not verify a claim, do not put it in something with your name on it. That single rule catches invented citations, drifted statistics and rules that changed, and it costs nothing to apply.
A working setup
Keep it in the roles above and it earns its place. Use it for drafts you will rewrite, for restructuring your own material, for explanations you will check, and for options you will choose among. Paste nothing you would not email to a stranger. Verify anything that leaves your desk. Check the current terms of the service you use, rather than trusting a description of them, and check your employer's policy before work material goes anywhere near a personal account. And keep the domains where being confidently wrong is expensive — health, law, money — in the preparation role rather than the answering one.
None of this is a verdict on whether these tools are good. It is a description of a specific shape: strong at arrangement, unreliable at recall, and unable to tell you which mode you just got. Work with that shape and the output is useful; work against it and the failures arrive without a warning label.
Questions people ask
Why does it make things up so confidently?
Because fluency and accuracy are produced by the same process. The system generates text that fits the pattern of an answer, and when it has solid material that text is correct; when it does not, it still produces something answer-shaped, with the same fluency and formatting. The deeper issue is that it has no reliable internal signal separating a well-supported answer from an invented one, so it cannot flag the difference and asking whether it is sure just generates another fluent reply. The practical consequence: confidence in the output carries no information about whether it is right.
Is it safe to paste work documents into these tools?
It depends on the service's current terms and on your employer's policy, and both are things to check rather than assume. Depending on the service, submitted content may be retained for a period, may be used to improve the system by default, may be excluded by a setting, or may be covered by a contractual commitment not to use it. Those terms differ between services and change over time. A workable rule regardless: do not paste anything you would not be comfortable having sit on a company's servers indefinitely, which rules out other people's personal information, anything under a confidentiality agreement, credentials, and unredacted documents.
Can I use it for medical, legal or financial questions?
For preparation, yes. For answers, no. Three things make these the worst case: the output sounds authoritative because the source material is authoritative, the person asking usually cannot evaluate it — which is why they asked — and acting on a wrong answer is expensive and sometimes irreversible. A fourth factor is that correct answers in these fields depend on particulars the tool does not have: your other medications, the rest of the contract, your full financial situation, and rules that changed this year. Use it to learn the vocabulary and draft the questions you want to ask a professional.
How do I check whether an answer is right without redoing the work?
Scale the checking to the consequence. Ask what happens if this specific claim is wrong — a tone adjustment on an internal email needs nothing, a figure someone else will act on needs a source, and a deadline or a dosage needs the authority that publishes it. Two techniques earn their time: ask for the source and then go and find that source yourself, since a fabricated reference simply will not exist; and supply the material and ask it to work only from what you gave it, which converts the task from recall to reading. Asking the same tool whether it is certain adds nothing.
What is it actually good at?
Transforming material you already have, rather than supplying material you do not. Four shapes work reliably: drafting something you will then rewrite, where the value is escaping the blank page; restructuring text you supplied, such as tightening a paragraph or turning notes into an outline; explaining a term or a process so that you have the vocabulary to verify it against a real source; and generating options rather than answers — twenty candidate titles, five approaches, counterarguments to your own position — where judgment stays with you and a wrong option costs nothing. The moment you are accepting output as fact, you have left the reliable zone.