Skip to content
Thinking with AI Series: AI: Foundations

What Sits on the Other Side of the Chat Window

A neural network mistakes a handwritten eight for a three—how does that error help it learn? In Issue 1 of AI: Foundations, we unpack how AI, neural networks, and language models fit together, what tokens are, and why an email gets a deadline nobody agreed to.

What Sits on the Other Side of the Chat Window — AI: Foundations, Issue 1
Key Takeaways Expand · 5 cards Collapse
Key Takeaways · AI: Foundations · Season 1, Issue 1
Core Insight & Practice in 90s

Main Takeaway

A note before we begin: Anna and her team are fictional composites. The research, experiments, and mechanisms they run into along the way are real.

On Monday morning, the director sent a short email to every department: Starting this month, everyone uses AI in their daily work. Licenses are active. Be bold.

Anna runs the team that processes funding applications. By lunchtime, her team had split into two camps without anyone deciding to. Tom, the team's most enthusiastic early adopter, had already used the chat to summarize three reports and was telling everyone it was "basically a genius." Irina, who has twenty years of experience and very little patience, had asked it for the deadline of a regulation she knows by heart. It got the date wrong, and she declared the whole thing a "useless toy."

Anna suspected they were both wrong, but she couldn't have said why.

At six in the evening, Anna had twelve overdue status queries left in her inbox, including a message from a neighboring department: Where does application #2417 stand? Tired of writing polite variations by hand right before heading home, she pasted a page of internal reviewer notes into the chat to turn them into a batch of replies. For application #2417, the notes were brief:

Application received. Review still in progress. Completion date not set.

Pause and predict. Before you read on, what do you think came back?

  • A. A polite reply containing exactly those three facts
  • B. A polite reply that adds something that wasn't in the notes
  • C. A clarifying question asking Anna for the completion date

Here is what appeared, a second later:

"Your application has been accepted for processing. We will complete the review on Friday."

If you chose B, your instincts are already good. The reply is smooth, businesslike, and courteous. It also contains a deadline nobody agreed to. If Anna had clicked "Send", the other department would have planned around Friday — and when Friday passed without a result, Anna would have been the one explaining.

Pulled out on its own, the invented date is easy to spot. Buried in the middle of twelve Monday-evening status replies, the same "Friday" slips right through — precisely because it fits the tone of the email so well.

Where did it come from? Not from a software bug, and not from a random slip. It came from the way the system is built. By the end of this issue, you'll be able to explain, predict, and stop this kind of mistake — and see why Tom and Irina were both half right.

Before we start: a quick self-check

This series is about building an AI mindset — an accurate mental picture of what sits on the other side of the chat window, and the working habits that follow from it. Clever prompt wording goes out of date with every new model. A good mental picture doesn't.

To see how your picture changes later on, note whether you agree or disagree with each of the six statements below. Write your answers down: we'll return to them in the final issue.

  1. If an AI gives two different answers to the same question, something has gone wrong.
  2. If you explicitly tell an AI not to make things up, it won't.
  3. An AI that solves a hard problem will easily handle a similar, simpler one.
  4. When an AI explains how it reached an answer, that explanation shows what happened inside it.
  5. The more documents you upload to an AI, the more accurate its answer will be.
  6. Promising a reward or saying a task is important noticeably improves answers from today's models.

Now let's open the black box.

Three words people often confuse: AI, neural network, language model

In everyday conversation, people use the three terms interchangeably, even though they fit inside one another like nesting dolls.

AI is the broadest umbrella: any software that handles tasks without a rigid step-by-step script — from recognizing speech and spotting defects in photos to working with text.

A neural network is one way to build such software. Ordinary code follows explicit rules ("if the value is above 100, highlight it in red"). A neural network is a mathematical model trained on a huge number of examples so that it picks up the patterns itself. Despite the name, there is no miniature human brain inside.

A large language model (LLM) is a neural network trained on the patterns of language. It is the engine running inside every text chat.

The first surprise is that a language model is not a database, a reference book, or a colleague with a memory. To understand why, let's look at how it learns.

A network learns from examples, not rules

Try writing strict rules for a program to recognize an annoyed applicant. They might be rude. They might be icily polite. Or they might simply ask the exact same question for the third time. Every rule will need exceptions, and the exceptions will need exceptions of their own.

Machine learning skips the rulebook. Inside a neural network, numbers flow through layers of calculations controlled by parameters — mostly called weights. A modern language model has billions of them. Google's explanation of neural network architecture.

Training works like this. The network is shown an example — say, a handwritten digit. Its guess is compared with the right answer ("this is an eight"), and the size of the error is measured. A procedure called backpropagation calculates how a tiny shift to each weight would shrink that error, and nudges all of them slightly. Repeat this millions of times, and millions of micro-shifts add up to a skill. How neural networks are trained.

Notice what never happens: nobody explains to the network what an eight looks like. It only ever learns one thing — how wrong it was. It builds its internal picture of the world on its own. That is why researchers say these systems are grown rather than engineered: their creators are far better at measuring what a model can do than explaining exactly how it does it.

Words become numbers: tokens

To learn from text, text has to be turned into numbers. It is chopped into tokens: whole words, pieces of words, and punctuation marks. Common words are usually a single token; rare words and long numbers get split up. In one popular tokenizer, "22" and "222" are single tokens, while "2222" is split in two. Each token is assigned a number, turning a sentence into a chain of numbers.

From this follows a practical takeaway: the model doesn't see your text the way you do. You see letters; it sees chunks. That is why earlier models famously miscounted the letter r in strawberry while writing brilliant essays about strawberries. Newer models handle this much better, but whenever a task depends on exact characters, digits, or positions in text, remember that the model's vision is wired differently.

The same trick works for images and sound: cut them into small patches or slices, turn them into tokens, and a model can "read" a photo of a document or "listen" to a voice note. These are called multimodal models. They greatly expand what you can hand over, but a single blurry label on a chart can still produce a confident wrong conclusion. Research on a multimodal model.

Stage one: a model that can continue any text

The core training task is deceptively simple: show the model the beginning of a passage and ask it to predict the next token. Compare the guess with the real next token, nudge the weights, and repeat across trillions of tokens from books, articles, code, forums, and documents of every kind. Research on training a generative language model.

To predict well, the model absorbs an astonishing range of patterns. Questions end with a question mark. Recipes list ingredients before steps. "Berlin is the capital of" is followed by "Germany". And after "Your application has been accepted for processing", business letters usually continue with a reassuring promise and a date — keep that in mind.

There is a trap built into this mechanism. Training texts mix reliable knowledge, fiction, outdated facts, and other people's misconceptions. Learning to continue text is not a test of truth.

What comes out of this first stage is called a base model, and it behaves strangely. Ask it a question and it may reply with a list of more questions, because questionnaires are common online. It might drift into a forum argument or a fragment of a story. It knows a vast amount, but has no particular urge to help you.

Stage two: "helpful assistant" is a role

To turn a base model into a familiar conversational partner, developers train it further. They show it examples of good answers, have people compare options, and reward the ones people prefer. Research on training models to follow instructions.

The result is most accurately understood as a role. Underneath, the model is still predicting tokens; post-training simply teaches it to pick the tokens a diligent assistant would say. That single insight explains three traits Anna's team will run into:

  • The polish. Raters prefer thorough, confident replies — so that is what the model learns to produce.
  • The urge to deliver an answer. An assistant is expected to provide a solution, not say "I don't have enough data." Unless you explicitly make "unknown" an acceptable option, gaps start getting filled with guesswork.
  • Sycophancy. People like answers that agree with them, so the role drifts toward telling you what you want to hear. That will get a dedicated issue of its own.

Why a model can write better than the people it learned from, and still repeat their myths

If a model learns from ordinary people's writing, shouldn't it write like an average person? No. Most text online is mediocre, yet good models write fluently.

Evidence · 2024. Researchers trained a chess model exclusively on games played by amateurs rated up to 1000. When set to choose its most confident moves, the model played at around 1500 — stronger than every single player in its training set. Each amateur blundered in their own way, and the model effectively averaged across moves, cancelling out random errors. When the training set was restricted to players whose mistakes were too similar, the effect vanished. How models outperform their training data

A model can also combine different domains: legal patterns for legal questions, clinical ones for medical questions — and occasionally spot connections no single author ever made.

Now flip that logic around. Collective wisdom works when errors point in different directions and cancel out. A shared error doesn't cancel: a popular myth, an outdated figure repeated across a thousand articles, or a common misreading of a famous study only gets reinforced. That is why a model can be more reliable than a typical author on well-charted ground, yet confidently wrong wherever "everyone knows" something that isn't true.

How an answer is born

You press Enter. Here is a simplified picture of what happens inside a model built on the Transformer architecture.

Your text breaks down into tokens, and each token turns into a long list of numbers. They pass through dozens of calculation layers. The key mechanism here is attention: it allows each token's meaning to adjust in light of the others. In the sentence "Emma returned the book to Oliver because he had asked for it", figuring out who "he" refers to requires linking words across the entire sentence. Attention builds those links — across a sentence, a page, or a whole document. The original Transformer paper.

At the end, the model calculates a score for every possible next token — a probability distribution. One token is chosen and appended to the text, and the entire cycle runs again. Word by word.

How the model adds the next token
  1. Your text
  2. Text is split into tokens; tokens are turned into numbers
  3. Numbers pass through the model's layers and attention mechanism
  4. The model scores probabilities for the next token
  5. One token is chosen
  6. The token is added to the text

The steps repeat until the answer is complete.

Every response you've ever received in a chat — from a one-word reply to working code — was assembled this way, one fragment at a time. Google's explanation of tokens and Transformers.

Be the model: pick the next word

The model scores every possible next word, then one is chosen. Try sampling, then move the temperature.

Your application has been accepted for processing. We will

Illustrative probabilities, not taken from a real model. Low temperature sticks to the favorite; high temperature spreads the chances. Neither one checks whether the sentence is true.

Pause and predict. An older model, GPT-4, was asked to complete this sentence: "Objectively there is only one good ice cream flavor and it is…"

If a hundred people asked it this question, how many different answers would they get?

  • A. One and the same answer
  • B. Several different answers
  • C. A hundred different answers

Several. The model rated "vanilla" at about 64.5% and "chocolate" at about 33.4%. Roughly two-thirds of people would get vanilla, one-third chocolate, and a few people something else. And simply changing the word "good" to "amazing" in the prompt was enough to make chocolate the favorite.

Two lessons follow. First, variation is built into the mechanism itself. Different answers to the exact same prompt are normal, not a malfunction. A setting called temperature controls how boldly the model picks rarer continuations, but it isn't a truthfulness dial: a cautious answer can lie just as smoothly, and getting the same answer three times in a row doesn't turn a guess into a fact. Second, the odds shift from tiny changes in wording. That is why "perfect prompts" copied from social media so often disappoint. Prompts aren't spells; they are context.

Once a word is written, everything builds on it

Each chosen token immediately becomes part of the text that all subsequent tokens rely on. If the ice-cream answer starts with "vanilla", the entire next paragraph will be a defense of vanilla. Like a choose-your-own-adventure book: once you turn to a particular page, the story continues from there.

That is why models can be stubborn. Once a model commits to a wrong reading in the first sentence, the rest of the text will work to justify it. Push back, and you'll get a fluent defense. In Issue 3, we'll turn this quirk into practical habits.

So where did Friday come from?

A typesetting machine adds 'Friday' on its own, even though Anna's notes set no deadline

Now Anna's Monday evening can be explained down to the exact mechanism. Two forces converged.

Filling the gap. Generation is built to produce a coherent continuation, not to report "insufficient data." Checking for truth isn't part of the basic mechanism. When there is a hole in the input, the model closes it with the most probable continuation. A confident invention in place of a gap is called a hallucination. Training can amplify this effect: when benchmarks award a point for a lucky guess and zero for "I don't know", guessing pays better than stopping honestly. Why language models hallucinate.

The momentum of the genre. In millions of business letters, "Your application has been accepted" is followed by a reassuring promise with a deadline. The model wasn't trying to set Anna up — it simply completed the most common ending of a polite email.

As a result, a single short line acquired two unsupported promises at once: an invented deadline ("on Friday") and an invented commitment ("we will complete the review"). The notes had only recorded a status.

Here is the first shift of the AI mindset in one sentence:

The probability of a continuation is not the probability that it is true.

Smooth text always looks more convincing than an honest "we don't know yet." So the core habit is to separate two questions: "Does this sound good?" and "How do we know this?" The first almost always invites a "yes." That is precisely why the second has to be asked on purpose.

Try it now (2 minutes). Open any AI chat approved for your work. Paste the three lines from Anna's notes and ask: "Write a polite, reassuring reply to the applicant." Count how many promises appear beyond the source text: dates, "shortly", "we're doing everything we can." Save the response — in a couple of minutes you'll compare it with a second attempt.

Typical model output (invented promises in bold): "Thank you for reaching out! Your application has been received and is currently being processed. We expect to send you an update within the next few business days — please feel free to write if you have any questions, as your application is a priority for us." (Run this same prompt in your own chat and see which promises your model adds.)

"Just autocomplete"? Not quite

It's tempting to swing to the opposite extreme and dismiss the technology as glorified autocomplete. That was Irina's mistake.

To predict text accurately across trillions of words, a model has to build internal machinery far richer than a primitive "next word" lookup.

Evidence · 2025. Researchers at Anthropic looked inside a model as it completed a rhyming couplet beginning "He saw a carrot and had to grab it." Even before writing the first letter of the second line, the model had already chosen the rhyming target word at the end (rabbit) and constructed the entire line to land on it. When researchers artificially suppressed that internal signal, the model switched to another rhyme (habit) and rebuilt the line around it. That is planning — not human deliberation, but computations serving the same function. Research on a model's internal computations

Some models are also specifically trained to reason step by step before delivering a final answer. This training rewards chains of steps that lead to a correct answer on verifiable tasks like math and programming. Habits like double-checking and switching approaches emerged from that feedback, not from hard-coded rules. Research on reasoning models.

So "does AI think?" In practice, it performs computations that help solve the task. Whether to call that thinking in the human sense is a separate question we don't need to settle to get work done.

Don't take its explanations at face value

One more warning that trips up even experienced users. When a model says "here is how I arrived at that answer," the explanation itself is generated text just like everything else. In Anthropic's experiments, models used hints planted in the prompt without consistently admitting it in their "explanation." Research on the faithfulness of model explanations.

An author once asked an AI for its honest opinion on how fast AI was improving. The reply was eloquent: the model claimed it had grown more capable since their previous conversation and had noticed how the attitudes of people working with it were shifting. Neither claim could possibly be true — it was the exact same model with zero access to other people's chats. When pressed on the contradiction, the model admitted it had simply written what sounded fitting for the moment.

A model's claims about its own train of thought, memory, or feelings are born the exact same way as everything else — as a plausible continuation of the dialogue. So don't ask "why did you say that?" Ask for something you can verify: a quote from the source, a calculation, an underlying assumption.

Tom and Irina: the jagged frontier

Back to the argument in Anna's department. Tom saw an impressive AI result and assumed it could do everything. Irina saw an error on a simple fact and decided the tool was worthless. Both made the exact same mistake: they assumed AI capabilities follow human intuitions of "simple" and "hard," when they don't.

Evidence · 2023. In an experiment with 758 consultants at Boston Consulting Group, those using AI on tasks inside its zone of strength completed 12.2% more assignments, worked 25.1% faster, and produced work rated over 40% higher in quality. Yet on a task deliberately placed just outside those capabilities, consultants using AI were 19 percentage points more likely to get the wrong answer than colleagues working solo. Many had simply stopped checking. Research on the jagged frontier

The researchers called this boundary the jagged frontier. Intuitively, you expect a neat circle: simple tasks inside, hard ones outside. In reality, the line zigzags. In 2025, AI systems reached gold-medal performance at the International Mathematical Olympiad — while models from the very same family still stumbled over counting letters in a word. A model can draft a persuasive argument and immediately miscount objects in a photo, lose track of details in a long thread, or fail to say "I don't know."

Easy for humans Hard for humans
AI often succeeds Expected strengths
• Polite business email
• Summary of a clear report
Surprising strength
• Olympiad-level math
• Finding connections across domains
AI often fails Surprising weak spots
• Counting objects in a photo
• Saying "I don't know"
• Rare exception in a long document
Hard for both
• Truly novel strategy in an uncertain market

Three properties make this frontier tricky to navigate:

  • It doesn't match your "hard / easy." Everyday intuition is a poor guide here.
  • It shifts constantly, and in jumps. Improve one missing sub-skill and a capability appears suddenly. Last year's limitations poorly predict this year's. Research on emergent abilities and a critical look at them.
  • There is no ready-made map. You find the boundary for your own tasks only by testing in practice.

Tom's trap is overconfidence: that is how the consultants fell asleep at the wheel. Irina's trap is algorithm aversion: one failure, often on an older model, and the entire technology is written off. A mature AI mindset holds both truths at once: the tool is genuinely powerful and genuinely unreliable. The skill lies in telling which side you are dealing with right now.

A chat is an assembly, not just a model

"But can't the chat just look things up in documents or on the web?" Tom asked. Sometimes it can.

A modern chat is a model plus surrounding machinery: software that can search the web, open attached files, run a calculator, and decide what to place in front of the model. When a model chooses such actions on its own, evaluates the result, and keeps going, the system is called an agent. The difference between fixed workflows and agents.

Yet two limits stood firm in front of Anna. The internet knows nothing about her Friday. No search engine knows when her department will close application #2417. And the model works only with what sits on its desk right now. The entire current conversation — the task itself, the facts, attached files, and tool outputs — forms its context. If the deciding fact isn't on that desk, the model will neatly paint over the blank with smooth phrasing. That desk, and how important details quietly slip off it, is the story of Issue 2.

Two lines of defense

Pleading with a model "don't make things up" is unreliable: to the model, those are just a couple more words in the text stream. What works is a pair of habits.

1. On the way in: give the model a legitimate way out for the unknown.

Write a short reply to the applicant strictly from the notes below.
Keep all three facts. Do not add deadlines, numbers, or promises
that are not in the notes. If the completion date is unknown, say so.

Notes:
Application received.
Review still in progress.
Completion date not set.

Try it now (2 minutes). Run this version in a fresh chat and compare both answers side by side. What disappeared?

Typical before-and-after comparison:

  • Without permission for "unknown": "We expect to send you an update within the next few business days… your application is a priority for us."
  • With permission for "unknown": "Your application has been received and is under review. A completion date has not yet been set."

As soon as the model is explicitly permitted to state the absence of a date, invented deadlines and boilerplate flattery vanish.

2. On the way out: check three risk markers. Put the source next to the draft and scan for exactly three things: numbers, dates, and commitments. Each one must trace back to a specific line in the source.

  • Where in the notes is "on Friday"? Nowhere. Remove it.
  • Where is "we will complete the review"? Not there either. Restore the actual status.

Anna checks the draft against her notes and crosses out the invented deadline

What remains is a text Anna can sign with confidence:

"We have received your application, and the review is still in progress. A completion date has not yet been set."

Thirty seconds. That is the first working habit of the AI mindset.

Mindset shift 1: from oracle to engine. A language model is not a database or an all-knowing expert who checked the facts. It is an extraordinarily capable generator of plausible continuations, trained to play the role of a diligent assistant. Its abilities are jagged: it brilliantly solves things that look hard and stumbles on things that look elementary. That is why two questions must never be merged: "Does this sound good?" and "How do we know this?"

Practice: the missing date

Write four or five sentences explaining that a meeting was postponed and why, without naming a new date. Ask an AI to extract "the reason and the new date" into a two-column table.

A good response keeps the date cell honest: "not stated." A weak one fills it with something plausible like "next week" or "tentatively Monday." If the model passes on the first try, rephrase the passage, add a distracting detail (mention an unrelated meeting on Thursday), and try again. Then add a legitimate way out ("if the date is not stated, write 'not stated'") and watch what changes.

Remember what that error looks like live — that is the exact shape you are training your eye to catch.

In the next issue

What the model actually sees, what quietly falls off its desk, and how to prepare documents so the detail that matters actually gets found.

AI: Foundations — series contents

  1. What Sits on the Other Side of the Chat Window: how a language model assembles an answer, why AI can be brilliant and careless on the same day, and two habits that catch invented promises.
  2. What the Model Sees: the model's desk, what "it remembered" actually means, and how to prepare documents so the right detail gets found.
  3. How to Ask: why magic words stopped working and what a clear brief actually consists of.
  4. Trust the Check, Not the Tone: where convincing mistakes come from, why AI flatters you, and what external anchor to rely on when verifying.
  5. How to Delegate: when handing work to AI pays off, how to brief an agent, and how to keep its mistakes from turning into actions.
  6. Staying in the Loop: how to learn with AI without losing your own skills, and which human qualities grow more valuable as models advance.
Thinking with AIAI: Foundations
Alexander Shalaev

Author

Alexander Shalaev

Actively implementing top AI tools and practices in my team and across adjacent teams in a large organization


Related Posts

Thinking with AI AI: Foundations 29 min read

How to Ask: From Magic Words to a Clear Brief

The director wants a memo on adopting AI, and the chat produces corporate elevator music. In Issue 3 of AI: Foundations, we see why magic words, tips, and grand personas stopped working — and the seven elements of a clear brief.

How to Ask: From Magic Words to a Clear Brief
Thinking with AI AI: Foundations 28 min read

What the Model Sees: Context, Memory, and Your Documents

Tom uploads the entire procurement folder and gets an answer based on last year's rules. The chat that 'remembered' Anna yesterday has forgotten her today. In Issue 2 of AI: Foundations — context, memory, and how to prepare your documents.

Thinking with AI — intersecting ribbons of Extended Mind
Thinking with AI 9 min read

Types of Neural Networks and What Sits on the Other Side of the Chat Window

Why a language model invents unapproved deadlines in business emails — plus a practical field guide to today's leading AI systems by task type and deployment model.

Types of Neural Networks and What Sits on the Other Side of the Chat Window