← drewkhoury.com about drew
Report · Interactive

How AI Works

A visual walk through the parts of an AI system: the message list a conversation really is, the context window it has to fit inside, the tools a model can reach for, and the loop a harness runs to tie all of it together.

Use the chapter buttons to move between diagrams. Each one animates through its steps as you go.

August 24, 2026
How AI Works
PDF
{{ dnum }} / {{ dcount }} · {{ dtitle }}
{{ caption }}
The conversation is a list of messages
system prompt · hidden
“You are ChefBot. Only discuss cooking — politely decline everything else.”
you
“What's a good pasta recipe?”
assistant
“Try cacio e pepe — three ingredients…”
you
“Make it vegetarian?”
↻ every send = this ENTIRE list again, top to bottom
the context window
{{ ctxTokens }}
tokens in the window
{{ ctxNote }}
system prompt
you
assistant
Never just your latest message — the whole list, from the top, every single time. The model has no memory of its own.
Text becomes tokens becomes numbers
1 · your text
The spaghetti was unbelievably good.
2 · chopped into tokens
{{ tk.t }}
common words = 1 token · rare words get split
3 · each token is just a number (ids below are illustrative)
{{ tk.n }}
~4 characters
per token, on average
~750 words
≈ 1,000 tokens of English
$ / token
pricing and limits are counted in tokens
not all text is equal “good” = 1 token · “unbelievably” splits into 2 · and dense code, JSON or logs use roughly twice as many tokens per character as plain prose
The context window — working memory with a hard limit
oldest messages — pushed out of the boxstill saved in your app · just no longer sent
↑ overflow falls out here
…the chat keeps growing…
conversation history
system prompt
{{ tokCount }} tokens in the boxa common limit today: ~200,000 tokens
only what's inside the box is sent to the model — nothing else exists for it
the key idea
{{ d3note }}
Every message you've sent — shipped to a sealed-off model
Your device
holds the conversation
system prompt + every message so far
the ENTIRE list →
every message, every time
← the reply streams back,
token by token
copy of the whole window
{{ t.text }}
a datacenter, somewhere else
{{ preLabel }}
public web text + licensed data
up to a cut-off date
→
months of training
on huge GPU clusters
↓ baked in once, then sealed
The model
frozen weights on racks of GPUs — a sealed box that turns text into text
{{ crunchLabel }}
✓ can see
only the messages you just sent
✕ cannot see
the live internet, your files, today's news
No memory between turns. No internet. Nothing past its training.
Tool calls — when the model needs the outside world
on your computer
in the datacenter · the model
you
“What's today's date?”
sent across →
model · has no clock and no calendar, so it asks
← “please run get_current_time() and send me the result”
your computer runs the tool · the model just waits
reads your system clock → 2026-08-23, 14:02 local
appended to the list as a message · whole conversation resent →
model · now with real data
← “Today is Sunday, 23 August 2026.”
the date is the clearest case — nothing in training could contain today. Same shape for your files, live prices, your calendar.
Sub-agents — splitting a job too big for one context window
lead agent · the one you're talking to
“Research this 300-page report”
{{ l.text }}
{{ b.text }}
{{ u.text }}
{{ wireLabel }}
sub-task {{ sub.n }}
{{ sub.job }}
{{ sub.doing }}
{{ sub.used }}
↑ each sub-agent read tens of thousands of tokens — only these few lines climb back into the lead's window
…which is why the lead can still answer without ever holding the 300 pages
the key idea
Three fresh windows instead of one overflowing one. The lead holds only the summaries — that's the real reason to split work, not magic.
Worth it when the pieces are genuinely independent. For small, tightly connected work one agent is clearer — and the lead stays accountable either way.
Why it sometimes makes things up
“The capital of France is ”
the model scores every possible next token
{{ p.word }}
{{ p.pct }}
it picks a likely one, adds it to the text, and repeats — one token at a time, forever forward
so what's a hallucination?
When it doesn't know, plausible tokens still score highest — so out comes a confident, fluent, wrong answer.
There's no built-in “I don't know”. Hence tools and sources.
Your own documents — the point: your files never go into the model
your insurance policy
300 pages — far too big to send
you ask: “am I covered for a cracked phone screen?”
every page is scored for closeness in meaning — not keywords, so “accidental damage to handsets” scores high without the word “cracked”
top 3 by score · scores illustrative
accidental damage, p.410.91
handsets & screens, p.880.87
excess & limits, p.2120.83
…297 other pagesdropped
→
the context window
pasted in alongside your question
Nothing was “trained on your data”. Three pages were pasted into the window, read once, and forgotten — the model is answering from text you handed it a second ago.
Which explains the failure you'll actually hit: if the search picks the wrong pages, the answer is wrong — and the model has no idea it's missing anything.
Putting it together — watch one job go round the loop
your computer
holds the list · runs tools
{{ loopLact }}
{{ loopArrow }}
the message list{{ loopTok }} tokens
{{ m.label }}
it only ever gets longer — every trip resends all of it
the model
reads the list · replies once
{{ loopRact }}
trip {{ loopTrip }} of the loop
An “AI agent” is only this: the list grows, it gets resent, the model answers or asks for a tool. The loop lives on your side.
When the box fills up
92% full · 40 turns of chat
slower · pricier · easier to distract
1 · Auto-compact
clears the oldest tool results, then summarises
creates a summary of the conversation — detail doesn’t survive
2 · Compact
ask the model to summarise the old part, keep the summary
30 turns → one paragraph
the same thing, but you guide what’s worth keeping
3 · Start fresh
new chat, paste back only what matters
cheapest and sharpest — you decide what's worth carrying over
A clean, compact window beats a big messy one: everything in the box competes for the model's attention, so irrelevant history actively makes answers worse.
So can it do maths? — recall vs. calculation
“1 + 1 = 2”
✓ right — but not because it added anything.
“1 + 1 =” is followed by “2” countless times in its training text, so “2” is overwhelmingly the likeliest next token. This is recall.
“4,391 × 7,208 = 31,648,528”
✕ a typical wrong answer. The true value is 31,650,328.
That exact sum appears nowhere in its training data, so it predicts digit by digit: right length, plausible shape, wrong number.
the fix is the same as always
model asks: calculate("4391*7208")
→
your computer computes it exactly
→
31,650,328
The rule of thumb: it's strong at language-shaped tasks it has seen patterns of, and unreliable at anything needing exact computation or live facts — which is precisely what tools are for.
Recap — the whole thing in four ideas
The list
A chat is a list of messages. It's resent in full every turn — that's the only reason it seems to remember.
The box
The window is a fixed box measured in tokens. Only what's inside exists; the overflow is genuinely gone.
The guess
It only predicts the next token. Fluent and confident even when wrong — and useless at exact sums.
The loop
look
→
act
→
check
↺
Everything else — tools, skills, agents, sub-agents — is software running that loop around the box.
A frozen text predictor, handed a list of messages in a fixed box, run in a loop by ordinary software that does all the fetching.
Hold onto that and the rest is detail: hallucinations are the predictor, forgetting is the box, and anything it “does” is the loop calling a tool.
A skill is a text file · nothing more
monthly-report.md
# Monthly report
Always pull figures from the finance sheet, never from chat.
## Steps
1. Check last month's file for the format
2. Fill the four sections in order
3. Flag anything down more than 5%
## Done when
Every number has a source noted beside it.
That's the whole artefact — a plain markdown file someone wrote once. No code, no training, no new abilities. When the work matches, the harness pastes it into the window as instructions, and behaviour changes because the instructions changed.
why bother
It saves you re-explaining your own process every time — and it means two people get the same result out of the same tool.
and you don't have to write it yourself
Describe the process in chat and ask for a skill file. Most tools have a skill for making skills — you talk, it writes the markdown, you correct it.
If you can write a good handover note for a colleague, you can write a skill. It's the same document.
Who actually does the work — one plugin, step by step
the same map, lighting up
the harness
skills
plugins
MCP
the window
the model
the world · your calendar · the web
your side
you connect the calendar plugin. Your side keeps the login and tells the model only that a tool called list_events exists.
the model
all it can emit is a line of text asking for that tool. It cannot call anything.
your side
signs in, calls the service, retries on failure, cuts 200 events down to tomorrow’s six.
→ the model
those six lines go back as a message, read like any other text.
so…
it never held your password or opened a connection. When a plugin misbehaves it is the plumbing, not the model.
Start here
How AI works, end to end
What actually happens between pressing Enter and reading the reply — and why AI behaves the way it does.
part one
What the model is
A chat is a list of messages in a fixed-size box. It has no memory of its own.
part two
What it can't do
It guesses the next word. That's why it invents things, and can't do sums.
part three
How it reaches the world
model
⇄
tool
→
files, web
It can't reach anything itself, so it asks the software around it to fetch things.
part four
Agents at work
look
→
act
→
check
↺
An agent is that same model, run in a loop until the job is done.
The words you'll hear · 1 of 2 · the basics
{{ g.term }}
{{ g.def }}
The words you'll hear · 2 of 2 · the machinery around the model
{{ g.term }}
{{ g.def }}
The harness — the software wrapped around the model
the harness · the app, on your side
{{ h.text }}
text
tokens
someone else’s datacenter
the model
frozen · stateless · text in, text out
the world your files · the web · your services
✕ not from here
why the word matters
Almost everything you think of as “the AI” is the harness, not the model: ChatGPT, Claude, Copilot and Codex are harnesses.
Same model, different harness → very different experience. It explains why one product can browse and another can't, why one remembers your files, and why “the AI can't do X” is usually the harness, not the model.
It's also who does the forgetting: trimming, compacting, and choosing what to send are all harness decisions.
Three words people mix up — tools, skills, plugins
a capability
Tool · tool call
A tool is something the harness can do — read a file, search, run a sum. A tool call is the model asking for one and getting the result back as text.
gives it new abilities
instructions
Skill
A written playbook for a kind of work — “here's how we build a spreadsheet”. Just text, pulled into the window when it's relevant.
gives it know-how, no new powers
plumbing
Plugin · connector
A package you switch on that hands the harness a bundle of new tools — connect your email, and “search inbox” and “send draft” appear.
delivers more tools
Skills teach · plugins add · tool calls do.
And none of them change the model itself — it's still the same frozen weights reading text. They change what's in the window and what the harness can run.
Products name these differently — “actions”, “connectors”, “extensions”, “MCP servers”. Ask which of the three it is and the confusion clears.
What is an “agent”, exactly?
a plain chatbot reply
you ask
→
it answers
■
one pass. Whatever it knew, you get. No looking, no checking.
an agent · same model, run in a loop
{{ a.name }}
{{ a.doing }}
↺ {{ agentNote }}
“acting” = calling a tool
read a file · run the tests · search the docs · click through a web app in a real browser · take a screenshot · call an API — each result comes back as text in the window
You talk to one agent per conversation. It may hand bounded pieces of work to helper agents — but they're workers, not colleagues you have to manage.
The one you're talking to stays accountable for the final answer. It's not “one agent per task” — it's one agent per request, plus however many helpers that request needs.
Anything that must outlive the window has to be written down
the context window
temporary by design
the decision you made 30 turns ago is gone. As far as the model is concerned, it never happened.
write it out →
← read it back in
next session
durable artifacts · outside the window
a spec or design doc
source files
a commit or pull request
an issue / ticket
a project instructions file
test output, screenshots
a decision record
This is why “write the spec first” works so well with agents: the spec is a persistent artifact. It outlives the chat, and it gives you and the agent one shared definition of “done”.
Rule of thumb: if losing it would hurt, it doesn't belong in the chat — it belongs in a file.
The three things you actually control
1 · Your prompt
everything it knows about the job arrives this way
the goal — what you want
the constraints — what it must respect
what “done” looks like
leave one out and it will cheerfully invent it
2 · Skills
reusable instruction packs, written once
“how we build a spreadsheet”
a playbook — steps, conventions, sometimes scripts
it gets pulled into the window when the work matches, so you don't re-explain the same process every time. It's instructions — not a smarter model.
3 · Which model
and you can switch part-way
fast + cheap → sorting, drafting, bulk edits
strong reasoning → the tricky bug, the design call
the catch: a new model reads the same window differently — so keep the plan and decisions written down, not implied.
Notice what all three have in common: they're just text you put in the window. That's the whole lever — there's nothing else you can reach.
three places — and almost all of it is yours
your side
the harness
☰ skills/files on disk
→
⊞ pluginsswitched on, register tools
→
● MCP serverssmall programs, running
→
the window
text
tokens
a datacenter
the modelfrozen weights · reaches nothing
↓ real calls leave here
carrying your credentials
the world your calendar · your repo · a payments API · the web
✕ never from here
Three different kinds of thing — a file, a package, a running program — all doing one job: getting a say in the message before it leaves your machine.
your message stops here first
the same map, lighting up
the harness
skills
plugins
MCP
the window
the model
the world · your calendar · the web
you
“what’s on my calendar tomorrow?”
harness
adds system prompt · skill descriptions · tool names
→ model
reads the whole assembled window
model
asks: run get_events for tomorrow
connector
calls the calendar service — 6 events back
→ you
appended as text, the model reads again, answer arrives
The model never touched your calendar. Every real thing that happened — the network call, the credentials, the six events — happened in the software around it.
what we just covered
{{ dtitle }}
{{ r.n }}
{{ r.text }}
coming next
{{ nextTitle }}
{{ nextTeaches }}
{{ dnum }} · {{ dtitle }}
{{ actProgress }}
teaches: {{ dteaches }}