Where it breaks: the real bottlenecks
Compounding errors, the cost staircase, context rot and trust.
Learning goal: You can tell a demo from a product in one question.
1 / 13
Every demo works. This lesson is about the gap between a demo and a Tuesday afternoon.
Before you start
- You need: to have finished Level 3 - this lesson assumes you already know what an agent's loop looks like.
- You need: access to any AI chat tool, ideally one where you can start a long conversation.
- You need: no technical background; the only maths is multiplying a decimal by itself, done for you above.
- You need: about 15 minutes, plus the patience to have one deliberately long conversation.
- Mindset: you're hunting for where a good demo quietly turns into a bad Tuesday.
Do this first:
- Start a new AI chat and ask it to do a five-step task (e.g. "plan a 3-day trip, step by step, checking in with me after each step").
- Let it run all five steps without correcting it, even if something looks slightly off.
- Look back at step 5 and compare it to what you actually wanted from step 1 - notice how far it drifted.
Bottleneck 0: the one nobody puts on the slide
An agent that is right 95% of the time sounds excellent. Now run it ten steps in a row:
0.95 ^ 1 = 95% great 0.95 ^ 5 = 77% annoying 0.95 ^ 10 = 60% unusable 0.95 ^ 20 = 36% a random number generator with good manners
Errors compound. This single line of arithmetic explains why long autonomous agents keep disappointing people, and it is why the best products in the world today are short loops with a human glancing at the result. If you take one thing from this whole course, take this.
The fixes are unglamorous and they work: fewer steps, checkpoints where code verifies the model's output, and a human in the loop at the expensive moments.
Bottleneck 1: time
animated · where the time goes
Stack up a single agent turn and see which stage eats the seconds.
running total 0.07s · a human gives up at ~3s
the lesson
Every stage is fast except the two that involve a model. Waiting is the default state of an agent.
Play it through. The shape is always the same - the model stages dwarf everything else, and a multi-step agent pays that cost repeatedly.
What actually helps:
- Streaming. Showing words as they arrive doesn't make it faster, it makes it feel three times faster. Perception is a real engineering target.
- Parallel tools. If it needs the weather and the flights, fetch both at once. Free seconds.
- Small model for small jobs. Classifying "is this spam?" does not need the flagship model.
- Caching. The same long instructions get re-sent every turn - providers will cache them for you, often around 90% cheaper on the repeated part.
Bottleneck 2: money, and the curve nobody warns you about
You pay per token, in and out. Because the whole transcript is re-sent every turn, cost per conversation does not grow in a line - it grows like a staircase getting taller.
turn 1 : 1k tokens → 1k paid
turn 5 : 5k on the desk → 5k paid
turn 20: 20k on the desk → 20k paid
─────
total ≈ 210k for 20 turns, not 20kA chat that costs a fraction of a cent at turn one costs twenty times that at turn twenty. Nobody notices until the monthly bill arrives. Summarising old turns is not a nice-to-have; it is the economics of the product.
Bottleneck 3: the context window, again
Bigger windows did not solve this. Two things stay true no matter how big the window gets:
- Attention is uneven. Models read the beginning and the end carefully and skim the middle. Bury your key rule in the middle of 80 pages and it may as well not be there.
- More context can make answers worse. Ten mediocre documents crowd out the two great ones. This is called context rot, and it is why serious systems spend most of their effort deciding what not to include.
Practical version for you as a user: paste less, but paste better. Two relevant pages beat a whole handbook.
Bottleneck 4: retrieval - the quiet killer
When a product "chats with your documents", something searched for chunks of text and pasted them onto the desk. If that search returned the wrong five paragraphs, no model on earth saves you. You get a fluent, confident, well-formatted wrong answer.
Retrieval is one possible failure point. Check whether the right evidence reached the model before changing the model itself. Reasoning, tool use and unclear requirements can also fail.
Bottleneck 5: reliability you inherit
Your agent calls three APIs. Each is up 99.9% of the time. Your agent is up 99.7% - before your own bugs. Real systems need retries with a random wait, timeouts, and a plan for "the tool is down" that isn't "the model invents a plausible answer instead". (That last failure mode is real and it is nasty.)
Bottleneck 6: trust, which is the actual product
An agent that is usually right but must always be checked has saved you nothing. It has moved the work from doing to reviewing, and reviewing is often the more tiring half.
This is why the winning designs are so often narrow:
- do one job
- show the evidence - the source, the number, the receipt
- make it obvious when it is unsure
- make undo easy and total
A hard-won rule: never automate the step a human would want to double-check anyway. Automate everything around it and put the human at that one moment. That is a product; full autonomy on the same task is a liability.
Bottleneck 7: safety, in the only framing that stays useful
Danger appears when three things meet in the same agent:
- it can read your private data, and
- it can be reached by untrusted content (a web page, an email, a PDF someone sent), and
- it can act on the outside world (send, post, pay, delete).
Any two are fine. All three and a malicious web page can contain instructions your agent obediently follows - it cannot reliably tell your words from words it found. Serious systems break one leg of that triangle on purpose. If you are evaluating an AI product, ask which leg they broke. If they don't understand the question, that is your answer.
Where things stand (as of 2026)
Prompt caching and cheaper, smaller models for simple sub-tasks are now routine ways of controlling cost, and most serious products use them. What is still unreliable: nobody has a general fix for compounding errors in long agent chains - the honest answer remains "keep the loop short and put a human at the expensive step." Context windows keep growing, but context rot (mediocre information crowding out the good stuff) has not gone away just because the window got bigger. Retrieval quality - finding the right paragraphs to paste onto the desk - is still the single most common cause of "the AI is stupid" complaints. Exact prices, context sizes, and caching discounts change every few months and vary by provider, so treat any number you read here or elsewhere as something to verify against the current pricing page before relying on it.
How to read the docs and look things up
The pricing and limits pages on a provider's own site are the primary source for cost and context-size claims - not a blog post from a year ago. Changelogs and release notes are where you'll find out that a model got cheaper, faster, or gained a bigger context window; they're usually more current than the main marketing pages. To spot marketing vs. spec: a spec gives you a table of exact token prices and limits with a last-updated date; marketing says "blazing fast" and "incredibly affordable" with no numbers. To test a reliability claim yourself in ten minutes: run the same multi-step task three times and see how much the final answer varies, or run a long conversation and see when it starts ignoring your first instruction. Keep dated notes - "as of March 2026, X model cost Y per million tokens" - because these numbers are exactly the ones that go stale fastest.
Check yourself
- Why does a 95%-accurate agent become useless over 20 steps, and what are two ways to fight it?
- Why does turn 20 of a chat cost so much more than turn 1?
- The answer was fluent and completely wrong about your own company's policy. Which bottleneck was it, most likely?
- Name the three ingredients that, together, make an agent genuinely unsafe.
Recap: what changed in your head
- You understand why a 95%-accurate agent becomes nearly useless after twenty steps.
- You know why long chats get quietly, dramatically more expensive.
- You know a bigger context window doesn't fix uneven attention or context rot.
- You know most "the AI is dumb" complaints are actually retrieval failures.
- You know real danger needs all three of private data, untrusted input, and the ability to act.