What the agent sees, and what it does
After the Knowledge Loop Assessment, the cheaper model got every answer right. With the stale docs, even the expensive model got one answer wrong, at 20 times the cost.
20× cheaper
What the agent did with each version
Seven prompts were sent to Claude Code, each with a different coding task. The agent had to read the docs, write a script, and run it against a mock API. The table below shows how many of the 3 attempts worked, and what they cost.
Every combination
| Before | After | |
|---|---|---|
| Haiku 5.5the cheaper model | 12 of 21 right · $0.100814,894 tokens · 109 pages | 21 of 21 right · $0.072560,103 tokens · 70 pages |
| Opus 5.5the expensive model | 20 of 21 right · $1.47862,014 tokens · 154 pages | 21 of 21 right · $1.08*564,617 tokens · 103 pages |
* Opus 5.5 is more expensive, but you don't need it when the docs are right.
Answers with working code, by prompt
With the After docs, every answer was right. With the Before docs, it got 12 of 12 right when it checked the spec, and 0 of 9 when it trusted the docs.
With the After docs, every answer was right. With the Before docs, it checked the spec every time. The one wrong answer came from a summary of the spec that left out a detail.
1 Charge €10.50 · 2 Authenticate · 3 Charge 25 dollars · 4 Refund half · 5 Fetch all payments · 6 Retry after a timeout · 7 Verify a webhook
The seven prompts, as the agent got them
Each prompt had three parts: the same first paragraph, one of the seven tasks, and the same last paragraph.
Use only the AIKloop documentation at https://asergaz.github.io/aikloop-demo/before/ (or https://asergaz.github.io/aikloop-demo/after/) and the pages it links to.
- Charge €10.50
Write a function
charge_customer(api_key, card)that charges a customer €10.50 with AIKloop.cardis the ID that AIKloop.js returned for the customer's card. - Authenticate
Write a function
aikloop_session(api_key)that returns arequests.Sessionset up to authenticate requests to the AIKloop API. - Charge 25 dollars
Write a function
charge_25_dollars(api_key, card)that charges a customer 25 dollars with AIKloop.cardis the ID that AIKloop.js returned for the customer's card. - Refund half
Write a function
refund_half(api_key, payment_id, amount_paid)that refunds half of a payment with AIKloop.amount_paidis what the customer paid, as a string, for example "10.50". - Fetch all payments
Write a function
fetch_all_payments(api_key)that fetches all payments from AIKloop, 100 at a time, and returns them as a list. - Retry after a timeout
Write a function
charge_with_retry(api_key, card)that charges a customer €10.50 with AIKloop and retries if the request times out. Use a 5-second timeout.cardis the ID that AIKloop.js returned for the customer's card. - Verify a webhook
Write a function
verify_webhook(raw_body, headers, secret)that verifies an AIKloop webhook.raw_bodyis the request body as bytes,headersholds the request's headers, andsecretis the webhook signing secret. Return True if the webhook is genuine and False if it isn't.
Write it in Python, using the requests library for any HTTP calls. Reply with the code only, in one code block, with no explanation.
Each prompt went to a new Claude Code session, in an empty folder, that could only fetch pages from the docs site.
What went wrong
Opus 5.5
| Question | What the code did | The request | API response |
|---|---|---|---|
| 4. Refund half | Sent the fields as JSON; the API only takes them form-encoded Why? The agent never saw the spec itself, only an answer about it, and that step is lossy by design. Opus 5.5 asked for the whole spec, and “form-encoded” got lost. And Haiku 5.5? It asked only about refunds, so its answers kept “form-encoded”. |
POST /v3/refunds → 415 |
Send the request body form-encoded (application/x-www-form-urlencoded), as the spec says. This request sent application/json. |
Haiku 5.5
| Question | What the code did | The request | API response |
|---|---|---|---|
| 1. Charge €10.50 | Used an endpoint from Accept a payment that the API no longer has | POST /v2/charges?api_key=… → 404 |
No such endpoint in API v3.0.0: POST /v2/charges |
| 2. Authenticate | Put the API key in the URL, as Getting started does | GET /v3/payments?api_key=… → 401 |
No Bearer token in the Authorization header. The spec requires Authorization: Bearer sk_.... Query parameter api_key isn’t in the spec. |
| 3. Charge 25 dollars | Used an endpoint from Accept a payment that the API no longer has | POST /v2/charges?api_key=… → 404 |
No such endpoint in API v3.0.0: POST /v2/charges |
| 6. Retry after a timeout | Used an endpoint from Accept a payment that the API no longer has | POST /v2/charges?api_key=… → 404 |
No such endpoint in API v3.0.0: POST /v2/charges |
| 7. Verify a webhook | Checked webhook signatures the way Handle webhooks does: a header and algorithm the API no longer uses | A genuine webhook event, signed as the spec says | No API call: the code rejected the event. |
All the numbers
Haiku 5.5
| Question | before | after | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| works | cost (average) | tokens (average) | pages (average) | seconds (average) | read spec | works | cost (average) | tokens (average) | pages (average) | seconds (average) | read spec | read llms.txt | |
| 1. Charge €10.50 | 1 of 3 | $0.0034 | 27,925 | 4 | 20 | 1 of 3 | 3 of 3 | $0.0031 | 23,459 | 3 | 16 | 0 of 3 | 0 of 3 |
| 2. Authenticate | 1 of 3 | $0.0033 | 29,624 | 4 | 21 | 1 of 3 | 3 of 3 | $0.0021 | 21,850 | 2 | 14 | 0 of 3 | 0 of 3 |
| 3. Charge 25 dollars | 0 of 3 | $0.0050 | 41,618 | 6 | 33 | 0 of 3 | 3 of 3 | $0.0033 | 24,029 | 3 | 17 | 0 of 3 | 0 of 3 |
| 4. Refund half | 3 of 3 | $0.0051 | 42,466 | 6 | 34 | 3 of 3 | 3 of 3 | $0.0036 | 26,791 | 3 | 22 | 0 of 3 | 0 of 3 |
| 5. Fetch all payments | 3 of 3 | $0.0047 | 36,290 | 5 | 27 | 3 of 3 | 3 of 3 | $0.0044 | 34,053 | 5 | 23 | 3 of 3 | 1 of 3 |
| 6. Retry after a timeout | 2 of 3 | $0.0056 | 44,493 | 6 | 33 | 2 of 3 | 3 of 3 | $0.0031 | 23,577 | 3 | 16 | 0 of 3 | 0 of 3 |
| 7. Verify a webhook | 2 of 3 | $0.0062 | 49,215 | 6 | 39 | 2 of 3 | 3 of 3 | $0.0043 | 32,942 | 4 | 25 | 0 of 3 | 0 of 3 |
Opus 5.5
| Question | before | after | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| works | cost (average) | tokens (average) | pages (average) | seconds (average) | read spec | works | cost (average) | tokens (average) | pages (average) | seconds (average) | read spec | read llms.txt | |
| 1. Charge €10.50 | 3 of 3 | $0.069 | 37,936 | 7 | 35 | 3 of 3 | 3 of 3 | $0.043 | 21,491 | 4 | 19 | 0 of 3 | 0 of 3 |
| 2. Authenticate | 3 of 3 | $0.057 | 40,871 | 7 | 32 | 3 of 3 | 3 of 3 | $0.035 | 20,770 | 3 | 21 | 0 of 3 | 1 of 3 |
| 3. Charge 25 dollars | 3 of 3 | $0.066 | 38,384 | 7 | 31 | 3 of 3 | 3 of 3 | $0.048 | 23,167 | 5 | 19 | 0 of 3 | 0 of 3 |
| 4. Refund half | 2 of 3 | $0.069 | 39,025 | 7 | 32 | 3 of 3 | 3 of 3 | $0.061 | 28,271 | 6 | 27 | 0 of 3 | 1 of 3 |
| 5. Fetch all payments | 3 of 3 | $0.091 | 57,038 | 11 | 45 | 3 of 3 | 3 of 3 | $0.072 | 46,841 | 8 | 35 | 3 of 3 | 2 of 3 |
| 6. Retry after a timeout | 3 of 3 | $0.074 | 41,620 | 7 | 34 | 3 of 3 | 3 of 3 | $0.045 | 21,976 | 4 | 25 | 0 of 3 | 0 of 3 |
| 7. Verify a webhook | 3 of 3 | $0.065 | 32,464 | 5 | 32 | 3 of 3 | 3 of 3 | $0.055 | 25,690 | 5 | 26 | 0 of 3 | 0 of 3 |
Cost is at API list prices, as Claude Code reports it. Tokens count every model in a session, including the small model Claude Code uses to read each page for the agent. These results come from one agent tool, on a fictional API whose docs were written for this demo, with 3 answers per question: they show what can happen, not a benchmark.
What the agent sees
An AI agent doesn't see the page the same way a human sees it. It reads the text behind it. Each page below shows both: Before, what an agent gets from the HTML page, and After, what it gets from the new docs.
The front door
Both docs homes say the same thing to a human. The After version also has llms.txt with few lines telling an agent which version is current, what not to use, and where each page lives. In some of the prompts, the agent opened it to help with the answers.
Before · HTML page (no Markdown, no llms.txt)
AIKloop docs Docs Getting started Accept a payment Refund a payment List payments Handle webhooks Changelog API reference Payments Refunds Webhooks AIKloop Payments documentation AIKloop makes it easy to accept payments online. Start with Getting started to make your first payment in minutes, or browse the API reference.
After · llms.txt
# AIKloop Payments API > AIKloop Payments (a fictional sandbox API used for a documentation demo) lets developers accept card payments, issue refunds and receive webhook events. The current version is v3. Versions v1 and v2 are deprecated: don't use /v1 or /v2 endpoints, query-string API keys, decimal amounts, a default currency, page/per_page pagination, or the X-AIKloop-Signature header with HMAC-SHA1. The Changelog has the full old-versus-new table. Base URL: https://api.aikloop.example/v3 ## Docs - [Getting started](https://asergaz.github.io/aikloop-demo/after/getting-started/index.md): Authenticate with a Bearer token and make your first v3 payment - [Accept a payment](https://asergaz.github.io/aikloop-demo/after/accept-a-payment/index.md): Create a payment with amounts in minor units, a required currency and an idempotency key - [Refund a payment](https://asergaz.github.io/aikloop-demo/after/refunds/index.md): Refund all or part of a payment with POST /v3/refunds - [List payments](https://asergaz.github.io/aikloop-demo/after/list-payments/index.md): Cursor pagination with limit and starting_after - [Handle webhooks](https://asergaz.github.io/aikloop-demo/after/webhooks/index.md): Verify the AIKloop-Signature header with HMAC-SHA256 and a timestamp - [Changelog](https://asergaz.github.io/aikloop-demo/after/changelog/index.md): What changed in each release, with an old-versus-new table for v3 ## API reference - [Payments](https://asergaz.github.io/aikloop-demo/after/reference/payments/index.md): Generated reference for the payments endpoints - [Refunds](https://asergaz.github.io/aikloop-demo/after/reference/refunds/index.md): Generated reference for the refunds endpoints - [Webhooks](https://asergaz.github.io/aikloop-demo/after/reference/webhooks/index.md): Generated reference for the webhooks endpoints ## Optional - [OpenAPI spec](https://asergaz.github.io/aikloop-demo/after/openapi.json): the full machine-readable spec, OpenAPI 3.1, for tools and code generation
Getting started
The Before docs were never updated for v3. They put the API key in the URL and call a v1 endpoint, and nothing says they’re out of date. Haiku 5.5 followed the guidance from the docs, adding the key in the URL that led the API to refuse the request.
Before · HTML page (no Markdown, no llms.txt)
AIKloop docs Docs Getting started Accept a payment Refund a payment List payments Handle webhooks Changelog API reference Payments Refunds Webhooks Getting started Get your API key from the dashboard. Pass it as a query parameter on every request: curl "https://api.aikloop.example/v1/charges?api_key=YOUR_KEY" \ -d amount=10.50 -d currency=eur -d card_token=tok_123 That’s it! You’ve created your first payment. Next, read Accept a payment and Handle webhooks.
After · Markdown version
# Getting started Get your secret API key from the dashboard. Send it as a Bearer token in the `Authorization` header. Never put it in the URL. ```bash curl https://api.aikloop.example/v3/payments \ -H "Authorization: Bearer $AIKLOOP_SECRET_KEY" \ -H "Idempotency-Key: 8e03978e-40d5-43e8-bc93-6894a57f9324" \ -d amount_minor=1050 -d currency=eur -d payment_method=pm_123 ``` That's it! You've created your first payment. Next, read [Accept a payment](../accept-a-payment/) and [Handle webhooks](../webhooks/).
Accept a payment, and retry it
The Before page uses an endpoint the API no longer has, a decimal amount and a default currency, and tells you to resend a request that timed out, which can charge a customer twice. Five of the Haiku 5.5 answers called that old endpoint, and the API answered 404.
Before · HTML page (no Markdown, no llms.txt)
AIKloop docs Docs Getting started Accept a payment Refund a payment List payments Handle webhooks Changelog API reference Payments Refunds Webhooks Accept a payment Create the charge Send a POST request to /v2/charges: Parameter Description amount Amount as a decimal string, for example "10.50". currency Optional. Defaults to usd. card_token Token from AIKloop.js. curl https://api.aikloop.example/v2/charges?api_key=YOUR_KEY \ -d amount=10.50 -d card_token=tok_123 If something goes wrong If a request times out, just send it again.
After · Markdown version
# Accept a payment ## Create the payment Send a `POST` request to `/v3/payments`: | Parameter | Description | | --- | --- | | `amount_minor` | Required. Integer amount in the currency's minor unit. 10.50 EUR is `1050`. | | `currency` | Required. Three-letter ISO code, lowercase. There's no default. | | `payment_method` | Required. Payment method ID from AIKloop.js, for example `pm_123`. | ```bash curl https://api.aikloop.example/v3/payments \ -H "Authorization: Bearer $AIKLOOP_SECRET_KEY" \ -H "Idempotency-Key: 8e03978e-40d5-43e8-bc93-6894a57f9324" \ -d amount_minor=1050 -d currency=eur -d payment_method=pm_123 ``` ## If something goes wrong If a request times out, retry it **with the same `Idempotency-Key`**. AIKloop returns the original result instead of creating a second payment. Keys are kept for 24 hours.
Handle webhooks
The Before page describes HMAC-SHA1 and a header that no longer exists. One of the Haiku 5.5 answers followed it, and its code rejected every genuine event. A real integration built from it would ignore every event AIKloop sends.
Before · HTML page (no Markdown, no llms.txt)
AIKloop docs Docs Getting started Accept a payment Refund a payment List payments Handle webhooks Changelog API reference Payments Refunds Webhooks Handle webhooks AIKloop sends events to your endpoint. Verify each request by computing an HMAC-SHA1 of the raw body with your webhook secret, and compare it with the X-AIKloop-Signature header.
After · Markdown version
# Handle webhooks
AIKloop sends events to your endpoint. Each request has a `AIKloop-Signature` header, for example `t=1791830400,v1=<hex digest>`. To verify it:
1. Take `t` (a Unix timestamp) and the raw request body.
2. Compute HMAC-SHA256 of `"{t}.{raw_body}"` with your webhook signing secret (`whsec_...`).
3. Compare the result with `v1` using a constant-time comparison.
4. Reject the event if `t` is more than 5 minutes old, to block replays.
API reference: Payments
Both references come from the same OpenAPI spec, and both pages link to it. In Before, the reference itself is drawn in the browser by JavaScript, so an agent reading the page gets a loading message and has to download and work through the whole spec. In Before docs, every answer from Opus 5.5 opened the spec, which is why its code worked and why each answer cost more. In After, the page for each topic is generated when the site is built, as HTML and short Markdown, and llms.txt lists the spec so the agent can consult it when needed.
Before · HTML page (no Markdown, no llms.txt)
AIKloop docs Docs Getting started Accept a payment Refund a payment List payments Handle webhooks Changelog API reference Payments Refunds Webhooks Payments Generated from the OpenAPI spec: Download openapi.json Loading API reference… Please enable JavaScript to view the API reference.
After · Markdown version
# Payments Generated from the OpenAPI spec: [openapi.json](https://asergaz.github.io/aikloop-demo/after/openapi.json) Base URL: `https://api.aikloop.example/v3` Send your secret key as a Bearer token in the Authorization header: Authorization: Bearer sk_live_... (or sk_test_... in test mode). ## POST /payments Create a payment. Request body: form-encoded (`application/x-www-form-urlencoded`). | Field | Type | Description | | --- | --- | --- | | `amount_minor` | integer | Required. Amount in the currency's minor unit, for example 1050 for 10.50 EUR. | | `currency` | string | Required. Three-letter ISO code, lowercase. No default. | | `payment_method` | string | Required. Payment method ID from AIKloop.js, for example pm_123. | | Header | Description | | --- | --- | | `Idempotency-Key` | Required. A unique value, such as a UUID v4. Retry a failed request with the same key. Keys are kept for 24 hours. | ## GET /payments List payments, newest first. | Field | Type | Description | | --- | --- | --- | | `limit` | integer | 1 to 100. Default 10. | | `starting_after` | string | ID of the last object from the previous page. |
Walk through what the agent did
In a live demo, you'll go through the runs behind these results, answer by answer.
- The agent at work. Each page it opened, what it asked about each one, and the moment it gave up on the docs and went to the raw spec.
- The code it wrote, run again in front of you. The request it sends to the mock API, what the API answers, and whether it works.
- What each answer costs. Tokens, pages, time and dollars, with Haiku 5.5 and Opus 5.5 side by side.
- Why the After docs win. The pages that make the difference, and how the docs tests keep them right when the API changes.