Docs tests
These tests read the docs the way a developer would, code and text alike. Every code sample runs against a mock of the API. Every claim in the text, such as "this endpoint exists" or "retrying is safe", is checked against the API's spec. When the next release arrives, they show what it breaks, before your customers do.
| Page | Test | Current release · v3.0.0 openapi-3.0.0.json | Incoming release · v3.1.0 openapi-3.1.0.json |
|---|---|---|---|
| Getting started✓ 4 of 4 passing | Endpoints exist in the spec | ✓ Pass | ✓ Pass |
| Authentication matches the spec | ✓ Pass | ✓ Pass | |
| Parameters and headers match the spec | ✓ Pass | ✓ Pass | |
| Sample runs against the mock API | ✓ Pass | ✓ Pass | |
| Accept a payment✗ 1 of 5 failing | Endpoints exist in the spec | ✓ Pass | ✓ Pass |
| Authentication matches the spec | ✓ Pass | ✓ Pass | |
| Parameters and headers match the spec | ✓ Pass | ✓ Pass | |
| Sample runs against the mock API | ✓ Pass | ✓ Pass | |
| Retrying after a timeout is safe | ✓ Pass | ✗ Fail
Says keys are kept for 24 hours, but the spec keeps them for 1 hour. A retry 23 hours and 59 minutes later created a second payment: the customer is charged twice.after/accept-a-payment.md, line 27: If a request times out, retry it **with the same `Idempotency-Key`**. AIKloop returns the original result instead of creating a second payment. Keys are kept for 24 hours.
|
|
| Refund a payment✓ 4 of 4 passing | Endpoints exist in the spec | ✓ Pass | ✓ Pass |
| Authentication matches the spec | ✓ Pass | ✓ Pass | |
| Parameters and headers match the spec | ✓ Pass | ✓ Pass | |
| Sample runs against the mock API | ✓ Pass | ✓ Pass | |
| List payments✗ 2 of 4 failing | Endpoints exist in the spec | ✓ Pass | ✓ Pass |
| Authentication matches the spec | ✓ Pass | ✓ Pass | |
| Parameters and headers match the spec | ✓ Pass | ✗ Fail
limit=100 is above the maximum of 50.after/list-payments.md, line 11: curl "https://api.aikloop.example/v3/payments?limit=100&starting_after=pay_123" \
|
|
| Sample runs against the mock API | ✓ Pass | ✗ Fail
400: limit=100 is above the maximum of 50.
|
|
| Handle webhooks✗ 1 of 1 failing | Signature header and rules match the spec | ✓ Pass | ✗ Fail
Says to reject events after 5 minutes; the spec says 3, so replayed events are accepted for longer than they should be.after/webhooks.md, line 13: 4. Reject the event if `t` is more than 5 minutes old, to block replays.
|
| Changelog✗ 1 of 1 failing | Latest release matches the spec | ✓ Pass | ✗ Fail
The latest entry is v3.0; the spec is v3.1.0.after/changelog.md, line 8: ## v3.0 (2026-03-02)
|
How the tests work
The tests read each page's Markdown file. Every curl sample runs as written against a mock of the API, so a sample that's wrong for the spec, fails here before a user runs it. The text around the samples is checked as well, for example: how long idempotency keys are kept, the webhook time window or the latest version.
| Test | What it checks |
|---|---|
| Endpoints exist in the spec | Every curl sample calls an endpoint that's in the spec. |
| Authentication matches the spec | Samples send a Bearer token in the Authorization header. |
| Parameters and headers match the spec | Every field and query parameter is in the spec, required ones are there, values have the right type and are within limits, and required headers such as Idempotency-Key are sent. |
| Sample runs against the mock API | The sample, run as written, gets a successful response from the mock. |
| Retrying after a timeout is safe | The page says to retry with the same Idempotency-Key, and sending its sample twice creates one payment, not two. If the page says how long keys are kept, a retry just before that time ends must still create only one payment. |
| Signature header and rules match the spec | The webhooks page names the header, the algorithm and the time window that the spec gives. |
| Latest release matches the spec | The changelog's latest entry is the version in the spec. |
See it catch the next release
The table above shows the docs as they are today, and what the unreleased v3.1.0 breaks. The interesting part happens before the release ships: the unreleased spec is tested privately against the same docs, Claude drafts the fixes as a pull request, and a human reviews them.
In a live demo, you'll see:
- What the next release breaks. Which pages, which lines and why, before any customer reads them.
- Claude fixing it. In VS Code with one command, or on GitHub as a pull request that opens on its own.
- The release going out. A human merges the fix, and the site updates with the release.
The API reference pages aren't tested here because they're generated from the spec. The Before docs aren't tested either: they're HTML pages that were never kept as code, so there's nothing to test them with. To see the tests run, ask for a demo.