What does a usable
API result actually cost?

Divide the cost of all recorded attempts, including failures and retries, by the number of outputs that meet your declared rules. Keep each provider and currency separate. When no output passes, the cost per accepted output is undefined, not zero.

A low per-call price can hide repeated unusable responses. Agent Delivery Lab checks a batch of captured results against the same contract and shows the entered cost beside acceptance. It is a local calculation over your records, not a benchmark of live providers.

Try the fictional example →

A cheaper call can cost more per usable output

The example below is fictional. Both providers attempt two invoice tasks. One of Example A's responses has the wrong currency; its retry times out. Example B returns two responses that meet the rules.

This arithmetic does not establish which provider is better. Real comparisons need the same tasks, input difficulty, contract, time window and cost scope. Even equal task labels do not prove equal inputs. A returned result can meet every rule and still be factually wrong.

Prepare the attempt records

Use a JSON array with one object per attempt. Every retry gets its own unique id; attempts for the same task share task. Record all relevant attempts, not only the ones that succeeded. For timeouts or request errors, provide error instead of response.

[
  {
    "id": "run-001",
    "provider": "My API",
    "task": "invoice-01",
    "cost": "0.01",
    "currency": "USD",
    "response": {"invoice": {"id": "INV-001", "currency": "USD", "total": 125.5}}
  },
  {
    "id": "run-002",
    "provider": "My API",
    "task": "invoice-02",
    "cost": "0.01",
    "currency": "USD",
    "error": "Request timed out"
  }
]

Download example attempts ↓ · Download example contract ↓

Define acceptance before comparing providers

The guided rule editor can add required paths, types, exact values and number bounds. For invoice extraction, require an ID string, the expected currency and a total within the task's known bounds. A rule like “total is nonnegative” only checks plausibility; it cannot show the extracted total matches the source invoice.

All attempts in a batch use the same contract. Split batches when different tasks need different contracts. Empty contracts and unsupported conditions are rejected. You can export the contract for future runs.

How the cost calculation works

• Exact entered totals: costs are nonnegative decimal strings with up to 12 whole digits and 8 decimal places. Internal arithmetic uses scaled integers rather than floating-point sums.
• Failures count: all entered costs stay in the total, even if response rules fail or a request error is recorded. Enter actual charged cost; an uncharged failure can use "0".
• Separate units: USD, EUR and USDC are separate groups. A currency label does not identify a network or token contract. The checker does not convert currencies or verify token value.
• Explicit rounding: cost per accepted output is rounded upward to 8 decimal places. Reports retain the exact entered total and acceptance count so you can reproduce the ratio.
• Distinct tasks: accepted outputs and accepted tasks are both shown. Three accepted retries of one task count as three outputs and one accepted task.
• No missing-cost inference: include fees, gas or other costs in the entered figure if they belong to your comparison. Missing records and unentered costs cannot be recovered by this tool.

Repeat the check in your own CI

Download runner.mjs and engine.mjs to the same folder. Keep a sanitized batch and acceptance contract in your own project. Use Node.js 22 or later.

node runner.mjs batch attempts.json contract.json > report.json

Exit 0 means every recorded response met the contract. Exit 1 means one or more attempts failed, even when other attempts passed. Exit 2 means the input or contract was invalid. Use the same runner for individual responses and payment snapshots; see the full guide.

Limits and privacy

The beta accepts 1–100 attempts per batch, up to 128 KiB of decoded JSON and 50 acceptance rules. Inputs, file imports and calculations stay in this tab. Switching tools retains inputs in memory; reloading clears them. Nothing is sent unless you explicitly submit anonymous feedback, which does not include inputs or reports.

Exported reports may contain your attempt IDs, task labels, provider labels, cost figures and error messages. Review them before sharing. Do not paste secrets into error messages. This is a deterministic diagnostic tool, not a security audit, provider reputation score, novel consensus algorithm or guarantee of financial value.

中文说明：关注合格结果成本

Check a batch →