Live in public beta — base URL https://api.agentic.forwardlane.com · metered, no charges during beta.
SWE-bench-style evaluation instances minted from private, never-public financial repositories — leak-free by construction, while public SWE-bench sits in every frontier model’s training set. Browse task inventory and model evaluation results through the API.

​
Authentication

All product endpoints require a tenant member API key sent as a bearer token. Both a missing and an invalid key return 401 with a docs_url field pointing at the authentication guide.

​
Quickstart

The second call reads your usage meters — every product call is metered against your organization.

​
Endpoints

​
GET /api/benchmark/tasks

List benchmark task instances. Returns: items (task instances) and total.

​
GET /api/benchmark/tasks/{instance_id}

Fetch a single task by instance id. Returns: The full instance: problem statement, base commit, verifying tests, difficulty.

​
GET /api/benchmark/eval-results

Model evaluation result summaries. Returns: Per-run summaries: model, evaluated, solved, solve_rate, timestamp.

​
Data provenance

intelligence/benchmark mines real bug-fix commits from private repo history (git) + test-to-function mapping (DuckDB tests 7,587 rows, test_edges) to mint SWE-bench-format instances: problem statement, base_commit, patch, verifying tests, difficulty. 3 instances materialized; generator runs on demand. What the data is: SWE-bench-format evaluation instances from 151 never-public financial repos — uncontaminated by construction (public SWE-bench is in frontier training sets).

​
Beta notes

  • This API is in public beta. Credits run in shadow mode: usage is metered and visible at GET /v1/billing/usage, but no charges are made during the beta.
  • Authentication is tenant member API keys only; per-call x402 payments are coming soon and not yet accepted.
  • Machine discovery: GET /v1/catalog (public, no auth) lists all products with live/coming-soon status and docs links.
  • Response shapes may still change during the beta; breaking changes will be announced in the changelog before they land.