How well it works
Every number below comes from one run of pnpm eval: 30 questions the docs answer and 12 they do not, each sent through the same route a visitor uses.
run 2026-10-22 09:42 UTC · commit 93cbf2a-dirty · answers by llama-3.3-70b-instruct-fp8-fast (workers-ai) · threshold 0.55 · prompt 2026-10-22.1
Faithfulness and correctness are scored by llama-3.3-70b-instruct-fp8-fast (workers-ai), a model grading a model. Treat them as an estimate, not a verdict. The same model wrote the answers, so it is grading its own work, which tends to flatter it.
Retrieval
30 gold questions · the right document in the passages the model saw
- Hit at 1
- 0.833
- Hit at 3
- 0.967
- Hit at 8
- 1.000
- Mean reciprocal rank
- 0.903
Declining
12 questions the docs cannot answer · gold questions wrongly declined
- Out of scope, declined
- 100%
- Stopped by the gate alone
- 67%
- Answerable, wrongly declined
- 0%
Answer quality
graded 1 to 5 by workers-ai/@cf/meta/llama-3.3-70b-instruct-fp8-fast
- Faithfulness
- 4.97 / 5claims backed by the passages
- Correctness
- 4.73 / 5against the authors' reference
- Citations sound
- 100%every [n] maps to a passage
Speed and cost
measured from the build workspace, one question at a time
- Answer time, median
- 2,696 msp95 4,610 ms
- First token, median
- 1,831 msp95 3,608 ms
- Tokens in the run
- 105,323103,550 in · 1,773 out
- Billed for the run
- $0.0000$0.0112 on claude-haiku-5-5
Question by question
41 passed · 1 failed of 42
A gold question passes when the right document is in the top 3, the assistant answered, and faithfulness is 4 or more. An out-of-scope question passes when it is declined. Open a question to read the answer and the judge’s note.
| Fail | q11 | How long does business verification take before I can use live keys?Answer: Business verification typically takes 1 to 2 business days to complete, but it can take up to 5 business days for businesses that require enhanced due diligence [6]. Retrieved documents, best first: going-live, api-keys, kyb-verification, sandbox. Expected: kyb-verification. Judge: answer accurately reflects passage information | #4 | 0.91 | Answered | 5 | 5 | 2,404 |
| Pass | q01 | How do I verify the signature on a Ledgerline webhook?Answer: To verify the signature on a Ledgerline webhook, you should read the raw request body, split the `Ledgerline-Signature` header, extract the timestamp and `v1` value, build the signed string, compute the HMAC-SHA256 of that string with your signing secret, and compare your value to the `v1` value using a constant-time comparison [1]. You can also use the official SDKs which implement all of the steps [2]. The docs do not cover any additional steps beyond these. Retrieved documents, best first: webhook-signatures, sdk-python, webhooks-overview, webhook-delivery-and-retries, sdk-node. Expected: webhook-signatures. Judge: answer accurately reflects the passages | #1 | 0.93 | Answered | 5 | 5 | 6,244 |
| Pass | q02 | What is the API rate limit?Answer: The default API rate limit is 600 requests per minute per API key [2]. Retrieved documents, best first: rate-limits, limits-and-quotas, sdk-node, changelog. Expected: rate-limits. Judge: answer is mostly correct but does not mention the organization-wide cap | #1 | 0.92 | Answered | 5 | 4 | 3,414 |
| Pass | q03 | What is the daily cut-off time for sending a payout so it goes out the same day?Answer: The daily cut-off time for sending a payout so it goes out the same day is 15:00 UTC for `ach`, `ach_same_day`, and `sepa` rails [1], but 18:00 UTC for domestic `wire` payouts [1]. Instant rails have no cut-off time [1]. The docs do not cover every possible rail or payout type, but they do provide information on the cut-off times for several specific rails. Retrieved documents, best first: payment-rails, payouts, limits-and-quotas, plans-and-pricing. Expected: payment-rails or payouts. Judge: answer accurately reflects passage information | #1 | 0.92 | Answered | 5 | 5 | 3,318 |
| Pass | q04 | How long does Ledgerline remember an idempotency key?Answer: Ledgerline remembers an idempotency key for 24 hours from the first use [1]. Retrieved documents, best first: idempotency, api-keys, retries-and-backoff, security-overview. Expected: idempotency. Judge: answer directly supported by passage | #1 | 0.92 | Answered | 5 | 5 | 2,830 |
| Pass | q05 | How many times will Ledgerline retry a webhook that my server fails to acknowledge?Answer: Ledgerline makes up to **8 attempts** in total: the first delivery and seven retries [1]. Retrieved documents, best first: webhook-delivery-and-retries, retries-and-backoff, webhook-signatures, webhooks-overview. Expected: webhook-delivery-and-retries. Judge: answer accurately reflects the passages | #1 | 0.92 | Answered | 5 | 5 | 2,237 |
| Pass | q06 | I retried a payout with the same Idempotency-Key but changed the amount. What happens?Answer: You cannot fix a bad request and resend it under the same key, because a `4xx` is replayed for 24 hours [1]. The docs do not cover what happens when you change the amount, but they do say that you should change the key whenever you change the request [1]. Retrieved documents, best first: idempotency, payouts, retries-and-backoff, returns-and-failed-payouts, sdk-node. Expected: idempotency. Judge: answer accurately reflects passage content | #1 | 0.88 | Answered | 5 | 5 | 2,822 |
| Pass | q07 | What is the difference between the posted, pending and available balance of an account?Answer: The `posted` balance answers what has definitely happened, the `pending` balance answers what is on its way, and the `available` balance answers what can be spent right now [3]. The `available` balance is the posted balance minus pending outflows [1]. The docs do not cover any other differences between these balances beyond their definitions and calculations. Retrieved documents, best first: account-balances, transactions-and-entries, core-concepts, ledgers-and-accounts, troubleshooting-common-errors, quickstart. Expected: account-balances. Judge: Answer accurately reflects the passages | #1 | 0.92 | Answered | 5 | 5 | 3,110 |
| Pass | q08 | My transaction request fails with entries_unbalanced. How do I fix it?Answer: To fix the `entries_unbalanced` error, sum the debits and credits per currency in your payload to ensure they match exactly [5]. Also, check that amounts are integers in minor units [5]. For a multi-currency movement, balance each currency separately [5]. The error message will tell you the totals [6]. The docs do not cover other potential causes of the error beyond these steps. Retrieved documents, best first: transactions-and-entries, quickstart, troubleshooting-common-errors, error-format-and-codes, invoices-and-billing. Expected: troubleshooting-common-errors or transactions-and-entries. Judge: Answer accurately reflects the passages. | #1 | 0.91 | Answered | 5 | 5 | 3,277 |
| Pass | q09 | Can I edit or delete a transaction after it has posted?Answer: No, posted entries are immutable, meaning nothing about them can change, not the amounts, the accounts or the description, and there is no update and no delete [1]. To fix a mistake, you must reverse the transaction [1]. Retrieved documents, best first: transactions-and-entries, quickstart, ledgers-and-accounts, core-concepts, counterparties-and-screening, troubleshooting-common-errors. Expected: transactions-and-entries. Judge: answer accurately reflects passage information | #1 | 0.89 | Answered | 5 | 5 | 2,561 |
| Pass | q10 | How far in the past can I set effective_at on a transaction?Answer: You can set `effective_at` up to **90 days in the past** [1][3]. Retrieved documents, best first: transactions-and-entries, quickstart, limits-and-quotas, invoices-and-billing, statements, reconciliation. Expected: transactions-and-entries. Judge: Answer accurately reflects the passages | #1 | 0.92 | Answered | 5 | 5 | 2,129 |
| Pass | q12 | Which people do I have to list as owners during KYB?Answer: You have to list every individual who owns **25% or more** of the business as owners during KYB [1]. Retrieved documents, best first: kyb-verification, glossary, counterparties-and-screening, going-live, rate-limits. Expected: kyb-verification. Judge: answer is mostly correct but incomplete | #1 | 0.79 | Answered | 5 | 3 | 2,159 |
| Pass | q13 | How can I rotate an API key without causing downtime?Answer: You can roll a key on a schedule, and immediately if you suspect a leak, which creates a replacement and retires the old key on a timer, so you can deploy without downtime [1]. By default the old key keeps working for 24 hours after a roll [1]. You can set `expire_old_after_hours` anywhere from 0 to 72 [1]. The docs do not cover how to handle the roll in your application code. Retrieved documents, best first: api-keys, restricted-keys-and-scopes, error-format-and-codes, idempotency, retries-and-backoff. Expected: api-keys. Judge: Minor omission of the POST endpoint in the answer | #1 | 0.81 | Answered | 5 | 4 | 3,886 |
| Pass | q14 | What permissions can I give a restricted API key?Answer: You can give a restricted API key permissions such as `ledgers:read`, `ledgers:write`, `transfers:read`, `transfers:write`, `payouts:read`, `payouts:write`, `webhooks:manage`, and `reports:read` [4]. These scopes are additive, meaning a key can do anything covered by at least one of its scopes and nothing else [4]. The docs do not cover custom permissions beyond these predefined scopes. Retrieved documents, best first: restricted-keys-and-scopes, api-keys, security-overview, error-format-and-codes. Expected: restricted-keys-and-scopes. Judge: Answer accurately reflects the predefined scopes in the passages. | #1 | 0.92 | Answered | 5 | 5 | 3,939 |
| Pass | q15 | How long can I use an FX quote before it expires?Answer: You can use an FX quote for 30 seconds before it expires [1]. After `expires_at`, booking against it fails with `quote_expired` [1][2]. Retrieved documents, best first: fx-quotes-and-conversions, error-format-and-codes, troubleshooting-common-errors, limits-and-quotas, multi-currency-ledgers, api-keys. Expected: fx-quotes-and-conversions. Judge: Answer accurately reflects the passages | #1 | 0.91 | Answered | 5 | 5 | 2,219 |
| Pass | q16 | Why does my currency conversion fail on Saturdays?Answer: Currency conversions fail on Saturdays because foreign exchange markets are closed at the weekend, from Friday 22:00 UTC until Sunday 22:00 UTC, and quote requests fail with HTTP `422` and the code `fx_market_closed` [1]. Retrieved documents, best first: fx-quotes-and-conversions, multi-currency-ledgers, troubleshooting-common-errors, invoices-and-billing. Expected: fx-quotes-and-conversions or troubleshooting-common-errors. Judge: answer accurately reflects passage content | #1 | 0.91 | Answered | 5 | 5 | 3,153 |
| Pass | q17 | How much does the Growth plan cost and what does it include?Answer: The Growth plan costs $499 per month [1] and includes 500,000 ledger entries per month, with each entry above the included amount costing $0.0015 [1]. It also includes other features such as 99.9% uptime SLA, 1 business day ticket response, and IP allowlists [1]. The docs do not cover all possible details of the plan, such as any potential discounts or promotions. Retrieved documents, best first: plans-and-pricing, limits-and-quotas, fx-quotes-and-conversions. Expected: plans-and-pricing. Judge: minor omission of FX spread detail | #1 | 0.73 | Answered | 5 | 4 | 3,835 |
| Pass | q18 | What is the largest ACH payout I can make on the Starter plan?Answer: The largest ACH payout you can make on the Starter plan is $25,000 [2][3]. Retrieved documents, best first: plans-and-pricing, payouts, limits-and-quotas, payment-rails. Expected: limits-and-quotas. Judge: answer is correct but does not mention the daily limit | #3 | 0.74 | Answered | 5 | 4 | 2,737 |
| Pass | q19 | How can I simulate a returned payout in the sandbox?Answer: You can simulate a returned payout in the sandbox by creating a counterparty bank account with routing number `110000000` and account number `000111111113` [4]. Retrieved documents, best first: sandbox, payouts, returns-and-failed-payouts, going-live. Expected: sandbox. Judge: Answer directly supported by passage 4. | #1 | 0.87 | Answered | 5 | 5 | 2,530 |
| Pass | q20 | Which data regions are available and can I move my data to another region later?Answer: The available data regions are `us` (Virginia, United States), `eu` (Frankfurt, Germany), and `ap` (Singapore) [2]. You can choose a region when creating your organization, but moving an organization to another region later requires the Scale plan and a support ticket, and takes about 2 weeks [1]. The docs do not cover any potential limitations or requirements for moving data between specific regions. Retrieved documents, best first: data-residency-and-retention, security-overview, changelog, account-balances, troubleshooting-common-errors. Expected: data-residency-and-retention. Judge: answer accurately reflects the passages | #1 | 0.85 | Answered | 5 | 5 | 3,231 |
| Pass | q21 | How long is ledger data stored after I close my account?Answer: Ledger data is kept for seven years after the account is closed [2]. The docs do not cover what happens to the data after the seven-year period in detail, but it is deleted from live systems within 30 days and from backups within 90 [2]. Retrieved documents, best first: ledgers-and-accounts, data-residency-and-retention, core-concepts, account-balances, quickstart. Expected: data-residency-and-retention. Judge: Answer accurately reflects the passages | #2 | 0.91 | Answered | 5 | 5 | 2,696 |
| Pass | q22 | When is my monthly statement available and in what file formats?Answer: Your monthly statement is available by **06:00 UTC** on the first business day of each month [1]. It is available in `PDF`, `CSV`, `CAMT.053`, `MT940`, and `BAI2` file formats [1]. Retrieved documents, best first: statements, invoices-and-billing, glossary, reconciliation, plans-and-pricing. Expected: statements. Judge: answer accurately reflects the passages | #1 | 0.85 | Answered | 5 | 5 | 2,328 |
| Pass | q23 | How does Ledgerline match my bank statement lines to ledger entries?Answer: Ledgerline matches bank statement lines to ledger entries by applying three rules in order: Exact, Fuzzy, and Manual [1]. The Exact rule matches when the reference and amount are both identical, the Fuzzy rule matches when the amount is identical, the booking dates are within 3 days, and the counterparty names have a similarity score of at least 0.85, and the Manual rule matches when you link the lines yourself in the dashboard or API [1]. The docs do not cover how to implement these rules in the API. Retrieved documents, best first: reconciliation, statements, ledgers-and-accounts, account-balances, glossary, transactions-and-entries, core-concepts. Expected: reconciliation. Judge: Answer accurately describes matching rules | #1 | 0.92 | Answered | 5 | 5 | 4,116 |
| Pass | q24 | What happens to my account if my invoice payment fails?Answer: Your account works normally for 7 days after the first payment attempt fails, which is considered a grace period [1]. After 7 days, write operations are suspended, and you can still read data, download statements, and receive webhooks, but creating payouts, transactions, and keys returns `billing_suspended` with HTTP `403` [1]. The docs do not cover what happens after the suspension is lifted. Retrieved documents, best first: invoices-and-billing, returns-and-failed-payouts, ledgers-and-accounts, payouts. Expected: invoices-and-billing. Judge: Minor omission about writes resuming after payment | #1 | 0.92 | Answered | 5 | 4 | 4,610 |
| Pass | q25 | How fast does support respond to tickets on each plan?Answer: Support responds to tickets within2 business days on the `Starter` plan, within 1 business day on the `Growth` plan, and within 4 business hours on the `Scale` plan [1]. The docs do not cover any exceptions to these response times. Retrieved documents, best first: plans-and-pricing, rate-limits, status-and-incidents, limits-and-quotas, invoices-and-billing. Expected: plans-and-pricing. Judge: Minor omission of 24/7 SEV1 incident coverage for Scale plan | #1 | 0.80 | Answered | 4 | 4 | 4,603 |
| Pass | q26 | What service credit does a Growth customer get if monthly uptime falls below 99%?Answer: A Growth customer gets a25% service credit if monthly uptime falls below 99% [2]. The docs do not cover how to claim this credit beyond the general procedure in [1]. Retrieved documents, best first: status-and-incidents, plans-and-pricing, invoices-and-billing, ledgers-and-accounts, limits-and-quotas. Expected: status-and-incidents. Judge: answer accurately reflects the passages | #1 | 0.77 | Answered | 5 | 5 | 2,933 |
| Pass | q27 | Which Node.js versions does the Ledgerline SDK support?Answer: The Ledgerline SDK supports **Node.js 18 and newer** [2]. Retrieved documents, best first: api-versioning, sdk-node, sdk-python, changelog. Expected: sdk-node. Judge: answer directly quotes the passage | #2 | 0.92 | Answered | 5 | 5 | 2,063 |
| Pass | q28 | How do I loop through every payout with the Python SDK without handling pages myself?Answer: You can loop through every payout with the Python SDK using the `auto_paging_iter()` method [1]. This method walks every page, requesting 100 items at a time. The docs do not cover how to handle potential errors that may occur during this process [1]. Retrieved documents, best first: sdk-python, sdk-node, payouts, returns-and-failed-payouts. Expected: sdk-python. Judge: answer accurately reflects the passage | #1 | 0.92 | Answered | 5 | 5 | 2,893 |
| Pass | q29 | Why do I get insufficient_funds when the account balance looks high enough?Answer: You get `insufficient_funds` when the account's **available** balance is lower than the outflow [4]. The **available** balance is the posted balance minus pending outflows [2][3], so a balance that looks large can still be fully reserved by earlier payouts. The docs do not cover how to handle this situation beyond checking the available balance and waiting for pending transactions to post or void [4]. Retrieved documents, best first: ledgers-and-accounts, account-balances, troubleshooting-common-errors, core-concepts, transactions-and-entries, multi-currency-ledgers. Expected: troubleshooting-common-errors or account-balances. Judge: Answer accurately reflects the passages | #2 | 0.84 | Answered | 5 | 5 | 3,523 |
| Pass | q30 | Why is my first payout to a new supplier sitting in pending_review?Answer: The payout is held in `pending_review` because it is over $25,000 and the counterparty was created less than 7 days ago [1]. Retrieved documents, best first: counterparties-and-screening, payouts, going-live, returns-and-failed-payouts, payment-rails. Expected: counterparties-and-screening or payouts. Judge: answer accurately reflects passage content | #1 | 0.86 | Answered | 5 | 5 | 2,795 |
| Pass | o01 | Do you have a Go SDK?Answer: I don't have that in the Ledgerline docs. Retrieved documents, best first: sdk-node, sdk-python, going-live, webhook-signatures. | n/a | 0.31 | Declined (gate) | — | — | 787 |
| Pass | o02 | How do I handle credit card chargebacks and disputes?Answer: I don't have that in the Ledgerline docs. Retrieved documents, best first: invoices-and-billing, returns-and-failed-payouts, error-format-and-codes, ledgers-and-accounts, transactions-and-entries, troubleshooting-common-errors, status-and-incidents. | n/a | 0.51 | Declined (gate) | — | — | 672 |
| Pass | o03 | Can I send payouts in Bitcoin or other cryptocurrency?Answer: I don't have that in the Ledgerline docs. Retrieved documents, best first: payouts, plans-and-pricing, multi-currency-ledgers, payment-rails, returns-and-failed-payouts, webhooks-overview. Judge: Answer correctly states that the docs do not cover the question | n/a | 0.84 | Declined (model) | 5 | — | 2,607 |
| Pass | o04 | Is there a QuickBooks or Xero integration?Answer: I don't have that in the Ledgerline docs. Retrieved documents, best first: going-live, sandbox, troubleshooting-common-errors, api-versioning, sdk-python, changelog, kyb-verification. | n/a | 0.18 | Declined (gate) | — | — | 877 |
| Pass | o05 | Does Ledgerline issue physical or virtual debit cards?Answer: I don't have that in the Ledgerline docs. Retrieved documents, best first: invoices-and-billing, ledgers-and-accounts, quickstart, kyb-verification, core-concepts, api-keys, sdk-python. Judge: Answer correctly states that the information is not in the docs. | n/a | 0.80 | Declined (model) | 5 | — | 5,091 |
| Pass | o06 | Is there a Terraform provider for managing webhook endpoints?Answer: I don't have that in the Ledgerline docs. Retrieved documents, best first: webhook-delivery-and-retries, webhooks-overview, webhook-signatures, api-versioning, sdk-python. | n/a | 0.52 | Declined (gate) | — | — | 1,998 |
| Pass | o07 | How do I pay suppliers in Brazil using PIX?Answer: I don't have that in the Ledgerline docs. Retrieved documents, best first: multi-currency-ledgers, core-concepts, invoices-and-billing, counterparties-and-screening, sdk-python, limits-and-quotas, payment-rails, quickstart. | n/a | 0.23 | Declined (gate) | — | — | 525 |
| Pass | o08 | Does Ledgerline comply with HIPAA for healthcare payments?Answer: I don't have that in the Ledgerline docs. Retrieved documents, best first: invoices-and-billing, webhook-signatures, glossary, sdk-node, webhooks-overview, sdk-python, api-keys. Judge: Answer correctly states that the docs do not cover the question | n/a | 0.81 | Declined (model) | 5 | — | 1,803 |
| Pass | o09 | What is the best recipe for sourdough bread?Answer: I don't have that in the Ledgerline docs. Retrieved documents, best first: quickstart, sdk-node, statements, going-live, reconciliation. | n/a | 0.04 | Declined (gate) | — | — | 889 |
| Pass | o10 | Who won the football match last night?Answer: I don't have that in the Ledgerline docs. Retrieved documents, best first: reconciliation, fx-quotes-and-conversions, returns-and-failed-payouts, payment-rails, error-format-and-codes. | n/a | 0.08 | Declined (gate) | — | — | 688 |
| Pass | o11 | Can I embed the dashboard in my own app with a white-label theme?Answer: I don't have that in the Ledgerline docs. Retrieved documents, best first: security-overview, going-live, restricted-keys-and-scopes, changelog, api-keys. | n/a | 0.13 | Declined (gate) | — | — | 669 |
| Pass | o12 | How do I set up GraphQL subscriptions for ledger changes?Answer: I don't have that in the Ledgerline docs. Retrieved documents, best first: quickstart, ledgers-and-accounts, changelog, api-versioning, core-concepts, data-residency-and-retention. Judge: Answer correctly states that the docs do not cover the question | n/a | 0.73 | Declined (model) | 5 | — | 1,812 |
History
1 run
one run so far
one run so far
one run so far
| 2026-10-22 09:42 | 93cbf2a-dirty | llama-3.3-70b-instruct-fp8-fast (workers-ai) | 0.967 | 0.903 | 4.97 | 100% | 2,696 |
What this does not show
- The question sets are small (30 and 12) and were written by the people who wrote the docs. One question moves hit@3 by 3 points.
- The confidence threshold was fitted on these same questions, so the decline numbers are in-sample. They are an operating point, not a forecast.
- A model judges faithfulness and correctness. It can be generous or wrong; the answers are in the table so you can check.
- Latency includes the database and the model, measured from the build workspace, which is far from both. The deployed region will differ.