Skip to content
Ledgerline Help

How well it works

Every number below comes from one run of pnpm eval: 30 questions the docs answer and 12 they do not, each sent through the same route a visitor uses.

run 2026-10-22 09:42 UTC · commit 93cbf2a-dirty · answers by llama-3.3-70b-instruct-fp8-fast (workers-ai) · threshold 0.55 · prompt 2026-10-22.1

Faithfulness and correctness are scored by llama-3.3-70b-instruct-fp8-fast (workers-ai), a model grading a model. Treat them as an estimate, not a verdict. The same model wrote the answers, so it is grading its own work, which tends to flatter it.

Retrieval

30 gold questions · the right document in the passages the model saw

Hit at 1
0.833
Hit at 3
0.967
Hit at 8
1.000
Mean reciprocal rank
0.903

Declining

12 questions the docs cannot answer · gold questions wrongly declined

Out of scope, declined
100%
Stopped by the gate alone
67%
Answerable, wrongly declined
0%

Answer quality

graded 1 to 5 by workers-ai/@cf/meta/llama-3.3-70b-instruct-fp8-fast

Faithfulness
4.97 / 5claims backed by the passages
Correctness
4.73 / 5against the authors' reference
Citations sound
100%every [n] maps to a passage

Speed and cost

measured from the build workspace, one question at a time

Answer time, median
2,696 msp95 4,610 ms
First token, median
1,831 msp95 3,608 ms
Tokens in the run
105,323103,550 in · 1,773 out
Billed for the run
$0.0000$0.0112 on claude-haiku-5-5

Question by question

41 passed · 1 failed of 42

A gold question passes when the right document is in the top 3, the assistant answered, and faithfulness is 4 or more. An out-of-scope question passes when it is declined. Open a question to read the answer and the judge’s note.

Every evaluation question and its result. Select a column heading to sort.
Failq11
How long does business verification take before I can use live keys?

Answer: Business verification typically takes 1 to 2 business days to complete, but it can take up to 5 business days for businesses that require enhanced due diligence [6]. Retrieved documents, best first: going-live, api-keys, kyb-verification, sandbox. Expected: kyb-verification. Judge: answer accurately reflects passage information

#40.91Answered552,404
Passq01
How do I verify the signature on a Ledgerline webhook?

Answer: To verify the signature on a Ledgerline webhook, you should read the raw request body, split the `Ledgerline-Signature` header, extract the timestamp and `v1` value, build the signed string, compute the HMAC-SHA256 of that string with your signing secret, and compare your value to the `v1` value using a constant-time comparison [1]. You can also use the official SDKs which implement all of the steps [2]. The docs do not cover any additional steps beyond these. Retrieved documents, best first: webhook-signatures, sdk-python, webhooks-overview, webhook-delivery-and-retries, sdk-node. Expected: webhook-signatures. Judge: answer accurately reflects the passages

#10.93Answered556,244
Passq02
What is the API rate limit?

Answer: The default API rate limit is 600 requests per minute per API key [2]. Retrieved documents, best first: rate-limits, limits-and-quotas, sdk-node, changelog. Expected: rate-limits. Judge: answer is mostly correct but does not mention the organization-wide cap

#10.92Answered543,414
Passq03
What is the daily cut-off time for sending a payout so it goes out the same day?

Answer: The daily cut-off time for sending a payout so it goes out the same day is 15:00 UTC for `ach`, `ach_same_day`, and `sepa` rails [1], but 18:00 UTC for domestic `wire` payouts [1]. Instant rails have no cut-off time [1]. The docs do not cover every possible rail or payout type, but they do provide information on the cut-off times for several specific rails. Retrieved documents, best first: payment-rails, payouts, limits-and-quotas, plans-and-pricing. Expected: payment-rails or payouts. Judge: answer accurately reflects passage information

#10.92Answered553,318
Passq04
How long does Ledgerline remember an idempotency key?

Answer: Ledgerline remembers an idempotency key for 24 hours from the first use [1]. Retrieved documents, best first: idempotency, api-keys, retries-and-backoff, security-overview. Expected: idempotency. Judge: answer directly supported by passage

#10.92Answered552,830
Passq05
How many times will Ledgerline retry a webhook that my server fails to acknowledge?

Answer: Ledgerline makes up to **8 attempts** in total: the first delivery and seven retries [1]. Retrieved documents, best first: webhook-delivery-and-retries, retries-and-backoff, webhook-signatures, webhooks-overview. Expected: webhook-delivery-and-retries. Judge: answer accurately reflects the passages

#10.92Answered552,237
Passq06
I retried a payout with the same Idempotency-Key but changed the amount. What happens?

Answer: You cannot fix a bad request and resend it under the same key, because a `4xx` is replayed for 24 hours [1]. The docs do not cover what happens when you change the amount, but they do say that you should change the key whenever you change the request [1]. Retrieved documents, best first: idempotency, payouts, retries-and-backoff, returns-and-failed-payouts, sdk-node. Expected: idempotency. Judge: answer accurately reflects passage content

#10.88Answered552,822
Passq07
What is the difference between the posted, pending and available balance of an account?

Answer: The `posted` balance answers what has definitely happened, the `pending` balance answers what is on its way, and the `available` balance answers what can be spent right now [3]. The `available` balance is the posted balance minus pending outflows [1]. The docs do not cover any other differences between these balances beyond their definitions and calculations. Retrieved documents, best first: account-balances, transactions-and-entries, core-concepts, ledgers-and-accounts, troubleshooting-common-errors, quickstart. Expected: account-balances. Judge: Answer accurately reflects the passages

#10.92Answered553,110
Passq08
My transaction request fails with entries_unbalanced. How do I fix it?

Answer: To fix the `entries_unbalanced` error, sum the debits and credits per currency in your payload to ensure they match exactly [5]. Also, check that amounts are integers in minor units [5]. For a multi-currency movement, balance each currency separately [5]. The error message will tell you the totals [6]. The docs do not cover other potential causes of the error beyond these steps. Retrieved documents, best first: transactions-and-entries, quickstart, troubleshooting-common-errors, error-format-and-codes, invoices-and-billing. Expected: troubleshooting-common-errors or transactions-and-entries. Judge: Answer accurately reflects the passages.

#10.91Answered553,277
Passq09
Can I edit or delete a transaction after it has posted?

Answer: No, posted entries are immutable, meaning nothing about them can change, not the amounts, the accounts or the description, and there is no update and no delete [1]. To fix a mistake, you must reverse the transaction [1]. Retrieved documents, best first: transactions-and-entries, quickstart, ledgers-and-accounts, core-concepts, counterparties-and-screening, troubleshooting-common-errors. Expected: transactions-and-entries. Judge: answer accurately reflects passage information

#10.89Answered552,561
Passq10
How far in the past can I set effective_at on a transaction?

Answer: You can set `effective_at` up to **90 days in the past** [1][3]. Retrieved documents, best first: transactions-and-entries, quickstart, limits-and-quotas, invoices-and-billing, statements, reconciliation. Expected: transactions-and-entries. Judge: Answer accurately reflects the passages

#10.92Answered552,129
Passq12
Which people do I have to list as owners during KYB?

Answer: You have to list every individual who owns **25% or more** of the business as owners during KYB [1]. Retrieved documents, best first: kyb-verification, glossary, counterparties-and-screening, going-live, rate-limits. Expected: kyb-verification. Judge: answer is mostly correct but incomplete

#10.79Answered532,159
Passq13
How can I rotate an API key without causing downtime?

Answer: You can roll a key on a schedule, and immediately if you suspect a leak, which creates a replacement and retires the old key on a timer, so you can deploy without downtime [1]. By default the old key keeps working for 24 hours after a roll [1]. You can set `expire_old_after_hours` anywhere from 0 to 72 [1]. The docs do not cover how to handle the roll in your application code. Retrieved documents, best first: api-keys, restricted-keys-and-scopes, error-format-and-codes, idempotency, retries-and-backoff. Expected: api-keys. Judge: Minor omission of the POST endpoint in the answer

#10.81Answered543,886
Passq14
What permissions can I give a restricted API key?

Answer: You can give a restricted API key permissions such as `ledgers:read`, `ledgers:write`, `transfers:read`, `transfers:write`, `payouts:read`, `payouts:write`, `webhooks:manage`, and `reports:read` [4]. These scopes are additive, meaning a key can do anything covered by at least one of its scopes and nothing else [4]. The docs do not cover custom permissions beyond these predefined scopes. Retrieved documents, best first: restricted-keys-and-scopes, api-keys, security-overview, error-format-and-codes. Expected: restricted-keys-and-scopes. Judge: Answer accurately reflects the predefined scopes in the passages.

#10.92Answered553,939
Passq15
How long can I use an FX quote before it expires?

Answer: You can use an FX quote for 30 seconds before it expires [1]. After `expires_at`, booking against it fails with `quote_expired` [1][2]. Retrieved documents, best first: fx-quotes-and-conversions, error-format-and-codes, troubleshooting-common-errors, limits-and-quotas, multi-currency-ledgers, api-keys. Expected: fx-quotes-and-conversions. Judge: Answer accurately reflects the passages

#10.91Answered552,219
Passq16
Why does my currency conversion fail on Saturdays?

Answer: Currency conversions fail on Saturdays because foreign exchange markets are closed at the weekend, from Friday 22:00 UTC until Sunday 22:00 UTC, and quote requests fail with HTTP `422` and the code `fx_market_closed` [1]. Retrieved documents, best first: fx-quotes-and-conversions, multi-currency-ledgers, troubleshooting-common-errors, invoices-and-billing. Expected: fx-quotes-and-conversions or troubleshooting-common-errors. Judge: answer accurately reflects passage content

#10.91Answered553,153
Passq17
How much does the Growth plan cost and what does it include?

Answer: The Growth plan costs $499 per month [1] and includes 500,000 ledger entries per month, with each entry above the included amount costing $0.0015 [1]. It also includes other features such as 99.9% uptime SLA, 1 business day ticket response, and IP allowlists [1]. The docs do not cover all possible details of the plan, such as any potential discounts or promotions. Retrieved documents, best first: plans-and-pricing, limits-and-quotas, fx-quotes-and-conversions. Expected: plans-and-pricing. Judge: minor omission of FX spread detail

#10.73Answered543,835
Passq18
What is the largest ACH payout I can make on the Starter plan?

Answer: The largest ACH payout you can make on the Starter plan is $25,000 [2][3]. Retrieved documents, best first: plans-and-pricing, payouts, limits-and-quotas, payment-rails. Expected: limits-and-quotas. Judge: answer is correct but does not mention the daily limit

#30.74Answered542,737
Passq19
How can I simulate a returned payout in the sandbox?

Answer: You can simulate a returned payout in the sandbox by creating a counterparty bank account with routing number `110000000` and account number `000111111113` [4]. Retrieved documents, best first: sandbox, payouts, returns-and-failed-payouts, going-live. Expected: sandbox. Judge: Answer directly supported by passage 4.

#10.87Answered552,530
Passq20
Which data regions are available and can I move my data to another region later?

Answer: The available data regions are `us` (Virginia, United States), `eu` (Frankfurt, Germany), and `ap` (Singapore) [2]. You can choose a region when creating your organization, but moving an organization to another region later requires the Scale plan and a support ticket, and takes about 2 weeks [1]. The docs do not cover any potential limitations or requirements for moving data between specific regions. Retrieved documents, best first: data-residency-and-retention, security-overview, changelog, account-balances, troubleshooting-common-errors. Expected: data-residency-and-retention. Judge: answer accurately reflects the passages

#10.85Answered553,231
Passq21
How long is ledger data stored after I close my account?

Answer: Ledger data is kept for seven years after the account is closed [2]. The docs do not cover what happens to the data after the seven-year period in detail, but it is deleted from live systems within 30 days and from backups within 90 [2]. Retrieved documents, best first: ledgers-and-accounts, data-residency-and-retention, core-concepts, account-balances, quickstart. Expected: data-residency-and-retention. Judge: Answer accurately reflects the passages

#20.91Answered552,696
Passq22
When is my monthly statement available and in what file formats?

Answer: Your monthly statement is available by **06:00 UTC** on the first business day of each month [1]. It is available in `PDF`, `CSV`, `CAMT.053`, `MT940`, and `BAI2` file formats [1]. Retrieved documents, best first: statements, invoices-and-billing, glossary, reconciliation, plans-and-pricing. Expected: statements. Judge: answer accurately reflects the passages

#10.85Answered552,328
Passq23
How does Ledgerline match my bank statement lines to ledger entries?

Answer: Ledgerline matches bank statement lines to ledger entries by applying three rules in order: Exact, Fuzzy, and Manual [1]. The Exact rule matches when the reference and amount are both identical, the Fuzzy rule matches when the amount is identical, the booking dates are within 3 days, and the counterparty names have a similarity score of at least 0.85, and the Manual rule matches when you link the lines yourself in the dashboard or API [1]. The docs do not cover how to implement these rules in the API. Retrieved documents, best first: reconciliation, statements, ledgers-and-accounts, account-balances, glossary, transactions-and-entries, core-concepts. Expected: reconciliation. Judge: Answer accurately describes matching rules

#10.92Answered554,116
Passq24
What happens to my account if my invoice payment fails?

Answer: Your account works normally for 7 days after the first payment attempt fails, which is considered a grace period [1]. After 7 days, write operations are suspended, and you can still read data, download statements, and receive webhooks, but creating payouts, transactions, and keys returns `billing_suspended` with HTTP `403` [1]. The docs do not cover what happens after the suspension is lifted. Retrieved documents, best first: invoices-and-billing, returns-and-failed-payouts, ledgers-and-accounts, payouts. Expected: invoices-and-billing. Judge: Minor omission about writes resuming after payment

#10.92Answered544,610
Passq25
How fast does support respond to tickets on each plan?

Answer: Support responds to tickets within2 business days on the `Starter` plan, within 1 business day on the `Growth` plan, and within 4 business hours on the `Scale` plan [1]. The docs do not cover any exceptions to these response times. Retrieved documents, best first: plans-and-pricing, rate-limits, status-and-incidents, limits-and-quotas, invoices-and-billing. Expected: plans-and-pricing. Judge: Minor omission of 24/7 SEV1 incident coverage for Scale plan

#10.80Answered444,603
Passq26
What service credit does a Growth customer get if monthly uptime falls below 99%?

Answer: A Growth customer gets a25% service credit if monthly uptime falls below 99% [2]. The docs do not cover how to claim this credit beyond the general procedure in [1]. Retrieved documents, best first: status-and-incidents, plans-and-pricing, invoices-and-billing, ledgers-and-accounts, limits-and-quotas. Expected: status-and-incidents. Judge: answer accurately reflects the passages

#10.77Answered552,933
Passq27
Which Node.js versions does the Ledgerline SDK support?

Answer: The Ledgerline SDK supports **Node.js 18 and newer** [2]. Retrieved documents, best first: api-versioning, sdk-node, sdk-python, changelog. Expected: sdk-node. Judge: answer directly quotes the passage

#20.92Answered552,063
Passq28
How do I loop through every payout with the Python SDK without handling pages myself?

Answer: You can loop through every payout with the Python SDK using the `auto_paging_iter()` method [1]. This method walks every page, requesting 100 items at a time. The docs do not cover how to handle potential errors that may occur during this process [1]. Retrieved documents, best first: sdk-python, sdk-node, payouts, returns-and-failed-payouts. Expected: sdk-python. Judge: answer accurately reflects the passage

#10.92Answered552,893
Passq29
Why do I get insufficient_funds when the account balance looks high enough?

Answer: You get `insufficient_funds` when the account's **available** balance is lower than the outflow [4]. The **available** balance is the posted balance minus pending outflows [2][3], so a balance that looks large can still be fully reserved by earlier payouts. The docs do not cover how to handle this situation beyond checking the available balance and waiting for pending transactions to post or void [4]. Retrieved documents, best first: ledgers-and-accounts, account-balances, troubleshooting-common-errors, core-concepts, transactions-and-entries, multi-currency-ledgers. Expected: troubleshooting-common-errors or account-balances. Judge: Answer accurately reflects the passages

#20.84Answered553,523
Passq30
Why is my first payout to a new supplier sitting in pending_review?

Answer: The payout is held in `pending_review` because it is over $25,000 and the counterparty was created less than 7 days ago [1]. Retrieved documents, best first: counterparties-and-screening, payouts, going-live, returns-and-failed-payouts, payment-rails. Expected: counterparties-and-screening or payouts. Judge: answer accurately reflects passage content

#10.86Answered552,795
Passo01
Do you have a Go SDK?

Answer: I don't have that in the Ledgerline docs. Retrieved documents, best first: sdk-node, sdk-python, going-live, webhook-signatures.

n/a0.31Declined (gate)——787
Passo02
How do I handle credit card chargebacks and disputes?

Answer: I don't have that in the Ledgerline docs. Retrieved documents, best first: invoices-and-billing, returns-and-failed-payouts, error-format-and-codes, ledgers-and-accounts, transactions-and-entries, troubleshooting-common-errors, status-and-incidents.

n/a0.51Declined (gate)——672
Passo03
Can I send payouts in Bitcoin or other cryptocurrency?

Answer: I don't have that in the Ledgerline docs. Retrieved documents, best first: payouts, plans-and-pricing, multi-currency-ledgers, payment-rails, returns-and-failed-payouts, webhooks-overview. Judge: Answer correctly states that the docs do not cover the question

n/a0.84Declined (model)5—2,607
Passo04
Is there a QuickBooks or Xero integration?

Answer: I don't have that in the Ledgerline docs. Retrieved documents, best first: going-live, sandbox, troubleshooting-common-errors, api-versioning, sdk-python, changelog, kyb-verification.

n/a0.18Declined (gate)——877
Passo05
Does Ledgerline issue physical or virtual debit cards?

Answer: I don't have that in the Ledgerline docs. Retrieved documents, best first: invoices-and-billing, ledgers-and-accounts, quickstart, kyb-verification, core-concepts, api-keys, sdk-python. Judge: Answer correctly states that the information is not in the docs.

n/a0.80Declined (model)5—5,091
Passo06
Is there a Terraform provider for managing webhook endpoints?

Answer: I don't have that in the Ledgerline docs. Retrieved documents, best first: webhook-delivery-and-retries, webhooks-overview, webhook-signatures, api-versioning, sdk-python.

n/a0.52Declined (gate)——1,998
Passo07
How do I pay suppliers in Brazil using PIX?

Answer: I don't have that in the Ledgerline docs. Retrieved documents, best first: multi-currency-ledgers, core-concepts, invoices-and-billing, counterparties-and-screening, sdk-python, limits-and-quotas, payment-rails, quickstart.

n/a0.23Declined (gate)——525
Passo08
Does Ledgerline comply with HIPAA for healthcare payments?

Answer: I don't have that in the Ledgerline docs. Retrieved documents, best first: invoices-and-billing, webhook-signatures, glossary, sdk-node, webhooks-overview, sdk-python, api-keys. Judge: Answer correctly states that the docs do not cover the question

n/a0.81Declined (model)5—1,803
Passo09
What is the best recipe for sourdough bread?

Answer: I don't have that in the Ledgerline docs. Retrieved documents, best first: quickstart, sdk-node, statements, going-live, reconciliation.

n/a0.04Declined (gate)——889
Passo10
Who won the football match last night?

Answer: I don't have that in the Ledgerline docs. Retrieved documents, best first: reconciliation, fx-quotes-and-conversions, returns-and-failed-payouts, payment-rails, error-format-and-codes.

n/a0.08Declined (gate)——688
Passo11
Can I embed the dashboard in my own app with a white-label theme?

Answer: I don't have that in the Ledgerline docs. Retrieved documents, best first: security-overview, going-live, restricted-keys-and-scopes, changelog, api-keys.

n/a0.13Declined (gate)——669
Passo12
How do I set up GraphQL subscriptions for ledger changes?

Answer: I don't have that in the Ledgerline docs. Retrieved documents, best first: quickstart, ledgers-and-accounts, changelog, api-versioning, core-concepts, data-residency-and-retention. Judge: Answer correctly states that the docs do not cover the question

n/a0.73Declined (model)5—1,812

History

1 run

Right document in the top 3
0.967

one run so far

Out-of-scope questions declined
100%

one run so far

Faithfulness, out of 5
4.97

one run so far

Evaluation runs, newest first. Select a column heading to sort.
2026-10-22 09:4293cbf2a-dirtyllama-3.3-70b-instruct-fp8-fast (workers-ai)0.9670.9034.97100%2,696

What this does not show

  • The question sets are small (30 and 12) and were written by the people who wrote the docs. One question moves hit@3 by 3 points.
  • The confidence threshold was fitted on these same questions, so the decline numbers are in-sample. They are an operating point, not a forecast.
  • A model judges faithfulness and correctness. It can be generous or wrong; the answers are in the table so you can check.
  • Latency includes the database and the model, measured from the build workspace, which is far from both. The deployed region will differ.