Accuracy study
The same tables the landing page shows, here as the primary reference. Every figure is regenerated from the committed measurement files by npm run results:export, so the document cannot drift from the run that produced it.
The study
What a 4B model on four CPU cores actually gets right
Twenty synthetic invoices and receipts, hand-authored ground truth, a schema-constrained prompt and temperature: 0. Every number below was measured on the hardware in the footnote and is regenerated from the committed run by npm run results:export. Nothing here is estimated, and the weakest field is shown rather than averaged away.
Local gemma3:4b, CPU only
| Field accuracy | 66.2% |
|---|---|
| Exact-record accuracyevery field on a document correct | 5.3% |
| p50 latency per document | 57.0s |
| p95 latency per document | 103.5s |
| Throughput | 60.7/hour |
| Schema violations reaching the table | 0 across 19 documents |
| Cost per 20 invoices | $0.00 |
What parameter count buys, on identical hardware
| Model | Field accuracy | p50 |
|---|---|---|
| gemma3:1b | 59.4% | 37.7s |
| gemma3:4b | 66.2% | 57.0s |
Four times the parameters bought 6.8 points of accuracy for 1.5× the latency. The 1B model is not merely worse, though: it filled 96.2% of the fields that had a correct answer, against 80.9% for the 4B — it fills in more, and gets more of what it fills in wrong. That is a real failure mode for an unattended pipeline, and it is why extraction here is review-first rather than autopilot.
Per-field accuracy, worst first
| Field | Accuracy | n | |
|---|---|---|---|
| Vendor | 94.7% | 19 | |
| Invoice no. | 68.4% | 19 | |
| Issue date | 89.5% | 19 | |
| Due date | 89.5% | 19 | |
| Currency | 78.9% | 19 | |
| Subtotal | 73.7% | 19 | |
| Tax | 100.0% | 19 | |
| Total | 84.2% | 19 | |
| Category | 15.8% | 19 | |
| Billable | 47.4% | 19 | |
| lineItems | 58.3% | 19 |
Category scores in the teens and it was predicted to: it is the one genuinely open-vocabulary field, and the scorer accepts only what the document permits. A closed category list would raise this number and make the field useless, which is not a trade worth making quietly. Tax is at the top because it is almost always printed explicitly, and getting a field wrong by returning null instead of a number is the single most expensive mistake in this table.
Against a hosted 31B, at the same $0.00
| Model | Field accuracy | Cost / 20 | Content leaves machine |
|---|---|---|---|
| gemma3:4b local | 66.2% | $0.00 | No |
| gemma-4-31b-it hosted | not run | $0.00 | Yes, to a data_collection: "deny" endpoint |
Not run. The free OpenRouter tier allows 50 requests per day and the local arms produce the headline numbers without touching that allowance. Two things stay true either way: the local arm makes zero external requests, and so does the hosted arm’s price. And the comparison would not be a clean ablation even if it were run — a hosted Gemma 4 against a local Gemma 3 confounds parameter count with model generation. That gap is disclosed rather than papered over with a number that would invite the wrong conclusion.
Measured on an i7-1165G7, 4 cores / 8 threads, 16 GB RAM, no GPU, Windows 11. Ollama gemma3:4b Q4_K_M. 20 fixtures, one ambiguous fixture excluded from headline figures and scored leniently. Ground truth authored before the first run and never edited to match model output.
Payment prediction
Which invoices to chase first
TabPFN-3.5 forecasts days-to-payment from invoice history, with a q10–q90 interval on every row. It is scored against two naive baselines on the same held-out invoices, because a forecast you cannot compare against this client’s own average is an assertion rather than a result.
TabPFN-3.5
7.91 days
mean absolute error
Client historical mean
8.12 days
the baseline that matters
Global mean
13.48 days
ignore who the client is
Paid invoices in history
84
Distinct clients
8
Features per row
12
TabPFN wins, by 3%. It is a real margin and it is also a narrow one — who pays late is mostly a property of who the client is, so a well-built per-client mean is a genuinely strong baseline rather than a straw man. The model’s contribution is in the residual: invoice size, timing seasonality and the client’s own variance. Its q10–q90 interval contained the true value in 12 of 17 held-out rows, against an 80% nominal target, so the intervals run a little narrow and are reported that way.