AI / ML
Evaluating LLM Extraction: Measure the Rows That Matter
By Vivek Jalondhara · 5 min read
Published
Build a useful evaluation set for price-sheet extraction, with row matching, field accuracy, abstentions, and document-level failure analysis.
An extraction model can return valid JSON while placing a price against the wrong product. The product question is more specific: did the import produce the right draft records, and can a reviewer find the remaining mistakes?
Consider an illustrative evaluation for Sheet Parser, a React and Node.js application that interprets vendor sheets through OpenRouter. The examples below use invented documents and counts. They describe an evaluation design, not measured accuracy or an existing private test dataset.
Define what one correct row means
Start with an annotation contract. A reference row might contain the document ID, source row, SKU, quantity basis, decimal price, and currency. Decide whether the amount is per item or per carton. A correct number with the wrong quantity basis is an incorrect usable price.
Allow an explicit unresolved label where the source is ambiguous. If a footer says that prices exclude an unspecified charge, the reference should preserve that uncertainty. Asking annotators to guess creates a benchmark that rewards confident invention. Record disagreements and settle the interpretation rule before scoring predictions.
Build a small set with different failure modes
Include sheets with repeated headers, blank prices, wrapped descriptions, multiple currencies, and discontinued items. Keep a clean ordinary sheet too. An evaluation consisting entirely of difficult edge cases cannot describe the typical workload, while only testing tidy tables hides the reasons the feature needs review.
Separate examples used to adjust prompts from examples reserved for evaluation. Group related sheets by vendor template when splitting them: nearly identical revisions on both sides can make generalization look stronger than it is. Scikit-learn's guidance on held-out evaluation and grouped data provides the underlying distinction; the vendor grouping is an application-specific choice.
Count omissions and invented rows separately
A hypothetical fixture contains ten product rows. The parser returns nine candidates: eight match, one mistakes a repeated header for a product, and two genuine rows are absent. Precision is eight divided by nine; recall is eight divided by ten. These expose different problems.
Match records with a declared rule, such as a server-assigned source location plus SKU, rather than array position. Conversely, deduplicating repeated predictions silently can hide a bug. Report duplicate candidates separately and decide how ambiguous matches are resolved before scoring.
Score fields and the review outcome
After matching rows, compare price, currency, and quantity basis separately. Normalize only differences the product considers equivalent: a decimal policy may treat 12.5 and 12.50 equally, but must not erase currency differences. Report an exact-row score requiring every required field to match as well as individual field results.
Track abstentions, source-reference validity, and the share of drafts needing correction. Missing a value and inventing one should remain distinguishable. A model's self-reported confidence is another prediction to evaluate, not a calibrated probability by default. Keep acceptance by a human separate from the original extraction result so corrections do not inflate the model's score.
Compare changes on the same documents
Version the fixtures, annotations, instructions, schema, and normalization rules. Record the requested and returned model identities. Compare prompts on the same reserved documents and inspect changed rows. An aggregate improvement can conceal a new currency error in a smaller group.
For variable outputs, repeat a bounded subset and report the observed spread with the sample size. Do not present one successful run as a reliability estimate. Keep malformed responses and failed requests in the report; excluding them answers how well completed extractions scored, not how often an import produced a usable draft.
Turn errors into an engineering decision
If errors cluster around wrapped descriptions, inspect document preprocessing before changing models. If prices shift by one row, inspect chunk boundaries and row matching. If a new template causes omissions, add a representative fixture after understanding the failure. The evaluation should point to a repairable component rather than just produce a leaderboard.
Detailed labels take time, so begin with a reviewed set covering the workflow's main risks and expand it when failures appear. Keep the reserved set useful by avoiding endless tuning against it. The tradeoff is deliberate: a smaller trustworthy evaluation can guide changes, but it cannot establish performance across every unseen vendor format.
Score matched source-row keys without hiding duplicates
This JavaScript helper expects canonical string keys produced by the fixture matcher. It reports duplicate candidates separately. Its null values mean a denominator is empty; the reporting layer must label that case instead of silently treating it as success.
Illustrative example · JavaScript · Simplified for explanation
function scoreRows(expectedKeys, predictedKeys) {
const expected = new Set(expectedKeys);
const predicted = new Set(predictedKeys);
if (expected.size !== expectedKeys.length) {
throw new Error("Reference keys must be unique");
}
const matched = [...predicted].filter(key => expected.has(key)).length;
return {
precision: predicted.size ? matched / predicted.size : null,
recall: expected.size ? matched / expected.size : null,
duplicates: predictedKeys.length - predicted.size,
missing: expected.size - matched,
unexpected: predicted.size - matched,
};
}- Annotate source rows
- Extract versioned drafts
- Match rows and score fields
- Inspect failures before accepting a change
Conceptual flow for the example above.
Tradeoff to keep in mind
Set-based row scores measure unique candidates and must be read alongside the duplicate count. They say nothing about field correctness; a matched row with the wrong currency still fails the field and exact-row checks.