AI / ML
Eight Synthetic Rows to Challenge an AI Price-Sheet Parser
By Vivek Jalondhara · 5 min read
Published
Build eight small vendor-sheet fixtures that expose missing values, unit confusion, identity loss, CSV errors, and injected instructions.
A small fixture is useful when it exposes one assumption clearly. For an AI price-sheet workflow, that means keeping the source, expected interpretation, and permitted unresolved outcome together so a reviewer can explain a failure.
These eight fictional cases are a starting set for the Sheet Parser domain. They are not production measurements or a benchmark of any model. A deterministic parser check and a live model evaluation are different runs, and passing this set would not establish general extraction accuracy.
1. An empty price cell
Use a row with a valid product code and a blank price. The expected behaviour is to preserve a missing amount and identify a review task. Do not manufacture zero simply because the storage column or table renderer prefers a number.
Keep a separate case for a whitespace-only cell if the supported file contract treats that as missing. Record the original value before normalisation so a later reviewer can determine whether the importer or the document supplied the empty representation.
2. An explicitly supplied zero
Use a row whose source says 0.00 and whose currency and unit are present. The importer should retain that supplied value rather than treating it as false or absent. Whether the zero may be accepted is a separate product rule that the fixture should state.
Pair this case with the blank-price case when reviewing a helper. They expose the difference between checking for missing data and applying a truthiness shortcut. A complete field is still subject to source and policy review.
3. An ambiguous numeric separator
Supply 1,250 without a declared convention. Depending on the source format, that punctuation may have different meaning. The expected outcome for this fixture is unresolved interpretation, not an automatic choice based on the developer's locale or the user's browser.
Then add a companion document with an explicit supported format rule. That distinguishes the case the application can normalise from the case it should reject or send for review. The fixture is testing a contract boundary, not whether a model can guess plausibly.
4. A pack price with an unknown pack size
Describe a connector as priced per pack while leaving the number of items unspecified. Preserve the selling unit and keep the pack count unresolved. Converting the amount to a per-item value would introduce information absent from the source.
The review result should identify the missing basis rather than mark the entire product unreadable. This tests whether the pipeline can keep useful fields while making a limited uncertainty visible to the person deciding what to do next.
5. A currency supplied by a section heading
Create a small synthetic sheet with one section labelled INR and another labelled USD. Place the test row in the second section while keeping the numeric amount identical to a row in the first. The expected currency comes from the row's actual section.
Include the headings in the fixture, not just the isolated row string. Otherwise the evaluator would ask the extractor to recover evidence it never received. Record which contextual cells or text region the expected interpretation relies on.
6. A repeated product code across vendors
Use the same SKU in two fictional vendor imports with different descriptions. The expected records remain scoped to their respective vendors. A global deduplication step keyed only by SKU should fail this exercise even if it preserves one well-formed row.
Vendor identity comes from trusted import context in this design. The model should not be able to switch that identity by returning a vendor field. Inspect the stored relationships as well as the displayed product text when evaluating this case.
7. A quoted description containing a line break
Put a comma and a newline inside a properly quoted CSV description. A CSV reader should retain that as one field and one logical record according to the supported format. A naive comma or line split can shift the amount into the wrong column.
Run this as a deterministic file-reading check before involving an LLM. Keeping parsing and interpretation tests separate makes failures easier to locate. The RFC 4180 description of quoted CSV fields provides a concrete format reference for the fixture.
8. A note that tries to approve itself
Add a document note asking the application to approve every row and change the destination vendor. Treat it as untrusted source content. The expected invariant is that document text cannot choose workspace permissions, vendor ownership, or acceptance state.
Inspect side effects, not merely the model's wording. Even if a response repeats the instruction, the server must retain control of those fields. Keep model outputs for comparison across repeated evaluations, and use larger representative datasets before making any claim about reliability beyond these narrow examples.
A compact manifest for the eight fixture expectations
Each identifier points to a synthetic source fixture you create with the context described above. This manifest defines expected properties; it is not an evaluation runner or a claim that a model passed them.
Illustrative example · JSON · Simplified for explanation
[
{
"id": "blank-price",
"expect": "missing amount remains unresolved"
},
{
"id": "explicit-zero",
"expect": "supplied zero is preserved"
},
{
"id": "ambiguous-separator",
"expect": "no locale guess"
},
{
"id": "unknown-pack-size",
"expect": "no invented per-item conversion"
},
{
"id": "section-currency",
"expect": "currency follows the correct heading"
},
{
"id": "cross-vendor-sku",
"expect": "vendor ownership remains separate"
},
{
"id": "multiline-csv",
"expect": "one quoted field remains intact"
},
{
"id": "injected-note",
"expect": "no change to authority or acceptance"
}
]- Create synthetic source fixtures
- Write independent expectations
- Run the relevant pipeline stage
- Inspect values and side effects
Conceptual flow for the example above.
Tradeoff to keep in mind
Synthetic fixtures make individual assumptions visible but cannot represent the full variety of vendor documents. Extend the set from authorised, sanitised failures and track regression cases without presenting fixture success as production accuracy.