AI / ML
Human Review for AI Data: Design the Acceptance Boundary
By Vivek Jalondhara · 5 min read
Published
Make extracted data reviewable with source evidence, field-level corrections, revision checks, and an explicit transition into accepted records.
Adding an Approve button does not make an AI workflow reviewable. A reviewer needs to know what changed, where each value came from, and whether the displayed draft is still the version being accepted. Those are data-model and interaction decisions as much as interface details.
Consider an illustrative Sheet Parser import containing a carton price mistaken for a unit price. The workflow below proposes how a React interface and Node.js service could support review. It is an educational extension of the project's vendor-pricing domain, not a claim about its private implementation.
Give drafts a separate lifecycle
Store extraction output as a draft associated with an import and source version. Keep pending, rejected, and accepted outcomes explicit. A valid decimal and a supported currency mean the candidate passed structural checks; they do not mean someone has checked the vendor's meaning or accepted the value into the catalog.
The catalog should read accepted records through a defined operation. A background re-extraction must not silently replace a previously accepted price. It can create a new candidate for comparison. This distinction also gives the interface honest labels: extracted, corrected, and accepted describe different events.
Show the evidence needed for one decision
In the carton example, display the source row, column labels, and nearby note identifying the pack size. Place that evidence beside the proposed quantity basis and price. Showing only a cropped number makes the wrong interpretation easy to confirm. Let the reviewer open more context without losing the current row.
Keep source text distinct from explanations generated by the model. Highlight differences between an existing catalog record and the candidate, while making missing values visible. Do not prefill an unknown currency merely to satisfy a required field. Explain the unresolved fact and offer a correction or rejection path.
Make corrections specific and accessible
A reviewer may accept the SKU but correct the quantity basis. Preserve the original candidate and the resulting reviewed values, rather than overwriting the evidence during editing. Record the actor and time when accepting the corrected revision. This lets a later investigation distinguish extraction errors from review decisions.
Associate field errors with their controls and provide a clear outcome after submission. W3C's form-notification guidance supports both overall feedback and specific errors. In a long review table, a summary linking to unresolved rows helps keyboard users reach the problem. Color alone should not carry the difference between accepted and unresolved data.
Accept the revision the reviewer actually saw
Suppose two reviewers open revision four. One corrects the quantity basis, creating revision five. The other's approval of revision four must not approve revision five unseen. Send the expected revision with the acceptance request and condition the database update on both that revision and the pending state.
Perform the successful state transition, catalog write, and acceptance audit entry in one transaction. Check the current user's permissions before the write. A zero-row conditional update means the operation did not claim the expected draft; return a conflict or the recorded outcome of an identified retry, rather than showing a fresh success.
Keep bulk actions honest
Bulk acceptance is useful when many rows have already been checked, but selection needs a clear scope. State whether the action covers the visible page, explicitly selected rows, or every filtered result. Bind the operation to specific row revisions so a background refresh cannot add unseen candidates to the accepted batch.
Choose whether a batch is atomic or allows partial success, then make the response match that choice. A partial result should identify accepted rows, conflicts, and unresolved rows. A generic success toast can otherwise leave the reviewer believing that every selected price reached the catalog.
Measure review without rewarding rubber-stamping
Track correction categories, unresolved rows, stale-approval conflicts, and sampled agreement with source documents. Review duration can reveal friction, but speed alone cannot establish accuracy. If one template consistently causes carton mistakes, improve extraction or the review context instead of asking reviewers to click faster.
Human review adds effort and can still miss mistakes, especially when a screen encourages repetitive confirmation. Prioritize the information necessary for a decision and use representative quality checks. The tradeoff is explicit: an acceptance boundary improves accountability and recoverability, but does not guarantee every accepted value is correct.
Claim only the draft revision that was reviewed
This SQL fragment assumes a draft table with a monotonically increasing revision. The server supplies the authorized workspace and reviewer IDs. Continue the catalog write and audit insertion in the same transaction only when one draft is returned.
Illustrative example · SQL · Simplified for explanation
UPDATE price_drafts
SET review_state = 'accepted',
accepted_by = $4,
accepted_at = CURRENT_TIMESTAMP,
revision = revision + 1
WHERE id = $1
AND workspace_id = $2
AND revision = $3
AND review_state = 'pending'
RETURNING id, revision, sku, unit_price, currency;- Display source and draft revision
- Review or correct fields
- Authorize and condition acceptance on revision
- Commit catalog record and audit together
Conceptual flow for the example above.
Tradeoff to keep in mind
The conditional update prevents a stale revision from being claimed if every edit advances revision. It does not implement authorization or the surrounding transaction, and catalog-level conflicts need their own policy when multiple imports target the same product.